Venue: NAACL 2024

Exploring the Impact of Table-to-Text Methods on Augmenting LLM-based Question Answering with Domain Hybrid Data

Dehai Min\({}^{1,4}\) Nan Hu\({}^{1,4}\) Rihui Jin\({}^{1,4}\) Nuo Lin\({}^{1}\) Jiaoyan Chen\({}^{2}\) Yongrui Chen\({}^{1,4}\) Yu Li\({}^{1,4}\) Guilin Qi\({}^{1,4}\) Yun Li\({}^{3}\) Nijun Li\({}^{3}\) Qianren Wang\({}^{3}\)

###### Abstract

Augmenting Large Language Models (LLMs) for Question Answering (QA) with domain specific data has attracted wide attention. However, domain data often exists in a hybrid format, including text and semi-structured tables, posing challenges for the seamless integration of information. Table-to-Text Generation is a promising solution by facilitating the transformation of hybrid data into a uniformly text-formatted corpus. Although this technique has been widely studied by the NLP community, there is currently no comparative analysis on how corpora generated by different table-to-text methods affect the performance of QA systems. In this paper, we address this research gap in two steps. First, we innovatively integrate table-to-text generation into the framework of enhancing LLM-based QA systems with domain hybrid data. Then, we utilize this framework in real-world industrial data to conduct extensive experiments on two types of QA systems (DSFT and RAG frameworks) with four representative methods: Markdown format, Template serialization, TPLM-based method, and LLM-based method. Based on the experimental results, we draw some empirical findings and explore the underlying reasons behind the success of some methods. We hope the findings of this work will provide a valuable reference for the academic and industrial communities in developing robust QA systems.

+
Footnote †: Corresponding author.

**Paperable title=Permanent title=Permanent title=Permanent title=Permanent title=Permanent title=Permanent title=Permanent title=Permanent title=Permanent title=Permanent title=Permanent title=Permanent title=Permanent title=Permanent title=Permanent title=Permanent title=Permanent title=Permanence title=Perman title=Permanent title=Perman title=Permanent title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Perman title=Permanperman title=Permanperman title=ermanpermanp title=ermanpermanp title=ermanpermanp title=ermanp title=ermanpermanp title=ermanp title=ermanp title=ermanpermanp title=ermanpermanp title=ermanpermanp title=ermanpermanp title=ermanpermanp title=ermanpermanp title=ermanpermanp title=ermanpermanp title=ermanpermanp title=ermanpermanp title=ermanp title=ermanpermanp title=ermanp title=ermanpermanp title=ermanp title=ermanpermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp title=ermanp item=ermanp title=ermanp title=ermanpname=ermanp title=ermanp title=ermanp title=ermanpermanp title=ermanp title=ermanpermanp title=ermanp title=ermanperman title=ermanp title=ermanpermanp title=ermanp title=ermanpermanp title=ermanp title=ermanpermanp title=ermanp title=ermanpermanp title=ermanperman title=ermanp title=ermanpermanp title=ermanperman title= 

though table-to-text generation has been widely studied by the NLP community, there is currently no comparative analysis on how corpora generated by different table-to-text methods affect the performance of domain-specific QA systems.

In this work, we address this research gap by two steps. First, we innovatively integrate table-to-text generation into the framework of enhancing LLM-based QA systems with domain hybrid data. Then, we utilize this framework to conduct extensive experiments on two types of QA systems (DSFT and RAG paradigms) with four representative table-to-text methods. We choose the following four strategies: 1) **Markdown** format; 2) **Template** serialization; 3) **TPLM-based** method; 4) **LLM-based** method. These strategies differ in complexity and underlying technology. The Markdown and Template serialization offer simplicity, while the TPLM-based and LLM-based methods leverage the capabilities of advanced language models to generate more nuanced text.

In terms of implementation, we collect a real-world hybrid dataset called ICT-DATA, by extracting text and tables from numerous documents about Information and Communication Technology (ICT) products. It is important to note that the text contained in tables accounts for approximately 18% of the total content in ICT-DATA (based on word count statistics). We employ different table-to-text methods to process the tables in ICT-DATA, obtaining different ICT corpora. These corpora are then utilized to build QA systems. Moreover, we create a benchmark dataset named ICTQA, which consists of QA pairs based on the knowledge of ICT-DATA. This dataset is particularly suitable for evaluating enhanced LLMs, as it includes some industry-specific knowledge not covered in the general LLMs training stage.

To our knowledge, our research is the first to comprehensively compare different table-to-text strategies on LLM-based QA systems enhanced by domain hybrid data. Our main findings are as follows:

* Table-to-text methods significantly impact the performance of QA systems, with relative score differences ranging from 2.8% to 9.0% in human evaluation and 4.8% to 16% in GPT-4 evaluation. In two systems, selecting the appropriate method can yield considerable benefits.
* In the DSFT paradigm, LLM-based and TPLM-based consistently outperform others across various model settings, demonstrating their superiority. In the RAG paradigm, while the LLM-based method still performs excellently, the Markdown has shown unexpected effectiveness.
* The varying frequency of domain-specific terms and verbs produced by these methods, alongside the differing quality of semantic representations in the generated text chunks, which appear to be pivotal factors influencing performance disparities across the two systems.

## 2 Table-to-Text Generation

Table-to-text generation Parikh et al. (2020); Chen et al. (2020); Cheng et al. (2022) aims to create natural language descriptions from semi-structured tabular data, such as web tables. As shown in Figure 1, we apply four representative table-to-text methods to textualize the tables in ICT-DATA, forming four different corpora. Formally: Let \(F_{i}:\text{Table}\rightarrow\text{Text}\) represent four table-to-text functions for \(i=1,2,3,4\). With the original ICT-DATA \(D=\{\text{Tab},\text{Text}\}\), each \(F_{i}\) converts tables Tab into text. The resulting ICT Corpora \(C_{i}\) are formed by combining these texts with Text:

\[C_{i}=F_{i}(\text{Tab})\cup\text{Text},\quad i=1,2,3,4\]

We next provide a detailed introduction of these four methods. Table 1 provides a comparative analysis of these methods in terms of their resource requirements, processing speeds, and text diversity.

* **Markdown** format: A straightforward method to represent tables in Markdown format. It does not involve model training and can be rapidly processed via scripts without manual intervention.
* **Template** serialization: This method uses a set of templates designed based on table features for textualization Li et al. (2023); Ye et al. (2019). It achieves slightly higher diversity in the generated text compared to the Markdown method, attributed to the use of multiple pre-prepared templates to accommodate different types of tables, which requires some manual involvement.
* **TPLM-based** method: This method involves fine-tuning Traditional Pre-trained Language Models (TPLMs), such as T5 Raffel et al. (2020)

\begin{table}
\begin{tabular}{l|c c c} \hline \hline
**Method** & **Resource** & **Speed** & **Diversity** \\ \hline
**Markdown** & CPU & Fast & Low \\
**Template** & CPU & Fast & Moderate \\
**TPLM-based** & GPU & Moderate & High \\
**LLM-based** & GPU or API & Low & Very High \\ \hline \hline \end{tabular}
\end{table}
Table 1: Comparison of table-to-text methods: resource usage, generation speed and diversity of generated text.

 

and BART (Lewis et al., 2020), on specific table-to-text generation task datasets Liu et al. (2022). In this paper, we utilize the MVP model Tang et al. (2023), which initially pre-trains the BART model on numerous natural language generation datasets, followed by fine-tuning on various cross-domain table-to-text datasets. It allows customized adjustment of the output through fine-tuning, offering higher flexibility and domain adaptability, while requiring more computational resources.
* **LLM-based** method: Recent endeavors employing LLMs for this task have drawn significant attention Bian et al. (2023). Impressively, Zhao et al. (2023) demonstrate that GPT-* models often outperform the best-performing fine-tuned models. We refer to their findings and utilize ChatGPT in a one-shot setting in our work. Similar to TPLM-based methods, this approach can be custom-tailored using In-Context Learning. Moreover, using the APIs of certain proprietary LLMs might pose risks of domain data leakage. Some examples of table-to-text, along with the specific templates and prompts for ChatGPT used in this paper, can be found in Appendix B.

## 3 Building LLM-based QA Systems with Domain Corpora

We will introduce separately how two LLM-based QA systems utilize these corpora. Their framework overview can be viewed in Figure 2.

**Domain-Specific Fine-Tuning.** In this approach, we first pre-train the LLM on the ICT corpus using next-token prediction Radford et al. (2018), enabling the model to incrementally learn domain knowledge. Subsequently, we adapt the model to the QA task through instruction tuning Ouyang et al. (2022). Formally, an original LLM \(M\), is pre-trained on each ICT Corpus \(C_{i}\), to obtain an updated foundation model \(M^{\prime}_{i}\):

\[M^{\prime}_{i}=\textrm{Pre-Train}(M,C_{i}),\quad i=1,2,3,4\]

The updated models are then further trained on the same instruction set \(I\) tailored for the QA task, resulting in the final QA oriented models \(M^{QA}_{i}\):

\[M^{QA}_{i}=\textrm{FineTune}(M^{\prime}_{i},I),\quad i=1,2,3,4\]

**Retrieval-Augmented Generation.** In this paradigm, we adopt the framework proposed by LangChain Chase (2022) with the Dense Passage Retriever (DPR) method Karpukhin et al. (2020), which consists of a multi-step process: 1) Splitting the large-sized Corpus \(C_{i}\) into smaller chunks \(\{p_{j}\}^{C_{i}}\); 2) Encoding each text chunk \(p_{j}\) into a d-dimensional vector by an encoder \(E_{P}(\cdot)\), which

Figure 1: Illustration of four domain corpora generation process. Different table-to-text methods are applied to tables of domain documents, generating different text. These generated texts are then merged with the original document texts, yielding different domain corpora.

Figure 2: Framework of domain-enhanced QA systems.

 

captures its semantic essence; 3) Building an indexed Vector Store for these vectors, optimizing the storage for efficient retrieval; 4) For each query \(Q\), retrieving the \(K\) most relevant text chunks, \(P=\{p_{k}\}^{K}_{k=1}\); 5) Using both the query \(Q\) and the retrieved prompts \(P\) to generate the final answer with the LLM.

## 4 Dataset and Evaluation Metrics

### Evaluation Dataset

**ICT-DATA.** We collect ICT-DATA based on 170 English technical documents related to ICT products. Each product document consists of tables and text, whose contents include product descriptions, configuration guides, terms, and definitions, etc. The total storage size is approximately 6GB. Moreover, the number of words in the table data accounts for about 18% of the total number of words in the dataset. In Appendix A.2, we provide detailed statistics and the preprocessing methods used for the table data.

**ICTQA.** We create the ICTQA dataset to evaluate the performance of domain QA systems, by collecting 9,000 questions with long-form answers from the actual ICT product technical support QA platform. All the answers are written by experts based on product documents. We manually select 500 questions as the test set, whose answers involve knowledge from both tables and text. The remaining QA pairs are used as the training set for the instruction fine-tuning phase in the DSFT paradigm. We show statistics and some examples in Appendix A.1.

### Evaluation Metrics

To evaluate the model's responses, we employ both automated and manual evaluation methods.

**Automated Evaluation Metrics.** Given that traditional lexical-overlap-based metrics (such as BLEU and ROUGE) are inadequate for evaluating the quality of long-form answers generated by LLMs Krishna et al. (2021); Kamalloo et al. (2023), we use GPT-4 as an evaluator with a demonstration setting, scoring responses based on their similarity to the golden answer Liu et al. (2023). The score ranges from 0 to 5 with discrete values; 0 indicates incoherent answers with repeated fields or responses like "I don't know the answer", 1 represents minimal similarity to the golden answer, and 5 denotes an accurate answer.

**Human Evaluation.** Given the limitations in evaluating long-form answers using existing automated metrics Wang et al. (2023b); Kamalloo et al. (2023), three evaluators with domain knowledge are asked to score responses based on the helpfulness and similarity to the golden answer, using the same scoring criteria with a range of 0 to 5 as the GPT-4 evaluator.

For fairness and to eliminate potential bias, responses are presented anonymously to both the GPT-4 and human evaluators. The full prompt, evaluation setup for human and scoring criteria are detailed in Appendix D.

## 5 Experimental Setup

**QA Systems of the DSFT Paradigm.** Within the DSFT paradigm, we utilize Meta's OPT (1.3B to 13B) Zhang et al. (2022) and Llama2-base (7B, 13B) Touvron et al. (2023) as foundation models. The OPT models offer variable sizes to enhance robustness. To mitigate training costs, we employ the QLoRA Dettmers et al. (2023) strategy for pretraining and instruction fine-tuning. The instruction template can be found in Appendix A.3.

**QA Systems of the RAG Paradigm.** We use the Llama2-chat models (7B, 13B, and 70B) and GPT-3.5-turbo for inference. We divide the corpus into smaller chunks, ensuring the integrity of sentences and keeping their lengths below 3000 characters. Subsequently, text chunks are vectorized using the BGE embedding model Zhang et al. (2023). We utilize the FAISS library Johnson et al. (2021) to retrieve the vectors of the top-3 relevant text chunks based on similarity. These chunks are input to the LLM with the corresponding questions for answering through the RAG-Chain from LangChain Chase (2022).

**Fair Comparison.** To maintain consistency and control variables, all models are trained or used under the same settings on four different corpora. Detailed training parameters and GPU costs are available in Appendix C.

## 6 Results

In the following subsections, we will discuss three research questions regarding our study.

### RQ1: How do these methods affect the performance of QA systems?

Table 2 shows the average scores for different QA system setups on the ICTQA test set. We can see



that there are significant differences in the performance of the two types of QA systems enhanced by corpora generated from different table-to-text methods. Their Relative Score Differences range from 2.8% to 9.0% in human evaluation and from 4.8% to 16% in GPT4 evaluation. For a more detailed observation, we present the score distribution from human evaluation of the DSFT QA models based on OPT-6.7B in Figure 3. From this figure, we can observe significant differences in score distribution among different QA models, reflecting their performance variations. From Table 2, we note that in the DSFT paradigm, both TPLM-based and LLM-based methods, which utilize language models for table-to-text generation, perform well across different models. Particularly, the LLM-based method shows the best performance in many models. On the other hand, the RAG paradigm provides a different observation. While the LLM-based method continues to exhibit excellent performance, the Markdown format shows a significant and unexpected improved performance in the RAG paradigm compared to DSFT, even best-performing in some models. To further illustrate these findings, we show the competition results of some QA system scores in Figure 4. We can clearly observe that the methods with higher average scores also have a higher probability of achieving better scores for each question. These observations underscore the necessity of choosing the appropriate method for processing table data when building domain-specific QA systems.

### RQ2: What are the potential reasons for their different performances?

Since DSFT and RAG systems utilize domain corpora in different ways, we will discuss them separately in this section.

**For the DSFT paradigm.** Inspired by the findings of Biderman et al. (2023); Razeghi et al. (2022); Elazar et al. (2023), which suggest a correlation and causal relationship between the ability of LLMs to answer factual questions and the frequency of salient entities found in their pre-training corpora, we also observe that different table-to-text methods have inconsistent preferences for domain verbs when describing tables. Following the approach of Zevallos et al. (2023); Wang et al. (2023c), we extract domain term sets and related verb sets from the QA pairs in the ICTQA test set. We then calculate the absolute frequency of these terms and verbs as they appear in the corpora generated by different table-to-text methods. In Table 3, we can clearly see significant differences in these frequencies across different corpora. For example, LLM-based methods show a term frequency more than twice that of Template methods, with verb frequency quadrupling. This is because LLM-based methods tend to supplement the subject with the

\begin{table}
\begin{tabular}{l|p{56.9pt}|p{28.5pt} p{28.5pt} p{28.5pt} p{28.5pt} p{28.5pt} p{28.5pt} p{28.5pt} p{28.5pt} p{28.5pt}} \hline \multirow{2}{*}{**Metrics**} & \multicolumn{2}{c}{**Table-to-Text**} & \multicolumn{2}{c}{**Domain Specific Time-Timing**} & \multicolumn{2}{c}{**Retrieval-Augmented Generation**} \\  & **Method** & **OPT-1.38** & **OPT-2.78** & **OPT-4.78** & **OPT-1.33** & **Llaman-2.78** & **Linear-1.30** & **GPT-3.5 turbo** & **Linear-2.78** & **Linear-1.30** & **Islama-2.70b** \\ \cline{2-13} \multirow{5}{*}{**Enum**} & **Markdown** & 2.05 & 2.41 & 2.38 & 2.51 & 2.82 & 3.05 & 3.29 & **3.72** & 3.98 & 3.94 \\  & **Template** & 2.04 & 2.40 & 2.26 & 2.47 & 2.82 & 3.04 & 3.26 & 3.44 & 3.96 & 3.76 \\  & **TPLM-based** & 2.12 & **2.43** & 2.43 & 2.58 & **3.30** & 3.13 & 3.26 & 3.27 & 3.92 & 3.64 \\  & **LLM-based** & **2.18** & **2.57** & **2.51** & **2.62** & 2.96 & **3.19** & **3.62** & 3.71 & **4.26** & **4.09** \\ \cline{2-13} \multirow{5}{*}{**Evid.**} & **RSD(v)** & 2.80 & 3.40 & 5.00 & 3.00 & 7.60 & 3.00 & 7.20 & 9.00 & 6.80 & 9.00 \\ \cline{2-13}  & **Markdown** & 1.74 & 2.16 & 2.27 & 2.25 & 2.77 & 3.06 & 3.28 & **3.46** & 3.67 & **3.74** \\  & **Template** & 1.81 & 2.22 & 2.39 & 2.34 & 2.84 & 3.08 & 3.27 & 3.06 & 3.38 & 3.37 \\  & **TPLM-based** & 2.33 & 2.46 & 2.45 & 2.53 & **3.29** & 3.19 & 3.28 & 2.9 & 3.41 & 3.30 \\  & **LRM-based** & **2.57** & **2.69** & **2.73** & **2.86** & 3.06 & **3.30** & **3.64** & 3.59 & **3.69** & 3.54 \\ \cline{2-13}  & **RSD(v)** & 16.60 & 10.60 & 9.20 & 12.20 & 10.00 & 4.80 & 7.40 & 15.20 & 6.20 & 8.80 \\ \hline \end{tabular}
\end{table}
Table 2: The average scores from Human Evaluation and GPT-4 Evaluation of the QA systems with four representative table-to-text methods. In each setting, the best result is shown in bold, and the second-best result is underlined. Relative Score Difference (RSD) is calculated using the formula \((\text{Highest Score}-\text{Lowest Score})/5\).

Figure 3: The scores distribution from human evaluation for the DSFT QA systems based on OPT-6.7B.

 

domain entity corresponding to the attribute when describing tables, and exhibits greater diversity in verbs. In contrast, Template methods use more pronouns, such as 'it', and monotonous predicates (usually 'be' verbs). By comparing these frequency rankings with the performance shown in Table 2, we can observe a positive correlation between them: methods with higher frequencies, especially the TPLM and LLM-based methods, correspond to superior QA capabilities in the DSFT systems.

**For the RAG paradigm.** Under the same LLM reader setup, retrieval accuracy in this semantic space crucially impacts RAG performance Ma et al. (2023). The retrieval process involves selecting the vectorized chunks with the highest similarity scores to the query vector. To investigate the impact of different methods on retrieval effectiveness, we use t-SNE Van der Maaten and Hinton (2008) to visualize the clustering of a query and related chunks in the semantic space at Figure 5. It could be clearly seen that chunks generated by the LLM-based and Markdown methods, which perform well in Table 2, are closer to the query in the semantic space. This makes the chunks related to the query more likely to be retrieved, thereby improving the system's performance. This suggests that in the RAG framework with the DPR method, the texts generated by these methods have more retrieval-friendly semantic representations and better alignment between queries and documents.

### RQ3: Are there practical suggestions for choosing table-to-text methods?

Through the analysis of RQ1 and RQ2, we know that the LLM-based strategy with ChatGPT is outstanding and reliable in both frameworks. In case its drawbacks mentioned in Section 2 are unacceptable, the TPLM-based strategy (i.e., selecting a well-tuned table-to-text model) is a good alternative in the DSFT paradigm. In the RAG paradigm, the simple and easy-to-use Markdown strategy is also a viable substitute. Additionally, although RAG systems using these four methods significantly outperform DSFT systems in terms of performance, building a vector retrieval library demands substantial memory resources. Therefore, referring to Table 4, choosing methods that generate more concise texts, such as LLM-based and Markdown strategies, is a wise decision.

### Additional discussion on experimental results

As shown in Table 2, under the ICT dataset and the experimental setup of this study, the RAG method outperforms the DSFT method in Llama2 models. This demonstrates that RAG has an excellent performance as a lower cost method. We attribute this result to two main reasons: 1). The ICT data used in this study covers dense domain knowledge, and it is still challenging to adapt the LLM well to this complex domain data through incremental pre-training. 2). As the statistical analysis in Ap

\begin{table}
\begin{tabular}{l|c c c c} \hline \hline
**Freq (Avg.)** & **Markdown** & **Template** & **TPLM-based** & **LLM-based** \\ \hline
**Text Len** & 998 & 1259 & 1138 & 897 \\ \hline \end{tabular}
\end{table}
Table 4: The average length of text generated by different methods for each table.

Figure 4: Comparison of human evaluation scores between QA models using different Table-to-Text methods. ‘A vs. B win’ indicates the percentage of test set instances where Model A’s score surpasses Model B’s.

Figure 5: A t-SNE visualization of chunk clusters in the embedding space of the RAG system. ’X Chunks’ represents chunks related to the query (red star) from the corpus generated by X table-to-text method.

 

pendix A.1, most of the questions in the ICTQA are quizzes on the knowledge of product manuals. In this scenario, the existing excellent dense vector retrievers have high recall accuracy. The studies of (Gupta et al., 2024) and (Soudani et al., 2024) have respectively conducted detailed experiments on the choice between Fine-Tuning and RAG under the agricultural domain data and Less Popular Knowledge scenarios. Our experimental results in this work further validate their viewpoints. It is also worth noting that in this study, the bge-large-en embedding model (Zhang et al., 2023) embeds text chunks into 1024-dimensional vectors. During the retrieval of relevant chunks based on the questions, the peak running memory requirement is approximately 280G.

Another interesting experimental result is that GPT-3.5-turbo performs worse than the Llama2 family in the RAG paradigm. We manually observe the QA cases and find that GPT-3.5-turbo has a significantly higher probability of outputting "I don't know the answer.", even if the retriever finds text chunks containing the correct answer.

## 7 Related Work

### Domain Augmented Large Language Models.

In order to enhance the capabilities of LLMs in domain-specific tasks, some works develop LLMs through incremental training on an extensive domain corpus, inheriting the benefits of both the emergent abilities of LLMs and domain-specific knowledge (Luo et al., 2023; Huang et al., 2023). This technology yields significant results, but it demands substantial computational resources and incurs high costs (Wang et al., 2023). In order to overcome this difficulty, a prompt-based solution that does not require updating model parameters has been proposed. They retrieve relevant domain information from external knowledge bases before answering questions with LLMs (Gao et al., 2023; Wang et al., 2023; Xu et al., 2023).

### Question Answering over Hybrid Data

Some works study QA tasks on hybrid data that contain both tables and text Zhu et al. (2021); Chen et al. (2020, 2020). Popular approaches often involve designing a complex system that has independent modules to process text and tables separately. The information from these two modules is then merged and fed into a language model to generate answers (Zhong et al., 2022). Additionally, some of these methods not only require annotations of metadata identifying text and tables relevant to the question, but they also rely on the formulation of executable languages to access tables, such as SQL or SPARQL (Nan et al., 2022; Li et al., 2021). These executable languages often have strict assumptions about the structure of the tables. These limitations make these approaches ill-suited for the real-world LLM-based scenario domain QA systems. Therefore, the results of this study were not compared with these baseline models in the experiments.

## 8 Conclusion

This paper studies the impact of different table-to-text methods on LLM-based QA systems enhanced by domain hybrid data. Specifically, we meticulously compared four representative methods: Markdown formatting, Template serialization, TPLM-based, and LLM-based approaches. Through experiments, we show the superiority of the LLM-based and TPLM-based methods in the DSFT framework, and the excellence of the LLM-based and Markdown methods in the RAG framework. A key discovery is the varying frequency of domain-specific terms and verbs produced by these methods, alongside the differing quality of semantic representations in the generated text chunks, which appear to be pivotal factors influencing performance disparities across the two systems. These insights not only shed light on the nuances of table-to-text generation methods but also have profound implications for the enhancement of LLMs. Furthermore, they offer practical guidance for tailoring domain-specific QA systems to meet particular needs.

## Acknowledgements

This work is partially supported by National Nature Science Foundation of China under No. U21A20488. We thank the Big Data Computing Center of Southeast University for providing the facility support on the numerical calculations in this paper 



## References

* Bian et al. (2023) Junyi Bian, Xiaolei Qin, Wuhe Zou, Mengzuo Huang, and Weidong Zhang. 2023. Hellama: Llama-based table to text generation by highlighting the important evidence.
* Biderman et al. (2023) Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Halahan, Mohammad Afah Khan, Shivanshu Purohit, Usvson Sai Prasanthh, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar Van Der Wal. 2023. Pythia: A suite for analyzing large language models across training and scaling. In _Proceedings of the 40th International Conference on Machine Learning_, volume 202 of _Proceedings of Machine Learning Research_, pages 2397-2430. PMLR.
* Chase (2022) Harrison Chase. 2022. Langchain.
* Chen et al. (2020a) Wenhu Chen, Ming-Wei Chang, Eva Schlinger, William Yang Wang, and William W Cohen. 2020a. Open question answering over tables and text. In _International Conference on Learning Representations_.
* Chen et al. (2020b) Wenhu Chen, Jianshu Chen, Yu Su, Zhiyu Chen, and William Yang Wang. 2020b. Logical Natural Language Generation from Open-Domain Tables. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 7929-7942, Online. Association for Computational Linguistics.
* Chen et al. (2020c) Wenhu Chen, Hanwen Zha, Zhiyu Chen, Wenhan Xiong, Hong Wang, and William Yang Wang. 2020c. Hybridqa: A dataset of multi-hop question answering over tabular and textual data. In _Findings of the Association for Computational Linguistics: EMNLP 2020_, pages 1026-1036.
* Cheng et al. (2022) Zhoujun Cheng, Haoyu Dong, Zhiruo Wang, Ran Jia, Jiaqi Guo, Yan Gao, Shi Han, Jian-Guang Lou, and Dongmei Zhang. 2022. HiTab: A hierarchical table dataset for question answering and natural language generation. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1094-1110, Dublin, Ireland. Association for Computational Linguistics.
* Dettmers et al. (2023) Tim Dettmers, Artidoto Papnoini, Ari Holtzman, and Luke Zettlemoyer. 2023. QLoRA: Efficient Finetuning of Quantized LLMs. ArXiv:2305.14314 [cs].
* Elazar et al. (2023) Yanai Elazar, Nora Kassner, Shauli Ravfogel, Amir Feder, Abhilasha Ravichander, Marius Mosbach, Yonatan Belinkov, Hinrich Schutze, and Yoav Goldberg. 2023. Measuring Causal Effects of Data Statistics on Language Model's 'Factual' Predictions. ArXiv:2207.14251 [cs].
* Gao et al. (2023) Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. 2023. Retrieval-augmented generation for large language models: A survey.
* Gupta et al. (2024) Aman Gupta, Anup Shirgaonkar, Angels de Luis Balaguer, Bruno Silva, Daniel Holstein, Dawei Li, Jennifer Marsman, Leonardo O Nunes, Mahsa Rouzabman, Morris Sharp, et al. 2024. Rag vs fine-tuning: Pipelines, tradeoffs, and a case study on agriculture. _arXiv preprint arXiv:2401.08406_.
* Gururangan et al. (2020) Suchin Gururangan, Ana Marasovic, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don't Stop Pretraining: Adapt Language Models to Domains and Tasks. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 8342-8360, Online. Association for Computational Linguistics.
* Huang et al. (2023) Huang Huang, Fei Yu, Jianqing Zhu, Xuening Sun, Hao Cheng, Dingjie Song, Zhihong Chen, Abdulmohsen Alharthi, Bang An, Juncai He, Ziche Liu, Zhiyi Zhang, Junying Chen, Jianquan Li, Benyou Wang, Lian Zhang, Ruoyu Sun, Xiang Wan, Haizhou Li, and Jinchao Xu. 2023. Acegpt, localizing large language models in arabic.
* Huang et al. (2022) Junjie Huang, Wanjun Zhong, Qian Liu, Ming Gong, Daxin Jiang, and Nan Duan. 2022. Mixed-modality Representation Learning and Pre-training for Joint Table-and-Text Retrieval in OpenQA. In _Findings of the Association for Computational Linguistics: EMNLP 2022_, pages 4117-4129, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
* Johnson et al. (2021) Jeff Johnson, Matthijs Douze, and Herve Jegou. 2021. Billion-scale similarity search with gpus. _IEEE Transactions on Big Data_, 7(3):535-547.
* Kamalloo et al. (2023) Ehsan Kamalloo, Nuha Dziri, Charles Clarke, and Davood Rafiei. 2023. Evaluating open-domain question answering in the era of large language models. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 5591-5606, Toronto, Canada. Association for Computational Linguistics.
* Karpukhin et al. (2020) Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense Passage Retrieval for Open-Domain Question Answering. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 6769-6781, Online. Association for Computational Linguistics.
* Krishna et al. (2021) Kalpesh Krishna, Aurko Roy, and Mohit Iyyer. 2021. Hurdles to Progress in Long-form Question Answering. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 4940-4957, Online. Association for Computational Linguistics.
* Lewis et al. (2020a) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020a.



BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, pp. 7871-7880. External Links: Link Cited by: SS2.
* P. Lewis, E. Perez, A. Picktus, F. Petroni, V. Karpukhin, N. Goyal, H. Kuttler, M. Lewis, W. Yih, T. Rocktaschel, S. Riedel, and D. Kiela (2020)BERTBERT: open multimodal generative pre-trained transformer for bioMedicine. Note: arXiv:2009.03649 Cited by: SS2.
* A. Hanbo Li, P. Ng, P. Xu, H. Zhu, Z. Wang, and B. Xiang (2021)Dual reader-parser on hybrid textual and tabular evidence for open domain question answering. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Vol. , pp. 4078-4088. External Links: Document, ISSN 1536-80XCited by: SS2.
* P. Li, Y. He, D. Yashar, W. Cui, S. Ge, H. Zhang, D. Rifinski Fainman, D. Zhang, and S. Chaudhuri (2023)Table-gpt: table-tuned gpt for diverse table tasks. arXiv preprint arXiv:2310.09263. Cited by: SS2.
* C. Ling, X. Zhao, J. Lu, C. Deng, C. Zheng, J. Wang, T. Chowdhury, Y. Li, H. Cui, X. Zhang, T. Zhao, A. Panalkar, W. Cheng, H. Wang, Y. Liu, Z. Chen, H. Chen, C. White, Q. Gu, J. Pei, and L. Zhao (2023)Domain specialization as the key to make large language models disruptive: a comprehensive survey. ArXiv:2305.18703[cs]. External Links: Link Cited by: SS2.
* A. Liu, H. Dong, N. Okazaki, S. Han, and D. Zhang (2022)PLOG: Table-to-logic pre-training for logical table-to-text generation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, Abu Dhabi, United Arab Emirates, pp. 5531-5546. External Links: Link Cited by: SS2.
* Y. Liu, D. Iter, Y. Xu, S. Wang, R. Xu, and C. Zhu (2023)G-eval: nlg evaluation using gpt-4 with better human alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Singapore, pp. 2511-2522. External Links: Link Cited by: SS2.
* Y. Luo, J. Zhang, S. Fan, K. Yang, Y. Wu, M. Qiao, and Z. Nie (2023)BioMedGPT: Open Multimodal Generative Pre-trained Transformer for BioMedicine. ArXiv:2308.09442[cs]. External Links: Link Cited by: SS2.
* Y. Luo, M. Lu, G. Liu, and S. Wang (2022)Few-shot Table-to-text Generation with Prefix-Controlled Generator. In Proceedings of the 29th International Conference on Computational Linguistics, Gyeongju, Republic, and Korea, pp. 6493-6504. External Links: Link Cited by: SS2.
* X. Ma, Y. Gong, P. He, H. Zhao, and N. Duan (2023)Query rewriting in retrieval-augmented large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Singapore, pp. 5303-5315. External Links: Link Cited by: SS2.
* L. Nan, Ch. Hsieh, Z. Mao, X. Victoria Lin, N. Verma, R. Zhang, W. Kryscinski, H. Schoelokopf, R. Kong, X. Tang, M. Mutuma, B. Rosand, I. Trindade, R. Bandaru, J. Cunningham, C. Xiong, D. Radev, and D. Radev (2022)FeTaQA: Free-form Table Question Answering. Transactions of the Association for Computational Linguistics10, pp. 35-49. External Links: Link, Document Cited by: SS2.
* L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. (2022)Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems35, pp. 27730-27744. External Links: Link, 2202.0206 Cited by: SS2.
* A. Parikh, X. Wang, S. Gehrmann, M. Paruqui, B. Dingra, D. Yang, and D. Das (2020)Totto: a controlled table-to-text generation dataset. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1173-1186. External Links: Link, Document Cited by: SS2.
* A. Paszke, S. Gross, F. Massa, A. Lecer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al. (2019)Pytorch: an imperative style, high-performance deep learning library. Advances in neural information processing systems32. Cited by: SS2.
* A. Radford, K. Narasimhan, T. Salimans, I. Sutskever, et al. (2018)Improving language understanding by generative pre-training. External Links: 1806.02789 Cited by: SS2.
* C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu (2020)Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research21 (140), pp. 1-67. External Links: Link, Document Cited by: SS2.
* S. Rajbhandari, J. Rasley, O. Ruwase, and Y. He (2020)Zero: memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 1-16. External Links: ISBN 978-1-467-387-3, Document, Link Cited by: SS2.
* J. Rasley, S. Rajbhandari, O. Ruwase, and Y. He (2020)DeepSpeed: system optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th 

ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, pages 3505-3506.
* Razeghi et al. (2022) Yasaman Razeghi, Robert L Logan IV, Matt Gardner, and Sameer Singh. 2022. Impact of pretraining term frequencies on few-shot numerical reasoning. In _Findings of the Association for Computational Linguistics: EMNLP 2022_, pages 840-854, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
* Soudani et al. (2024) Heydar Soudani, Evangelos Kanoulas, and Faegheh Hasibi. 2024. Fine tuning vs. retrieval augmented generation for less popular knowledge.
* Sui et al. (2023) Yuan Sui, Mengyu Zhou, Mingjie Zhou, Shi Han, and Dongmei Zhang. 2023. GPT4Table: Can Large Language Models Understand Structured Table Data? A Benchmark and Empirical Study. ArXiv:2305.13062 [cs] version: 3.
* Tang et al. (2023) Tianyi Tang, Junyi Li, Wayne Xin Zhao, and Ji-Rong Wen. 2023. MVP: Multi-task Supervised Pretraining for Natural Language Generation. In _Findings of the Association for Computational Linguistics: ACL 2023_, pages 8758-8794, Toronto, Canada. Association for Computational Linguistics.
* Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. LIama 2: Open foundation and fine-tuned chat models. _arXiv preprint arXiv:2307.09288_.
* Van der Maaten and Hinton (2008) Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. _Journal of machine learning research_, 9(11).
* Wang et al. (2023a) Cunxiang Wang, Xiaoze Liu, Yuanhao Yue, Xiangru Tang, Tianhang Zhang, Cheng Jiayang, Yunzhi Yao, Wenyang Gao, Xuming Hu, Zehan Qi, Yidong Wang, Linyi Yang, Jindong Wang, Xing Xie, Zheng Zhang, and Yue Zhang. 2023a. Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificity. ArXiv:2310.07521 [cs].
* Wang et al. (2023b) Jiaan Wang, Yunlong Liang, Fandong Meng, Zengkui Sun, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. 2023b. Is ChatGPT a good NLG evaluator? a preliminary study. In _Proceedings of the 4th New Frontiers in Summarization Workshop_, pages 1-11, Singapore. Association for Computational Linguistics.
* Wang et al. (2023c) Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Aliisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023c. Self-instruct: Aligning language models with self-generated instructions. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 13484-13508, Toronto, Canada. Association for Computational Linguistics.
* Wang et al. (2023d) Yubo Wang, Xueguang Ma, and Wenhu Chen. 2023d. Augmenting black-box llms with medical textbooks for clinical question answering.
* Wu et al. (2023e) Chaoyi Wu, Weixiong Lin, Xiaoman Zhang, Ya Zhang, Yanfeng Wang, and weidi Xie. 2023. PMC-LLaMA: Towards Building Open-source Language Models for Medicine. ArXiv:2304.14454 [cs].
* Xie et al. (2022) Tianbao Xie, Chen Henry Wu, Peng Shi, Ruqiq Zhong, Torsten Scholak, Michihiro Yasunaga, Chien-Sheng Wu, Ming Zhong, Pengcheng Yin, Sida I. Wang, Victor Zhong, Bailin Wang, Chengzu Li, Connor Boyle, Ansong Ni, Ziyu Yao, Dragromir Radev, Caiming Xiong, Lingpeng Kong, Rui Zhang, Noah A. Smith, Luke Zettlemoyer, and Tao Yu. 2022. UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Models. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 602-631, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
* Xu et al. (2023) Benfeng Xu, Chunxu Zhao, Wenbin Jiang, PengFei Zhu, Songtai Dai, Chao Pang, Zhuo Sun, Shuohuan Wang, and Yu Sun. 2023. Retrieval-augmented domain adaptation of language models. In _Proceedings of the 8th Workshop on Representation Learning for NLP (RepL4NLP 2023)_, pages 54-64, Toronto, Canada. Association for Computational Linguistics.
* Yang et al. (2023) Fangkeai Yang, Pu Zhao, Zezhong Wang, Lu Wang, Bo Qiao, Jue Zhang, Mohit Garg, Qingwei Lin, Saravan Rajmohan, and Dongmei Zhang. 2023. Empower large language model to perform better on industrial domain-specific question answering. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: Industry Track_, pages 294-312, Singapore. Association for Computational Linguistics.
* Ye et al. (2019) Rong Ye, Wenxian Shi, Hao Zhou, Zhongyu Wei, and Lei Li. 2019. Variational template machine for data-to-text generation. In _International Conference on Learning Representations_.
* Zevallos et al. (2023) Rodolfo Zevallos, Mireia Farras, and Nuria Bel. 2023. Frequency Balanced Datasets Lead to Better Language Models. In _Findings of the Association for Computational Linguistics: EMNLP 2023_, pages 7859-7872, Singapore. Association for Computational Linguistics.
* Zhang et al. (2023) Peitian Zhang, Shitao Xiao, Zheng Liu, Zhicheng Dou, and Jian-Yun Nie. 2023. Retrieve Anything To Augment Large Language Models. ArXiv:2310.07554 [cs].
* Zhang et al. (2022) Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihalov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjal Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022. OPT: Open Pre-trained Transformer Language Models. ArXiv:2205.01068 [cs].
* Zhao et al. (2022) Xujiang Zhao, Jiaying Lu, Chengyuan Deng, Can Zheng, Junxiang Wang, Tanmy Chowdhury, Li Yun, 

Hejie Cui, Zhang Xuchao, Tianjiao Zhao, et al. 2023a. Domain specialization as the key to make large language models disruptive: A comprehensive survey. _arXiv preprint arXiv:2305.18703_.
* Zhao et al. (2023b) Yilun Zhao, Haowei Zhang, Shengyun Si, Linyong Nan, Xiangru Tang, and Arman Cohan. 2023b. Investigating table-to-text generation capabilities of large language models in real-world information seeking scenarios. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: Industry Track_, pages 160-175, Singapore. Association for Computational Linguistics.
* Zhong et al. (2022) Wanjun Zhong, Junje Huang, Qian Liu, Ming Zhou, Jiahai Wang, Jian Yin, and Nan Duan. 2022. Reasoning over hybrid chain for table-and-text open domain question answering. In _International Joint Conference on Artificial Intelligence (IJCAI)_, pages 4531-4537.
* Zhu et al. (2021) Fengbin Zhu, Wenqiang Lei, Youcheng Huang, Chao Wang, Shuo Zhang, Jiancheng Lv, Fuli Feng, and Tat-Seng Chua. 2021. Tat-qa: A question answering benchmark on a hybrid of tabular and textual content in finance. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 3277-3287.

## Appendix A ICT Datasets

### Ictqa

To analyze various question types, we follow the classification method of Yang et al. (2023). This approach categorizes questions based on their first interrogative word and assigns tags reflecting the nature of the information sought. The statistical data of this classification is detailed in Table 5. Specifically, the ICTQA dataset labels questions under tags like 'Parameter', 'Configuration', and 'Command', each indicating the type of information requested. For instance, 'Parameter' relates to queries about specific values or settings, while 'Configuration' pertains to questions regarding the setup of systems or processes. Additionally, questions are grouped by their first interrogative word. This categorization sheds light on user inquiries: 'What' typically seeks factual details, and 'How' focuses on procedures or techniques. The average length of questions and answers in ICTQA is 75.13 characters and 160.25 characters, respectively. Table 6 shows examples from ICTQA questions.

### ICT-data

In the process of collecting the ICT-DATA, we perform preprocessing on the table data. Specifically,

\begin{table}
\begin{tabular}{l l|l l} \hline \multicolumn{2}{c|}{**Question Tag (\%)**} & \multicolumn{2}{c}{\(1^{st}\)**Question word (\%)**} \\ \hline Parameter & 19.55 & What & 29.33 \\ Configuration & 17.94 & How & 16.84 \\ Command & 12.25 & Why & 11.6 \\ Other & 50.26 & Which & 9.14 \\ \hline \multicolumn{2}{c|}{**Avg # of length**} & Can & 6.57 \\ \hline Question & 75.13 & Is & 4.24 \\ Answer & 160.25 & Other & 22.28 \\ \hline \end{tabular}
\end{table}
Table 5: Statistics of ICTQA

Figure 6: Top 15 Frequent Cell Contents in the Header Row of Tables.

 

to standardize the tables from the dataset, we transform them into N x M arrays. For tables with merged cells, we expand the col-span or row-span attributes, copying the content into individual cells. Additionally, to illustrate the characteristics of tables in the ICT domain, Figure 6 shows the top 15 frequent cell contents in the header rows of all tables in the ICT-DATA dataset. Table 7 provides a detailed statistical overview of the ICT-DATA. The total number of words in the dataset reaches 987 million, of which there are 178 million words in the tables, accounting for about 18% of the total dataset. On average, each table contains about 477 words, about 13 cells, and the average text length of each cell is about 36 words.

### Instruction Template

Table 8 shows the instruction template we use. We fill the question and answer slots in the template with the QA pairs from the ICTQA dataset to form a set of instructions.

## Appendix B Table-to-Text Generation Setups

### Template Design for Table Serialization

The tables in the ICT-DATA dataset consist of two types: relational tables and key-value pair tables. These two types of tables can be easily distinguished by matching keywords in the header row cells and considering the number of columns in the table. As illustrated in Table 9, we develop distinct templates for each type of table: 1) Key-value pair tables, as shown in Table 15, contain \(m\) key-value pairs that describe entities mentioned in the table's title. 2) Relational tables, as shown in Table 16, include a main column (MC) and \(n\) attribute columns (AC), where the cells in the main column represent the entities described by the table. The main column can be identified through simple rules, including the uniqueness of its content and the presence of specific keywords in the header row. To enhance the diversity of the text produced through the Template serialization, we compile a specialized glossary. When a term from this glossary is found in a table's header, the corresponding template content is adjusted accordingly. For example, if the string in the main column is "Name", "The [\(AC_{1}\)] of the [\(MC\)] named [\(C_{M_{1}}\)] is [\(C_{M_{1}A_{1}}\)]." will be changed as "The [\(AC_{1}\)] of the [\(C_{M_{1}}\)] is [\(C_{M_{1}A_{1}}\)]."

### Prompt for LLM-based Method

In Table 10, we present a prompt template specifically designed for the LLM-based table-to-text method. This template is tailored for generating natural language descriptions from tables in a two-dimensional array format, and includes a demonstration. In the prompt, we instruct LLMs not to

\begin{table}
\begin{tabular}{|p{142.3pt}|c|} \hline
**Statistics** & **Value** \\ \hline Total Word Count in ICT-DATA & 987M \\ Total Word Count in ICT Tables & 178M \\ Average Word Count per Table & 476.71 \\ Average Number of Cells per Table & 13.29 \\ Average Text Length per Cell & 35.87 \\ \hline \end{tabular}
\end{table}
Table 6: Three examples from the ICTQA dataset.

\begin{table}
\begin{tabular}{|l|} \hline Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request. \\
**Instruction:** Please answer the following questions concerning ICT products. \\
**Input:** [Question] \\
**Response:** [Answer] \\ \hline \end{tabular}
\end{table}
Table 8: The instruction template.

 

output any additional information (which does not appear in the table, but comes from the internal knowledge of the LLM), regardless of whether this information is relevant to the table content.

### Table-to-Text Generation Examples

In Table 15, we showcase the conversion of a simple two-column table into text using four different methods. For a more complex scenario involving a multi-column table with empty cells, refer to the example provided in Table 16. Lastly, the adaptation of these methods for a table featuring both multiple columns and merged cells is displayed in Table 17.

## Appendix C Training Setup and GPU costs

This paper involves model training within the DSFT QA framework. We utilize an A100 40GB node equipped with 4 GPUs for both pre-training and fine-tuning of the Llama2-13B model. The pre-training and fine-tuning phases for other models are performed on a V100 32GB node with 8 GPUs. These processes leverage the DeepSpeed framework Rasley et al. (2020). Table 11 provides a detailed overview of the GPU costs for training each model under the markdown method setting.

During the pre-training stage, we adopt an unsupervised learning approach, focusing on next-token prediction. To optimize memory usage, the DeepSpeed Zero Redundancy Optimizer Zero Stage 2) is employed Rajbhandari et al. (2020). Training parameters include a per-GPU batch size of 16 and a fixed seed value (1234) to ensure reproducibility. Each model undergoes a single epoch of training, utilizing a cosine learning rate scheduler with an initial rate of 2e-4. The learning rate warmup ratio is set at 0.05, accompanied by a weight decay rate of 0.01. Data preprocessing is performed with a block size of 512. In the QLoRA configuration, trainable parameters include the transformer's query, key, value, and output projection matrices, along with token embeddings and the language model head. Other parameters include: a LoRA rank of 64, an alpha value of 128, a dropout rate of 0.05 for the LoRA layer, and float16 for PyTorch Paszke et al. (2019) tensors.

For the instruction fine-tuning phase, we derive the instruction dataset from the ICTQA training

\begin{table}
\begin{tabular}{|p{28.5pt}|p{28.5pt}|} \hline  & **Number of iterations** \\ \hline  & **Number of iterations 

set. The training parameters for this phase are: 5 training epochs, a maximum token length of 512, a batch size of 8 per GPU, and a learning rate of 1e-4 with a cosine decaying scheduler. The configuration of QLoRA remains the same as in the pre-training phase.

## Appendix D Evaluation Setup

### Prompt for LLM as Evaluator

Table 12 details the prompt template used for evaluating responses via LLM (GPT-4) in a one-shot setting. The task of the GPT-4 is to compare four responses to a grounded answer and assigning each a score ranging from 0 to 5. Additionally, Table 13 provides an example of the one-shot demonstration used in this evaluation.

### Setup and Criteria for Human Evaluation

For the human evaluation, three co-authors of this paper, all with domain expertise in ICT products, are designated as evaluators. Each sample receives three independent evaluations from these qualified evaluators. We analyze the consistency of scoring across the three evaluations. If the ranking order of the four responses remains consistent and the score difference for the same response across different evaluators does not exceed one point, the evaluation is deemed reliable. In cases of significant discrepancies, evaluators are requested to reassess the sample. In all evaluation documents, the sources of the responses are anonymously presented as 'A', 'B', 'C', and 'D'. Table 14 shows the scoring criteria for human evaluation, which is consistent with the scoring criteria for LLM as evaluator presented in Table 12.

## Appendix E Case Analysis

We demonstrate a QA case of a RAG QA system built upon the LLaMA-70B-chat model. As shown in Table 18, our RAG QA system successfully retrieves the correct context information containing the query answer from the corpora generated by the Markdown and LLM-based methods. However, it fails to retrieve correct information from the corpora generated by the TPLM-based and Template methods. We show their t-SNE visualization in the semantic space in Figure 7. In this case, it can be clearly seen that the misleading text chunks generated by the TPLM-based and template method, which are related to entities in the query but do not contain the correct answer, are semantically closer to the query. This leads to the failure to retrieve the correct chunks (i.e., Target chunks in the figure) containing the query's answer, indicating that the text generated by these two methods has poor semantic representations.

\begin{table}
\begin{tabular}{l c c c l} \hline \hline \multirow{2}{*}{**Model**} & **Pre-training Cost** & **Fine-Tuning Cost** & **Total Cost** \\  & (GPU hours) & (GPU hours) & (GPU hours) & **GPU Node** \\ \hline OPT-1.3B & 102 & 1 & 103 & 8*V100-32G \\ OPT-2.7B & 176 & 2 & 178 & 8*V100-32G \\ OPT-6.7B & 339 & 5 & 344 & 8*V100-32G \\ OPT-13B & 660 & 5 & 665 & 8*V100-32G \\ Llama2-7B & 519 & 5 & 524 & 8*V100-32G \\ Llama2-13B & 549 & 7 & 556 & 4*A100-40G \\ \hline \hline \end{tabular}
\end{table}
Table 11: The GPU cost of training each model under the QLoRA strategy using a corpus generated by the **Markdown** table-to-text generation method. The training cost with corpora produced by other methods is close to that of Markdown. The GPU hours are computed as follows: (iteration time (seconds) \(\times\) number of iterations \(\times\) number of GPUs \(\div\) 3600 seconds/hour).

Figure 7: T-SNE visualization of chunk clusters in the embedding space for the four table-to-text methods in the RAG system case study. ‘Random Chunks’ represent chunks randomly selected from the corpus.

 

[System] You will be provided with a question, the correct answer to the question, and four candidate answers. Your responsibility is to evaluate the consistency between the candidate answers and the correct answer. The focus should be on understanding the correlation or similarity of the content, rather than grammar or style. Please make sure you understand these guidelines before proceeding.

Consult this guide whenever needed:

0, penalty:

The candidate answers have issues such as repetitive sentences, which can significantly impair the helpfulness of the response.

1, Very low correlation:

Indicates that the candidate answer is almost entirely unrelated or opposite to the correct answer.

2, Low correlation:

Indicates that the candidate answer significantly deviates from the correct answer.

3, Moderate correlation:

Suggests that the candidate answer shares some similarities with the correct answer but may lack several key points or include extra unrelated content.

4, High correlation:

Indicates that the candidate answer is largely consistent with the correct answer, missing only minor points or details.

5, Very high correlation:

Signifies that the candidate answer is almost identical to or captures the complete essence of the correct answer.

You will need to categorize the four candidate answers A, B, C, and D based on their relevance.

For example, A:1, B:2, C:3, D:5 means: A's correlation with the correct answer falls into 1. Very low correlation. B's correlation with the correct answer is higher than A's, at 2. Low correlation. C's correlation with the correct answer is 3. Moderate correlation. D's correlation with the correct answer is 5. Very high correlation.

<A demonstration of the evaluation>

[User] Evaluation Form (only score, do not output any other explanation):

<Question>: question

<Correct answer> (Standard Answer): golden answer

A answer: response_1

B answer: response_2

C answer: response_3

D answer: response_4

\begin{table}
\begin{tabular}{|p{286.9pt}|} \hline \multicolumn{2}{|c|}{**\textless{}Question\textgreater{}: How long can the Information field be in a PPP data packet?**} \\ \multicolumn{2}{|c|}{**\textless{}Correct answer\textgreater{} (Standard Answer): The maximum length for the Information field, including the Padding field, is the maximum receive unit (MRU). The MRU defaults to 1500 bytes and can be negotiated.**} \\ \end{tabular} & A answer: The length of the information field is limited to 1500 bytes. \\
**B answer: The information field can be up to 1500 bytes in length. The maximum size of the information field is specified by RFC 1661 and is set at 1500 bytes. This allows for the transmission of large packets, but also ensures that the protocol remains efficient and reliable.**} \\
**C answer: The length of the information field is variable and can range from 0 to 1536 bytes.**} \\
**D answer: The length of the information field is not specified, but it must be at least 1 byte. The information field is variable length and contains the protocol specific information that the peer requires to establish the link. The length of the information field is not specified, but it must be at least one byte. The format of the information field is defined by the protocol being used. For example, IPX uses the information field to specify the IPX network number and the IPX node address.** \\ \end{tabular}
\end{table}
Table 13: One-Shot Demonstration for the LLM Evaluator.

 

The scoring criteria for human evaluation:

0, penalty:

 The candidate answers have issues such as repetitive sentences, which can significantly impair the helpfulness of the response.

1, Very low correlation:

 Indicates that the candidate answer is almost entirely unrelated or opposite to the correct answer.

2, Low correlation:

 Indicates that the candidate answer significantly deviates from the correct answer.

3, Moderate correlation:

 Suggests that the candidate answer shares some similarities with the correct answer but may lack several key points or include extra unrelated content.

4, High correlation:

 Indicates that the candidate answer is largely consistent with the correct answer, missing only minor points or details.

5, Very high correlation:

 Signifies that the candidate answer is almost identical to or captures the complete essence of the correct answer.

Table Example 1: Basic information about the PLCh-Power-1

\begin{table} \end{table}
Table 15: A table-to-text example of four methods for a simple table with only two columns.

 

Table Example 2: Indicators on the PLC-IH-1

\begin{tabular}{|c|c|c|c|c|} \hline \hline Regional & Min. & CQR.M & Grid. \\ \hline - & PWR indicator & Green & Steady on & The module is powered on. \\ \hline - & PWR indicator & - & Off & The module is powered off. \\ \hline - & PLC,/R indicator & Red & Blinking & The module is receiving data. \\ \hline - & PLC,/R indicator & Green & Blinking & The module is sending data. \\ \hline - & Broadband carrier data sending status indicator of phase A & Green & Steady on & Broadband carrier data is sent through phase B. \\ \hline - & Broadband carrier data sending status indicator of phase B & Green & Steady on & Broadband carrier data is sent through phase C. \\ \hline - & Broadband carrier data sending status indicator of phase C & Green & Steady on & Broadband carrier data is sent through phase C. \\ \end{tabular}

Markdown:

Table Caption: Indicators on the PLC-IH-1

[0pt] 



\begin{table} \end{tabular}
* Markdown: Table Caption: Relationship between the device group type and device networking reliability

| Type I Networking | Remarks | | 1 :- | :- | -- | | Multiple-active device group | ToR devices configured with M-LAG | Two to eight devices that use the same device model and VTEP IP address can be added to a device group. | Multiple-active device group | Multiple-active gateway or gateways configured with M-LAG | Two to eight devices that use the same device model and VTEP IP address can be added to a device group. | Multiple-active device group | Multiple-active NE routers | Two to eight devices that use the same device model and VTEP IP address can be added to a device group. | VTEP IP address can be added to a device group. | Multiple-active device group | Multiple-active vDHCP device group | Two to eight devices that use the same device model and VTEP IP address can be added to a device group. | Template: The following sentences describe about Relationship between the device group type and device networking reliability. The Networking of the Type named Multiple-active device group is ToR devices configured with M-LAG. Its Remarks is Two to eight devices that use the same device model and VTEP IP address can be added to a device group. The Networking of the Type named Multiple-active device group is Multiple-active gateway or gateways configured with M-LAG. Its Remarks is Two to eight devices that use the same device model and VTEP IP address can be added to a device group. The Networking of the Type named Multiple-active device group is Multiple-active NE routers. Its Remarks is Two to eight devices that use the same device model and VTEP IP address can be added to a device group. | TPLM-based: The multiple-active device group type is ToR devices configured with M-LAG. The multiple-active device group type is Two to eight devices that use the same device model and VTEP IP address can be added to a device group. The multiple-active device group type is Multiple-active gateway or gateways configured with M-LAG. The multiple-active device group type is Two to eight devices that use the same device model and VTEP IP address can be added to a device group. | Added to a device group.. The multiple-active device group type is Multiple-active NE routers. The multiple-active device group type is Multiple-active vDHCP device group. The multiple-active device group type is Two to eight devices that use the same device model and VTEP IP address can be added to a device group. LLM-based: The table provides information about different types of device groups and their configurations. The "Multiple-active device group" type can be configured with various networking options such as ToR devices with M-LAG, multiple-active gateway or gateways with M-LAG, multiple-active NE routers, and multiple-active vDHCP device group. Each of these types allows for the addition of two to eight devices that share the same device model and VTEP IP address.





**Question**: How can I modify the entries in the pimBsrElectedBSRRPSetTable?

**Gold Answer**: The entries in this table cannot be modified.

\begin{table}
\begin{tabular}{l l} \hline \hline
**Methodown Retrieved Passage**: ... & Title: pimBsrElectedBSRRPSetTable This table describes RP-Set information in the elected BSR ... Creation Restriction: No entry can be created in this table. Modification Restriction: The entries in this table cannot be modified. Deletion Restriction: The entries in this table cannot be deleted ... \\  & cannot be modified. Deletion Restriction: The entries in this table cannot be deleted ... \\  & **Answer**: The modification restriction for the pimBsrElectedBSRRPSetTable is set to “not-accessible,” indicating that the entries in this table cannot be modified. This restriction is defined in the MIB file and cannot be changed through the command line or any other means. Therefore, it is not possible to modify the entries in this table. ✓ \\ \hline
**Template** & **Retrieved Passage**: ... <Page_Start>: Title: Configuration Precautions for VRRP ¡Page_End> <Page_Start>: Title: vrrpRouterStatsTable <Page_End> <Page_Start>: Title: Configuring VRRP ... \\
**Answer**: I don’t know the answer. ✗ \\ \hline
**TPLM-based Retrieved Passage**: ... <Page_Start>: Title: Configuration Precautions for VRRP ¡Page_End> <Page_Start>: Title: vrrpRotateStatsTable <Page_End> <Page_Start>: Title: Configuring VRRP ... \\
**Answer**: I don’t know the answer. The knowledge provided does not contain any information about modifying entries in the pimBsrElectedBSRRPSetTable. The title of the knowledge suggests that it may be related to PIM (Protocol Independent Multicast) and BSR (Bootstrap Router), but without additional context or information, it is not possible to provide a helpful answer to the question.If you have any additional information or context, please provide it and I will do my best to assist you. ✗ \\ \hline
**LLM-based Retrieved Passage**: ... pimBsrElectedBSRRPSetTable This table describes RP-Set information in the elected BSR ... Creation Restriction: No entry can be created in this table. Modification Restriction: The entries in this table cannot be modified. Deletion Restriction: The entries in this table cannot be deleted... \\
**Answer**: According to the provided knowledge, it is not possible to modify the entries in the pimBsrElectedBSRTPSetTable. The table provides information about various objects related to PIM BSR (Protocol Independent Multicast Bootstrap Router) elected BSR (Best Source Router) group mapping, and the entries in this table cannot be modified.The knowledge states that the creation restriction for this table is set to “No entry can be created in this table,” and the modification restriction is set to “The entries in this table cannot be modified.” Additionally, the deletion restriction is set to “The entries in this table cannot be deleted.” Therefore, it is not possible to modify the entries in the pimBsrElectedBSRRPSetTable, and any attempts to do so will likely result in an error message or a notification that the modification was unsuccessful. ✓ \\ \hline \hline \end{tabular}
\end{table}
Table 18: A QA example of the RAG QA system based on using a corpus generated by each of the four table-to-text methods as a retrieval source. The red font indicates text in the retrieved passage that is relevant to the answer.