Rethinking Table Instruction Tuning
Authors: Naihao Deng, Rada Mihalcea
Accepted at: acl25

Rethinking Table Instruction Tuning

Naihao Deng and Rada Mihalcea
University of Michigan
{dnaihao, mihalcea}@umich.edu

Abstract

Recent advances in table understanding have focused on instruction-tuning large language models (LLMs) for table-related tasks. However, existing research has overlooked the impact of hyperparameter choices, and also lacks a comprehensive evaluation of the out-of-domain table understanding ability and the general capabilities of these table LLMs. In this paper, we evaluate these abilities in existing table LLMs, and find significant declines in both out-of-domain table understanding and general capabilities as compared to their base models. Through systematic analysis, we show that hyperparameters, such as learning rate, can significantly influence both table-specific and general capabilities. Contrary to the previous table instruction-tuning work, we demonstrate that smaller learning rates and fewer training instances can enhance table understanding while preserving general capabilities. Based on our findings, we introduce TAMA, a Table LLM instruction-tuned from LLaMA 3.1 8B Instruct, which achieves performance on par with, or surpassing GPT-3.5 and GPT-4 on table tasks, while maintaining strong out-of-domain generalization and general capabilities. Our findings highlight the potential for reduced data annotation costs and more efficient model development through careful hyperparameter selection. We open-source the project and our models.1

1 Introduction

Recent years have witnessed a paradigm shift to data-driven methods for table understanding. Researchers have instruction-tuned various LLMs, particularly the open-source models from the LLaMA family (Touvron et al., 2023; Dubey et al., 2024), to improve their ability on handling table-related tasks (Chen et al., 2019; Nan et al., 2022),

Figure 1: Performance comparison between our model TAMA and the existing table LLMs on out-of-domain table understanding and general benchmarks.

and to advance the state-of-the-art performance on various table benchmarks (Zhang et al., 2024a,b).

However, existing research has been influenced by the lack of transparency on closed-source LLMs, which often claim to be trained on large-scale datasets without revealing the detailed training process. As a result, open-source efforts have tended to follow these closed-source models by focusing primarily on large-scale datasets (Zhang et al., 2024a), while overlooking the crucial influence of hyperparameter choices. In addition, existing work lacks a discussion of how these table LLMs perform on out-of-domain table understanding tasks, and how their general capabilities get compromised when specializing on table tasks. We argue that out-of-domain table understanding is crucial for table LLMs, as it reflects how well these models generalize to unseen table tasks. In addition, the general capabilities of these models are still important for handling table-related tasks. For instance, instruction following is crucial in real-world applications where end-users may request specific input-output formats (e.g., The user may request the model to return the answer in JSON). Additionally, stronger reasoning capabilities and comprehensive general knowledge

1We open-source our code at https://github.com/MichiganNLP/TAMA. We open-source our models at https://huggingface.co/MichiganNLP/tama-5e-7 and https://huggingface.co/MichiganNLP/tama-1e-6.
can enhance these models’ ability to handle diverse scenarios, such as interpreting user queries and reasoning over complex data. Therefore, having an understanding of these table LLMs’ general capabilities gives us a comprehensive understanding of these models’ limitations in our practical usage.

In this paper, we first evaluate the existing table LLMs in terms of their out-of-domain table understanding ability and their general abilities. We reveal that existing table LLMs suffer from a significant decline in terms of these abilities compared to their base models. Sometimes, the performance decline on general reasoning benchmarks, such as AI2ARC, can be up to 20 percentage.

We then select the latest LLaMA 3.1 8B Instruct model, and proceed to explore how hyperparameter choices influence the model’s performance. Our analysis reveals that the learning rate plays a crucial role in shaping the model’s table understanding ability and influencing the model’s general ability. A large learning rate, as seen in the existing table LLMs, compromises the model’s general capabilities and leads to suboptimal table understanding performance. On the other hand, a small learning rate, while effectively preserving the model’s general capabilities, fails to sufficiently improve its table understanding ability. In addition, we find that it is possible to achieve strong table understanding ability with a much smaller amount of training data – for instance, 2,600 in Section 4. Our training size is significantly smaller compared to the two million instances used by TableLLaMA (Zhang et al., 2024a), and ten times smaller than that of TableBenchLLM (Wu et al., 2024), highlighting the potential to reduce annotation costs in future model development. We also explore the effects of the number of epochs and the synergy of the task, and discuss our findings in Section 3.

Based on our findings, we carefully select the hyperparameters and instruction-tune the LLaMA 3.1 8B Instruct model, resulting in TAMA, which demonstrates strong table understanding ability and general capabilities (Figure 1).

In summary, our contributions are three fold:

• We examine the existing table LLMs and reveal that these table LLMs do not generalize to out-of-domain table tasks and show compromised general capabilities compared to their base model.

• We reveal the impact of the often-ignored hyperparameter selection such as the learning rate, number of training instances, and so on. We find that the commonly-adopted learning rate can be too large, may lead to suboptimal table understanding performance, and can compromise the model’s general capabilities. In addition, we can achieve strong table understanding ability with a much smaller amount of training data compared to the existing works.

• Based on our findings, with careful hyperparameter selection, we instruction-tune LLaMA 3.1 8B Instruct model with 2,600 table instruction data. As an 8B size model, our resulting model, TAMA achieves performance on par with, or even exceeding GPT-3.5 in table understanding tasks, and in some cases surpassing GPT-4, while retaining the general capabilities of its base model. Moreover, TAMA exhibits strong out-of-domain table understanding and general capabilities (Figure 1).

In the following sections, Section 2 evaluates the existing table LLMs in terms of their out-of-domain table understanding ability and general capabilities. Section 3 explores how the hyperparameter choices shape the model’s ability. Based on our findings in Section 3, we build our model, TAMA in Section 4.

2 Evaluation of Existing Table LLMs

2.1 Experimental Setup

Models to Evaluate. Table 1 provides a comprehensive overview of the existing table LLMs. As we do not have access to the closed-source table LLMs, we focus on the evaluation of the open-source ones, including TableLLaMA (Zhang et al., 2024a), TableLLM (Zhang et al., 2024b), and TableBenchLLM (Wu et al., 2024). All of these open models are fine-tuned with all parameters being updated.

Evaluation Datasets. Table 2 provides the datasets on which we test these table LLMs in terms of their out-of-domain table understanding ability and their general capabilities. We choose Table-Syn (Li et al., 2023) to test these table LLMs’ out-of-domain table understanding ability, as none of them has been fine-tuned on this dataset.

2.2 Findings

Existing Table LLMs possess limited out-of-domain table understanding ability. In Table 3, all the existing table LLMs suffer from performance drops on
<table>
  <tr>
    <th>Model</th>
    <th>Base Model</th>
    <th>Learning Rate</th>
    <th>Epochs</th>
    <th>Data Size</th>
    <th>Data Source</th>
    <th>Open-Source?</th>
  </tr>
  <tr>
    <td>TableGPT (2023)</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>X</td>
  </tr>
  <tr>
    <td>Table-GPT (2023)</td>
    <td>GPT-3.5</td>
    <td>-</td>
    <td>-</td>
    <td>13K</td>
    <td>S</td>
    <td>X</td>
  </tr>
  <tr>
    <td>TableLLaMA (2024a)</td>
    <td>LongLoRA 7B†</td>
    <td>2e-5</td>
    <td>6</td>
    <td>2M</td>
    <td>R</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>TableLLM (2024b)</td>
    <td>CodeLLaMA 7B & 13B Instruct</td>
    <td>2e-5</td>
    <td>6</td>
    <td>309K</td>
    <td>R + S</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>TableBenchLLM (2024)</td>
    <td>LLaMA 3.1-8B & others</td>
    <td>2e-5</td>
    <td>3</td>
    <td>20K</td>
    <td>S</td>
    <td>✓</td>
  </tr>
</table>

Table 1: Existing table instruction tuned models. For “Data Source”, “S” and “R” represent synthesized data and real data, respectively. †: a variant based on the LLaMA 2 7B model.

<table>
  <tr>
    <th>Evaluation Datasets</th>
    <th>Category</th>
    <th># Shots</th>
    <th>Task Type</th>
    <th>Metrics</th>
  </tr>
  <tr>
    <td>Table-Syn* (2023)</td>
    <td>Table Understanding</td>
    <td>-</td>
    <td>Gen</td>
    <td>Acc</td>
  </tr>
  <tr>
    <td>IFEval (2023)</td>
    <td>Instruction Following</td>
    <td>-</td>
    <td>Gen</td>
    <td>Instance-level Acc</td>
  </tr>
  <tr>
    <td>MMLU (2021)</td>
    <td>General</td>
    <td>5-shot</td>
    <td>MC</td>
    <td>Acc</td>
  </tr>
  <tr>
    <td>MMLU pro (2024)</td>
    <td>General</td>
    <td>5-shot</td>
    <td>MC</td>
    <td>Acc</td>
  </tr>
  <tr>
    <td>AI2ARC (2018)</td>
    <td>Reasoning</td>
    <td>0-shot</td>
    <td>MC</td>
    <td>Acc</td>
  </tr>
  <tr>
    <td>GPQA (2023)</td>
    <td>Reasoning</td>
    <td>0-shot</td>
    <td>MC</td>
    <td>Acc</td>
  </tr>
</table>

Table 2: Details of the benchmarks upon which we evaluate the existing table LLMs. We report the performance on the main set for GPQA and the challenge set for AI2ARC. “Gen” and “MC” stand for generation and multi-choice, respectively.

Table-Syn compared to their base models. Though these table LLMs achieve SOTA performance on various benchmarks (Zhang et al., 2024a,b), such a performance decline reveals their limited out-of-domain table understanding capabilities, which aligns with the findings by Zheng et al. (2024a).

Existing Table LLMs demonstrate poor instruction-following ability. In Table 3, both TableLLaMA and TableLLM show significant drops in performance on IFEval (Zhou et al., 2023), with accuracy declines of 5.63 and 17.86, resulting in a score of 25.78 and 30.46, respectively. While TableBenchLLM maintains a similar score to its base model (32.85 compared to 32.13 for LLaMA 3.1-8B), this performance is still limited compared to 83.57 by GPT-4 reported by Zhou et al. (2023). At such low instruction following scores, existing table LLMs cannot consistently follow instructions such as “return the answer in JSON format” as shown in Table 7 in Section 4.3 and Tables 17 to 19 in Appendix E, limiting the model’s usage if the end users need data extraction that requires certain answer format.

Existing table instruction tuning compromises models’ general capabilities. Existing table instruction-tuning methods lead to significant drops in accuracy on general benchmarks such as MMLU, AI2ARC, GPQA as shown in Table 3. For instance, compared to their base models, TableLLaMA experiences a decline of 13.95 accuracy score on MMLU, while TableLLM and TableBenchLLM lose 8.79 and 9.41, respectively. Appendix B provides further discussion of the model’s performance corresponding to each category in MMLU benchmark. On the general reasoning benchmarks such as AI2ARC, the drop can be as large as 20.90 for TableBenchLLM, showing that the existing table instruction tuning hurts their base model’s reasoning ability. This limits the existing table LLMs’ usage if there are general knowledge or reasoning involved in end users’ request.

3 Hyperparameter Exploration

Table 1 reports the hyperparameters used in the existing table instruction tuning works. While often overlooked or treated as technical details, hyperparameter selection plays a critical role. The impact of factors such as learning rate, and number of epochs should not be underestimated, as they significantly influence both the table understanding and general ability. In the following subsections, Section 3.1 introduces the model and datasets used in our analysis experiments, Section 3.2 provides the findings and the choices we make that lead to our model in Section 4.

3.1 Experimental Setup

Models. We conduct full parameter table instruction tuning using the 8B version of the LLaMA 3.1 Instruct model (Dubey et al., 2024) because of its superior general capabilities, especially its strong instruction following ability. Appendix C.1 provides detailed reasons for the choice of LLaMA 3.1 and the choice between the base versus the
<table>
  <tr>
    <th></th>
    <th>Table-Syn</th>
    <th>IFEval</th>
    <th>MMLU</th>
    <th>MMLU<sub>Pm</sub></th>
    <th>AI2ARC</th>
    <th>GPQA</th>
  </tr>
  <tr>
    <td>LongLoRA 7B<sup>†</sup></td>
    <td>2.40</td>
    <td>31.41</td>
    <td>44.22</td>
    <td>17.51</td>
    <td>42.24</td>
    <td>23.66</td>
  </tr>
  <tr>
    <td>TableLLaMA</td>
    <td>0.00</td>
    <td>25.78</td>
    <td>30.27</td>
    <td>12.33</td>
    <td>30.89</td>
    <td>23.44</td>
  </tr>
  <tr>
    <td>△</td>
    <td>↓ 2.40</td>
    <td>↓ 5.63</td>
    <td>↓ 13.95</td>
    <td>↓ 5.18</td>
    <td>↓ 11.35</td>
    <td>↓ 0.22</td>
  </tr>
  <tr>
    <td>CodeLLaMA 13B Instruct</td>
    <td>33.40</td>
    <td>48.32</td>
    <td>44.69</td>
    <td>19.66</td>
    <td>48.72</td>
    <td>24.78</td>
  </tr>
  <tr>
    <td>TableLLM</td>
    <td>18.40</td>
    <td>30.46</td>
    <td>35.90</td>
    <td>15.36</td>
    <td>34.81</td>
    <td>24.11</td>
  </tr>
  <tr>
    <td>△</td>
    <td>↓ 15.00</td>
    <td>↓ 17.86</td>
    <td>↓ 8.79</td>
    <td>↓ 4.30</td>
    <td>↓ 13.91</td>
    <td>↓ 0.67</td>
  </tr>
  <tr>
    <td>LLaMA 3.1-8B</td>
    <td>13.40</td>
    <td>32.13</td>
    <td>62.08</td>
    <td>13.86</td>
    <td>74.40</td>
    <td>28.12</td>
  </tr>
  <tr>
    <td>TableBenchLLM</td>
    <td>9.00</td>
    <td>32.85</td>
    <td>52.67</td>
    <td>17.84</td>
    <td>53.50</td>
    <td>27.01</td>
  </tr>
  <tr>
    <td>△</td>
    <td>↓ 4.40</td>
    <td>↑ 0.72</td>
    <td>↓ 9.41</td>
    <td>↑ 3.98</td>
    <td>↓ 20.90</td>
    <td>↓ 1.11</td>
  </tr>
</table>

Table 3: Performance comparison between the existing table LLMs (second row) and their base models (first row). †: A variant of LLaMA 2 7B model.

instruction-tuned version of the model. In addition, we expand our analyses across various LLMs and learning setups detailed in Section 3.2.

Datasets. We draw training data from three representative table understanding datasets in this section, FeTaQA (Nan et al., 2022), a free-form table question answering (Table QA) dataset; HiTab (Cheng et al., 2022), a short-answer Table QA dataset; TabFact (Chen et al., 2019), a table fact verification dataset. In Figure 2, we also report the model’s performance on FEVEROUS (Aly et al., 2021), another table fact checking dataset, and on two general benchmarks, MMLU and IFEval introduced in Table 2.

3.2 Analysis

We first analyze the effects of the learning rate and the number of examples.

Learning Rate. In Figure 2, we fine-tune the LLaMA 3.1 8B Instruct model using instruction data from TabFact, HiTab, and FeTaQA.

We find that the learning rate plays a crucial role in determining model performance, as well as how well the model preserves its general capabilities. In general, LLaMA 3.1 8B Instruct achieves the best performance when the learning rate is around 1.0e-6 and 5.0e-7. For instance, on TabFact, LLaMA 3.1 8B Instruct achieves its best performance (73.10) at a learning rate of 1.0e-6 with 1500 examples. Moreover, there is little to no decrease in LLaMA 3.1 8B Instruct performance on MMLU and IFEval with such learning rates (Figures 2e and 2f). With a smaller learning rate such as 1.0e-7, though the model’s performance on MMLU and IFEval can be well-preserved, the model’s performance on table tasks such as FEVEROUS is suboptimal under the same setup (66.86 compared to 74.63 at a learning rate of 5.0e-6). In contrast, when the learning rate is too large, such as 1.0e-5, we observe a significant decline in the model’s performance on both MMLU and IFEval, suggesting that a larger learning rate may hurt the model’s general capabilities. We note that all the existing table LLMs use a large learning rate of 2e-5 (Table 1), which explains their compromised out-of-domain table understanding ability and general capabilities compared to their base models in Table 3.

Number of Examples. As the number of training instances increases, we find that there is a period of quick learning followed by a period of marginal performance improvement.

We observe in Figure 2 that on table tasks such as FeTaQA and HiTab, there is a period where the model’s performance boosts up quickly, typically happening when tuning on the first 200 examples. Later, the performance improvement appears marginal. This aligns with the findings of Zhou et al. (2024) that the foundational performance of the LLM can be improved with a limited amount of high-quality data in the instruction tuning stage. We hypothesize that with the first few hundred examples, the model is able to enhance its table reasoning ability quickly. After this point, the model’s performance increase may primarily come from fitting the nuanced patterns in these datasets. Therefore, unlike the existing table LLMs that may involve up to two million training instances as seen in Table 1, we choose to train on 200 instances for each dataset in Section 4.

In addition, we can achieve competitive or even SOTA performance with limited data. On HiTab,
Figure 2: LLaMA 3.1 8B Instruct’s performance (y-axis) with respect to the number of training instances (x-axis). We fine-tune the model for three epochs. We note that the learning rate plays a crucial role in shaping the model’s capabilities, and the performance improvement beyond 200 examples seems marginal.

with a learning rate of 1.0e-6 and 1,500 examples, we achieve an accuracy score of 66.29, outperforming the previous SOTA performance of 64.71 by TableLLaMA. On FEVEROUS, with 1,500 examples, we achieve a better score of 74.63 compared to 73.77 by TableLLaMA. Although the credit also comes from the LLaMA 3.1 Instruct model, which is much stronger compared to the LLaMA 2 model that TableLLaMA is tuned from, we highlight that TableLLaMA has used two million data points in its table instruction tuning stage, including the entire training set of TabFact, FeTaQA, and HiTab, while here we use around 7% of the entire training data for HiTab. Our analysis demonstrates that with a strong foundational model and a good choice of learning rate, we can achieve competitive performance on table understanding tasks with limited training instances.

Full Analysis. We provide a pointer to our full analysis and a short summary of the findings here.

1. Learning rate across LLMs under full parameter setup (Appendix C.4), and LoRA and QLoRA setups (Appendix C.5).

Finding: We list our recommended learning rate per model in Table 4 for the full parameter setup. Please refer to Appendix C.5 for recommendations for LoRA and QLoRA setups.

2. Number of examples across LLMs under full parameter tuning setup (Appendix C.4), and LoRA and QLoRA setups (Appendix C.5).

Finding: Under all these setups, there is a diminishing return as the number of training examples increases.

3. Number of epochs (Appendix C.2).

Finding: We do not see significant performance gains when we increase the number of epochs. Therefore, we choose to train our model for two epochs in Section 4.

4. Multi-task training (Appendix C.3).

Finding: There are synergy effects on these table tasks, therefore, we decide to fine-tune our

<table>
  <tr>
    <th>Model</th>
    <th>Learning Rate</th>
  </tr>
  <tr>
    <td>Llama 2 7B Instruct</td>
    <td>1.0e-6 / 5.0e-7</td>
  </tr>
  <tr>
    <td>Llama 3.1 8B Instruct</td>
    <td>1.0e-6 / 5.0e-7</td>
  </tr>
  <tr>
    <td>Qwen 2.5 7B Instruct</td>
    <td>1.0e-6 / 5.0e-7</td>
  </tr>
  <tr>
    <td>Mistral v0.3 7B Instruct</td>
    <td>5.0e-7 / 1.0e-7</td>
  </tr>
  <tr>
    <td>Phi 3 small 8K Instruct (7B)</td>
    <td>5.0e-6 / 1.0e-6</td>
  </tr>
</table>

Table 4: Recommended learning rate across different LLMs on table-specific tasks.
<table>
  <tr>
    <th>Task Category</th>
    <th>Task Name</th>
    <th>Dataset</th>
    <th>Shorthand</th>
    <th>#Size (Table/Sample)</th>
    <th>Data Split</th>
    <th>Metrics</th>
  </tr>
  <tr>
    <td rowspan="6">Question Answering</td>
    <td>Table QA</td>
    <td>WikiTQ (2015)</td>
    <td>W-T</td>
    <td>0.4K/4K</td>
    <td>Test</td>
    <td>Acc</td>
  </tr>
  <tr>
    <td>Table QA</td>
    <td>WikiSQL (2017)</td>
    <td>W-S</td>
    <td>5K/16K</td>
    <td>Test</td>
    <td>Acc</td>
  </tr>
  <tr>
    <td>Hybrid Table QA</td>
    <td>HybridQA (2020)</td>
    <td>Hyb</td>
    <td>3K/3K</td>
    <td>Test</td>
    <td>Acc</td>
  </tr>
  <tr>
    <td>Table QA</td>
    <td>TATQA (2021)</td>
    <td>TAT</td>
    <td>0.2K/0.7K</td>
    <td>Test</td>
    <td>Acc</td>
  </tr>
  <tr>
    <td>Highlighted Cells QA</td>
    <td>FcTaQA (2022)</td>
    <td>FeT</td>
    <td>2K/2K</td>
    <td>Dev</td>
    <td>BLEU</td>
  </tr>
  <tr>
    <td>Hierarchical Table QA</td>
    <td>HiTab (2022)</td>
    <td>HiT</td>
    <td>1K/1K</td>
    <td>Dev</td>
    <td>Acc</td>
  </tr>
  <tr>
    <td>Hierarchical Table QA</td>
    <td>AIT-QA (2022)</td>
    <td>AIT</td>
    <td>0.1K/0.3K</td>
    <td>Test</td>
    <td>Acc</td>
  </tr>
  <tr>
    <td>Table QA</td>
    <td>TABMWP (2023)</td>
    <td>TAB</td>
    <td>7K/7K</td>
    <td>Test</td>
    <td>Acc</td>
  </tr>
  <tr>
    <td rowspan="3">Table Fact Verification</td>
    <td>Fact Verification</td>
    <td>TabFact (2019)</td>
    <td>TaF</td>
    <td>2K/12K</td>
    <td>Dev</td>
    <td>Acc</td>
  </tr>
  <tr>
    <td></td>
    <td>InfoTabs<sup>2</sup> (2020)</td>
    <td>Inf</td>
    <td>0.06K/0.5K</td>
    <td>Test</td>
    <td>Acc</td>
  </tr>
  <tr>
    <td></td>
    <td>FEVEROUS (2021)</td>
    <td>FEV</td>
    <td>4K/7K</td>
    <td>Dev</td>
    <td>Acc</td>
  </tr>
  <tr>
    <td>Dialogue Generation</td>
    <td>Table Grounded Dialogue Generation</td>
    <td>KVRET (2017)</td>
    <td>KVR</td>
    <td>0.3K/0.8K</td>
    <td>Test</td>
    <td>Micro F1</td>
  </tr>
  <tr>
    <td>Data-to-Text</td>
    <td>Highlighted Cells Description</td>
    <td>ToTTo (2020)</td>
    <td>ToT</td>
    <td>7K/8K</td>
    <td>Test</td>
    <td>BLEU</td>
  </tr>
</table>

Table 5: Datasets where we sample the instruction pairs to fine-tune the LLaMA 3.1 8B Instruct model. We randomly select 200 data points from each of these datasets in our table instruction tuning stage. We denote these datasets by their shorthands in Table 6.

model on a diverse range of tasks in Section 4.

In addition, we provide analysis regarding how the data features affect the model’s performance degradation on general benchmarks in Appendix C.7.

4 TAMA

Based on our findings from Section 3, we build our general table understanding model TAMA by instruction tuning the LLaMA 3.1 8B Instruct model.

4.1 Experimental Setup

Hyperparameter Selection. In Section 3, we find that with 200 instruction pairs, the model has already achieved competitive table understanding ability, and the performance gain after such a point is marginal. Moreover, tuning the model at a learning rate of 1.0e-6 for two epochs would enhance the model’s table understanding ability while still maintaining its general ability. Therefore, we select 200 instruction pairs in the training set from each of the datasets in Table 5, and train the model at the learning rate of 1.0e-6 for two epochs.

Dataset Splits. As we use FeTaQA, HiTab, Tab-Fact, FEVEROUS, MMLU, and IFEval in Section 3 for hyperparameter selection, we report their scores under the “Dev” category. In the test time, we test our model on the additional nine table understanding datasets in Table 5. Moreover, we test our model on the two synthesized table understanding datasets from Table-Syn (Li et al., 2023) and from Wu et al. (2024) (denoted as S1 and S2 in Table 8, respectively) to assess its out-of-domain table understanding ability. To assess the model’s general ability, apart from MMLU and IFEval, we test our model on MMLU$_{Pro}$, AI2ARC, and GPQA introduced in Table 2.

Appendix A provides more details of our experimental setup including the information of GPU server, generation hyperparameters, data processing, and our evaluation setup. Appendix F provides examples from datasets that we evaluate upon.

4.2 Results and Analyses

Table 6 shows TAMA’s performance on datasets listed in Table 5. Table 8 shows TAMA’s performance on the two out-of-domain table benchmarks and the general benchmarks.

TAMA demonstrates strong table understanding ability. We notice that there is a significant performance boost for TAMA compared to its base model, LLaMA 3.1 8B Instruct, on almost every dataset. For instance, on Table QA tasks such as HybridQA, TAMA achieves an accuracy of 60.86 compared to LLaMA 3.1 8B Instruct’s 32.83. When compared to the commercial closed-source LLMs such as GPT-3.5 and GPT-4, TAMA surpasses the performance of GPT-3.5 model on almost every ta-

1https://machinelearning.apple.com/research/introducing-apple-foundation-models
2Due to budget limit for prompting GPT models, we uniformly sample 500 data points from the original test set as our test set.
<table>
  <tr>
    <th rowspan="2">Models</th>
    <th colspan="4">Dev</th>
    <th colspan="3">Test</th>
    <th rowspan="2">ToT</th>
  </tr>
  <tr>
    <th>FeT</th>
    <th>HiT</th>
    <th>TaF</th>
    <th>FEV</th>
    <th>W-T</th>
    <th>W-S</th>
    <th>Hyb</th>
    <th>TAT</th>
    <th>AIT</th>
    <th>TAB</th>
    <th>Inf</th>
    <th>KVR</th>
  </tr>
  <tr>
    <td>GPT-3.5</td>
    <td>26.49<sup>†</sup></td>
    <td>43.62<sup>†</sup></td>
    <td>67.41<sup>†</sup></td>
    <td>60.79</td>
    <td>53.13<sup>†</sup></td>
    <td>41.91<sup>†</sup></td>
    <td>40.22<sup>†</sup></td>
    <td>31.38<sup>†</sup></td>
    <td>84.13</td>
    <td>46.30<sup>†</sup></td>
    <td>56.00</td>
    <td>54.56<sup>†</sup></td>
    <td>16.81<sup>†</sup></td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>21.70<sup>†</sup></td>
    <td>48.40<sup>†</sup></td>
    <td><b>74.40<sup>†</sup></b></td>
    <td><b>68.40<sup>†</sup></b></td>
    <td><b>47.60<sup>†</sup></b></td>
    <td><b>58.60<sup>†</sup></b></td>
    <td><b>55.81<sup>†</sup></b></td>
    <td><b>88.57</b></td>
    <td><b>67.10<sup>†</sup></b></td>
    <td><b>58.60</b></td>
    <td><b>56.46<sup>†</sup></b></td>
    <td><b>12.21</b></td>
  </tr>
  <tr>
    <td>base</td>
    <td>15.33</td>
    <td>32.83</td>
    <td>58.44</td>
    <td>66.37</td>
    <td>43.46</td>
    <td>20.43</td>
    <td>32.83</td>
    <td>26.70</td>
    <td>82.54</td>
    <td>39.97</td>
    <td>48.39</td>
    <td>50.80</td>
    <td>13.24</td>
  </tr>
  <tr>
    <td><b>TAMA</b></td>
    <td><b>35.37</b></td>
    <td><b>63.51</b></td>
    <td><b>73.82</b></td>
    <td><b>77.39</b></td>
    <td><b>52.88</b></td>
    <td><b>68.31</b></td>
    <td><b>60.86</b></td>
    <td><b>48.47</b></td>
    <td><b>89.21</b></td>
    <td><b>65.09</b></td>
    <td><b>64.54</b></td>
    <td><b>43.94</b></td>
    <td><b>37.94</b></td>
  </tr>
</table>

Table 6: Evaluation results on the datasets listed in Table 5. “Base” denotes the LLaMA 3.1 8B Instruct model. We make the number bold if it is the best among the four, we underline the number if it is at the second place. † indicates the performance reported by Gou et al. (2023); Srivastava et al. (2024); Zhang et al. (2024a).

ble task in Table 6 except for KVRET and WikiTQ. And on WikiTQ, the two yields a similar performance (TAMA achieves 52.81 and GPT-3.5 achieves 53.13).

On tasks such as WikiSQL, HybridQA, InfoTabs, FEVEROUS, TAMA yields a superior performance than GPT-4. Notably, on two out-of-domain synthesized table understanding datasets in Table 8, TAMA surpasses the performance of GPT-3.5 (on S1, TAMA yields 64.93 while GPT-3.5 yields 54.80, on S2, TAMA yields 28.60 while GPT-3.5 yields 27.75). These two datasets are comprised of diverse table understanding tasks, and the domain distribution is significantly different from all the in-domain training data we use. The competitive performance TAMA demonstrates on these two datasets indicates its strong general table understanding ability.

This suggests that while pre-training imparts a foundational understanding of table-related knowledge, table-specific fine-tuning plays a crucial role in further enhancing the model’s capability in handling table data.

TAMA preserves the general capabilities. Table 8 indicates that TAMA preserves the original LLaMA 3.1 8B Instruct’s performance on almost every general benchmark. For instance, on MMLU, TAMA yields an accuracy of 66.99 compared to the base model’s 66.04; on Al2ARC, TAMA yields an accuracy of 81.23 compared to the base model’s 80.89. We leave the discussion of the slight performance improvements on these general benchmarks to Section 4.3. On IFEval, TAMA preserves most of its instruction following ability compared to the base model (74.70 compared to the base model’s 79.62). Thanks to the strong instruction following ability of the original LLaMA 3.1 8B Instruct model, TAMA even yields a similar instruction following score on IFEval to GPT-3.5 (74.70 for TAMA compared to 74.80 for GPT-3.5). Table 7 provides two examples from TAMA’s predictions versus existing table LLMs’ on IFEval and Table-Syn (S1 in Table 6). Existing table LLMs fail to return their answers in JSON formats in most cases, while TAMA returns the correct format.

TAMA is data efficient. We highlight that for each dataset, we use 200 training instances, which is less than 5% of the size of the original training dataset. For instance, on HiTab, we use 2.67% of the original 7,417 training instances, and on TabFact, we use 0.21% of the original 92,283 training instances. In total, we use 2,600 table instruction-answer pairs. When tuned on such a limited number of training instances, with carefully selected hyperparameters, the model can still advance its table understanding ability while maintaining its general capabilities.

4.3 Hindsight Analysis

In hindsight, we want to validate that our selected hyperparameters indeed work the best. Therefore, we run the experiments on the same training set with the learning rate ranging from 1.0e-7 to 1.0e-5, and the number of epochs from one to six. Figure 3 reports part of the results, and Appendix D reports the complete results and provide further discussion.

As shown in Figure 3a, in the table understanding tasks, the learning rate of 1.0e-6 and 5.0e-7 yields the best overall performance, which coincides with our findings in Section 3. In addition, the model achieves its best aggregated performance around two to three epochs for both learning rate.

On S2, one of the out-of-domain table understanding datasets, the learning rate of 1.0e-6 maintains an overall best ROUGE-L score (around 28 to 29), and the learning rate of 5.0e-7 underperforms 1.0e-6, with the best ROUGE-L score of 23.64 achieved at the second epoch.

For MMLU, both 1.0e-6 and 5.0e-7 maintain their performance, sometimes even slightly better than the original LLaMA 3.1 8B Instruct model. As revealed in Table 9, the performance boost is more pronounced on STEM category. We hypothe-
<table>
  <tr>
    <th>PROMPT:</th>
    <th>Please provide ... in JSON format.</th>
    <th>Correct?</th>
  </tr>
  <tr>
    <td>TableLLaMA</td>
    <td>&lt;Mommy&gt;, &lt;Dad&gt; ... &lt;/s&gt;</td>
    <td>✗</td>
  </tr>
  <tr>
    <td>TableLLM</td>
    <td>...df = pd.read_csv('data.csv')...</td>
    <td>✗</td>
  </tr>
  <tr>
    <td>TableBenchLLM</td>
    <td>...1. Sarah Palin...</td>
    <td>✗</td>
  </tr>
  <tr>
    <td><i>TAMA</i> (ours)</td>
    <td>{"famous_moms": [{"name": ... }]</td>
    <td>✓</td>
  </tr>
  <tr>
    <th colspan="2">PROMPT: # Task Description: determine the semantic type ... Return in JSON format... [Table] [Candidates]...</th>
    <th>Correct?</th>
  </tr>
  <tr>
    <td>TableLLaMA</td>
    <td>&lt;Blue Blazer (mask)&gt;...&lt;/s&gt;</td>
    <td>✗</td>
  </tr>
  <tr>
    <td>TableLLM</td>
    <td>{"chosen_semantic_type": "Film"}</td>
    <td>✗</td>
  </tr>
  <tr>
    <td>TableBenchLLM</td>
    <td>...Loser (wager)*Let's consider...</td>
    <td>✗</td>
  </tr>
  <tr>
    <td><i>TAMA</i> (ours)</td>
    <td>{"chosen_semantic_type": "Wrestler"}</td>
    <td>✓</td>
  </tr>
</table>

Table 7: Table LLMs’ predictions on the prompts from IFEval and Table-Syn (S1 in Table 6). We omit parts of the examples for readability. Appendix E provides the complete examples.

<table>
  <tr>
    <th rowspan="2">Models</th>
    <th colspan="4">OOD Table</th>
    <th colspan="5">General</th>
  </tr>
  <tr>
    <th colspan="2">Test</th>
    <th colspan="2">Dev</th>
    <th colspan="3">Test</th>
  </tr>
  <tr>
    <th></th>
    <th>S1<sup>†</sup></th>
    <th>S2</th>
    <th>MMLU</th>
    <th>IFEval</th>
    <th>M<sub>pro</sub></th>
    <th>GPQA</th>
    <th>ARC</th>
  </tr>
  <tr>
    <td>Acc</td>
    <td>R-L</td>
    <td>Acc</td>
    <td>Acc</td>
    <td>Acc</td>
    <td>Acc</td>
    <td>Acc</td>
    <td>Acc</td>
  </tr>
  <tr>
    <td>GPT-3.5</td>
    <td>54.80</td>
    <td>27.75<sup>†</sup></td>
    <td>70.00<sup>†</sup></td>
    <td>74.80<sup>†</sup></td>
    <td>-</td>
    <td>29.80<sup>†</sup></td>
    <td>-</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td><b>80.20</b></td>
    <td><b>40.38<sup>†</sup></b></td>
    <td>86.40<sup>†</sup></td>
    <td>92.00<sup>†</sup></td>
    <td>63.71<sup>†</sup></td>
    <td>32.10<sup>†</sup></td>
    <td>-</td>
  </tr>
  <tr>
    <td>base</td>
    <td>53.60</td>
    <td>23.47<sup>†</sup></td>
    <td>66.04</td>
    <td>79.62</td>
    <td>22.10</td>
    <td>32.14</td>
    <td>80.89</td>
  </tr>
  <tr>
    <td><i>TAMA</i></td>
    <td>64.93</td>
    <td>28.60</td>
    <td>66.99</td>
    <td>74.70</td>
    <td>31.84</td>
    <td>31.92</td>
    <td>81.23</td>
  </tr>
</table>

Table 8: Evaluation results on the out-of-domain (OOD) table understanding benchmarks and general benchmarks. For the two out-of-domain table understanding datasets, we make the number bold if it is the best among the four, we underline the number if it is at the second place. M<sub>pro</sub> and ARC denote MMLU<sub>pro</sub> and AI2ARC, respectively. R-L denotes the ROUGE-L score. † indicates results reported by Achiam et al. (2023); Zhou et al. (2023); Rein et al. (2023); Wang et al. (2024); Wu et al. (2024), and the report from Apple†.

![Line plots showing performance scores (y-axis) with respect to the number of epochs (x-axis) and learning rates for five datasets: All table tasks, S2 (ROUGE-L), MMLU (Acc), IFEval (Acc), AI2ARC (Acc)](page_1012_1342_1022_340.png)

Figure 3: Performance scores (y-axis) with respect to the number of epochs (x-axis) and learning rates. In Figure 3a, we aggregate the performance scores for all the datasets listed in Table 5.

size that this is because table-related tasks typically involve data analysis that requires math reasoning, which belongs to the STEM category. Therefore, training on table-related tasks would lead to better
<table>
  <tr>
    <th></th>
    <th>STEM</th>
    <th>Social Science</th>
    <th>Humanities</th>
    <th>Others</th>
    <th>Overall</th>
  </tr>
  <tr>
    <td>base</td>
    <td>56.03</td>
    <td>76.15</td>
    <td>61.57</td>
    <td>72.27</td>
    <td>66.04</td>
  </tr>
  <tr>
    <td><i>TAMA</i></td>
    <td>58.25</td>
    <td>76.37</td>
    <td>62.42</td>
    <td>72.86</td>
    <td>66.99</td>
  </tr>
</table>

Table 9: Performance breakdown for the four categories in the MMLU dataset. The performance corresponds to the learning rate of 1.0e-6 and two training epochs.

STEM performance. This also explains the performance boost for MMLU<sub>pro</sub> in Table 8.

For IFEval, AI2ARC, the smaller the learning rate is, the less it affects the model’s general capabilities. For instance, on IFEval, at the smallest learning rate of 1.0e-7, the model maintains the base model’s performance, while 5.0e-7 and 1.0e-6 maintain most of the base model’s performance.

Generally, the trends we observe here resemble the trends we have observed in Section 3. A learning rate that is too large or too small would lead to suboptimal performance on table understanding tasks, and fine-tuning the model with one or two epochs would result in a competitive model without the risk of sacrificing its general capabilities. Moreover, we demonstrate here that with preliminary experiments, we can find a set of good or even the best hyperparameters to train the final model. Therefore, we encourage researchers to prioritize hyperparameter selection and conduct preliminary experiments when developing their models.

5 Related Work

Table Understanding Methods. The past decade has witnessed a paradigm shift in approaches to table understanding. Before the advent of LLMs, researchers typically adapt model structures to better interpret table data (Lebret et al., 2016; Liu et al., 2018; Yang et al., 2022). As language models demonstrate promising performance on various tasks (Devlin et al., 2019), researchers gradually shift their attention towards data-driven methods for table understanding. For instance, Yin et al. (2020); Herzig et al. (2020) pre-train BERT (Devlin et al., 2019) or BERT-derived model on large-volume of table data from sources such as Wikipedia to acquire better table representations. Xie et al. (2022) reveal the synergy effects of various structured tasks, including many table tasks, laying foundations to build a generalist model for structured data. In the era of LLMs, as LLMs possess innate table-understanding abilities, researchers also explore prompt engineering techniques to optimize LLMs for table tasks (Chang & Fosler-Lussier, 2023; Deng et al., 2024).

Table Instruction Tuning. Recently, researchers have increasingly focused on instruction tuning to enhance LLMs’ table understanding ability. As demonstrated by Touvron et al. (2023); Dubey et al. (2024); Chung et al. (2024); Su et al. (2024), instruction-tuning can improve model performance and generalization to unseen tasks. Meanwhile, models from the open-source LLaMA family (Touvron et al., 2023) demonstrate strong capabilities, leading researchers to instruction-tune these models for better table understanding. For instance, TableLLaMA (Zhang et al., 2024a) is instruction-tuned from a variant of LLaMA 2 model (Touvron et al., 2023), TableLLM (Zhang et al., 2024b) is instruction-tuned from CodeLLaMA, Wu et al. (2024) instruction-tune various foundational models such as LLaMA 3.1 (Dubey et al., 2024), resulting in their TableBenchLLM model. Moreover, Zheng et al. (2024a) treat tables as images and instruction-tune Vicuna (Chiang et al., 2023), a vision model that is originally fine-tuned from the LLaMA model, for table understanding. However, as revealed by Zheng et al. (2024a); Deng et al. (2024), treating tables as texts rather than images yields better performance. In this paper, we focus on table instruction tuning with tables fed as texts.

6 Conclusion

In this paper, we highlighted the limited out-of-domain table understanding ability and limited general capabilities of existing table LLMs. From our analysis, we found that the commonly adopted hyperparameters in existing table LLMs are suboptimal, and hyperparameter choices in table instruction tuning are crucial in shaping the model’s capabilities. Based on our analysis, we selected a set of hyperparameters and fine-tuned our own model, <i>TAMA</i>. Notably, as an 8B model, <i>TAMA</i> demonstrates strong table understanding ability, outperforming GPT-3.5 on most of the table understanding benchmarks, even achieving performance on par or better than GPT-4. Moreover, <i>TAMA</i> preserves strong general capabilities. We hope our findings and our model <i>TAMA</i> can facilitate future research on structured data.
Limitations and Future Directions

Scope of the Study. Due to the space constraint, we provide further analysis across LLMs in Appendix C.4, including Llama 2 7B Instruct (Touvron et al., 2023), QWen 2.5 7B Instruct (Bai et al., 2023), Mistral v0.3 7B Instruct (Jiang et al., 2023), and Phi 3 small 8K Instruct (7B) (Abdin et al., 2024), and analysis in terms of LoRA and QLoRA in Appendix C.5. We provide further analysis regarding how the data features affect the model’s performance degradation on general benchmarks in Appendix C.7. However, due to the scope of one study, we cannot exhaust every possible foundational model and every possible dataset.

Better Hyperparameter Selection Methods. While our paper effectively demonstrates the importance of hyperparameter selection, we primarily rely on manual hyperparameter tuning. We encourage future efforts on automated approaches to hyperparameter optimization, specifically tailored for balancing domain adaptation and general capabilities.

Better Data Selection Methods. Our work employs random sampling from datasets, which, while proving effective, suggests redundancy in current tableQA datasets. We encourage future efforts to investigate more principled data selection strategies based on diversity, difficulty, or information content to further improve efficiency.

Ethical Considerations

In our experiments, the datasets we use are meant to evaluate the models’ capabilities on handling tabular data as well as the model’s general capabilities. Therefore, we assume there is no ethical consideration within the scope of the datasets. During our exploration and the construction of our own model, we employ foundational models like LLaMA 3.1 SB Instruct (Dubey et al., 2024). These foundational models may be subject to jail breaking (Zou et al., 2023) or other malicious user behaviors. We advocate practitioners to follow the intended usage of these foundational models and the result models after further fine-tuning such as our model introduced in this paper.

Acknowledgements

We leverage the LLaMA Factory library for model training and inference (Zheng et al., 2024b). We thank the anonymous reviewers for their valuable feedback and suggestions. We thank Yulong Chen, Xin Liu, Jie Ruan, Jung-Chun Liu for their feedback on the initial draft.

This project was partially funded by a grant from OpenAI and a Microsoft Foundational Model grant. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of OpenAI or Microsoft.

References

Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219, 2024.

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shaymal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.

Rami Aly, Zhijiang Guo, Michael Sejr Schlichtkrull, James Thorne, Andreas Vlachos, Christos Christodoulopoulos, Oana Cocarascu, and Arpit Mittal. The fact extraction and VERification over unstructured and structured information (FEVEROUS) shared task. In Rami Aly, Christos Christodoulopoulos, Oana Cocarascu, Zhijiang Guo, Arpit Mittal, Michael Schlichtkrull, James Thorne, and Andreas Vlachos (eds.), Proceedings of the Fourth Workshop on Fact Extraction and VERification (FEVER), pp. 1–13, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.fever-1.1. URL https://aclanthology.org/2021.fever-1.1.

Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023.

Shuaichen Chang and Eric Fosler-Lussier. How to prompt llms for text-to-sql: A study in zero-shot, single-domain, and cross-domain settings. arXiv preprint arXiv:2305.11853, 2023.

Wenhui Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. Tabfact: A large-scale dataset for table-based fact verification. arXiv preprint arXiv:1909.02164, 2019.

Wenhui Chen, Hanwen Zha, Zhiyu Chen, Wenhan Xiong, Hong Wang, and William Yang Wang. HybridQA: A dataset of multi-hop question answering over tabular and textual data. In Trevor Cohn, Yulan He, and Yang
Liu (eds.), Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 1026–1036, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.findings-emnlp.91. URL https://aclanthology.org/2020.findings-emnlp.91.

Zhoujun Cheng, Haoyu Dong, Zhiruo Wang, Ran Jia, Jiaqi Guo, Yan Gao, Shi Han, Jian-Guang Lou, and Dongmei Zhang. HiTab: A hierarchical table dataset for question answering and natural language generation. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1094–1110, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.acl-long.78. URL https://aclanthology.org/2022.acl-long.78.

Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Liannmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna.lmsys.org (accessed 14 April 2023), 2(3):6, 2023.

Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetuned language models. Journal of Machine Learning Research, 25(70):1–53, 2024.

Peter Clark, Isaac Cowen, Owen Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv:1803.05457v1, 2018.

Naihao Deng, Zhenjie Sun, Ruiqi He, Aman Sikka, Yulong Chen, Lin Ma, Yue Zhang, and Rada Mihalcea. Tables as texts or images: Evaluating the table reasoning ability of LLMs and MLLMs. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics ACL 2024, pp. 407–426, Bangkok, Thailand and virtual meeting, August 2024. Association for Computational Linguistics. URL https://aclanthology.org/2024.findings-acl.23.

Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems, 36, 2024.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4171–4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1423. URL https://aclanthology.org/N19-1423.

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.

Mihail Eric and Christopher D Manning. Key-value retrieval networks for task-oriented dialogue. arXiv preprint arXiv:1705.05414, 2017.

Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Nan Duan, and Weizhu Chen. Critic: Large language models can self-correct with tool-interactive critiquing. arXiv preprint arXiv:2305.11738, 2023.

Vivek Gupta, Maitrey Mehta, Pegah Nokhz, and Vivek Srikumar. INFOTABS: Inference on tables as semi-structured data. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 2309–2324, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.acl-main.210. URL https://aclanthology.org/2020.acl-main.210.

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021.

Jonathan Herzig, Pawel Krzysztof Nowak, Thomas Müller, Francesco Piccinno, and Julian Eisenschlos. TaPas: Weakly supervised table parsing via pre-training. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4320–4333, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.acl-main.398. URL https://aclanthology.org/2020.acl-main.398.

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021.

Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamtford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023.

Yannis Katsis, Sameem Chemmengath, Vishwajeet Kumar, Samarth Bharadwaj, Mustafa Canim, Michael Glass, Alfio Gliozzo, Feifei Pan, Jaydeep Sen, Karthik Sankaranarayanan, and Soumen Chakrabarti. AIT-QA: Question answering dataset over complex tables in the airline industry. In Anastassia Loukina,
Rashmi Gangadharai, and Bonan Min (eds.), Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Industry Track, pp. 305–314. Hybrid: Seattle, Washington + Online, July 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.naacl-industry.34. URL https://aclanthology.org/2022.nacl-industry.34.

Rémi Lebret, David Grangier, and Michael Auli. Neural text generation from structured data with application to the biography domain. In Jian Sha, Kevin Duh, and Xavier Carreras (eds.), Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 1203–1213, Austin, Texas, November 2016. Association for Computational Linguistics. doi: 10.18653/v1/D16-1128. URL https://aclanthology.org/D16-1128.

Peng Li, Yeye He, Dror Yashar, Weiwei Cui, Song Ge, Haidong Zhang, Danielle Rifinski Fanman, Dongmei Zhang, and Surajit Chaudhuri. Table-gpt: Table-tuned gpt for diverse table tasks. arXiv preprint arXiv:2310.09263, 2023.

Tianyu Liu, Kexiang Wang, Lei Sha, Baobao Chang, and Zhifang Sui. Table-to-text generation by structure-aware seq2seq learning. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.

Pan Lu, Liang Qiu, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, Tanmay Rajpurkar, Peter Clark, and Ashwin Kalyan. Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning. In International Conference on Learning Representations (ICLR), 2023.

Linyong Nan, Chiachun Hsieh, Ziming Mao, Xi Victoria Lin, Neha Verma, Rui Zhang, Wojciech Kryściński, Hailey Schoelkopf, Riley Kong, Xiangru Tang, Mutethia Mutuma, Ben Rosand, Isabel Trindade, Renusree Bandaru, Jacob Cunningham, Caiming Xiong, Dragomir Radev, and Dragomir Radev. Fe-TaQA: Free-form table question answering. Transactions of the Association for Computational Linguistics, 10:35–49, 2022. doi: 10.1162/tacl_a_00446. URL https://aclanthology.org/2022.tacl-1.3.

Ankur Parikh, Xuezhi Wang, Sebastian Gehrmann, Manaal Faruqui, Bhuwan Dhingra, Diyi Yang, and Dipanjan Das. ToTTTo: A controlled table-to-text generation dataset. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1173–1186, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.emnlp-main.89. URL https://aclanthology.org/2020.emnlp-main.89.

Panupong Pasupat and Percy Liang. Compositional semantic parsing on semi-structured tables. In Chengqing Zong and Michael Strube (eds.), Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 1470–1480, Beijing, China, July 2015. Association for Computational Linguistics. doi: 10.3115/v1/P15-1142. URL https://aclanthology.org/P15-1142.

David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark. arXiv preprint arXiv:2311.12022, 2023.

Pragya Srivastava, Manuj Malik, and Tanuja Ganu. Assessing llms’ mathematical reasoning in financial document question answering. arXiv preprint arXiv:2402.11194, 2024.

Aofeng Su, Aowen Wang, Chao Ye, Chen Zhou, Ga Zhang, Guangcheng Zhu, Haobo Wang, Haokai Xu, Hao Chen, Haoze Li, et al. Tablegpt2: A large multimodal model with tabular data integration. arXiv preprint arXiv:2411.02059, 2024.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.

Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyuan Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. arXiv preprint arXiv:2406.01574, 2024.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022.

Xianjie Wu, Jian Yang, Linzheng Chai, Ge Zhang, Jiahein Liu, Xinrun Du, Di Liang, Daixin Shu, Xianfu Cheng, Tianzhen Sun, et al. Tablebench: A comprehensive and complex benchmark for table question answering. arXiv preprint arXiv:2408.09174, 2024.

Tianbao Xie, Chen Henry Wu, Peng Shi, Ruqi Zhong, Torsten Scholak, Michihiro Yasunaga, Chien-Sheng Wu, Ming Zhong, Pengcheng Yin, Sida I. Wang, Victor Zhong, Bailin Wang, Chengzu Li, Connor Boyle, Ansong Ni, Ziyu Yao, Dragomir Radev, Caiming Xiong, Lingpeng Kong, Rui Zhang, Noah A. Smith, Luke Zettlemoyer, and Tao Yu. Unified-SKG: Unifying and multi-tasking structured knowledge grounding with text-to-text language models. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 602–631, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.
doi: 10.18653/v1/2022.emnlp-main.39. URL https://aclanthology.org/2022.emnlp-main.39.

Jingfeng Yang, Aditya Gupta, Shyam Upadhyay, Luheng He, Rahul Goel, and Shachi Paul. TableFormer: Robust transformer modeling for table-text encoding. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 528–537, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.acl-long.40. URL https://aclanthology.org/2022.acl-long.40.

Pengcheng Yin, Graham Neubig, Wen-tau Yih, and Sebastian Riedel. TaBERT: Pretraining for joint understanding of textual and tabular data. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 8413–8426, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.acl-main.745. URL https://aclanthology.org/2020.acl-main.745.

Liangyu Zha, Junlin Zhou, Liyao Li, Rui Wang, Qingyi Huang, Saisai Yang, Jing Yuan, Changbao Su, Xiang Li, Aofeng Su, et al. Tablegpt: Towards unifying tables, nature language and commands into one gpt. arXiv preprint arXiv:2307.08674, 2023.

Tianshu Zhang, Xiang Yue, Yifei Li, and Huan Sun. TableLlama: Towards open large generalist models for tables. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 6024–6044, Mexico City, Mexico, June 2024a. Association for Computational Linguistics. doi: 10.18653/v1/2024.naacl-long.335. URL https://aclanthology.org/2024.naacl-long.335.

Xiaokang Zhang, Jing Zhang, Zeyao Ma, Yang Li, Bohan Zhang, Guanlin Li, Zijun Yao, Kangli Xu, Jinchang Zhou, Daniel Zhang-Li, et al. Tablellm: Enabling tabular data manipulation by llms in real office usage scenarios. arXiv preprint arXiv:2403.19318, 2024b.

Mingyu Zheng, Xinwei Feng, Qingyi Si, Qiaoqiao She, Zheng Lin, Wenbin Jiang, and Weiping Wang. Multimodal table understanding. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 9102–9124, Bangkok, Thailand, August 2024a. Association for Computational Linguistics. URL https://aclanthology.org/2024.acl-long.493.

Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, and Zheyuan Luo. LlamaFactory: Unified efficient fine-tuning of 100+ language models. In Yixin Cao, Yang Feng, and Deyi Xiong (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), pp. 400–410, Bangkok, Thailand, August 2024b. Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-demos.38. URL https://aclanthology.org/2024.acl-demos.38/.

Victor Zhong, Caiming Xiong, and Richard Socher. Seq2sql: Generating structured queries from natural language using reinforcement learning. arXiv preprint arXiv:1709.00103, 2017.

Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36, 2024.

Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911, 2023.

Fengbin Zhu, Wenqiang Lei, Youcheng Huang, Chao Wang, Shuo Zhang, Jiancheng Lv, Fuli Feng, and Tat-Seng Chua. TAT-QA: A question answering benchmark on a hybrid of tabular and textual content in finance. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 3277–3287, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.acl-long.254. URL https://aclanthology.org/2021.acl-long.254.

Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043, 2023.
Table 10: Temperature and top_p value for table LLMs.

<table>
  <tr>
    <th>Model</th>
    <th>Temperature</th>
    <th>Top_p</th>
  </tr>
  <tr>
    <td>TableLLaMA</td>
    <td>0.6</td>
    <td>0.90</td>
  </tr>
  <tr>
    <td>TableLLM</td>
    <td>0.8</td>
    <td>0.95</td>
  </tr>
  <tr>
    <td>TableBenchLLM</td>
    <td>0.0</td>
    <td>0.95</td>
  </tr>
  <tr>
    <td><i>TAMA</i> (ours)</td>
    <td>0.01</td>
    <td>0.95</td>
  </tr>
</table>

A  Experiment Details

A.1 GPU Details
We run our experiments on 1 server node with 4 A40, each with 48 GB GPU memory, and 1 server node with 8 A100, each with 48 GB GPU memory.

A.2 Generation Details.
Table 10 shows the generation hyperparameters for table LLMs.

A.3 Batch Sizes
We conduct preliminary experiments to evaluate the impact of batch size on model performance, testing batch sizes of 4, 8, 16, 32, and 64. These experiments are performed on a total of 600 training instances from FeTaQA, HiTab, and TabFact (200 from each dataset, respectively as detailed in Section 3). We train the Llama 3.1 8B Instruct model for three epochs with a learning rate of 1e-6.

Our results indicate no significant performance variation across batch sizes on table tasks (e.g., TabFact) and general benchmarks (MMLU, IFEval). However, at batch size 64, we observe a substantial drop in BLEU score to 15.84, whereas all other batch sizes achieve BLEU scores above 30. With only 600 training instances, a batch size of 64 leads to approximately 25 parameter updates. We hypothesize that each update is based on a highly aggregated gradient that lacks sufficient variation, therefore the model may struggle to learn meaningful representations.

Based on these findings, we select a batch size of 16 for our experiments in Sections 3 and 4.

A.4 Details of Prompting GPT Models
We prompt the GPT-3.5-turbo and GPT-4-turbo models and set the temperature to 0.

A.5 Details of Data Processing
We follow the format of the dataset if the dataset is used by Zhang et al. (2024a). We add instructions for the datasets used by Xie et al. (2022). For datasets not used by Zhang et al. (2024a); Xie et al. (2022), we process them from their original source, and add an instruction per dataset.

A.6 Details of Evaluation
For datasets such as WikiTQ, TATQA, we follow their original evaluation scripts. For datasets such as WikiSQL, we follow Xie et al. (2022); Zhang et al. (2024a) to evaluate the exact match accuracy. For datasets such as ToTTo and FeTaQA, we follow Xie et al. (2022) and use the SacreBLEU loaded from the Hugging Face library to calculate the BLEU-4 score. For ToTTo, following Xie et al. (2022), we calculate the BLEU-4 score given all the references in the test set. For S2, we report the ROUGE-L following Wu et al. (2024) loaded from the Hugging Face library.

For MMLU, MMLU<sub>pro</sub>, AI2ARC and GPQA, our objective is to select the most appropriate completion among a set of given options based on the provided context. Following Touvron et al. (2023), we select the completion with the highest likelihood given the provided context. As we evaluate the model based on their selection of choice “A”, “B”, etc. We do not normalize the likelihood by the number of characters in the completion. We note that our setup for MMLU<sub>pro</sub> is different from the chain-of-thought (CoT) (Wei et al., 2022) setup in the original LLaMA 3.1 report, as many of the existing table LLMs exhibit poor instruction-following ability, making it challenging to evaluate their performance through generation-based tasks. For IFEval, we report the instance-level strict accuracy defined by Zhou et al. (2023), which reports the percentage of verifiable instructions that are followed.

B  Evaluation of the Existing Table LLMs.

MMLU Performance Breakdown in Terms of Categories. We provide the performance breakdown in terms of the category for MMLU in Table 11.

On STEM subjects, TableLLaMA experiences a decline of 7.05, while TableLLM and TableBenchLLM drop by 5.40 and 7.36, respectively. STEM subjects, including abstract algebra and mathematics at various levels (elementary, high school, and college), typically require strong logical reasoning and analytical capabilities, which are highly relevant to data analysis in table tasks. The drop in performance across these models indicates that current table instruction tuning compromises such reasoning abilities of their base models, limiting
their application in table analytical scenarios.

There is even more pronounced performance degradation in other categories. Though these categories may not directly align with table understanding, they assess model capabilities that are still critical for end-user applications. For instance, the “Others” category includes subjects like global facts, which are essential for users seeking reliable information during queries. The decline in performance across these broader categories suggests that the current table instruction tuning methods may compromise the model’s ability to handle general knowledge tasks effectively, which limits its practical usefulness for diverse real-world applications.

C Model and Hyperparameter Exploration

C.1 Model Selection

Reasons to Select LLaMA 3.1. LLaMA 3.1 (Dubey et al., 2024) provides a set of foundational models for language. Compared to the prior LLaMA models, LLaMA 3.1 claims to improve both the quantity and the quality of the data used for pre-training and post-training (15T multilingual pre-training tokens for LLaMA 3.1 compared to 1.8T tokens for LLaMA 2). Such an enormous amount of training makes LLaMA 3.1 one of the most advanced open-source LLMs.

Reasons to Select the Instruct Version Rather than the Base Version. Currently, there are two kinds of model selections for table instruction tuning, instruction-tuning the base version of the model, as seen in works like TableLLaMA(Zhang et al., 2024a) and TableBenchLLM(Wu et al., 2024), or continuing instruction-tuning an already instruction-tuned version, as done with TableLLM(Zhang et al., 2024b) as listed in Table 1.

As the end user may come up with their own set of instructions, we expect table instruction-tuned models to possess a strong general instruction-following ability. Imparting general instruction-following ability through table instruction-tuning to the base model is challenging, as there is a lack of diversity in the table instruction-tuning data. For instance, TableLLaMA employs six specific instruction templates across two million data points, which pales in comparison to the diverse instruction datasets in broader instruction tuning efforts such as those by Chung et al. (2024), which include 1,836 tasks, each with a set of instruction templates. As shown in Figure 4c, when tuning the base version of the LLaMA 3.1 8B model on instruction pairs on FeTaQA, HiTab, and TabFact, the instruction following ability of the model does not improve significantly. Moreover, with a large learning rate such as 1.0e-5, the model’s instruction following ability drops significantly when there is more training data coming in.

We argue that the instruction-tuned version possesses strong general instruction-following capabilities, eliminating the need to repeat the general instruction-tuning stage. Therefore, a more effective strategy is to table instruction-tune an already instruction-tuned model, focusing on enhancing its table understanding ability while preserving its general instruction-following capabilities. As shown in Figure 2f, with proper hyperparameter selection, we can maintain the inherent strong instruction following ability of the LLaMA 3.1 8B Instruct model.

In terms of specific table understanding tasks, tuning LLaMA 3.1 8B Instruct model yields better performance than its base version on TabFact (73.10 in Figure 2a v.s. 71.10 in Figure 4a) under the same experimental setup. Therefore, we select the LLaMA 3.1 8B Instruct model as our starting model.

C.2 Effects of Epochs

Figure 5 illustrates the relationship between the performance of LLaMA 3.1 8B Instruct model and the number of epochs when we fine-tune the model on the 1,500 instruction pairs at a learning rate of 1.0e-6. The model demonstrates a decent performance on these table tasks within just one or two epochs. In the meantime, the model mostly preserves its performance on MMLU and IFEval, indicating that its general capabilities are not compromised too much while acquiring table reasoning ability. Beyond this point, there is no significant performance improvement, suggesting that extending training for more epochs yields diminishing returns or may even lead to overfitting. Therefore, we choose to train our model for two epochs in Section 4 instead of the commonly adopted six epochs by existing table LLMs as seen in Table 1.

C.3 Effects of Multi-Task

In Figure 6, we present the heatmap of model performance when fine-tuning the LLaMA 3.1 8B Instruct model on a single dataset (one of the datasets among FeTaQA, HiTab, and TabFact). We fine-
Table 11: Performance (accuracy scores) comparison between existing table LLMs (second row) and their base models (first row) with respect to the four categories in MMLU (e.g. “STEM” column) and their overall MMLU performance (“Overall” column). †: A variant of LLaMA 2 7B model.

<table>
  <tr>
    <th></th>
    <th>STEM</th>
    <th>Social Science</th>
    <th>Humanities</th>
    <th>Others</th>
    <th>Overall</th>
  </tr>
  <tr>
    <td>LongLoRA 7B†</td>
    <td>35.65</td>
    <td>50.70</td>
    <td>40.66</td>
    <td>51.20</td>
    <td>44.22</td>
  </tr>
  <tr>
    <td>TableLLaMA</td>
    <td>28.60</td>
    <td>31.49</td>
    <td>29.59</td>
    <td>31.65</td>
    <td>30.27</td>
  </tr>
  <tr>
    <td>△</td>
    <td>↓ 7.05</td>
    <td>↓ 19.21</td>
    <td>↓ 11.07</td>
    <td>↓ 19.55</td>
    <td>↓ 13.95</td>
  </tr>
  <tr>
    <td>CodeLLaMA 13B Instruct</td>
    <td>37.57</td>
    <td>50.24</td>
    <td>42.64</td>
    <td>49.05</td>
    <td>44.69</td>
  </tr>
  <tr>
    <td>TableLLM</td>
    <td>32.17</td>
    <td>39.52</td>
    <td>34.77</td>
    <td>37.57</td>
    <td>35.90</td>
  </tr>
  <tr>
    <td>△</td>
    <td>↓ 5.40</td>
    <td>↓ 10.72</td>
    <td>↓ 7.87</td>
    <td>↓ 11.48</td>
    <td>↓ 8.79</td>
  </tr>
  <tr>
    <td>LLaMA 3.1-8B</td>
    <td>52.85</td>
    <td>73.94</td>
    <td>55.43</td>
    <td>69.06</td>
    <td>62.08</td>
  </tr>
  <tr>
    <td>TableBenchLLM</td>
    <td>45.49</td>
    <td>62.56</td>
    <td>46.18</td>
    <td>59.38</td>
    <td>52.67</td>
  </tr>
  <tr>
    <td>△</td>
    <td>↓ 7.36</td>
    <td>↓ 11.38</td>
    <td>↓ 9.25</td>
    <td>↓ 9.68</td>
    <td>↓ 9.41</td>
  </tr>
</table>

![Line plots showing accuracy scores for TabFact, MMLU, and IFEval across epochs](page_101_1012_1242_312.png)

Figure 4: LLaMA 3.1 8B’s accuracy scores (y-axis) on TabFact, MMLU, and IFEval with respect to the number of training instances (x-axis). We fine-tune the model for three epochs.

tune the model for two epochs at a learning rate of 1.0e-6 with 500 instruction pairs, and then test it against the six datasets. Additionally, Figures 7 and 8 in Appendix C.6 present heatmaps across varying learning rates (from 1.0e-7 to 1.0e-5) and number of epochs (from one to six).

There are synergy effects on these tasks. The model achieves better performance when trained on the instruction pairs combined from all three datasets, compared to being trained on each of them separately. For instance, the accuracy on HiTab increases to 66.29, compared to 64.84 when trained only on HiTab as shown in Figure 7.

There are inter-connections between different tasks. In Figure 6, we note that fine-tuning solely on HiTab leads to a performance of 67.80 on TabFact, and fine-tuning solely on TabFact leads to a performance of 55.62 on HiTab, demonstrating a transfer of learned capabilities between these two tasks. However, this relationship is not universal as training on HiTab yields poor performance on FeTaQA, indicating that the overlap between certain tasks may be limited.

Based on these observations, we choose to fine-tune our model on a diverse range of tasks and datasets in Section 4.

C.4 Hyperparameter Exploration Across Models

We conduct experiments to validate our findings across different models in the full-parameter setup, including Llama 2 7B Instruct (Touvron et al., 2023), QWen 2.5 7B Instruct (Bai et al., 2023), Mistral v0.3 7B Instruct (Jiang et al., 2023), and Phi 3 small 8K Instruct (7B) (Abdin et al., 2024).

Learning Rate. We train each model on 500 examples from HiTab, FeTaQA, and TabFact (1,500 examples total) to explore the effects of the learning rate. Table 12 presents our results.

We observe a significant performance drop happens for every model on the two general benchmarks. Interestingly, for models such as QWen 2.5, when we increase the learning rate from 1.0e-6 to 5.0e-6, it would primarily affect the IFEval dataset rather than MMLU, suggesting that the compromises may happen at different speeds with respect
Figure 5: LLaMA 3.1 8B Instruct model’s performance (y-axis) across different numbers of epochs (x-axis). We fine-tune the model on the 1,500 instruction pairs, with 500 pairs each from FeTaQA, HiTab, and TabFact, at a learning rate of 1.0e-6.

Figure 6: Heatmap when we fine-tune the LLaMA 3.1 8B Instruct model on a single dataset (y-axis) and test against the others (x-axis). In this plot, we fine-tune the model for two epochs at a learning rate of 1.0e-6 with 500 instruction pairs.

to different aspects of the model’s general capability.

The Phi model shows a pronounced performance drop from 1.0e-5 to 5.0e-5, in contrast to Llama, Mistral and QWen models, where the “breakdown point” on the learning rate is slightly smaller, especially for Mistral model, where we see 5 points lose on IFEval from 5.0e-7 to 1.0e-6.

Table 4 lists the learning rate we would suggest for practitioners to use if they would fine-tune the LLMs on table-specific tasks.

Number of Examples. We further experiment with various training sizes for each model to observe its impact on performance. Table 13 reports the results for Llama 2 7B, QWen 2.5, Mistral v0.3, and Phi 3 8K models at one of the learning rates we select based on our results in Table 12.

<table>
  <tr>
    <th>Learning Rate</th>
    <th>FeTaQA</th>
    <th>TabFact</th>
    <th>MMLU</th>
    <th>IFEval</th>
  </tr>
  <tr>
    <th colspan="5">Llama 2 7B Instruct</th>
  </tr>
  <tr>
    <td>5.0e-7</td>
    <td>26.54</td>
    <td>52.63</td>
    <td>47.12</td>
    <td>47.84</td>
  </tr>
  <tr>
    <td>1.0e-6</td>
    <td>29.03</td>
    <td>53.80</td>
    <td>47.07</td>
    <td><b>47.84</b></td>
  </tr>
  <tr>
    <td>5.0e-6</td>
    <td>33.86</td>
    <td>51.05</td>
    <td>46.58</td>
    <td><b>35.25</b></td>
  </tr>
  <tr>
    <td>1.0e-5</td>
    <td>34.77</td>
    <td>53.79</td>
    <td>45.99</td>
    <td>39.93</td>
  </tr>
  <tr>
    <th colspan="5">QWen 2.5 7B Instruct</th>
  </tr>
  <tr>
    <td>5.0e-7</td>
    <td>33.14</td>
    <td>71.09</td>
    <td>73.66</td>
    <td>76.02</td>
  </tr>
  <tr>
    <td>1.0e-6</td>
    <td>34.50</td>
    <td>72.66</td>
    <td>73.52</td>
    <td><b>75.78</b></td>
  </tr>
  <tr>
    <td>5.0e-6</td>
    <td>34.04</td>
    <td>72.81</td>
    <td>73.81</td>
    <td><b>49.28</b></td>
  </tr>
  <tr>
    <td>1.0e-5</td>
    <td>33.84</td>
    <td>71.51</td>
    <td>73.49</td>
    <td>41.61</td>
  </tr>
  <tr>
    <th colspan="5">Mistral v0.3 7B Instruct</th>
  </tr>
  <tr>
    <td>1.0e-7</td>
    <td>31.91</td>
    <td>64.32</td>
    <td>61.32</td>
    <td>62.83</td>
  </tr>
  <tr>
    <td>5.0e-7</td>
    <td>36.44</td>
    <td>70.35</td>
    <td>60.76</td>
    <td>57.79</td>
  </tr>
  <tr>
    <td>1.0e-6</td>
    <td>36.99</td>
    <td>71.88</td>
    <td>60.45</td>
    <td><b>52.28</b></td>
  </tr>
  <tr>
    <td>5.0e-6</td>
    <td>35.71</td>
    <td>53.64</td>
    <td>34.96</td>
    <td><b>33.09</b></td>
  </tr>
  <tr>
    <td>1.0e-5</td>
    <td>32.14</td>
    <td>50.87</td>
    <td>24.93</td>
    <td>27.70</td>
  </tr>
  <tr>
    <th colspan="5">Phi 3 SK Instruct (7B)</th>
  </tr>
  <tr>
    <td>1.0e-6</td>
    <td>33.10</td>
    <td>72.04</td>
    <td>70.48</td>
    <td>71.22</td>
  </tr>
  <tr>
    <td>5.0e-6</td>
    <td>37.26</td>
    <td>73.82</td>
    <td>74.89</td>
    <td>68.71</td>
  </tr>
  <tr>
    <td>1.0e-5</td>
    <td>38.13</td>
    <td>73.92</td>
    <td>73.30</td>
    <td><b>62.95</b></td>
  </tr>
  <tr>
    <td>5.0e-5</td>
    <td>34.46</td>
    <td>50.90</td>
    <td>49.08</td>
    <td><b>28.78</b></td>
  </tr>
  <tr>
    <td>1.0e-4</td>
    <td>30.66</td>
    <td>50.33</td>
    <td>49.17</td>
    <td>23.02</td>
  </tr>
</table>

Table 12: LLMs’ performance scores corresponding to different learning rate. In this experiment, we train each model on 500 examples from HiTab, FeTaQA, and TabFact (1,500 examples total) for three epochs.

Across all models, performance improvement becomes marginal from 600 to 1500 examples, suggesting diminishing returns with larger datasets.

In addition, we find that given the same number of training instances, Llama 3.1 8B Instruct achieves better performance than Llama 2 7B Instruct. For instance, when trained with the same 1,500 examples at the learning rate of 1.0e-6, Llama 3.1 8B Instruct yields 73.10 on TabFact (Section 3) while Llama 2 7B Instruct only yields 53.80 (Table 13). Therefore, models with stronger general capabilities require less tuning data in our fine-tuning process.

C.5 Hyperparameter Exploration for LoRA and QLoRA

We conduct experiments using LoRA (Hu et al., 2021) and QLoRA (Dettmers et al., 2024) based on Llama 3.1-8B-Instruct. Specifically, we use hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4 2 as the base model for our QLoRA experiments.

We replicate the experiments we conduct in Ap-

2https://huggingface.co/hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4
<table>
  <tr>
    <th># Size</th>
    <th>FeTaQA</th>
    <th>TabFact</th>
    <th>MMLU</th>
    <th>IFEval</th>
  </tr>
  <tr>
    <th colspan="5">Llama 2 7B Instruct (1.0e-6)</th>
  </tr>
  <tr><td>30</td><td>13.32</td><td>31.68</td><td>47.07</td><td>45.08</td></tr>
  <tr><td>90</td><td>13.86</td><td>49.51</td><td>46.96</td><td>46.16</td></tr>
  <tr><td>150</td><td>14.79</td><td>46.24</td><td>47.09</td><td>47.48</td></tr>
  <tr><td>300</td><td>14.47</td><td>50.27</td><td>47.09</td><td>45.56</td></tr>
  <tr><td>600</td><td>24.12</td><td>50.74</td><td>47.11</td><td>45.56</td></tr>
  <tr><td>1500</td><td>29.03</td><td>53.80</td><td>47.07</td><td>47.84</td></tr>
  <tr>
    <th colspan="5">Qwen 2.5 7B Instruct (1.0e-6)</th>
  </tr>
  <tr><td>30</td><td>14.2</td><td>8.42</td><td>73.91</td><td>70.43</td></tr>
  <tr><td>90</td><td>16.45</td><td>8.47</td><td>73.76</td><td>70.43</td></tr>
  <tr><td>150</td><td>21.14</td><td>69.66</td><td>73.83</td><td>69.5</td></tr>
  <tr><td>300</td><td>22.1</td><td>69.65</td><td>73.72</td><td>68.95</td></tr>
  <tr><td>600</td><td>32.12</td><td>70.86</td><td>73.71</td><td>68.21</td></tr>
  <tr><td>1500</td><td>34.5</td><td>72.66</td><td>73.52</td><td>66.73</td></tr>
  <tr>
    <th colspan="5">Mistral v0.3 7B Instruct (5.0e-6)</th>
  </tr>
  <tr><td>30</td><td>23.84</td><td>0.28</td><td>61.39</td><td>49.72</td></tr>
  <tr><td>90</td><td>10.67</td><td>60.29</td><td>61.34</td><td>51.76</td></tr>
  <tr><td>150</td><td>19.79</td><td>49.82</td><td>61.34</td><td>52.87</td></tr>
  <tr><td>300</td><td>33.93</td><td>61.91</td><td>61.13</td><td>51.02</td></tr>
  <tr><td>600</td><td>34.28</td><td>66.34</td><td>61.12</td><td>52.31</td></tr>
  <tr><td>1500</td><td>36.44</td><td>70.35</td><td>60.76</td><td>47.69</td></tr>
  <tr>
    <th colspan="5">Plu 3 8K Instruct (7B) (5.0e-6)</th>
  </tr>
  <tr><td>30</td><td>17.19</td><td>9.62</td><td>75.43</td><td>52.31</td></tr>
  <tr><td>90</td><td>24.01</td><td>67.32</td><td>75.43</td><td>63.96</td></tr>
  <tr><td>150</td><td>24.67</td><td>68.00</td><td>75.43</td><td>62.11</td></tr>
  <tr><td>300</td><td>34.81</td><td>71.30</td><td>75.61</td><td>62.85</td></tr>
  <tr><td>600</td><td>37.74</td><td>72.91</td><td>75.50</td><td>61.18</td></tr>
  <tr><td>1500</td><td>37.26</td><td>73.82</td><td>75.26</td><td>59.70</td></tr>
</table>

Table 13: LLMs’ performance scores corresponding to different sizes of the training data. We specify the learning rate we use for each model in the bracket next to the model names. Here we train each model for three epochs.

pendix C.4, and here we present our results in two aspects, the learning rate and the number of examples.

Learning Rate. Table 14 presents the results. We find that there is still a “breakdown point” where further increasing the learning rate causes a sharp decline in overall performance for both LoRA and QLoRA. However, such “breakdown point” for LoRA and QLoRA (around 5.0e-5) is larger than the full parameter tuning (usually around 1.0e-6). When the learning rate does not surpass such a “breakdown point”, both methods demonstrate competitive in-domain performance on table tasks.

Number of Examples. Table 15 presents the results. Similar to what we have found for full parameter fine-tuning, both LoRA and QLoRA show diminishing returns as the number of training examples increases. While performance improves with more examples, the rate of improvement slows beyond 600 examples for LoRA. For QLoRA, the rate of improvement slows beyond 90 examples. We find that with 1,500 examples, QLoRA and LoRA perform similarly on the in-domain table tasks, and on FeTaQA, QLoRA even outperforms LoRA by 1 point. This suggests that practitioners may leverage such parameter-efficient fine-tuning methods like QLoRA in practice, especially when they have limited table data.

<table>
  <tr>
    <th>Learning Rate</th>
    <th>FeTaQA</th>
    <th>TabFact</th>
    <th>MMLU</th>
    <th>IFEval</th>
  </tr>
  <tr>
    <th colspan="5">LoRA</th>
  </tr>
  <tr><td>1.0e-6</td><td>16.63</td><td>63.21</td><td>66.06</td><td>80.22</td></tr>
  <tr><td>5.0e-6</td><td>23.69</td><td>66.80</td><td>65.97</td><td>80.94</td></tr>
  <tr><td>1.0e-5</td><td>29.66</td><td>68.58</td><td>66.03</td><td>80.58</td></tr>
  <tr><td>5.0e-5</td><td>35.33</td><td>73.80</td><td>67.04</td><td>76.98</td></tr>
  <tr><td>1.0e-4</td><td>35.81</td><td>75.63</td><td>67.42</td><td>71.22</td></tr>
  <tr><td>5.0e-4</td><td>36.04</td><td>73.88</td><td>66.36</td><td>60.67</td></tr>
  <tr><td>1.0e-3</td><td>35.54</td><td>73.64</td><td>59.02</td><td>38.73</td></tr>
  <tr>
    <th colspan="5">QLoRA</th>
  </tr>
  <tr><td>1.0e-7</td><td>20.36</td><td>63.06</td><td>64.56</td><td>80.22</td></tr>
  <tr><td>5.0e-7</td><td>19.07</td><td>66.42</td><td>64.68</td><td>80.46</td></tr>
  <tr><td>1.0e-6</td><td>27.44</td><td>67.18</td><td>64.68</td><td>79.98</td></tr>
  <tr><td>5.0e-6</td><td>34.64</td><td>70.98</td><td>64.76</td><td>78.66</td></tr>
  <tr><td>1.0e-5</td><td>36.86</td><td>73.20</td><td>65.22</td><td>77.58</td></tr>
  <tr><td>5.0e-5</td><td>36.52</td><td>74.11</td><td>65.76</td><td>76.02</td></tr>
  <tr><td>1.0e-4</td><td>35.94</td><td>74.91</td><td>65.76</td><td>74.22</td></tr>
  <tr><td>5.0e-4</td><td>33.72</td><td>50.50</td><td>42.76</td><td>32.85</td></tr>
  <tr><td>1.0e-3</td><td>0.01</td><td>50.16</td><td>22.95</td><td>23.86</td></tr>
</table>

Table 14: Performance scores corresponding to using LoRA and QLoRA. In this experiment, we train each model on 500 examples from HiTab, FeTaQA, and TabFact (1,500 examples total) for three epochs.

C.6 Individual Task’s Influence on Model Performance

Figures 7 and 8 present heatmaps across varying learning rates (from 1.0e-7 to 1.0e-5) and epochs (from one to six). We can see that the patterns coincide with what we have discussed in Section 3, that a learning rate that is too large such as 1.0e-5 or too small such as 1.0e-7 leads to suboptimal table understanding ability, and the large learning rate also compromises the model’s general capabilities. Moreover, we do not observe significant performance gain when we fine-tune the model for more epochs. Across these hyperparameters, we can observe the inter-connections between tasks such as HiTab and TabFact, as training solely on one often leads to good performance on the other. But this is not universally true, as tasks such as FeTaQA and FEVEROUS seem to not have strong
<table>
  <tr>
    <th># Size</th>
    <th>FeTaQA</th>
    <th>TabFact</th>
    <th>MMLU</th>
    <th>IFEval</th>
  </tr>
  <tr>
    <th colspan="5">LoRA (5.0e-5)</th>
  </tr>
  <tr>
    <td>30</td>
    <td>17.36</td>
    <td>63.89</td>
    <td>66.14</td>
    <td>71.90</td>
  </tr>
  <tr>
    <td>90</td>
    <td>19.83</td>
    <td>66.50</td>
    <td>66.03</td>
    <td>70.98</td>
  </tr>
  <tr>
    <td>150</td>
    <td>14.69</td>
    <td>68.62</td>
    <td>66.10</td>
    <td>73.01</td>
  </tr>
  <tr>
    <td>300</td>
    <td>26.01</td>
    <td>67.96</td>
    <td>66.20</td>
    <td>72.09</td>
  </tr>
  <tr>
    <td>600</td>
    <td>34.08</td>
    <td>72.13</td>
    <td>66.65</td>
    <td>70.61</td>
  </tr>
  <tr>
    <td>1500</td>
    <td>35.33</td>
    <td>73.80</td>
    <td>67.04</td>
    <td>68.39</td>
  </tr>
  <tr>
    <th colspan="5">QLoRA (5.0e-5)</th>
  </tr>
  <tr>
    <td>30</td>
    <td>18.02</td>
    <td>66.55</td>
    <td>64.78</td>
    <td>72.46</td>
  </tr>
  <tr>
    <td>90</td>
    <td>35.33</td>
    <td>68.44</td>
    <td>65.08</td>
    <td>69.32</td>
  </tr>
  <tr>
    <td>150</td>
    <td>33.50</td>
    <td>69.78</td>
    <td>65.36</td>
    <td>74.31</td>
  </tr>
  <tr>
    <td>300</td>
    <td>35.95</td>
    <td>69.46</td>
    <td>65.63</td>
    <td>71.72</td>
  </tr>
  <tr>
    <td>600</td>
    <td>36.25</td>
    <td>73.68</td>
    <td>65.80</td>
    <td>69.13</td>
  </tr>
  <tr>
    <td>1500</td>
    <td>36.52</td>
    <td>74.11</td>
    <td>65.82</td>
    <td>65.62</td>
  </tr>
</table>

Table 15: Performance scores corresponding to different sizes of the training data for LoRA and QLoRA. We specify the learning rate we use for LoRA and QLoRA in the bracket next to the method names.

inter-connections.

In addition, we observe that the learning rate works the best for an individual task does not necessarily work the best for other tasks. For instance, in Figures 7 and 8, the learning rate of 5.0e-6 yields the best performance for FeTaQA, but is suboptimal for HiTab and TabFact. This highlights that when multiple tasks are involved in the training process, researchers need to consider beyond a single task to decide their hyperparameters.

C.7 Trade-off Analysis for Data Properties

We expand our analysis to assess how features in the training data may influence model performance. To investigate this, we train the Llama 3.1 8B Instruct model for three epochs using 500 examples on each dataset, respectively.

Appendix C.6 presents the results. We find that the performance degradation is most significant on TabFact. Interestingly, despite TabFact having intermediate numeric density and table-to-question token ratios, it still shows the fastest performance decline.

We hypothesize that this is due to the nature of the task rather than the table-specific features examined. Since FeTaQA and HiTab are table QA tasks, they may possess similar QA form that the model has encountered in its general instruction tuning stage, this may ease the decay of the model’s general capabilities in our fine-tuning stage. However, TabFact is about fact-checking, the input form includes both the table and the claim to be verified, which we suspect may not be as common as the QA data in its general instruction tuning stage. Therefore, the model suffers a more significant performance decay because it needs to update more of its internal knowledge to handle such a task.

D Hindsight Analysis

Figure 9 provides the complete results of the model performance versus the learning rate and the number of epochs.

Apart from what we have discussed in Section 4.3, we find that on S1, the learning rate of 5.0e-7 yields a consistent good accuracy scores (around 64 to 65) across all the epochs, while 1.0e-6 maintains a good accuracy score (around 64 to 65) for the first two epochs, but starting from the third epoch, it experiences a performance decline (from 64.93 to 52.56).

In terms of the general benchmarks, GPQA resembles similar trends as the trends for IFEval and AI2ARC that the smaller the learning rate is, the less it affects the model’s general capabilities.

E Model Prediction Examples

Table 17 provides an example for table LLMs’ generation on IFEval dataset. Tables 18 and 19 provide two examples for table LLM’s generation on Table-Syn dataset. Apart from the limited out-of-domain table reasoning ability, we find that existing table LLMs also exhibit limited instruction-following capabilities, and often struggle with consistently returning answers in specified formats, such as JSON. Such a limitation poses challenges in the practical use cases, where the end-users may request specific output formats to extract answers from the model’s predictions.

F Dataset Examples

F.1 WikiTQ

Input:

[TAB] col: | description losses | 1939/40 | 1940/41 | 1941/42 | 1942/43 | 1943/44 | 1944/45 | total | [SEP] | direct war losses | 360,000 | | | 183,000 | 543,000 | [SEP] | murdered | 75,000 | 100,000 | 116,000 | 133,000 | 82,000 | 506,000 | [SEP] | deaths in prisons & camps | 69,000 | 210,000 | 220,000 | 266,000 | 381,000 | 1,146,000 | [SEP] | deaths outside of prisons & camps | 42,000 | 71,000 | 142,000 | 218,000 | 473,000 | [SEP] | murdered in eastern regions | | | | 100,000 | 100,000 | [SEP] | deaths other countries | | | | | 2,000 | [SEP] | total | 504,000 | 352,000 | 407,000 |
Figure 7: Heatmap when we fine-tune LLaMA 3.1 8B Instruct model on a single dataset (y-axis) and test against the others (x-axis). We fine-tune the model for one to three epochs (horizontal directions) at a learning rate of 1.0e-5, 5.0e-6, 1.0e-6, 5.0e-7, 1.0e-7 (vertical direction) with 500 instruction pairs.

541,000 | 681,000 | 270,000 | 2,770,000 | \n\nhow many people were murdered in 1940/41?

Instruction:
This is a table QA task. The goal of this task is to answer the question given the table.

Output:
100,000

F.2 FeTaQA
Input:
[TLE] The Wikipedia page title of this table is Gerhard Bigalk. The Wikipedia section title of this table is Ships attacked. [TAB] | Date | Name | Nationality | Tonnage (GRT) | Fate | [SEP] | 14 June 1941 | St. Lindsay | United Kingdom | 5,370 | Sunk | [SEP] | 21 December 1941 | HMS Audacity | Royal Navy | 11,000 | Sunk | [SEP] | 2 February 1942 | Corilla | Netherlands | 8,096 | Damaged | [SEP] | 4 February 1942 | Silveray | United Kingdom | 4,535 | Sunk | [SEP] | 7 February 1942 | Empire Sun | United Kingdom | 6,952 | Sunk | [SEP] | 16 May 1942 | Nicarao | United States | 1,445 | Sunk | [SEP] | 19 May 1942 | Isabela | United States | 3,110 | Sunk |
\n\nThe highlighted cells of the table are: [HIGHLIGHTED_BEGIN][11,000], [Sunk], [8,096], [Damaged] [HIGHLIGHTED_END] What happened to the two heaviest ships Gerhard Bigalk attacked?

Instruction:
This is a free-form table question answering task. The goal for this task is to answer the given question based on the given table and the highlighted cells.

Output:
Gerhard Bigalk damaged one ship of 8,096 GRT, and sunk one warship of 11,000 tons.

F.3 TabFact
Input:
[TLE] The table caption is about tony lema. [TAB ] | tournament | wins | top - 5 | top - 10 | top - 25 | events | cuts made [SEP] | masters tournament | 0 | 1 | 2 | 4 | 4 | 4 | [SEP] | us open | 0 | 2 | 3 | 4 | 6 | 5 | [SEP] | the open championship | 1 | 2 | 2 | 2 | 3 | 3 | [SEP] | pga championship | 0 | 0 | 1 | 2 | 5 | 4 | [SEP] | totals | 1 | 5 | 8 | 12 | 18 | 16 | \n\nThe statement is: <tony lema be in the top 5 for the master tournament , the us open , and the open championship>. Is it entailed or refuted by the table above?
Figure 8: Heatmap when we fine-tune LLaMA 3.1 8B Instruct model on a single dataset (y-axis) and test against the others (x-axis). We fine-tune the model for four to six epochs (horizontal directions) at a learning rate of 1.0e-5, 5.0e-6, 1.0e-6, 5.0e-7, 1.0e-7 (vertical direction) with 500 instruction pairs.

Instruction:
This is a table fact verification task. The goal of this task is to distinguish whether the given statement is entailed or refuted by the given table.

Output:
entailed

F.4 KVRET
Input:
col : event | time | date | room | agenda | party\n\nThe dialogue history is: <remind me to take my pills || >. Please generate the response based on the given table and the given dialogue history.

Instruction:
This is a dialogue response generation task grounded on tables. The goal of this task is to generate response based on the given dialogue history and the given table. The dialogues are grounded through underlying tables and span three distinct tasks in the in-car personal assistant space: calendar scheduling, weather information retrieval, and point-of-interest navigation.

Output:
what time do you need to take your pills ?

F.5 ToTTo
Input:
<table><tr><th>page_title</th><th>List of Governors of South Carolina</th></tr><tr><th>section_title</th><th>Governors under the Constitution of 1868</th></tr><tr><th>table</th><th>76</th></tr><tr><th>col_header</th><th>#</th></tr><tr><th>col_header</th><th>74</th></tr><tr><th>col_header</th><th>75</th></tr><tr><th>col_header</th><th>Daniel Henry Chamberlain</th></tr><tr><th>col_header</th><th>Governor</th></tr><tr><th>row_header</th><th>76</th></tr><tr><th>December 1, 1874</th><th>Took Office</th></tr><tr><th>row_header</th><th>76</th></tr><tr><th>Please generate one natural language description to describe the given highlighted table cells.</th></tr></table>

Instruction:
This is a highlighted cells description task. The goal of this task is to generate the language description given table cells.

Output:
Daniel Henry Chamberlain was the 76th Governor of South Carolina from 1874.
<table>
  <tr>
    <th>D<sub>num, tab</sub> (%)</th>
    <th>No. Cells : Num. Cells</th>
    <th>TT tokens</th>
    <th>Tab tokens</th>
    <th>Q tokens</th>
    <th colspan="6">Tab tokens MMLU MMLU MMLU IFEval IFEval IFEval</th>
  </tr>
  <tr>
    <th></th>
    <th></th>
    <th></th>
    <th></th>
    <th></th>
    <th>Q tokens</th>
    <th>(1e-6)</th>
    <th>(5e-6)</th>
    <th>(1e-5)</th>
    <th>(1e-6)</th>
    <th>(5e-6)</th>
    <th>(1e-5)</th>
  </tr>
  <tr>
    <td>TabFact</td>
    <td>73.03</td>
    <td>1.34 : 1</td>
    <td>292,822</td>
    <td>264,520</td>
    <td>19,286</td>
    <td>13.72 : 1</td>
    <td>66.74</td>
    <td><b>64.51</b></td>
    <td>29.95</td>
    <td>77.70</td>
    <td><b>49.40</b></td>
    <td><b>25.66</b></td>
  </tr>
  <tr>
    <td>FcTaQA</td>
    <td>57.99</td>
    <td>1.68 : 1</td>
    <td>309,624</td>
    <td>251,697</td>
    <td>42,492</td>
    <td>5.92 : 1</td>
    <td>65.79</td>
    <td>65.66</td>
    <td>63.73</td>
    <td>77.82</td>
    <td>53.36</td>
    <td>31.41</td>
  </tr>
  <tr>
    <td>HiTab</td>
    <td>80.60</td>
    <td>1.19 : 1</td>
    <td>452,149</td>
    <td>424,941</td>
    <td>11,030</td>
    <td>38.53 : 1</td>
    <td>66.37</td>
    <td>66.77</td>
    <td>62.91</td>
    <td>78.18</td>
    <td>49.40</td>
    <td>29.74</td>
  </tr>
</table>

Table 16: Llama 3 8B Instruct’s performance on the general benchmarks MMLU and IFEval corresponding to different learning rates (Numbers in the bracket). We train the model for three epochs using 500 examples on each dataset, respectively. “D<sub>num, tab</sub>” represents the density of the number cells in the table. “No. Cells : Num. Cells” denotes the cells containing no number versus cells containing numbers. “TT tokens”, “Tab tokens”, “Q tokens” represent the total number of input tokens, table tokens, and question tokens.

![Line graphs showing performance scores (y-axis) with respect to the number of epochs (x-axis) and learning rates for various datasets.](page_320_670_1047_496.png)

Figure 9: Performance scores (y-axis) with respect to the number of epochs (x-axis) and learning rates. In Figure 9a, we aggregate the performance scores for all the datasets listed in Table 5.

F.6 S1
Input:
# Task Description: Please determine whether Citation A and Citation B refer to the same entity or not. Your final answer should be 'Yes' or 'No'. Return the final result as JSON in the format "{\"answer\": \"<Yes or No>\}". Let's think step by step and show your reasoning before showing the final result.\n## Input:\nCitation A is:\n|title|authors|venue|year|\n|---|---|---|---|\n|algorithms for mining association rules in bag databases|py hsu , yl chen , cc ling|information sciences\u00e2 ?? informatics and computer science : an & hellip ; ,|2004.0|\n\nLet's think step by step and show your reasoning before showing the final result. Return the final result as JSON in the format "{\"answer\": \"<Yes or No>\}".\n## Output:\n

Instruction:
You are a helpful assistant that specializes in tables.

Output:
{"answer": "No"}
<table>
  <tr>
    <th>PROMPT:</th>
    <td>Please provide the names of 5 famous moms in JSON format.</td>
    <th>Follow Instruction?†</th>
  </tr>
  <tr>
    <td>TableLLaMA</td>
    <td>&lt;Mommy&gt;, &lt;Dad&gt;, &lt;Baby&gt;, &lt;Baby&gt;, &lt;Dad&gt;, &lt;Mommy&gt;&lt;/s&gt;</td>
    <td>x</td>
  </tr>
  <tr>
    <td></td>
    <td>
      import pandas as pd<br>
      # Load the data<br>
      df = pd.read_csv('data.csv')<br>
      # Select the first 5 rows<br>
      TableLLM<br>
      df_first_5_rows = df.head(5)<br>
      # Convert the dataframe to JSON<br>
      json_output = df_first_5_rows.to_json()<br>
      # Print the JSON output<br>
      print(json_output)
    </td>
    <td>x</td>
  </tr>
  <tr>
    <td>TableBenchLLM</td>
    <td>Here are the names of 5 famous moms in JSON format: 1. Sarah Palin 2. Jill Biden 3. Michelle Obama 4. Amy Schumer 5. Madonna...</td>
    <td>x</td>
  </tr>
  <tr>
    <td>TAMA (ours)</td>
    <td>
      {"famous_moms": [{"name": "Oprah Winfrey"}, {"name": "Michelle Obama"},{"name": "Angelina Jolie"},{"name": "Madonna"},{"name": "J.K. Rowling"}]}
    </td>
    <td>✓</td>
  </tr>
</table>

Table 17: Table LLMs’ predictions on the prompt in IFEval. For TableBenchLLM, it starts to repeat itself and we omit the repeated part for better readability. †: We note that IFEval checks the instruction following ability rather than the correctness of the answer.

F.7 S2

Input:
You are a table analyst. Your task is to answer questions based on the table content.\n\nThe answer should follow the format below:\n[Answer Format]\nFinal Answer: AnswerName1, AnswerName2 ...\n\nEnsure the final answer format is the last output line and can only be in the "Final Answer: AnswerName1, AnswerName2..." form, no other form. Ensure the "AnswerName" is a number or entity name, as short as possible, without any explanation.\n\nGive the final answer to the question directly without any explanation.\n\nRead the table below in JSON format:\nTABLE \n<table><tr><th>season</th><th>tropical lows</th><th>tropical cyclones</th><th>severe tropical cyclones</th><th>strongest storm</th></tr><tr><td>["1990 - 91"]</td><td>10, 10, 7, "marian"</td><td>["1991 - 92"]</td><td>11, 10, 9, "jane - irma"</td><td>["1992 - 93"]</td><td>6, 3, 1, "oliver"</td><td>["1993 - 94"]</td><td>12, 11, 7, "theodore"</td><td>["1994 - 95"]</td><td>19, 9, 6, "chloe"</td><td>["1995 - 96"]</td><td>19, 14, 9, "olivia"</td><td>["1996 - 97"]</td><td>15, 14, 3, "pancho"</td><td>["1997 - 98"]</td><td>10, 9, 3, "tiffany "</td><td>["1998 - 99"]</td><td>21, 14, 9, "gwenda"</td><td>["1999 - 00"]</td><td>13, 12, 5, "john / paul"]}</td></tr></table>\nLet's get start!\nQuestion: What is the average number of tropical cyclones per season?\n

Instruction:
You are a helpful assistant that specializes in tables.

Output:
10.6

F.8 MMLU

Input:

{5-shot examples}
Find the degree for the given field extension Q( sqrt(2), sqrt(3), sqrt(18)) over Q.
\nA. 0\nB. 4\nC. 2\nD. 6\nAnswer:

Instruction:
The following are multiple choice questions (with answers) about abstract algebra.\n\n

Output:
B

F.9 IFEval

Input:
Can you help me make an advertisement for a new product? It's a diaper that's designed to be more comfortable for babies and I want the entire output in JSON format.

Instruction:
You are a helpful assistant.

Output:
[JSON formatted answer]
Task Description: Please check the following table, there is one and exactly one cell in the table that is missing. When you find this missing cell, please point it out using the row id shown in the first column. Return the final result as JSON in the format {"row_id": "<row_id of the row with missing cell>"}.

## Input:
|row_id|Date|Sales|Cost of Sales|Expenses|
|---|---|---|---|---|
|1|13542817791180685|286681|
|2|113328173456268921|
|3|136406116405588108284571|
|4|136418115111870952291381|
|5|1371411688371702023798|

Return the final result as JSON in the format {"row_id": "<row_id of the row with missing cell>"}.

PROMPT: ## Output:

<table>
  <tr>
    <th></th>
    <th></th>
    <th>Correct?</th>
  </tr>
  <tr>
    <td>TableLLaMA</td>
    <td>&lt;177911.0&gt;, &lt;133281.0&gt;, &lt;164055.0&gt;, &lt;151118.0&gt;, &lt;37141.0&gt;&lt;/s&gt;</td>
    <td>X</td>
  </tr>
  <tr>
    <td>TableLLM</td>
    <td>{"row_id": 2}</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>TableBenchLLM</td>
    <td>Final Answer: 2 // 2php2://...</td>
    <td>X</td>
  </tr>
  <tr>
    <td>TAMA (ours)</td>
    <td>{"row_id": 2}</td>
    <td>✓</td>
  </tr>
</table>

Table 18: An example of table LLMs’ predictions on Table-Syn.

# Task Description: Please look at the input column and determine the semantic type that can describe “every single* instance the input column. Please only choose one semantic type from the candidate list, and remember that the type you choose has to accurately describe every single entity in the column. If no candidate column type can suitably describe every single instance in the column, please return 'None'. Please only choose one type from the candidate list below, and “do not” create new types. Return the final result as JSON in the format {"chosen_semantic_type": "<an entry from the candidate list or None>"}.

## Input:
**Column:**
Loser (wager)
I
Ultratumba (mask)
Ultratumba (hair)
El Noruego (hair)
El Destroyer (mask)
Dory Dixon (hair)
Humberto Garza (hair)
CÁsar Valentino (hair)
La Momia (mask)
...
**Candidate column type:**
AcademicJournal, AdministrativeRegion, Airport...Film, Wrestler...
Return the final result as JSON in the format {"chosen_semantic_type": "<an entry from the candidate list or None>"}.

PROMPT: ## Output:

<table>
  <tr>
    <th></th>
    <th></th>
    <th>Correct?</th>
  </tr>
  <tr>
    <td>TableLLaMA</td>
    <td>&lt;Blue Blazer (mask)&gt;, &lt;Don Corleone (mask)&gt;...&lt;/s&gt;</td>
    <td>X</td>
  </tr>
  <tr>
    <td>TableLLM</td>
    <td>{"chosen_semantic_type": "Film"}</td>
    <td>X</td>
  </tr>
  <tr>
    <td>TableBenchLLM</td>
    <td>...Loser (wager)*Let’s consider the new type “Loser (wager)” to be a role...</td>
    <td>X</td>
  </tr>
  <tr>
    <td>TAMA (ours)</td>
    <td>{"chosen_semantic_type": "Wrestler"}</td>
    <td>✓</td>
  </tr>
</table>

Table 19: An example of table LLMs’ predictions on Table-Syn. We omit part of the prompt for better readability.

Metareview score: 3
Metareview: This paper explores how different hyperparameter settings impact table understanding performance of LLMs. The authors discovered that using a small learning rate and a limited amount of training data can enhance a model’s ability to interpret tables without hurting its general capabilities. Leveraging this insight, they instruction-tuned a LLaMA-3.1-8B-Instruct model, which reached performance levels on par with GPT-3.5 and GPT-4.
Summary of reasons to publish: By systematically evaluating hyperparameters across various LLM architectures, this work lays a solid foundation for future advancements in table understanding research. The nuanced examination of how different model families respond to learning rate adjustments offers key guidance for effective model tuning.