StructLM: Towards Building Generalist Models for Structured Knowledge Grounding
Authors: Alex Zhuang
Ge Zhang
Tianyu Zheng
Xinrun Du
Junjie Wang
Weiming Ren
Wenhao Huang
Jie Fu
Xiang Yue
Wenhu Chen
Venue: COLM (2024)
Keywords: Structured Data；Instruction Tuning Dataset；Model Scalability Analysis
TL;DR: Our CodeLlama-based models, bolstered by a dataset of 1.1 million examples, set new standards in structured knowledge tasks, showing that model size has a minimal effect on enhancing these skills.

Abstract: Structured data sources, such as tables, graphs, and databases, are ubiquitous knowledge sources. Despite the demonstrated capabilities of large language models (LLMs) on plain text, their proficiency in interpreting and utilizing structured data remains limited. Our investigation reveals a notable deficiency in LLMs' ability to process structured data, e.g., ChatGPT lags behind state-of-the-art (SoTA) model by an average of 35\%. To augment the Structured Knowledge Grounding (SKG) capabilities in LLMs, we have developed a comprehensive instruction tuning dataset comprising 1.1 million examples. Utilizing this dataset, we train a series of models, referred to as $\texttt{structlm}$, based on Mistral and the CodeLlama model family, ranging from 7B to 34B parameters. Our $\texttt{structlm}$ series surpasses task-specific models~\citep{UnifiedSKG2022} on 16 out of 18 evaluated datasets and establishes new SoTA performance on 8 SKG tasks. Furthermore, $\texttt{structlm}$ demonstrates strong generalization across 6 novel held-out SKG tasks, outperforming TableLlama by an average of 35\% and Flan-UL2 20B by an average of 10\%. Contrary to expectations, we observe that scaling model size offers marginal benefits, with $\texttt{structlm}$-34B showing only slight improvements over $\texttt{structlm}$-7B. This suggests that structured knowledge grounding is still a challenging task and requires more innovative design to push to a new level. We release the model weights and training dataset to the community, along with relevant code on Github.

Published as a conference paper at COLM 2024
StructLM: Towards Building Generalist Models for Structured Knowledge Grounding
Alex Z Ge Zhangl:27;
Tianyu Zheng", Xinrun Du?, Junjie Wang Weiming Ren Stephen W. Huang", Jie Fu Xiang Yue Wenhu Chen 1.277 Unniversity of Waterloo, Multimodal Art Projection Research Community, Waseda University, "HKUST, 'Oio State University, "harmony,a,, Veector Institute https://tiger-att -ab. gi github. io/
Abstract
Structured data sources, such tables, graphs, and databases, are ubiq- uitous knowledge sources. Despite the demonstrated capabilities large language models (LLMs) plain text, their proficiency in interpreting and utilizing structured data remains limited. Our investigation reveals a notable deficiency in LLMs' ability to process structured data, e.g., Chat- GPT lags behind state-of-the-ar (SoTA) model by an average of 35%. To augment the Structured Knowledge Grounding (SKG) capabilities in LLMs, we have developed comprehensive instruction tuning dataset comprising 1.1 million examples. Utilizing this dataset, we train a series of models, referred to as StructLM, based on Mistral and the CodeLlama model family, ranging from 7B 34B parameters. Our StructLM series surpasses task- specific models (Xie et al., 2022) on 16 out of 18 evaluated datasets and establishes new SoTA performance on 8 SKG tasks. Furthermore, StructLM demonstrates strong generalization across 6 novel held-out SKG tasks, out- performing TableLlama by an average of 35% and Flan-UL2 20B by an average of 10%. Contrary to expectations, we observe that scaling model size offers marginal benefits, with StructLM-34B showing only slight im- provements over StructLM-7B. This suggests that structured knowledge grounding is still a challenging task and requires more innovative design to push to a new level. We release the model weights and training dataset to the community, along with relevant code on Github.
1 Introduction
Traditionally, users need to write programs to interface with structured data like tables, databases, knowledge graphs, etc. This often requires that they master the domain-specific language like SQL, SPARQL, etc. or develop domain specific skills for that structured type. Recently, researchers have explored the possibility of automating the interface with natural language enable potential use cases in question-answering (Pasupat & Liang, 2015; Zhong et al., 2017; Nan et al., 2022), summarization (Parikh et al., 2020; Nan et al., 2021; Bao et al., 2018), and fact verification (Aly et al., 2021; Chen al., 2019; Gupta et al., 2020b), among others, all grounded to structured knowledge source. This effort can lower the barrier for end users to access a massive amount of structured data.
Previous work (Yu et al., 2020; Liu et al., 2021; Xie et al., 2022; Zhang et al., 2023) has been mostly focused on building task-specific models for different tasks with rather limited generalization ability. Building a generalist structure knowledge grounding (SKG) system across a wide range of tasks proves to be challenging. This is mainly due to the heterogeneity of data format and use cases. We evaluated GPT-3.5-Turbe (Jiang et al., 2023) on 18 SKG tasks and observed that its performance is on average 35% lower than the SoTA specialized models. It shows that the L.M''s ability on SKG is heavily overlooked during pre-training.
1

---

Published as a conference paper at COLM 2024
In this paper, we explore the possibility of building generalist model based on LLMs that can ground on diverse types of structure and unstructured knowledge to interface with humans. Specifically,
construct large data set of over million instruction-follooing examples, a majority of which is SKG data, along with additional general instruction- following data, which we find improves generalizability. We fine-tune models at three scales: 7B, 13B, and 34B, based on Mistral and the CodeLlama family of code foundation models. When compared USKG, we find that our models surpass these single-task models on 16 of 18 tasks. StructLM achieves SoTA on 8 of 18 evaluated tasks, beating ChatGPT by huge margin. Morever, we show that compared with TableLlama, a recent open language model finetuned for tabular inference tasks, our held-out performance is vastly superior. Compared with Flan-UL.2-20B, our models also generalize better overall on the held out tasks by as much as 10% on average.
We study the performance of StructLM, namely whether the model experiences cross- task generalization benefits from the dataset mixture, and find that our multi-task model performs significantly better overall than single-task models the exact same parameter scale. We also study the effect of different pretraining data determine whether special pretraining regimes, such as code or math, contribute effective SKG reasoning ability. We find that code pretraining is the most effective. We perform additional ablations to confirm our results and support our claims. Our contributions are:
finetuned performance to
We construct a large SKG instruction train and release our models that outperform the previous 3B USKG fine-tuned on individual tasks on total of 16 of 18 tasks. StructLM also achieves SoTA results 8 of them.
dataset with 1.1 million samples. We
We show that StructLM is able to show strong zero-shot generalization capability unseen structure knowledge grounding tasks, which was not shown by previous models. StructLM outperforms Flan-UL22 20B on of tasks and TableLlama on all 6 tasks. The average absolute improvement over them are 10% and 35%.
We find that mixing in general-domain instruction-tuning data during finetuning preserves generalization ability, and that code-pretrained base models can improve model performance on the SKG tasks.
2 Related Works
2.1 Solving SKG tasks
Structured knowledge, such web tables, knowledge graphs, and databases, have long been the subject of study in knowledge grounding. However, SKG tasks have heteroge- neous data formats which have inspired methods that leverage specific training setups to
StructLM: generalist model knowledge grounding
Who got the points champion FIBA, 2023
100
ChatGPT
Previous SoTA
StructLM
80
Luka Dončić
60
Grounding
40
Structured & Unstructured Knowledge
20
ToTTo
GraiiIIA
KVRet
CWebOA FEVEROUS
Table
XB
DB
Text
Figure 1: StructLM can ground on structured and unstructured knowledge to respond to human queries. The previous SoTA was attained by many different task-specific models like TAPEX (Liu et al., 2021), USKG (Xie et al., 2022), TableLiama (Zhang et al., 2023), BINDER- Codex (Cheng et al., 2022), etc. StructLM beats the SoTAS on seven SKG tasks.
2

---

Published as a conference paper at COLM 2024
the following question with the help of the given data table.
Held-in Evaluation
data table: clnearrized table>
Unseen Held-out Tasks
List out the answers to the latest question the dialog based on the information in the given table.
question: election, and how nany votes was the nargin?
1982 illinois guhernatorial
thompson prevailed In the 1982 illinois gubernatorial election 5,874 vote nargin.
table: <Linearized
the totals all these teans? || what were all the teans?
Input Types:
knowledge graphs
tables
schema
16, 12, 12, 14, 15, 14, 13, 14, 15, 22
Output Types:
code / formal grammar
text (code translation)
summary
single answer
bookean
Figure 2: Overview of StructLM. This figure illustrates the prompting structure of StructL highlighting its capability to process various forms of structured data beyond linearized data tables, including linearized database schemas and knowledge graphs.
learn those representations. For example, PTab (Liu et al., 2022) and Multil (Zhao et al., 2022) learn the contextual representation of tabular data by incorporating semantic information through specific training methods or reasoning approaches. RASAT (Qi et al., 2022a) integrates relation-awas self-attention with the Transformer seq2seq architecture and utilizes various relational structures to address SQL problems. TAPEX (Liu et al., 2021) conducts pretraining over tabular cabular/database database data with the help of an SQL executor to provide supervision.
More recently, methods have begun to move away from these auxiliary task-specific struc- tures. USKG (Xie et al., 2022) were the first to unify many SKG tasks into sequence-to- sequence format, allowing them to be aggregated into the same data mixture. However, they were not able to show strong performance improvements to constructing multi-task mix of SKG data over task-specific tuning methods. StructGPT (Jiang et al., 2023) represents line of work that uses prompting frameworks on powerful LLMs to solve tasks with more robustness and accuracy. In contrast, our work examines open models and tries to assess their fundamental capabilities. Contemporary 2023) has argued that tabular data deserves special attention. Focusing on this domain, their method fine-tunes on several new tabular tasks improve table understanding, and operates on longer 8k context length. These improvements can be additive to our work.
our work, Lable (Zhang al.,
2.2 LLMs with Instruction Tuning
Instruction-tuning (IT) has been popularized method address the gap between train- ing objectives and user goals in LLMs. This technique involves additional training of LLMs using pairs of instructions and outputs. IT enhances both the controllability and the pre- dictability of the models, aligning them more closely with user expectations. Furthermore, recent studies such as FLAN (Wei et al., 2022), UL2 (Tay al., 2023a), and Llama2 (Touvron et al., 2023) have shown that IT can improve the performance of downstream tasks through multi-task learning across diverse data types. While FLAN-UL2 trains on subset of 11 SKG tasks, it also trains many more unrelated language tasks. In work, by focusing on SKG data, we hope to provide focused study that can act as a reference for future work to improve performance this task type.
2.3 Reasoning Capability in LLMs
Reasoning stands as a pivotal skill for LLMs in the development of real-world AI ap- plications which would enable the autonomous completion of many thought-intensive tasks viewed traditionally to require human thinking, like programming or mathematical problem-solving (Li et al., 2022). Recent studies (Li et al., 2022; 2023c; Rozière et al., 2023;
3

---

Published as a conference paper at COLM 2024
None
General Instruction Turing 512,089
512,069
Table
267.026
436,432
Fact
132,952
Domain Specific Grammar
72,335
Summartzation
Knowledge Graph
183,420
71,976
Code Lannerttion
Knowladiee Triples
060
82,659
Interproiation
Datebase
14, 166
19,059
Speatic Lenguage 14, 188
Figure 3: Breakdown of Structured Knowledge Types and Tasks in the Training Data. On the left side, we see a coarse breakdown of the different categories of structured inputs in our dataset. On the right side, we see an overview of the task groups that are represented for those structured knowledge types.
Azerbayev al., 2023a) indicate that LLMs trained code and mathematical datasets exhibit profound reasoning skills, and can even achieve performance on par with human levels. For example, Codellama (Rozière et al., 2023), foundation model trained more programming data, has significantly improved reasoning capabilities across variety of pro- gramming and mathematical benchmarks. Furthermore, Llemma (Azerbayev al., 2023a) continues to pretrain the CodeLlama model on mix of scientific papers, math-related web data, and mathematical code. Its results show excellent reasoning capabilities on the MATH benchmark (Hendrycks et al., 2021) and the ability to prove theorems without further fine-tuning. On the fine-tuning side, Wizard (Luo et al., 2023a), and WizardCoder (Luo et al., 2023c) have shown the effectiveness of instruction tuning on reasoning capabilities, given high quality data.
In this work, we view structured data third testbed for different kind of reasoning within LLMs. We posit that in addition to mathematical or logical reasoning, the ability to recognize and make use of patterns within structured input indicates that model has robust representations of relationships in data. These representations may serve as a strong prior for further reasoning downstream.
3 Method
3.1 Dataset Curation
Motivated by the goal of training language model generally capable of a wide range of structured data tasks, we select a total of 25 SKG tasks to study. We report results on 18 held-in and 6 held-out tasks, where each held-out task meant to roughly evaluate the generalization capability of held-in task group. In total, our held-in training dataset contains approximately 700k SKG examples. We describe the held-in dataset groups below.
This group of datasets deals with the summarization or interpreta- tion of structured data from tables to knowledge triples formal languages. Their inclusion is motivated by the idea that useful LMs should be able to make sense of wide variety of structured information and map it to meaning in natural language. The corresponding held-out dataset for this task group is intended to be WikiTableText.
Data to Text Generation.
Table based Question Answering. This group of datasets deals specifically with tabular data, optionally combined with text passages. LMs which are able to accurately answer questions and retrieve information from tables can be widely useful as assistants. The corresponding held-out dataset for this task group is SQA.
4

---

Published as a conference paper at COLM 2024
Knowledge-groundd Conversations. This group of tasks evaluates knowledge grounding in-conversation. Humans naturally interface with LMs through chat, and enabling this capability can lower the barrier to accessing the information in stored structured data. These tasks track user intention through provided dialogue and ask the model to provide an answer to the latest question. The held-out dataset for this task group is CoSQL..
Fact verification. One common use case for tables is to reference facts. In addition to question answering, the ability to reliably determine if data in table supports a statement signals the existence of robust representation of the table's data. The held-out dataset for this task group InfoTTbs.
SQL or domain-specific languages SQL is the language most commonly used to interface with structured data today. Understanding how to write SQL also requires understanding of abstractions of tables and how they are linked together. In other domain-specific languages, the MTOP task measures a model's ability to parse specification and generate an API call, which sees potential in LLM tool use (e.g., (Qin et al., 2023)). The corresponding held-out dataset for this task group is intended to be BIRD (Li al., 2023b), which further tests SQL generation abilities.
Mathematical reasoning An analysis of tabular data may also require performing quick mathematical computations over their contents. Performance on these datasets tells us how well models can combine both structured knowledge and mathematical reasoning. As there are currently limited number of datasets that combine mathematical reasoning with SKG, this category includes just TabMWP in the held-in corpus. We set FinQA as challenging held-out dataset analogue. Not only does it require financial domain knowledge, but it combines tabular information with long text passages, and requires the generation of mathematical code.
General instruction data. In addition to the SKG datasets within the held-in dataset mixture, we also included general instruction tuning data without any structured knowledge component, to maintain the instruction-folloiing ability of our model. We use Orca (Lian al., 2023), which constructed from cleaned GPT-4 responses a number of prompts from existing general large-scale instruction-folloviii of data contamination for our held-out datasets based on our ablation results. We give detailed overview of all dataset statistics in Table 1.
datasets. We detect no signs
3.2 Instruction Finetuning Approach
To instruction tune model, each example in our dataset consists of system prompt, instruction, input, and output. For all SKG data examples, we use the same system prompt. For each dataset, we write 10 instruction variations, which are randomized when con- structing the training samples. For SKG data, the input is composed of a combination of structured knowledge input and accompanying text that could be question, statement, or anything that would be required to specify the task. The prompt is provided in Figure 6.
3.3 Training and Evaluation Details
The base models for StructLM are the LodeLlama-Instruct family of models (Rozière et al., 2023). We finetune all models with batch size of 512 for epochs on A800 gpus. We train our 7-34B models on 16-64 GPUs using DeepSpeed ZeRO-3 (Rasley et al., 2020) such that the total training time for each model between 3-5 days. This training setup is largely in line with community conventions, such as the settings used for the WizardLM (Xu et al., 2023), WathrdMath (Luo et al., 2023a), and WizardCoder (Luo et al., 2023c) models.
We follow the structured data linearization conventions in USKG (Xie et al., 2022). However, we use a different truncation scheme as described below. During training, we maintain maximum sequence length of 2048. To preserve much input context possible, when truncating we consider the combined token length of the prompt input and output label. We truncate only the structured knowledge portion of the input that the example becomes at most 2048 tokens long. As shown in the dataset statistics in Table 1, setting the max token length of the examples in dataset to 2048 allows nearly all examples to fit within the

---

Published as a conference paper at COLM 2024
Overall Length
Train
Test
Dataset
Input (avg)
Output (avg)
Count
Input (max)
Output (max)
# Trunc.
Count
Input (max)
Output (max)
Trunc.
TahMWP
207.8
4.5
23059
719
33
7686
7113
31
ToTTo
251.8
31.0
120761
2040
155
467
7700
2048
119
GrailQA
281.0
44.1
44337
884
134
6163
546
123
SQL2Text
122.3
18.1
5600
337
61
1034
245
38
MMQA
656.2
7.7
15688
2047
146
234
1501
2048
94
11
Spider
36.0
7000
226
1034
453
KVRet
573.4
17.1
6288
1217
161
1147
82
HybridQA
700.4
6.8
62682
2047
91
200
3466
2048
79
SParC
276.3
32.6
12059
1417
226
1625
467
146
Q
1350.3
27639
321
321
2816
256
TabFaact
660.1
4.6
92283
2045
12779
1687
WikiTQ
831.8
5.8
11321
2028
273
4344
2048
148
10
WikisQL
689.2
7.1
56355
2047
518
16
15878
2048
244
FeTaQA
653.2
7326
158
2003
1548
FEVEROUS
799.3
3.4
40669
2047
2052
4285
2048
195
MultWOZ
777.2
154.5
56668
1656
196
7368
1344
185
DART
133.7
30.3
62659
406
258
5097
261
109
Logic2Text
166.1
26.9
8566
358
67
1092
347
60
MTOP
961.0
34.4
15667
1002
215
4386
990
113
SlimOrca
278.9
152.4
512069
2047
1808
BIRD
439.8
63.3
9428
1992
347
99
1534
1214
386
CoSQL.
287.4
34.9
9502
1640
226
1300
535
190
SQA
12275
1012
769
Infotabs
276.9
3.7
16538
1009
5400
1105
WikiTableText
149.6
27.4
10000
313
97
2000
226
89
Finqa 1230.3
21.0
6251
2040
72
186
1147
2048
Table 1: Token sequence length statistics for each dataset in our train and test sets. Input and output statistics are in tokens. We report the number of examples which have been truncated in each dataset.
context window with rare truncations. We discard examples for which even this structured input truncation is insufficient (e.g. the output is too long). During inference, we set the input token length to 2048, to allow even more structured information be placed within the input context. We set the maximum generation length to 1024, which is sufficient for all correct responses in all datasets. For each model, including our single-task finetuned models, we choose the best performing checkpoint of the 3-epoch checkpoints.
4 Experiments
Firstly, to illustrate the current performance of language models on SKG tasks, we evaluate ChatGPT (GPT-3.5-turbo) and the base model Code 1-shot setting. Our prompting scheme, using the same linearized knowledge structures as in our held-in training, sees them struggle across the board with many tasks due to the unseen structure knowledge format. Although ChatGPT is superior text-based tasks, its performance lackluster on SKG tasks. Its gap with SoTA models as significant as 35%.
Baselines
under
Held-in Results To evaluate the benefits of instruction-tuning dataset mix, also run single-task baseline (each 7B model) on each task and report their individual performance. We again use CodeLlama--7--nnttrice as the base model for each, and match each single task model on the same number of epochs (3) that was used to train the multitask models, ensuring that each model has seen the same data the same number of times. We observe that our multi-task models outperform these single-task models on nearly every task, with some by considerable margin of up to 7%. This demonstrates the effectiveness of our instruction tuning dataset and supports the presence of cross-task generalization.
When compared to the 18 task-specific USKG models, StructLM-7B can surpass USKG by average of 2%. From parameter-count perspective, each of the USKG models is T5-3B model, which means over the entire held-in set, these results require 54B parameters. Our 7B-M StructLM in comparison be viewed being nearly 8x parameter efficient while

---

Published as a conference paper at COLM 2024
Metric
ChaiGPT
FLAN-U12
Tablel
7B-M
Held
LoTTo
BLEU
$5.9
2117
17.9
17.5
48.5
49.0
49.8
49.4
49.3
50.2
10.3
GailIA
EM
77.1
9.3
80.4
SQU 2Txt
Blee
94.8
88.6
90.7
82.9
95.2
94.8
98.3
93.8
68.5
92.6
+0.4
MMQA
F1
85.3
59.6
41.5
30.7
81.5
85.3
85.5
85.2
86.0
58.1
+2.8
Spider
50.5
72.4
72.4
KVRet
All Micro
67.9
52.9
34.4
39.5
70.9
67.9
72.2
72.6
69.5
69.3
14.7
ElybridQA
Acc
68.4
23.7
12.9
2.3
58.4
61.0
38,4
62.6
59
59.1
61.1
-5.8
SParC
CempWebQ
Acc
76,8
08.9
30,9
3.1
75.6
75,9
73.3
79.9
78.3
80.4
51.9
+5.1
TabFact
Ace
93.0
62.4
25.7
79
87.1
82.5
84.6
80.8
84.7
WIKTTQ
546
56.5
60.3
WIKSSQI
All Ex
93.0
31.5
21.5
0.4
86.5
87.3
86.0
87.0
88.7
67.2
87.6
-4.
FeTQAA
39,0
38,0
37.5
36.0
FEVEROUS
Ace
65.6
57.8
73.2
58.4
78.1
85.6
62.4
85.9
84.4
85.0
85.7
+0.3
MullWaz
Joint Ace
60.6
8.9
0.3
0.U
53.0
53.4
55.4
54.5
53.0
53.8
-8.2
63.2
LogkzZest
Blae
95.3
73.5
51.5
99.1
89.5
91.4
89.5
89.9
90.1
89.1
-5.2
MTOP
TM
37.5
1.4
0.8
77.4
87.5
36.8
75.8
81.2
81.6
52.1
Average
74.9
39.5
30.8
20.2
68.2
69.3
72.3
71.1
71.3
72.6
-1.2
Held Out
BIRD
Acc
36.6*
22.6
22.3
+2.9
CosQU
EM
58.3"
33.7
26.5
0.2
32.4"
5.1
11
52.8
49.8
52.2
55.0
-21.3
Ace
70.5
70.1"
42.6
49.77
36.1
Infotabs
Acc
75.6*
40.2
68.7
20.3
16.2
47.2
55.3
58.1
-8.5
WikiTableTTest
BLEU
33.7"
3.8
3,9
577
39.8*
19:4
3.4
17.1
8.3
9.3
8.8
-2.3
Finga
71.1"
29.5
27.3
36.2
Average
57.6
26.1
78.6*
3.7
35.3
35.5
+8.2
Table The overall evaluation results of our model against other baselines. 7B-M was trained with Mistral-7B as the base model. Cells with "_" in the held-in part mean that the model did not train on this dataset, and results are not comparable. USKG models are overfit to the held-in dataset labels, and thus cannot generalize comparably. Cells in the held-out section with "*" are held-in results. SoTA results are copied from the original papers for reference. ST refers to the single-task fine-tuning result of CodeLlama- Jama-Instruct-7B on each dataset. BASE and BASE-M refer to the 1-shot performance of CodeLLlam--Instruc/// Mistral-7B-Instrucc-ee
and respectively. A refers to the difference between StructLM and the best known result. score denotes the state-of-the-art score on specific tasks. All StructLM held-out results are U-shot. Specifications as how SoTA results are selected are given in Table 4.
still surpassing USKG models on 16 of 18 datasets. is worth noting that although the single-task (ST) models are more than double the size in parameters compared to USKG, they do not perform much better on average. This fact indicates that there may be significant unused model capacity that can be better utilized via more effective training regimes, such as our instruction tuning. Regarding FLAN-UL2-208 (Tay et al., 2023b), which was also extensively trained on structure knowledge grounding tasks, StructLM outperforms it on 7 of the mutually held-in datasets. Our results on held-in datasets (Tabfact and FaTaQA) are par with TableLlama (Zhang et al., 2023), which is an LLM pre-trained on 2.6M table understanding tasks. We are vastly outperforming TableLlama on the held-out datasets.
Held-out Results On held out tasks, StructLM shows strong generalization performance, outperforming ChatGPT on of 6 tasks. These novel tasks contain remarkably different data format than the training dataset. For example, FinQA (Chen et al., 2021) requires the model to generate a mathematical expression to answer question in the financial domain, while BIRD (Li et al., 2023b) requires the model to interface with multiple databases. These skills are not exactly covered in the training set, thus they put high demand for the models' generalization capabilities. Such generalization capabilities are non-existent in USKG models as they are task-specific Another table foundation model TableLlama (Zhang et al., 2023) would also fail on most of the held-out tasks with performance close to zero. In contrast, StructLM is much more capable of generalization on these novel tasks. The average improvement over TableI is 35%. Compared to another strong foundation
7

---

Published as a conference paper at COLM 2024
Codeelama
Llemms
Uama2
70
Average Pertomannce
50
60
60.4
53.4
57.8
55.4
57.7
54.8
45.4
30
34.8
20
10
Math related (n-2)
Code related (n=5)
Tabular 1.3)
Qverall (n=
Figure 4: Effect of different pretraining curricula on SKG finetuning performance in relevant task groupings. We can observe the advantages of CodeLIma over the others.
Purpose
Train
Eval
FT
Result
Schema task transfer
Spider, SParC, Logic2Text
Logie2Text
89.47
89.93
KT task transfer
CompWebQ, WebQSP, GrailQa, DART
DART
60.28
60.34
FetaQA, HybridQA, WikiTQ,
TabFact,
TabMWP, ToTTo, MMQA, WikisQL, KVRet, Tab Fact, Feverous, Infotabs
Table task transfer
Feverous Infotabs
75.46
80.81
Summ. data type transfer
ToTTo, DART
DART
60.28
61.42
QA data type transfer
CompWebQ, WikiSQL
WikiSQL
85.49
86.36
Table 3: Cross task and cross datatype transfer results. FT is average of single-task performance over the datasets in the Eval column.
held in
71.4
Avg. Perf.
71.2
29:
71.0
held out
28
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.3
0.1
0.5
SKG proportion of data mix!
Figure 5: Effect of general instruction-followinng data on averaged held-out SKG dataset performance. Performance is measured as the average over evaluation metrics across all tasks within held-in or held-out groups.
structur-knowlllggg-gome
model Flan-UL2 20B (Tay et al., 2023b), we also outperform on held-out tasks by an average of 10%.
Additionally, results in Table support that the Mistral base model has stronger gener- alization and ability to handle SKG tasks than CodeLlama, with about a 10% advantage on metrics over all tasks. We see that this strength transfers to the fine-tuned versions of the models.
5 Ablation Studies
Effect of base model pretraining data. We ablate our choice of base model, CodeLlama-
7b-Instruct, by finetuning the unspecialized Llama2-7b base model and Llemma, which is further pretrained mathematical texts (Azerbayev et al., 2023b). Intuitively, one might guess that coding ability has the most transferability to performance on the types SKG tasks studying due to the symbolic nature of programming languages and code-
8

---

Published as a conference paper at COLM 2024
writing scenarios. However, is possible that other types of pretraining to boost reasoning ability, such as math, have even greater transferability.
Our ablation results in Table 6 can be broken down into groupings of tasks, in Figure 4. Models pretrained on code indeed perform slightly better, and these gains are not necessarily limited to tasks which explicitly involve grammatically regular input, or require the generation of code. Math pretraining does seem to improve the performance of the Llama2 base model, but not by as much as code pretraining. Overall, it seems that code pretraining is a useful step in training a performant model in this SKG setting, which may be due to conceptual similarity on certain tasks.
Effect of general instruction data in mixture As we see in Figure 5, the held-in performance is relatively unaffected by the added general examples, but held-out performance improves significantly with more general data. Empirically, we also observe that when training large volume of task-specific input and output formats, the model becomes less capable of following instructions on new tasks in a zero-shot setting. We hypothesize that training on this general mixture helps zero-shot performance because it can reduce overfitting to the task formats in the training set.
Cross-task and cross-format transferability We ablate the transferability of performance between input structure knowledge types and between output task types. To test this, train number of tasks together and compare them to their single-task finetuned models. Our results are indicative that there is cross-task transferability of performance occurring. On tables, we see this effect clearly. This may be explained by the volume and variety of table tasks included in the training mix. In schema and knowledge triples, the performance improvement is not pronounced, perhaps due the limited size of those datasets. Nevertheless, evidence supports that diversifying tasks on the same structured knowledge type benefits performance.
On the other hand, we see that finetuning on different datatypes with the same task (i.e. summarization) also yields benefits performance. On the summarization and question- answering (QA) experiments, we train on both tabular and knowledge graph data. We evaluate summarization with DART and QA with WikiSQL. We see that both cases, the extra dataset yielded about 1% improvement. Considering that the added datasets in each case organize information in a completely different way, this result suggests diversifying data types for similar tasks do indeed benefit each other as well.
Discussion
We argue that SKG is an important capability for future language models. We have seen through our experiments on ChatGPT and the Llama2 family that there is significant room for improvement. We found that we could produce a strong model by focused instruction- tuning on SKG tasks, however, we also observe that the performance difference between 7B to 34B StructLM models was not dramatic. This raises a concern about the state of SKG data: could we be approaching performance ceiling? Combined with the fact that we were able to outperform UT.2-20b, a much larger model, with our 7B model on 3 tasks, it seems that L.MSs at various scales are struggling with SKG capabilities.
Indeed, grounding to structured knowledge directly in a model's input represents chal- lenge in reasoning and input sensitivity. However, has a wide range potential benefits. To meaningfully improve SKG capability, we propose that future work may explore con- tinued pretraining of open foundation models on more structured data formats. Similar to current attempts at code math pretraining, it is possible that pretraining models text interleaved with tables or other types of regular data formatting will help us move towards establishing SKG as foundational model capability.

---

Published as a conference paper at COLM 2024
7 Conclusion
In this paper, we explore the current capabilities of open language models on structured knowledge grounding tasks. We show that LLMs are currently weak at SKG tasks. To address this gap, we construct an instruction-tuning dataset mixture of 1.1M examples and release models that and achieve SOTA on of 18 held-in tasks, and that outperform strong existing models such as TableLlama and Flan UL2 on held-out tasks. We also study the effects of various factors that influence the performance of our model on these task types. We hope that our work provides an updated understanding of what is achievable in the SKG domain, and can serve as a strong baseline for future improvements.
References
Rami Aly, Zhijiang Guo, Michael Sejr Schlichtkrull, James Thorne, Aedseas Vlachos, Christos Christodoulopoulls, Oana Cocarascu, and Arpit Mittal. FEVEROUS: Fact extraction and VERification over unstructured and structured information. 2021.
Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen McAleer, Albert Q. Jiang, Jia Deng, Stella Biderman, and Sean Welleck. Llemma: An open language model for mathematics. CoRR, abs /5300.02330.03 2023a.
Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen McAleer, Albert Q. Jiang, Jia Deng, Stella Biderman, and Sean Welleck. Llemma: An open language model for mathematics, 2023b.
Junwei Bao, Duyu Tang, Nan Duan, Zhao Yan, Yuanhua Lv, Ming Zhou, and Tiejun Zhao. Table-to-text: Describing table region with natural language. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1), April 2018. ISSN 2159-5399. doi: 10.1609. /aaai. v32i1.11944. URL http://dxx doi org/10. 1609//6ai. v32i1. 11944.
Pawel Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Iñigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gašić. MultiWOZ large-scale multi-domain Wizard- of-Oz dataset for task-oriented dialogue modelling. In Ellen Riloff, David Chiang, Julia Hockenmaier, and un'ichi Tsujii (eds.), Proceedings of the 2018 Conference Empirical Meth- ods in Natural Language Processing, pp. 5016-5026, Brussels, Belgium, October-November URL
2018. Association for Computational Linguistics. doi: https://aclanthollog. Drg
Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. Tabfact: A large-scale dataset for table-based fact verification. In International Conference on Learning Representations, 2019.
Wenhu Chen, Hanwen Zha, Zhiyu Chen, Wenhan Xiong, Hong Wang, and William Wang. Hybridqa: A dataset of multi-hop question answering over tabular and textual data. Findings of EMNLP 2020, 2020a.
Zhiyu Chen, Wenhu Chen, Hanwen Zha, Xiyou Zhou, Yunkai Zhang, Sairam Sundaresan, and William Yang Wang. Logic2Text: High-fidelity natural language generation from logical forms. In Trevor Cohn, Yulan He, and Yang Liu (eds.), Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 2096-2111 Online, November 2020b. findings-emnlp.ll
Linguuttts Association for Computational Linguistics. doi: 10.18653/vi
URL https://aclannholopge
2020. iindings-emnyy 190.
Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan Routledge, and William Yang Wang. Finqa: A dataset of numerical reasoning over financial data. Proceedings of EMNLP 2021, 2021.
Zhoujun Cheng, Tianbao Xie, Peng Shi, Chengzu Li, Rahul Nadkarni, Yushi Hu, Caiming Xiong, Dragomir Radev, Mari Ostendorf, Luke Zettlemoyer, et al. Binding language models in symbolic languages. In The Eleventh International Conference on Learning Repre- sentations, 2022.
10

---

Published as a conference paper at COLM 2024
Jordan Clive, Kris Cao, and Marek Rei. Control prefixes for parameter-efficient text genera- tion. arXiv preprint arXiv:2110.08329, 2021.
Yinpei Dai, Hangyu Li, Yongbin Li, Jian Sun, Fei Huang, Luo Si, and Xiaodan Zhu. Preview, attend and review: Schema-awane curriculum learning for multi-domain dialog state tracking, 2021.
Mihail Eric, Lakshmi Krishnan, Francois Charette, and Christopher D. Manning. Key-value retrieval networks for task-oriented dialogue. In Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue. Association for Computational Linguistics, 2017. doi: 10.18653/v1 /v/1///5/1w111/666 URL http://dx //dp: org/1e. 18653/v1//17-5555
Yu Gu, Sue Kase, Michelle Vanni, Brian Sadler, Percy Liang, Xifeng Yan, and Yu Su. Be- yond iid: three levels of generalization for question answering on knowledge bases. In Proceedings of the Web Conference 2021, pp. 3477-3488. ACM.
Deepak Gupta, Surabhi Kumari, Asif Ekbal, and Pushpak Bhattacharyya. MMQA: A multi-domain multi-lingual question-answeringg framework for English and Hindi. In Nicoletta Calzolari, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Koiti Hasida, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Asun- cion Moreno, Jan Odijk, Stelios Piperidis, and Takenobu Tokunaga (eds.), Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan, May 2018. European Language Resources Association (ELRA). URL https://aclanthoolog// wtg.L1/...../----- 1440.
Vivek Gupta, Maitrey Mehta, Pegah Nokhiz, and Vivek Srikumar. INFOTABS: Inference on tables as semi-structured data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 2339923222 Online, July 2020a. Association for Computa- moll-mmain.. 210.
tional Linguistics. URL https: wwwww..www.wwwbb.../////ooo/ org/ orgg/anthooogy//ann
Vivek Gupta, Maitrey Mehta, Pegah Nokhiz, and Vivek Srikumar. Infotabs: Inference on tables as semi-structured data. arXiv preprint #xXi:20055001117 2020b.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset, 2021.
Mohit lyyer, Wen-tau Yih, and Ming-Wei Chang. Search-based neural structured learning for sequential question answering. In Proceedings of the 55th Annual Meeting of the Association for Computational Liaienaalnninnnnn Linguistics (Volume 1: Long Papers), pp- 1821-1831, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 00.18633/v1/P17-111 URL org/P17-1161
Jinhao Jiang, Kun Zhou, Zican Dong, Keming Ye, Wayne Xin Zhao, and Ji-Rong Wen. StructGPT: A General Framework for Large Language Model to Reason over Structured Data, October 2023. URL
www.wxwww....o....eeee www.orrg//bb////ooee 09645. arXiv:2305.0964 00.42009599555. [cs].
Sung-Min Lee, Eunhwan Park, Daeryong Seo, Donghyeon Jeon, Inho Kang, and Seung- Hoon Na. MAFiD: Moving average equipped fusion-in-decoder for question answering over tabular and textual data. In Andreas Vlachos and Isabelle Augenstein (eds.), Findings of the Association for Computational Linguistics: EACL 2023, pp. 2337-2344, Dubrovník, Croatia, May 2023. Association for Computational Linguistics. doi: (0.18633vvl. / findings-eacl.11 URL
org/ 2023. findings-eat catss-eacl.111 177.
Haoran Li, Abhinav Arora, Shuohui Chen, Anchit Gupta, Sonal Gupta, and Yashar Mehdad. MTOP: A comprehensive multilingual task-oriented semantic parsing benchmark. In Paola Merlo, Jorg Tiedemann, and Reut Tsarfaty (eds.), Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 2950-2962, Online, April 2021. Association for Computational Linguistics. doi: 10.18653/ v1 v1/2021.ea 2-0011eccllmall main.257. URL MtR.hhtts:///claathooo /awpaaaaaaaaaaayy...caaa org/ 2221.
257.
11

---

Published as a conference paper at COLM 2024
Haoyang Li, Jing Zhang, Cuiping Li, and Hong Chen. Resdsql: Decoupling schema linking and skeleton parsing for text-to-sq. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 13067-13075, 2023a.
Jinyang Li, Binyuan Hui, Ge Qu, Binhua Li, Jiaxi Yang, Bowen Li, Bailin Wang, Bowen Qin, Ruiying Geng, Nan Huo, Xuanhe Zhou, Chenhao Ma, Guoliang Li, Kevin C. C. Chang, Fei Huang, Reynold Cheng, and Yongbin Li. Can Ilm already serve database interface? big bench for large-scale database grounded text-to-sqls, 2023b.
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Cheng- hao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj, Joel Lamy-Poirier, João Monteiro, Oleh Shliazhko, Nicolas Gontier, Nicholas Meade, Armel Zebaze, Ming-Ho Yee, Logesh Kumar Umapathi, Jian Zhu, Benjamin Lipkin, Muh- tasham Oblokulov, Zhiruo Wang, Rudra Murthy V, Jason Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang, Nour Moustafa-Fahmy, Ur- vashi Bhattacharyya, Wenhao Yu, Swayam Singh, Sasha Luccioni, Paulo Villegas, Maxim Kunakov, Fedor Zhdanov, Manuel Romero, Tony Lee, Nadav Timor, Jennifer Ding, Claire Schlesinger, Hailey Schoelkopf, Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu, Jennifer Robinson, Carolyn Jane Anderson, Brendan Dolan-Gavitt, Danish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. Starcoder: may the source be with you! CoRR, abs/ 2305.06161, 2023c.
Shujie Li, Liang Li, Ruiying Geng, Min Yang, Binhua Li, Guanghu Yuan, Wanwei He, Shao Yuan, Can Ma, Fei Huang, and Yongbin Li. Unifying structured data as graph for data-to- text pre-training. ArXiv, abs/2401.01183, 2024. ÜRL https://apt wewp..ssatatisccoll..... org/
Yujia Li, David H. Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d' Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. Competition-level code generation with alphacode. CoRR, 220000200,78, 2022.
Wing Lian, Guan Wang, Bleys Goodson, Eugene Pentland, Austin Cook, Chanvichet Vong, and "Teknium". Slimorca: An open dataset of gpt-4 augmented flan reasoning traces, with verification, 2023. URL htttps://httts:////eee h/tgttig////uggggggaa..c/. wtt./aacccccoooooo/eeeee
co co/Open-Orca/ immrraa
Guang Liu, Jie Yang, and Ledell Wu. Ptab: Using the pre-trained language model for modeling tabular data. CoRR, abs/ 22...200000 2022.
Qian Liu, Bei Chen, Jiaqi Guo, Morteza Ziyadi, Zeqi Lin, Weizhu Chen, and Jian-Guang Lou. Tapex: Table pre-training via learning neural sql executor, 2021.
Pan Lu, Liang Qiu, Kai-Wei Chang, Ying Nian Wu, Song-Chun
Chun Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan. Dynamic prompt learning via policy gradient for semi- structured mathematical reasoning. In International Conference on Learning Representations (ICLR), 2023.
Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct. arv preprint arXio:2308.0955 095833, 2023a.
Haoran Luo, Haihong E, Zichen Tang, Shiyao Peng, Yikai Guo, Wentai Zhang, Chenghao Ma, Guanting Dong, Meina Song, and Wei Lin. Chatkbqa: A generate- cennerate-the-reetee then-retrieve framework for knowledge base question answering with fine-tuned large language models, 2023b.
12

---

Published as a conference paper at COLM 2024
Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. Wizardcoder: Empowering code large language models with evol-instruct. arviv preprint XXii::300600866633 2023c.
Linyong Nan, Dragomir Radev, Rui Zhang, Amrit Rau, Abhinand Sivaprasad, Chiachun Hsieh, Xiangru Tang, Aadit Vyas, Neha Verma, Pranav Krishna, Yangxiaokang Liu, Nadia Irwanto, Jessica Pan, Faiaz Rahman, Ahmad Zaidi, Mutethia Mutuma, Yasin Tarabar, Ankit Gupta, Tao Yu, Yi Chern Tan, Xi Victoria Lin, Caiming Xiong, Richard Socher, and Nazneen Fatema Rajani. DART: Open-domain structured data record to text generation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 432-447, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/ 11/2021..aacl-main https://aclantthll.g org/2021 naacl-main. 37.
URL
Linyong Nan, Chiachun Hsieh, Ziming Mao, Xi Victoria Lin, Neha Verma, Rui Zhang, Wojciech Kryšciñskii, Hailey Schoelkopf, Riley Kong, Xiangru Tang, Mutethia Mutuma, Ben Rosand, Isabel Trindade, Renusree Bandaru, Jacob Cunningham, Caiming Xiong, Dragomir Radev, and Dragomir Radev. Fetaqa: Free-form table question answering. Transactions of the Association for Computational Linguistics, 10:35-49, 2022. ISSN 2307-387X. http:/dxxx.ooo.ogg////// wttp////....//aa.ll......a 2tt2/tacl.a_0044
doi: 10.1162 ttao///aaa...044...... URL
Ankur P Parikh, Xuezhi Wang, Sebastian Gehrmann, Manaal Faruqui, Bhuwan Dhingra, Diyi Yang, and Dipanjan Das. ToTTo: A controlled table-to-text generation dataset. In
Proceedings of EMNLP, 2020.
Panupong Pasupat and Percy Liang. Compositional semantic parsing on semi-structured tables. In Chengqing Zong and Michael Strube (eds.), Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 1470-1480, Beijing, China, July 2015. Association for Computational Linguistics. doi: 10.3115/v1/P15111 https:///ccanttooogg.orrr///////
URL
Jiexing Qi Jingyao Tang, Ziwei He, Xiangpeng Wan, Yu Cheng, Chenghu Zhou, Xinbing Wang, Quanshi Zhang, and Zhouhan Lin. RASAT: integrating relational structures into pretrained seq2seq model for text-to-sql. In EMNLP, pp. 3215-3229, Association for Computational Linguistics, 2022a.
Jiexing Qi, Jingyao Tang, Ziwei He, Xiangpeng Wan, Yu Cheng, Chenghu Zhou, Xinbing Wang, Quanshi Zhang, and Zhouhan Lin. Rasat: Integrating relational structures into pretrained seq2seq model for text-to-sql. arXiv preprini ar arXio:2205.0698. 06688883,, 2022b.
Cheng Qian, Chi Han, Yi R. Fung, Yujia Qin, Zhiyuan Liu, and Heng Ji. Creator: Tool creation for disentangling abstract and concrete reasoning of large language models, 2023.
Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. Toollim: Facilitating large language models to master 16000+ real-world apis, 2023.
Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD '20 pp. 3505-3506, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 9781450379984. doi: 00.1145/399488.34007777 https://doi.org/ 10. 1145/3394486. 3406703.
URL
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton-Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. Code llama: Open foundation models for code. CoRR, abs/ 2208. 12900, 2023.
13

---

Published as a conference paper at COLM 2024
Chang Shu, Yusen Zhang, Xiangyu Dong, Peng Shi, Tao Yu, and Rui Zhang. Logic- consistency text generation from semantic parses. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Findings of the Association for Computational Linguistics: ACI-IJJNLPP 2021, pp. 4414-4426, Online, August 2021. Association for Computational URL https:///clanntoology/ org/
Linguistics. doi: (0.18853/v1. 111/222...fnnnngge.. 2021. feid.finninng--ae aclls-acl..0 388.
Yiheng Shu and Zhiwei Yu. Data distribution bottlenecks in grounding language models to knowledge bases, 2023.
Jiashuo Sun, Hang Zhang, Chen Lin, Yeyun Gong, Jian Guo, and Nan Duan. Apollo: An optimized training approach for long-form numerical reasoning. arXiv preprint arXtt:2212.07229, 2022.
Alon Talmor and Jonathan Berant. The web as a knowledge-base for answering com- plex questions. In Proceedings of the 2018 Conference of the North American Chapler of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers). Association for Computational Linguistics, 2018. doi: 00.1863/vv1/18--1
URL /tdx: doi. org/10. 1663/v1/N18-11 1059.
Yi Tay, Mostafa Dehghani, Vinh Q. Tran, Xavier Garcia, Jason Wei, Xuezhi Wang, Hyung Won Chung, Dara Bahri, Tal Schuster, Huaixiu Steven Zheng, Denny Zhou, Neil Houlsby, and Donald Metzler. UL2: unifying language learning paradigms. In ICLR. OpenReview.net, 2023a.
Yi Tay, Mostafa Dehghani, Vinh Q. Tran, Xavier Garcia, Wei, Xuezhi Wang, Hyung Won Chung, Siamak Shakeri, Dara Bahri, Tal Schuster, Huaixiu Steven Zheng, Denny Zhou, Neil Houlsby, and Donald Metzler. U12: Unifying language learning paradigms, 2023b.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, lliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. CoRR, abs/ 2000/230..000 .092080988..... 2023.
Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc Le. Finetuned language models are zero-shot learners. In ICLR. OpenReview.net, 2022.
Dongling Xiao, Linzheng Chai, Qian-Wen Zhang, Zhao Yan, Zhoujun Li, and Yunbo Cao. Cqr-sql: Conversational question reformulation enhanced context-dependent text-to-sql parsers, 2022.
Tianbao Xie, Chen Henry Wu, Peng Shi, Ruiqi Zhong, Torsten Scholak, Michihiro Ya- sunaga, Chien-Sheng Wu, Ming Zhong, Pengcheng Yin, Sida I. Wang, Victor Zhong, Bailin Wang, Chengzu Li, Connor Boyle, Ansong Ni, Ziyu Yao, Dragomir Radev, Caim- ing Xiong, Lingpeng Kong, Rui Zhang, Noah A. Smith, Luke Zettlemoyer, and Tao Yu. Unifiedskg: Unifying and multi-tasking structured knowledge grounding with text-to-text language models. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, 2022. doi: 10.18653 v11/2222.mm/. mr0222emlll-main main.39. URL http://dx.. foi.org//0) 18653/ /22/33/vv/2222222 wwt.:22.eenlpmmmmeeeme conlp-main.3.
14

---

Published as a conference paper at COLM 2024
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXit:2304.12244, 2023.
Kuan Xu, Yongbo Wang, Yongliang Wang, Zujie Wen, and Yang Dong, Sead: End-to-end text-to-sql generation with schema-aware denoising. ar arv preprint ar arXit:2105.07911, 2021.
Yunhu Ye, Binyuan Hui, Min Yang, Binhua Li, Fei Huang, and Yongbin Li. Large language models are versatile decomposers: Decomposing evidence and questions for table-based reasoning. In Proceedings of the 46th International ACM SIGIR Conference Research and Development in Information Retrieval, pp. 174-184, 2023.
Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. Spider: A large- scale human-labeled dataset for complex and cross-domain semantic parsing and text- to-sql task. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2018. doi: 10.18653 1/////11/// /s/////d11111/// URL http: //xx www.ii.rr/.... 18653/v1/D18- 1425.
Tao Yu, Rui Zhang, Michihiro Yasunaga, Yi Chern Tan, Xi Victoria Lin, Suyi Li, Heyang Er, Irene Li, Bo Pang, Tao Chen, Emily Ji, Shreya Dixit, David Proctor, Sungrok Shim, Jonathan Kraft, Vincent Zhang, Caiming Xiong, Richard Socher, and Dragomir Radev. semantic parsing in context. In Proceedings of the 57th Annual Meeting of the Association for Computationnl Linguistics. Association for Computational Linguistics, 2019. doi: 10.18653 /v1/ h111pp--114/ URL http: //dx. doi www.drg/.....oooo org/10 18653/v1/P19-1444
Sparc: Cross-
Tao Yu, Chien-Sheng Wu, Xi Victoria Lin, Yi Chern Tan, Xinyi Yang, Dragomir Radev, Caiming Xiong, et al. Grappa: parsing. In International Conference on Learning Representations, 2020.
pre-training for table semantic
Tianshu Zhang, Xiang Yue, Yifei Li, and Huan Sun. TableLlama: Towards Open Large wttp://wwwrrgaa///1.......
Generalist Models for Tables, November 2023. URL http://///rxx abt//31//111111 09206. arXiv:2311.09206 [cs].
Yilun Zhao, Yunxiang Li, Chenying Li, and Rui Zhang. Multihiertt: Numerical reasoning over multi hierarchical tabular and textual data. In ACL (1), pp. 6588-6600. Association for Computational Linguistics, 2022.
Victor Zhong, Caiming Xiong, and Richard Socher. Seq2sql: Generating structured queries from natural language using reinforcement learning. CoRR, abs/ 1709.00103, 2017.
15

---

Published as a conference paper at COLM 2024
A SoTA Results
Dataset
Metric
Split
SOTA Score
Best Performing Model
labMWP
Acc
test
94.7
CREATOR (ChatGPT((iinn et al., 2023)
ToTTo
BLEU
test
49.9
UniK2G (Li al, 2024)
GrailQA
val
77.1
TIARA + GAIN (T5-3B) (Shu Yu, 2023)
SQLZlext
Blec
94.78
LnifiedSKG
MMQA
Fl
85.28
UnifiedSKG
Spider
EM
dev
80.5
RESDSQL el,
KVRet
All Micro
test
67.88
UnifiedSKG
HybridQA
Acc
dev
68.4
212)
SParC
test
68.2
al 2022b)
CompWebQ
76.8
ChatKBQALao et (2023b)
labFact
Acc
test.small
93.0
Dater(Ye et,, 2023)
WikiTQ
All Ex
test
65.9
al,
WikiSQI.
All Ex
test
93.0
GeaD=FeecutionnCCuiied Decoding(Xu et al,, 2021)
FelaQA
BLEU
test
39.0
IableLlama (Zhang et al., 2023)
Feverous
Acc
85.6
FLAN UL2 20b(Tay et al., 2023b)
MultiWOZ
Joint Acc
60.6
TripPy SaCIoog(Dai et al., 2021)
Dart
BLEU
52.0
Control Pretixes (I5-large))Clive al., 2021)
Logic2Text
Blec
test
95.3
UnifiedSKG
МТОP
EM
test
87.5
FLAN 20b(Tay et al., 2023b)
BIRD
Acc
dev
36.6
ChatGPT COT(L et al., 2023b)
CoSQL
EM
tust
58.3
CQR-SQL(Xiao et al., 2022)
SQA
Acc
test
70.5
UL2 20y et al., 2023b)
Infotabs
Acc
dev
75.6
Infotabs paper(Gupta et al., 2020b)
WikiTableText
BLEU
33.7
Unified al., 2024)
Finqa
Acc
private.tess test
71.1
AP'OLOOSSun et al., 2022)
Table 4: Specification of SoTA scores and their sources on the most prevalent metrics used for assessment.
Across the datasets that we evaluate, the SoTA scores are chosen based on methods that do not use agent based methods models large as GPT-4 for fairer comparison. Across these datasets, we can see that many of the best performing methods are purpose-built for the type of data structure used in the task. For example, RESDSQL focuses exclusively controlled SQL generation, DATER uses SQL-based reasoning for tabular tasks, and S3HQA focuses on table and text multi-hop QA. Compared these methods, the performance of StructLM can be seen a strong baseline to determine if these domain-specific designs yield real benefits.
16

---

Published as a conference paper at COLM 2024
B
SlimOrca Dataset Mixture Details
Metric
0%
10%
20%
50%
57%
Held-In Datasets
TabMWP
Ace
71.14
70.35
70.52
69.01
69.36
ToTTo
BLEU
49.78
49.51
49.47
49.31
49.38
GrailQA
EM
81.09
80.46
80.29
80.89
80.38
SQL2Text
Blec
95.07
94.39
94.49
94.97
93.81
MMQA
F1
84.26
84.31
84.11
83.40
85.15
Spider
EM
72.92
71.57
73.40
72.73
72.44
KVRet
All Micro
71.60
73.90
70.34
72.25
72.61
HybridQA
Acc
59.23
59.09
59.09
59.03
59.17
SParC
EM
63.09
62.34
63.26
61.93
CompWebQ
Acc
80.61
79.15
78.76
78.73
78.34
TabFact
Acc
83.41
81.09
81.42
80.92
80.77
WikiTQ
All Ex
50.02
48.50
49.24
48.30
50.09
WikisQL
All Ex
87.33
86.45
86.73
86.68
88.67
FeTaQA
BLEU
36.58
37.26
36.55
36.72
36.03
Feverous
Acc
85,02
84.13
84.11
83.73
84.41
MultiWOZ
Joint Acc
54.66
54.10
53.73
53.92
54.49
Dart
BLEU
61.38
61.89
61.08
62.24
62.24
Logic2Text
Blec
88.83
89.47
89.19
90.57
88.92
MIOP
EM
82.44
81.71
81.19
80.92
81.21
Held-Out Datasets
BIRD
Acc
21.30
22.30
22.30
23.00
22.30
CoSQL
EM
51.24
49.95
50.84
50.74
49.75
SQA
Acc
49.02
46.03
43.11
48.39
49.72
Infotabs
Acc
38.00
56.26
57.87
62.35
62.46
WikiTableText
BLEU
14.78
13.51
6.66
7.27
8.27
Finqa
Acc
19.70
24.32
27.55
25.37
27.29
Table 5: Ablation results for the mixtures of general data in the training set.
In total, we train 5 models, where the percentage represents the percent of the training data that is general. In the held out data, we see noticeable gains in generalization performance for FinQA and InfoTabs datasets. Notably, FinQA requires the generation of python- executable math expression and InfoTabs requires an exact match to 3 previously unseen (boolean) options. WikiTable performance seems suffer, but is evaluated based on the BLEU score with only one target sentence. As result, we place more emphasis on the model's 0-shot adaptation ability new output specifications unseen the training data.
17

---

Published as a conference paper at COLM 2024
C
Per-dataset Pretraining Data Comparison
Tasks
Metric
Code-LM
LLaMA
Math-IM
Held-In Datasets
TabMWP
Acc
71.14
62.96
66.5
ToTTo
BLEU
49.78
48,26
47.4
GrailQA
EM
81.09
75.72
77.66
SQL2Text
Blec
95.07
94.49
94.58
MMQA
84.26
83.96
82.13
Spider
EM
72.92
65.96
71.95
KVRet
All Micro
71.6
70.36
70.03
HybridQA
Acc
59.23
59.26
57.04
SParC
EM
63.09
56.94
60.35
CempWebQ
Acc
80.61
77.31
76.6
TabFact
Acc
83.41
80,46
79.47
WikiTQ
All Ex
50.02
45.6
46.89
WikiSQL
All Ex
87.33
83.93
85.49
FeTaQA
BLEU
36.58
34.37
34.1
Feverous
Acc
85.02
83.2
82.52
MultiWOZ
Joint Acc
54.66
55.43
53.79
Dart
BLEU
61.38
61.52
61.24
Logic2Text
Blee
88.83
88.0
90.38
МТОP
EM
82.44
77.18
75.56
Held- Out Datasets
BIRD
Acc
21.3
15.9
18.8
CoSQL
EM
51.24
48.76
SQA
Acc
49.02
37.03
49.05
Infotabs
Acc
38.0
4.44
32.54
WikiTableeText
BLEU
14.78
13.0
14.82
Finqa
Acc
19.7
6.63
21.53
Table 6: Fine-grained evaluation results comparing finetuning done on different base models. Code refers to CodeLlamm-Instrucct77 Math refers Llemma-7b. LLaMA refers to Llama2- 7b.
In these fine-grained results we can compare the performance of Llemma, Llama, and CodeLlama in more detail. Llemma does seem to hold advantage tasks that involve math (TabMWP, FinQA). Math pretraining does not seem to benefit tabular tasks overall (WikiSQL, WikiTQ, HybridQA, MMQA, etc.), but does show advantages SQL coding tasks (Spider, SparC). CodeLlama, however, seems to show performance improvement the base Llama model not just coding tasks, but also math and tabular tasks as well. These findings may underscore the need to understand what constitutes "reasoning" language model.
18

---

Published as a conference paper at COLM 2024
D Prompt Format

 

You are an AI assistant that specializes in analyzing and reasoning over structured information. You will be given task, optionally with some structured knowledge input. Your answer must strictly adhere to the output format, specified. 

(instruction) (input} 

Figure Prompt format for all SKG examples. This formatting convention follows LLama2 Touvron et al. (2023). The input contains the linearized structured data, together with any other context, question or statement. The instruction specifies the task.
19

---

Published as a conference paper at COLM 2024
E Held-Out Generation Examples
For illustration purposes, we provide examples of successful and unsuccessful responses of StructLM-13B on the FinQA held-out dataset.
E.1
Successful StructLM-13B Response on FinQA
Use the financial data in the following passage and table to calculate the answer the question. table:
performance graph the following graph compares the yearly change in the cumulative total stockholder return for our last five full fiscal years, based upon the market price of our common stock, with the cumulative total return on nasdaq composite index ( u.s. companies and a peer group, the nasdaq medical equipment-sic code 3840-3849 index, which is comprised of medical equipment companies for that period... the performance graph assumes the investment of $ 100 on march 31, 2007 in our common stock, the nasdaq composite index ( u.s. companies and the peer group index, and the reinvestment of any and all dividends..
col: /3/31/ /2007 3/31/2000 / 3/31/2011 3/31/2011 /3/1//2012 row 1: abiomed inc [100 196.19 [35.87 /75.55 [106.37 [162.45 row 2 nasdaq composite index [100 994111 |63.12 99.02 114.84 1127.66 row 3 nasdaq medical equipment sic code 3840- |100 |82.91 |41.56 77.93 94.54 |74.40
this graph is not 201csoliciting material 201d under regulation 14a or 14c of the rules promulgated under the securities exchange act of 1934, is not deemed filed with the securities and exchange commission and is not to be incorporated by reference in any our filings under the securities act of 1933, amended, or the exchange act whether made before or after the date hereof and irrespective of any general incorporation language in any such filing. transfer agent american stock transfer & trust company 59 maiden lane, new york, ny 10038, is our stock transfer agent..
question:
what is the roi of an investment in nasdaq composite index from march 2007 to march 2010? write down short python expression.
(99.02-1 100) / 100
In the above example, StructLM-13B identified the relevant quantity within the given lin- earized table, then determined how calculate the required ROI quantity in python- executable expression.
20

---

Published as a conference paper at COLM 2024
E.2
Unsuccessful StructLM-13B Response FinQA
Use the financial data in the following passage and table to calculate the answer the question. table:
performance graph the following graph and table compares the cumulative five-year total return provided to shareholders on our class common stock relative to the cumulative total returns of the s&p 500 index and our customized peer group- the peer group includes cboe holdings inc. intercontinentalexchang group, inc. and nasdaq inc investment of $ 100 ( with reinvestment of all dividends is assumed to have been made our class common stock, in the peer group and the s&zp 500 index december 31 2012, and its relative performance is tracked through december 31 2017 comparison of 5 year cumulative total return* among cme group inc. the s&p 500 index, and peer group 12/12 12/13 12/14 14 12/15 12/16 cme group inc s&p 500 peer group $ 100 invested on 12/31/11 in stock or index, including reinvestment of dividends. fiscal year ending december 31 copyright a9 2018 standard & poor 2019s, a division of s&p global all rights reserved. the stock price performance included in this graph is not necessarily indicative of future stock price performance..
col: 2013 |2014 [2015 22016 [2017 row 1: cme group inc |S 164.01 |$ 194.06 |$ 208.95 |$ 279.85 $ 370.32 row 2 s&p 500 |132.39 |150.51 [152.59 170.84 [208.14 14 row 3 peer group [176.61 [187.4 48 [219.99 |249.31 |323.23
unregistered sales of equity securities during the past three years there have not been any unregistered sales by the company of equity securities..
question:
in 2017 what was the ratio of the the cme group inc. stock perfomamce to the s&p write down short python expression.
(cme group inc.stocck......o.ooooooo.o... performance / s&p 500 stock performance)
In the above example, StructLM-132 understood how to calculate the required quantity, but could not select the relevant numbers from the given table.
21

---

Published as a conference paper at COLM 2024
Limitations
The collection process used to construct the training data for StructLM tries to include wide a variety of data types. As we have seen, there is evidence that this diversity is capable of affording transferable benefits to each dataset in the mixture. However, the tasks that we train and evaluate on are still academic datasets which have each been curated and designed for a specific purpose. While these are indeed diverse, the SKG domain relies specific formatting and prompting conventions, which may result our models having unnecessary specificity towards the conventions within our train set. To develop clearer picture how SKG performs as its own domain, we may require larger scale datasets with more heterogeneous formatting conventions. Further opportunities for training more robust SKG models may lie increasing the diversity of structured data types in this way.
Additionally, while have tried to evaluate
models to the best of our ability, many of the tasks of our held-out datasets measure accuracy through heuristic matching step of model's output. In zero or few-shot settings, it is quite challenging to exactly control the generations of an autoregressive transformer to be adherent to a certain rule or grammar, and this has been subject of study in may of the other works cited in Table We note that because of this reality, poor results in zero or few-shot context may betray the existence of useful representations that the model has already learned. Without further prompting or finetuning efforts, it may be difficult to bring these capabilities to light. As such, another opportunity for improvement upon our methods may involve more flexible constrained methods of language model evaluation.
22


Reviews:
[{"review_id": "F17dmxbHlR", "reviewer": "Reviewer_qYA4", "summary": "This work introduces StructLM, which provides a series of unified fine-tuned language models (ranging from 7B, 13B and 34B) aiming to solve structured knowledge grounding tasks. The SFT data are collected from 19 existing hold-in SKG tasks (whether 18 or 19 is not sure, typos in Section 3.1), and general instruction tuning corpus. In total, the collected data size is 1.1 M. Experiments demonstrate the effectiveness of StructLM on 18 tasks, outperforming UnifiedSKG on 11 of them using 7B model and 14 of them with 34B model. And they achieve SOTA on 7 tasks among them.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Section 3.1: \"We report results on 18 held-in and 6 held-out tasks\". It seems the actual number of held-in tasks/datasets is 19, not 18, according to Table 1, which gives 19+6=25 SKG tasks.\n\nIt is correct that we have 19 tasks in our held-in dataset, however, we omit reporting TabMWP results in the main table (Table 2) because we could not find baselines that also have TabMWP as a held-in dataset during finetuning. Nevertheless, below we are providing the results of TabMWP for StructLM models. Please feel free to compare with the other results on the leaderboard here.\n\n| Metric | SOTA (Chameleon GPT-4) | StructLM-Mistral-7b | StructLM-7b | StructLM-13b | StructLM-34b |\n| --- | --- | --- | --- | --- | --- |\n| Accuracy | 98.78 | 74.6 | 69.4 | 71.2 | 75.8 |\n\n> Will the collected and preprocessed 1.1 M data be open-sourced? This may promote the development of further SFT work on SKG tasks in future.\n\nyes, the data has been open sourced. \n\nThanks for reading our work and giving your feedback. Our intention with this work was to provide an updated strong baseline for SKG tasks and transparently help the community track progress of the possible state of the art on this important task category. We hope that the dataset we construct and the models that we have trained can have value in this regard.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This work presents abundant experiements on many SKG tasks to demonstrate the effectiveness of SFT on LLMs in solving SKG tasks.\n\n2. The results are satisfactory and achieve SOTA on 7 tasks, while beating other prompting method and similar unified multi-tasking model competitors.", "reasons_to_reject": "1. Although the author emphasizes that StructLM outperforms UnifiedSKG, it is worth mentioning that the largest model of UnifiedSKG is 3B, less than one half of the smallest StructLM. This gives rather unfair comparison.\n\n2. From the engineering perspective, this work is self-contained and complete. However, from the perspective of academics, this work makes little contribution, which merely collects abundant SKG data to train a larger model. And all SKG data are collected from existing benchmarks. This further discounts the effort of this work.", "questions_to_authors": "1. Section 3.1: \"We report results on 18 held-in and 6 held-out tasks\". It seems the actual number of held-in tasks/datasets is 19, not 18, according to Table 1, which gives 19+6=25 SKG tasks.\n\n2. Will the collected and preprocessed 1.1 M data be open-sourced? This may promote the development of further SFT work on SKG tasks in future.", "extra_scores": {}}, {"review_id": "Px3nHJQRVz", "reviewer": "Reviewer_dWUP", "summary": "The paper develops a 1,1Ml example dataset to finetune LLMs for Structured Knowledge Grounding. Extensive experimentation shows that the SKG-specific finetuning effectively improves the sota on a few benchmarks, indicating that SKG may have been neglected in current instruction-tuned models; the study also identifies interesting limitations.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for reading our work and your positive outlook on the value of our contributions. We have released all our models and datasets already, and this should make the concerns about the dataset details more transparent.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The topic is very important. Reasoning with tables is Achilles’s heel of many strong mainstream models. The authors state that they will release the dataset; this could be a valuable resource for the community. Experimentation is extensive; the paper is well written. Interesting discussion and limitation sections.", "reasons_to_reject": "The technological or scientific novelty is somehow limited.  information about how the dataset was created is not very specific", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "r18vYGqGCV", "reviewer": "Reviewer_BN8n", "summary": "This work explores the problem of how large language models (LLMs) handle structured data such as that found in tables, databases, and knowledge graphs. This issue is crucial because, as Jiang et al. (2023) have shown, current powerful LLMs struggle with structured data.\n\nThe paper develops a vast and diverse dataset for instruction tuning, comprising over 1 million samples, and trains various models, termed StructLM, ranging from 7B to 34B parameters. Importantly, these models are general-purpose and are not fine-tuned on specific sources of structured data, yet they perform quite strongly on novel tasks.\n\nAdditionally, the use of code-trained LLMs is an interesting exploration. The insight that code training is beneficial for structured data tasks can be valuable for others, although this insight has already been recognized as the inclusion of code in LLM pre-training data seems to improve reasoning capabilities in LLMs.\n\nThe evaluation is thorough, and appropriate baselines are compared. Overall, this paper appears to be a good fit for COLM, contributing to the community by releasing the model weights, training dataset, and relevant code, which encourages further research and application in structured knowledge grounding (SKG).", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reading our work and your positive outlook on the value of our contributions. \n\n> Increasing the model size from 7B to 34B parameters results in only marginal improvements. Why do the authors think this is the case with reference to scaling laws? Is more data required, or is something else at play here?\n\nWe do observe that the model performance improves as the parameter size increases. However, the curve is more flat than on other coding and math tasks. The problem is in the existing pre-training corpus, which largely omits the structured text (like tables, graphs) on the web. This leads to underfitting in structure text data. Therefore, our argument is that: with the current pre-training data, simply enlarging the model size cannot lead to significant improvement. We advocate adding more structured web data to the pre-training corpus to help LLMs achieve better performance on SKG tasks.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Overall, this paper appears to be a good fit for COLM, contributing to the community by releasing the model weights, training dataset, and relevant code, which encourages further research and application in structured knowledge grounding (SKG).", "reasons_to_reject": "There are not many conceptually novel ideas, and the paper is largely a model training effort, but this is not a show-stopper, and is indeed common and valuable given the weights and dataset that are contributed to the community.", "questions_to_authors": "Increasing the model size from 7B to 34B parameters results in only marginal improvements. Why do the authors think this is the case with reference to scaling laws? Is more data required, or is something else at play here?", "extra_scores": {}}, {"review_id": "qkC2uo6Hjs", "reviewer": "Reviewer_3aLT", "summary": "*Post-rebuttal update*: Based on all the responses, my score remains unchanged. Code/data is already released which is a plus.\n\nThe is a good empirical study of LLM performance on tasks involving structured data (such as knowledge graphs, databases, tables, etc.). The idea is somewhat simple -- measure Structured Knowledge Grounding capabilities analogous to how performance is measured on math related or other knowledge tasks. Nevertheless similar benchmarks don't exist in the literature.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for reading our work, and your kind words on the paper and the value of our contributions. We have already released the code and datasets but unfortunately have not anonymized them so that they can be attached here.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The paper is well written, clear and has all the necessary \"ingredients\". \n* A well evaluated and formulated empirical study \n* Substantial experiments on several different tasks and datasets. I also like that the authors bring in related tasks such as fact verification into the scope of knowledge grounding. This makes a lot of sense but not enough attention has been paid to this task in the LLM evaluation literature. \n* Interesting findings and a fresh view of combining LLMs with structured knowledge.", "reasons_to_reject": "* Code is not shared as part of the submission but I assume it would be released?", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Xx1Z5EhQlB", "reviewer": "Reviewer_fFrk", "summary": "This paper presents a large SKG instruction-tuning dataset which contains 1.1 million data samples. By fine-tuning on both the structured data and general instruction data with several tasks, the derived StructLM outperforms other LMs on structured knowledge related tasks. Besides, it also exhibits generalization capability which could be transferred to unseen structure data.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for reading our work. \n\n> Why don't we use a RAG when using LM for structured data? What's the advantage of StructLM than RAG?\n\nStructLM and RAG are not mutually exclusive. StructLM is actually an important part of RAG.\nRAG is retriever + augmented generation. StructLM focuses on the \"augmented generation\", where the augmented knowledge is presented in structured format. \n\nOn some of our evaluation tasks, we already used retrievers to narrow down to the subgraph and then feed the subgraph as augmented knowledge to StructLM.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed dataset is useful for future researchers since it contains enough structured data which is scarce in previous datasets.\n2. The StructLM outperforms other LM baselines by a large margin, which demonstrates the dataset's practicality and provides a useful LM for structure data prompting.", "reasons_to_reject": "1. The authors seems not testing StructLM on general-sense datasets, which is crutial for a LM.\n2. The performance doesn't surpass the SoTA.", "questions_to_authors": "1. Why don't we use a RAG when using LM for structured data? What's the advantage of StructLM than RAG?", "extra_scores": {}}]

Author rebuttal:

Meta-review: Summary: 
The paper introduces "StructLM," a series of fine-tuned language models aimed at improving performance on Structured Knowledge Grounding (SKG) tasks. The models based on the CodeLlama, ranging from 7B to 34B parameters, are trained on a dataset comprising 1.1 million examples collected from existing SKG tasks and general instruction tuning corpus. The paper demonstrates that StructLM outperforms the current state-of-the-art (SOTA) on several SKG benchmarks.

Strengths:
- The paper presents a thorough evaluation of StructLM across multiple SKG tasks, showing substantial improvements over existing models, including achieving SOTA on several benchmarks.
- The introduction of a large, diverse dataset for instruction tuning is a valuable contribution to the field. The authors also commit to releasing the dataset, model weights, and code, which will benefit the broader research community.
- The paper is well-written and clearly presents its methods, findings, and implications. The empirical study is well-formulated, and the integration of related tasks like fact verification into SKG is a noteworthy approach.

Weaknesses:
- As pointed out by multiple reviewers, the paper's primary contribution lies in the engineering and dataset collection rather than introducing novel scientific concepts. The approach of using code-trained LLMs for structured data tasks, while beneficial, is not entirely new.
- Also, some reviewers mentioned that the comparison between StructLM and other prior models such as UnifiedSKG may be considered unfair due to the significant difference in model sizes. The largest UnifiedSKG model is 3B, whereas the smallest StructLM model is 7B.

Overall, the paper makes a solid contribution to the field of structured knowledge grounding by providing a comprehensive dataset and fine-tuned models that achieve significant improvements over existing approaches. The paper is well-structured and offers practical resources for future research, which enhances its value to the community.
Decision: Accept