StructLM: Towards Building Generalist Models for Structured Knowledge Grounding

Authors: alex  zhuang
ge  zhang
tianyu  zheng
xinrun  du
junjie  wang
weiming  ren
stephen w huang
jie  fu
xiang  yue
wenhu  chen
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-22 16:01:14

Abstract: Structured data sources, such as tables, graphs, and databases, are ubiquitous knowledge sources. Despite the demonstrated capabilities of large language models (LLMs) on plain text, their proficiency in interpreting and utilizing structured data remains limited. Our investigation reveals a notable deficiency in LLMs' ability to process structured data, e.g., ChatGPT lags behind state-of-the-art (SoTA) model by an average of 35\%. To augment the Structured Knowledge Grounding (SKG) capabilities in LLMs, we have developed a comprehensive instruction tuning dataset comprising 1.1 million examples. Utilizing this dataset, we train a series of models, referred to as $\texttt{structlm}$, based on Mistral and the CodeLlama model family, ranging from 7B to 34B parameters. Our $\texttt{structlm}$ series surpasses task-specific models~\citep{UnifiedSKG2022} on 16 out of 18 evaluated datasets and establishes new SoTA performance on 8 SKG tasks. Furthermore, $\texttt{structlm}$ demonstrates strong generalization across 6 novel held-out SKG tasks, outperforming TableLlama by an average of 35\% and Flan-UL2 20B by an average of 10\%. Contrary to expectations, we observe that scaling model size offers marginal benefits, with $\texttt{structlm}$-34B showing only slight improvements over $\texttt{structlm}$-7B. This suggests that structured knowledge grounding is still a challenging task and requires more innovative design to push to a new level. We release the model weights and training dataset to the community, along with relevant code on Github.

Main paper: 
 introduction: 
 Traditionally, users need to write programs to interface with structured data like tables, databases, knowledge graphs, etc. This often requires that they master a the domain-specific language like SQL, SPARQL, etc. or develop domain specific skills for that structured type. Recently, researchers have explored the possibility of automating the interface with natural language to enable potential use cases in question-answering (Pasupat & Liang, 2015; Zhong et al., 2017; Nan et al., 2022) , summarization (Parikh et al., 2020; Nan et al., 2021; Bao et al., 2018) , and fact verification (Aly et al., 2021; Chen et al., 2019; Gupta et al., 2020b) , among others, all grounded to a structured knowledge source. This effort can lower the barrier for end users to access a massive amount of structured data.Previous work (Yu et al., 2020; Liu et al., 2021; Xie et al., 2022; Zhang et al., 2023) has been mostly focused on building task-specific models for different tasks with rather limited generalization ability. Building a generalist structure knowledge grounding (SKG) system across a wide range of tasks proves to be challenging. This is mainly due to the heterogeneity of data format and use cases. We evaluated GPT-3.5-Turbo (Jiang et al., 2023) on 18 SKG tasks and observed that its performance is on average 35% lower than the SoTA specialized models. It shows that the LLM's ability on SKG is heavily overlooked during pre-training.In this paper, we explore the possibility of building a generalist model based on LLMs that can ground on diverse types of structure and unstructured knowledge to interface with humans. Specifically, we construct a large data set of over a million instruction-following examples, a majority of which is SKG data, along with additional general instructionfollowing data, which we find improves generalizability. We fine-tune models at three scales: 7B, 13B, and 34B, based on Mistral and the CodeLlama family of code foundation models. When compared to USKG, we find that our models surpass these single-task models on 16 of 18 tasks. StructLM achieves SoTA on 8 of 18 evaluated tasks, beating ChatGPT by a huge margin. Morever, we show that compared with TableLlama, a recent open language model finetuned for tabular inference tasks, our held-out performance is vastly superior. Compared with Flan-UL2-20B, our models also generalize better overall on the held out tasks by as much as 10% on average.We study the performance of StructLM, namely whether the model experiences crosstask generalization benefits from the dataset mixture, and find that our multi-task model performs significantly better overall than single-task models of the exact same parameter scale. We also study the effect of different pretraining data on our finetuned performance to determine whether special pretraining regimes, such as code or math, contribute to effective SKG reasoning ability. We find that code pretraining is the most effective. We perform additional ablations to confirm our results and support our claims. Our contributions are:• We construct a large SKG instruction-tuning dataset with 1.1 million samples. We train and release our models that outperform the previous 3B USKG fine-tuned on individual tasks on a total of 16 of 18 tasks. StructLM also achieves SoTA results on 8 of them.• We show that StructLM is able to show strong zero-shot generalization capability on unseen structure knowledge grounding tasks, which was not shown by previous models. StructLM outperforms Flan-UL2 20B on 3 of 5 tasks and TableLlama on all 6 tasks. The average absolute improvement over them are 10% and 35%.• We find that mixing in general-domain instruction-tuning data during finetuning preserves generalization ability, and that code-pretrained base models can improve model performance on the SKG tasks. 
solving skg tasks: 
 Structured knowledge, such as web tables, knowledge graphs, and databases, have long been the subject of study in knowledge grounding. However, SKG tasks have heterogeneous data formats which have inspired methods that leverage specific training setups to Figure 1 : StructLM can ground on structured and unstructured knowledge to respond to human queries. The previous SoTA was attained by many different task-specific models like TAPEX (Liu et al., 2021) , USKG (Xie et al., 2022) , TableLlama (Zhang et al., 2023) , BINDER-Codex (Cheng et al., 2022) , etc. StructLM beats the SoTAs on seven SKG tasks. 6, 6, 9, 10, 10, 12, 12, 11, 13, 14, 15, 14, 13, 14, 15, 22 Figure 2: Overview of StructLM. This figure illustrates the prompting structure of StructLM, highlighting its capability to process various forms of structured data beyond linearized data tables, including linearized database schemas and knowledge graphs.learn those representations. For example, PTab (Liu et al., 2022) and MultiHiertt (Zhao et al., 2022) learn the contextual representation of tabular data by incorporating semantic information through specific training methods or reasoning approaches. RASAT (Qi et al., 2022a) integrates relation-aware self-attention with the Transformer seq2seq architecture and utilizes various relational structures to address SQL problems. TAPEX (Liu et al., 2021) conducts pretraining over tabular/database data with the help of an SQL executor to provide supervision.More recently, methods have begun to move away from these auxiliary task-specific structures. USKG (Xie et al., 2022) were the first to unify many SKG tasks into a sequence-tosequence format, allowing them to be aggregated into the same data mixture. However, they were not able to show strong performance improvements to constructing a multi-task mix of SKG data over task-specific tuning methods. StructGPT (Jiang et al., 2023) represents a line of work that uses prompting frameworks on powerful LLMs to solve tasks with more robustness and accuracy. In contrast, our work examines open models and tries to assess their fundamental capabilities. Contemporary to our work, TableLlama (Zhang et al., 2023) has argued that tabular data deserves special attention. Focusing on this domain, their method fine-tunes on several new tabular tasks to improve table understanding, and operates on a longer 8k context length. These improvements can be additive to our work.Instruction-tuning (IT) has been popularized as a method to address the gap between training objectives and user goals in LLMs. This technique involves additional training of LLMs using pairs of instructions and outputs. IT enhances both the controllability and the predictability of the models, aligning them more closely with user expectations. Furthermore, recent studies such as FLAN (Wei et al., 2022) , UL2 (Tay et al., 2023a) , and Llama2 (Touvron et al., 2023) have shown that IT can improve the performance of downstream tasks through multi-task learning across diverse data types. While FLAN-UL2 trains on a subset of 11 SKG tasks, it also trains on many more unrelated language tasks. In our work, by focusing on SKG data, we hope to provide a focused study that can act as a reference for future work to improve performance on this task type.Reasoning stands as a pivotal skill for LLMs in the development of real-world AI applications which would enable the autonomous completion of many thought-intensive tasks viewed traditionally to require human thinking, like programming or mathematical problem-solving (Li et al., 2022) . Recent studies (Li et al., 2022; 2023c; Rozière et al., 2023 ; Azerbayev et al., 2023a) indicate that LLMs trained on code and mathematical datasets exhibit profound reasoning skills, and can even achieve performance on par with human levels. For example, CodeLlama (Rozière et al., 2023) , a foundation model trained on more programming data, has significantly improved reasoning capabilities across a variety of programming and mathematical benchmarks. Furthermore, Llemma (Azerbayev et al., 2023a) continues to pretrain the CodeLlama model on a mix of scientific papers, math-related web data, and mathematical code. Its results show excellent reasoning capabilities on the MATH benchmark (Hendrycks et al., 2021) and the ability to prove theorems without further fine-tuning. On the fine-tuning side, WizardMath (Luo et al., 2023a) , and WizardCoder (Luo et al., 2023c) have shown the effectiveness of instruction tuning on reasoning capabilities, given high quality data.In this work, we view structured data as a third testbed for a different kind of reasoning within LLMs. We posit that in addition to mathematical or logical reasoning, the ability to recognize and make use of patterns within a structured input indicates that a model has robust representations of relationships in data. These representations may serve as a strong prior for further reasoning downstream. 
dataset curation: 
 Motivated by the goal of training a language model generally capable of a wide range of structured data tasks, we select a total of 25 SKG tasks to study. We report results on 18 held-in and 6 held-out tasks, where each held-out task meant to roughly evaluate the generalization capability of a held-in task group. In total, our held-in training dataset contains approximately 700k SKG examples. We describe the held-in dataset groups below.Data to Text Generation. This group of datasets deals with the summarization or interpretation of structured data from tables to knowledge triples to formal languages. Their inclusion is motivated by the idea that useful LMs should be able to make sense of a wide variety of structured information and map it to meaning in natural language. The corresponding held-out dataset for this task group is intended to be WikiTableText.Answering. This group of datasets deals specifically with tabular data, optionally combined with text passages. LMs which are able to accurately answer questions and retrieve information from tables can be widely useful as assistants. The corresponding held-out dataset for this task group is SQA.Knowledge-grounded Conversations. This group of tasks evaluates knowledge grounding in-conversation. Humans naturally interface with LMs through chat, and enabling this capability can lower the barrier to accessing the information in stored structured data. These tasks track user intention through provided dialogue and ask the model to provide an answer to the latest question. The held-out dataset for this task group is CoSQL.Fact verification. One common use case for tables is to reference facts. In addition to question answering, the ability to reliably determine if data in a table supports a statement signals the existence of a robust representation of the table's data. The held-out dataset for this task group is InfoTabs.SQL or domain-specific languages SQL is the language most commonly used to interface with structured data today. Understanding how to write SQL also requires understanding of abstractions of tables and how they are linked together. In other domain-specific languages, the MTOP task measures a model's ability to parse a specification and generate an API call, which sees potential in LLM tool use (e.g., (Qin et al., 2023) ). The corresponding held-out dataset for this task group is intended to be BIRD (Li et al., 2023b) , which further tests SQL generation abilities.An analysis of tabular data may also require performing quick mathematical computations over their contents. Performance on these datasets tells us how well models can combine both structured knowledge and mathematical reasoning.As there are currently a limited number of datasets that combine mathematical reasoning with SKG, this category includes just TabMWP in the held-in corpus. We set FinQA as a challenging held-out dataset analogue. Not only does it require financial domain knowledge, but it combines tabular information with long text passages, and requires the generation of mathematical code.In addition to the SKG datasets within the held-in dataset mixture, we also included general instruction tuning data without any structured knowledge component, to maintain the instruction-following ability of our model. We use SlimOrca (Lian et al., 2023) , which is constructed from cleaned GPT-4 responses to a number of prompts from existing general large-scale instruction-following datasets. We detect no signs of data contamination for our held-out datasets based on our ablation results. We give a detailed overview of all dataset statistics in Table 1 .To instruction tune our model, each example in our dataset consists of a system prompt, instruction, input, and output. For all SKG data examples, we use the same system prompt.For each dataset, we write 10 instruction variations, which are randomized when constructing the training samples. For SKG data, the input is composed of a combination of a structured knowledge input and accompanying text that could be a question, statement, or anything that would be required to specify the task. The prompt is provided in Figure 6 .The base models for StructLM are the CodeLlama-Instruct family of models (Rozière et al., 2023) . We finetune all models with a batch size of 512 for 3 epochs on A800 gpus. We train our 7-34B models on 16-64 GPUs using DeepSpeed ZeRO-3 (Rasley et al., 2020 ) such that the total training time for each model is between 3-5 days. This training setup is largely in line with community conventions, such as the settings used for the WizardLM (Xu et al., 2023) , WizardMath (Luo et al., 2023a) , and WizardCoder (Luo et al., 2023c) models.We follow the structured data linearization conventions in USKG (Xie et al., 2022) . However, we use a different truncation scheme as described below. During training, we maintain a maximum sequence length of 2048. To preserve as much input context as possible, when truncating we consider the combined token length of the prompt input and output label. We truncate only the structured knowledge portion of the input so that the example becomes at most 2048 tokens long. As shown in the dataset statistics in context window with rare truncations. We discard examples for which even this structured input truncation is insufficient (e.g. the output is too long). During inference, we set the input token length to 2048, to allow even more structured information to be placed within the input context. We set the maximum generation length to 1024, which is sufficient for all correct responses in all datasets. For each model, including our single-task finetuned models, we choose the best performing checkpoint of the 3-epoch checkpoints. 
experiments: 
 Baselines Firstly, to illustrate the current performance of language models on SKG tasks, we evaluate ChatGPT (GPT-3.5-turbo) and the base model CodeLlama-7B-Instruct under a 1-shot setting. Our prompting scheme, using the same linearized knowledge structures as in our held-in training, sees them struggle across the board with many tasks due to the unseen structure knowledge format. Although ChatGPT is superior on text-based tasks, its performance is lackluster on SKG tasks. Its gap with SoTA models is as significant as 35%.To evaluate the benefits of our instruction-tuning dataset mix, we also run single-task baseline (each a 7B model) on each task and report their individual performance. We again use CodeLlama-7B-Instruct as the base model for each, and match each single task model on the same number of epochs (3) that was used to train the multitask models, ensuring that each model has seen the same data the same number of times. We observe that our multi-task models outperform these single-task models on nearly every task, with some by a considerable margin of up to 7%. This demonstrates the effectiveness of our instruction tuning dataset and supports the presence of cross-task generalization.When compared to the 18 task-specific USKG models, StructLM-7B can surpass USKG by a average of 2%. From a parameter-count perspective, each of the USKG models is a T5-3B model, which means over the entire held-in set, these results require 54B parameters. Our 7B-M StructLM in comparison can be viewed as being nearly 8x as parameter efficient while Table 2 : The overall evaluation results of our model against other baselines. 7B-M was trained with Mistral-7B as the base model. Cells with "-" in the held-in part mean that the model did not train on this dataset, and results are not comparable. USKG models are overfit to the held-in dataset labels, and thus cannot generalize comparably. Cells in the held-out section with "*" are held-in results. SoTA results are copied from the original papers for reference. ST refers to the single-task fine-tuning result of CodeLlama-Instruct-7B on each dataset. BASE and BASE-M refer to the 1-shot performance of CodeLlama-Instruct-7B, and Mistral-7B-Instruct-v0.2 respectively. ∆ refers to the difference between StructLM and the best known result. score denotes the state-of-the-art score on specific tasks. All StructLM held-out results are 0-shot. Specifications as to how SoTA results are selected are given in Table 4 .still surpassing USKG models on 16 of 18 datasets. It is worth noting that although the single-task (ST) models are more than double the size in parameters compared to USKG, they do not perform much better on average. This fact indicates that there may be significant unused model capacity that can be better utilized via more effective training regimes, such as our instruction tuning. Regarding FLAN-UL2-20B (Tay et al., 2023b) , which was also extensively trained on structure knowledge grounding tasks, StructLM outperforms it on 7 of the 9 mutually held-in datasets. Our results on held-in datasets (Tabfact and FeTaQA) are on par with TableLlama (Zhang et al., 2023) , which is an LLM pre-trained on 2.6M table understanding tasks. We are vastly outperforming TableLlama on the held-out datasets.On held out tasks, StructLM shows strong generalization performance, outperforming ChatGPT on 5 of 6 tasks. These novel tasks contain remarkably different data format than the training dataset. For example, FinQA (Chen et al., 2021) requires the model to generate a mathematical expression to answer a question in the financial domain, while BIRD (Li et al., 2023b) requires the model to interface with multiple databases. These skills are not exactly covered in the training set, thus they put high demand for the models' generalization capabilities. Such generalization capabilities are non-existent in USKG models as they are task-specific. Another table foundation model TableLlama (Zhang et al., 2023) would also fail on most of the held-out tasks with performance close to zero. In contrast, StructLM is much more capable of generalization on these novel tasks. The average improvement over TableLlama is 35%. Compared to another strong foundation Table 3 : Cross task and cross datatype transfer results. FT is an average of single-task performance over the datasets in the Eval column.Figure 5 : Effect of general instruction-following data on averaged held-out SKG dataset performance. Performance is measured as the average over evaluation metrics across all tasks within held-in or held-out groups.structure-knowledge-grounding model Flan-UL2 20B (Tay et al., 2023b) , we also outperform it on held-out tasks by an average of 10%.Additionally, our results in Table 2 support that the Mistral base model has stronger generalization and ability to handle SKG tasks than CodeLlama, with about a 10% advantage on metrics over all tasks. We can see that this strength transfers to the fine-tuned versions of the models. 
ablation studies: 
 Effect of base model pretraining data. We ablate our choice of base model, CodeLlama-7b-Instruct, by finetuning the unspecialized Llama2-7b base model and Llemma, which is further pretrained on mathematical texts (Azerbayev et al., 2023b) . Intuitively, one might guess that coding ability has the most transferability to performance on the types of SKG tasks we are studying due to the symbolic nature of programming languages and code-writing scenarios. However, it is possible that other types of pretraining to boost reasoning ability, such as math, have even greater transferability.Our ablation results in Table 6 can be broken down into groupings of tasks, as in Figure 4 . Models pretrained on code indeed perform slightly better, and these gains are not necessarily limited to tasks which explicitly involve a grammatically regular input, or require the generation of code. Math pretraining does seem to improve the performance of the Llama2 base model, but not by as much as code pretraining. Overall, it seems that code pretraining is a useful step in training a performant model in this SKG setting, which may be due to conceptual similarity on certain tasks.As we see in Figure 5 , the held-in performance is relatively unaffected by the added general examples, but held-out performance improves significantly with more general data. Empirically, we also observe that when training a large volume of task-specific input and output formats, the model becomes less capable of following instructions on new tasks in a zero-shot setting. We hypothesize that training on this general mixture helps zero-shot performance because it can reduce overfitting to the task formats in the training set.We ablate the transferability of performance between input structure knowledge types and between output task types. To test this, we train a number of tasks together and compare them to their single-task finetuned models. Our results are indicative that there is cross-task transferability of performance occurring. On tables, we see this effect clearly. This may be explained by the volume and variety of table tasks included in the training mix. In schema and knowledge triples, the performance improvement is not pronounced, perhaps due to the limited size of those datasets. Nevertheless, evidence supports that diversifying tasks on the same structured knowledge type benefits performance.On the other hand, we see that finetuning on different datatypes with the same task (i.e. summarization) also yields benefits to performance. On the summarization and questionanswering (QA) experiments, we train on both tabular and knowledge graph data. We evaluate summarization with DART and QA with WikiSQL. We see that in both cases, the extra dataset yielded about 1% improvement. Considering that the added datasets in each case organize information in a completely different way, this result suggests diversifying data types for similar tasks do indeed benefit each other as well. 
discussion: 
 We argue that SKG is an important capability for future language models. We have seen through our experiments on ChatGPT and the Llama2 family that there is significant room for improvement. We found that we could produce a strong model by focused instructiontuning on SKG tasks, however, we also observe that the performance difference between 7B to 34B StructLM models was not dramatic. This raises a concern about the state of SKG data: could we be approaching a performance ceiling? Combined with the fact that we were able to outperform UL2-20b, a much larger model, with our 7B model on 3 tasks, it seems that LLMs at various scales are struggling with SKG capabilities.Indeed, grounding to structured knowledge directly in a model's input represents a challenge in reasoning and input sensitivity. However, it has a wide range of potential benefits.To meaningfully improve SKG capability, we propose that future work may explore continued pretraining of open foundation models on more structured data formats. Similar to current attempts at code or math pretraining, it is possible that pretraining models on text interleaved with tables or other types of regular data formatting will help us move towards establishing SKG as a foundational model capability. 
conclusion: 
 In this paper, we explore the current capabilities of open language models on structured knowledge grounding tasks. We show that LLMs are currently weak at SKG tasks. To address this gap, we construct an instruction-tuning dataset mixture of 1.1M examples and release models that and achieve SOTA on 7 of 18 held-in tasks, and that outperform strong existing models such as TableLlama and Flan UL2 on held-out tasks. We also study the effects of various factors that influence the performance of our model on these task types. We hope that our work provides an updated understanding of what is achievable in the SKG domain, and can serve as a strong baseline for future improvements. unregistered sales of equity securities during the past three years there have not been any unregistered sales by the company of equity securities. . 
 
 Appendix: in 2017 what was the ratio of the the cme group inc . stock perfomamce to the s&p write down a short python expression.(cme group inc stock performance / s&p 500 stock performance)In the above example, StructLM-13B understood how to calculate the required quantity, but could not select the relevant numbers from the given table.The collection process used to construct the training data for StructLM tries to include a wide a variety of data types. As we have seen, there is evidence that this diversity is capable of affording transferable benefits to each dataset in the mixture. However, the tasks that we train and evaluate on are still academic datasets which have each been curated and designed for a specific purpose. While these are indeed diverse, the SKG domain relies on specific formatting and prompting conventions, which may result in our models having unnecessary specificity towards the conventions within our train set. To develop a clearer picture of how SKG performs as its own domain, we may require larger scale datasets with more heterogeneous formatting conventions. Further opportunities for training more robust SKG models may lie in increasing the diversity of structured data types in this way.Additionally, while we have tried to evaluate our models to the best of our ability, many of the tasks of our held-out datasets measure accuracy through a heuristic matching step of a model's output. In zero or few-shot settings, it is quite challenging to exactly control the generations of an autoregressive transformer to be adherent to a certain rule or grammar, and this has been a subject of study in may of the other works cited in Table 4 . We note that because of this reality, poor results in zero or few-shot context may betray the existence of useful representations that the model has already learned. Without further prompting or finetuning efforts, it may be difficult to bring these capabilities to light. As such, another opportunity for improvement upon our methods may involve more flexible constrained methods of language model evaluation. 


Decision: Summary: 
The paper introduces "StructLM," a series of fine-tuned language models aimed at improving performance on Structured Knowledge Grounding (SKG) tasks. The models based on the CodeLlama, ranging from 7B to 34B parameters, are trained on a dataset comprising 1.1 million examples collected from existing SKG tasks and general instruction tuning corpus. The paper demonstrates that StructLM outperforms the current state-of-the-art (SOTA) on several SKG benchmarks.

Strengths:
- The paper presents a thorough evaluation of StructLM across multiple SKG tasks, showing substantial improvements over existing models, including achieving SOTA on several benchmarks.
- The introduction of a large, diverse dataset for instruction tuning is a valuable contribution to the field. The authors also commit to releasing the dataset, model weights, and code, which will benefit the broader research community.
- The paper is well-written and clearly presents its methods, findings, and implications. The empirical study is well-formulated, and the integration of related tasks like fact verification into SKG is a noteworthy approach.

Weaknesses:
- As pointed out by multiple reviewers, the paper's primary contribution lies in the engineering and dataset collection rather than introducing novel scientific concepts. The approach of using code-trained LLMs for structured data tasks, while beneficial, is not entirely new.
- Also, some reviewers mentioned that the comparison between StructLM and other prior models such as UnifiedSKG may be considered unfair due to the significant difference in model sizes. The largest UnifiedSKG model is 3B, whereas the smallest StructLM model is 7B.

Overall, the paper makes a solid contribution to the field of structured knowledge grounding by providing a comprehensive dataset and fine-tuned models that achieve significant improvements over existing approaches. The paper is well-structured and offers practical resources for future research, which enhances its value to the community.

Reviews: review: paper_summary: paper_summary: This work introduces StructLM, which provides a series of unified fine-tuned language models (ranging from 7B, 13B and 34B) aiming to solve structured knowledge grounding tasks. The SFT data are collected from 19 existing hold-in SKG tasks (whether 18 or 19 is not sure, typos in Section 3.1), and general instruction tuning corpus. In total, the collected data size is 1.1 M. Experiments demonstrate the effectiveness of StructLM on 18 tasks, outperforming UnifiedSKG on 11 of them using 7B model and 14 of them with 34B model. And they achieve SOTA on 7 tasks among them.
strength_weakness: strength_weakness: 1. Although the author emphasizes that StructLM outperforms UnifiedSKG, it is worth mentioning that the largest model of UnifiedSKG is 3B, less than one half of the smallest StructLM. This gives rather unfair comparison.

2. From the engineering perspective, this work is self-contained and complete. However, from the perspective of academics, this work makes little contribution, which merely collects abundant SKG data to train a larger model. And all SKG data are collected from existing benchmarks. This further discounts the effort of this work.
questions: questions: 1. Section 3.1: "We report results on 18 held-in and 6 held-out tasks". It seems the actual number of held-in tasks/datasets is 19, not 18, according to Table 1, which gives 19+6=25 SKG tasks.

2. Will the collected and preprocessed 1.1 M data be open-sourced? This may promote the development of further SFT work on SKG tasks in future.
paper_summary: paper_summary: The paper develops a 1,1Ml example dataset to finetune LLMs for Structured Knowledge Grounding. Extensive experimentation shows that the SKG-specific finetuning effectively improves the sota on a few benchmarks, indicating that SKG may have been neglected in current instruction-tuned models; the study also identifies interesting limitations.
strength_weakness: strength_weakness: The technological or scientific novelty is somehow limited.  information about how the dataset was created is not very specific
paper_summary: paper_summary: This work explores the problem of how large language models (LLMs) handle structured data such as that found in tables, databases, and knowledge graphs. This issue is crucial because, as Jiang et al. (2023) have shown, current powerful LLMs struggle with structured data.

The paper develops a vast and diverse dataset for instruction tuning, comprising over 1 million samples, and trains various models, termed StructLM, ranging from 7B to 34B parameters. Importantly, these models are general-purpose and are not fine-tuned on specific sources of structured data, yet they perform quite strongly on novel tasks.

Additionally, the use of code-trained LLMs is an interesting exploration. The insight that code training is beneficial for structured data tasks can be valuable for others, although this insight has already been recognized as the inclusion of code in LLM pre-training data seems to improve reasoning capabilities in LLMs.

The evaluation is thorough, and appropriate baselines are compared. Overall, this paper appears to be a good fit for COLM, contributing to the community by releasing the model weights, training dataset, and relevant code, which encourages further research and application in structured knowledge grounding (SKG).
strength_weakness: strength_weakness: There are not many conceptually novel ideas, and the paper is largely a model training effort, but this is not a show-stopper, and is indeed common and valuable given the weights and dataset that are contributed to the community.
questions: questions: Increasing the model size from 7B to 34B parameters results in only marginal improvements. Why do the authors think this is the case with reference to scaling laws? Is more data required, or is something else at play here?
paper_summary: paper_summary: *Post-rebuttal update*: Based on all the responses, my score remains unchanged. Code/data is already released which is a plus.

The is a good empirical study of LLM performance on tasks involving structured data (such as knowledge graphs, databases, tables, etc.). The idea is somewhat simple -- measure Structured Knowledge Grounding capabilities analogous to how performance is measured on math related or other knowledge tasks. Nevertheless similar benchmarks don't exist in the literature.
strength_weakness: strength_weakness: * Code is not shared as part of the submission but I assume it would be released?
paper_summary: paper_summary: This paper presents a large SKG instruction-tuning dataset which contains 1.1 million data samples. By fine-tuning on both the structured data and general instruction data with several tasks, the derived StructLM outperforms other LMs on structured knowledge related tasks. Besides, it also exhibits generalization capability which could be transferred to unseen structure data.
strength_weakness: strength_weakness: 1. The authors seems not testing StructLM on general-sense datasets, which is crutial for a LM.
2. The performance doesn't surpass the SoTA.
questions: questions: 1. Why don't we use a RAG when using LM for structured data? What's the advantage of StructLM than RAG?

Comments: title: Revision of scores
comment: I have read the response and willing to improve the score to 6: Marginally above acceptance threshold. The reason is that:

1.  The author explains the reason why he/she misses one entry in the tasks table, which seems reasonable.

2. The author promises that their preprocessed data will be open-sourced which will greatly promote the development of community on table tasks.

3. However, I still maintain my opinion that the novelty of this work is eaten by previous work UnifiedSKG, and the author offers an unfair comparison regarding model size with that work.