Structure-aware Domain Knowledge Injection for Large Language Models

## Abstract

Abstract This paper introduces a pioneering methodology, termed StructTuning , to efficiently transform foundation Large Language Models (LLMs) into domain specialists.
It significantly reduces the training corpus requirement to a mere 0.3% , while achieving an impressive 50% of traditional knowledge injection performance.
Our method is inspired by the educational processes of human students, particularly how structured domain knowledge from textbooks is assimilated and subsequently applied to tackle real-world challenges through specific exercises.
Based on this, we propose a novel two-stage strategy for knowledge injection and alignment: Structure-aware Continual Pre-Training (SCPT) and Structure-aware Supervised Fine-Tuning (SSFT).
In the SCPT phase, we automatically extract the domain knowledge taxonomy and reorganize the training corpora, enabling LLMs to effectively link textual segments to targeted knowledge points within the taxonomy.
In the SSFT phase, we explicitly prompt models to elucidate the underlying knowledge structure in their outputs, leveraging the structured domain insight to address practical problems.
Our ultimate method has undergone extensive evaluations across model architectures and scales, using closed-book question-answering tasks on LongBench and MMedBench datasets.
Remarkably, our method demonstrates the potential of comparable improvement against the state-of-the-art MMedLM2 on MMedBench, while significantly reducing the training costs to 5% .
This breakthrough paves the way for scaling up our StructTuning for stronger domain-specific LLMs with comprehensive data utilization.
Code is available at https://github.com/alibaba/struxgpt .

## 1 Introduction

Large language models (LLMs) have recently seen extensive deployment across various applications (Vaswani et al., ; Achiam et al., ; Jiang et al., ; Bi et al., ).
When adapting foundational models (*e.g.*, Llama series (Touvron et al., ; ; Dubey et al., )) for specialized AI assistants in distinct domains (Qiu et al., ; Guo et al., ), developers usually employ two techniques to enhance LLMs’ proficiency: retrieval-augmented generation (RAG) (Lewis et al., ) and domain knowledge injection (Gururangan et al., ).
While RAG effectively utilizes an external knowledge base to augment information, the retrieval process’s inherent noise poses challenges to generating reliable responses, especially in scenarios requiring logical reasoning where there is a semantic gap between the user’s query and the knowledge base.(Zhang et al., ; Chen et al., ).
Thus, another avenue of research focuses on injecting new knowledge into models via training techniques (Gu et al., ; Hu et al., ; Mecklenburg et al., ).

Continual pre-training (Sun et al., ; Ibrahim et al., ) has been preferred for integrating new, domain-specific knowledge into existing LLMs (Cui et al., ; Wang et al., ; Qiu et al., ).
Nevertheless, it often entails resource-intensive auto-regressive training on billions of tokens from the internet to learn fragmented knowledge points, rather than absorbing structured knowledge from a few domain-specific textbooks (Jin et al., ).
For example, MMedLM (Qiu et al., ) curates 25.5B tokens to derive a medical model, and DeepSeek-Coder (Guo et al., ) uses 2T tokens for coding adaptation.
The common failure to learn effectively from limited textbook content has been attributed to insufficient data diversity (Zhu & Li, ), which however violates the observation during the human education process in Fig. 1: students gain knowledge by sequentially studying from textbooks, reviewing knowledge points and structures, and applying this knowledge through proper exercises.
In this process, all the new data to learn are textbooks (structured content) and exercising examples (question-answering pairs), and students just adopt their world knowledge to memorize, understand, and apply the knowledge to become domain experts (Krathwohl, ; Yu et al., ).

Figure: Figure 1: Discrepancy between human learning process and vanilla LLM adaptation paradigm. Human students learn structured knowledge through textbooks section by section, with particular exercises on related knowledge points. Traditional LLM adaptation involves continual pre-training on data chunks from randomly concatenated text segments, with aimless supervised fine-tuning for conversation alignment. The inherent property of structured knowledge is ignored.
Refer to caption: x1.png

Inspired by this, we propose to inject the domain knowledge from textbooks into LLMs, as educating a human student, through a novel two-stage training strategy: Structure-aware Continual Pre-Training (SCPT) and Structure-aware Supervised Fine-Tuning (SSFT).

In the SCPT stage, we argue that high-quality textbook data can adequately infuse the domain knowledge (Gunasekar et al., ), where the organization of training corpora is crucial.
In conventional paradigms, as illustrated in Fig. 1, text corpora are simply concatenated and divided into chunks of 2048 (Qiu et al., ) or 4096 (Guo et al., ), while the inherent structure of the texts (*e.g.*, catalogs of textbooks) is disregarded.
Instead, we propose an automatic approach to maintain each chunk’s knowledge structure.
We view each chunk as a knowledge point, and employ advanced LLMs to efficiently extract domain knowledge taxonomy from the corpus, bypassing the need for manual annotation.
Subsequently, LLMs are trained to predict the textual content (corresponding to the knowledge point) under the condition of the knowledge path within the domain structure, linking individual training chunks with the entire knowledge architecture.
Finally, models are asked to memorize the knowledge structure to review the whole domain knowledge system.

In the SSFT stage, the goal shifts from knowledge injection to enabling LLMs to recall and utilize their acquired knowledge to tackle real-world challenges.
We explicitly elicit knowledge paths in LLMs’ responses, as a beacon for models to targeted information retrieval or logical reasoning for reliable responses.
To this end, we derive a scalable strategy to generate question-answer pairs as practice exercises by powerful LLMs such as GPT4 (Achiam et al., ) or LLaMA3 (Dubey et al., ).
In the scenarios with existing QA pairs like MMedBench (Qiu et al., ), we retrieve related knowledge structure and content, instructing LLaMA3 to provide explanations from questions to answers based on the knowledge paths.
For datasets lacking specific QA samples like LongBench (Bai et al., ), we randomly select knowledge paths from the domain taxonomy and prompt LLaMA3 to craft question-answer-explanation triplets for training exercises.

Our ultimate approach, termed StructTuning, outperforms conventional methods in domain knowledge injection by emulating human learning processes through SCPT and SSFT phases.
We extensively evaluate StructTuning’s effectiveness across different model architectures and sizes.
For domain-adapted language models, we first examine their capability to recall the injected knowledge through open-ended QA on the LongBench (Bai et al., ) dataset, then assess their application of this knowledge in addressing real-world issues via multiple-choice QA on MMedBench (Qiu et al., ).
Both evaluations underscore the superiority of StructTuning.
Remarkably, we achieve a 50% improvement in knowledge injection compared to the SOTA MMedLM2 in the medical domain, using merely 0.3% of the training data requirement. And StructTuning illustrates the potential of comparable performance with only 5% of training costs. These findings reveal the scalability of our method for enhancing domain-specific AI assistants with further comprehensive data utilization.

Our contribution is summarized as follows:

- •
We proposed a novel two-stage training strategy, SCPT and SSFT, to inject domain knowledge into LLMs by preserving and utilizing the inherent structure of the training corpus.
- •
We developed a scalable data construction framework to generate structure-aware training samples from original corpora, so as to facilitate the SCPT and SSFT stages.
- •
We conducted extensive investigations on our StructTuning strategy on various data and model settings, and comprehensively illustrate our superiority in knowledge injection.

## 2 Related Work

Domain Adaptation for Large Language Models.
While pre-trained LLMs possess promising capabilities, their performance is often hampered by the scope and recency of their training data, which particularly affects smaller models in downstream applications (Zhao et al., ; Wang et al., ).
Continual Pre-Training (CPT) addresses this by perpetually updating a pre-trained model with domain-specific content (Sun et al., ; Xu et al., ).
To keep pace with the latest information, models can be fine-tuned with supervised instruction-response pairs (SFT), thus staying current with the advancing knowledge landscape (Mecklenburg et al., ; Qiu et al., ).
Existing literature confirms that combining CPT and SFT is effective for LLMs to remain precise and up-to-date in dynamic fields like law (Cui et al., ; Nguyen, )
and medicine (Wang et al., ; Qiu et al., ).
Our study builds upon this CPT-SFT framework, innovating with SCPT-SSFT strategies to efficiently and effectively infuse domain knowledge with the inherent structure hierarchy.

Structure-aware Knowledge Aggregation.
A branch of researchers follows the conventional paradigm to extract entity-relation-entity triplets from texts to construct knowledge graphs (Pan et al., ), to enhance LLMs’s factual knowledge and logical reasoning by feature aggregation (Liu et al., ; Zhang et al., ), prompt engineering (Wen et al., ; Wang et al., ),
training data synthesis (Tang et al., ), etc.
In these cases, each node corresponds to either a specific entity or an abstract concept, lacking the capability to present an informative and self-contained knowledge point.
Some works have recently related a piece of descriptive text to a knowledge point, and constructed the knowledge structure for LLMs’ retrieval-augmented generation (Sarthi et al., ; Dong et al., ), where the top-to-down retrieval provides precise information-seeking paths along the knowledge structure.
In this paper, we extend the structure-aware knowledge aggregation to LLMs’ training phase, injecting the whole domain knowledge structure into LLMs’ by linking the training samples into corresponding knowledge points and reasoning paths.

Data Augmentation and Synthesis.
Traditional methods aim to artificially expand the training dataset size (Xu et al., ; Mukherjee et al., ) or generate entirely new samples that could help models learn better or adapt to specific tasks (Tang et al., ).
Yet, they often overlook the structured nature of domain knowledge, and the aimlessly generated samples may also lack diversity (Ovadia et al., ; Mecklenburg et al., ), leading to potentially suboptimal training outcomes when applied for domain adaptations (Mecklenburg et al., ; Tang et al., ).
By contrast, our SSFT design is an innovative departure to address the challenge of retaining and utilizing the structured knowledge inherent in domain-specific content.

## 3 Methodology

Fig. 2 depicts our StructTuning methodology to inject domain knowledge into pre-trained LLMs using the inherent knowledge structure.
With curated domain corpora (typically a few textbooks), we first develop an automatic approach to extract the knowledge structure, and associate text chunks to corresponding knowledge paths and points (Sec. 3.1).
Then, we design a two-stage training strategy to inject the highly structured domain knowledge into language models by mimicking the human education process, comprising the SCPT (Sec. 3.2) and SSFT (Sec. 3.3) techniques.

Figure: Figure 2: Framework for structure-aware knowledge injection. We extract the inherent knowledge structure in the training corpus, and associate training chunks to corresponding knowledge points. Models are continually pre-trained on data chunks in the condition of the knowledge structure, and fine-tuned with supervised QA samples to elicit their learned knowledge to solve real-world questions (including knowledge-intensive (KI) QA, 2- or multi-hop QA, *etc.*).
Refer to caption: x2.png

### 3.1 Automatic Extraction of Knowledge Structure

For web-crawled corpus, previous data pre-processing focuses on quality assessment for individual documents (Bi et al., ), while the meta-info of knowledge structures (*e.g.*, the table content for a textbook) is usually neglected or filtered out, and all we have are those sequentially arranged text segments (*e.g.*, page-by-page-chunked content).
As shown in Fig. 2 (a), we aim to extract (or, recover) the knowledge structure from the raw corpus for subsequent domain knowledge injection.

First, we use spaCy(^1^11https://github.com/explosion/spaCy) to split the content from a textbook at the paragraph-level, and merge the sentences to form training chunks within a maximum size (*e.g.*, 2048 tokens (Qiu et al., )).
After that, we prompt the advanced Llama3-70B (Dubey et al., ) model to summarize the title for each chunk, where the textual content with the abstractive title jointly contributes to a “knowledge point”.

Then, we automatically aggregate knowledge points and extract the inherent structure hierarchy by leveraging advanced language models.
Inspired by Liu et al. (), we take the title list to instruct a specifically developed 7B model to identify the inherent knowledge structure within the text chunks.
A prompt example is displayed in Fig. 3, and the detailed implementation is presented in Sec. B.1.
In particular,
Sec. B.1 and Sec. B.2 verify that our specialized 7B model can identify sufficiently precise knowledge structure for effective and efficient domain adaptation, as more powerful LLMs like LLaMA3-70B (Dubey et al., ) and GPT-3.5 (Brown et al., ) cannot bring significant enhancement while largely increase the inference costs.

Fig. 3 presents an example of the extracted structure, where we use the tree-like mindmap structure (Wen et al., ) to present the knowledge taxonomy from a textbook.
And Fig. A1 showcases how to deal with non-textbook data.
The whole process does not involve human annotation, which reduces the cost and makes our method scalable for larger domain training corpora.

Figure: Figure 3: Left: prompt template to extract hierarchical knowledge structures from the given content. Right: example of the extracted knowledge structure presented by a mindmap format.
Refer to caption: x3.png

After automatically extracting the domain knowledge structure and associating the original training chunks to related knowledge points, we delve into injecting domain knowledge through structure-aware continual pre-training (SCPT) and structure-aware supervised fine-tuning (SSFT).

### 3.2 Structure-aware Continual Pre-Training

In conventional knowledge injection methods, training corpora are randomly concatenated and chunked into text segments without distinguishing the original content, leading to the fact that models can only absorb domain knowledge that is emergent in the data diversity (Ovadia et al., ; Mecklenburg et al., ; Qiu et al., ).
In this section, we present another solution to inject knowledge from limited pieces of textbooks by leveraging the highly abstractive and exhaustive domain knowledge structures for continual pre-training.

Figure: Figure 4: Example of prompt templates to bridge mindmap structure and textual contents.
Refer to caption: x4.png

We first transform the knowledge structure into natural languages using the same mindmap template in Fig. 3, and prepend it to each training chunk, forcing LLMs to memorize the textual content (knowledge points) in the condition of the associated knowledge path in the structure hierarchy.
We collected 20 diversified templates from GPT-4 (Achiam et al., ) to bridge mindmap structures and training chunks, one of which is displayed in Fig. 4.
The prepended mindmap, as well as the template, does not produce auto-regressive loss.
Losses are only calculated in the content part.
Formally, we turn the original language modeling in vanilla CPT to conditioned modeling (Keskar et al., ) in our SCPT stage:

$$ $\displaystyle p({\bm{x}}^{k})=\prod_{i=1}^{n}{p(x^{k}_{i}|x^{k}_{<i})}\quad \Longrightarrow\quad p({\bm{x}}^{k}|{\bm{s}}^{k})=\prod_{i=1}^{n}{p(x^{k}_{i}| x^{k}_{<i},{\bm{s}}^{k})}$ (1) $$

where $p({\bm{x}}^{k})$ models the probability distribution for the $k$-th chunk ${\bm{x}}^{k}=(x^{k}_{1},\cdots,x^{k}_{n})$ via the chain rule of probability (Bengio et al., ) on each token $x^{k}_{i}$, and ${\bm{s}}^{k}$ denotes the associated knowledge mindmap paths.

As illustrated in Fig. 2 (b), after traversing the $m$ knowledge points in extracted structures, models are asked to recall the whole knowledge hierarchy, *i.e.*, to model the composed probability distribution:

$$ $\displaystyle p(\bar{{\bm{s}}})=\prod_{k=1}^{m}{p({\bm{s}}^{k})}$ (2) $$

In SCPT, we mimic the human education process to inject knowledge into LLMs in a section-by-section manner, and replay the entire knowledge structure for the models to review and summarize the learned domain knowledge.
These two steps iteratively alternate throughout training epochs.

Next, we will introduce how to teach LLMs to explicitly utilize their domain knowledge, which is learned in the SCPT stage, to solve practical problems by doing exercises with our SSFT technique.

### 3.3 Structure-aware Supervised Fine-Tuning

In traditional knowledge injection paradigms, supervised fine-tuning serves as a tool to align the (continually) pre-trained models to interactive chatbots, *e.g.*, through massive question-answering exercises (Cui et al., ; Qiu et al., ).
However, most QA data augmentation or generation strategies aim at enlarging the quantity and enhancing the diversity of training samples (Xu et al., ; Mukherjee et al., ; Liu et al., ), which neglects the nature of the highly structured domain knowledge.
Therefore, our structure-aware supervised fine-tuning (SSFT) technique focuses on eliciting models’ structured knowledge learned during the SCPT stage, adapting LLMs to interactive and reliable domain experts.

Fig. 2 (c) illustrates the idea of synthesizing SSFT samples guided by domain knowledge structures.
Specifically, we use the random walk algorithm to create knowledge paths with $1$ to $l$ branches in the original mindmap.
For paths linking to a single knowledge point, we use the corresponding text content to prompt Llama3-70B (Dubey et al., ) to generate knowledge-intensive question-answering pairs.
For paths with two or more branches, we prompt Llama3-70B with the knowledge path and textual contents to synthesize 2- or multi-hop QA samples, which require specific reasoning along the knowledge structure to derive from questions to answers.
Fig. 5 presents several examples.

Figure: Figure 5: QA samples synthesized for SSFT. We instruct Llama3-70B to generate (a) knowledge-intensive and (b) multi-hop questions and derive the diagnosis answers with explicit reasoning.
Refer to caption: x5.png

For every synthesized QA sample (${\bm{z}}$), we will prepend the relevant mindmap hierarchy to the answer, and add a CoT prompt in the question to construct another type of QA data (${\bm{z}}^{\prime}$) for SFT alignment.
This design explicitly elicits the learned knowledge in models’ responses, teaching them how to apply the structured knowledge to address real-world problems.
We use the two types of QA samples for training, as recommended by Qiu et al. ().
During testing, we can either use the vanilla question as input to efficiently gather models’ answers, or take the CoT prompt to probe to what extent LLMs can memorize and leverage the injected knowledge to answer the questions.

Integrating with SCPT and SSFT, our StructTuning approach translates into remarkable efficacy and efficiency in domain knowledge injection, as comprehensively evaluated in the following sections.

## 4 Experiments

We design a comprehensive evaluation of our StructTuning through several experiments on two benchmarks.
First, we investigate the free-form question-answering task on the LongBench (Bai et al., ) dataset, in order to verify the memorization and understanding of injected knowledge (the answer can be directly found in training corpora).
Then, we delve into the multi-choice question-answering task on MMedBench (Qiu et al., ), to explore how LLMs apply the injected knowledge in basic medicine to determine the real-world diagnosis for patients with logical reasoning.

### 4.1 Preliminary Investigation on Free-form Question-Answering

**Table 1: Recall evaluation of Open-Book QA (OBQA) and Closed-Book QA (CBQA) tasks on the LongBench (Bai et al., ) dataset. The best results are marked in bold, and the secondary results are marked with underlines. The backbone model is Llama2-7B (Touvron et al., ).**
| Task | Adaptation | SingleDoc-QA | MultiDoc-QA | Average |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Qasper | MFQA | MFQAzh | HpQA | 2Wiki | Musiq | Duzh |  |  |  |
| OBQA | - | 39.7 | 44.3 | 17.5 | 30.1 | 35.5 | 11.9 | 9.6 | 26.9 |
| CBQA | CPT+SFT | 20.7 | 35.3 | 20.6 | 29.9 | 32.1 | 18.9 | 12.0 | 24.2 |
| SCPT+SFT | 18.8 | 42.5 | 17.7 | 35.7 | 36.4 | 20.5 | 15.3 | 26.7 |  |
| SCPT+SSFT | 30.5 | 44.6 | 24.3 | 40.8 | 42.0 | 21.8 | 16.8 | 31.5 |  |

Datasets and Tasks.
LongBench (Bai et al., ) is a multi-task benchmark tailored for open-book reading comprehension evaluation, where LLMs generate answers to given questions based on one or several input passages.
To focus on knowledge injection, we turn the open-book evaluation into a closed-book QA task, where LLMs are trained on contextual passages and queried with questions only.
We choose 7 subsets with 1,350 test examples from LongBench for single- and multi-document question-answering evaluation, and the remaining synthetic or code-orientated tasks are eliminated.
More details are described in Sec. A.1.

Evaluation Metrics.
To quantify the knowledge memorization degree, we mainly report the recall (Zhu & Li, ) for models’ outputs against ground-truth answers for the free-form QA tasks.
In Sec. B.3 we also evaluate our method with the F1-score measure for a thorough comparison.

Investigated Models.
For LongBench, we mainly investigate the knowledge injection to Llama2-7B (Touvron et al., ) to compare the open- and closed-book QA performance.

Implementation Details.
For closed-book QA, the Llama2-7B model is continually pre-trained on 10,476 passages for 3 epochs using a batch size of 128. The initial learning rate is 2e-5, decayed to 0 at the end of training with a cosine scheduler. Then, we query Llama3-70B to generate 2,700 SSFT QA examples and remove those with over 0.5 F1-Score similarity to test samples to prevent data leakage.
We train the knowledge-injected models for 1 epoch to avoid overfitting.

Figure: Figure 6: An example for structure-aware responses.
Refer to caption: x6.png

Main Results.
First, we use a CoT instruction of “Answer the question and explain why” to set a baseline for Open-Book QA (OBQA).
As shown in Tab. 1, it presents a moderate knowledge comprehending capability in SingleDoc QA tasks (*e.g.*, over 40% recall on Qasper and MFQA), and relatively poor knowledge coordination in MultiDoc QA subsets (*e.g.*, around 10% recall on Musiq and Duzh) due to the attention drift on longer inputs.

Then, we try to inject passage content into LLMs with a conventional CPT+SFT paradigm to benchmark the Closed-Book QA (CBQA) baseline.
Note that we use two types of question templates (refer to Sec. 3.3) during training, and use the CoT version during testing to elicit models’ memorized knowledge in their responses.
However, Tab. 1 indicates such a knowledge injection approach is ineffective, as the knowledge recall is even lower than the OBQA baseline (*i.e.*, 24.2% v.s. 26.9%).

On the contrary, our SCPT strategy has already achieved a considerable knowledge recall of 26.7%, which is higher than the CPT-SFT paradigm of 24.2% and approaching the OBQA baseline of 26.9%.
It implies our structure-aware continually pre-trained model has successfully associated the relevant passages with their entire knowledge structure for the given question, which provides the knowledge path to seek targeted information to derive the answer.

Furthermore, our SSFT technique explicitly teaches the model to recall the learned knowledge and answer the questions, which further improves the knowledge recall to 31.5% and even surpasses the Open-Book QA setting.
The results indicate the vanilla SFT strategy can only regularize LLMs’ response styles, while our SSFT could teach LLMs to utilize their knowledge (injected in the SCPT stage) to answer corresponding questions, as exemplified in Fig. 6.

Particularly, our method receives significant enhancements on MultiDoc QA subsets, which implies even though the total text content may exceed the models’ attention window and influence the OpenBook-QA, we can still chunk the content into several pieces and successfully inject the knowledge points into LLMs meanwhile preserving the whole knowledge structure.

Figure: Figure 1: MindMap Recall

In addition, we also use lexical ROUGE-L (Lin, ) and semantic BERTScore (Zhang et al., ) to quantify the memorization of injected knowledge structures, by comparing the mindmap in models’ responses (as Fig. 6 displays) with ground-truth answers.
The results in Fig. 1 indicate a relatively good memorization of the injected knowledge mindmap, emphasizing the efficacy of our SCPT strategy.

### 4.2 In-depth Evaluation for Multi-choice QA Application

Datasets and Tasks.
MMedBench (Qiu et al., ) is a multilingual medical multi-choice QA benchmark, with 45,048 QA pairs for adapting LLMs to medical experts and 8,518 for testing.
We collect 76M textbook corpora from MedTextBooks (Jin et al., ) and MMedC (Qiu et al., ) for medical knowledge injection and use the training/test split from MMedBench for SFT/evaluation.
We also curate another two sizes of training sets (with 30M and 132M tokens) to validate our method’s scalability.
Detailed setup is in Sec. A.2.

Evaluation Metrics.
For multi-choice QA in MMedBench (Qiu et al., ), we follow the default setting to calculate the accuracy on six language subsets, as well as the averaged scores.
Metrics are computed by lexical exact-matching on models’ responses, rather than maximum token probabilities.

Investigated Models.
For MMedBench, we extend the investigated LLMs across model scales and architectures including Llama2-7B/13B (Touvron et al., ), InternLM2-7B (Zheng et al., ), and the recent Llama3-8B (Dubey et al., ).
We also compare our knowledge-injected models with other popular domain-specified LLMs, such as MedAlpaca (Han et al., ), ChatDoctor (Yunxiang et al., ), PMC-LLaMA (Wu et al., ), and MMedLM (Qiu et al., ) models.

Implementation Details.
We first train LLMs for 3 epochs on medical textbooks with a batch size of 128. Then, we ask Llama3-70B to create structure-aware explanations for existing 45K QA samples in MMedBench’s training split and 33K extra entries by traversing extracted knowledge structures. Syntheses with overlapped options in the test set are removed.
In the SFT phase, the learning rate is set as 1e-6 to avoid overfitting on such an amount of SFT samples, as suggested by Qiu et al. ().

Main Results.
In Tab. 2, we present the overall performance across a series of LLMs on the testing split of MMedBench.
The results demonstrate the promising enhancement achieved by our StructTuning technique, which translates into consistent improvements on the advanced InternLM2-7B (Zheng et al., ) and Llama3-8B (Dubey et al., ) models, and largely outperforms the previous domain-specific LLMs like PMC-LLaMA (Wu et al., ) and MedAlpaca (Han et al., ).
Notably, our structure-aware knowledge injection approach, using merely 76M tokens curated from medical textbooks, achieves over 50% performance (4.46% v.s. 8.71%, 2.57% v.s. 4.96%) against the state-of-the-art MMedLM (Qiu et al., ) method, which is trained on a huge MMedC (Qiu et al., ) corpora of 25.5B tokens.
Our approach shows much more efficiency in transforming pre-trained LLMs into domain experts by structured-aware knowledge injection,
*e.g.*, we only use 0.3% corpus to achieve comparable or even slightly better performance on English and Spanish subsets on top of Llama3-8B.

However, our method is currently unable to achieve 100% of knowledge injection efficacy against MMedLM, which may comprise two major reasons: (1) some knowledge in the training corpus has already been learned by the foundation model (*e.g.*, a slight decrease in the Chinese subset), and (2) knowledge in the curated 76M training data cannot cover all testing scenarios (*e.g.*, a considerable gap on the Japanese subset due to the absence of Japanese corpus, see Tab. A2).
Therefore, we make a step to evaluate our approach’s scalability on various corpora sizes in Fig. 8.

**Table 2: Multiple-choice accuracy evaluation on MMedBench (Qiu et al., ). We report each model’s accuracy across six languages separately, with “Average” denoting the mean score over six languages. We also compare the data requirement (“#Token”) for medical knowledge injection.**
| Model | English | Chinese | Japanese | French | Russian | Spanish | Average | #Token |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| ChatDoctor | 43.52 | 43.26 | 25.63 | 18.81 | 62.50 | 43.44 | 39.53 | - |
| PMC-LLaMA | 47.53 | 42.44 | 24.12 | 20.74 | 62.11 | 43.29 | 40.04 | - |
| MedAlpaca | 46.74 | 44.80 | 29.64 | 21.06 | 59.38 | 45.00 | 41.11 | - |
| Llama2-7B | 43.36 | 50.29 | 25.13 | 20.90 | 66.80 | 47.10 | 42.26 | - |
| InternLM-7B | 44.07 | 64.62 | 37.19 | 24.92 | 58.20 | 44.97 | 45.67 | - |
| InternLM2-7B | 57.27 | 77.55 | 47.74 | 41.00 | 68.36 | 59.59 | 58.59 +0.00 | - |
| +MMed | 61.74 | 80.01 | 61.81 | 52.09 | 80.47 | 67.65 | 67.30 +8.71 | 25.5B |
| +Ours | 60.80 | 79.19 | 50.75 | 45.34 | 75.39 | 66.85 | 63.05 +4.46 | 76M |
| Llama3-8B | 63.86 | 78.23 | 48.24 | 50.80 | 71.48 | 64.15 | 62.79 +0.00 | - |
| +MMed | 66.06 | 79.25 | 61.81 | 55.63 | 75.39 | 68.38 | 67.75 +4.96 | 25.5B |
| +Ours | 66.77 | 77.44 | 53.27 | 51.61 | 74.61 | 68.49 | 65.36 +2.57 | 76M |

Figure: Figure 8: Comparison of knowledge injection approaches.
Refer to caption: x7.png

Approach’s Scalability.
In Tab. 2, we use roughly 0.3% of the 25.5B tokens in MMedC Qiu et al. () to evaluate knowledge injection methods.
Here, another two training corpora sizes are curated: 30M and 132M, which take around 0.1% and 0.5% of 25.5B tokens respectively.
We compare the vanilla CPT-SFT paradigm and our SCPT-SSFT strategy across those data settings
The backbone LLM is InternLM2-7B (Zheng et al., ), and the training settings follow the main experiments.
According to Fig. 8, our method consistently surpasses the vanilla paradigm by a large margin, emphasizing the efficacy and efficiency of domain knowledge injection.
In particular, we can fit two performance-ratio scaling curves from the data points in Fig. 8 as:

$$ $\displaystyle p_{v}\approx-0.04(\log r)^{2}+13.3\log r+100.0;\quad p_{s} \approx-1.11(\log r)^{2}+7.63\log r+133.0$ (3) $$

where $p_{v}$ and $p_{s}$ denote the relative performance enhancement (%) for vanilla and structure-aware knowledge injection, and $r$ refers to the corpus ratio.
The scaling law indicates we can achieve comparable performance (100%) with only 5% of the total training corpus, significantly reducing the costs for LLMs’ domain adaptation.
On the other hand, it also indicates our method may lead to 133% enhancement with a further 100% comprehensive data utilization.
However, we have not empirically verified those predictions due to time and resource constraints, which we view as a temporal limitation of our current work and leave it to future investigations.

Approach’s Generalization.
In addition to the InternLM2 and Llama3 models in previous experiments, we also investigate the generalization ability of our SCPT+SSFT paradigm on Llama2 (Touvron et al., ) model series. As shown in Tab. 3, our method leads to consistently significant improvements on 7B (+8.78%) and 13B (+6.17%) backbone models.
The results further demonstrate the generalizability and scalability of our StructTuning strategy across model architectures and sizes.

**Table 3: Structure-aware knowledge injection to Llama2 (Touvron et al., ) model series.**
| Model | English | Chinese | Japanese | French | Russian | Spanish | Average |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Llama2-7B | 43.36 | 50.29 | 25.13 | 20.90 | 66.80 | 47.10 | 42.26 |
| +Ours | 49.41 | 65.15 | 36.68 | 35.21 | 69.14 | 50.62 | 51.04 |
| Llama2-13B | 51.37 | 57.97 | 32.66 | 25.08 | 69.92 | 52.99 | 48.33 |
| +Ours | 53.02 | 68.30 | 37.78 | 41.71 | 70.70 | 55.51 | 54.50 |

Ablation Studies.
To further demonstrate the efficacy of each component of our StructTuning method, we conduct a comprehensive ablation study with the English split of the MMedBench dataset.
Specifically, we take Llama2-7B as the backbone model, select the English textbooks (Jin et al., ) (with 26M tokens) for vanilla and our structure-aware continual pre-training, and use different QA samples for supervised fine-tuning.
In Tab. 4, “SFT” refers to vanilla SFT with 10K English QA samples provided in MMedBench’s (Qiu et al., ) training split,
“SSFT” indicates structure-aware SFT on 10K training data, where the questions are the same as “SFT” while the answers are enhanced with knowledge structure explanation by Llama3-70B, as described in Sec. 3.3.
“SSFT*” includes another 8K structure-aware QA syntheses by Llama3-70B, consisting of 18K entries for training.
The training hyper-parameters follow the main experiment.

**Table 4: Ablation studies with Llama2-7B on the English subset of MMedBench.**
| Adaptation | English | Chinese | Japanese | French | Russian | Spanish | Average |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| - | SFT | 44.54 | 32.81 | 26.63 | 15.27 | 53.91 | 42.30 | 35.91 |
| CPT | SFT | 46.27 | 32.57 | 26.13 | 17.36 | 50.00 | 40.63 | 35.49 |
| SCPT | SFT | 46.50 | 32.14 | 20.10 | 18.17 | 53.91 | 39.97 | 35.13 |
| SCPT | SSFT | 49.96 | 32.63 | 22.11 | 17.52 | 51.17 | 41.28 | 35.78 |
| SCPT | SSFT* | 49.10 | 33.92 | 18.33 | 27.14 | 57.42 | 43.73 | 38.27 |
| RAG | 38.12 | 29.22 | 22.61 | 23.34 | 53.91 | 36.47 | 33.95 |  |

In Tab. 4, vanilla CPT+SFT injection paradigm does bring an improvement of 1.73% in the English split, while our SCPT technique with vanilla SFT strategy presents a slightly higher accuracy of 46.50%.
However, when combining SSFT with SCPT, our method immediately leads to a significant boost in the English split (49.96% v.s. 44.54%), demonstrating the efficacy and necessity of eliciting the learned structured knowledge to solve practical problems.
Moreover, the supplementation of 8K extra QA syntheses (“SSFT*”) surprisingly enhances the model performance on the other five subsets, resulting in a valuable gain on the average accuracy to 38.27%.
This result demonstrates the knowledge transferability across different languages (Lai et al., ; Qin et al., ).
After training with SSFT, LLMs can actively utilize the knowledge injected in one language to solve the problem in another language, evidencing our superiority against the traditional SFT technique.
In Sec. B.4, we follow Liu et al. () to randomly generate another 8K SFT pairs for further comparison, where our structure-aware SFT syntheses are verified to better enhance LLMs’ knowledge application.

In addition, we observe that the commonly used RAG (Lewis et al., ) strategy does not bring significant advantages to the MMedBench evaluation.
The main reason lies in the gap between the pre-training corpus (comprising official knowledge statements from textbooks) and evaluated QA samples (originating from practical diagnosis records).
Knowledge injection by (S)CPT and (S)SFT shows more advantages in this situation.

## 5 Conclusion

This work pioneers in incorporating structure-aware methodologies to enhance domain knowledge injection into large language models.
Through a novel SCPT-SSFT paradigm, we have set a new precedent for adapting LLMs to specialized domains, and the promising and scalable results underscore the viability and potential of our method.
We hope to inspire further research in efficient and effective domain adaptation, moving a step closer to models that can truly emulate human intelligence.

Limitation.
A noteworthy limitation of our work is the under-explored full potential of our StructTuning strategy for knowledge injection.
Although we have estimated a scaling law in Eq. 3, the data requirement to achieve 100% of the desired effectiveness remains unverified.
Besides, our method introduces negligible data preprocessing costs to extract the domain knowledge structure. However, Sec. B.2 indicates our method can still reduce the overall adaptation cost to 10% since only a small part of training corpus is required.
We will delve into the investigations in our future work.

## Appendix A Implementation Details

### A.1 Detailed Setup on LongBench

To focus on the investigation of knowledge injection, we choose 7 subsets from LongBench (Bai et al., ) across single- and multi-document QA tasks in English and Chinese, and the remaining synthetic or code-orientated tasks are eliminated:

- •
Single-Doc QA. For single-document QA, we take three subsets from LongBench: (1) Qasper (Dasigi et al., ), featured by question-answering over NLP technical papers and annotated by NLP practitioners; (2) MultiFieldQA (Bai et al., ), manually curated from multiple data sources and annotated by Ph.D. students; and (3) MultiFieldQA-zh, the Chinese split also provided by Bai et al. (), covering multiple Chinese scenarios. MultiFieldQA contains 150 Context-Question-Answer triplets to test, and the others adopted subsets include 200 pieces of test samples respectively.
- •
Multi-Doc QA. Multi-document QA requires LLMs to extract and combine information from multiple documents to derive the answer, which is generally more challenging than single-doc QA. We take four multi-hop QA datasets: (1) HotpotQA (Yang et al., ), containing 2-hop questions written by native speakers given two related paragraphs; (2) 2WikiMultihopQA (Ho et al., ), involving up to 5-hop questions synthesized through manually designed templates on Wikipedia passages; (3) MuSiQue (Trivedi et al., ), carefully composed with up to 4-hop reasoning on an increased number of supporting and distracting context evidence; and (4) Dureader (He et al., ), developed based on Baidu Search and Baidu Zhidao and filtered by Bai et al. () to reduce the data noise.
Each subset has 200 test samples.

For data entries in Single-Doc QA, we extract the knowledge structure for each single passage; in Multi-Doc QA, we identify the knowledge structure across multiple passages for each test sample. There are ultimate 1350 question-answer-passage(s)-(knowledge)structure quadruples to evaluate knowledge injection approaches on LongBench.

### A.2 Detailed Setup on MMedBench

Data for Evaluation.
The Multilingual Medical Benchmark (MMedBench) (Qiu et al., ) represents a comprehensive and diverse multilingual medical Question and Answering (QA) benchmark designed to evaluate models’ capabilities of understanding and processing medical content.

MMedBench’s robust dataset extends across 6 languages (*i.e.*, English, Chinese, Japanese, French, Russian, and Spanish) and 21 medical fields, which include, but are not limited to, Internal Medicine, Biochemistry, Pharmacology, Psychiatry, and many others. It provides 45,048 training pairs and 8,518 testing pairs for diverse learning and testing scenarios. The training split is specifically designed for domain-specific finetuning of large language models (LLMs), while the entire testing set allows for a precise assessment of multi-choice question-answering performance.
Statistics on six languages are displayed in Tab. A1.
Notably, the benchmark includes scenarios where questions may have multiple correct answers (*i.e.*, in Japanese and French subsets), introducing additional complexity for the model evaluation process.

**Table A1: Sample statistics on MMedBench.**
| Split | English | Chinese | Japanese | French | Russian | Spanish | Total |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Train | 10,178 | 27,400 | 1,590 | 2,171 | 1,052 | 2,657 | 45,048 |
| Test | 1,273 | 3,426 | 199 | 622 | 256 | 2,742 | 8,518 |

Data for Continual Pre-Training.
To investigate high-quality domain knowledge injection for LLMs, we collect 18 English textbooks and 33 Chinese textbooks from the National Medical Board Examination in the USA and Mainland China, respectively (Jin et al., ).
All collected textbooks are originally in PDF format and Jin et al. () converted them into digital text via OCR and performed some clean-up pre-processing strategies to reduce the data noise.
The English and Chinese textbooks count for around 26.1M and 21.5M tokens by Llama2 tokenizer (Touvron et al., ).
Then, we randomly sample an extra 28M textbook corpora from MMedC (Qiu et al., ) for the other languages (except the unavailable Japanese textbooks), and the statistics are displayed in Tab. A2.

**Table A2: Sample statistics on training data. “$\dagger$” and “$\ddagger$” refer to different sizes.**
| Stage | English | Chinese | Japanese | French | Russian | Spanish | Total |
| --- | --- | --- | --- | --- | --- | --- | --- |
| CPT | 26.1M | 21.5M | - | 8.1M | 10.3M | 10.1M | 76.1M |
| CPT$\dagger$ | 6.9M | 8.8M | - | 4.1M | 4.9M | 5.4M | 30.1M |
| CPT$\ddagger$ | 35.9M | 27.2M | 4.5M | 14.0M | 24.9M | 26.1M | 132.6M |
| SFT | 18.8K | 39.1K | 1.6K | 5.3K | 5.9K | 7.5K | 78.2K |

Furthermore, to evaluate the scalability of our knowledge injection strategy, we randomly sample 30.1M tokens from the 76M data as a smaller training set (denoted as “CPT$\dagger$”), and also collect extra textbooks from MMedC (Qiu et al., ) to extend a larger training split to 132.6M tokens in total (denoted as “CPT$\ddagger$”).
For the extended 123.6M corpus, the newly collected textbooks are processed as Sec. 3 states for structure-aware knowledge injection.
In particular, as MMedC does not provide Japanese textbooks, we take a part of the Wikipedia data as the knowledge supplementation for the Japanese language. Here a clustering-based technique (Sarthi et al., ) is adopted to recursively build the knowledge structure from fragmented text segments (the others are sequential chunks from textbooks), and the processed Japanese corpus is mixed with other languages for comprehensive knowledge injection.
Fig. A1 presents an example to illustrate these two kinds of knowledge structure extraction processes.

Data for Supervised Fine-Tuning.
As introduced in Sec. 3.3, we prompt Llama3-70B (Dubey et al., ) to build the structure-aware answer explanations on top of the raw SFT samples in MMedBench’s training split, and generate extra QA pairs by traversing the extracted knowledge structure from textbooks. The final quantity statistics are presented in Tab. A2.

Knowledge Structures.
We extract the domain knowledge structure for each textbook, where Fig. 3 presents an example, and combine the medical knowledge for six languages in MMedBench. As the (S)CPT corpus for Japanese is collected from Wikipedia rather than textbooks, we derive a single knowledge structure for Japanese medicine.

Figure: Figure A1: Knowledge structure extraction from (a) sequential chunks (*e.g.*, from textbooks) by our specialized 7B model and (b) separated trunks (*e.g.*, from websites) by clustering-based methods (Sarthi et al., ). Here the terms of “Section”, “Chapter”, and “book” are just examples to help illustrate the knowledge structure.
Refer to caption: x8.png

### A.3 Resource Requirement

We use 8 NVIDIA A100-80G GPUs to train all the models, and leverage 1-2 NVIDIA A100-80G GPUs for inference.

## Appendix B Additional Experiments

### B.1 Ablation on Knowledge Structure Extraction

Extracting domain knowledge structure is a prerequisite for subsequent knowledge injection (including both SCPT and SSFT) for language models. In Sec. 3.1, we propose a bottom-up strategy to re-chunk the texts from domain textbooks, summarize a title for each chunk, and send the title list to a specialized 7B model to derive the knowledge structure.
The prompt template is displayed in Appendix C.

In fact, we argue that due to the language diversity, a perfectly recovered table of contents of textbooks is unnecessary for domain knowledge injection. A reasonable knowledge structure is sufficient enough.
In Tab. A3, we individually adopt few-shot GPT-3.5-Turbo (Brown et al., ) and LLaMA3-70B (Dubey et al., ) models to extract medical knowledge structure from 18 English textbooks (Jin et al., ) (with 26.1M tokens) for subsequent knowledge injection (the backbone LLM is LLaMA2-7B (Touvron et al., )).
Although they present 3.6%-3.7% enhancement on MMedBench (Qiu et al., )’s English test set (denoted as “improvement”), leveraging GPT-3.5-Turbo and LLaMA3-70B is either expensive or time-consuming.
GPT-3.5-Turbo costs around 15 dollars to process 26M tokens, while LLaMA3-70B takes around 1.5 hours on 2 A100-80G GPUs, of which both limit the scalability of data pre-processing for structure-aware knowledge injection.

**Table A3: Comparison of models to extract knowledge structures on 26M English corpus.**
| Model | Improvement | Time Consumption | Extra Cost |
| --- | --- | --- | --- |
| GPT-3.5-Turbo | +3.58 | - | 15$ |
| LLaMA3-70B | +3.72 | 1.5h | - |
| Ours-7B | +3.69 | 0.2h | - |

Inspired by Liu et al. (), we distilled the knowledge structure extraction capability from giant LLMs to a LLaMA2-7B model via supervised fine-tuning.
In particular, we instruct LLaMA3-70B to generate 22K training examples (pairs of raw knowledge points and extracted knowledge structures) from Wikipedia, and train a LLaMA2-7B model at a batch size of 128 and a learning rate of 2e-5 for 1 epoch.
After utilizing the specialized 7B model to identify the knowledge structure in medical textbooks, as shown in Tab. A3, the results translate to comparable performance on structure-aware knowledge injection.
Meanwhile, the inference cost significantly decreases to 0.3 hours, which is more scalable to handle a larger amount of domain corpus.

### B.2 Comparison on Training Costs

In Tab. A4, we quantify the total training cost on 8 A100-80G GPUs. According to Qiu et al. (), the conventional CPT+SFT paradigm on 25.5B medicine corpus takes more than 30 days to derive the SOTA MMedLM model. In our SCPT+SSFT framework, although the pre-processing (*i.e.*, knowledge structure extraction) introduces an extra 0.6 hours to process 0.3% data (around 76M tokens), the total training process only costs 4.5 hours. As suggested in Fig. 8, when 5% training data is leveraged for knowledge injection to achieve 100% improvement, the overall cost is limited to 3 days, much less than the CPT+SFT approach with more than a month. Those analyses further demonstrate the efficacy and efficiency of our structure-aware knowledge injection framework.

**Table A4: Comparison of training costs for knowledge injection.**
| Paradigm | Corpus | Improvement | Pre-process | Total |
| --- | --- | --- | --- | --- |
| CPT+SFT | 100% | 100% | - | $>$30d |
| SCPT+SSFT | 0.3% | 50% | 0.6h | 4.5h |
| SCPT+SSFT | 5% | 100% | 9.7h | 3d |

### B.3 F1-Score Evaluation on LongBench

In Sec. 4.1, we mainly follow Zhu & Li () to investigate the memorization and understanding of injected knowledge by calculating the knowledge recall in models’ responses.
Here we report the F1-score measure over the Open-Book QA (OBQA) and Closed-Book QA (CBQA) settings for a thorough comparison.
Note that here we use the vanilla question prompt to obtain concise answers, instead of the CoT prompt used in Sec. 4.1 to elicit models’ memorized knowledge. The evaluated models are the same as Sec. 4.1.

In Tab. A5, we report the Open-Book QA (OBQA) baseline for Llama2-7B with passages as inputs, which shows the best performance on MultiFieldQA (MFQA) (Bai et al., ) and 2WikiMultihopQA (2Wiki) (Ho et al., ) subsets.
Then, we establish the Closed-Book QA (CBQA) baseline by traditional CPT+SFT to inject passage contents into model parameters, and supplement the experiment of our SCPT+SSFT technique for comparison.
According to the results shown in Tab. A5, CPT+SFT slightly improves the QA performance on several subsets (such as MultiFieldQA-zh (MFQAzh) (Bai et al., )), while the overall F1-Score measure is still inferior to the OBQA performance.
In contrast, our SCPT+SSFT approach successfully boosts the closed-book QA performance to 20.1% on average, even surpassing the open-book QA baseline of 18.7%.

The results are consistent with Sec. 4.1, which jointly demonstrate the effectiveness of structure-aware knowledge injection for large language models.

**Table A5: F1 Score evaluation of Open-Book QA (OBQA) and Closed-Book QA (CBQA) tasks on the LongBench (Bai et al., ) dataset. The best results are marked in bold, and the secondary results are marked with underlines. The backbone model is Llama2-7B (Touvron et al., ).**
| Task | Adaptation | SingleDoc-QA | MultiDoc-QA | Average |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Qasper | MFQA | MFQAzh | HpQA | 2Wiki | Musiq | Duzh |  |  |  |
| OBQA | - | 19.2 | 36.8 | 11.9 | 25.4 | 32.8 | 9.4 | 5.2 | 18.7 |
| CBQA | CPT+SFT | 16.8 | 23.1 | 13.2 | 21.3 | 19.1 | 10.4 | 13.4 | 16.8 |
| SCPT+SFT | 15.2 | 21.5 | 15.2 | 14.9 | 19.8 | 5.6 | 14.1 | 15.2 |  |
| SCPT+SSFT | 19.7 | 23.5 | 19.5 | 26.4 | 24.1 | 12.2 | 15.4 | 20.1 |  |

### B.4 Ablation on SFT Data Synthesis

In Sec. 4.2, we compared our structure-aware knowledge injection with conventional CPT+SFT paradigm on MMedBench. On its English subset, we ablated the training components of our method, and found that the newly synthesized 8K SSFT data (by traversing the extracted knowledge structure) can inspire LLMs’ cross-language capability to apply the learned structured knowledge to solve practical diagnosis problems. Here, we follow Liu et al. () to randomly generate another 8K QA pairs for SFT alignment for further comparison, denoted as “SFT*”. We randomly sample medical texts and instruct Llama3-70B (Dubey et al., ) for data synthesis, without the knowledge structure provided. Tab. A6 indicates that “SFT*” brings slight enhancement to the English test subset, but the average accuracy drops to 34.51% instead. The results further demonstrate our method’s efficacy in the application of the injected, structured domain knowledge.

**Table A6: Comparison of SFT data synthesis strategies on MMedBench. The backbone LLM is the same Llama2-7B model after SCPT on English textbooks.**
| SFT synthesis | English | Chinese | Japanese | French | Russian | Spanish | Average |
| --- | --- | --- | --- | --- | --- | --- | --- |
| - | 46.50 | 32.14 | 20.10 | 18.17 | 53.91 | 39.97 | 35.13 |
| SFT* | 47.13 | 32.49 | 16.58 | 16.72 | 51.95 | 42.16 | 34.51 |
| SSFT* | 49.10 | 33.92 | 18.33 | 27.14 | 57.42 | 43.73 | 38.27 |

## Appendix C Prompt Template for Knowledge Structure Extraction

Fig. A2 displays the prompt template to query our specialized 7B model to extract knowledge structure on given knowledge points, which introduces the task definition, detailed instruction, and output formats to illustrate the process.

Figure: Figure A2: Prompt template for knowledge structure identification.
Refer to caption: x9.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: 1. Is the extracted structure always a tree? How do you guarantee this?
2. Figure 3: To confirm, you give the model the prompt on the left of Figure 3 and the model outputs the contents on the right, right?
3. L237: Can you provide more details? How come the prompt on the left of Figure 3 now allows the model to output natural language, instead of the structured output as shown in Figure 3 right?
4. L239: It appears that, during structure-aware continual pre-training, you will train the model to generate all the texts chunks from the raw corpus; just that you will condition the generation on the mindmap. Then how is it possible that in the MMedBench experiment, your corpus is much smaller than the raw corpus? L420: How do you convert a 25B-token corpus to 74M?
5. Figure 4: Does {section} means {title}?
6. L296: Could you define “path” and “branch”?
7. Table 1: Perhaps I missed this. How are SFT data generated?
8. L342: Maybe I missed this. For LongBench experiments, do you compute exact match of the answer? What if the model output has a long chain of thought? Or do you use model-based evaluation?
9. L350: Do you compute F1 on the QA, question, or answer tokens? It seems that there could still be semantically similar examples.
10. In Table 3, does +MMed correspond to CPT or CPT+SFT?
soundness: 2
strengths: - This paper proposes a novel method that substantially improve the data efficiency and effectiveness of knowledge injection from domain-specific corpora into pretrained LLMs. The method is intuitive and has the nice feature of exploiting the structure in domain knowledge, organizing text chunks as a mindmap before letting the model learn the contents based on the mindmap.
- The paper is well-written, easy-to-follow, and detailed.
- The main experimental results are strong and the ablation demonstrates the helpfulness of both training stages.
summary: - This paper proposes to inject knowledge from a domain-specific corpus into LLMs by StructTuning, a two-stage training strategy that comprises a Structure-aware Continual Pre-Training stage and a Structure-aware Supervised Fine-Tuning stage, to improve data efficiency (L075).
- The former stage extracts domain knowledge taxonomy from the corpus and trains LLMs to memorize the taxonomy (a tree-like mindmap of content) and predict text chunks given paths in the taxonomy.
- In the latter stage, the model is trained on question-answer-explanation triplets where the explanations are based on knowledge paths from the domain knowledge taxonomy.
- The authors examine the knowledge recall of domain-adapted language models on LongBench and their abilities to apply injected knowledge on MMedBench. On MMedBench, the proposed method can achieve 50% of the improved performance of the baseline knowledge injection method using only 0.3% data.
weaknesses: The Questions below need to be addressed. Q4 about corpus size is important and needs clarification. Otherwise the paper is in a good shape overall.

After Author Rebuttal: While I appreciate the author responses and my clarification questions have been mostly addressed, the Q4 remains concerning. A main result in the paper is that the proposed LLM+Ours that uses 76M tokens can achieve 50% of the improvement over the baseline LLM compared to LLM+MMed that uses 25.5B tokens. However, "Ours" uses a different knowledge source than "MMed", as Reviewer kHUD also mentioned, and it is unclear whether the performance comes from the fact that the extra knowledge source is high quality. The scaling law predicted results is also not perfectly convincing, and the authors could have run the experiment with the 5% data to show the argued 100% performance. Otherwise, I believe the paper has its merits.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: What specific modifications were made to the loss function to accommodate the conditional modelling aspect of SCPT? How sensitive is SCPT performance to the choice of templates used to bridge mindmap structures and text chunks?  Were different template strategies compared? How do you control the length and complexity of the mindmaps to avoid overwhelming the model's context window?

The paper mentions removing synthetic QA pairs similar to the test set.  What similarity metric and threshold were used? How much data was filtered out?

How is the random walk algorithm used to generate the knowledge paths utilized in SSFT? What is the rationale for the path length parameter l? What is the impact of having too long path lengths?

Can you elaborate on the limitations of RAG observed in the MMedBench evaluation, providing concrete examples where knowledge injection proves superior?
soundness: 3
strengths: - Efficiency: The paper demonstrates impressive efficiency gains, achieving comparable performance with significantly less training data (0.3% compared to existing methods). The proposed scaling law hints at even greater potential for efficiency with larger datasets.

- Comprehensive Evaluation: The experiments cover different model architectures and scales, and utilize diverse datasets and tasks, including both recall-based and reasoning-based evaluations. The ablation study provides valuable insights into the contribution of individual components.
summary: This paper introduces StructTuning, a novel two-stage framework designed to efficiently adapt large language models (LLMs) to specific domains by mimicking the structured learning process of humans.  Inspired by how students learn from structured textbook content and apply that knowledge through exercises, StructTuning comprises Structure-aware Continual Pre-Training (SCPT) and Structure-aware Supervised Fine-Tuning (SSFT).  In the SCPT stage, the method automatically extracts a domain knowledge taxonomy from the corpus (primarily textbooks) and trains the LLM to associate text chunks with their corresponding knowledge points within this taxonomy. This process effectively links fragmented information into a cohesive knowledge structure.  The subsequent SSFT stage focuses on applying the acquired knowledge.  It utilizes synthetically generated question-answer pairs, designed to prompt the LLM to explicitly leverage the learned knowledge structure in its reasoning and explanations.  The paper evaluates StructTuning on closed-book question-answering tasks using the LongBench and MMedBench datasets.  Results show substantial improvements in knowledge recall and application compared to conventional Continual Pre-Training (CPT) followed by Supervised Fine-Tuning (SFT).  Remarkably, StructTuning achieves these improvements using only a fraction (0.3%) of the training data required by standard methods, showcasing its potential for efficient and scalable domain adaptation in LLMs.
weaknesses: Commonality in Code Training: While the paper claims originality in its structure-aware approach, this technique is already prevalent in contexts where training data consists of code or hierarchical data structures. In code training, for instance, it’s common to replace long sequences with structured, layered representations that summarize relationships across hierarchies. This may limit the novelty of StructTuning’s proposed structure-aware methodology.

Scaling Law Verification: The proposed scaling law, while intriguing, relies heavily on extrapolation.  The lack of empirical validation for corpus ratios between 5% and 100% significantly undermines this claim.  While acknowledging resource constraints, exploring at least a few intermediate data points would considerably strengthen the paper.

Clarity on Knowledge Structure Extraction:  The paper would benefit from greater clarity on the knowledge structure extraction process using the specialized 7B model. Specifically, more information on training details and quantitative performance metrics, such as precision and recall against human-annotated structures, would strengthen the analysis. Additionally, including representative examples of extracted structures across different domains would illustrate the model’s effectiveness and adaptability. An ablation study could help evaluate the impact of the extracted structures on downstream tasks like multi-hop QA. For instance, removing different levels of the extracted structure (e.g., top-level categories versus deeper hierarchies) and analyzing the effect on QA performance would clarify whether certain structural elements are critical or if the performance degrades gracefully without them.

Lack of comparison with other knowledge injection methods: The paper would benefit from a more comprehensive comparison with a broader range of knowledge injection techniques beyond the primary comparisons to vanilla CPT+SFT and MMedLM. Including comparisons with methods like knowledge distillation tailored for domain adaptation, recent approaches integrating external knowledge graphs during fine-tuning, and hybrid methods combining retrieval-augmented generation with parameter-efficient fine-tuning would strengthen the paper’s claims of superiority. This expanded evaluation would provide a clearer picture of the proposed method’s advantages in terms of efficiency and domain adaptation capabilities.

Comparison with Retrieval-Based Methods: While RAG is briefly mentioned and experimented with (sec 4.2 & 5). The paper provides very limited details about how this RAG baseline was implemented.  It doesn't specify which retrieval method was used (e.g., dense or sparse retrieval and embedding model), the size and composition of the knowledge base, and any preprocessing or indexing steps applied. Additionally, details on retrieval hyperparameters—like the number of retrieved passages and any reranking strategies—would clarify the setup. A more rigorous comparison with state-of-the-art retrieval-augmented methods is needed to properly contextualize the contribution.  This is particularly important given the focus on knowledge-intensive tasks.

Small Typo:
Table A4, line 2 & line 3's 1st column.
P19, line 999, "self-conatined".

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 2
questions: In the analysis of experimental results, like the one on line 468, are the conclusions drawn reliable? With such a small sample of results, is it possible to draw a regular curve? Can other arguments be added to strengthen the reliability of the conclusion?
soundness: 3
strengths: 1.Refer to the educational processes of human students, the authors propose a novel two-stage strategy for knowledge injection and alignment: Structure-aware Continual Pre-Training (SCPT) and Structure-aware Supervised Fine-Tuning (SSFT). This skillfully introduces structured data into continuous learning, and makes use of the characteristics of structured data to effectively improve the learning effect.

2.This paper first introduces the current challenges faced by LLM in learning domain-specific knowledge, and proposes an ingenious and refined method to effectively improve the understanding and application of LLM to domain-specific knowledge.
summary: This paper introduces a pioneering methodology, termed StructTuning, to efficiently transform foundation Large Language Models (LLMs) into domain specialists. Refer to the educational processes of human students, particularly how structured domain knowledge from textbooks is assimilated and subsequently applied to tackle real-world challenges through specific exercises, the authors propose a novel two-stage strategy for knowledge injection and alignment: Structure-aware Continual Pre-Training (SCPT) and Structure-aware Supervised Fine-Tuning (SSFT). The ultimate method has undergone extensive evaluations across model architectures and scales, using closed-book question-answering tasks on LongBench and MMedBench datasets, and demonstrates the potential of comparable improvement against the state-of-the-art MMedLM2 on MMedBench, while significantly reducing the training costs to 5%.
weaknesses: The analysis of experimental results is too arbitrary: for example, the author directly fits the regular curve of line 468 after obtaining the performance of the large language model under three experimental conditions: 0.1, 0.3, and 0.5. In the absence of sufficient experimental data, it seems impossible to draw such a universal conclusion. Such an arbitrary conclusion may cast doubt on the reliability of the conclusions of this paper.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: 1. In the open-ended question answering task on the LONGBENCH dataset, could you clarify how the training and test sets are divided? Specifically, is there a distinction between the “10,476 passages” mentioned in the paper and those referenced in the appendix?
2. In line 351, when you state “remove those with over 0.5 F1-Score similarity to test samples to prevent data leakage,” does this imply that similar questions in test set are generated? If so, does filtering out entries with an F1 score over 0.5 sufficiently prevent the performance boost from similar questions in test se?
3. In the multiple-choice task, RAG’s performance is consistently lower than fine-tuning methods. Could you provide specific experimental settings to help clarify why these results differ from those reported in existing literature?
4. The average passage length in LONGBENCH’s single- and multi-document tasks far exceeds the 4K context limit of Llama-2. How did you account for the impact of information truncation on baseline performance? Have you considered using models more suited for long-context settings as comparisons?
5. In the multiple-choice task, the MMedBench dataset uses MedTextBooks as the knowledge source instead of MMedC. Could you explain the reasoning behind this choice? Does comparing training costs across different datasets impact the fairness of the experiments?
6. On line 468, you derive the scaling law (Equation 3) based on only four data points from Figure 7. Is this approach sufficient to reach a generalizable conclusion? Have you considered introducing more data points to validate the universality of the formula?
soundness: 2
strengths: 1. Significance of the Research Problem: This paper addresses the current challenges in enabling LLMs to learn specific domain knowledge. By designing the pre-training and fine-tuning strategy, it aims to improve the LLMs’ understanding and application of domain-specific knowledge, offering a approach to meet practical need.
2. Interesting Methodology: The proposed two-stage StructTuning approach ingeniously emulates human learning processes, gradually injecting structured domain knowledge in stages.
summary: The paper introduces “StructTuning”, a methodology designed to efficiently transform foundational Large Language Models (LLMs) into domain specialists. The authors are inspired by the human educational process and propose an interesting two-stage learning strategy: Structure-aware Continual Pre-Training (SCPT) and Structure-aware Supervised Fine-Tuning (SSFT). Evaluations on LongBench and MMedBench datasets demonstrate that StructTuning achieves comparable improvements to state-of-the-art models like MMedLM2, while significantly reducing training costs.
weaknesses: 1. Unclear Experimental Setup: (1) In Open-ended Question Answering: For the LONGBENCH dataset, the division between the training and test sets is unclear. For example, as described in Section A.1, the authors generate a knowledge structure for each passage, but the sum of individual documents—reported as 200+200+150+200+200+200=1150—does not align with the 1350 entries stated on line 781 of this paper, raising concerns about the fairness and consistency of comparisons. Additionally, the reference to “10,476 passages” on line 349 is unclear in terms of how these passages differ from those in the appendix (line 781). Could the authors clarify the dataset composition, providing a detailed breakdown of how the 1350 entries and the 10,476 passages correspond to each other and to the original LongBench dataset? (2) The paper mentions “removing entries with over 0.5 F1-Score similarity to test samples to prevent data leakage.” This raises questions about whether the SSFT data generation process creates questions similar to those in the bechmark’s test set, and if so, whether filtering out sample with an F1 score over 0.5 is sufficient to prevent SSFT from merely serving as data augmentation by introducing similar samples to the test set. Could the authors provide more details on the SSFT data generation process, specifically how similarity to test samples is measured? Additionally, an analysis of the similarity distribution between generated and test samples could help justify the choice of a 0.5 F1-Score threshold. Futhermore, I am also curious about the results about CPT+SSFT, which can measure whether the performance improvement is just because SSFT introduced data augmentation. (3) In the Multiple-choice Tasks, the RAG approach consistently underperforms compared to other fine-tuning methods, which contrasts with findings in existing studies [1][2]. However, the paper does not provide specific settings for the RAG experiments, which could impact the reliability of these comparisons. Could the authors include detailed information about the RAG experimental setup, including the retrieval method, the number of retrieved documents, and how these documents were incorporated into the model’s decision-making process? This additional context would help clarify the reasons for the observed performance discrepancy.

[1] Ovadia, Oded, et al. “Fine-tuning or retrieval? comparing knowledge injection in LLMs.” arXiv preprint arXiv:2312.05934 (2023).

[2] Soudani, Heydar, Evangelos Kanoulas, and Faegheh Hasibi. “Fine Tuning vs. Retrieval Augmented Generation for Less Popular Knowledge.” arXiv preprint arXiv:2403.01432 (2024).

2. Inappropriate Dataset: (1) In the LONGBENCH Dataset: The average passage length in both single- and multi-document settings ranges from 3K to 18K, exceeding the 4K context limit of Llama-2. This makes it unsuitable for the baseline OBQA evaluation, as much of the content is truncated. While the intent might be to showcase the advantage of knowledge structuring on lengthy documents, the baseline’s context limitations lead to an unfair comparison, necessitating a more appropriate choice of long-context models for fair assessment. (2) In the Multiple-choice Task: The paper evaluates on the MMedBench dataset but uses MedTextBooks as the knowledge source rather than MMedC. The reason for not applying SCPT on MMedC is unclear. Comparing training costs across different datasets might not be valid, as smaller, high-quality datasets can yield better training outcomes, making this comparison potentially misleading.
3. Overly Conclusive Experimental Results: On line 468, the scaling law in Equation 3 is derived by fitting only four points from Figure 7. This limited dataset does not provide a sufficient basis for such a definitive conclusion, making the generalization seem premature. Could the authors consider collecting more data points to better validate the scaling law? Alternatively, if further data collection is not feasible, presenting it as a preliminary observation rather than a definitive conclusion would strengthen the reliability of this claim.