SciLitLLM: How to Adapt LLMs for Scientific Literature Understanding

## Abstract

Abstract Scientific literature understanding is crucial for extracting targeted information and garnering insights, thereby significantly advancing scientific discovery.
Despite the remarkable success of Large Language Models (LLMs), they face challenges in scientific literature understanding, primarily due to (1) a lack of scientific knowledge and (2) unfamiliarity with specialized scientific tasks.
To develop an LLM specialized in scientific literature understanding, we propose a hybrid strategy that integrates continual pre-training (CPT) and supervised fine-tuning (SFT), to simultaneously infuse scientific domain knowledge and enhance instruction-following capabilities for domain-specific tasks.
In this process, we identify two key challenges: (1) constructing high-quality CPT corpora, and (2) generating diverse SFT instructions.
We address these challenges through a meticulous pipeline, including PDF text extraction, parsing content error correction, quality filtering, and synthetic instruction creation.
Applying this strategy, we present a suite of LLMs: SciLitLLM , specialized in scientific literature understanding.
These models demonstrate promising performance on scientific literature understanding benchmarks.
Our contributions are threefold:
(1) We present an effective framework that integrates CPT and SFT to adapt LLMs to scientific literature understanding, which can also be easily adapted to other domains.
(2) We propose an LLM-based synthesis method to generate diverse and high-quality scientific instructions, resulting in a new instruction set – SciLitIns – for less-represented scientific domains.
(3) SciLitLLM achieves promising performance in scientific literature understanding benchmarks.
We release the data processing codes 1 1 1 https://github.com/dptech-corp/Uni-SMART/tree/main/SciLitLLM , model weights 2 2 2 https://huggingface.co/Uni-SMART/SciLitLLM-1.5 and the synthetic dataset 3 3 3 https://huggingface.co/datasets/Uni-SMART/SciLitIns .

## 1 Introduction

Figure: Figure 1: Average scores on SciRIFF of models with varying parameter sizes.
Refer to caption: extracted/5936497/figures/performance_plot.png

Figure: Figure 2: An example of scientific literature understanding in SciRIFF. It involves extracting accurate entities from a biomedicine paper. SciLitLLM-7B demonstrates sufficient scientific knowledge and instruction-following ability to accurately identify and extract these entities.
Refer to caption: x1.png

Scientific literature understanding involves the systematic evaluation and interpretation of scientific texts and publications, to identify trends, extract targeted information, and garner insights (AI4Science & Quantum, ; Zheng et al., ), significantly contributing to scientific discovery.
Concurrently, Large Language Models (LLMs) (OpenAI, ; Team, ; Touvron et al., ) have achieved remarkable success in natural language processing, prompting the development of domain-specific LLMs across various fields (Cui et al., ; Wu et al., ; Clusmann et al., ).
However, recent studies (Wadden et al., ; Cai et al., ; Singh et al., ) indicate that LLMs face challenges when specializing in scientific literature understanding, particularly in context understanding and question answering.
Take Figure 2 as an example, where the LLM is asked to understand the content of a biomedical research paper and then extract the targeted information.
LLMs’ potential might be hindered by two major barriers: (1) a lack of scientific knowledge, which results in errors such as the missing important entities in Llama-3.1-8B (AI@Meta, ), and (2) unfamiliarity with scientific tasks, leading to the inability of Galactica-6.7B (Taylor et al., ) to follow task instructions accurately.

To make LLMs specialized in science-relevant tasks, existing studies mostly adopt two strategies, as illustrated in Figure 4:
(1) Fine-tuning with scientific instructions (Wadden et al., ; Zhang et al., ; Singhal et al., ).
A general-purpose LLM is fine-tuned with collected domain-specific instructions to adapt it to science-relevant tasks. However, instruction fine-tuning alone is insufficient to imbue the models with comprehensive scientific knowledge.
(2) Pre-training on scientific corpora (Beltagy et al., ; Zeng et al., ; Taylor et al., ).
This approach involves training models on vast scientific corpora.
While this method equips LLMs with domain knowledge, the lack of instruction-tuning confines them to solving relevant tasks.
Moreover, it is hampered by substantial computational costs and data requirements (Ling et al., ; Yang et al., ).
To address these obstacles while balancing efficiency, we propose a hybrid strategy that incorporates continual pre-training (CPT) and supervised fine-tuning (SFT), to simultaneously infuse domain knowledge and enhance domain-specific instruction-following capabilities.

However, as illustrated in Figure 4, developing a scientific literature understanding model using this CPT and SFT pipeline presents two critical requirements:
(1) High-quality CPT Corpora.
Scientific corpora, predominantly in PDF format such as textbooks and research papers, are not directly digestible for LLM training.
Converting these documents to text with PDF parsing tools introduces formatting and syntax errors, degrading corpus quality.
Worse still, scientific documents often contain segments that contribute little information (*e.g., *references), necessitating quality control to filter them out.
See the first row in Figure 4 for a comparison of high- and low-quality CPT texts.
(2) Diverse Scientific Instructions.
Effective instruction following for scientific literature understanding requires a large, high-quality, and diverse set of task-related instructions.
However, to the best of our knowledge, there is a scarcity of well-designed instruction datasets for scientific literature understanding, and hiring human annotators to curate such a dataset from scratch is prohibitively expensive (Erdmann et al., ; Qiu et al., ).
See the second row in Figure 4 for an illustration of high- and low-quality instructions.

To address these challenges, we devise an effective pipeline to construct high-quality domain corpora for CPT and diverse scientific instructions for SFT, as illustrated in Figure 5:

In the CPT stage for domain knowledge injection, we start with an extensive in-house corpus consisting of 73k textbooks and 625k academic papers in the scientific field, all in PDF format.
Initially, we leverage PyPDF2(^4^44[https://pypdf2.readthedocs.io](https://pypdf2.readthedocs.io)), a widely used open-source PDF parsing tool, to extract raw texts from these documents.
We then employ a moderate yet powerful model, Llama3-7B-Instruct, to correct the format and spelling errors introduced by PDF parsing (*cf. *Section 3.1.1).
Subsequently, we train a small text quality classifier to score the corpus and filter out texts of low educational value(^5^55Phi models (Gunasekar et al., ; Li et al., ; Abdin et al., ) propose to determine the quality of a pre-training text by its educational value for a student whose goal is to learn basic domain concepts.) in the scientific field (*cf. *Section 3.1.2).
These two simple yet effective textual refinement and quality control measures ensure the high quality of our CPT corpus, culminating 12.7 billion tokens for CPT via the Qwen2.5 tokenizer.

In the SFT stage for domain instruction fine-tuning, to overcome the scarcity of domain-specific instructions and the high cost of human annotations, we propose a novel instruction synthesis method (*cf. *Section 3.2.1).
It enables us to generate diverse instructions to better equip the model for domain-specific tasks.
Moreover, we sequentially apply instruction duplication based on Levenshtein distance and an LLM-based filtering method to ensure the quality of synthetic instructions (*cf. *Section 3.2.2).

Figure: Figure 3: Comparison of strategies to adapt LLMs to scientific tasks.
Refer to caption: x2.png

Having established such high-quality datasets, we apply the CPT-SFT integration strategy on a general-purpose LLM – Qwen2.5 (Team, ) and obtain SciLitLLM of two scales: 7B and 14B.
Evaluations on benchmarks of scientific literature understanding demonstrate the effectiveness of our strategy.
We observe promising performance enhancements, with an average improvement of 4.0% on SciAssess (Cai et al., ) and 10.1% on SciRIFF (Wadden et al., ), compared to the leading LLMs under 10B parameters.
Notably, SciLitLLM-7B even outperforms Llama3.1 and Qwen2.5 with 70B parameters on SciRIFF.
Additionally, SciLitLLM-14B achieves leading results on both benchmarks, surpassing other open-source LLMs.
Further ablation studies demonstrate the effectiveness of each module in our pipeline.

In summary, our contributions are threefold:
(1) We devise an effective and comprehensive pipeline to adapt general LLMs to a specific domain – scientific literature understanding.
It combines continual pre-training (CPT) and supervised fine-tuning (SFT), to enhance scientific knowledge base and instruction-following capabilities for specialized domain tasks.
(2) We propose a novel domain instruction synthesis method to curate instructions for scientific literature understanding, resulting in a new dataset – SciLitIns.
(3) SciLitLLM, trained through the proposed pipeline, outperforms leading open-source LLMs on scientific literature understanding.

## 2 Related Works

### 2.1 Knowledge Injection via Continual Pre-training

Pre-training a language model is usually conducted on a large corpus of textual data to learn the statistical properties of language (Radford et al., ; Brown et al., ).
To further inject domain knowledge into a general LLM after pre-training, researchers engage in continual pre-training (CPT) on high-quality domain-specific corpora (Jin et al., ; Sun et al., ), sometimes combined with general corpora.
This process enhances the model’s fundamental understanding abilities in specific downstream domains while mitigating catastrophic forgetting of general knowledge (Wu et al., ; Ke et al., ; Mehta et al., ).
See the comprehensive study (Gupta et al., ) for different warm-up strategies for CPT.
Additionally, the CPT corpora can be augmented by transforming them into an instruction-response format (Cheng et al., ; ).
Furthermore, the scaling law (Hoffmann et al., ) of domain-specific CPT (Que et al., ) is explored to determine the optimal mix of data.
However, these studies primarily focus on training dynamics and data recipes, leaving the pre-processing for scientific data, especially raw PDF files, largely unexplored.
Exhibiting such steps is essential for generating high-quality domain corpora and effectively injecting domain knowledge, representing a significant challenge for practitioners.

### 2.2 Domain Adaptation via Supervised Fine-tuning

Supervised fine-tuning (SFT) modifies a pre-trained language model to follow specific instructions or perform designated tasks by fine-tuning it on a targeted, task-specific dataset (Raffel et al., ; Wei et al., ).
Applying SFT to a general LLM for specific domain adaptation has demonstrated effectiveness in various fields: in medicine (Clusmann et al., ), corpora of medical literature and clinical notes are used; in law (Cui et al., ), legal documents and case law are compiled; and in finance (Wu et al., ), financial reports and market data are utilized.
In the scientific domain, several studies have specialized LLMs for scientific tasks, often necessitating the construction of a substantial domain-specific dataset with SFT.
For example, SciGLM (Zhang et al., ) leverages existing LLMs to generate step-by-step reasoning for unlabelled scientific instructions.
ChemLLM (Zhang et al., ), a more specified LLM in the chemistry field, collects structured chemical data from a vast selection of online databases and transforms this structured data into a question-answering format for SFT.
SciRIFF (Wadden et al., ) converts existing literature understanding datasets into natural language input-output pairs suitable for instruction-tuning using pre-defined templates.
However, benchmark studies (Feng et al., ; Cai et al., ) indicate that SFT alone may not provide adequate scientific knowledge to excel in relevant tasks.
This suggests the need for a more comprehensive approach that combines domain knowledge infusion with instruction-following enhancements.

### 2.3 LLMs for Scientific Literature Understanding

In the scientific domain, existing strategies for developing specialized LLMs mostly fall into two categories:
(1) Supervised fine-tuning with scientific instructions.
This approach requires a large, high-quality, and diverse set of instructions to cultivate problem-solving abilities for scientific tasks.
Representative works (*e.g., *SciGLM (Zhang et al., ), ChemLLM (Zhang et al., ), and SciRIFF (Wadden et al., )) have been detailed in Section 2.2.
(2) Pre-training with scientific corpora.
This approach involves pre-training on a large corpus of scientific texts to improve performance on downstream scientific tasks.
Early attempts, such as SciBert (Beltagy et al., ) and KV-PLV (Zeng et al., ), are based on BERT (Devlin et al., ) and pre-trained
on a large corpus of scientific text for downstream scientific task enhancement.
More recently, Galactica (Taylor et al., ) is pre-trained on a vast corpus of scientific literature, including research papers, scientific articles, and other relevant scientific texts.
Despite these advances, two major limitations hinder these models from excelling in scientific literature understanding: (1) lack of scientific knowledge, and (2) inability to follow task instructions.
To address these challenges, we propose a combined pipeline of CPT and SFT to devise a specialized LLM for scientific literature understanding.
It injects domain-specific knowledge through CPT while enhancing task-specific instruction-following abilities through SFT, leading to a more capable LLM for scientific literature understanding.

## 3 Method

Figure: Figure 5: The pipeline of SciLitLLM consists of two key stages: continual pre-training (CPT) for scientific knowledge injection and supervised fine-tuning (SFT) for scientific instruction following. Specifically, the CPT stage involves several modules: PDF parsing, format & grammar correction (*cf. *Section 3.1.1), and quality filtering (*cf. *Section 3.1.2) modules. These modules ensure the model is equipped with high-quality scientific domain knowledge. The SFT stage includes LLM-based instruction generation (*cf. *Section 3.2.1) and instruction quality control (*cf. *Section 3.2.2) measures. These steps are designed to fine-tune the model’s ability to follow scientific instructions accurately and effectively.
Refer to caption: x4.png

In this section, we present the details of our proposed pipeline (*cf. *Figure 5): continual pre-training for scientific knowledge injection and supervised fine-tuning for scientific tasks enhancement.

### 3.1 CPT for Scientific Knowledge Injection

What are high-quality pre-training corpora?
Researchers (Gunasekar et al., ; Li et al., ; Abdin et al., ) suggest that language models benefit from corpora that possess the same qualities as an exemplary textbook for human learners: clarity, self-containment, instructiveness, and balance.
Recognizing the wealth of high-quality scientific textbooks and research papers over the past decades, we have curated a substantial collection of over 73,000 textbooks and 625,000 research papers within the scientific domain, ensuring all documents are copyright-compliant.

However, we still face two practical obstacles when dealing with those textbooks and research papers:
(1) Formatting and syntax errors.
Most textbooks and research paper documents are in PDF format, which is not directly digestible by LLMs.
Converting these documents using tools like PyPDF2 often introduces formatting and syntax errors, which degrade the quality of the corpus.
(2) Corpus quality control.
Despite their overall high quality, textbooks and research papers also contain segments with little useful information, such as references and garbled text introduced during the PDF parsing process.
To tackle these obstacles, we devised the following modules: format & grammar correction and CPT quality filter.

#### 3.1.1 Format & Grammar Correction

A parsed text from a PDF document often contains many formatting and syntax errors.
To address this issue, we prompt Llama3-8B-Instruct, to correct these errors introduced during the PDF parsing process.
Utilizing the vLLM (Kwon et al., ) backend, Llama3-8B-Instruct can process approximately 2.52 million tokens per Nvidia A100 GPU hour.
The process takes over 5,000 A100 GPU hours to handle all the textbooks and research papers.
Example texts – both before and after processing – along with the prompt template are provided in Appendix A to demonstrate the improvements through this correction process.

#### 3.1.2 CPT Quality Filter

During CPT, maintaining the quality of the training corpus is crucial for effective knowledge injection.
Given the extremely large scale of pre-training corpora, assessing quality through human annotation is not feasible (Erdmann et al., ; Qiu et al., ).
Consequently, leading LLMs (*e.g., *Phi (Gunasekar et al., ), Llama (Touvron et al., ), and Qwen2.5 (Team, )) employ model-based quality filters.
The typical process involves using larger LLMs to score the quality of a subset of texts, which then serve as labels for training small classifiers (*e.g., *random forest (Gunasekar et al., ) and Bert (AI@Meta, )) to annotate the entire training corpus.
Inspired by this approach, we design a resource-efficient method based on a lightweight text quality classifier.

We first annotate a random subset of 50k CPT texts using Llama3-70B-Instruct.
We adapt the quality assessment prompt from fineweb-edu-classifier (Anton et al., ), a widely-used quality classifier for web data, to evaluate the educational value (Gunasekar et al., ) of the scientific knowledge in each sampled text, assigning scores ranging from 0 (lowest quality) to 5 (highest quality).
After annotation, we perform supervised transfer learning on the fineweb-edu-classifier checkpoint.

This process results in a scientific text quality classifier tailored for scientific corpus assessment.
We then utilize this classifier to assess the quality of the entire CPT dataset and exclude the lowest-scoring 25% (See Figure 4 for concrete samples).
See Appendix B for more details about classifier training and hyperparameters.

#### 3.1.3 CPT Training Settings

**Table 1: Data statistics of CPT and SFT stages. Underlined datasets are curated by us.**
| Stage | Data source | Domain | #Doc/#Ins | # Tokens |
| --- | --- | --- | --- | --- |
| CPT | In-house Textbooks | Science | 73k | 10B |
| In-house Journals | Science | 625k | 2.7B |  |
| Redpajama (Computer, ) | General | - | 11B |  |
| SFT | SciLitIns | Science | 93k | 86M |
| SciRIFF (Wadden et al., ) | Science | 70k | 40M |  |
| Infinity-Instruct | General | 3M | 1.7B |  |

After preparing all the corpus, we perform CPT on Qwen2.5-Base (Team, ) for one epoch, encompassing 23.7 billion tokens (*cf. *Table 1), with a sequence length of 2,048 tokens.
To maintain the model’s general knowledge, we also include a similar scale of general corpus tokens from Redpajama (Computer, ).
To stabilize the learning procedure, we gradually decrease the learning rate from $1\times 10^{-5}$ to 0 with a cosine scheduler.
To address overfitting, we apply a weight decay of $0.1$ and gradients were clipped at a maximum value of $1.0$.
The CPT training took approximately 3 days on 32 Nvidia A100 GPUs for SciLitLLM-7B-Base and about 7 days for the 14B model.

### 3.2 SFT for Scientific Instruction Following

After performing CPT on an extensive scientific corpus to incorporate domain knowledge, we subsequently conduct SFT on domain-specific instructions to enhance the model’s ability to understand scientific literature.
We identify two major challenges in SFT for scientific instruction following:

- •
Existing instruction-tuning datasets in the scientific domain (Feng et al., ; Fang et al., ; Li et al., ) primarily focus on fields such as physics, chemistry, and biology.
Manually collecting instruction-tuning data for other less-represented vertical domains (*e.g., *biomedicine, and material) is both time-consuming and costly (Erdmann et al., ; Qiu et al., ).
- •
Few instruction-tuning datasets adequately reflect the scenario of scientific literature understanding, which typically involves a segment of scientific literature accompanied by a question that requires deriving an answer from the text.

To address these challenges, we draw inspiration from leading models (*e.g., *Nemotron-4 (Adler et al., ), Phi (Gunasekar et al., ), and Qwen (Yang et al., )), which leverage existing LLMs to construct synthetic instruction sets.
We propose a novel instruction synthesis method to curate instructions specifically for scientific literature understanding.

#### 3.2.1 Instruction Synthesis of Less-represented Domains

Unlike typical question-answer pairs, an instruction for a scientific literature understanding task comprises three components (Wadden et al., ): (1) a segment of scientific literature, (2) a question pertaining to the context, and (3) the corresponding answer.
Simply prompting an LLM to generate a scientific context along with an associated question-answer pair – without variations in the instructions or parameters – often yields similar or repeated contents.
Thus, we design a simple yet effective three-step pipeline to generate diverse and high-quality scientific contexts and corresponding question-answer pairs, consisting of the following:

- 1.
Probability table of domain keywords.
For a target scientific domain (*e.g., *biomedicine, and material), we collect dozens of high-impact research papers via Google Scholar(^6^66[https://scholar.google.com/](https://scholar.google.com/)) and count the word frequency appearing in these papers.
Then, we obtain a probability table of domain keywords.
- 2.
Scientific task list.
Since LLMs are expected to handle various types of scientific tasks, an instruction set with task diversity is essential.
Therefore, we compile a list of task descriptions by including representative tasks from existing scientific NLP datasets (Wadden et al., ; Cai et al., ; Feng et al., ), covering as many scenarios as possible that an LLM may encounter in real applications.
- 3.
Instruction Generation.
Given a word probability table and the task list for a specific scientific domain, we sample 20 keywords and a task description each time.
Subsequently, GPT-4o (OpenAI, ) is prompted to generate a scientific context containing the sampled keywords and a question-answer pair according to the provided task description.

Utilizing this pipeline, we obtain over 100k synthetic instructions covering less-represented scientific domains and various types of specialized tasks (details are presented in Appendix C.1).

#### 3.2.2 Instruction Quality Control

To ensure the diversity and quality of generated instructions, effective measures for quality control are essential.
Specifically, we incorporate heuristic deduplication and LLM-based filtering.

- 1.
Heuristic deduplication.
Despite the measures taken during the generation process to prevent high homogeneity in the instructions, the generated data points may still contain similar questions or identical answers.
To eliminate such redundancy, we implement a deduplication process to remove similar data points. See Appendix C.2 for details.
- 2.
LLM-based filtering.
Inspired by recent efforts (Eldan & Li, ; Chiang & Lee, ; Zhang et al., ) to measure the quality of generated content using LLMs, we leverage Llama-3-70B-Instruct to assess the quality of generated instructions for five aspects: clarity, complexity, correctness, usefulness, and adaptability.
We show the quality statistics of synthetic instructions in Figure 7.
The detailed recipe for instruction quality evaluation with concrete examples is included in Appendix C.3.

Through instruction synthesis and quality control pipeline, we obtain SciLitIns, consisting of 93,894 high-quality and diverse instructions for scientific literature understanding.

#### 3.2.3 SFT Training Settings

Our SFT training dataset consists of three parts: SciLitIns, SciRIFF (Wadden et al., ) and Infinity-Instruct(^7^77[https://huggingface.co/datasets/BAAI/Infinity-Instruct](https://huggingface.co/datasets/BAAI/Infinity-Instruct)), as shown in Table 1. Infinity-Instruct is a collection of more than twenty open-source instructions datasets, covering various general domains. SciRIFF and SciLitIns contain specialized instructions for scientific literature understanding.
For both scales, we train for one epoch on Infinity-Instruct to cultivate their general instruction-following abilities, then for five epochs on SciLitIns and SciRIFF for scientific literature understanding enhancement.
The training is conducted with a sequence length of 4,096, a maximum learning rate of $1\times 10^{-5}$, and a cosine scheduler.
The SFT training takes approximately 32 hours for the 7B and 70 hours for the 14B model on 32 A100 GPUs, resulting in SciLitLLM-7B-Instruct and SciLitLLM-14B-Instruct.

## 4 Experiments

**Table 2: Performance comparison of base models. SciLitLLM outperforms Llama3.1 and Qwen2.5 with similar scales, demonstrating improved domain-specific understanding through continual pretraining on scientific corpora.**
| Models | MMLU-Pro-Bio | MMLU-Pro-Chem | MMLU-Pro-Heal | MaScQA | Avg. |
| --- | --- | --- | --- | --- | --- |
| # Parameters $<$ 10B |  |  |  |  |  |
| Llama3.1-8B | 56.35 | 25.88 | 44.50 | 53.28 | 45.00 |
| Qwen2.5-7B | 68.36 | 48.37 | 51.64 | 56.23 | 56.15 |
| SciLitLLM-7B | 70.45 | 51.21 | 54.06 | 59.27 | 58.75 |
| # Parameters $>$ 10B |  |  |  |  |  |
| Qwen2.5-14B | 79.78 | 62.73 | 62.22 | 66.92 | 67.91 |
| SciLitLLM-14B | 80.62 | 64.44 | 64.24 | 68.06 | 69.34 |

**Table 3: Model performances on scientific literature understanding benchmarks: SciAssess and SciRIFF. SciLitLLM-7B and SciLitLLM-14B achieve leading performance compared with open-source models of similar scales. The best-performing models smaller than 10B and larger than 10B are highlighted in bold. Results for SciTulu-7B, GPT-3.5, and GPT-4o on SciRIFF are taken from its original papers, while all other results are produced by us.**
| Dataset | Domain/<br>Task | # Parameter $<$ 10B | # Parameter $>$ 10B | API |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| SciTulu-7B | Mistral-7B | Llama3.1-8B | Qwen2.5-7B | SciLitLLM-7B | Qwen2.5-14B | Llama3.1-70B | Qwen2.5-72B | SciLitLLM-14B | GPT3.5 | GPT4o |  |  |
| SciAssess | Biology | 45.3 | 52.0 | 63.4 | 60.9 | 65.3 | 65.9 | 69.4 | 67.2 | 67.0 | 55.4 | 68.9 |
| Chemistry | 19.0 | 34.0 | 46.1 | 47.9 | 55.4 | 60.0 | 62.5 | 63.7 | 63.4 | 37.3 | 68.0 |  |
| Material | 31.3 | 36.5 | 50.9 | 48.9 | 53.7 | 57.9 | 59.2 | 59.4 | 62.6 | 37.0 | 62.0 |  |
| Medicine | 19.9 | 25.1 | 30.2 | 28.3 | 32.4 | 31.5 | 37.4 | 37.8 | 39.9 | 31.8 | 45.8 |  |
| \hdashline | Mean | 28.9 | 36.9 | 47.7 | 46.5 | 51.7 | 53.8 | 57.1 | 57.0 | 58.2 | 40.4 | 61.2 |
| SciRIFF | BioASQ | 37.5 | 43.2 | 45.0 | 45.3 | 51.2 | 48.0 | 43.5 | 46.7 | 53.9 | 47.3 | 46.7 |
| BioR | 55.7 | 48.9 | 48.1 | 43.9 | 76.6 | 56.8 | 61.5 | 57.8 | 83.3 | 53.9 | 61.0 |  |
| DiscMT | 61.5 | 44.5 | 68.8 | 73.7 | 71.1 | 70.5 | 77.3 | 54.7 | 77.7 | 67.9 | 78.3 |  |
| EI | 11.6 | 17.2 | 17.2 | 23.3 | 23.5 | 27.2 | 26.0 | 26.3 | 30.8 | 19.2 | 24.7 |  |
| MC | 34.6 | 47.5 | 49.7 | 50.8 | 70.7 | 54.2 | 60.9 | 58.0 | 72.1 | 47.8 | 58.7 |  |
| MuP | 72.1 | 87.5 | 87.0 | 84.9 | 67.5 | 91.5 | 90.9 | 94.0 | 67.7 | 76.8 | 86.9 |  |
| Qasper | 54.2/38.6 | 48.8/43.2 | 46.5/42.7 | 38.1/25.7 | 50.7/54.1 | 48.4/46.8 | 49.1/40.2 | 61.9/50.5 | 54.0/55.1 | 54.7/39.8 | 67.8/50.5 |  |
| SciERC | 35.6 | 31.1 | 32.9 | 30.1 | 49.9 | 33.7 | 36.2 | 34.1 | 54.7 | 28.6 | 42.2 |  |
| SciFact | 66.0/49.2 | 68.8/53.8 | 63.8/53.7 | 77.8/60.3 | 83.5/67.3 | 83.1/65.6 | 87.3/70.3 | 86.5/69.6 | 91.0/68.6 | 69.7/53.3 | 84.3/68.7 |  |
| \hdashline | Mean | 47.0 | 48.6 | 50.5 | 50.3 | 60.6 | 56.9 | 58.5 | 58.2 | 64.4 | 50.8 | 60.9 |

In this section, we perform experiments to answer the following research questions: (Q1) How does SciLitLLM perform on scientific literature understanding tasks? (Q2) Can CPT with domain-specific corpora aid in scientific knowledge injection? (Q3) Can SFT with SciLitIns improve performance on scientific literature understanding tasks?

### 4.1 Experimental Setup

#### 4.1.1 Benchmarks

To evaluate the performance of LLMs regarding scientific knowledge base and specialized task-solving abilities, our benchmarks include:

- •
Base model benchmarks.
To assess their scientific knowledge, which serves as the foundation for scientific literature understanding, we evaluate the base models on scientific subjects – biology, chemistry, and health in MMLU-Pro (Wang et al., ) and MaScQA (Zaki et al., ) – a question answering dataset of materials science.
- •
Instruction model benchmarks.
We evaluate the instruct models on scientific literature understanding benchmarks: SciRIFF (Wadden et al., ) and SciAssess (Cai et al., ).
Brief descriptions of them are provided in Appendix D.

#### 4.1.2 Baselines

- •
Base model baselines.
We compare SciLitLLM-base against leading open-source base models: Llama-3.1 (AI@Meta, ), Qwen-2.5 (Team, ).
- •
Instruction model baselines. We benchmark leading instruction LLMs including GPT-[3.5, 4o] (Brown et al., ; OpenAI, ), Llama3.1-[8B, 70B] (AI@Meta, ) and Qwen2.5-[7B, 14B, 72B] (Yang et al., ; Team, ) and Mistral-7B (Jiang et al., ).
We also report the performance of SciTulu-7B (Wadden et al., ), which is a fine-tuned Llama2-7B (Touvron et al., ) model on SciRIFF.

### 4.2 Performance Overview (Q1)

#### 4.2.1 Base model performance

The performance comparison of base models is shown in Table 2.
SciLitLLM-base consistently outperforms other general base models across four scientific benchmarks.
Specifically, compared with LLMs of less than 10 billion parameters, SciLitLLM-7B-Base shows an average accuracy improvement of 2.60% over Qwen2.5-7B.
The results demonstrate that CPT on high-quality scientific corpora can significantly improve the LLMs’ ability to understand scientific contexts.

#### 4.2.2 Instruct model performance

As shown in Table 3, SciLitLLM-7B-Instruct achieves the highest performance in all 4 domains on SciAssess, outperforming the second-best model (*i.e., *Llama3.1-8B) by 4.0%.
Notably, on SciRIFF, it surpasses baseline models by a substantial margin of 10.1%.
Additionally, SciLitLLM-14B-Instruct shows a 4.6% and 7.5% performance improvement over Qwen2-14B-Instruct on SciAssess and SciRIFF, respectively.
It even outperforms other open-source models with five times more parameters (*i.e., *Llama3.1-70B and Qwen2.5-72B) on both benchmarks. This demonstrates the effectiveness of our proposed pipeline in adapting a general-purpose LLM to a specialized model for scientific literature understanding.

Detailed model performance on SciAssess is presented in Table 7, where SciLitLLM-7B lead in 15 out of 27 sub-tasks.
Specifically, SciLitLLM excels in tasks such as table extraction and molecule generation, likely benefiting from the comprehensive task coverage in our synthetic instruction dataset – SciLitIns.
On SciRIFF, SciLitLLM-7B and SciLitLLM-14B both rank first in 8 out of 11 evaluations(^8^88In SciRIFF, the Qasper and SciFact tasks have two different evaluation methods and thus two results. For LLM-based evaluations in MuP and Qasper, we use GPT-4o instead of GPT-3.5, as was done in the original paper.).

### 4.3 Ablation Study (Q2 & Q3)

We conducted ablation experiments on three key components in our pipeline: the CPT stage, the SFT data recipe, and the instruction quality filter, to investigate their effectiveness.
Note that all ablation experiments were performed on SciLitLLM-7B due to budget constraints.

#### 4.3.1 Scientific knowledge injection via CPT (Q2)

**Table 4: Ablation study of the CPT stage. The results shows that the CPT stage is essential to improve scientific literature understanding ability.**
| Model | SciAssess | SciRIFF |
| --- | --- | --- |
| Qwen2.5-Instruct | 46.5 | 50.3 |
| Qwen2.5-Base+SFT | 49.8 | 57.0 |
| Qwen2.5-Base+CPT+SFT | 51.7 | 60.6 |

We investigate the contribution of the CPT stage for SciLitLLM.
We compare the three variants: (1) Qwen2.5-7B-Instruct: official instruct-model checkpoint; (2) Qwen2.5-7B-base + SFT: applying our SFT stage directly to Qwen2.5-7B-base without CPT; (2) Qwen2.5-7B-base + CPT + SFT: SciLitLLM-7B-Instruct.

As shown in Table 6, applying SFT alone to the Qwen2.5-7B-Base model yields improvements on SciAssess (+3.3%) and SciRIFF (+6.7%).
Further, incorporating CPT results in an extra 1.9% improvement on SciAssess and a 3.6% gain on SciRIFF.
This shows the effectiveness of our CPT pipeline in curating high-quality CPT corpora and injecting scientific knowledge to LLMs.

#### 4.3.2 SFT with SciLitIns (Q3)

We explore how fine-tuning with our synthetic instruction set, SciLitIns, impacts performance in scientific literature understanding.
We incrementally add three datasets to the SFT training set: Infinity-Instruct, SciRIFF, and SciLitIns.
As shown in Table 6, using only the general purpose instruction set (*i.e., *Infinity-Instruct) results in the lowest performance on both SciAssess (47.1%) and SciRIFF (51.2%).
This indicates that fine-tuning LLMs on general instructions alone is insufficient for scientific literature understanding.

Adding the training set of SciRIFF to Infinity-Instruct improves performance on SciRIFF significantly (+5.5%) but decreases performance on SciAssess (-0.3%).
This drop may be due to SciRIFF’s relatively narrower focus, primarily covering biomedicine and clinical medicine, while SciAssess spans broader fields such as biology, chemistry, and material science. This highlights the need for a diverse instruction set that covers diverse scientific domains.
Finally, including SciLitIns boosts performance on both benchmarks, with SciAssess at 51.7% and SciRIFF at 60.6%.
This demonstrates that SciLitIns, which is designed to contain instructions from less-represented scientific domains, is useful for improving model performance in scientific literature understanding.

#### 4.3.3 Impact of Instruction filter

We conduct an ablation study to assess the impact of quality filter for synthetic instructions by varying whether the dataset SciLitIns  was filtered.
As discussed in Section 3.2.2, this filter removes low-quality instructions evaluated from five key aspects. Table 6 shows that applying the filter improves the performance of SciLitLLM-7B on SciAssess (+0.8%) and SciRIFF (+0.5%).
This demonstrates that SFT quality filtering process effectively excludes low-quality instructions, thereby boosting the performance of SciLitLLMon scientific literature understanding.

## 5 Limitations

Despite the promising results achieved by SciLitLLM, there are several limitations that should be acknowledged as follows:

- •
Insufficient data volume. Compared with existing pre-training datasets (AI@Meta, ; Yang et al., ; Taylor et al., ), the amount of data used for CPT is comparatively small.
Future work should consider incorporating a larger scientific corpus, potentially including scientific blogs or purely synthetic data.
- •
Lack of reasoning enhancement.
The current pipeline does not explore advanced reasoning techniques such as Chain-of-Thought (Wei et al., ) or Tree-of-Thought (Yao et al., ) in the data construction or model inference stages.
- •
Lack of preference alignment.
Due to a limited financial budget, the model lacks Reinforcement Learning from Human Feedback (RLHF) (Ouyang et al., ).
RLHF has shown significant improvements in aligning models with human preferences and ensuring more reliable outputs.
Implementing RLHF in future iterations could further enhance the model’s reliability.

Addressing these limitations in future research will be crucial developing a more robust and capable LLM specialized in scientific literature understanding.

## 6 Conclusion and Future Works

In this paper, we introduce SciLitLLM, a specialized model for scientific literature understanding.
It is initialized with a general base model – Qwen2.5 (Yang et al., ), and trained through continual pre-training (CPT) and supervised fine-tuning (SFT).
For effective scientific knowledge injection during CPT, we propose model-based format and grammar correction method, along with text quality filtering measures.
To ensure high-quality and diverse instructions during SFT, we devise instruction synthesis and quality control approaches.
Our experiments on widely-used benchmarks demonstrate the effectiveness of this pipeline in adapting a general model to the field of scientific literature understanding.
Specifically, SciLitLLM-7B achieves a 4.0% improvement on the SciAssess (Cai et al., ) and a 10.1% improvement on the SciRIFF (Wadden et al., ) compared to leading models with fewer than 10 billion parameters.
SciLitLLM-14B also surpasses leading open-source LLMs with around 70B parameters.
We note that this pipeline could be easily adapted to other specialized domains, particularly those lacking adequate high-quality corpora and instruction sets.

Our future work will focus on expanding the diversity and quality of the training data, as well as exploring more efficient methods for domain-specific knowledge injection and high-quality instruction generation.
Moreover, we plan to expand our pipeline to include the RLHF (Ouyang et al., ) stage for better human preference alignment and safety.

## Ethics Statement

In developing SciLitLLM, we prioritized ethical considerations to ensure the responsible use of our models and methodologies.
First, our research does not involve human subjects, and all data used for continual pre-training (CPT) are copyright-compliant.
We employed a rigorous quality filtering process to minimize the risk of incorporating biased or misleading content.
Nevertheless, we acknowledge that biases inherent in scientific literature, including historical underrepresentation of certain research domains, may propagate into the model’s outputs.
We also adhere to all applicable legal and ethical research guidelines, such as respecting copyright policies during dataset construction and providing comprehensive model documentation.
Our work is conducted with a commitment to research integrity, ensuring that our contributions remain beneficial to the scientific community while addressing ethical responsibilities associated with developing AI technologies.

## Reproducibility Statement

We have made extensive efforts to ensure the reproducibility of our work. Our proposed SciLitLLM models are available anonymously. The corresponding training pipeline have been described in Sections 3. The source code for all modules shown in Figure 5 is included in the supplementary material. The training datasets for both CPT and SFT stages are shown in Table 1. We will release the generated instruction sets SciLitIns after administrative procedure.

## Appendix A Format & Grammar Correction Examples

## Appendix B CPT Quality Filter

We randomly select 50k samples from our CPT data. These selected samples are then scored using the Llama3-70B-Instruct model.
The prompt utilized for this scoring process is as follows:

We train a Scientific Texts Quality Classifier on these labeled data samples. The classifier is a 109M BERT (Devlin et al., ) classifier, fine-tuned from the checkpoint of fineweb-edu-classifier (Anton et al., ). The model is trained for 20 epochs with a learning rate of 0.001 and a batch size of 1024. Ninety percent of the 50K samples are used as the training set, and the rest 10% are used as the validation set. The training process costs approximately 50 minutes on 4 A100 GPUs. We select the checkpoint from the epoch that yields the highest validation micro F1 score as our final checkpoint.

During Inference, we set batch size to 2048, and beam number to 1. The inference process costs 90 minutes on 4 A100 GPUs. We utilize the generated to filter out 25% data with the lowest quality. The distribution of the scores is demonstrated in Figure 6. The filtered-out 25% data are marked gray, while the remaining 75% CPT data are marked orange.

Figure: Figure 6: Score distribution of the CPT Data
Refer to caption: x5.png

## Appendix C SFT Details

### C.1 Instruction Generation Pipeline

In SciLitIns, we focus on generating instructions for three less-represented domains (materials science, medicine, and drug discovery) and five question types:

- •
Table Extraction: Table Extraction tasks evaluate a model’s proficiency in extracting, summarizing, and structuring data from an article into a table format.
- •
Entity Extraction: Entity Extraction tasks are designed to evaluate a model’s ability to extract specific information, such as entities or relationships, from the text.
- •
Molecule Translation: Molecule Translation tasks evaluate a model’s ability to translate molecules between different SMILES formats.
- •
Molecule Extraction: Molecule Extraction tasks ask a model to extract an appropriate molecule from a scientific paragraph that contains multiple molecules.
- •
Multiple Choice and True-or-False: Multiple Choice and True-or-False questions assess a model’s ability to select the correct answer from a set of options, testing its knowledge and reasoning on both simple and complex scenarios.

For each of the three scientific domains, we collect a set of high-impact research papers and construct a word frequency table. To generate a question in a given domain, we sample 20 keywords from the corresponding word table and insert them into the prompt for that question. To ensure fair representations of less frequent keywords, we use random sampling with a temperature setting of 3. We will release our code, prompt templates, and word frequency tables. Below is an example of generating a table extraction question:

### C.2 Instruction Deduplication

The generated synthetic data may contain similar questions or identical answers. To eliminate redundancy, we implement a fuzzy deduplication process using the Levenshtein distance to calculate the similarity score between question-answer pairs. Specifically, for two pairs $(q_{1},a_{1})$ and $(q_{2},a_{2})$, their textual similarity is defined as $(1-\text{lev}(q_{1},q_{2}))(1-\text{lev}(a_{1},a_{2}))$, where $\text{lev}(\cdot,\cdot)$ denotes the Levenshtein distance. Due to significant differences between texts from different question types, we compute similarity matrices separately for each type. We then use a disjoint-set data structure to merge highly similar data points. We use this process to remove approximately 5% to 10% of duplicated data for each question type.

### C.3 Quality Assessment of Generated SFT Instructions

In section 3.2.2, we sample 10k instruction pairs from SciLitIns and evaluate them by Llama-3-70B using the below prompt. Specifically, the quality is evaluated from five aspects: clarity, complexity, correctness, usefulness, and adaptability, assigning each instruction a score from 0 (lowest quality) to 5 (highest quality).

Below is an example of SciLitIns, which will be sent to Llama-3-70B for evaluation.

Below is the evaluation by Llama-3-70B on the example from  SciLitIns. Each criterion is scored out of 5 points, making the total possible points 25. The total score is then averaged, resulting in an overall score out of 5.

We show the quality statistics of synthetic instructions in Figure 7.
Instructions with an average score of less than 4 are filtered out.

Figure: Figure 7: The quality of SciLitIns is evaluated from five aspects: clarity, complexity, correctness, usefulness, and adaptability (the higher the better). Instructions with an average score of less than 4 are filtered out.
Refer to caption: x6.png

## Appendix D Benchmark Details

To the best of our knowledge, there are two commonly-adopted datasets for scientific literature understanding:

- •
SciRIFF (Wadden et al., ) evaluates essential scientific literature understanding capabilities, including information extraction, summarization, question answering, claim verification, and classification.
Data points in SciRIFF are notable for their long input contexts and complicated structured outputs. The Qasper and SciFact tasks have two different evaluation methods and thus two results. For MuP and one evaluation method of Qasper, we use GPT-4o as language model evaluator, while the original paper uses GPT-3.5.
We note that SciRIFF contains a separate training set used in the SFT stage in our study.
- •
SciAssess (Cai et al., ) features an end-to-end benchmark of understanding PDF content.
It includes 28 tasks from four scientific domains: biology, chemistry, materials and medicine.
SciAssess is used exclusively for testing in our evaluation.

Overall, SciRIFF provides basic benchmarks for comprehending short scientific segments and various instructions, while SciAssess presents more challenging tasks involving longer contexts from raw PDFs.

## Appendix E Detailed Performance on SciAssess

The detailed results on each tasks in SciAssess are shown in Table 7.

**Table 7: Detailed model performance on SciAssess tasks.**
| Domain | Task | SciTulu-7B | Mistral-7B | Llama3.1-8B | Qwen2.5-7B | SciLitLLM-7B | Qwen2.5-14B | Llama3-70B | Qwen2.5-72B | SciLitLLM-14B | GPT3.5 | GPT4o |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Biology | Average | 45. 3 | 52.0 | 63.4 | 60.9 | 65.3 | 65.9 | 69.4 | 67.2 | 67.0 | 55.4 | 68.9 |
| MmluPro | 41.4 | 34.4 | 68.9 | 70.2 | 72.2 | 77.7 | 81.5 | 84.0 | 79.6 | 65.0 | 87.4 |  |
| ChartQA | 35.2 | 40.7 | 46.7 | 47.7 | 48.2 | 51.8 | 47.7 | 48.7 | 56.3 | 31.2 | 55.8 |  |
| ChemRecog | 54.9 | 56.0 | 73.7 | 66.7 | 84.3 | 80.1 | 83.6 | 76.4 | 77.8 | 64.9 | 79.5 |  |
| CompRecog | 41.6 | 72.4 | 69.3 | 66.8 | 72.2 | 74.5 | 76.8 | 71.2 | 71.1 | 63.6 | 73.3 |  |
| DiseRecog | 58.0 | 67.6 | 76.2 | 57.4 | 76.1 | 64.7 | 79.3 | 79.3 | 74.1 | 68.8 | 76.3 |  |
| GeneFunc | 40.6 | 40.7 | 45.8 | 56.6 | 38.9 | 46.4 | 47.4 | 43.8 | 43.2 | 39.1 | 41.0 |  |
| Chemistry | Average | 19.0 | 34.0 | 46.1 | 47.9 | 55.4 | 60.0 | 62.5 | 63.7 | 63.4 | 37.3 | 68.0 |
| MmluPro | 13.8 | 17.9 | 40.8 | 46.2 | 52.6 | 63.7 | 67.6 | 72.3 | 63.2 | 30.3 | 74.5 |  |
| ElecQA | 19.6 | 41.7 | 53.8 | 63.3 | 70.4 | 75.9 | 75.5 | 78.5 | 78.4 | 30.5 | 85.5 |  |
| OledExtr | 0.9 | 22.7 | 13.7 | 36.3 | 30.6 | 42.0 | 56.3 | 49.9 | 37.4 | 28.0 | 43.8 |  |
| ChartQA | 33.3 | 46.7 | 86.7 | 46.7 | 66.7 | 80.0 | 80.0 | 80.0 | 80.0 | 66.7 | 86.7 |  |
| PolyQA | 25.0 | 56.7 | 59.1 | 76.9 | 88.0 | 90.9 | 85.2 | 91.4 | 91.3 | 33.0 | 93.8 |  |
| PolyExtr | 7.4 | 35.8 | 43.1 | 42.6 | 66.4 | 52.9 | 69.0 | 69.2 | 78.1 | 56.2 | 75.9 |  |
| SolExtr | 31.5 | 33.7 | 33.2 | 40.5 | 36.1 | 41.1 | 44.7 | 43.7 | 41.3 | 40.8 | 44.4 |  |
| ReactQA | 21.0 | 23.1 | 30.3 | 32.8 | 33.3 | 39.0 | 38.5 | 37.9 | 51.3 | 27.2 | 48.7 |  |
| MechQA | 18.2 | 27.3 | 54.5 | 45.5 | 54.5 | 54.5 | 45.5 | 50.0 | 50.0 | 22.7 | 59.1 |  |
| Material | Average | 31.3 | 36.4 | 50.9 | 48.9 | 53.7 | 57.9 | 59.2 | 59.4 | 62.6 | 37.0 | 62.0 |
| MatQA | 36.9 | 43.0 | 59.7 | 56.7 | 58.6 | 66.5 | 73.8 | 71.9 | 70.3 | 52.1 | 76.8 |  |
| ChartQA | 33.3 | 66.7 | 66.7 | 73.3 | 46.7 | 60.0 | 46.7 | 53.3 | 53.3 | 40.0 | 46.7 |  |
| CompExtr | 8.0 | 8.9 | 9.9 | 13.9 | 40.6 | 37.1 | 45.7 | 43.0 | 42.9 | 18.9 | 46.2 |  |
| TempQA | 24.2 | 32.9 | 68.1 | 43.0 | 72.0 | 69.6 | 65.2 | 64.7 | 79.2 | 29.5 | 80.7 |  |
| SampDiff | 29.1 | 20.7 | 36.3 | 39.7 | 44.3 | 52.7 | 62.4 | 57.8 | 60.8 | 32.9 | 62.4 |  |
| TreatSeq | 56.4 | 46.5 | 64.9 | 66.8 | 59.9 | 61.4 | 61.4 | 65.8 | 68.8 | 48.5 | 59.4 |  |
| Medicine | Average | 19.9 | 25.1 | 30.2 | 28.3 | 32.4 | 31.5 | 37.4 | 37.8 | 39.9 | 31.8 | 45.8 |
| MmluPro | 21.3 | 30.3 | 58.4 | 51.7 | 57.2 | 62.0 | 71.0 | 68.5 | 65.8 | 53.1 | 76.3 |  |
| AffiExtr | 2.7 | 3.7 | 3.7 | 2.6 | 5.6 | 4.0 | 4.7 | 7.1 | 3.9 | 5.5 | 10.1 |  |
| ChartQA | 33.3 | 40.0 | 26.7 | 40.0 | 33.3 | 33.3 | 40.0 | 33.3 | 53.3 | 33.3 | 46.7 |  |
| TagMol | 1.2 | 0.2 | 7.1 | 0.7 | 2.8 | 6.2 | 14.3 | 13.6 | 7.3 | 2.3 | 22.9 |  |
| MarkMol | 6.9 | 20.2 | 35.2 | 20.8 | 37.5 | 23.6 | 42.5 | 44.3 | 47.1 | 52.3 | 58.5 |  |
| MolDoc | 54.0 | 56.0 | 50.0 | 54.0 | 58.0 | 60.0 | 52.0 | 60.0 | 62.0 | 44.0 | 60.0 |  |

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Are you planning to release the textbook datasets? 
Can you elaborate on the pipeline on what type of textbooks impact and improve task performance in SciRiff? 
Can you provide contamination studies in textbook datasets and the test/eval cases?
soundness: 2
strengths: - The paper introduces a strong pipeline in collecting textbook data and SFT data. This is aligned with most recent LM papers, showcasing the importance of data in the success of LM training. 
- Improved result in science instruction following
summary: This paper introduces a method to improve scientific instruction following through post-training Qwen models. The main contribution is to collect science textbook data and improved SFT data mix. Their evaluation on a recent SciRiff dataset shows improvement.
weaknesses: The authors don't seem to be planning to release their textbook dataset. This raises the question of data contamination in evaluating the proposed model.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: 1.  Are there special reasons why we should group the results based on
    the 10B model size? (table 2/3). I think it's more reasonable to
    organize based on pre-trained only/with instruction tuning/with
    domain specific tuning?

2.  Also in this paper there is only fine-tuning results on the Qwen
    model family but not others. It would be interesting to compare the
    fine-tuning effects on llama or other model families.

    1.  the author compared the performance with the SciTulu model,
        which is trained based on Llama-2 families. I don't think it's a
        fair comparison in table 3.
soundness: 2
strengths: 1.  Overall the paper is nicely written and the pipeline is nicely
    presented.

2.  The curated dataset and models could be helpful.
summary: This paper proposes a synthetic data generation pipeline for training
LMs for scientific literature understanding tasks. The main
contributions include:

1.  a pipeline for curating continued pre-training corpus based on
    textbooks and research papers (including document parsing,
    formatting, and filtering),

2.  another pipeline for creating instruction fine-tuning data for
    scientific literature understanding tasks via prompting GPT-4o.

3.  the authors show that finetuning Qwen-2.5 model (7B and 14B) on the
    CPT and SFT data can improve the performance on scientific
    literature understanding tasks.
weaknesses: 1.  The primary contribution of the paper seems to be focused on the
    dataset construction; overall the method resembles similar works for
    synthetic data generation and there are some limitations:

    1.  The PDF processing pipeline can be improved. Scientific PDF are
        known to contain complex layout and structures, and previous
        work have identified that using simple PDF parsers can lead to
        suboptimal training results (S2ORC, Lo and Wang, and ). However,
        the author primarily uses a simple PyPDF parser (\"Converting
        these documents using tools like PyPDF2 often introduces
        formatting and syntax errors, which degrade the quality of the
        corpus (line 246)\"). I'd suggest the authors investigate the
        text quality issues and check other libraries like papermage, Lo
        et al.

2.  The results are not very strong. I'd imagine a domain-specifically
    distilled model can have a substance gain in performance compared to
    GPT-4o, especially the instruction fine-tuning dataset is generated
    via GPT-4o (line 331); however, as shown in table 3, the trained
    models (SciLitLLM-7B and SciLitLLM-14B) are on par with GPT-4o. Also
    the experimental design and presentation could be improved (see my
    suggestions in questions).

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: What is the difference between the new CPT dataset and the Dolma Dataset (https://allenai.github.io/dolma/)?
soundness: 3
strengths: 1. The paper proposes a new framework that includes continual pretraining and supervised fine-tuning. The proposed framework and dataset can be very useful for other LLMs specialized in scientific understanding. The released dataset seems to be very comprehensive compared to the existing dataset.
2. The paper shows that with the new framework, the paper can further improve the performance of general LLM. The framework is especially useful in small models. Additionally, the paper includes an ablation study for CPT, SFT, and instruction quality filters. 
3. The paper provides the code and its model. In the appendix, the paper shows the improvement of format & grammar correction, CPT quality filter, SFT details, benchmark details, and detailed performance on SciAssess.
summary: The paper proposes a new strategy to improve LLM for scientific literature understanding, which includes continual pretraining and supervised fine-tuning. The paper uses Llama3-8B to correct parsing errors and filters the dataset with Llama3-70B. The paper continues pretraining the Qwen2.5.The paper designs a three-step pipeline to generate diverse scientific contexts and corresponding QA pairs. The paper then incorporates heuristic deduplication and LLM-based filtering for the instructions. The proposed framework seems to improve the performance compared to SciRIFF. The paper also includes an ablation study.
weaknesses: 1. The creation of the CPT and SFT datasets seems to rely on LLMs. The paper can randomly sample a small subset of the created dataset to check its quality with humans to show the effectiveness of the proposed framework.
2. The experiment and ablation study is not comprehensive. The paper fails to show that the model is fine-tuned to other existing scientific understanding datasets, such as the Dolma dataset (https://allenai.github.io/dolma/). The paper did include SciTulu-7B dataset; however, scitulu is based on LLama2-tb. The paper should also show the model's performance only finetuned with SciLitIns. The analysis in the experiment is also rather simple. The paper needs to provide some explanation instead of just repeating the results in the table.
3. Some details are not very clear. What is the score used in Tables 2 and 3? Many additional evaluation results and analysis are put in the Appendix. Authors should move some of them to the main paper.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: 1. How sensitive is the model performance to the quality filtering threshold (currently set at 25%)? Was this choice empirically validated?

2. The instruction synthesis method uses GPT-4 - have you explored using smaller models or your own models in a bootstrapping approach?
soundness: 3
strengths: ### Well-Motivated Approach
- The hybrid CPT+SFT strategy effectively addresses both domain knowledge and task-specific capabilities.
- The pipeline is well-designed with clear motivation for each component.
- The approach is generalizable to other specialized domains

### Technical Contributions
>Pipeline includes innovative components like LLM-based format correction (Section 3.1.1) and quality filtering (Section 3.1.2)
The instruction synthesis method (Section 3.2.1) is clever and tackles the challenge of limited scientific instruction data.

### Solid empirical results
- SciLitLLM-7B outperforms similar-sized models by significant margins \~4% on SciAssess, +\~10% on SciRIFF.
- SciLitLLM-14B surpasses larger proprietary instruction tuned models (70B+ parameters).
summary: The paper introduces SciLitLLM, a specialized language model for scientific literature understanding, built using a hybrid approach combining continual pre-training (CPT) and supervised fine-tuning (SFT). The key contributions include 1) A pipeline that combines CPT with high-quality scientific corpora and SFT with diverse scientific instructions, 2) Novel methods for improving scientific text quality and generating domain-specific instructions, and 3)  Empirical results showing improved performance on scientific understanding benchmarks SciRIFF and SciAssess
weaknesses: ### Weaknesses
The CPT corpus (12.7B tokens) is relatively small compared to standard pre-training datasets The paper acknowledges this limitation but could discuss potential impacts more thoroughly. For example, how this affect the representation from different scientific subject domains.

Otherwise I don't see clear weakness of paper of such kind. The paper appears comprehensive and well-executed for the research scope.

Meta-review: This paper presents SciLitLLM, a language model tailored for scientific literature understanding, utilizing a hybrid approach that combines continual pre-training (CPT) with supervised fine-tuning (SFT). The proposed pipeline emphasizes constructing high-quality CPT datasets and generating diverse domain-specific instructions for SFT. The paper demonstrates competitive results on benchmarks like SciAssess and SciRIFF and introduces datasets and pipelines with potential applicability beyond scientific domains.

*Strengths*:  
-Motivation and design: The paper provides a well-motivated approach addressing both domain knowledge enhancement and task-specific instruction alignment through its hybrid CPT+SFT strategy. Reviewers aknowledged the pipeline's clarity and the methodological rigor (fcV1, TiCd, and W9HG)  
-Contributions: Reviewers appreciated the overall pipeline, including LLM-based data processing, format correction, and quality filtering mechanisms. The instruction synthesis method effectively addresses the challenge of limited scientific data. In addition the released models and data are valuable to the community.  
-Empirical results: SciLitLLM achieves strong results compared with baselines and reviewers (fcV1 and W9HG) also noted its generalizability to other domains.   

*Weaknesses*:

-Dataset processing: Reviewer 7iRT raised concerns about the use of PyPDF2 for document parsing. The authors acknowledged this limitation and clarified their choice was driven by computational constraints.   
-Computational efficiency: Reviewer 7iRT questioned the computational cost-benefit ratio of achieving GPT-4o comparable performance
The authors provided context about the practical value of their approach for organizations needing private, domain-specific models.    
-Comparative Analysis: Reviewers suggested additional comparisons with other model families and datasets. The authors conducted contamination studies and quality filtering threshold experiments during the rebuttal period to address these concerns.  
-Closed nature of the textbook data: A major weakness is the closeness of the textbook data used in CPT. The authors do not have any plans to release such data, they might have legitimate copyright reasons though and they plan to release the list of the books.   
-Some reviewers suggested comparisons (e.g., with Llama3). Such additional comparisons were acknowledged as valuable future work but were constrained by computational resources.

Decision: Accept (Poster)