SciLitLLM: How to Adapt LLMs for Scientific Literature Understanding

Authors: Sihang Li, Jin Huang, Jiaxi Zhuang, Yaorui Shi, Xiaochen Cai, Mingjun Xu, Xiang Wang, Linfeng Zhang, Guolin Ke, Hengxing Cai
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
Keywords: Large Language Model, Pre-training, Supervised Fine-tuning, Scientific Literature Understanding

Abstract:
Scientific literature understanding is crucial for extracting targeted information and garnering insights, thereby significantly advancing scientific discovery.
Despite the remarkable success of Large Language Models (LLMs), they face challenges in scientific literature understanding, primarily due to (1) a lack of scientific knowledge and (2) unfamiliarity with specialized scientific tasks.
To develop an LLM specialized in scientific literature understanding, we propose a hybrid strategy that integrates continual pre-training (CPT) and supervised fine-tuning (SFT), to simultaneously infuse scientific domain knowledge and enhance instruction-following capabilities for domain-specific tasks.
In this process, we identify two key challenges: (1) constructing high-quality CPT corpora, and (2) generating diverse SFT instructions. 
We address these challenges through a meticulous pipeline, including PDF text extraction, parsing content error correction, quality filtering, and synthetic instruction creation.
Applying this strategy, we present a suite of LLMs: SciLitLLM, specialized in scientific literature understanding.
These models demonstrate promising performance on scientific literature understanding benchmarks.
(1) We present an effective framework that integrates CPT and SFT to adapt LLMs to scientific literature understanding, which can also be easily adapted to other domains.
(2) We propose an LLM-based synthesis method to generate diverse and high-quality scientific instructions, resulting in a new instruction set -- SciLitIns -- for less-represented scientific domains. 
(3) SciLitLLM achieves promising performance in scientific literature understanding benchmarks.

Decision: accept (poster)
Reviewer scores: 6, 5, 6, 8

Reviews and discussion:
<thread>
creation date:
1730656976247

thread_type:
official_review

<thread_object>
creator: reviewer_fcv1

<content>
summary:
The paper introduces SciLitLLM, a specialized language model for scientific literature understanding, built using a hybrid approach combining continual pre-training (CPT) and supervised fine-tuning (SFT). The key contributions include 1) A pipeline that combines CPT with high-quality scientific corpora and SFT with diverse scientific instructions, 2) Novel methods for improving scientific text quality and generating domain-specific instructions, and 3)  Empirical results showing improved performance on scientific understanding benchmarks SciRIFF and SciAssess

strengths:
### Well-Motivated Approach
- The hybrid CPT+SFT strategy effectively addresses both domain knowledge and task-specific capabilities.
- The pipeline is well-designed with clear motivation for each component.
- The approach is generalizable to other specialized domains

### Technical Contributions
>Pipeline includes innovative components like LLM-based format correction (Section 3.1.1) and quality filtering (Section 3.1.2)
The instruction synthesis method (Section 3.2.1) is clever and tackles the challenge of limited scientific instruction data.

### Solid empirical results
- SciLitLLM-7B outperforms similar-sized models by significant margins \~4% on SciAssess, +\~10% on SciRIFF.
- SciLitLLM-14B surpasses larger proprietary instruction tuned models (70B+ parameters).

weaknesses:
### Weaknesses
The CPT corpus (12.7B tokens) is relatively small compared to standard pre-training datasets The paper acknowledges this limitation but could discuss potential impacts more thoroughly. For example, how this affect the representation from different scientific subject domains.

Otherwise I don't see clear weakness of paper of such kind. The paper appears comprehensive and well-executed for the research scope.

questions:
1. How sensitive is the model performance to the quality filtering threshold (currently set at 25%)? Was this choice empirically validated?

2. The instruction synthesis method uses GPT-4 - have you explored using smaller models or your own models in a bootstrapping approach?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730667761305

thread_type:
official_review

<thread_object>
creator: reviewer_w9hg

<content>
summary:
The paper proposes a new strategy to improve LLM for scientific literature understanding, which includes continual pretraining and supervised fine-tuning. The paper uses Llama3-8B to correct parsing errors and filters the dataset with Llama3-70B. The paper continues pretraining the Qwen2.5.The paper designs a three-step pipeline to generate diverse scientific contexts and corresponding QA pairs. The paper then incorporates heuristic deduplication and LLM-based filtering for the instructions. The proposed framework seems to improve the performance compared to SciRIFF. The paper also includes an ablation study.

strengths:
1. The paper proposes a new framework that includes continual pretraining and supervised fine-tuning. The proposed framework and dataset can be very useful for other LLMs specialized in scientific understanding. The released dataset seems to be very comprehensive compared to the existing dataset.
2. The paper shows that with the new framework, the paper can further improve the performance of general LLM. The framework is especially useful in small models. Additionally, the paper includes an ablation study for CPT, SFT, and instruction quality filters. 
3. The paper provides the code and its model. In the appendix, the paper shows the improvement of format & grammar correction, CPT quality filter, SFT details, benchmark details, and detailed performance on SciAssess.

weaknesses:
1. The creation of the CPT and SFT datasets seems to rely on LLMs. The paper can randomly sample a small subset of the created dataset to check its quality with humans to show the effectiveness of the proposed framework.
2. The experiment and ablation study is not comprehensive. The paper fails to show that the model is fine-tuned to other existing scientific understanding datasets, such as the Dolma dataset (https://allenai.github.io/dolma/). The paper did include SciTulu-7B dataset; however, scitulu is based on LLama2-tb. The paper should also show the model's performance only finetuned with SciLitIns. The analysis in the experiment is also rather simple. The paper needs to provide some explanation instead of just repeating the results in the table.
3. Some details are not very clear. What is the score used in Tables 2 and 3? Many additional evaluation results and analysis are put in the Appendix. Authors should move some of them to the main paper.

questions:
What is the difference between the new CPT dataset and the Dolma Dataset (https://allenai.github.io/dolma/)?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730670050434

thread_type:
official_review

<thread_object>
creator: reviewer_7irt

<content>
summary:
This paper proposes a synthetic data generation pipeline for training
LMs for scientific literature understanding tasks. The main
contributions include:

1.  a pipeline for curating continued pre-training corpus based on
    textbooks and research papers (including document parsing,
    formatting, and filtering),

2.  another pipeline for creating instruction fine-tuning data for
    scientific literature understanding tasks via prompting GPT-4o.

3.  the authors show that finetuning Qwen-2.5 model (7B and 14B) on the
    CPT and SFT data can improve the performance on scientific
    literature understanding tasks.

strengths:
1.  Overall the paper is nicely written and the pipeline is nicely
    presented.

2.  The curated dataset and models could be helpful.

weaknesses:
1.  The primary contribution of the paper seems to be focused on the
    dataset construction; overall the method resembles similar works for
    synthetic data generation and there are some limitations:

    1.  The PDF processing pipeline can be improved. Scientific PDF are
        known to contain complex layout and structures, and previous
        work have identified that using simple PDF parsers can lead to
        suboptimal training results (S2ORC, Lo and Wang, and ). However,
        the author primarily uses a simple PyPDF parser (\"Converting
        these documents using tools like PyPDF2 often introduces
        formatting and syntax errors, which degrade the quality of the
        corpus (line 246)\"). I'd suggest the authors investigate the
        text quality issues and check other libraries like papermage, Lo
        et al.

2.  The results are not very strong. I'd imagine a domain-specifically
    distilled model can have a substance gain in performance compared to
    GPT-4o, especially the instruction fine-tuning dataset is generated
    via GPT-4o (line 331); however, as shown in table 3, the trained
    models (SciLitLLM-7B and SciLitLLM-14B) are on par with GPT-4o. Also
    the experimental design and presentation could be improved (see my
    suggestions in questions).

questions:
1.  Are there special reasons why we should group the results based on
    the 10B model size? (table 2/3). I think it's more reasonable to
    organize based on pre-trained only/with instruction tuning/with
    domain specific tuning?

2.  Also in this paper there is only fine-tuning results on the Qwen
    model family but not others. It would be interesting to compare the
    fine-tuning effects on llama or other model families.

    1.  the author compared the performance with the SciTulu model,
        which is trained based on Llama-2 families. I don't think it's a
        fair comparison in table 3.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730705026477

thread_type:
official_review

<thread_object>
creator: reviewer_ticd

<content>
summary:
This paper introduces a method to improve scientific instruction following through post-training Qwen models. The main contribution is to collect science textbook data and improved SFT data mix. Their evaluation on a recent SciRiff dataset shows improvement.

strengths:
- The paper introduces a strong pipeline in collecting textbook data and SFT data. This is aligned with most recent LM papers, showcasing the importance of data in the success of LM training. 
- Improved result in science instruction following

weaknesses:
The authors don't seem to be planning to release their textbook dataset. This raises the question of data contamination in evaluating the proposed model.

questions:
Are you planning to release the textbook datasets? 
Can you elaborate on the pipeline on what type of textbooks impact and improve task performance in SciRiff? 
Can you provide contamination studies in textbook datasets and the test/eval cases?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732152345366

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Common Response

comment:
We sincerely thank all reviewers for their thoughtful and constructive feedback. We are encouraged by the positive comments on the motivation of our work (Reviewer fcV1), clarity of writing (Reviewer 7iRT), pipeline design (Reviewers TiCd, W9HG, and fcV1), model performance (Reviewers TiCd, W9HG, and fcV1), and dataset contributions (Reviewers 7iRT and W9HG). These insights have greatly helped us refine the paper, and we appreciate the opportunity to address the reviewers’ concerns. Below, we summarize the major updates made in response to the reviews:

- **CPT Dataset Release**: We have released the list of 73,000 textbooks used in constructing the CPT dataset. Unfortunately, due to copyright constraints, we cannot directly release the electronic textbook files. However, we woule like to highlight the value of our open-sourced data processing pipeline, which enables researchers in academia and industry to train private domain-specific models. This pipeline may be particularly helpful for those with access to large, domain-specific corpora but limited expertise in domain adaptation. We believe this resource extends beyond scientific literature and has broader applications for private domain modeling.

- **Contamination Studies**: To evaluate potential contamination effects, we tested SciLitLLM on SciAssess using both the full evaluation set and a clean evaluation set. Results indicate minimal differences in performance between the two sets. 

- **Quality Filtering Threshold Experiment**: We performed a sensitivity analysis to examine how different quality filtering thresholds (the percentage of data filtered out from the pre-training corpus) impact model performance. The results showed that the current threshold of 25% yields slightly better performance compared to other tested thresholds. 

We have made extensive efforts to address the reviewers’ main concerns, and the corresponding revisions are highlighted in orange throughout the manuscript. Additionally, detailed point-by-point responses to each reviewer’s comments are provided in the following sections.

Again, we appreciate reviewers' invaluable contributions toward improving the quality of this work.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper presents SciLitLLM, a language model tailored for scientific literature understanding, utilizing a hybrid approach that combines continual pre-training (CPT) with supervised fine-tuning (SFT). The proposed pipeline emphasizes constructing high-quality CPT datasets and generating diverse domain-specific instructions for SFT. The paper demonstrates competitive results on benchmarks like SciAssess and SciRIFF and introduces datasets and pipelines with potential applicability beyond scientific domains.

*Strengths*:  
-Motivation and design: The paper provides a well-motivated approach addressing both domain knowledge enhancement and task-specific instruction alignment through its hybrid CPT+SFT strategy. Reviewers aknowledged the pipeline's clarity and the methodological rigor (fcV1, TiCd, and W9HG)  
-Contributions: Reviewers appreciated the overall pipeline, including LLM-based data processing, format correction, and quality filtering mechanisms. The instruction synthesis method effectively addresses the challenge of limited scientific data. In addition the released models and data are valuable to the community.  
-Empirical results: SciLitLLM achieves strong results compared with baselines and reviewers (fcV1 and W9HG) also noted its generalizability to other domains.   

*Weaknesses*:

-Dataset processing: Reviewer 7iRT raised concerns about the use of PyPDF2 for document parsing. The authors acknowledged this limitation and clarified their choice was driven by computational constraints.   
-Computational efficiency: Reviewer 7iRT questioned the computational cost-benefit ratio of achieving GPT-4o comparable performance
The authors provided context about the practical value of their approach for organizations needing private, domain-specific models.    
-Comparative Analysis: Reviewers suggested additional comparisons with other model families and datasets. The authors conducted contamination studies and quality filtering threshold experiments during the rebuttal period to address these concerns.  
-Closed nature of the textbook data: A major weakness is the closeness of the textbook data used in CPT. The authors do not have any plans to release such data, they might have legitimate copyright reasons though and they plan to release the list of the books.   
-Some reviewers suggested comparisons (e.g., with Llama3). Such additional comparisons were acknowledged as valuable future work but were constrained by computational resources.