SciLitLLM: How to Adapt LLMs for Scientific Literature Understanding
Authors: Sihang Li
Jin Huang
Jiaxi Zhuang
Yaorui Shi
Xiaochen Cai
Mingjun Xu
Xiang Wang
Linfeng Zhang
Guolin Ke
Hengxing Cai
Venue: ICLR 2025 Poster (2025)
Keywords: Large Language Model
Pre-training
Supervised Fine-tuning
Scientific Literature Understanding
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)

Abstract: Scientific literature understanding is crucial for extracting targeted information and garnering insights, thereby significantly advancing scientific discovery.
Despite the remarkable success of Large Language Models (LLMs), they face challenges in scientific literature understanding, primarily due to (1) a lack of scientific knowledge and (2) unfamiliarity with specialized scientific tasks.
To develop an LLM specialized in scientific literature understanding, we propose a hybrid strategy that integrates continual pre-training (CPT) and supervised fine-tuning (SFT), to simultaneously infuse scientific domain knowledge and enhance instruction-following capabilities for domain-specific tasks.
In this process, we identify two key challenges: (1) constructing high-quality CPT corpora, and (2) generating diverse SFT instructions. 
We address these challenges through a meticulous pipeline, including PDF text extraction, parsing content error correction, quality filtering, and synthetic instruction creation.
Applying this strategy, we present a suite of LLMs: SciLitLLM, specialized in scientific literature understanding.
These models demonstrate promising performance on scientific literature understanding benchmarks.
(1) We present an effective framework that integrates CPT and SFT to adapt LLMs to scientific literature understanding, which can also be easily adapted to other domains.
(2) We propose an LLM-based synthesis method to generate diverse and high-quality scientific instructions, resulting in a new instruction set -- SciLitIns -- for less-represented scientific domains. 
(3) SciLitLLM achieves promising performance in scientific literature understanding benchmarks.

Published as a conference paper at ICLR 2025
SCILITLLM: How TO ADAPT LLMS FOR SCIENTIFIC LITERATURE UNDERSTANDING
Sihang Li .in Huang , Jiaxi Zhuang Yaorui Shi Xiaochen Cai², Mingjun Xu Xiang Wang Linfeng Zhang Guolin Ke², Hengxing Cai
University of Science and Technology of China, DD Technology
{sihang0520, xiangwang
1223 223 ( /egmaii. com, {huangjin, caibengxing @dp tech "Equal contribution. Work done while interning at DP Technology. *Corresponding author.
ABSTRACT
Scientific literature understanding is crucial for extracting targeted information and garnering insights, thereby significantly advancing scientific discovery. De- spite the remarkable success of Large Language Models (LLMs), they face chal- lenges in scientific literature understanding, primarily due to (I) lack of scientific knowledge and (2) unfamiliarity with specialized scientific tasks. To develop LLM specialized in scientific literature understanding, we propose a hybrid strat- egy that integrates continual pre-training (CPT) and supervised fine-tuning (SFT), to simultaneously infuse scientific domain knowledge and enhance instruction- following capabilities for domain-specift tasks. In this process, we identify two key challenges: (1) constructing high-quality CPT corpora, and (2) gener- ating diverse SFT instructions. We address these challenges through a meticulous pipeline, including PDF text extraction, parsing content error correction, quality filtering, and synthetic instruction creation. Applying this strategy, we present suite of LLMs: SciLitLLM, specialized in scientific literature understanding. These models demonstrate promising performance on scientific literature under- standing benchmarks. Our contributions framework that integrates CPT and SFT to adapt LLMs to scientific literature un- derstanding, which can also be easily adapted to other domains. (2) We propose an I.LM-based synthesis method to generate diverse and high-quality scientific instructions, resulting in scientific domains. (3) SciLitLLM achieves promising performance in scientific literature understanding benchmarks. We release the data processing codes' and model weights?.
threefold: (1) We present effective
instruction set SciL
for less-represented
INTRODUCTION
66
ScILLLLM-144
SciLitLLM-7B
GPT-40
Scientific literature understanding involves the systematic evaluation and interpretation of sci- entific texts and publications, to identify trends,
SciRIFF Score
60
Liamas: 1-70B
Owen2 Qwen2.5-148 2-00
Qwin2.5722
54-
extract targeted information, and garner in- sights (A14Science & Quantum, 2023; Zheng al., 2023), significantly contributing to sci- entific discovery. Concurrently, Large Lan- guage Models (LLMs) (OpenAI, 2023; Qwen, 2024; Touvron et al., 2023) have achieved re- markable success natural language process- ing, prompting the development of domain-specific LLMs across various fields (Cui et al., 2024; Wu et al., 2023; Clusmann al., 2023). However, recent studies (Wadden et al., 2024; Cai et al., 2024; Singh et al., 2023) indicate that L.LMs face challenges when specializing in scientific litera- ture understanding, particularly in context understanding and question answering. Take Figure 2 as
Qwen2.5- 5-78
BPT-3.5
Llama8.1 1-.........
48
Wistral-78
Saiuuu
7B
70B
API
Number of Parameters
Figure 1: Average scores on SciRIFF of models with varying parameter sizes.
thub: com wtt.://ptcccccoo//ooeaeooo SMART
/https://hugginggfffe co/colocccoooooooooo lecti ena/ -nni-MMAR//cct 1111 1m11m1 5- 07-66377333388 975ba 929

---

Published as a conference paper at ICLR 2025
Scientific Literature Understandlng
Abstrect: .e invertipaedd whether single nucestide polymorphems [SVPs] of HNFA encoding HNFF4slpha influenced the conversion at evaluating
impaired glucose tolerance (IGT) type diabetes melitusin subjects of STOP-NIDDN
presentin of type diabetes melitus Three SNPs 92 promoter region |n44810424, r113846 14, and 122 144908) were in almest complete assocition ID 20.57, (12 and, therefore, only 1548 10424 was included in further onalyses. No assecation was found in men, Hapletype analysts based on three SNPs Irs4810424, r22771197 and r53818247) indicated that conversion
effect arathhue compared placebo almmestcomppess
1
Scientific Document
dabetes melituus dependent the rumber nik alleles...
You will be shown an abstradt from a alooreica reswarch paper. Gien this abstract your tesk is extrect unique entities of the fellowing types.| ["Celliin",, "Dsease", "Variant") "I. "OEsease" ("Dlabetes", ... "Variant" "Vrraa"."""P"7SA""" ("Pro7SAla" ,...). // Rest of the task.
culputas ISON object the format: ("Celliin"' (hBPP"" ...
2 Ta sk
Galactica-6.77
Llama-3.1-88
SciLitLMM78 (Ours)
Answer
"CalLine"
"CelLinw":
Ittink you can use spaCy to do this.
"Diseas""
"IGT"
speCy any for natural language
processing (NLP) Python. is
"Dioaase::
"Impared gucose tolerance"
"type diaette melituss, "dlabotes"
"Ty- 2 disbatos milituss
estend, to be fast and memory efficient.
"Vari "Pro75Al" 7s1188414" 1r548 10424 *ss21 +22144800", +620711977
"Varant". 1x.31 7235162477 "w207 1197" "r51086614" "122146007; Tro48
Prediction by LLMs
You can use it to extract entities from your abstract.
3
X
X
Ucdommber gith Soconiie
Soleands Fouesiddu
Figure 2: An example of scientific literature understanding in SciRIFF. It involves extracting accu- rate entities from a biomedicine paper. Scil.itII.MM77 demonstrates sufficient scientific knowledge and instruction-followii ability to accurately identify and extract these entities.
an example, where the LLM is asked to understand the content of a biomedical research paper and then extract the targeted information. LLMs' potential might be hindered by two major barriers: (1) a lack of scientific knowledge, which results in errors such as the missing important entities in Llama-3.1-80 (AI Meta, 2024), and (2) unfamiliarity with scientific tasks, leading to the inability of Galactica-6.78 (Taylor et al., 2022) to follow task instructions accurately.
To make LLMs specialized in science-relevant tasks, existing studies mostly adopt two strategies, as illustrated in Figure 3: (1) Fine-tuning with scientific instructions (Wadden et al., 2024; Zhang LLM is fine-tuned with collected domain-
et al., 2024b; Singhal et al., 2022). A specific instructions to adapt it to science-relevan tasks. However, instruction fine-tuning alone insufficient imbue the models with comprehensive scientific knowledge.
(2) Pre-training on scientific corpora (Beltagy et al., 2019; Zeng et al., 2022; Taylor et al., 2022). This approach involves training models on vast scientific corpora. While this method equips LLMS with domain knowledge, confines them to solving relevant tasks. Moreover, it is hampered by
the lack of instruction-tuning substantial computational costs and data requirements (Ling et al., 2023; Yang et al., 2023). To address these obstacles while balancing efficiency, we propose hybrid strategy that incorporates continual pre-training (CPT) and supervised fine-tuning (SFT), to simultaneously infuse domain knowledge and enhance domain-specific instroctionnffllloinng capabilities.
However, as illustrated in Figure 4, developing a scientific literature understanding model using this CPT and SFT pipeline presents two critical requirements: (1) High-quality CPT Corpora. Scientific corpora, predominantly in PDF format such as textbooks and research papers, are not directly digestible for LLM training. Converting these documents to text with PDF parsing tools introduces formatting and syntax errors, degrading corpus quality. Worse still, scientific documents often contain segments that contribute little information (e.g., references), necessitating quality con- trol to filter them out. See the first row in Figure 4 for comparison of high- and low-quality CPT texts. (2) Diverse Scientific Instructions. Effective instruction following for scientific literature understanding requires a large, high-quality, and diverse set of task-related instructions. However, to the best of our knowledge, there is a scarcity of well-designed instruction datasets for scientific literature understanding, and hiring human annotators to curate such dataset from scratch is pro- hibitively expensive (Erdmann et al., 2019; Qiu et al., 2023). See the second row in Figure 4 for an illustration of high- and low-quality instructions.
To address these challenges, we devise an effective pipeline to construct nigh-quality domain corpora for CPT and diverse scientific instructions for SFT, as illustrated in Figure 5:
In the CPT stage for domain knowledge injection, we start with an extensive in-house corpus con- sisting of 73k texthooks and 625k academic papers in the scientific field, all in PDF format. Initially,
2

---

Published as a conference paper at ICLR 2025
Teigh Quality
Quality
Instrction Taniin
aathrre Tatuer salrng
Inc aos
UM
nousie utts
Pre-taaiing Corpora
Score 4.7
0166
Pro
Gesiralor Untramed Luk)
Selvvttic UMM
Quption our an CVSal
Queatic les 2n o00
(Dung)
UMM Gese
Genrral
U.M
Valovallee
LLM
Superoledd
deaer: Sile 373
Figure 3: Comparison of strategies to adapt Figure 4: Examples of high and low-quality CPT text and SFT instructions.
LLMs scientific tasks.
we leverage PyPDF23, widely used open-source PDF parsing tool, extract raw texts from these documents. We then employ moderate yet powerful model, Llama3-8B-Instee format and spelling errors introduced by PDF parsing (ef. Section 2.1.1). Subsequently, we train a small text quality classifier to score the corpus and filter out texts of low educational value" in the scientific field (cf. Section 2. 2.1.2). These two simple yet effective textual refinement and quality con- trol measures ensure the high quality of our CPT corpus, culminating 12.7 billion tokens for CPT via the Qwen2.5 tokenizer.
to correct the
In the SFT stage for domain instruction fine-tuning, to overcome the scarcity of domain-specifee in- structions and the high cost of human annotations, we propose a novel instruction synthesis method (cf. Section 2.2.1). 1). It enables us to generate diverse instructions to better equip the model for domain-specific tasks. Moreover, we sequentially apply instruction duplication based on Leven- shtein distance and an LLM-based filtering method to ensure the quality of synthetic instructions (cf. Section 2.2.2).
Having established such high-quality datasets, apply the CPT-SFT integration strategy on general-purpose I.IM - Qwen2.5 (Qwen, 2024) and obtain SciL.it.L.M of two scales: 7B and 14B. Evaluations on benchmarks of scientific literature understanding demonstrate the effectiveness of our strategy, We observe promising performance enhancements, with an average improvement of 4.0% on SciAssess (Cai al., 2024) and 10.1% on SciRIFF (Wadden al., 2024), compared to the leading LLMs under 10B parameters. Notably, SciLitLLM-7D even outperforms Llama3. 1 and Qwen2.5 with 70B parameters on SciRIFF. Additionally, SciLitLLM- 14B achieves leading results on both benchmarks, surpassing other open-source LLMs. Further ablation studies demonstrate the effectiveness of each module in our pipeline.
In summary, our contributions are threefold: (1) We devise an effective and comprehensive pipeline to adapt general LLMs to specific domain scientific literature understanding. combines con- tinual pre-training (CPT) and supervised fine-tuning (SFT), to enhance scientific knowledge base and instruction- wow.looinnoooooinnggoooooooon capabilities for specialized domain tasks. (2) We propose a novel domain instruction synthesis method to curate instructions for scientific literature understanding, resulting in a new dataset SciLitIns. (3) SciLitLLLM trained through the proposed pipeline, outperforms leading open-source LLMs scientific literature understanding.
2 METHOD
In this section, present the details of proposed pipeline (cf. Figure 5): continual pre-training for scientific knowledge injection and supervised fine-tuning for scientific tasks enhancement, We refer Appendix A for comprehensive literature review on knowledge injection, domain adapta- tion, and scientific literature understanding for LLMs.
2.1 CPT FOR SCIENTIFIC KNOWLEDGG INTECTION
Phtpps: : h/pppd/2/ www.rreddddccs...
'Phi models (Gunasekar al., 2023; Li et al., 2023; Abdin al., 2024) propose to determine the quality of pre-training text by its educational value for student whose goal is learn basic domain concepts.
3

---

Published as a conference paper at ICLR 2025
CPT: Scientific Knowledge Injection
Data Collection
Reformat Grammar Correction
CPT Quality Filtoring
have estracted the relowiin coe proses intoducedd formatting Roues, Please help me connec these
books
Owen22.5 Base
lires Dela Ouurity
10M Journal Paragraphs
seadable version of the
50k Se
Quality Clasdller
CPT Data
Baw Test: glaw Text!)
Open-tource Data
cornected
Scientific Instruction Following
Word Frequency Table
Keywords
SFT Quality Filtering
Instruction Generation
coid
Conerate an Enility Extraction question using lowingg key
Lirraa--T0 TEO
word Feegencc foer
Pre-defined Tasks
Dusetion: operaprap bayotonden. rebronndpp: (patieme chanical reaction)
SFT Data
e ScLLttLMM-instuue
Tuble Entruction Moleczte Generation
Care
Dedapication
Figure 5: The pipeline of SciLitLLM consists of two key stages: continual pre-training (CPT) for scientific knowledge injection and supervised fine-tuning (SFT) for scientific instruction following. Specifically, the CPT stage involves several modules: PDF parsing, format & grammar correction (ef. Section 2.,.1), and quality filtering (cf. Section 2.1.2) modules. These modules ensure the model equipped with high-quality scientific domain knowledge. The SFT stage includes LLM- based instruction generation (cf. Section 2.2.1) and instruction quality control (cf. Section 2.2.2) measures. These steps are designed to fine-tune the model's ability follow scientific instructions accurately and effectively.
What high-quality pre-training corpora? Researchers (Gunasekar et al., 2023; Li et al., 2023; Abdin al., 2024) suggest that language models benefit from corpora that possess the same qualities as an exemplary textbook for human learners: clarity, self-containment, in- structiveness, and balance. Recognizing the wealth of high-quality scientific textbooks and research papers over the past decades, we have curated substantial collection of over 73,000 textbooks and 625,000 research papers within the scientific domain, ensuring all documents are copyright-compliant.
Stage sounce
Domain
#Dociila
In-house Textbooks
Science
73k
10B
CPT
In-house Joumals
Science
625k
2,7B
Rodpajama (Computer, 2023)
General
11B
SciLitins
Science
93k
86M
SET
SciRIFF (Wadden et al., 2024)
Science
70k
40M
Infinity-Instree
General
3M
1.7B
Table 1: Data statistics of CPT and SFT stages.
Underlined datasets are curated by
However, we still face two practical obstacles when dealing with those textbooks and research pa- pers: (1) Formatting and synlax errors. Most textbooks and research paper documents are in PDF format, which is not directly digestible by LLMs. Converting these documents using tools like PyPDF2 often introduces formatting and syntax errors, which degrade the quality the corpus. (2) Corpus quality control. Despite their overall high quality, textbooks and research papers also con- tain segments with little useful information, such as references and garbled text introduced during the PDF parsing process. To tackle these obstacles, we devised the following modules: format & grammar correction and CPT quality filter.
2.1.1 FORMAT & GRAMMAR CORRECTION
A parsed text from PDF document often contains many formatting and syntax errors. To address this issue, we prompt Mlama--8B-Instruss to correct these errors introduced during the PDF parsing process. Utilizing the vLLM (Kwon et al., 2023) backend, Blama3--BBIInstee imately 2.52 million tokens per Nvidia A100 GPU hour. The process takes over 5,000 A100 GPU hours handle all the textbooks and research papers. Example texts both before and after process- ing along with the prompt template are provided in Appendix to demonstrate the improvements through this correction process.
can process approx-
4

---

Published as a conference paper at ICLR 2025
Sumplally
Quality
Figure 6: Score distribution of the CPT Data. The filtered-out 25% data marked gray, while the re- maining 75% are marked orange.
Figure 7: The quality of SciLitlns is evaluated from five aspects: clarity, complexity, correctness, useful- ness, and adaptability. Instructions with average score of less than 4 are filtered out.
2.1.2 CPT QUALITY FILTER
During CPT, maintaining the quality of the training corpus crucial for effective knowledge in- jection. Given the extremely large scale of pre-training corpora, assessing quality through human annotation is feasible (Erdmann et al., 2019; Qiu et al., 2023). Consequently, leading LLMs (e.g., Phi (Gunasekar et al., 2023), Llama (Touvron et al., 2023), and Qwen2.5 (Qwen, 2024)) em- ploy model-based quality filters. The typical process involves using larger LLMs to score the quality of subset of texts, which then serve labels for training small classifiers (e.g., random forest (Gu- nasekar et al., 2023) and Bert (AI@ Meta, 2024)) to annotate the entire training corpus. Inspired by this approach, design resource-efficient method based
lightweight text quality classifier.
We first annotate a random subset of 50k CPT texts using Llama3-70BB 70---777--nnntucct We adapt the quality assessment prompt from for web data, to evaluate the educational value (Gunasekar et al., 2023) of the scientific knowledge in each sampled text, assigning scores ranging from (lowest quality) annotation, we perform supervised transfer learning on the fineweb-edu-classffe checkpoint.
(Anton et al., 2024), widely-used quality classifier
(highest quality). After
This process results in scientific text quality classifier tailored for scientific corpus assessment. We then utilize this classifier to assess the quality of the entire CPT dataset and exclude the lowest- scoring 25% (See Figure for conerete samples). The distribution of the scores illustrated in Figure 6. We refer to Appendix C for more details about classifier training and sensitivity analysis of the filtering threshold.
2.1.3 CPT TRAINING SETTINGS
After preparing all the corpus, we perform CPT on Qwen2.5-Base (Qwen, 2024) for one epoch, encompassing 23.7 billion tokens (ef. Table 1), with sequence length of 2,048 tokens. To maintain the model's general knowledge, we also include a similar scale of general corpus tokens from Red- pajama (Computer, 2023). To stabilize the learning procedure, we gradually decrease the learning rate from 1 x 10 to 0 with a cosine scheduler. To address overfitting, we apply weight decay of 0.1 and gradients were clipped at a maximum value of 1.0. The CPT training took approximately days on 32 Nvidia A100 GPUs for MGLitLLM-7B--Baatt Base and about 7 days for the 14B model.
2.2 SFT FOR SCIENTIFIC INSTRUCTION FOLLOWING
After performing CPT on extensive scientific corpus to incorporate domain knowledge, we sub- sequently conduct SFT on domain-speciff instructions to enhance the model's ability to understand scientific literature. We identify two major challenges in SFT for scientific instruction following:
Existing instruction-tumie -uuning datasets in the scientific domain (Feng al., 2024; Fang et al., 2023) primarily focus on fields such as physics, chemistry, and biology. Manually collecting instruction- tuning data for other less-represented vertical domains (e.g., biomedicine, and material) both time-consuming and costly (Erdmann et al., 2019; Qju et al., 2023).
Few instruction-tuning datasets adequately reflect the scenario of scientific literature understand- ing, which typically involves a segment of scientific literature accompanied by a question that requires deriving answer from the text.
5

---

Published as a conference paper at ICLR 2025
To address these challenges, draw inspiration from leading models (e.g.. Nemotron-4 (Adler et al., 2024), Phi (Gunasekar et al., 2023), and Qwen (Yang et al., 2024)), which leverage existing LLMs to construct synthetic instruction sets. We propose novel instruction synthesis method to curate instructions specifically for scientific literature understanding.
2.2.1 INSTRUCTION SYNTHESIS OF LESS-REPRESENTED DOMAINS
Unlike typical question-answer pairs, an instruction for a scientific literature understanding task comprises three components (Wadden et al., 2024): (I) segment of scientific literature, (2) a ques- tion pertaining to the context, and (3) the corresponding answer. Simply prompting an LLM to generate scientific context along with associated question-answe the instructions or parameters often yields similar or repeated contents. Thus, we design a simple yet effective three-step pipeline to generate diverse and high-quality scientific contexts and corre- sponding question-answer pairs, consisting of the following:
pair without variations in
Probability table of domain keywords. For a target scientific domain (e.g.. biomedicine, and ma- terial), collect dozens of high-impact research papers via Google Scholar" and count the word frequency appearing in these papers. Then, we obtain a probability table of domain keywords.
Scientific task list. Since LLMs are expected to handle various types scientific tasks, an compile list of task descriptions
struction set with task diversity is essential. Therefore, by including representative tasks from existing scientific NLP datasets (Wadden et al., 2024; Cai et al., 2024; Feng al., 2024), covering many scenarios possible that LLM may encounter in real applications.
Instruction Generation. Given a word probability table and the task list for specific scientific domain, we sample 20 keywords and a task description each time. Subsequently, GPT-4o (Ope- nAI, 2023) is prompted to generate a scientific context containing the sampled keywords and a question-answer pair according to the provided task description.
Utilizing this pipeline, we obtain over 100k synthetic instructions covering less-represented scien- tific domains and various types of specialized tasks (details are presented Appendix D.I).
2.2.2 INSTRUCTION QUALITY CONTROL
To ensure the diversity and quality of generated instructions, effective measures for quality control are essential. Specifically, we incorporate heuristic deduplication and LLM-based filtering.
1. Heuristic deduplication. Despite the measures taken during the generation process to prevent high homogeneity the instructions, the generated data points may still contain similar questions or identical answers. To eliminate such redundancy, we implement a deduplication process to remove similar data points. See Appendix D.2 for details.
2. LLM-based filtering. Inspired by recent efforts (Eldan & Li, 2023; Chiang & Lee, 2023; Zhang ct al., 2024c) to measure the quality of generated content using LLMs, we leverage Llama-3- Ins-ruttt to assess the quality of generated instructions for five aspects: clarity, complexity, correctness, usefulness, and adaptability. We show the quality statistics of synthetic instructions in Figure 7. Instructions with average score of less than are filtered out. The detailed recipe for instruction quality evaluation with concrete examples is included in Appendix D.3.
Through instruction synthesis and quality control pipeline, we obtain SciLitIns, consisting of 93,894 high-quality and diverse instructions for scientific literature understanding.
2.2.3 SFT TRAINING SETTINGS
Our SFT training dataset consists of three parts: SciLitIns, SciRIFF (Wadden et al., 2024) and as shown in Table 1. Infinity-Instrue is a collection of more than twenty open- source instructions datasets, covering various general domains. SciRIFF and SeiLitIns contain spe- cialized instructions for scientific literature understanding. For both scales, we train for one epoch
com/
'https: http:///gggggggaooooooc//aa////a
"https://hugginggface

---

Published as a conference paper at ICLR 2025
Models
MMLUU-Pro-Bio
MMLLU-Pro-Chem
MMMLU-Prr-Heal
MuSeQA
Avg.
Parameters < 10B
Llamm3.1 1-8 BB-B
$6.35
25.88
44.50
53.28
45.00
Qwen2.5.701 7B- Base
68.36
48.37
51.64
56.23
56.15
Scil COILHELLM--B-BasS
70.45
$1.21
54.06
89.27
58.75
Parameters >- 10B
Owen2.5-14B-BBeee CeILitLLMM14B-BBaag
79.78
66.92
80.62
64.44
64.24
68,06
69.34
Table 2: Accuracy(%) comparison of base models. Sc outperforms Llama3. and Qwen2.5 with similar scales, demonstrating improved domain cemain-specific understanding through continual pre- training on scientific corpora. Underline indicates those trained on domain domain-specific corpora.
Dusasel
Task
Misini-B
Quc12.5.78
SaTTule-88
Qwes2.5-14B
Lium33.1-00
Parameeer 2 Qwcn2.5:728
Biddggs
52.0
63.4
60.9
45.3
65.3
65.9
69.4
67.2
67.0
35.4
68.9
Chemisry
34.0
46.1
47.9
19.0
55.4
80.0
62.5
63.7
614
37.3
66.0
SciAssess
Materia
36,5
50.9
43.9
31.3
$3.7
$7.9
59.2
59.4
62.6
37.0
62.0
Malicin:
25.1
30.2
25.3
19.9
32.4
31.5
37.4
37.8
36.9
51.8
4558
Metn
369
27.7
46.5
"259
"51.7
53.
377
57.0"
362
20.3
"6._""
BioASQ
Biek
55.7
57.8
610
DiscMT
68.8
773
17:2
17.2
23.3
11.
23.5
21.2
2600
26.3
30.8
19.2
26.7
ScIRIFF
MuP
MC
87.5
47.5
NT.0
49.7
84.9
50.8
72.1
34.6
67.5
70.7
91.5
542
609
90.9
94.0
56.0
67.7
72.1
47.8
76.8
86.9
58.7
Queerr
48 48.844322
46 5442.7
18.1
$4.21866
50
50.7/8411
4844468
1440.2
61,98005
$4
14.7799
67 8/90.5
SciERC
51.1
32.9
30.1
35.6
49.9
33.7
36.2
34.1
$4.7
28.6
42.2
SciFad
68.8/53.8
63.853377
77.860.3
66,04992
83.5W7.3
53 18INS6
87,3003
86.5/99
9LAN36
60.7/53.
84 84.3998.7
Metn
"466
305
50.5
47.0
66.6
56.7
363
58.2"
6474
50.
60.5"
Table 3: Model performances on scientific literature understanding benchmarks: SciAssess and SciRIFF. SciLitLLM-7B and SciLitLLM-1pp achieve leading performance compared with open- source models of similar scales. The best-performing models smaller than 10B and larger than 10B highlighted in bold. Results for SciTulu-7B, GPT-3.5, and GPT-4o on SciRIFF taken from its original papers, while all other results are produced by us. Underline indicates those trained on dopain-specific instructions.
on Infinity-Instruus to cultivate their general cnttructionfollloviin abilities, then for five epochs on SciLiitnn and SciRIFF for scientific literature understanding enhancement. The training is conducted with a sequence length of 4,096, maximum learning rate of 1 x 10-5, 5, and a cosine scheduler. The SFT training takes approximately 32 hours for the 7B and 70 hours for the 14B model on 32 A100 GPUs, resulting in SciLitLLLM77 ----LM------nstrc Instruct and SciLitLLM- 14B-Instruct.
EXPERIMENTS
In this section, we perform experiments answer the following research questions: (Q1) How does SciLitLLM perform scientific literature understanding tasks? (Q2) Can CPT with domain- specific corpora aid in scientific knowledge injection? (Q3) Can SET with SciLitIns improve per- formance on scientific literature understanding tasks?
3.1
EXPERIMENTAL SETUP
3.1.1 BENCHMARKS
To evaluate the performance of LLMs regarding scientific knowledge base and specialized task- solving abilities, our benchmarks include:
Base model benchmarks. To scientific literature understanding, we evaluate the base models on scientific subjects - biology, chemistry, and health in MMLU-Pro (Wang al., 2024) and MaScQA (Zaki et al., 2023) question answering dataset of materials science.
their scientific knowledge, which serves as the foundation for
Instruction model benchmarks. We evaluate the instruct models on scientific literature understand- ing benchmarks: SciRIFF (Wadden al.. 2024) and SciAssess (Cai al., 2024). Due the space limitation, we refer to their original paper for the detailed metrics each task. Brief descriptions of them are provided in Appendix E.

---

Published as a conference paper at ICLR 2025
3.1.2 BASELINES
Base model baselines. We compare SciLitLLM-bass against leading open-source base models: Llama-3.1 (AI@ 2024). Qwen-2.5 (Qwen, 2024).
Instruction model baselines. We benchmark leading instruction LLMs including GPT-(3.5, 4o] (Brown et al., 2020; OpenAl, 2023), (lama3.1-[8B) 70BJ (AI@Meta, 2024) and Qwen2.5- [7B, 14B, 72B] (Yang et al., 2024; Qwen, 2024) and Mistral-7B (Jiang et al., 2023). We also report the performance of SciTulu-7B (Wadden et al., 2024), which is a fine-tuned Llama2-7B (Touvron al., 2023) model on SciRIFF.
3.2 PERFORMANCE OVERVIEW (QI)
3.2.1 BASE MODEL PERFORMANCE
The performance comparison of base models is shown in Table 2. Scilitt.LM-aase consistently outperforms other general base models across four scientific benchmarks, indicating its extensive domain knowledge. For instance, SciLiLLL-7B--Ba of 2.60% over Qwen2.5-7BB 5-B---BBBBB Base, despite both models having comparable parameter count. This improvement can be attributed the specialized pretraining on high-quality domain -speciii cor- pora. It benefits from exposure to scientific language patterns, terminology, and reasoning.
achieves an average accuracy improvement
3.2.2 INSTRUCT MODEL PERFORMANCE
As shown in Table 3, compared with models with similar scale, Sci. achieves the highest performance in all 4 domains on SciAssess, outperforming the second-best model (i.e., Ilama3. 8B) by notable 4.0%. On SciRIFF, it surpasses baseline models by even more substantial margin of 10.1% %. These results highlight the effectiveness of our pipeline in adapting a general-purpose LLM into specialized model for scientific literature understanding.
Specifically, one key contributor is our synthetic instruction dataset - SciLitIns, which offers com- prehensive task coverage across diverse domains. It ensures that the model can generalize across tasks with varying input data and reasoning complexities, such as table extraction and molecule generation. For example, Scil Lt.LittLLM--B leads in 15 out of 27 sub-tasks SciAssess, indicating its strong ability to adapt to the specialized requirements of these tasks.
Notably, SciLiLLLM-14B shows significant performance gains over Qwen2-14B on both bench- marks, with a 4.4% improvement on SciAssess and a 7.5% improvement on SciRIFF. These results suggest that approach effectively leverages the larger capacity of the 14B parameter model, enabling it to encode more nuanced scientific reasoning and domain-specine knowledge. Interest- ingly, SciLitLLM-14B also outperforms much larger open-source models, such as Llama3. 1-70B and Qwen2.5-72BB despite having only one-fifth the number of parameters. This demonstrates that our continual pretraining and instruction finettnning pipeline is not merely scale-dependent but strategically optimized for domain adaptation.
The performance on SciRIFF further illustrates this optimization. Comparing with open-source models with similar scale, both SciLitLLM-7B and SciLiLLLM-14B rank first in 8 out of 11 evalu- ations, including challenging tasks like Qasper and SciFact. These tasks require advanced abilities in scientific question answering and fact verification, respectively, which are likely bolstered by the diversity of high-quality examples SciLitIns.
These findings underscore the broader importance of designing domain-specific LLMs. While general-purpose purpose models, such as Llama and Qwen, perform well on broader tasks, their lack of domain specialization limits their effectiveness in fields like scientific literature understanding. The results demonstrate that the adaptation pipeline designed by us can yield promising performance without relying solely on increased model size.
general-
3.3 ABLATION STUDY (Q2 Q3)
We conducted ablation experiments three key components in our pipeline: the CPT stage, the SFT data recipe, and the instruction quality filter, to investigate their effectiveness. Note that all ablation experiments were performed on SciL GriLitLLL--77 due budget constraints.
8

---

Published as a conference paper at ICLR 2025
Benchmark
SciAssess SciRIFF
SFT Dataset
SeiAssess SciRIFF
Benchmark
SciAssess
SeiRIFF
Model
Infinit-Insmmit only
47.1
51.2
SciLitIns
Qwen2.5 5 Insstruct
Qwen2.5. Swve2.5-Base-SEE SET
49.8
46.5
57.0
50.3
SciRFF
allns
46.8
56.7
w/o filtering
50.9
60.1
Owen2.5Basee-CPTE -CPTSSTT
51.7
60.6
SeiRIFF SciLitins
51.7
60.6
w/ filtering
51.7
60.6
Table 6: Ablation study of SFT instruction quality fil- tering. This shows that our proposed filter refines the quality of SciLitIns.
Table 4: Ablation study of the CPT stage. The results shows that the CPT stage is essential to improve scientific literature un- derstanding ability.
Table 5: Ablation study of SFT data recipes. Our syn- thetic dataset SciLitIns effec- tively improves performance both benchmarks.
3.3.1 SCIENTIFIC KNOWLEDGE INJECTION VIA CPT (Q2)
We investigate the contribution the CPT stage for SciLitI.L.. We compare the three variants: (1) 1enrn..5-7-Instruaee official instruct-model checkpoint; (2) Qwen2.5-7B-buse SFT: applying
our SFT stage directly to Qwen2.5-7B-bar without CPT; (2) Qwen2.5-7B-bor + CPT + SFT: SciLiLLLM-7BIInttun
As shown in Table 4, applying SFT alone to the Qwen2.5-7B- on SciAssess (+3.3%) and SciRIFF (+6.7%). This demonstrates that task-specific instruction fine- tuning effectively enhances performance by adapting the model to the specialized tasks in scientific literature understanding. Further, incorporating CPT results in extra 1.9% improvement Assess and a 3.6% gain on SciRIFF. These results highlight the unique role of CPT in pre-adapting the base model scientific contexts. By curating high-quality scientific corpus and using it for domain-specific pretraining, the CPT stage injects foundational scientific knowledge into the model. This foundational understanding reduces the gap between general-purpose pretraining and the spe- cific requirements of downstream tasks, allowing the subsequent SFT stage to focus on fine-grained task adaptation rather than compensating for domain knowledge deficiencies.
model yields improvements
Sci-
3.3.2 SFT WITH SCILITINS (Q3)
We explore how fine-tuning with our synthetic instruction set, SciLitlns, impacts performance in scientific literature understanding. We incrementally add three datasets to the SFT training set: Instrity-Instruct, SciRIFF, and SciLitIns. As shown in Table 5, using only the general purpose instruction set, Infinity-Instrun results the lowest performance on both SciAssess (47.1% 1%) and SciRIFF (51.2%). This result highlights the limitations of general-purpose instructions in addressing the nuanced demands scientific literature tasks. General purepose datasets often lack the domain- specific structure, reasoning, and terminology required for tasks like scientific fact verification or molecule generation. These findings emphasize the need for specialized instructions to bridge the gap between general capabilities and domain-specific expertise.
Adding the training set of SciRIFF to Infinity-Instruet improves performance on SciRIFF signifi- cantly (+5.5% but decreases performance on SciAssess (-0.3%). This drop may be due SciRIFF's relatively narrower focus, primarily covering biomedicine and clinical medicine, while SciAssess spans broader fields such as biology, chemistry, and material science. This highlights the need for diverse instruction set that covers diverse scientific domains. Replacing SciRIFF with SciLitIns in the fine-tuning process yields better performance than Infmnit--Instruc on both benchmarks. achiev- ing 50.2% on SciAssess and 54.8% on SciRIFF. The diverse task coverage in ittns plays key role in this improvement. By incorporating synthetic instructions from underrepresented domains, SciLitIns broadens the model's exposure to scientific knowledge, enabling it to generalize across varied scientific disciplines.
Finally, combining SciRIFF and SciLitIns yields the best results, with SciAssess reaching 51.7% and SciRIFF achieving 60.6%. These gains suggest that SciLitIns and SciRIFF complement each other. addressing the limitations of each dataset individually. While SciRIFF focuses highly specific domains like biomedicine, SciLitIns fills the gaps by covering tasks from broader and less- represented scientific areas.

---

Published as a conference paper at ICLR 2025
3.3.3 IMPACT OF INSTRUCTION FILTER
We conduct an ablation study to assess the impact of quality filter for synthetic instructions by varying whether the dataset SciLitIns was filtered. As discussed in Section 2.2.2, this filter removes low-quality instructions evaluated from five key aspects. Table 6 shows that applying the filter improves the performance of Scil Lt-.L- SciAssess (+0.8%) and SciRIFF (+0.5%).
While the performance improvements from filtering are evident, the gains are relatively modest. This suggests that the dataset already of reasonably high quality before filtering, with only small proportion of low-quality instructions being removed. It also indicates that the fine-tuning process is robust enough tolerate some noise, although further improvements can be achieved with higher-quality inputs.
4 LIMITATIONS
Despite the promising results achieved by SciLitLLM, there are several limitations that should be acknowledged as follows:
Insufficient data volume. Compared with existing pre-training datasets (At@Meta, 2024; Yang et al., 2024; Taylor et al., 2022), the amount of data used for CPT is comparatively small. While the dataset has been carefully curated to ensure quality and domain relevance, the limited size poses challenges in achieving comprehensive representation across diverse scientific disciplines. With a smaller corpus, certain specialized may be underrepresented, limiting the model's ability perform well on tasks within these areas. While increasing dataset size is a logical step, it is crucial to maintain high-quality filter to avoid introducing noise. The effectiveness of the current CPT stage demonstrates that smaller but well-curated dataset can still deliver significant gains. Therefore, the challenge lies in scaling the corpus size while preserving its relevance and quality.
Lack of reasoning enhancement. The current pipeline does not explore advanced reasoning tech- niques such as Chain-offThought (Wei et al., 2022) or Tree-of- Thought (Yao et al., 2023) in the data construction or model inference stages.
Lack of preference alignment. Due to a limited financial budget, the model lacks Reinforcement Learning from Human Feedback (RLHF) (Ouyang et al., 2022). RLHF has shown significant improvements in aligning models with human preferences and ensuring more reliable outputs. Implementing RLHF in future iterations could further enhance the model's reliability.
Addressing these limitations in future research will be crucial developing a more robust and capable LLM specialized in scientific literature understanding.
5 CONCLUSION AND FUTURE WORKS
In this paper, introduce Sc,,itLLM, a specialized model for scientific literature understanding. initialized with a general base model Qwen2.5, and trained through continual pre-training (CPT) and supervised fine-tuning (SFT). For effective scientific knowledge injection during CPT, we propose model-based format and grammar correction method, along with text quality filtering measures. To ensure high-quality and diverse instructions during SFT, we devise instruction synthe- sis and quality control approaches. Our experiments on widely-used benchmarks demonstrate the effectiveness of this pipeline in adapting a general model the field of scientific literature under- standing. Specifically, It achieves a 4.0% improvement on the SciAssess (Cai et al., 2024) and a 10.1 1% improvement on the SeiRIFF (Wadden et al., 2024) compared to leading models with fewer than 10 billion parameters. SciLitLLM-144 also surpasses leading open-source LLMs with around 70B parameters. We note that this pipeline could be easily adapted to other specialized domains, particularly those lacking adequate high-quality corpora and instruction sets.
Our future work will focus on expanding the diversity and quality of the training data, as well as ex- ploring more efficient methods for domain -omain-specific knowledge injection and high-quality instruction generation. Moreover, we plan to expand our pipeline to include the RLHF (Ouyang et al., 2022) stage for better human preference alignment and safety.
10

---

Published as a conference paper at ICLR 2025
ETHICS STATEMENT
In developing SciLitLLM, we prioritized ethical considerations to ensure the responsible use of our models and methodologies. First, our research does not involve human subjects, and all data used for continual pre-training (CPT) are copyright-complian. We employed rigorous quality filter- ing process to minimize the risk of incorporating biased or misleading content. Nevertheless, we acknowledge that biases inherent in scientific literature, including historical underrepresenttaion of certain research domains, may propagate into the model's outputs. We also adhere to all applicable legal and ethical research guidelines, such as respecting copyright policies during dataset construc- tion and providing comprehensive model documentation. Our work is conducted with a commitment to research integrity, ensuring that contributions remain beneficial to the scientific community while addressing ethical responsibilities associated with developing Al technologies.
REPRODUCIBILITY STATEMENT
We have made extensive efforts to ensure the reproducibility of our work. Our proposed SciLitLLM models are available. The corresponding training pipeline have been described Sections 2. The source code for all modules shown in Figure 5 is included in the supplementary material. The training datasets for both CPT and SFT stages are shown in Table 1.
ACKNOWLEDGEMENT
The work of Sihang Li, Yaorui Shi, and Xiang Wang, is in part supported by the National Natural Science Foundation of China (92)7011
REFERENCES
Marah I Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat S. Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Caio César Teodoro Mendes, Weizhu Chen, Vishrav Chaudhary, Parul Chopra, Allie Del Giorno, Gustavo de Rosa, Matthew Dixon, Ro- nen Eldan, Dan Iter, Amit Garg, Abhishek Goswami, Suriya Gunasekar, Emman Haider, Junheng Hao, Russell J. Hewett, Jamie Huynh, Mojan Javaheripi, Xin Jin, Piero Kauffmann, Nikos Karam- patziakis, Dongwoo Kim, Mahoud Khademi, Lev Kurilenko, James Lee, Yin Tat Lee, Yuanzhi Li, Chen Liang. Weishung Liu, Eric Lin. Zeqi Lin, Piyush Madan, Arindam Mitra, Hardik Modi, Anh Nguyen, Brandon Norick, Barun Patra. Daniel Perez-Becker Thomas Portet, Reid Pryzant, Heyang Qin, Marko Radmilac, Corby Rosset, Sambudha Roy. Olatunji Ruwase, Olli Saarikivi, Amin Saied, Adil Salim, Michael Santacroce, Shital Shah, Ning Shang. Hiteshi Sharma, Xia Song. Masahiro Tanaka, Xin Wang, Rachel Ward, Guanhua Wang. Philipp Witte, Michael Wyatt, Can Xu, Jiahang Xu, Sonali Yadav, Fan Yang, Ziyi Yang. Donghan Yu, Chengruidong Zhang, Cyril Zhang, Jianwen Zhang, Li Lyna Zhang, Yi Zhang, Yue Zhang, Yunan Zhang, and Xiren Zhou, Phi-3 technical report: A highly capable language model locally on your phone. CoRR, abs/2404. 14219, 2024.
Bo Adler, Niket Agarwal, Ashwath Aithal, Dong H. Anh, Pallab Bhattacharya, Annika Brundyn, Jared Casper, Bryan Catanzaro, Sharon Clay, Jonathan M. Cohen, Sirshak Das, Ayush Dattagupta, Olivier Delalleau, Leon Derczynski, Yi Dong, Daniel Egert, Ellie Evans, Aleksander Ficek, Denys Fridman, Shaona Ghosh, Boris Ginsburg, Igor Gitman, Tomasz Grzegorzek, Robert Hero, Jining Huang. Vibhu Jawa, Joseph Jennings, Aastha Jhunjhunwala, John Kamalu, Sadaf Khan, Oleksii Kuchaiev, Patrick LeGresley, Hui Li, Jiwei Liu, Zihan Liu, Eileen Long, Ameya Sunil Maha- baleshwarkar, Somshubra Majumdar, James Maki, Miguel Martinez, Maer Rodrigues de Melo, Ivan Moshkov, Deepak Narayanan, Sean Narenthiran, Jesus Navarro, Phong Nguyen, Osvald Nitski, Vahid Noroozi, Guruprasad Nutheti, Christopher Parisien, Jupinder Parmar, Mostofa Pat- wary, Krzysztof Pawelec, Wei Ping, Shrimai Prabhumoye, Rajarshi Roy, Trisha Saar, Vasanth Rao Naik Sabavat, Sanjeev Satheesh, Jane Polak Scowcroft. Jason Sewall, Pavel Shamis, Ger- ald Shen. Mohammad Shoeybi, Dave Sizer, Misha Smelyanskiy, Felipe Soares, Makesh Nar- simhan Sreedhar, Dan Su, Sandeep Subramanian, Shengyang Sun, Shubham Toshniwal, Hao

---

Published as a conference paper at ICLR 2025
Wang, Zhilin Wang, Jiaxuan You, Jiaqi Zeng. Jimmy Zhang, Jing Zhang. Vivienne Zhang, Yian Zhang, and Chen Zhu. N 340b technical report. CoRR, abx/2406.11704, 2024. URL https: //oi:///oiii.// org/10. 4 arxiv. 407041704.
Microsoft Research A14Science and Microsoft Azure Quantum. The impact of large language mod- els on scientific discovery: a preliminary study using GPT-4. CoRR, abs/2311.07361, 2023.
AetMeta. Llama 3 model card. 2024. URL
11ama3/blob/mat http//maamm//mmm////////// Lt///MOL__AAAEEE md.
Lozhkov Anton, Ben Allal Loubna, Werra Leandro, and Thomas. Fineweb-edu, May 2024. URL www.htttps//hhggiiiiee
coadateecccaaaeeeeaoeeeeeee /Huggi htwFaccerr/fieeeeeeeeeeeeee httpe///eeeoooogooaoaoo/a//eeeee
Iz Beltagy, Kyle Lo, and Arman Cohan. Scibert: A pretrained language model for scientific text. In EMNLP/JJCNLP (I), pp.
Association for Computational Linguistics, 2019.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In NeurlPS, 2020.
Hengxing Cai, Xiaochen Cai, Junhan Chang, Sihang Li, Lin Yao, Changxin Wang, Zhifeng Gao, Hongshuai Wang. Yongge Li, Mujie Lin, Shuwen Yang. Jiankun Wang, Yuqi Yin, Yaqi Li, Lin- feng Zhang, and Guolin Ke. Sciassess: Benchmarking LLM proficiency in scientific literature analysis. CoRR, 2024. URL https:://///t org/ / 100 4 118550/arxiio 2403. 01.00.1197 01976.
Daixuan Cheng, Shaohan Huang, and Furu Wei. Adapting large language models via reading com- prehension. CoRR, abs/2200.09530 2023.
Daixuan Cheng. Yuxian Gu, Shaohan Huang, Junyu Bi, Minlie Huang, and Furu Wei. In- struction pre-training: Language models are supervised multitask learners. arXiv preprint arXiv:2406. 14491, 2024.
David Cheng-Han Chiang and Hung-yi Lee. Can large language models be an alternative to human evaluations? In Anna Rogers, Jordan L.. Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa-
pers). ACL 2023, Toronto, Canada, July 9-14, 2023, pp. 15607-15631, Association for Compu- tational Linguisties, 2023. URL https: /ooi org 10 1..0..1..000 www.1116511v1/ acl- aonn 87C.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sehastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Lev- skaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito. David Luan, Hyeontuek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowyez, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Bren- nan Sacta, Mark Diaz. Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstema Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways. JMLR, 2023. URL https://///. / jmlr orgg wtwpppaprraaaaaeeeeeeeeeeeeeeeee
pwppaspppssssae S 224 4 22 1144 html.
Jan Clusmann, Fiona R Kolbinger, Hannah Sophie Muti, Zunamys Carrero. Jan-Niklas Eckardt, Narmin Ghaffari Lalch, Chiara Maria Lavinia Löffler, Sophic-Carolinn Schwarzkopf, Michacla Unger, Gregory P Veldhuizen, al. The future landscape of large language models in medicine. Communications medicine, 3(1):141, 2023.
Together Computer. Redpajama: an open dataset for training large language models, 2023. URL https: wtthhb...ommeeooo httpm/u/.ommmttttteerroooooooooe /ReddP jama taa
12

---

Published as a conference paper at ICLR 2025
Jiaxi Cui, Munan Ning, Zongjian Li, Bohua Chen, Yang Yan, Hao Li, Bin Ling, Yonghong Tian, and Li Yuan. Chatlaw: A multi-agent collaborative legal assistant with knowledge graph enhanced mixture-of-experts large language model, 2024. URL https://arxiv/ org/ http://ggggaaaaa////ee//a abs//bbb///3 1699..
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In NAACL-HILT (1), pp. 4171-4186. As- sociation for Computational Linguistics, 2019.
Ronen Eldan and Yuanzhi Li. Tinystories: How small can language models be and still speak coherent english? CoRR, abs/2305.07759, 2023. URL https://doi. org/o0gg///.ooo /00g//1..0000 48550/ arXiv. 2305. 07759.
Alexander Erdmann, David Joseph Wrisley, Benjamin Allen, Christopher Brown, Sophie Cohen- Bodénès, Micha Elsner, Yukun Feng, Brian Joseph, Béatrice loyeux-Prunel, and Marie- de Marneffe. Practical, efficient, and customizable active learning for named entity recognition in the digital humanities. In NAACL-HLT (1), pp. 2223-2234. Association for Computational Linguistics, 2019.
Yin Fang, Xiaozhuan Liang, Ningyu Zhang, Kangwei Liu, Rui Huang, Zhuo Chen, Xiaohui Fan, and Huajun Chen. Mol-instructions: A large-scale biomolecular instruction dataset for large language models. CoRR, abs/2306.08018, 2023. URL https: /doii org/ 1000010110 48550/
arXiv. 2tt.v233008008.. 08018.
Kehua Feng, Keyan Ding, Weijie Wang, Xiang Zhuang, Zeyuan Wang, Ming Qin, Yu Zhao, Jian- hua Yao, Qiang Zhang, and Huajun Chen. Sciknoweval: Evaluating multi-level scientific knowl- edge of large language models. CoRR, 2024. URL https: h/tas://dooo http::///ooo/g///ooo..otoo 10. 48550/arXix 2406. 09.98
Suriya Gunasekar, Yi Zhang. Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Harkirat Singh Behl, Xin Wang, Sébastien Bubeck, Ronen Eldan, Adam Tauman Kalai, Yin Tat Lee, and Yuanzhi Li. Textbooks are all you need. CoRR, abs/2306. 11644, 2023.
Kshitij Gupta, Benjamin Thérien, Adam Ibrahim, Mats L. Richter, Quentin Anthony, Eugene Belilovsky, Irina Rish, and Timothée Lesort. Continual pre-training of large language models: How to (re)warm your model? CoRR, abs/2308.04014, 2023.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hen- nigan, Eric Noland, Katic Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. Training compute-optimal large language models. CoRR, abs/2203. 15556, 2022.
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Ann Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b. CoRR, abs/2310.06825, 2023. URL attps://doii http:///oo.ogg///////// 48550/85540 arxii 2310. 06825.
Xisen Jin, Dejiao Zhang, Henghui Zhu, Wei Xiao, Shang-Wer Wen Li, Xiaokai Wei, Andrew O. Amold, and Xiang Ren. Lifelong pretraining: Continually adapting language models to emerging corpora. In NAT. pp. 4764-4780 Association for Computational Linguistics, 2022.
Zixuan Ke, Yijia Shao, Haowei Lin, Tatsuya Konishi, Gyuhak Kim, and Bing Liu. Continual pre- training of language models. In ICLR. OpenReview.net, 2023.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In SOSP, pp. 611-626. ACM, 2023.
Sihang Li. Zhiyuan Liu, Yanchen Luo, Xiang Wang, Xiangnan He, Kenji Kawaguchi, Tat-Seng Chua, and Qi Tian. Towards molecule-text interpretation in language models. In ICLR. Open- Review.net, 2024.
13

---

Jiayi Liao, Ruobing Xie, Sihang Li, Xiang Wang, Xingwu Sun, Zhanhui Kang, and Xiangnan Published as a conference paper at ICLR 2025
Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. Textbooks are all you need II: phi-1.5 technical report. CoRR, 0ts/2309.05463,, 2023.
Jiayi Liao, Sihang Li, Zhengyi Yang. Jiancan Wu, Yancheng Yuan, Xiang Wang, and Xiangnan He. Llara: Large language-recommendatie assistant. In Proceedings of the 47th International ACM SIGIR Conference Research and Development in Information Retrieval, pp. 1785-1795, 2024.
He. Patchrec: Multi-grained patching for efficient Ilm-based sequential recommendation. arXiv preprint arXiv:2501. /5087, 2025.
Chen Ling, Xujiang Zhao, Jiaying Lu, Chengyuan Deng, Can Zheng, Junxiang Wang, Tanmoy Chowdhury, Yun Li, Hejie Cui, Xuchao Zhang, Tianjiao Zhao, Amit Panalkar, Wei Cheng. Haoyu Wang, Yanchi Liu, Zhengzhang Chen, Haifeng Chen, Chris White, Quanquan Gu, Carl Yang. and Liang Zhao. Beyond ane-model-fite
survey of domain specialization for large language models. CoRR, abs/2305. 18703, 2023. doi: 10.4850/ARRXIV2205.1188 doi. doi.org//0. 10rg/11..0 45550/ arxiv. 2305. 18703.
URL https://
Zhiyuan Liu, Sihang Li, Yanchen Luo, Hao Fei, Yixin Cao, Kenji Kawaguchi, Xiang Wang, and Tat-Seng Chua. Molca: Molecular Lauph-language modeling with cross-modal projector and uni-modal adapter. In Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pp. 15623-15638, 2023.
Zhiyuan Liu, An Zhang, Hao Fei, Enzhi Zhang, Xiang Wang, Kenji Kawaguchi, and Tat-Seng Chua. Prott3: Protein-to-text generation for text-based protein understanding. ACL (1), pp. 5949-5966. Association for Computational Linguistics, 2024.
Zhiyuan Liu, Yanchen Luo, Han Huang, Enzhi Zhang, Sihang Li, Junfeng Fang, Yaorui Shi, Xiang Wang, Kenji Kawaguchi, and Tat-Seng Chua. NEXT-MOL: 3d diffusion meets 1d language modeling for 3d molecule generation. In The Thirteenth International Conference on Learning Representations, 2025. URL https: http///ppppeeeeeee/ee/e/ee net/ feree//ooummtt wtt.:/oooommmmmmmmte.
Kyle Lo, Lucy Lu Wang. Mark Neumann, Rodney Kinney, and Daniel S. Weld. S2ORC: the seman- tic scholar open research corpus. In ACL, pp. 4969-4983, 2020. URL https: /tttps::/ce //doi.org//////// 10. 1.0.18653//1 v acl ma in. 447.
Yanchen Luo, Junfeng Fang, Sihang Li, Zhiyuan Liu, Jiancan Wu, An Zhang, Wenjie Du, and Xiang Wang. Text-guided small molecule generation via diffusion model. iScience, 27(11), 2024.
Sanket Vaibhav Mehta, Darshan Patil, Sarath Chandar, and Emma Strubell. An empirical investi- gation of the role of pre-training in lifelong learning. J. Mach. Learn. Res., 24:214:1-214:50, 2023.
OpenAl. GPT-4 technical report. CoRR, abs/2303.08774, 2023. URL htRLLttt:///d...gg//// 28550/arXiva 2303.08774.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In NeurlPS, 2022. URL /ttpp:pppppppp...eeeepe nips. wc/pape:pppppeeeeeceeeeeeeeeeeee fwwes/ wwwps//pppeee///222222 / bleffee5bbe 64.73443942 (001011455500000000000
Abst www.cctttoooofoeneceeee html.
Hang Qiu, Krishna Chintalapudi, and Ramesh Govindan. MCAL: minimum cost human-machine active labeling. In ICLR. OpenReview.net, 2023.
Haoran Que, Jiaheng Liu, Ge Zhang, Chenchen Zhang, Xingwei Qu, Yinghao Ma, Feiyu Duan, Zhiqi Bai, Jiakai Wang, Yuanxing Zhang, et al. D-ept law: Domain-specific continual pre-training scaling law for large language models. arXiv preprint 200.22066.11755. 2024.
Team Qwen. Qwen2.5: A party of foundation models, September 2024, URL https: / qwenlm. github. githubio/bbl ht/bll/blloooooooo og/ qww.//wen255... 5/.
14

---

Published as a conference paper at ICLR 2025
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21: 10:14140:00 2020.
Yaorui Shi, Sihang Li, Taiyan Zhang. Xi Fang, Jiankun Wang, Zhiyuan Liu, Guojiang Zhao, Zheng- dan Zhu, Zhifeng Gao, Renxin Zhong, et al. Intelligent system for automated molecular patent infringement assessment. arXiv preprint arXiv:2412.07819, 2024.
Amanpreet Singh, Mike D' Arey, Arman Cohan, Doug Downey, and Sergey Feldman. Scirepeval: A multi-format benchmark for scientific document representations. In EMNLP, pp. 2023. URL https: / org/ 1 10. 210653/v1/222
emnlp-main 338.
Karan Singhal, Shekoofeh Azizi, Tao Tu, S. Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Kumar Tanwani, Heather Cole-Lewis, Stephen Pfohl, Perry Payne, Martin Senevi- ratne, Paul Gamble, Chris Kelly, Nathaneal Schärli, Aakanksha Chowdhery, Philip Andrew Mans- field, Blaise Agtiera y Arcas, Dale R. Webster, Gregory S. Corrado, Yossi Matias, Katherine Chou, Juraj Gottweis, Nenad Tomasev, Yun Liu, Alvin Rajkomar, Joelle K. Barral, Christopher Semturs, Alan Karthikesalingam, and Vivek Natarajan. Large language models encode clinical knowledge. CoRR, abs/2212.13138,, 2022. URL https: /oi. org /00 10.00........ 48550/55444 arxiv. 2212. 13138.
Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Au- thur. Ben Bogin, Khyathi Raghavi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Harsh Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, Ian Magnusson, Ja- cob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Abhi- lasha Ravichander, Kyle Richardson, Zejiang Shen, Emma Strubell, Nishant Subramani, Oyvind Tafjord, Evan Pete Walsh, Luke Zettlemoyer Noah A. Smith, Hannaneh Hajishirzi, Iz Belt- agy, Dirk Groeneveld, Jesse Dodge, and Kyle Lo. Dolma: an open corpus of three tril- lion tokens for language model pretraining research. In ACL, pp. 15725-15788, 2024. URL https:/ //wi. org/ 10. 18 65333/6666611 3/5533//// vll 2024 acl-iong, 840.
Yu Sun, Shuohuan Wang. Yu-Kun Li, Shikun Feng, Hao Tian, Hua Wu, and Haifeng Wang. ERNIE 2.0: A continual pre-training framework for language understanding. In AAAI, pp. AAAI Press, 2020.
Ross Taylor, Marein Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshom, Elvis Sar- avia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. Galactica: A large language model for science. CoRR, abs/2211.0900 1000055, 2022. URL https: http://.....rgg//.//// /0550/arxiiv 2211. 09085.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. CoRR, abs/2302. 13971, 2023.
David Wadden, Kejian Shi, Jacob Morrison, Aakanksha Naik, Shruti Singh, Nitzan Barzilay, Kyle Lo, Tom Hope, Luca Soldaini, Shannon Zejiang Shen, Doug Downey, Hannaneh Hajishirzi, and Arman Cohan. Sciriff: A resource to enhance language model instruction-follooinn over scien- tific literature. CoRR, abs/2406.07835, 2024. URL https: /ttp://///..//////////// 48550/4555500 arxiv.. 2406. 07835.
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj. Xuan He, Ziyan Jiang. Tianle Li, Max Ku, Kai Wang. Alex Zhuang, Ronggi Fan, Xiang Yue, and Wenhu Chen. Mmlu-pro: more robust and challenging multi-task language understanding benchmark, 2024.
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc Le. Finetuned language models are zero-shot learners. arXiv preprint 2021.
15

---

Published as a conference paper at ICLR 2025
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In NeurIPS, 2022.
Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prab- hanjan Kambadur, David S. Rosenberg, and Gideon Mann. Bloomberggpt: A large language model for finance. CoRR, abs/2303. 17564, 2023. doi: 10.48550/ARXIV.230... 175644 URL
https: http://dooo..ggg//// 48550/ a 230.311175550
Tongtong Wu, Massimo Caccia, Zhuang Li, Yuan-Fang Li, Guilin Qi, and Gholamreza Haffari. Pretrained language model in continual learning: A comparative study, In ICLR. OpenReview.net, 2022.
An Yang, Baosong Yang. Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li. Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang. Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma. Jianxin Yang, Jin Xu, Jin- gren Zhou, Jinze Bai. Jinzheng He, Junyang Lin, Kai Dang. Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng. Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wen- bin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Xuejing Liu, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yugiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan. Qwen2 technical report, 2024. URL https: /awxivv.vvvv. org/ abs/ 2407.
Hongyang Yang. Xino-Yang Liu, and Christina Dan Wang. Fingpt: Open-source financial large 48550/
language models. CoRR, abs/2306.06031, 2023. URL https: / //555/4
arXiv. 2.1iv.2306.. 2306.3066020 06031.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. In NeurlPS, 2023.
Mohd Zaki, NM Krishnan, et al. Mascqu: A question answering dataset for investigating materials science knowledge of large language models. arXiv preprint arXiv:2308. 09115, 2023.
Zheni Zeng. Yuan Yao, Zhiyuan Liu, and Maosong Sun. A deep-learning system bridging molecule structure and biomedical text with comprehension comparable to human professionals. Nature communications, 13(862), 2022.
Dan Zhang. Ziniu Hu, Sining Zhoubian, Zhengxiao Du, Kaiyu Yang. Zihan Wang. Yisong Yue, Yuxiao Dong, and Jie Tang. Sciglm: Training scientific language models with self-reflective instruction annotation and tuning. CORR, 1as24010079500 2024a. URL https: //ds://////.eee/ /ttg//r/////ooo////////ooo/oo/
10. 18.48550/ aarvvv...vvvvv////// 2401.01 07950.
Di Zhang, Wei Liu, Qian Tan, Jingdan Chen, Hang Yan, Yuliang Yan, Jiatong Li, Weiran Huang, Xiangyu Yue, Dongzhan Zhou, Shufei Zhang, Mao Su, Hansen Zhong, Yuqiang Li, and Wanli Ouyang. Chemllm: chemical large language model. CORR, abs/2402.06852, 2024b. URL
1tl.r///.......01 48550/ arxiv. 2402 06852.
Xingjian Zhang, Yutong Xie, Jin Huang. Jinge Ma, Zhaoying Pan, Qijia Liu, Ziyang Xiong, Tolga Ergen, Dongsub Shim, Honglak Lee, and Qiaozhu Mei. MASSW: A new dataset and benchmark tasks for ai-assisted scientific workflows. CoRR, abs/2406.06637,, 2024c. URL httptt:: //doi.
org/10. 4884885555 335555/axxii 2406. 06357.
Yizhen Zheng, Huan Yee Koh, Jiaxin Ju, Anh T. N. Nguyen, Lauren T. May, Geoffrey I. Webb, and Shirui Pan. Large language models for scientific synthesis, inference and explanation. CoRR, abs/2310.07984, 2023.
16

---

Published as a conference paper at ICLR 2025
A RELATED WORKS
A.1 KNOWLEDGE INJECTION VIA CONTINUAL PRE-TRAINING
Pre-train Pre-training language model is usually conducted on large corpus of textual data to learn the statistical properties of language (Radford et al., 2019; Brown et al., 2020; Lo et al., 2020; Soldaini al., 2024). To further inject domain knowledge into a general L.LM after pre-training, researchers engage in continual pre-training (CPT) on high-quality domain-specific corpora (Jin et al., 2022; Sun et al., 2020), sometimes combined with general corpora. This process enhances the model's fundamental understanding abilities in specific downstream domains while mitigating catastrophic forgetting of general knowledge (Wu et al., 2022; Ke et al., 2023; Mehta et al., 2023). See the com- prehensive study (Gupta et al., 2023) for different warm-up strategies for CPT. Additionally, the CPT corpora can be augmented by transforming them into an instruction-respone format (Cheng et al., 2023; 2024). Furthermore, the scaling law (Hoffmann et al., 2022) of domain-specifie CPT (Que et al., 2024) is explored to determine the optimal mix of data. However, these studies primarily fo- cus training dynamics and data recipes, leaving the pre- processing for scientific data, especially raw PDF files, largely unexplored. Exhibiting such steps is essential for generating high-quality do- main corpora and effectively injecting domain knowledge, representing significant challenge for practitioners.
A.2 DOMAIN ADAPTATION VIA SUPERVISED FINE-TUNING
Supervised fine-tuning (SFT) modifies a pre-trained language model to follow specific instructions or perform designated tasks by fine-tuning it on targeted, task-specific dataset (Raffel et al., 2020; Wei et al., 2021; Liu et al., 2023; Li et al., 2024; Liu et al., 2024; Liao et al., 2024; 2025). Apply- ing SFT to general LLM for specific domain adaptation has demonstrated effectiveness in various fields: in medicine (Clusmann et al., 2023), corpora of medical literature and clinical notes are used; in law (Cui et al., 2024), legal documents and case law are compiled; and in finance (Wu et al., 2023), financial reports and market data are utilized. In the scientific domain, several stud- ies have specialized LLMs for scientific tasks, often necessitating the construction of a substantial domain-specific dataset with SFT. For example, SciGLM (Zhang et al., 2024a) leverages existing LLMs to generate step-by-step reasoning for unlabelled scientific instructions. C. (Zhang et al., 2024b), a more specified LLM in the chemistry field, collects structured chemical data from a vast selection of online databases and transforms this structured data into a question-answeriigg format for SFT. SciRIFF (Wadden et al., 2024) converts existing literature understanding datasets into natural language input-output pairs suitable for cotuuctiontttnnnng using pre-defined templates. However, benchmark studies (Feng et al., 2024; Cai et al., 2024) indicate that SFT alone may not provide adequate scientific knowledge to excel in relevant tasks. This suggests the need for a more comprehensive approach that combines domain knowledge infusion with follooi en- hancements.
A.3 L.MSS FOR SCIENTIFIC LITERATURE UNDERSTANDING
In the scientific domain (Luo et al., 2024; Liu et al., 2025; Shi et al., 2024), existing strategies for de- veloping specialized LLMs mostly fall into two categories: (1) Supervised fine-tuning with scientific instructions. This approach requires a large, high-quality, and diverse set of instructions to cultivate problem-solving abilities for scientific tasks. Representative works (e.g. SciGLM (Zhang et al., 2024a), ChemLLM (Zhang et al., 2024b), and SciRIFF (Wadden et al., 2024)) have been detailed in Section A.2. (2) Pre-training with scientific corpora. This approach involves pre-training on large corpus of scientific texts to improve performance on downstream scientific tasks. Early attempts, such as SciBert (Beltagy et al., 2019) and KV-PLV (Zeng et al., 2022), are based on BERT (Devlin et al., 2019) and pre-trained on a large corpus of scientific text for downstream scientific task en- hancement. More recently, Galactica (Taylor et al., 2022) is pre-trained on a vast corpus of scientific literature, including research papers, scientific articles, and other relevant scientific texts. Despite these advances, two major limitations hinder these models from excelling in scientific literature un- derstanding: (1) lack of scientific knowledge, and (2) inability to follow task instructions. To address these challenges, we propose a combined pipeline of CPT and SFT to devise a specialized LLM for scientific literature understanding. It injects domain-specifie knowledge through CPT while enhanc-
17

---

Published as a conference paper at ICLR 2025
ing task-specific ioslluction-foloowingg abilities through SFT, leading to capable LLM for scientific literature understanding.
B FORMAT & GRAMMAR CORRECTION EXAMPLES
Raw text parsed by PyPDF2
Highly entrrati a radiation, such as or fäst electorns, deposits
throghout the solid 1 g material. Gas production occurs h the solid phase and must di f to the surface to observed. The
yield II can depend on the rddollsis
gas remains the solid. Experiments have that the apparent e of H can vary by afccto o of in the r dadio o y s of polyethylene spheres
e the particle size if
2100 cm22g (about understood in the bubble formation, which could a e product quality.
mmn) [12],
gas trapping diffission sterage. Extremely h h dese rates in the
of
of certain materials may lead
The yie 2 in a d o ooly s is polymers w it ry-rys is well known for several types p mers
Format Grammar Correction Prompt
extracted following
extraction process introduced many formalting isses unnecessary
line breaks, extra spaces, and other artifacts that disrupt the text flow. Could you plense help me correct these formatting issues and provide clean, readable version of the text? Respond with the Corected Version only.
Raw Text:
(RawText)
Start your response with the corrected version of the
Text after Format & Grammar Correction
Highly penetrating radiation, sach y-rays or fast electrons, deposits energy throughout the solid target material. Gas production occurs within solid phase and must the particle size if some the gas remains in the solid. Experiments have shown that the apparent yield of H2 can vary factor of 3 in the rudiolysis of polyethy spheres understord the context which could alter product quality. The yield H2 in the radiolysis of polymers with 7-eays known for several types of polymers [2].
to the surface observed. apparent yield of H2 can depend on radiolysis procedure
2100 cm2/g (about 0.03 mmn) [12], The effects gas trapping and diffusion are not storage. Extremely high dose rates processing certain materials lead to bubble formation.
C CPT QUALITY FILTER
We randomly select 50k samples from our CPT data. These selected samples are then scored using the 1amma3-70B-Inssie model. The prompt utilized for this scoring process is follows:
18

---

Published as a conference paper at ICLR 2025
Filtering Threshold
MMLU-Pro-Blo
MMLU-Pro-Chem
MM-UUPrro-Hall
MaScQA
Avg.
15%
70.59
50.32
53.14
53.14
58.32
25%
70.45
51.21
$4.06
54.06
58.75
35%
69.82
49.52
51.64
61.64
57.67
Table 7: 7B base model performance (accuracy %) with different CPT data filtering thresholds.
Prompt for CPT Data Quality Labelling
Below is an extract from textbook. Evaluste whether the text has a high educational value and could be uscful in an educational setting teaching from primary school grade school levels using additive 5-point scoring system described below. Points are accumulated based on the satisfaction of each criterion:
Add I point the extract provides academic content like advertisements and promotioona material.
basic information relevant to educational lopics.
includes irvelevant
Add another point the extract addresses certain elements pertinent to education but does align closely with educational standards, It might mix oducational content with non-oducational material, offering a superficial overview of poteutially useful topics, or presenting information disorganized manner and incoherent writing style.
Award a third point if the extract is appropriate for educational use and introduces key concepts relevant to school curricula. is coherent though it may not be comprebensive or could include some extrancous information. It may resemble an introductory section of a textbook basic tutorial that suitable for leuming but has notable limilations like treating concepts that are loo complex for graie school students.
Grant fourth point if the extract highly relevant and beneficial for educational purposes for level not higher than grade school, exhibiting clear and cousistent writing style. It could be similar to a chapter from textbook or a tutorial, offering substantial educational content, including exercises and solutions, with minimal imelevant information, and the concepts aren't too advanced for grade school students. The content is coherent, focused, and valuahle for structured learning.
Bestow a fifth point the extract is outstanding in its educational value, and perfectly suited for teaching either at primary school or grade school. It follows detailed reasoning, the writing style easy follow, and offers profound and thorough insights into the subject matter, devoid of any con-educational or complex content.
After examining the extract:
Briefly justify your score, up Conclude with the score using the format: "Educational score:: itotal points;""
words.
We train a Scientific Texts Quality Classifier on these labeled data samples. The classifier is a 109M BERT (Devlin et al., 2019) classifier, fine-tuned from the checkpoint of finewebeduucllassffer (An- ton et al., 2024). The model is trained for 20 epochs with a learning rate of 0.001 and a batch size of 1024. Ninety percent of the 50K samples are used as the training set, and the rest 10% are used as the validation set. The training process costs approximately 50 minutes on 4 A100 GPUs. We select the checkpoint from the epoch that yields the highest validation micro Fl score as final checkpoint. During Inference, we set batch size to 2048, and beam number to 1. The inference process costs 90 minutes on 4 A100 GPUs. We utilize the generated to filter out 25% data with the lowest quality.
We also conduct empirical experiments investigate the sensitivity of model performance to the quality filtering threshold. The results in Table show that setting the filtering threshold at 25% achieves the best overall performance, effectively injecting scientific domain knowledge into the model. When the threshold is set too high (c.g.. 35%). the average quality of the corpus improves; however, the significant reduction in data volume limits the ability to fully capture domain knowl- edge. Conversely, when the threshold is set too low (e.g., 15%), the increase in data volume in- cludes more low-quality content, which negatively impacts model performance. Thus, the trade-off between data quality and quantity is crucial.
D SET DETAILS
D.1 INSTRUCTION GENERATION PIPELINE
In SciLitIns, we focus on generating instructions for three less-represented domains (materials sci- ence, medicine, and drug discovery) and five question types:
Table Extraction: Table Extraction tasks evaluate model's proficiency in extracting, summariz- ing, and structuring data from an article into table format.
Entity Extraction: Entity Extraction tasks are designed evaluate a model's ability extract specific information, such as entities or relationships, from the text.
Molecule Translation: Molecule Translation tasks evaluate a model's ability to translate molecules between different SMILES formats.
19

---

Published as a conference paper at ICLR 2025
Molecule Extraction: Molecule Extraction tasks ask model extract appropriate molecule from a scientific paragraph that contains multiple molecules.
Multiple Choice and True-or-False: Multiple Choice and True-or-False questions assess a model's ability to select the correct answer from set of options, testing its knowledge and rea- soning on both simple and complex scenarios.
For each the three scientific domains, we collect a set of high-impact research papers and construct word frequency table. To generate question in a given domain, sample 20 keywords from the corresponding word table and insert them into the prompt for that question. To ensure fair representations of less frequent keywords, We will release our code, prompt templates, and word frequency tables. Below is an example of generating table extraction question:
random sampling with temperature setting of
Prompt for Generating Table Extraction Question
training machine learning model extmacts from correctly. in JSON, with each catry containing "text" and "answer" attributes. You should gencrate paragraph that includes the kcywords: { (Keys }-
synthetic training
formatted
The "lext" purt contain enough information the table extracted! In "lext" part, You you include table description latex format.
Special
table content:
You should generate a table that has complicated numbers and characters, include non-standard characters, and have variety of values. Make sure the value you generated not follow simple patterns, example, never inclade deplicate values or values with constant interval columns.
Your unswer should contain as much details as possible. You should only generate one JSON, The value the two attributes should string. Use {f and }} warp your output. attention the escape characters the latex format. Remember put the end of the first string. Never use jaon block to wrap your outpur. Here is the format for your output:
"text": "Your paragraph here, remember to include table in latex format".
"answer": "Your answer table bere"
Now start your answer:
D.2 INSTRUCTION DEDUPLICATION
The generated synthetic data may contain similar questions identical answers. To eliminate re- dundancy, we implement fuzzy deduplication process using the Levenshtein distance to calculate the similarity score between question-ansee answer pairs. Specifically, for two pairs (Q..a)) and (92. 42), their textual similarity is defined as (1 lev(q1. 92))(1 lev(a1, 02)). where lev(..-) denotes the Levenshtein distance. Due to significant differences between texts from different question types, we compute similarity matrices separately for each type. We then use a disjoint-set data structure to merge highly similar data points. We use this process to remove approximately 5% 10% of duplicated data for each question type.
D.3 QUALITY ASSESSMENT GENERATED SFT INSTRUCTIONS
In section 2.2.2, we sample 10k instruction pairs from SciLitIns and evaluate them by Llama-3-70B using the below prompt. Specifically, the quality is evaluated from five aspects: clarity, complexity, correctness, usefulness, and adaptability, assigning each instruction a score from (lowest quality) to (highest quality).
20

---

Published as a conference paper at ICLR 2025
SFT Evaluation Prompt
You helpful and precise assistant for checking the quality instruction tuning data for large language models. Your task is evaluatte
the given instruction using the criterions described below.
Clarity: The sample should clear, specific, unambiguous, providing well-defined for model perform.
Complexity: The sample should be advanced complexity that necessitate a high level of comprehemsion and cognitive processing, chal- significanll. Correctness: The sample impeccably written, with flawless grammar, syntax, and stracture, demonstrating exceptional clarity and professionalism.
lenging the language
Usefulness; The sample should he highly usefal, and contribute expanding the model's knowledge hase.
Adaptability: The sample could be adapted to different contexts or use cases, showing some flexibility. examining the instruction-aespee response pair:
Brielly justify your scores with paragruph in the field "Explanation", to 500 words.
For each point of criterion above, assign
You should only provide the rest of your answer in structured format shown below, and make sure response be directly
parsed by computer programs.
Below template for your response:
Explanation: (string. your explanations to the scores;
"Clarity": <int, complexity score >, "Complexity": <inn, complexity scorc>> "Comectness": <int, quality score >.
"Usefulness" <int, usefulness
"Adaptability": <int, adaptability score
"Total": <ini, score>
Below an example SciLitIns, which will be sent Llama-3-70B for evaluation.
An Example in SciLitins
Confext
In studies regarding material science, ermatrix shown significant importance understanding natively, many researchers have proposed series of experiments analyze these phenomena. Even though the process is delicate, the cutbusiasm for discovering new findings understandable. The following Table presents schematic illustration the euthalpy changes observed during the different fracture plases. These studies indispensable for advancing our knowledge the material's resilience. In addition. in ocrtain cases, pronouneed crors in measurement can occur, but the references provided can guide through the correct methodology. among tools
hehaviors. Alter-
differential scanning calorimetry which helps measure these enhhlpy changes accurately.
Abegtn/tablej [ht] intoentee ing Abegi (tabclar) (lateicic)) Anth1 InMentreriall Tnitial Enthalpy (kJ/nol) Pinal Enthalpy (kJ/mol) Fracture Pheae MinntliinekMMs.riia A 25.4 47.8 s Brittle 28.9 9 50.2 & Semi-brittlo
intetterrree B 22.3 & 45.0 Ductile
WiNbhiine wwlnhbiinelaaeeddrenn. (atend An canccaptionn (Entkalpy changes during frecture
phases) www.llablllii (talleeeaaeeeenhaalagp (thend(tabla))
The process of accurately measuring these values is crncial for the material selection in various engineering applications, making it indispeusable research area.
Instruction
table paragraph.
Response
Material, Initial Enthalpy (KT/ool), Final Enthalpy (KI/mol),Fractu Fracture Phase
Material A.25 4.47.8 8. Britttle
Material 2..55.....,..mm..
Material C,28.9,50.,S Sc.ibbbbbiii
Below is the evaluation by Llama-3-70B on the example from SeiLitIns. Each criterion is scored out of points, making the total possible points 25, The total score then averaged, resulting in an overall score out of 5.
21

---

Published as a conference paper at ICLR 2025
Evaluation by Llama-3-70B
Explanation:
The instruction clear and specific, asking the model to extract table from given paragraph. The paragraph itself well-written, with proper grammar syntax, making casy understand. The table well-structured and provides valuable information about vnihalpy changes during fracture pluses. The lask requires good undersianding of the text and the ability to identily the table. However, particularly
complexity the relatively low, only involves extracting table, The task is useful for advancing knowledge in material science, and the table could be adopted to different contexts use cases.
ambiguous.
"Clarity":
"Complexity" 2.
"Correttness" "
"Usefulness": 4,
"Adaptability": "Tocal": 20
E BENCHMARK DETAILS
To the best of our knowledge, there are two commonly-adopted datasets for scientific literature understanding:
SciRIFF (Wadden et al., 2024) evaluates essential scientific literature understanding capabilities, including information extraction, summarization, question answering, claim verification, and clas- sification. Data points SciRIFF are notable for their long input contexts and complicated struc- tured outputs. The Qasper and SciFact tasks have two different evaluation methods and thus two results. For MuP and one evaluation method of Qasper, we use GPTT-4o as language model evalu- ator, while the original paper uses GPT-3.5. We note that SciRIFF contains a separate training used in the SFT stage in our study.
SciAssess (Cai al., 2024) features an end-to-end benchmark of understanding PDF content. It includes 28 tasks from four scientific domains: biology, chemistry, materials and medicine. SciAssess is used exclusively for testing in our evaluation.
Overall, SciRIFF provides basic benchmarks for comprehending short scientific segments and var- ious instructions, while SciAssess presents more challenging tasks involving longer contexts from raw PDFs.
F DETAILED PERFORMANCE ON SCIASSESS
The detailed results on each task in SciAssess are shown in Table 8.
43.
63.4
01.5
65.
35.4
Mrndm
41.4
54.6
56.9
10.2
12.2
81.5
195
66.0
UannAA
38.2
-0.7
45-7
0222
48.2
51.8
02.2
48.7
16.3
11.2
25.3
Biclgy
ChenReeng
51.
73.7
66.7
86.5
$0.1
83.6
17.8
Dackesy
$1.4
40.6
40.7
45.8
$6.6
36.9
46.4
47.4
41.5
45.2
20.1
41.0
F10
3430
45.1
+2.5
5000
12.5
51.3
15.4
37.5
1500
Matafro
13.8
17.9
405
45.2
$2.5
65.7
07.6
72.3
65.2
30.3
19.6
91.5
61.5
10.
Chemisty
Pay/n
ChanOA
25.0
33.5
56.7
59.1
76.9
88.
90.9
852
91.4
91.3
0
93.5
Polybber
7/4
35.8
43.1
12.6
66.6
51.
56.2
13.0
56
15.
Solloar
51.5
3132
40.
41.3
41.3
MadiQA
18.
54.5
05.5
500
25.7
10
Averaz
31.5
509
43.9
53.7
$1.
99.2
96.4
62.
$2.0
MarAA
5699
430
197
96.
556
66.1
718
71.
T11
40.1
76.1
51.
66.7
715
40.0
CuspEarr
29.5
Sumptiir
16.3
52.7
56.6
66.5
649
66.
909
614
61.
M5.5
66.
45%
1944
Averag
100
25.1
3022
26.5
35.4
31.3
37A
31.3
31.8
43.3
Atmurro
21.3
39.4
51.7
62.0
23.1
Modicie
ChunOA
53.5
367
35.3
lightel
12
0.2
7.1
80
6.2
14.
7.3
2.3
Marktte
59
20.2
35.2
208
315
25.8
425
44.3
40.
52.5
58.5
510
900
540
65.0
Table Detailed model performance on SciAssess tasks.
22

---

Published as a conference paper at ICLR 2025
Eval Datasets
In-house Texthooks (%)
In-house Journals (%)
SciRIFF
1.1
0.7
SciAssess
11.0
1.1
Table 9; Contamination rates in textbook and journal datasets.
Category
Contamination Rates (%)
SciL.it.LMM-B
SciL.tt.LM-144
Full Set (%) Clean Set (%) Full Set (%) Clean Set (%)
Biology
12.2
65.9
66.3
67.5
67.6
Chemistry
13.3
54.4
55.8
64.2
65.2
Material
12.5
52.5
50.7
61.8
62.9
Medicine
8.7
33.6
34.2
42.4
43.9
Mean
12.0
51.6
51.6
59.0
59.8
Table 10: Performance comparison of Scil.itLL.M on full and clean evaluation sets of SciAssess.
G CONTAMINATION STUDY
To investigate data contamination in our datasets and its potential impact on downstream perfor- mance, we conducted contamination analysis this section. Prior studies consistently indicate that data contamination in pre-training datasets has negligible effects on evaluation performance:
The GPT-4 technical report states: Contamination overall has very little effect on the reported results (OpenAI, 2023). For instance, tasks such as AP US History (73% contamination rate), AP World History (47% contamination rate), and LSAT (39% contamination rate) show negligible differences in performance between the full evaluation set and a clean evaluation set with all contaminated data points removed.
PaLM analyzed datasets with contamination rates ranging from 20% 75% and concluded that data contamination does not cause meaningful inflation of our reported results (Chowdhery et al., 2023).
For our contamination analysis, we adopt the contamination detection method described in the GPT- 4 technical report: For each evaluation instance, randomly extract three substrings of 50 char- acters each. match is determined if any of the three sampled substrings appears as a part of the processed training example.
The contamination rates for our textbook and journal datasets are summarized Table Our analy- sis reveals that contamination rates very low for SciRIFF but higher for SciAssess, particularly in the textbook dataset. To evaluate whether contamination impacts model performance, we assessed SciLitLLM SciAssess using both the full evaluation set and clean evaluation set (with all con- taminated examples removed). As shown in Table 10, the differences in performance between the full and clean evaluation sets are minimal, consistent with prior findings. These results reinforce the conclusion that data contamination does not significantly affect the performance of SciLitLLMM
H HUMAN EVALUATION OF DATASET QUALITY
To further validate the effectiveness of the proposed framework creating the CPT and SFT datasets, conducted human evaluation study. This study involved sampling random subsets of unfiltered CPT and SFT data and comparing human annotations to the scores generated by our quality filters.
For the CPT dataset, we sampled 50 entries from the textbook and journal datasets. For the SFT dataset, sampled 50 entries from SciLitIns. Each sampled entry was independently evaluated by four annotators using the same prompts provided to the LLMs. To assess the alignment between human annotations and the quality filter, computed measures:
23

---

Published as a conference paper at ICLR 2025
Human-Human Agreement: Calculated as the average Spearman correlation coefficient across all pairs of annotators.
Humann-Quality Filter Agreement: Calculated as the average Spearman correlation between each annotator's scores and the quality filter's score.
higher Spearman correlation indicates a stronger agreement between the compared sets of scores. The evaluation results are summarized in Table 11. The results show that for both datasets, the Human- Human-Quality Filter Agreement is comparable to, or even exceeds, the Human-Hummn Agrec- ment. This indicates that the quality filter closely aligns with human annotations, demonstrating its reliability in capturing quality as perceived by humans.
Agreement
Human-Human Human-Qualij Filter
CPT
0.58
0.76
SFT
0.89
0.88
Table 11: Comparison of Human- Human Agreement and Human Humman-Quality Filter Agreement for CPT and SFT datasets.
24


Reviews:
[{"review_id": "5EIeOPWDKB", "reviewer": "Reviewer_TiCd", "summary": "This paper introduces a method to improve scientific instruction following through post-training Qwen models. The main contribution is to collect science textbook data and improved SFT data mix. Their evaluation on a recent SciRiff dataset shows improvement.", "questions": "Are you planning to release the textbook datasets? \nCan you elaborate on the pipeline on what type of textbooks impact and improve task performance in SciRiff? \nCan you provide contamination studies in textbook datasets and the test/eval cases?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The paper introduces a strong pipeline in collecting textbook data and SFT data. This is aligned with most recent LM papers, showcasing the importance of data in the success of LM training. \n- Improved result in science instruction following", "weaknesses": "The authors don't seem to be planning to release their textbook dataset. This raises the question of data contamination in evaluating the proposed model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fic34co5v1", "reviewer": "Reviewer_7iRT", "summary": "This paper proposes a synthetic data generation pipeline for training\nLMs for scientific literature understanding tasks. The main\ncontributions include:\n\n1.  a pipeline for curating continued pre-training corpus based on\n    textbooks and research papers (including document parsing,\n    formatting, and filtering),\n\n2.  another pipeline for creating instruction fine-tuning data for\n    scientific literature understanding tasks via prompting GPT-4o.\n\n3.  the authors show that finetuning Qwen-2.5 model (7B and 14B) on the\n    CPT and SFT data can improve the performance on scientific\n    literature understanding tasks.", "questions": "1.  Are there special reasons why we should group the results based on\n    the 10B model size? (table 2/3). I think it's more reasonable to\n    organize based on pre-trained only/with instruction tuning/with\n    domain specific tuning?\n\n2.  Also in this paper there is only fine-tuning results on the Qwen\n    model family but not others. It would be interesting to compare the\n    fine-tuning effects on llama or other model families.\n\n    1.  the author compared the performance with the SciTulu model,\n        which is trained based on Llama-2 families. I don't think it's a\n        fair comparison in table 3.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1.  Overall the paper is nicely written and the pipeline is nicely\n    presented.\n\n2.  The curated dataset and models could be helpful.", "weaknesses": "1.  The primary contribution of the paper seems to be focused on the\n    dataset construction; overall the method resembles similar works for\n    synthetic data generation and there are some limitations:\n\n    1.  The PDF processing pipeline can be improved. Scientific PDF are\n        known to contain complex layout and structures, and previous\n        work have identified that using simple PDF parsers can lead to\n        suboptimal training results (S2ORC, Lo and Wang, and ). However,\n        the author primarily uses a simple PyPDF parser (\\\"Converting\n        these documents using tools like PyPDF2 often introduces\n        formatting and syntax errors, which degrade the quality of the\n        corpus (line 246)\\\"). I'd suggest the authors investigate the\n        text quality issues and check other libraries like papermage, Lo\n        et al.\n\n2.  The results are not very strong. I'd imagine a domain-specifically\n    distilled model can have a substance gain in performance compared to\n    GPT-4o, especially the instruction fine-tuning dataset is generated\n    via GPT-4o (line 331); however, as shown in table 3, the trained\n    models (SciLitLLM-7B and SciLitLLM-14B) are on par with GPT-4o. Also\n    the experimental design and presentation could be improved (see my\n    suggestions in questions).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QnfcgmlRBX", "reviewer": "Reviewer_W9HG", "summary": "The paper proposes a new strategy to improve LLM for scientific literature understanding, which includes continual pretraining and supervised fine-tuning. The paper uses Llama3-8B to correct parsing errors and filters the dataset with Llama3-70B. The paper continues pretraining the Qwen2.5.The paper designs a three-step pipeline to generate diverse scientific contexts and corresponding QA pairs. The paper then incorporates heuristic deduplication and LLM-based filtering for the instructions. The proposed framework seems to improve the performance compared to SciRIFF. The paper also includes an ablation study.", "questions": "What is the difference between the new CPT dataset and the Dolma Dataset (https://allenai.github.io/dolma/)?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper proposes a new framework that includes continual pretraining and supervised fine-tuning. The proposed framework and dataset can be very useful for other LLMs specialized in scientific understanding. The released dataset seems to be very comprehensive compared to the existing dataset.\n2. The paper shows that with the new framework, the paper can further improve the performance of general LLM. The framework is especially useful in small models. Additionally, the paper includes an ablation study for CPT, SFT, and instruction quality filters. \n3. The paper provides the code and its model. In the appendix, the paper shows the improvement of format & grammar correction, CPT quality filter, SFT details, benchmark details, and detailed performance on SciAssess.", "weaknesses": "1. The creation of the CPT and SFT datasets seems to rely on LLMs. The paper can randomly sample a small subset of the created dataset to check its quality with humans to show the effectiveness of the proposed framework.\n2. The experiment and ablation study is not comprehensive. The paper fails to show that the model is fine-tuned to other existing scientific understanding datasets, such as the Dolma dataset (https://allenai.github.io/dolma/). The paper did include SciTulu-7B dataset; however, scitulu is based on LLama2-tb. The paper should also show the model's performance only finetuned with SciLitIns. The analysis in the experiment is also rather simple. The paper needs to provide some explanation instead of just repeating the results in the table.\n3. Some details are not very clear. What is the score used in Tables 2 and 3? Many additional evaluation results and analysis are put in the Appendix. Authors should move some of them to the main paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dlYKDe9nBX", "reviewer": "Reviewer_fcV1", "summary": "The paper introduces SciLitLLM, a specialized language model for scientific literature understanding, built using a hybrid approach combining continual pre-training (CPT) and supervised fine-tuning (SFT). The key contributions include 1) A pipeline that combines CPT with high-quality scientific corpora and SFT with diverse scientific instructions, 2) Novel methods for improving scientific text quality and generating domain-specific instructions, and 3)  Empirical results showing improved performance on scientific understanding benchmarks SciRIFF and SciAssess", "questions": "1. How sensitive is the model performance to the quality filtering threshold (currently set at 25%)? Was this choice empirically validated?\n\n2. The instruction synthesis method uses GPT-4 - have you explored using smaller models or your own models in a bootstrapping approach?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "### Well-Motivated Approach\n- The hybrid CPT+SFT strategy effectively addresses both domain knowledge and task-specific capabilities.\n- The pipeline is well-designed with clear motivation for each component.\n- The approach is generalizable to other specialized domains\n\n### Technical Contributions\n>Pipeline includes innovative components like LLM-based format correction (Section 3.1.1) and quality filtering (Section 3.1.2)\nThe instruction synthesis method (Section 3.2.1) is clever and tackles the challenge of limited scientific instruction data.\n\n### Solid empirical results\n- SciLitLLM-7B outperforms similar-sized models by significant margins \\~4% on SciAssess, +\\~10% on SciRIFF.\n- SciLitLLM-14B surpasses larger proprietary instruction tuned models (70B+ parameters).", "weaknesses": "### Weaknesses\nThe CPT corpus (12.7B tokens) is relatively small compared to standard pre-training datasets The paper acknowledges this limitation but could discuss potential impacts more thoroughly. For example, how this affect the representation from different scientific subject domains.\n\nOtherwise I don't see clear weakness of paper of such kind. The paper appears comprehensive and well-executed for the research scope.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)