Curriculum Model Merging: Harmonizing Chemical LLMs for Enhanced Cross-Task Generalization
Authors: Baoyi He
Luotian Yuan
Ying Wei
Fei Wu
Venue: NeurIPS 2025 poster ()
Keywords: LLM
model merging
chemical
Primary area: machine_learning_for_sciences

Abstract: The emergence of large language models (LLMs) has opened new opportunities for AI-driven chemical problem solving. However, existing chemical LLMs are typically tailored to specific task formats or narrow domains, limiting their capacity to integrate knowledge and generalize across tasks. Model merging offers a promising route for efficiently combining specialized LLMs into a unified model without access to original training data, which is urgently needed in the chemical domain where in-house data and privacy preservation are critical. However, effective model merging in the chemical domain poses unique challenges: (1) significant disparities among chemical LLMs due to task-specific specialization, and (2) a highly imbalanced distribution of chemical LLMs in targeted downstream tasks, where some are over-benchmarked while others remain underexplored. These challenges intensify model inconsistencies such as parameter interference and accumulated fine-tuning noise, which collectively hinder effective model merging. To this end, we propose Curriculum Model Merging (CMM), a curriculum-based framework that progressively merges expert chemical LLMs in a moderate and continual manner. CMM aims to harmonize their inconsistencies while meantime preserve their domain-specific expertise. Comprehensive experiments on two benchmark datasets show that CMM effectively consolidates task-specific expertise and outperforms the state-of-the-art methods by 29.03\% in terms of overall average performance. Moreover, CMM facilitates chemical knowledge generalization across prediction and generative tasks without sacrificing robustness, exhibiting promising merging performance under both expert-abundant and expert-sparse scenarios.

Curriculum Model Merging: Harmonizing Chemical LLMs for Enhanced Pross-Tash Generalization
Baoyi He Zhejiang University 123210370zju. edu
Luotian Yuan" Zhejiang University 122212400zju. edu. cn
Ying Weit Zhejiang University ying. wei weiszju. edu. cn
Fei Wut
Zhejiang University wufei@zju. edu. cn
Abstract
The emergence of large language models (LLMs) has opened new opportunities for Al-driven chemical problem solving. However, existing chemical LLMs are typically tailored specific task formats or narrow domains, limiting their capacity to integrate knowledge and generalize across tasks. Model merging offers a promis- ing route for efficiently combining specialized LLMs into a unified model without access to original training data, which is urgently needed in the chemical domain where in-house data and privacy preservation are critical. However, effective model merging in the chemical domain poses unique challenges: (1) significant dispar- ities among chemical LLMs due to task-specific specialization, and (2) a highly imbalanced distribution of chemical LLMs in targeted downstream tasks, where some are over-benchin wer-benchmarked while others remain underexplored. These challenges intensify model inconsistencies such as parameter interference and accumulated fine-tuning noise, which collectively hinder effective model merging. To this end, we propose Curriculum Model Merging (CMM), curriculum-batt basedd framework that progressively merges expert chemical LLMs in a moderate and continual man- CMM aims harmonize their inconsistencies while meantime preserve their domain-specific expertise. Comprehensive experiments on two benchmark datasets show that CMM effectively consolidates task-specific expertise and outperforms the state-of-the-ar methods by 29.03% in terms of overall average performance. Moreover, CMM facilitates chemical knowledge generalization across prediction and generative tasks without sacrificing robustness, exhibiting promising merging performance under both expert-abundant and expert-sparse scenarios.
Introduction
The emergence of large language models (LLMs)) has profoundly reshaped the landscape of Chem- istry [9], demonstrating remarkable effectiveness across wide spectrum of real- world problems (e.g., property prediction [49). molecular generation 47), and retrosynthesis (15)). A prevailing strategy adapt foundation LLMs for chemical research is task-specific fine-tuning, which enhances downstream performance on specialized datasets. However, this paradigm inevitably fragments chemical intelligence, as each fine-tuned model becomes confined to narrow domains or data formats, limiting holistic knowledge integration and cross-tass generalization. Multi Multi-task learning partially alleviates the issues by training universal model multiple chemical datasets. Yet, its applicability
"quual contribution. Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

in chemistry domains remains constrained by the scarcity of publicly shareable data under in in-house data privacy regulations and the high computation cost required for large-scale joint optimization [62]
Recently, model merging has emerged as a compelling alternative for integrating multiple expert models into comprehensive base model without requiring access to the original training data or extensive retraining, an approach that has already achieved considerable success in natural language processing (NLP) [57]. However, transferring such success to the chemical domain remains highly challenging. First, there exist significant disparities among expert chemical LLMs. It is widely acknowledged that chemical tasks differ fundamentally from natural language tasks, often requiring larger shifts in representation space and greater parameter adjustments when fine-tuning general- purpose foundation models (e.g.. LLaMA [46], GPT [40] for chemical applications. Consequently, base and expert models in the chemical domain tend to exhibit greater variance due to task specciicc specializations, in stark contrast to the relatively homogeneous model landscape in NL.. Additionally, these greater fine-tuning adjustments accumulate noise and further amplify parameter inconsistencies, leading to interference and hidden conflicts among expert models that complicate the merging process. Second, the chemical domain exhibits a highly imbalanced distribution of targeted downstream tasks, models
Widely studied tasks such as property prediction are supported by numerous (i.e., expert-abundant scenarios), whereas many niche or emerging tasks such as retrosynthesis are covered by only a few or no published models (i.e., expert-sparse scenarios). This imbalanced distribution introduces additional challenges when merging models: knowledge critical expert- sparse tasks may be diluted by noise or overshadowed by dominant patterns from expert-abundant models. As result, the effectiveness of existing model merging methods remains limited in chemical domain. Merged models frequently suffer from poor knowledge generalization beyond individual model boundaries and degraded performance across diverse task types.
In this paper, we introduce Curriculum Model Merging (CMM), a framework designed to address the aforementioned challenges in merging chemical LLMs. Inspired by curriculum learning 14), we decompose the extremely complicated problem of simultaneously combining heterogeneous expert models into a progressive and continual merging process. At its core, CMM constructs a route-based merging curriculum. In the early stages, weaker expert models are merged first, which gradually enhances the base model's 's generalization and stability. This, in turn, prepares the base model to accommodate stronger and more specialized experts in subsequent rounds. The result is a merged model that incrementally strengthens task task-specific capabilities while maintaining robustness across tasks. Our CMM method consists of two main steps. (1) Curriculum construction: CMM ranks expert models based on their capabilities across various benchmarks, establishing an ordered curriculum that navigates a continual, capability-awar merging process. (2) Iterative merging: each expert model is progressively merged into the current base model through task vector extraction and composition, after which the merged model serves as the base model for the next iteration. Recognizing that the relative importance of each expert significantly influences the ultimate performance, we introduce multiple merging strategies that vary the degree of involvement for each expert. This is achieved by jointly controlling the merging order and assigning adaptive merging weights.
The key contributions of our paper are outlined below. (1) Practicability: We, for the first time, integrate model merging techniques to effectively consolidate task- asecificciiic specializations in the chemical domain, overcoming data unavailability and avoiding substantial computational costs. (2) Ef- ficacy: We introduce CMM, progressive and performance-awarr merging framework that addresses the significant challenges of merging heterogeneous chemical LLMs. Across two representative benchmarks. CMM exceeds the state-of-the-art merging method by 29.03%. (3) Generality: We evaluate CMM under both expert-abundant and expert-sparse scenarios. The results demonstrate that CMM facilitates robust chemical knowledge generalization across both predictive and generative tasks, without sacrificing performance consistency.
2 Related Work
Chemical LLMs We briefly summarize the recently published LLM studies and their special- ized versions in the chemistry domain in Appendix A Early models such as EMILES-BER' 44
and SMILES Transformer [22 have been employed to process SMILES information of molecules. Uni-Mol (4)) AGBT 5 and Molformer (52) focus on modeling 3D molecules. SeiGLM (1]. Scil.-
itL.LM 30. KALF-LM-Chem [9] and LLAMMT 37 primarily focus on natural language processing, information extraction, and scientific reasoning capabilities. GeLLMO [[] MolecularGPT [32], and
2

---

Standard Model Merging Methods
Pairwise Model Merging
Multiple Model Merging
M
M4
M
M
M
margo
Minerge
M
model
Merged model parametees
paramettrs
M
Curriculum Model Merging
Ranking considering expert capariic capability
Iterative merging procedure
M2
M2
O
4
Ba
Fs
Expert merging order
P:
R4
+
Dynamic merging weights
Mmerg=
f
82
Bg
Ba
M
Base model
Intermediate aserged moddel
Mergad model parameters
permmmttos
parameters
Base Model
Expert Models
Figure 1: Standard Model Merging Methods vs. Curriculum Model Merging Standard Model Merging merges all expert models simultaneously, while Curriculum Model Merging first ranks the expert models and then merges them progressively.
Mol-LLaMA 27 have been fine-tuned for molecule-related tasks, including multi-property molecule optimization tasks, molecular property prediction, and understanding molecules. ChemLLM [62], ChemDFM 633, LlaSMol [59] and Kolinst-molecule [15] are specialized for essential chemistry tasks, such as retrosynthesis, yield prediction, molecular property prediction, and description guided molecule design. CrystaLLM and Crystal-text-LLM 38 are specifically designed for crystal generation and stable materials generation.
Model Merging Methods Model merging methods are broadly categorized into: (1) pre- merging
methods, which align models before merging, and (2) during-merging methods, which resolve con- flicts during merging process. Pre-merging methods leverage the linear mode connectivity (LMC)
property [12] 14 [16] to align models prior to merging. Early approaches such CCAMerge 2 use
bCA-based neuron alignment, while MuDSC [54] jointly aligns weights and activations. C2MB
optimizes global permutations layer-wise. and Deep-Align [39] introduces Icarnable architecture for dynamic weight alignment. During-merging methods directly manipulate parameters to reduce interference. Model Soups (51] averages weights, while Task Arithmetic [25] and TIES (55) use task vectors with interference control. DARE [60] and DELLA [10] sparsify deltas to retain critical changes. Uncertainty-basee merging 8 uses second-order Hessians guide merging. Modular approaches such as Concrete [22. MMr-Mngging [24] Brin-Merging [4], and Weight-Ensemblie MoE [44) apply dynamic routing and task-specific control. PWE-MMoE (43) extends this multi- objective settings using preference vectors. Finally, Representation Surgery (58) calibrates merged models post hoc by aligning their representations with those of the original expert models.
Methodology
3.1 Preliminaries
Let M, represents a pretrained model and (M,, .2,..... ... My} represent models fine-tuned based on the pretrained model. The goal is to merge the combined strengths of these models into M,. We

---

follow the task arithmetic [25
approach and merge tasks using combined task vectors where the
task vector is defined
7 0h pre:
(1)
where Ppe represents the weights of the pretrained model and of represents weights after fine-tuning on task
Task Arithmetic Merging Given N expert models fine-tuned from the same base, this method aims to merge them into a single model merge by linearly combining their task vectors. A simple arithmetic average is used to obtain the merged task vector:
T new
TE:
8 mergg Opee À Tnew
where À is an optional scaling term controlling the influence of the aggregated task updates, and Tnew denotes the sum of all task vectors.
Drop And Rescale Merging [60] This is method reduce interference during merging. The first step, Drop, randomly resets a portion of task vectors to zero according to a drop rate p. The second step, Rescale, multiplies the remaining values by scaling factor of I-p" After applying Drop And Rescale (DARE), large number of parameters in the fine-tuned model become identical those in the base model. In other words, DARE updates only a small subset of parameters, retaining only a sparse set of parameter changes and thereby reducing redundancy.
3.2 Curriculum Model Merging
We evaluate the set of models {M1, M2...., MN} on validation set. For each model, we evaluate its performance on individual tasks and normalize the scores to ensure comparability across tasks. The normalized scores are then averaged and aggregated to compute overall performance score. Let the overall performance scores be denoted as {Si, S SN.. Note that we provide an ablation study in Appendix to investigate the impact of different various sizes of the validation set on final merging effectiveness. We sort the models in ascending order of performance to obtain ranking:
computation methods well as
M(1) M(2). ... M(N): such that S()) S S(2) S... ... S S(N),
where S(o) the score of model M(y- The coefficient 3x (0, 1) is the scaling term used to combine the k-th task vector. Specifically, we compute:
7(1) (11) ppc:
0 = Ppc + 31 T(1)
(3)
1(2)
(22 01
02 0 + 32 T(2):
(4)
(k) ((k) n 0 0: 0 + 3x T(k))
(5)
where k = 1,2,..., ... N. T(k) represents the difference between the fine-tuned parameters of the k-th model and the current merged model, and 0g denotes the parameters of the merged model after incorporating the k-th task vector. This recursive procedure continues until all task vectors have been merged.
Since model performance improves progressively, better- petter-perfommig models should not be assigned smaller weights. There are two strategies to increase the contribution of the later models. First, the distribution of 3 should be either flat or increasing based on practical needs, such as constant value, linear schedule, or exponential growth. We have explored these possibilities in Section 4.4 Second, CMM is capable of dynamically adjusting the degree of involvement of each expert. Since < Bx < 1, this property inherently allows greater weights for the later models. The proof is provided below.

---

Table 1: Performance of baseline methods and the proposed approach on Chembench. For all evaluation metries, higher values indicate better performance. We highlight the best performance as bold. Bold values denote the best results in each column.
Model
NC
Property
M2C
C2M Product
RS
YP
Average
Expert models
Alama-3-8B-Instruee
$1.19
27.79
90.30
40.88
34.00
29.33
45.33
60.89
33.67
45.93
Molinst-molecul--Bt
39.05
25.39
80.94
39.75
29.67
31.67
46.33
60.89
33.00
42.97
KALE-LM-Chem-1.5
61.33
43.72
90.30
53.75
72.67
53.67
45.67
47.52
45.00
57.07
Mccrkat-8b-vl.0
50.19
24.26
86.62
40.88
27.33
30.33
42.33
56.93
34.00
43,65
GeLLMO-PP
28.91
29.48
59.53
24.25
24.67
25.00
41.33
29.70
24.00
31.87
SOTA merge methods
TA
60.83
25.11
92.98
46.12
55,00
46.67
47.00
59.90
33.00
51.85
TIES
46.43
30.61
77.26
40.00
35.67
34.33
32.33
44.55
27.33
40.95
DARE
31.41
25.11
49.83
23.88
25.00
25.00
41.67
29.21
23.33
SCE
40.05
43.02
77.93
36.75
28.67
45.33
30.20
29.33
40.03
AF
58,95
29.76
91.97
47.5
47.00
42.00
46.00
63.37
37.33
51.54
Merged models through CMM
CMM
63.95
35,40
92.98
54.37
78.33
72.33
46.00
64.85
44.67
61.43
DARE_CMM
63.58
33.43
92.31
51.88
74.00
68.00
47.33
62.87
46.00
59.93
Substituting T(1) into 0 gives:
0 = -1-11-33))) 32) + ()
(6)
Similarly, substituting the previous expressions into 0₂ gives:
0 (1 -1) B2 B1 (1 B2)0 + 320
(7)
k
k
k
O₂ = [I 3i)
Opee +
Bg
(1 B:)
0
(8)
j=1
{=
During merging, since < < 1,0 < 3&) < 1, the involvement of top-ranked models that are merged earlier is dynamically reduced.
Our framework compatible with different and incoming advanced model merging methods. We denote the original version combined with Task Arithmetic as CMM, and the one with DARE as DARE_ CMM.
4 Experiments
4.1
Experimental Setup
Base & expert models We briefly summarize the recently published LLM studies in the chemistry domain in Appendix A Among them, expert models for merging are selected considering their enhanced performance on respective specialized tasks. The performance of each model on individual chemical tasks is presented in Table T and Table 2 Ultimately, we adopt the universal LLaMA3- 8B-Instruct (18 as the base model. GeL.LM-P66
[1]. Meerkat-8B-vi.. 28), KALE-LM-
Chem-1.5 9 and Molinst-Molecule-B81 [15 as expert models. Based on the method which is demonstrated in Section 3.2 and the overall performance across benchmarks of each expert models reported in Table 3 the merging order is assigned follow: GeLLMO-P6-LIee The merging weight coefficients for each model, 3, are assigned using linear strategy that increase from 0.3 to 0.6. We draw inspiration for this coefficient range from a conclusion in task arithmetic 25]: "Scaling coefficients in the range 0.3 to 0.5 produce close optimal results in many cases." Ablation studies and corresponding experimental results are provided in Section 4.4 to demonstrate the effectiveness of the chosen merging order and the 3 weighting strategy.
Mcerkat-8B.. Berrkatt-B--ll.. 8.0v
Molinst-Molecule--
Baseline algorithms We compare the performance of the merged model derived from our approach against variety of baselines, including: 1) the original base and expert models, and 2) merged
5

---

models obtained through existing perfformmee model merging approaches. Task Arithmetic (TA) [25] constructs task vector by subtracting the weights of the base model from the expert models and combines together the task vectors by concise arithmetic operations. mees-ng (TIES) [55] discards the task vectors with negligible changes and combines the remaining vectors that are aligned in sign. DARE_TE TA [60] considers the disparities between base and expert model parameters. dropping a subset of task vectors. SCE [48] allocates parameter matrix-leve coefficients based on the magnitude of parameter changes, enabling fine-grained merging. Arcee Fusion (AF) [17] assesses the importance of each parameter based on the Kullback-Leibler (KL) divergence and dynamically optimizes the parameters accordingly. Studies have shown that across different datasets and experimental settings, TA, DARE TA, and TIES alternately achieve the best performance and are regarded ofate-of-the-ase art methods. To ensure fair comparison, all approaches use the same base and expert models described above. While TA, TIES, DARE_TA and SCE support the simultaneous merging of multiple models, AF merges one expert model into the base model at time and performs iterative merging for multiple experts. Each method receives the same ranking information derived from the validation set. More specifically, the TA, TIES, DARE_ TA, and SCE methods also require assigning a weight to each expert model. The AF method requires specifying merging order. In all cases, the weights or orders used are the same as those used by CMM. The details and results of the comparison between CMM and other machine learning methods are provided in Appendix B
Benchmarks and evaluation metrics The performance of various models is evaluated on two repre- sentative chemical benchmarks. The first benchmark, Chembench [62], consists of 4,100 high-quality multiple-choice questions and answers spanning 9 core chemistry tasks: Name Conversion (NC), Property Prediction (Property P). Mol2Caption (M2C), Caption2Mol (C2M), Product Prediction (Product_ _). Retrosynthesis (RS), Yield Prediction (YP), Temperature Prediction (TP), and Solvent Prediction (SP). The evaluation metric used for this benchmark is accuracy. The second benchmark consists of two molecular generation tasks from Mol-Instructions 113: retrosynthesis and forward reaction prediction. To reduce computational overhead, 200 samples from each task are randomly selected as the test dataset. An ablation study in Appendix D exxamines the relationship between the number of evaluation samples and the resulting metrics. The results indicate that beyond 200 samples, the evaluation scores stabilize, showing only minor fluctuations without any consistent upward or downward trend. We employ Round-trip accuracy, SELFIES BLEU score, Validity rate and Exact match rate as the evaluation metrics to comprehensively evaluate the performance. SELF- IES BLEU score (SBS) quantifies the syntactic similarity between the generated SELFIES strings and the ground truth strings using the BLEU metric, which captures n-gram overlap. Validity rate (VR) is defined as the proportion of generated molecular structures that are chemically valid. Exact match rate (EMR) measures the proportion of generated molecular structures that exactly match the corresponding ground truth structures. However, given that multiple chemically plausible predictions may exist beyond the exact ground truth, retrosynthetic studies (26) have proposed a more chemically meaningful and robust evaluation metric, round-trip accuracy. Round-trip accuracy (RTA) measures the proportion of predicted reactants that, when input into forward reaction model, regenerate the original product. It reflects the chemical validity and semantic correctness of the predicted structures. Accordingly, we consider round-trip accuracy the most reliable metric on the second benchmark. The validation set used for ranking is composed of the ChemBench dev set and 100 samples from Mol-Instructions, both of which are entirely disjoint from the test set.
4.2 Results
Table T shows the prediction performance of models merged using our approach (CMM, DARE_ CMM), in comparison with individual expert models and models produced by baseline merging approaches across 9 multiple-choice chemical question tasks and their overall average on ChemBench. The expert models exhibit diverse strengths across different tasks, confirming their respective specializations. For example, MALE--LM-hhem-- erty P and strong results on C2M and Product P. while Molinst-Molecule perform competitively in YP and TP. In contrast, the general-purpose base model LLaMA3-8B- Instruct delivers moderate results and lacks task-speeific specialization, leading to a lower average accuracy of 45.93 compared to the best-performing expert, KALF-LMM-Chm-- 1.55, which achieves 57.07. This disparity underscores the value of leveraging expert models tailored to specific chemical domains. Our merged models, CMM and
achieves the best performance on Prop-
and Meerkat-8B-v1.0 -B-
consistently outperform individual experts

---

Table 2: Performance of baselines and our approach on Mol-Instructions. For all columns, higher values reflect better performance.
Retrosynthesis SBS
Forward Reaction Prediction RTA
Average RTA
Models
RTA
VR
EMR
Expert models
Llama-3-8B-Insteee
0
0.14
0.21
0.82
0.07
Molinst-molecle--0
0.42
0.50
0.66
0.22
0.31
0.55
0.69
0.33
0.37
ALLE-LM-Chem-L55
0
0
1.00
0
0.14
0.21
0.82
0
0.07
Meerkat-8h-vla,
0.02
0.61
0
0.13
0.45
0
0.08
0.16
0.93
0.08
SOTA merge methods
0.16
0.98
0.03
0.08
TIES
0.10
0.24
0.73
0.03
0.35
0.89
0.07
DARE TA
0.36
0.85
0..88
0.12
0.29
0.91
1.00
0.21
0.33
SCE
0.01
0.06
0.54
0
0.22
0.49
0.01
0.12
0.31
0.53
0.01
Merged models through CMM
CMM
0.42
0.66
0189
0.28
1.00
0.22
0.35
DARE_ CMM
0.31
0.64
0.84
0.05
0.28
0.92
1.0
0.23
0.30
by effectively integrating their complementary strengths. Notably, CMM achieves the highest average accuracy of 61.43, surpassing KALE-LM-Chem--I. by 7.6% and the base model by 33.7%, and ranks first in 6 out of 9 tasks. DARE_ _BMM also achieves strong results, with an average accuracy of 59.93. These results demonstrate that CMM not only concentrates task-specific expertise but also mitigates inconsistencies, such parameter interference and accumulated finettuning noise, that typically arise when combining heterogeneous Chemical LLMs. By progressively aligning expert models with the base model based on capability, CMM facilitates chemical knowledge generalization without sacrificing robustness. Compared to existing state-of-the-art model merging methods, including TA, TIES, AAR__TA, SCE, and AF, our approach delivers superior performance both in average and across individual tasks. While TA achieves a respectable average accuracies of 51.85, which is 18.5% lower than CMM, it also falls short several tasks where our methods excel, such Product P. RS, and TP. Moreover, all bascline approaches struggle to resolve hidden noise and parameter conflicts, resulting in merged models that underperform even the best individual expert -KALE-LM-Cheme This indicates their limited effectiveness preserving knowledge and managing inter-model incon- sistencies. In contrast, CMM and DARE CMM maintain consistently high performance across tasks, highlighting their effectiveness in preserving useful knowledge while avoiding performance degradation.
Table 3: The overall average performance across 9 prediction tasks from ChemBench and 2 generative tasks from
Table reports the performance the Mol- Instructions benchmark, which focuses on gen- erative chemical tasks, specifically retrosynthe- sis and forward reaction prediction. In contrast to ChemBench, expert-abundant bench- mark with multiple experts demonstrating su- perior performance, Mol-Instructions presents an expert-sparse scenario, where only a single expert, Molinst-Molecule-s results. Specifically, achieves round-trip ac- curacies of 0.42 and 0.31 on retrosynthesis and forward prediction, respectively, while other ex- pert models, including those that performed well on ChemBench, exhibit near-zero performance on round-trip accuracy. This large performance disparity highlights the unique specialization of Dolinst-Molecule-st and the uneven distribu- tion of expertise among the expert model group. Despite this imbalance, our proposed methods, CMM and DARE, CMM, successfully concen-
Models
Overall average
Expert models
Llama-3-8BBInnstee
38.85
Molinst-molecule-8
41.79
KALE-LM-Chem--15
47.97
Meerkat-8h-vv.00
35.71
shows promising
GeLLMO-P6
27.53
SOTA merge methods
TA
43.88
TIES
34.69
DARE_ TA
30.86
SCE
32.84
AF
42,26
Merged models through CMM
CMM
56.62
DARE DARE_CMM
54.40

---

trate the rare but critical generative capability of Molinst-Molecule-83 8B without being negatively impacted by the poor performance of other experts. CMM achieves a round-trip accuracy of 0.42 on retrosynthesis, matching the top-performing expert, and maintains strong validity and SELFIES BLEU scores. On forward reaction prediction, CMM attains round-trip accuracy of 0.28, closely approaching Nolinst-Molecule-88
0.31. Importantly, further analysis reveals that CMM correctly generates a distinct subset of samples compared to Molinst-Molecules model does not merely inherit expert knowledge in one-to-one fashion. While some knowledge from Molinst-Molecun
suggesting that the merged
is not retained during the merging process, CMM also appears to generalize relevant patterns beyond individual model boundaries, bridging predictive reasoning and generative modeling. This finding underscores the advantage of CMM's 's capability-awai aware merging strategy, which not only preserves task task-specialized knowledge but also supports meaningful high-level expert intelligence fusion. On the contrary, baseline model merging methods struggle in this expert-sparse setting. Their round-trip accuracy retrosynthesis rarely exceeds 0.1, and drops to near zero on forward reaction prediction. This failure stems from their inability to handle highly imbalanced con- ditions where most experts contribute noise rather than signal, leading poor knowledge integration and degraded performance.
To enable a unified comparison of model performance across the two benchmarks, we normalize the round-trip accuracy scores from Hol-Instructions by rescaling them to a 0-100 range. We then combine the results from the nine predictive tasks in ChemBench and the two generative tasks in Inol-Instructions to compute an overall average score, as shown in Table 3 CMM achieves the highest overall average score of 56.62, significantly outperforming all individual expert models and state-of-the -ott-h----t---ttt-e- model merging methods. Specifically, it exceeds the best-performing expert model, KALB-LM-Chem- 29.03%. The DARE-CMM model also performs strongly, achieving an overall average score of 54.40, ranking second among all merged models and outperforming all expert models and other baseline approaches. These results clearly demonstrate the effectiveness of our progressive, capability-awand merging strategy for integrating chemical LLMs. Notably, CMM not only concentrates spe--spcificc expertise but also mitigates inconsistencies and achieves strong generalization across both predictive and generative tasks under both expert- abbundnnt and expert-sparse conditions.
by 18.03% and outperforms the best baseline merging method, TA, by
4.3 Analysis
We use the Subspace Alignment Ratio (SAR) 36 to investigate how the expert models are integrated into the merged model during the merging process. The SAR is used to quantify the alignment between the subspaces of two task matrices. The formal definition and computation details of SAR are provided in Appendix F A higher SAR value indicates greater overlap between the subspaces, meaning that the merged model better inherits the capabilities of the expert model.
We observe that as more expert models are merged later in the process, the SAR values of those merged carlier tend to slightly decrease. Additionally, higher assigned weights are generally associated with higher SAR values. This suggests that placing better-periee setter-performing expert models later in the merging order, and assigning them higher weights, leads to higher SAR values. Take a model as an example. Table fable/Issowss shows the evolution of the SAR value for molinst-molecule-88 during the CMM process. This expert model was merged at step 3, where its SAR value increased significantly. At step 4, the SAR value slightly decreased due to the merging of KALE-LM L.MM--hhm-1.55 When reduced the weight assigned to Molinst-molecule-8B at step 3, we observed a corresponding decrease in its SAR value.
Table 4: SAR for Molinst-molecule.
Metric
Stepl
Step2
Step3
Step4
Step3(reduced weight)
SAR
0.042
0.042
0.650
0.647
0.506
4.4 Ablation study
Influence of the number of expert models Starting from the full expert model set (GeLLMO- Pa-mlama, Meerkat-8B-v1.o. Molinst-Molecule-SH and KAL.F-LM-Chem-I..)) we progressively reduce the number of source models to evaluate how performance degrades across both predictive and
8

---

Table 5: Ablation study on the number of expert models used in CMM. We progressively reduce the number of merged expert models evaluate the impact on CMM's performance across the ChemBench and Mol-Instructions benchmarks.
Chembench Average
Mol-Instructions
Overall Average
GeL.LM
Meerkat
Molinst-molecule
KALE-LM-Chem
Average
V
61.43
0.35
56.62
Y
48.47
0.13
42.02
40.00
49.34
40.37
Table Ablation study on the expert model merging order. We compare the performance of CMM under different merging sequences, including reverse and random orders.
Chembench Average
Mol-Instructions
Overall Average
Merge Sequence
Average
Default
61.43
0.35
56.62
Reverse
48.
0.13
41.67
Random
52.05
0.26
47.31
generative benchmarks, results presented in T The full CMM configuration, which includes all four expert models, achieves the most superior performance both benchmarks, with an average score of 61.43 on ChemBench and 0.35 on Mol-Instructions, resulting in the best overall average of 56.62. Removing the strongest predictive expert, drop in performance, ChemBench drops to 48.47 .47 and Mol-Instructions to 0.13. Although KALE- LM-Chem-1.5 is key contributor in boosting merging performance, the model after merging KALE-LM-Cheem-I35 achieves a siginificantly higher overall average score of 56.62 compared to KALE-LM-Chem's 47.97. This suggests that CMM enables a bi-directional improvement, where not only does the high-performing expert contribute significantly, but the inclusion of moderate- performing experts in the early stage also accumulate positive effect leading to the ultimate superior overall performance. As more experts are excluded, performance continues to decline, particularly in the generative domain where hol-Instructions scores fall to zero once Molinst-Molecule is removed.
leads to substantial
Influence of the expert model merging order To investigate the effect of merge sequence on model performance, we compared our default merge order in the main experiment with two alternative strategies: reverse order and random order. 6
an ablation study evaluating the impact of expert model merging order on the performance of CMM. The default merging sequence, determined based on model task-specific performance, achieves the best overall results, with an average score of 61.43 on ChemBench, 0.35 on Mol-Instructions, and an overall average of 56.62. In contrast, reversing the merging order results in a sharp decline in performance, particularly on ChemBench, dropping 48. 15, and overall, down 41.67. Similarly, using a random merging order yields a moderate degradation in performance, with overall accuracy reduced to 47.31. To further understand these effects, we visualize and compare the cumulative performance trajectories under the default and reverse merging orders. As shown in Figure 2 the default order produces a steady and consistent performance gain, culminating in the highest final score after all expert models are merged. In contrast, the reverse order exhibits a diminishing return pattern: although merging the strongest expert model (KALE-LM- (M-Cmmm first provides an early boost, subsequent merging steps result stagnant or reduced gains. This supports the hypothesis that progressively building the model's capacity using increasingly capable experts helps consolidate task-specific strengths while maintaining base model's adaptive capability for the next round.
Influence of the merging strategy and weight coefficient 3 Since the expert models are ordered from weakest to strongest based on their individual performance, the distribution of B is expected to be either uniform or gradually increasing. In this experiment, we evaluate three distribution strategies: constant, linear, and exponential. As shown in Table 7 the linear distribution yields the best overall performance. In the constant setting, all expert models are assigned the same 8g, which we set to 0.5-a a value empirically close to the optimal according to prior research. Assigning equal weights to all expert models results a notably lower overall average of 54.60. To further examine the effect of the coefficient range, we conducted an ablation study by varying 3 within different intervals. In our experiments, we found that slightly increasing the coefficient could lead to marginal performance improvements, whereas slightly decreasing it often resulted in noticeable drop in performance.

---

60
Overall average score
LLaMA 4B Instuct (bosS)
KALE JACC Mol-MMo
Meerkan
KALE LAEEEMMO co
LLOkW'3 68 Instruct (baso)
order
KALE: E-LM-Chem-11
45
LLakskk--iisinndae (bson) POL
-AA 2--2hem-1
Reverse order CMM
Mo inst Rittetuee 48
Maerbai--BBr11 B-11
LLaMAG 5B Instuct (base)
Ged Ged. hem
LLaMAk3-BB instruct (bass) MALLLM--PB-Llmm
Cedl MOD-PSs Liers
2
3
Merging steps
Figure 2: Performance comparison of the merged models along the trajectories of default/reverse order CMM.
Table Ablation study on the influence of the merging strategy and the coefficient range weight 3. We compare constant, linear, and exponential distributions of 3, and further analyze the effect of varying the coefficient range on model performance.
Chembench Average
Mol-Instructions
Overall Average
B distribution
Averuge
constant(0.3, 0.6]
60.51
0,28
54.60
linear(0.33 0.3, 0.6]
61.43
0.35
56.62
exponential(0.33
61.38
0.33
56.22
linear[0.2. 2, 0.5]
61.03
0.24
54.21
linear(0.4, 0.7)
60.38
0.41
56.86
4.5 Scalability
To evaluate the scalability and generality of CMM, we further apply to settings beyond the chemical domain. The results demonstrate that CMM remains effective and stable under these extended settings, indicating that the underlying merging mechanism is not limited chemistry-specifi tasks. Detailed experimental setups and results are provided in Appendix
5 Conclusion
In this work, we propose Curriculum Model Merging (CMM), progressive, capability- framework tailored to the challenges of merging heterogeneous chemical LLMs. CMM addresses the domain-specific issues of model disparity and imbalanced task coverage by structuring the merging process curriculum and applying adaptive weighting to expert contributions. Evaluations ChemBench and Mol-Instructions show that CMM consistently outperforms individual experts and state-of-the-ard baselines, while generalizing well across predictive and generative tasks in both expert-abundant and expert-sparse settings. These results demonstrate CMM's effectiveness as scalable and privacy-conseious approach for building versatile chemical language models from specialized experts.
Limitation
While our approach demonstrates strong empirical performance, several limitations remain. First, model merging currently requires all expert models to share the same architecture, which may limit its applicability to settings with different model architectures. Second, performance can also degrade when merging large number of expert models. This is known challenge observed in prior work (5) Lastly, although initial analyses suggest why CMM achieves improved results, deeper theoretical understanding remains future research.
10

---

Acknowledgements
We thank all the anonymous reviewers for their constructive suggestions on improving this paper. This paper is partially supported by National Natural Science Foundation of China (NSFC) grant 62441236.
References
[1] Luis M Antunes, Keith T Butler, and Ricardo Grau-Crespo. Crystal structure generation with autoregressive large language modeling. Nature Communications, 15(1):00570, 2024.
[2] Arcee.ai. Llama-3. --supernova-lite.
L-SuperNova-Litee 2024.
[3] Ashvini Kumar Jindal, Pawan Kumar Rajpoot, Ankur Parikh, Akshita Sukhlecha. Llama-3.1- storm-8b, 2024.
[4] Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th Annual International Conference on Machine Learning, ICML. '09, page 41-48, New York, NY, USA, 2009. Association for Computing Machinery.
[5] Dong Chen, Kaifu Gao, Duc Duy Nguyen, Xin Chen, Yi Jiang, Guo-Wei Wei Wei, and Feng Pan. Algebraic graph-assisted bidirectional transformers for molecular property prediction. Nature communications, 12)1223,,, June 2021.
[6] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jenry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers solve math word problems, 2021.
[7] Donato Crisostomi, Marco Fumero, Daniele Baieri, Florian Bernard, and Emanuele Rodolà. m: Cycle-consistent multi-model merging, 2024.
[8] Nico Daheim, Thomas Möllenhoff, Edoardo Maria Ponti, Iryna Gurevych, and Moham- mad Emtiyaz Khan. Model merging by uncertainty-basse gradient matching, 2024.
[9] Weichen Dai, Yezeng Chen, Zijie Dai, Zhijie Huang, Yubo Liu, Yixuan Pan, Baiyang Song, Chengli Zhong, Xinhe Li, Zeyu Wang, et al. Kale-Im: Unleash the power of ai for science via knowledge and logic enhanced large model. arXiv preprint arXiv:2409. 18695, 2024.
[10] Pala Tej Deep, Rishabh Bhardwaj. and Soujanya Poria. Della-merging: Reducing interference in model merging through magnitude-based sampling, 2024.
[11] Vishal Dey, Xiao Hu, and Xia Ning. G Generalizing large language models for multi- property molecule optimization, 2025.
[12] Felix Draxler, Kambis Veschgini, Manfred Salmhofer, and Fred A. Hamprecht. Essentially barriers in neural network energy landscape, 2019.
[13] Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers). pages 2368- 2378, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
[14] Rahim Entezari, Hanie Sedghi, Olga Saukh, and Behnam Neyshabur. The role of permutation invariance in linear mode connectivity of neural networks, 2022.
[15] Yin Fang. Xiaozhuan Liang. Ningyu Zhang, Kangwei Liu, Rui Huang, Zhuo Chen, Xiaohui Fan, and Huajun Chen. Mol-instructions: A large-scale biomolecular instruction dataset for large language models. ICLR. OpenReview.nee, 2024.
[16] Damien Ferbach, Baptiste Goujaud, Gauthier Gidel, and Aymeric Dieuleveut. Proving linear mode connectivity of neural networks via optimal transport, 2024.

---

[17] Charles Goddard, Shamane Siriwardhana, Malikeh Ehghaghi, Luke Meyers, Vladimir Karpukhin, Brian Benedict, Mark McQuade, and Jacob Solawetz. Arcee's MergeKit: A toolkit for merging large language models. In Franck Dernoncourt, Daniel Preotiuc-Pietns and Anastasia Shimorina, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 477-485, Miami, Florida, US, November 2024. Association for Computational Linguistics.
[18] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, and et al. The Ilama herd of models, 2024.
[19] Yang Han, Ziping Wan, Lu Chen, Kai Yu, and Xin Chen. From generalist to specialist: A survey of large language models for chemistry. Owen Rambow, Leo Wanner, Marianna Apidianaki, Hend Al-Khalifa, Barbara Di Eugenio, and Steven Schockaert, editors, Proceedings of the 31st International Conference on Computational Linguistics, pages 1106-1123, Abu Dhabi, UAE, January 2025. Association for Computational Linguistics.
[20] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, 2021.
[21] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. NeurlPS, 2021.
[22] Shion Honda, Shoi Shi, and Hiroki R. Ueda. Smiles transformer: Pre-trained molecular fingerprint for low data drug discovery. 2019.
[23] Stefan Horvi, Albert Manuel Orozco Camacho, Eugene Belilovsky, and Guy Wolf. Harmony in diversity: Merging neural networks with canonical correlation analysis, 2024.
[24] Chenyu Huang, Peng Ye, Tao Chen, Tong He, Xiangyu Yue, and Wanli Ouyang. Emr-merging: Tuning-fre free high- perforrmffm model merging, 2024.
[25] Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannanch Ha- jishirzi, and Ali Farhadi. Editing models with task arithmetic. In The Eleventh International Conference on Learning Representations, 2023.
[26] Yinjie Jiang, Yemin Yu, Ming Kong, Yu Mei, Luotian Yuan, Zhengxing Huang, Kun Kuang, Zhihua Wang, Huaxiu Yao, James Zou, Connor W. Coley, and Ying Wei. Artificial intelligence for retrosynthesis prediction. Engineering, 25:32-50, 2023.
[27] Dongki Kim, Wonbin Lee, and Sung Ju Hwang. Mol-llama: Towards general understanding of molecules large molecular language model, 2025.
[28] Hyunjae Kim, Hyeon Hwang, Jiwoo Lee, Sihyeon Park, Dain Kim, Taewhoo Lee, Chanwoong Yoon, Jiwoong Sohn, Donghee Choi, and Jaewoo Kang. Small language models learn enhanced reasoning skills from medical textbooks. arXiv preprint arXiv:2404.000. 240.04440.00376, 2024.
[29] Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang. Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luca Soldaini, Noah A. Smith, Yizhong Wang, Pradeep Dasigi, and Hannaneh Hajishirzi. Tulu 3: Pushing frontiers open language model post-training, 2025.
[30] Sihang Li, Jin Huang, Jiaxi Zhuang, Yaorui Shi, Xiaochen Cai, Mingjun Xu, Xiang Wang, Linfeng Zhang, Guolin Ke, and Hengxing Cai. Scilitllm: How to adapt 1lms for scientific literature understanding, 2024.
[31] Xianggen Liu, Yan Guo, Haoran Li, Jin Liu, Shudong Huang, Bowen Ke, and Jiancheng Lv. Drugllm: Open large language model for few-shot molecule generation, 2024.
[32] Yuyan Liu, Sirui Ding, Sheng Zhou, Wenqi Fan, and Qiaoyu Tan. Moleculargpt: Open large language model (Ilm) for sho-shot molecular property prediction, 2024.
12

---

[33] Zequn Liu, Wei Zhang, Yingce Xia, Lijun Wu, Shufang Xie, Tao Qin, Ming Zhang, and Tie- Yan Liu. MolXPT: Wrapping molecules with text for generative pre-training, In Anna Rogers. Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1606-1616, Toronto, Canada, July 2023. Association for Computational Linguistics.
[34] Zhenyi Lu, Chenghao Fan, Wei Wei, Xiaoye Qu, Dangyang Chen, and Yu Cheng. Twin-merging: Dynamic integration of modular expertise in model merging, 2024.
[35] Liuzhenghao Lv, Zongying Lin, Hao Li, Yuyang Liu, Jiaxi Cui, Calvin Yu-Chian Chen, Li Yuan, and Yonghong Tian. Prollama: A protein large language model for multi-task protein language processing. IEEE Transactions on Artificial Intelligence, pages 1-12, 2025.
[36] Daniel Marezak, Simone Magistri, Sebastian Cygert, Bartlomiej Twardowski, Andrew D. Bagdanov, and Joost van de Weijer. No task left behind: Isotropic model merging with common and task-specific subspaces, 2025.
[37] Vaibhav Mishra, Somaditya Singh, Dhruv Ahlawat, Mohd Zaki, Vaibhav Bihani, Hargun Singh Grover, Biswajit Mishra, Santiago Miret, Mausam, and N. M. Anoop Krishnan. Foundational large language models for materials research, 2025.
[38] Andrea Madotto Andrew Gordon Wilson C. Lawrence Zitnick Nate Gruver, Anuroop Sriram and Zachary Ward Ulissi. Fine-tuned language models generate stable inorganic materials as text. In International Conference on Learning Representations 2024, 2024.
[39] Aviv Navon, Aviv Shamsian, Ethan Fetaya, Gal Chechik, Nadav Dym, and Haggai Maron. Equivariant deep weight space alignment, 2024.
[40] OpenAl, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, et al. Gpt-4 technical report, 2024.
[41] Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoe V Le, Ed H Chi, Denny Zhou, et al. Challenging big- bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.088 2022.
[42] Anke Tang, Li Shen, Yong Luo, Liang Ding, Han Hu, Bo Du, and Dacheng Tao. Concrete subspace learning based interference elimination for multi-task model fusion, 2023.
[43] Anke Tang, Li Shen, Yong Luo, Shiwei Liu, Han Hu, and Bo Du. Towards efficient pareto set approximation via mixture of experts based model fusion, 2024.
[44] Anke Tang, Li Shen, Yong Luo, Nan Yin, Lefei Zhang, and Dacheng Tao. Merging multi-task models via weight-ensemblligg mixture of experts, 2024.
[45] Ryan Teknium, Jeffrey Quesnelle, and Chen Guang. Hermes technical report, 2024.
[46] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023.
[47] Umit V. Ucak, Islambek Ashyrmamatov, Junsu Ko, and Juyong Lee. Retrosynthetic reac- tion pathway prediction through neural machine translation of atomic environments. Nature Communications, 13(1):1186 2022.
[48] Fanqi Wan, Longguang Zhong, Ziyi Yang, Ruijun Chen, and Xiaojun Quan. Fusechat: Knowl- edge fusion of chat models, 2024.
[49] Sheng Wang, Yuzhi Guo, Yuhong Wang, Hongmao Sun, and Junzhou Huang. Smiles- bert: Large scale unsupervised pre-training for molecular property prediction. In Proceedings of the 10th ACM International Conference Bioinformatics, Computational Biology and Health Informatics, BCB '19, page 429-436, New York, NY, USA, 2019. Association for Computing Machinery.
13

---

[50] Zijing Wang, Xingle Xu, Yongkang Liu, Yiqun Zhang, Peiqin Lin, Shi Feng, Xiaocui Yang, Daling Wang, and Hinrich Schütze. Why do more experts fail? a theoretical analysis of model merging, 2025.
[51] Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo- Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 23965-23998. PMLR, 17-23 Jul 2022.
[52] Fang Wu, Dragomir Radev, and Stan Z. Li. Molformer: Motif-based transformer on 3d heterogeneous molecular graphs. Proceedings of the AAAI Conference on Artificial Intelligence, 37(4):5312-5320, Jun. 2023.
[53] Tong Xie, Yuwei Wan, Wei Huang, Zhenyu Yin, Yixuan Liu, Shaozhou Wang, Qingyuan Linghu, Chunyu Kit, Clara Grazian, Wenjie Zhang, Imran Razzak, and Bram Hoex. Darwin series:
Domain specific large language models for natural science, 2023.
[54] Zhengqi Xu, Ke Yuan, Huiqiong Wang, Yong Wang, Mingli Song, and Jie Song. Training-fre free pretrained model merging, 2024.
[55] Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. Ties-merging: Resolving interference when merging models. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, vol- ume 36, pages 7093-7115 Curran Associates, Inc., 2023.
[56] Vikas Yadav, Steven Bethard, and Mihai Surdeanu. Quick and (not so) dirty: Unsuper- vised selection of justification sentences for multi-hop question answering. arXiv preprini arXiv:1911.07176. 2019.
[57] Enneng Yang, Li Shen, Guibing Guo, Xingwei Wang. Xiaochun Cao, Jie Zhang, and Dacheng Tao. Model merging in Ilms, milms, and beyond: Methods, theories, applications and opportu- nities, 2024.
[58] Enneng Yang, Li Shen, Zhenyi Wang, Guibing Guo, Xiaojun Chen, Xingwei Wang, and Dacheng Tao. Representation surgery for multi-task model merging, 2024.
[59] Botao Yu, Frazier N. Baker, Ziqi Chen, Xia Ning, and Huan Sun. LlaSMol: Advancing large language models for chemistry with large-scale, comprehensive, high-quality instruction tuning dataset. In First Conference on Language Modeling. 2024.
[60] Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models are super mario: Absorbing abilities from homologous models as a free lunch. In Forty-first International Conference on Machine Learning, 2024.
[61] Dan Zhang, Ziniu Hu, Sining Zhoubian, Zhengxiao Du Kaiyu Yang, Zihan Wang, Yisong Yue, Yuxiao Dong, and Jie Tang. Sciglm: Training scientific language models with self-refiective instruction annotation and tuning, 2024.
[62] Di Zhang, Wei Liu, Qian Tan, Jingdan Chen, Hang Yan, Yuliang Yan, Jiatong Li, Weiran Huang, Xiangyu Yue, Dongzhan Zhou, Shufei Zhang, Mao Su, Hansen Zhong, Yuqiang Li, and Wanli Ouyang. Chemllm: A chemical large language model, 2024.
[63] Zihan Zhao, Da Ma, Lu Chen, Liangtai Sun, Zihao Li, Yi Xia, Bo Chen, Hongshen Xu, Zichen Zhu, Su Zhu, Shuai Fan, Guodong Shen, Kai Yu, and Xin Chen. Developing chemdfim as a large language foundation model for chemistry. Cell Reports Physical Science, 6(4): 102523. 2025.
[64] Gengmo Zhou, Zhifeng Gao, Qiankun Ding, Hang Zheng, Hongteng Xu, Zhewei Wei, Linfeng Zhang, and Guolin Ke. Uni-mol: A universal 3d molecular representation learning framework. In The Eleventh International Conference Learning Representations, 2023.
14

---

[65] Jeffrey Zhou, Tianjian Lu. Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. cetructioo-followwpie evaluation for large language models, arXiv preprint urXiv:2311.0791/ 2023.
[66] Le Zhuo, Zewen Chi, Minghao Xu, Heyan Huang, Jianan Zhao, Heqi Zheng, Conghui He, Xian-Ling Mao, and Wentao Zhang. ProtLLM: An interleaved protein-language LLM with pptein-ass-oors pre-training. In Lun-Wet Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8950- 99008993,,050 Bangkok, Thailand, August 2024, Association for Computational Linguistics.
15

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No]. or [NA].
[NA] means either that the question Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] providedd proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] or "[NA]" is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
See abstract and section 1
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: section 6
16

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't I acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: See section
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: see section 4
Guidelines:
The answer NA means that the paper does not include experiments.
17

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We will open code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (nttps://nips.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ /ttppp.cc/pbllc/guideeee http://oeeuuiiiiiioooooooooooooo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
18

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer:
[Yes]
Justification: section 4
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:[Yes)
Justification: see section
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: see section
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
19

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: The reaserch conform ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
section 5
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
20

---

Released models that have high risk for misuse or dual use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: see Reference.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:[Yes)
Justification:
We well documented them.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
21

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the L.M used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: the core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy /ectps://neurr cc/ //onfffrencees22255555 for what should or should be described.
22

---

A
Large language models (LLMs) in the chemical domain
Table 8 presents a brief overview of recently published chemical LLMs.
Table 8: Summary of LLMs in the chemical domain
Model
Time
#Parameters
Base model
Dataset
Capability
MolXPT
33
2023.05
350M
GPT-2
PubChem, PubMed
Mol. und.
Crystal.LM
2023.07
GPT-2
MP, OQMD, NOMAD
Crystal gen. Sci.
DARWIN-MDP
531
2023.08
7B
LLaMA
SeiQ, FAIR
Mol-Instmattions
2023.11
7B/88
L
Mol- Instructions
Mol. gen.
ChemDFM [
2024.01
8B/13B
LLaMA-3/LLMM
Chemical literature, textbooks
Mol. und. Sci.
SciGLMM 61
2024.01
6B
ChatGLM3
Scilnstruct
Ch 62
2024.02
7B
InternLM22
ChemData and Multi-Corpus
Prop. pred.
LlaSMol 59
2024.02
7B
Mistral
SMolInstruct
Prop. pred.
ProLLaaMA 35
2024.02
LLaMA-2
UniRef50
Prot. und.
ProcLLM 66
2024.02
7B
LLaMA
InterPT
Prot. gen.
Meerkat
28
2024.04
7B/8B/70B
Mistral/L Mistta//LLaMA-33LLLMMM
18 medical textbooks ZINC, ChEMBL
Med.
DrugLLM
2024.05
7B
LLaMA
MolecularGiPT
32
2024,06
7B
I.LMA-2
QM9, ChEMBL SciL.itlns
Prop. pred. Sci.
Sc 30
2024.08
7B/144
Qwen2
KALE-LM
2024.09
Llama-3.1
Prop. pred.
LLAMAT
37
2024.12
7B/8B
LLaMA-2/LLaMAS
R2CID MuMOInstruct
Mater, pred.
GcLLMO
[
2025.02
7B/SB
Mistral-vO.3 10..al-0..///a-aa--3.
Mol. gen.
Mol-LLaMA
27
2025.02
8B
Llama-3.1
Mol-LLaMA-Instuurt
Mol. und.
B Comparison between CMM and other machine learning methods
To demonstrate the advantage of CMM, we provide the results of comparisons between CMM and other machine learning methods. We conducted an additional experiment where we supervised fine-tuned (SFT) Biima-3-8B-Instruue on the dev set of ChemBench and compared it with our CMM model. Due to the limited training data, the SFT model performed poorly, as shown in Table 9 We also compare CMM with GPT-4 40), leading closed-soure source model, based on results reported in prior work [62]. This comparison reveals that our CMM model achieves competitive performance with GPT-4 While GPT-4 remains stronger overall performance665.0 vs. 61 .3), CMM offers a compelling trade-off by delivering strong performance with zero training cost, full reproducibility, and no reliance on proprietary APIs or data.
Table 9: Comparison between CMM and other machine learning methods
Model
Chembench Avg.
SFT
19.19
GPT-4
65.89
CMM
61.43
C Influence of different score computation methods
In the main experiment, the overall performance score for each model is computed by first normalizing its scores across all tasks and then taking the average. Here we propose an alternative method for computing the overall performance scores. For each model, we first compute the average score within each benchmark, where each benchmark consists of multiple tasks. These per-benchmark averages are then normalized ensure comparability, and their mean is taken the overall performance score. Under this alternative computation scheme, the ranking of expert models from weakest to strongest is as follows: GeLLMO-P6-LLamm Keeping other settings unchanged, we apply the Curriculum Model Merging (CMM) procedure based on this order. For comparison, we also recompute the overall scores of the CMM obtained the main experiment using the new scoring method. The results are summarized in Table TO, This method of computing overall scores overlooks the variation the number of tasks across different benchmarks and therefore cannot accurately reflect the capabilities of each model. As result, the performance of
Meerkat-8B-vv 8L.0 SALF-LM-Cheem-L 1.5, and Molinst-Molecule--ee 8B.
23

---

the merged model obtained in this setting is inferior to that of the original one. Specifically, while the average score on hol-Instructiins remains similar, there is a notable drop performance on ChemBench.
Table 10: Ablation study on different score computation methods
Chembench Average
Mol-Instructions Average
Overall
Model
Average
Expert models
Alama-3-8B-Ime
45.93
0.07
26.47
Molinst-molecule-88
42.97
0.37
39.99
57.07
0.07
32.04
Meerkat-8b-vl.0
43.65
21.83
GeLLMO-P6
31.87
0.08
19.94
Merged model
CMM
61.43 53.02
0.35
48.22
CMM_Alee AltRank
0.38
45.51
D Relationship between the number of samples and results
Prouidds more details of the impact of the number of evaluation samples on the resulting eval- uation metrics. The evaluation metrics do not exhibit significant upward or downward trends beyond 200 samples. Therefore, we randomly selected 200 evaluation samples to improve computational efficiency.
SELFIES
Validity
1.0
1.0
Score
0.4
0.4
100
200
300
400
600
700
100
200
300
400
500
600
700
Sample Size.
Sample Size
Retrosynthesis
Forward Reaction Prediction
Figure Relationship between the number of evaluation samples and the resulting metries
E Scalability of CMM
E.1 General model set
CMM is generalizable and applicable beyond the chemical domain. We also applied CMM general- purpose LLMs and still achieved the best performance. In this experiment, we adopted the universal [18] the base model, and Hermes-3-LL 3-Lmmma-Lmmm------
-lama-3.1-8B. -B--tttutttt
[45). Llama-3.1-Tulee
3.1too-r----t------- 8B [3. and 3. 1..1-SuerrNova-Lit 2 as expert models, following this merging order. The performance the models was evaluated across 7 benchmarks spanning multiple domains, including general, instruction following, mathematics, and coding: MMLU 20]
8B 229].
IFEval 655) ARC-C (56), DROP [13, BBH 41), GSM8K (6) and Math [2] For comparison, we also included the model generated using the Task Arithmetic (TA) method.

---

The model merged using CMM outperforms all expert models and the task arithmetic baseline in terms of average score across the seven benchmarks. Detailed results are presented in Table
Table 11: CMM on general model set
Model
MML.U
IFEval
ARC-C DROP
BBH
GSM8K
Math
Avg
Expert models
1lama-3.1-40 Instructt
69.35
68.11
81.69
82.33
57.61
84.15
39.76
69.00
Hermes-3-1 3-lama-33 3.1-
64.29
59.59
82.03
77.73
62.25
81.50
25.20
64.66
66.26
75.18
66.10
76.68
61.18
87.87
40.02
67.61
Sauna-.1-Sttom---
68.87
72.18
81.69
79.01
56.01
82.94
35.60
68.04
IJama-3. --SuperNova-...it
69.29
69.90
84.07
81.84
60.10
83.40
36.58
69.31
Merge methods
TA
67.96
67.03
82.03
80.02
69.80
81.43
41.10
69.91
CMM
69.81
70.26
81.69
83.07
61.11
84.61
40.96
70.22
E.2
Larger and more diverse model set
To evaluate the scalability of CMM, we extended our original experiment by introducing two additional tasks-reasoning and math also included two high- beigh--erorming expert models on these tasks: 1 1.upe-rrrNoaaa-Lii [2] and Llama-3. --Tuuu-----E [29]. For comparison, we also included the model generated using the Task Arithmetic (TA) method.
DROP [3] and GSM8K 66, respectively. We
The performance of the six expert models and the CMM-merged model across 13 tasks is shown in Table 12 The CMM-merged model outperforms all expert models as well as the TA-merged model in terms of overall performance. This demonstrates the effectiveness of CMM when applied to larger and more diverse model sets.
When examining individual benchmarks, we observe that the CMM merged model not only inherits but also surpasses the best expert performance on Chembench, indicating strong domain adaptation ca- DROP, and GSM8K-CH
pabilities. However, the other three benchmarks MoM-MMoooeeeeee does not outperform the nop-eefforming expert models, suggesting that while CMM effectively aggregates knowledge, certain task-specific expertise may be partially diluted during the merging process. We would like to clarify that this phenomenon is not specific to CMM, but rather a common limitation across model merging methods. Prior research has shown that even state-of- the-art merging methods often experience performance saturation after merging no more than six expert models [50]. This highlights open challenge in the field and calls for further theoretical understanding and methodological advances to improve the scalability of model merging.
Table 12: CMM larger and more diverse model
Model
Chembench Avg
Mol-Instructions Avg
DROP
GSM8K
Overall Avg
Expert models
Llama-3-8B-Instie
45,93
0.07
18.63
79.38
40.41
GeLLMO-P6
31.87
0.08
81.91
84.53
36.10
Soorkat--b-v1..
43.65
0
74.3
44.81
39.38
Molinst-molecule-8h
42.97
0.37
24.15
62.62
42.12
SuperNova-8B
52.5
0
81.84
83.40
49,06
-lama-Tul--3-8B
53.28
0
76.68
87.87
49.54
KALF-4MM-Chem-1.5
57.07
0.07
56.72
67.63
50.15
Merge methods
TA
27.35
0.15
62.50
66. 19
31.14
CMM
63.80
0.19
67.84
79.15
58.40
25

---

Subspace Alignment Ratio
Subspace Alignment Ratio(SAR) 36 defined as:
|| ar Aelle
SAR(A, Ai kM)
where 4 denotes a task vector, AM the difference between the parameters of the merged model and those of the base model, and IR M is the projection matrix onto the subspace spanned by the top kas left-singular vectors of Av. The number of singular vectors used (kM) formulated as:
Ei of
km min k:
E²
(10)
Ei=1 of
where os denotes the singular values of AM, and 0.05.
26


Reviews:
[{"review_id": "co4LaFQjvU", "reviewer": "Reviewer_2XdM", "summary": "The paper proposes Curriculum Model Merging (CMM), a method for integrating multiple specialized chemical large language models (LLMs) into a unified model without requiring access to original training data or significant computational resources. CMM addresses challenges such as model disparities and imbalanced task coverage by progressively merging expert models in a curriculum-based manner. The method demonstrates superior performance, outperforming state-of-the-art merging techniques by 29.03% on average across two benchmark datasets. The paper highlights CMM's ability to generalize across predictive and generative tasks, even in expert-sparse scenarios", "questions": "1. Could the authors provide more theoretical analysis or intuition behind why the proposed curriculum merging order (weakest to strongest) is optimal? Are there scenarios where a different order might be more effective?\n2. The paper mentions reduced computational costs, but could the authors provide concrete metrics (e.g., training time or resource usage) comparing CMM to other merging methods?\n3. How might CMM perform in non-chemical domains (e.g., biology or materials science)? Are there domain-specific adjustments needed?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The paper proposes Curriculum Model Merging (CMM), a method for integrating multiple specialized chemical large language models (LLMs) into a unified model without requiring access to original training data or significant computational resources. CMM addresses challenges such as model disparities and imbalanced task coverage by progressively merging expert models in a curriculum-based manner. The method demonstrates superior performance, outperforming state-of-the-art merging techniques by 29.03% on average across two benchmark datasets. The paper highlights CMM's ability to generalize across predictive and generative tasks, even in expert-sparse scenariosStrengths:\n\n1. The curriculum-based merging approach is novel, particularly in the context of chemical LLMs. This work adapts and extends ideas from NLP to a domain with unique challenges.\n2. This work addresses a critical gap in the chemical domain by enabling the integration of specialized LLMs while preserving privacy and reducing computational costs. The results demonstrate substantial improvements over existing methods.\n3. This work presents a well-designed and thorough experimental evaluation, validating CMM's effectiveness across diverse chemical tasks. The ablation studies provide insights into the impact of merging order and weight coefficients.\n\nWeaknesses:\n\n1. The theoretical foundation for the merging process could be expanded to better justify the chosen curriculum strategy and weight assignments.\n2. Some technical details, such as the exact computation of normalized scores for ranking models, could be more explicitly described.\n3. The paper could further discuss the scalability of CMM to larger or more diverse model sets.", "was_revised": true, "final_justification": "After considering the rebuttal and discussions, I find that the authors have satisfactorily addressed my main concerns. Theoretical justification for the “weakest-to-strongest” curriculum order is now supported by SAR-based analysis and concrete examples; the computation of normalized scores is clarified; and additional experiments with larger, more diverse expert sets strengthen the scalability claim. Some limitations remain—performance drop with >10 experts and lack of quantitative efficiency comparison with training-based methods—but these have limited impact on the core contributions. Overall, the rebuttal meaningfully improves the paper’s clarity and technical soundness, justifying my increased score.", "rebuttal": "We sincerely thank the reviewer for providing valuable feedback. We detail our response below point by point. Please kindly let us know whether you have any further concerns.\n\n**W1 & Q1**: Theoretical analysis for the chosen curriculum strategy and weight assignments. Could the authors provide more theoretical analysis or intuition behind why the proposed curriculum merging order (weakest to strongest) is optimal? \n>We appreciated for your concern regarding the limited theoretical analysis of curriculum model merging strategy. We provide further theoretical analysis below. We use the **Subspace Alignment Ratio (SAR)**[1] to investigate how the expert models are integrated into the merged model during the merging process.\n\n>1.\n>  The SAR is used to quantify the alignment between the subspaces of two task matrices. SAR is defined as:\n   $$\n\\mathrm{SAR}(\\Delta_t, \\Delta_M; k_M) = \\frac{ \\left\\| \\Pi_{k_M, M} \\Delta_t \\right\\|_F }{ \\left\\| \\Delta_t \\right\\|_F },\n$$ \n\n>    where $\\Delta_t$ denotes a task vector, $\\Delta_M$ is the difference between the parameters of the merged model and those of the base model, and $\\Pi_{k_M, M}$ is the projection matrix onto the subspace spanned by the top $k_M$ left-singular vectors of $\\Delta_M$. The number of singular vectors used ($k_M$) is formulated as:\n$$\nk_M = \\min \\{ k : \\frac{ \\sum_{i=k+1}^{r} \\sigma_i^2 }{ \\sum_{i=1}^{r} \\sigma_i^2 } \\leq \\epsilon^2 \\}\n$$\n\n>  where $\\sigma_i$ denotes the singular values of $\\Delta_M$, and $\\epsilon$ = 0.05.\n\n2.\n>  **A higher SAR value indicates a greater overlap between the subspaces, meaning that the merged model better inherits the capabilities of the expert model.** We observe that as more expert models are merged later in the process, the SAR values of those merged earlier tend to slightly decrease. Additionally, higher assigned weights are generally associated with higher SAR values. This suggests that placing better-performing expert models later in the merging order, and assigning them higher weights, leads to higher SAR values.\n\n3.\n>  For better intuition, we present an example below. The table shows the evolution of the SAR value for **Molinst-molecule-8B** during the CMM merging process. This expert model was merged at **step 3**, where its SAR value increased significantly. At **step 4**, the SAR value slightly decreased due to the merging of KALE-LM-Chem-1.5. When we reduced the weight assigned to Molinst-molecule-8B at step 3, we observed a corresponding decrease in its SAR value.\n>   \n>   | Metric   | Step1 | Step2 | Step3 | Step4 | Step3（reduced weight） |\n>   | -------- | ------ | ------ | ------ | ----- | ----- |\n>   | SAR   | 0.042   | 0.042   | 0.650   | 0.647  | 0.506 |\n>\n>[1] Marczak et al. \"No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspaces.\"\n\n\n**W2**: Some technical details, such as the exact computation of normalized scores for ranking models, could be more explicitly described.\n>Thank you for this question.\n> - We have clarified the computation details of normalized scores in **Section 4.2**. *“To enable a unified comparison of model performance across the two benchmarks, we normalize the round-trip accuracy scores from Mol-Instructions by rescaling them to a 0–100 range.”*\n> - This normalization is necessary because ChemBench scores are already reported on a 0–100 scale, whereas the Mol-Instructions round-trip accuracy scores are originally on a 0–1 scale. To ensure **fair averaging across benchmarks**, we rescale the Mol-Instructions scores to the 0–100 range, enabling a consistent comparison with ChemBench.\n\n\n**W3 & Q3**: The paper could further discuss the scalability of CMM to larger or more diverse model sets.\n>We thank this insightful question. \n> - We present an additional experiment using **a larger and more diverse collection of expert models**.\n>   - To evaluate the scalability of CMM, we extended our original experiment by introducing two additional tasks—**reasoning** and **math**—evaluated on **DROP**[1] and **GSM8K**[2], respectively. We also included two high-performing expert models on these tasks: **SuperNova-8B**[3] and **Llama-Tulu-3-8B**[4]. For comparison, we also included models generated using the Task Arithmetic (TA) method.\n>   - The performance of the six expert models and the CMM-merged model across 13 tasks is shown in the table below. The CMM-merged model **outperforms** all expert models as well as the TA-merged model in terms of overall performance. This demonstrates the effectiveness of CMM when applied to larger and more diverse model sets.\n> \n> | Model | Chembench Average | Mol-Instructions Average | DROP | GSM8K | Overall Average |\n> | :-: | :-: | :-: | :-: | :-: | :-: |\n> | Llama-3-8B-Instruct | 45.93 | 0.07 | 18.63 | 79.38 | 40.41 |\n> | GeLLMO-P6 | 31.87 | 0.08 | 81.91 | 84.53 | 36.10 |\n> | Meerkat-8b-v1.0 | 43.65 | 0 | 74.3 | 44.81 | 39.38 |\n> | Molinst-molecule-8b  | 42.97 | 0.37 | 24.15 | 62.62 | 42.12 |\n> | SuperNova-8B  | 52.5 | 0 | 81.84 | 83.40 | 49.06 |\n> | Llama-Tulu-3-8B  | 53.28 | 0 | 76.68 | 87.87 | 49.54 |\n> | KALE-LM-Chem-1.5  | 57.07  | 0.07 | 56.72 | 67.63 | 50.15 |\n> | TA  | 27.35  | 0.15 | 62.50 | 66.19 | 31.14 |\n> | CMM  | 63.80  | 0.19 | 67.84 | 79.15 | 58.40 |\n>\n> - When merging a large number of expert models (e.g., more than 10), we observe that performance tends to degrade. However, we would like to clarify that this phenomenon is **not specific to CMM**, but rather a **common limitation** across model merging methods. Prior research has shown that even state-of-the-art merging methods often experience **performance saturation after merging no more than six expert models** [5]. This highlights an open challenge in the field and calls for further theoretical understanding and methodological advances to improve the scalability of model merging.\n> \n>[1] Dheeru et al. \"Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs.\"\n>\n>[2] Karl et al. \"Training veriffers to solve math word problems.\"\n>\n>[3] Lucas et al. \"Arcee-SuperNova: Training Pipeline and Model Composition.\"\n>\n>[4] Nathan et al. \"Tulu 3: Pushing Frontiers in Open Language Model Post-Training.\"\n>\n>[5] Wang et al. \"Why Do More Experts Fail? A Theoretical Analysis of Model Merging. \"\n\n\n**Q2**: The paper mentions reduced computational costs, but could the authors provide concrete metrics (e.g., training time or resource usage) comparing CMM to other merging methods?\n>Thank you for this question.\n> - CMM, as well as all other merging methods in baselines are all **training-free**. The merging process takes only **a few minutes**.\n>   - The *“without computational costs”* mentioned in the paper refers to the comparison between model merging and other training-based methods such as knowledge distillation and supervised fine-tuning (SFT). Model merging integrates only homogeneous models and does not require any training or data, as it simply involves combining model parameters."}, {"review_id": "bHb6lzFrLE", "reviewer": "Reviewer_j7J5", "summary": "This paper first identifies some characteristics of chemical LLMs, including the prevalence of in-house training data in the chemical domain and significant disparities in parameters and downstream functionality, which highlight the necessity of novel model merging methods. Therefore, they propose Curriculum Model Merging (CMM), a method that constructs curriculum to progressively merge the expert chemical LLMs in a moderate and continual manner. Their comprehensive experiments on two benchmark datasets show that the proposed method concentrates task-specific expertise and outperforms the state-of-the-art methods by 29.03% in terms of an overall average performance score.", "questions": "See weaknesses.", "limitations": "yes", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n(1) This paper identifies a critical problem of merging chemical LLMs due to prevalence of in-house training data in this domain. \n\n(2) The authors propose a model merging method to tackle this problem.\n\n(3) Extensive experiments are conducted to demonstrate the effectiveness of the proposed method.\n\n**Weaknesses**\n\n(1) The experimental setting, especially those of baseline models, are not clearly described. In my understanding (with some of my best guesses), the Expert Models (e.g. Table 1) are actually evaluated by their zero-shot performance on ChemBench. The SOTA merge methods are evaluated similarly on their out-of-domain performance on ChemBench, as none of the expert models used training information from Chembench before testing. On the contrary, only the proposed CMM method gets access to Chembench information through the $S_i$ score. I would like to confirm with the authors that the understanding above is correct.\n\n(2) If so, the comparison to all baselines in this paper seems unfair. The baselines in this paper such as TA and TIES are actually not designed for scenarios like this. These methods serve as lightweight methods with generalizability to multiple tasks, while the CMM is constructing task-specific models which are tailored to a “selected benchmark suite”.\n\n(3) Of course, the application scenario proposed by the authors might be useful. But in such cases, the baselines to compare should not be limited to those model merging methods shown in this paper. For example, what is the performance of directly finetuning a LLM on Chembench, and what about directly using a cutting edge LLM like GPT-4o with in-context learning? In other words, this paper proposes to use model merging to obtain high performance on chemical tasks like Chembench, then it is necessary to show advantage of CMM to all kinds of machine learning methods. Only comparing CMM to model merging methods looks insufficient to me.", "was_revised": true, "final_justification": "The response mostly addresses my concerns. I therefore updated my score to 3.", "rebuttal": "We appreciate very much your constructive comments on our paper. Please kindly find our response to your comments below. We hope that our response satisfactorily addresses the issues you raised. Please feel free to let us know if you have any additional concerns or questions.\n\n**W1 & W2**: The experimental setting, especially those of baseline models, are not clearly described. The comparison to all baselines in this paper seems unfair. \n>Thank you for the insightful question regarding fairness.\n> - We use the **dev set of ChemBench** and **100 samples from Mol-Instructions** as our **validation set**, which is completely **disjoint** from the **test set**. The $S_i$ scores are computed based on the validation set. Consistent with the baseline models, the CMM-merged model does **not** have access to any information from the test set.\n> - The merged models produced by the SOTA methods get access to **the same information** as the CMM-merged model. \n>   - Our experimental process can be divided into two stages: **ranking** and **merging**. In the **ranking** stage, expert models are ranked based on their performance on the **validation set**, and corresponding weights are assigned accordingly. The **merging** stage then combines the expert models in the parameter space using different merging methods (e.g., CMM, TA, TIES, DARE_TA, SCE, AF).\n>   - The comparison is **fair and consistent** for both the SOTA merging methods and CMM, as each method receives the **same ranking information** derived from the validation set. More specifically, the TA, TIES, DARE_TA, and SCE methods also require assigning a weight to each expert model. The AF method requires specifying a merging order. In all cases, the weights or orders used are the same as those used by CMM.\n\n\n**W3**: The baselines to compare should not be limited to those model merging methods shown in this paper. In other words, this paper proposes to use model merging to obtain high performance on chemical tasks like Chembench, then it is necessary to show advantage of CMM to all kinds of machine learning methods.\n>We sincerely thank the reviewer for this insightful suggestion. \n> - **To address the point about stronger baselines**, we conducted an additional experiment where we **fine-tuned** Llama-3-8B-Instruct on the dev set of ChemBench and compared it with our CMM-merged model. Due to the limited training data, the fine-tuned model (SFT) performed poorly:\n>   | Model   | Chembench Average |\n>   | -------- | ------ |\n>   | sft model   | 19.19   |\n> - We also compare CMM with GPT-4, a leading closed-source model, based on results reported in prior work[1]:\n>   - Note that all results on ChemBench, including those for GPT-4 and our CMM-merged model, are obtained under the same **5-shot in-context learning** setting without any fine-tuning. This ensures a fair comparison across all methods.\n>   - This comparison reveals that while our CMM-merged model—constructed from **open-source** expert models without any additional training—achieves **competitive performance** on several tasks (e.g., NC, M2C, RS, TP). While GPT-4 remains stronger overall (65.89 vs. 61.43), CMM offers a compelling trade-off by delivering strong performance with **zero training cost, full reproducibility**, and **no reliance on proprietary APIs or data**.\n>   | Model  | NC    | Property_P | M2C   | C2M   | Product_P | RS    | YP    | TP    | SP    | Average |\n>   |--------|-------|------------|-------|-------|-----------|-------|-------|-------|-------|---------|\n>   | GPT-4  | 55    | 68         | 95    | 64    | 88        | 72    | 45    | 59    | 47    | 65.89   |\n>   | CMM    | 63.95 | 35.40      | 92.98 | 54.37 | 78.33     | 72.33 | 46.00 | 64.85 | 44.67 | 61.43   |\n> - Beyond its efficiency, model merging offers a key structural advantage. CMM operates in the parameter space, and is **orthogonal to the expert models themselves**—stronger expert models naturally lead to stronger merged models. This design decouples capability acquisition (via expert models) from integration (via merging), making CMM especially promising as open-weight models continue to improve.\n> \n>[1] Zhang et al. \"Chemllm: A chemical large language model.\""}, {"review_id": "EJWPz0vLrc", "reviewer": "Reviewer_1evb", "summary": "This paper presents Curriculum Model Merging (CMM), a novel framework for combining multiple, specialized Large Language Models (LLMs) in the chemical domain into a single, more versatile model. The authors identify that directly merging chemical LLMs is challenging due to significant disparities among models fine-tuned for specific tasks and an imbalanced distribution of expertise across the field.\nThe core idea of CMM is to merge models progressively, following a curriculum. The key contributions of the paper are --\n\nA Novel Merging Framework: CMM first evaluates and ranks the individual expert models based on their performance. It then iteratively merges them, starting from the weakest-performing model and gradually incorporating stronger ones. This step-by-step process aims to harmonize the models, reduce parameter interference, and preserve specialized knowledge.\n\nState-of-the-Art Performance: The paper demonstrates through extensive experiments that CMM significantly outperforms existing model merging methods. On a combined set of predictive and generative tasks, CMM achieves an overall score 29.03% higher than the next best baseline method.\n\nRobust Generalization: The proposed method proves effective in two distinct scenarios: an \"expert-abundant\" setting with many models available and an \"expert-sparse\" setting where only one model possesses the critical expertise for a task. CMM successfully consolidates knowledge for both predictive and generative tasks without significant performance degradation.\n\nPractical Application: This work is the first to systematically adapt model merging for the chemical domain, offering a practical, privacy-preserving solution that does not require access to original training data or incur substantial computational costs", "questions": "1. On the Generalizability of CMM to Architecturally Diverse Models --\nYour experiments compellingly demonstrate CMM's effectiveness when merging models from the LLaMA family (LLaMA-2, LLaMA-3, Mistral). This architectural homogeneity raises a question about the broader applicability of the CMM framework. Could you discuss the anticipated challenges or necessary modifications for applying CMM to a more architecturally diverse set of models? For instance, what would be the implications of merging a traditional transformer-based LLM with a model based on a graph neural network, another common architecture in the chemical domain?\n\n2. Providing Concrete Evidence for Synergistic Knowledge Generalization --\nThe paper makes the exciting claim that the CMM-merged model \"does not merely inherit expert knowledge\" but \"appears to generalize relevant patterns beyond individual model boundaries\". This suggests a synergistic effect where the whole is greater than the sum of its parts. To make this powerful claim more concrete, could you provide a few qualitative examples from your test set?\n\n3. Regarding the Robustness of the Single-Metric Ranking Curriculum --\nThe curriculum in CMM is constructed based on a single, overall performance score that is aggregated across multiple tasks. While your ablation study explores different score computation methods, it still relies on a single final metric for ranking. Have you considered scenarios where this single metric might be insufficient? For example, a model could have a low overall score but be the only expert on a rare but critical task. How would CMM ensure this vital, niche capability is not diluted or lost when it is merged early in the curriculum as a \"weaker\" model?\n\n4. Elucidating the Choice of Merging Weight Hyperparameters --\nYour ablation study on the merging weight distribution (constant, linear, exponential) is very informative. The paper states that the best-performing linear strategy used coefficients that increased from 0.3 to 0.6. Could you provide more intuition on how sensitive the model's performance is to this specific range? How was this range determined?", "limitations": "1. Discussion of Methodological and Experimental Limitations --\nA dedicated limitations section should be added to address the following points:\nArchitectural Homogeneity: The current study exclusively uses expert models from the LLaMA family. The authors should acknowledge that the effectiveness of CMM on architecturally diverse models (e.g., merging a transformer with a graph neural network) has not been tested and may present unforeseen challenges.\n\nScalability of Sequential Merging: The experiments involve merging four expert models. The paper should discuss the potential scalability of the sequential CMM approach. As the number of expert models grows into the dozens or hundreds, the linear, one-by-one merging process could become a computational bottleneck, and potential \"catastrophic forgetting\" of the earliest merged models might become more pronounced.\n\nRobustness of the Curriculum Ranking: The curriculum is based on a single aggregated performance score. The authors should discuss the potential brittleness of this approach. For example, a model that is weak overall but possesses a unique, critical capability might be merged too early, risking the dilution of its rare knowledge by subsequent, stronger models.\n\nHyperparameter Sensitivity: The paper shows a linear weighting strategy works best but could benefit from a discussion on the sensitivity and tuning process for the weight coefficients (e.g., the 0.3 to 0.6 range). This would provide a more complete picture of the practical implementation of CMM.\n\n2. Discussion of Potential Negative Societal Impact\n\nThe development of more powerful and versatile chemical language models carries potential risks that warrant discussion:\n\nDual-Use Applications: A model that effectively consolidates knowledge from disparate chemical domains could be a powerful tool for beneficial research like drug discovery. However, it could also be misused to accelerate the design of harmful substances, such as novel toxins, illicit drugs, or chemical weapons. The authors should acknowledge this dual-use potential as a significant societal risk associated with creating more powerful, generalist chemical AI.\n\nOver-Reliance and De-skilling: The availability of a highly capable, \"all-in-one\" chemical LLM could lead to over-reliance by researchers and practitioners. This could potentially de-skill human chemists or stifle creative, out-of-the-box human problem-solving. It also raises the risk of significant errors if the model's failure modes are not well-understood by its users.\n\nAccessibility and Malicious Use: The CMM method is designed to be practical and efficient. This accessibility means that malicious actors could potentially use it to merge their own privately-trained expert models to create a powerful tool for nefarious purposes, bypassing the safeguards of public models. A brief discussion on the importance of responsible model governance would be appropriate.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths -----\nQuality: The paper demonstrates high quality through its rigorous and comprehensive experimental validation. The authors use two distinct benchmarks, Chembench and Mol-Instructions, which together cover 11 different chemical tasks, spanning both prediction and generation. The inclusion of extensive ablation studies systematically validates the core components of the CMM framework; for example, the performance drops significantly from 56.62 to 41.67 when the merging order is reversed, confirming the curriculum's importance. Furthermore, the comparison against multiple relevant and high-performing baseline methods like TA, TIES, and DARE_TA provides strong evidence for CMM's superior performance.\n\nClarity: The paper is clear and well-written. The core concepts are introduced logically, and the methodology is presented with clear mathematical formalism. A major strength is the use of visualizations to aid understanding. Figure 1 provides an excellent intuitive illustration of the CMM process compared to standard methods , while Figure 2 effectively visualizes the performance gains at each step of the merging curriculum, making a compelling case for the chosen merging order.\n\nSignificance: The work addresses a problem of high practical significance: creating a single, versatile chemical LLM from multiple specialized models without needing the original data or incurring massive computational costs. The demonstrated success of CMM in both \"expert-abundant\" (Chembench) and \"expert-sparse\" (Mol-Instructions) scenarios underscores its robustness and broad utility. The finding that CMM can preserve and even generalize rare knowledge—as seen in its performance on retrosynthesis where only one expert model was competent—is a particularly significant result. The overall performance improvement of 29.03% over the best SOTA baseline is substantial and impactful.\n\nOriginality: The paper's primary originality lies in being the first to systematically adapt and tailor model merging techniques to the unique challenges of the chemical LLM domain. The authors originally identify the key problems of high model disparity and imbalanced task coverage in chemistry. The proposed solution, framing the merging process as a curriculum where models are progressively integrated based on their capability, is a novel and insightful contribution that directly addresses these challenges.\n\n\nWeaknesses -----\nQuality: A potential critique of the paper's quality could center on the homogeneity of the models used. All the expert models chosen are based on the LLaMA architecture (LLaMA-2, LLaMA-3, Mistral). An argument could be made that the method's effectiveness has not been demonstrated on a more diverse set of model architectures, leaving its generalizability beyond LLaMA-like models in question. Additionally, the entire curriculum is constructed based on a single aggregated performance score, which could be a point of weakness if a model performs poorly on average but possesses unique, valuable capabilities on a niche task not well-represented in the overall score.\n\nClarity: While the paper's results are clear, the analysis of why the merged model generalizes could be deeper. The paper makes the exciting claim that the CMM-merged model \"does not merely inherit expert knowledge\" and \"appears to generalize relevant patterns\". However, this is not substantiated with qualitative examples or deeper analysis, leaving the inner workings of the knowledge fusion somewhat opaque.\n\nSignificance:  As all experiments and validation are confined to the chemical domain, the paper does not provide evidence that the CMM approach would be beneficial for merging models in other domains like computer vision or robotics, where the nature of expert specialization might differ.\n\nOriginality: An argument could be made that the paper's originality is somewhat limited, as it combines two existing lines of research: task-vector-based model merging (like Task Arithmetic) and curriculum learning. From this perspective, the core contribution could be framed as a clever and effective application of known concepts to a new domain, rather than the invention of a fundamentally new merging paradigm.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you sincerely for your thoughtful and positive feedback on our work. We are particularly grateful for your recognition of the various aspects of our research. Below, we have provided a detailed explanation for your remaining concern as follows. Please do not hesitate to let us know if you have any further questions.\n\n**Q1**:Could you discuss the anticipated challenges or necessary modifications for applying CMM to a more architecturally diverse set of models? \n> We appreciate your inquiry about merging architecturally diverse set of models.\n> - CMM belongs to the category of model merging, which can only integrate models with the same architecture. However, model fusion methods could be explored to combine an architecturally diverse set of models.\n> - Approaches for integrating multiple model capabilities into a single model can be categorized into two types. \n>   - The first is **model fusion**, which integrates **heterogeneous** models in a manner similar to knowledge distillation, requiring both training and data during the process. \n>   - The second is **model merging**, which combines only **homogeneous** models and does not require any training or data, as it simply involves combining model parameters.\n\n**Q2**: The paper makes the exciting claim that the CMM-merged model \"does not merely inherit expert knowledge\" but \"appears to generalize relevant patterns beyond individual model boundaries\". To make this powerful claim more concrete, could you provide a few qualitative examples from your test set?\n> We sincerely thank the reviewer for this valuable suggestion. In the benchmark, there are cases where none of the expert models answered a question correctly, but the CMM-merged model produced the correct answer. Below is one such example.\n> - Question: What substances are commonly selected for the synthesis of NC(=O)c1ccc(N2CCN(Cc3cn4cc(Cl)ccc4n3)CC2)c(Cl)c1 ?\n>A. There's a likelihood of reactions happening, with COc1ccc(-c2cc(C)cc(NC(=O)C3(c4ccc5c(c4)OC(F)(F)O5)CC3)n2)c(C)n1.Cl>C1COCCO1>NC(=O)c1ccc(N2CCN(Cc3cn4cc(Cl)ccc4n3)CC2)c(Cl)c1.\n>B. Undergoing reactions is a possibility, and N#Cc1ccc(N2CCN(Cc3cn4cc(Cl)ccc4n3)CC2)c(Cl)c1.[OH-].[Na+].CC(C)(C)O.CO>OO.ClCCl>NC(=O)c1ccc(N2CCN(Cc3cn4cc(Cl)ccc4n3)CC2)c(Cl)c1.\n>C. Components used in the production of COc1cccc2c1nc(C(F)F)n2-c1nc(N2CCOCC2)nc(N(CCCN(C)C)C2CCN(S(=O)(=O)CCl)CC2)n1.\n>D. The potential for reactions to materialize exists, with C/C(=N\\O)c1ccc2c(c1)B(O)OC2(C)C>CC(=O)O.[Zn]>NC(=O)c1ccc(N2CCN(Cc3cn4cc(Cl)ccc4n3)CC2)c(Cl)c1.\n\n>   - The correct answer is B, while the expert models predicted C, C, A, and A, respectively. In contrast, the CMM-merged model selected the correct answer B, demonstrating that our merged model not only inherits knowledge from the experts, but also makes more accurate and informed decisions.\n\n**Q3**: A single final metric for ranking might be insufficient? For example, a model could have a low overall score but be the only expert on a rare but critical task. How would CMM ensure this vital, niche capability is not diluted or lost when it is merged early in the curriculum as a \"weaker\" model?\n> Thank you for your question. We agree that rare but critical capabilities should be preserved during the merging process, even if they come from expert models with relatively low overall scores.\n> - To address this question, we plan to adjust the ranking strategy by assigning **higher priority** to such expert models in order to prevent their capabilities from being merged too early and potentially diluted.\n> - Whether an early-merged rare capability is preserved also depends on its **orthogonality to the capabilities introduced by later models**. If the rare capability is largely independent of those introduced later, it tends to be less affected and better preserved in the merged model.\n\n\n**Q4**: The paper states that the best-performing linear strategy used coefficients that increased from 0.3 to 0.6. Could you provide more intuition on how sensitive the model's performance is to this specific range? How was this range determined?\n>We appreciate your inquiry about the impact of the merging coefficient $\\beta$, and will include the following sensitivity analysis in the updated submission.\n> - We draw inspiration from a conclusion in task arithmetic: “Scaling coefficients in the range 0.3 to 0.5 produce close to optimal results in many cases” [1].\n> - In our experiments, we found that **slightly increasing** the coefficient could lead to **marginal performance improvements**, whereas **slightly decreasing** it often resulted in a **noticeable drop** in performance. Based on these observations, we decided to select coefficients from the range [0.3, 0.6], which is close to the originally suggested [0.3, 0.5] interval.\n>   | Range   | Chembench Average | Mol-Instructions Average |  Overall Average |\n>   | -------- | ------ | ------ |  ----- |\n>   | [0.4,0.7]   | 60.38   | 0.41   |  56.86 |\n>   | [0.3,0.6]   | 61.43   | 0.35   |  56.62 |\n>   | [0.2,0.5]   | 61.03   | 0.24   |  54.21 |\n>\n>[1]Ilharco et al. \"Editing Models with Task Arithmetic.\"\n\n\n**Significance**: As all experiments and validation are confined to the chemical domain, the paper does not provide evidence that the CMM approach would be beneficial for merging models in other domains.\n> We appreciate the reviewer’s thoughtful comment.\n> - CMM is **generalizable** and **applicable** beyond the chemical domain. We also applied CMM to **general**-purpose LLMs and still achieved the best performance, as outlined below.\n>   - We adopted the universal **Meta-Llama-3.1-8B-Instruct**[1] as the base model, and used **Hermes-3-Llama-3.1-8B**[2], **Llama-3.1-Tulu-3-8B**[3], **Llama-3.1-Storm-8B**[4], and **Llama-3.1-SuperNova-Lite**[5] as expert models.\n>   - The performance of the models was evaluated across 7 benchmarks spanning multiple domains, including **general**, **instruction following**, **mathematics**, and **coding**. For comparison, we also included models generated using the Task Arithmetic (TA) method.\n>   - The model merged using CMM **outperforms** all expert models and the task arithmetic baseline in terms of average score across the seven benchmarks.\n>   | Model                             | MMLU  | IFEval | ARC-C | DROP  | BBH   | GSM8K | Math  | Avg   |\n>|----------------------------------|-------|--------|-------|-------|-------|-------|--------|--------|\n>| Meta-Llama-3.1-8B-Instruct       | 69.35 | 68.11  | 81.69 | 82.33 | 57.61 | 84.15 | 39.76 | 69.00 |\n>| Hermes-3-Llama-3.1-8B            | 64.29 | 59.59  | 82.03 | 77.73 | 62.25 | 81.50 | 25.20 | 64.66 |\n>| Llama-3.1-Tulu-3-8B_hf           | 66.26 | 75.18  | 66.10 | 76.68 | 61.18 | 87.87 | 40.02 | 67.61 |\n>| Llama-3.1-Storm-8B               | 68.87 | 72.18  | 81.69 | 79.01 | 56.01 | 82.94 | 35.60 | 68.04 |\n>| Llama-3.1-SuperNova-Lite         | 69.29 | 69.90  | 84.07 | 81.84 | 60.10 | 83.40 | 36.58 | 69.31 |\n>| TA                               | 67.96 | 67.03  | 82.03 | 80.02 | 69.80 | 81.43 | 41.10 | 69.91 |\n>| CMM                              | 69.81 | 70.26  | 81.69 | 83.07 | 61.11 | 84.61 | 40.96 | **70.22** |\n>\n>[1] Team Llama et al. \" The llama 3 herd of models.\"\n>\n>[2] Teknium et al. \"Hermes 3 Technical Report.\"\n>\n>[3] Nathan et al. \"Tulu 3: Pushing Frontiers in Open Language Model Post-Training.\"\n>\n>[4] Ashvini Kumar Jindal. \"Llama-3.1-Storm-8B: Improved SLM with Self-Curation + Model Merging.\"\n>\n>[5] Lucas et al. \"Arcee-SuperNova: Training Pipeline and Model Composition.\"\n\n**Scalability**: The paper should discuss the potential scalability of the sequential CMM approach. As the number of expert models grows into the dozens or hundreds, the linear, one-by-one merging process could become a computational bottleneck, and potential \"catastrophic forgetting\" of the earliest merged models might become more pronounced.\n>Thank you for raising this important point regarding the scalability of the sequential merging process in CMM.\n> - We would first like to clarify that the performance degradation when merging a large number of expert models is **not specific to CMM or its sequential nature**, but rather a common challenge observed in model merging research. Some researchers have observed that the performance of even state-of-the-art model merging methods tends to **saturate after merging no more than six expert models**[1].\n> - In fact, **most existing studies on model merging typically involve only a small number of expert models**. For instance, Dai et al. merged one expert model each from the mathematics, coding, and translation domains [2], while CABS merging combined only two general-domain models [3]. To the best of our knowledge, there has not yet been a practical scenario that requires merging dozens or hundreds of large-scale expert models.\n> - We agree that scaling model merging to a large number of models raises important challenges, including potential \"catastrophic forgetting\" and optimization instability. We view this as a **broader open problem in the model merging domain**, not limited to sequential merging. Developing theoretical tools and practical strategies to mitigate such issues is an important direction for future work.\n> \n> [1] Wang et al. \"Why Do More Experts Fail? A Theoretical Analysis of Model Merging. \"\n> [2] Dai et al. \"Leveraging Submodule Linearity Enhances Task Arithmetic Performance in LLMs.\"\n> [3] Yang et al. \"CABS: Conffict-Aware and Balanced Sparsiffcation for Enhancing Model Merging. \""}, {"review_id": "F2SXDkATqa", "reviewer": "Reviewer_Chme", "summary": "In this paper, the authors introduce a novel model merging approach known as Curriculum Model Merging (CMM), which progressively integrates expert chemical Large Language Models (LLMs) in a gradual and continuous manner. This method comprises two primary steps: First, the models are ranked according to their capabilities, and a merge order is established. Second, an iterative merging procedure is executed. The experimental results on ChemBench and Mol-instructions demonstrate the effectiveness of the proposed approach.", "questions": "1. The notation $\\lambada$ and $\\tau_{new}$ in Sec.3.1 should be clearly described.\n2. According to Tab. 1, the capabilities of Meerkat and GeLLMO are both worse than the base model LLaMA-3. Why were they selected for merging in the experiments? Would the merged model perform better if they were excluded?\n3. It can be observed in Tab. 4 that the performance on ChemBench improves after removing MolInst-Molecule and Meerkat. More discussion on this observation would be beneficial.\n4. There are some typos. For instance, \"Tab. 4.2\" in lines 170 and 284 should be corrected to \"Tab. 3.\" The authors are advised to thoroughly review the paper for consistency and accuracy.", "limitations": "The authors claim to have discussed the limitations in Section 5. However, the reviewer did not notice any such discussion in the conclusion section. It would be beneficial to include a clear discussion of the limitations in the appropriate section, such as the conclusion or a dedicated limitations section.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "The paper is well-organized, and the proposed approach is technically sound. Model merging is an interesting yet challenging task that enables the integration of multiple expert LLMs into a unified model without requiring access to the original training data or incurring additional computational costs. The proposed approach is simple but effective for practical applications.\n\nHowever, the reviewer still has some concerns. The notation in Sec. 3.1 should be clearly described, for example, $\\lambda$ and $\\tau_{new}$. It would be beneficial to provide more in-depth discussion on the experimental results. Additionally, the proposed approach has only been evaluated on LLaMA-3 based models for chemical tasks. The paper should be carefully reviewed to avoid minor errors.", "was_revised": true, "final_justification": "After considering the rebuttal and discussions, I find that the experimental results address my concerns to a large extent. I have carefully read the authors' response as well as the discussion with other reviewers. The rebuttal meaningfully improves the paper’s clarity and technical soundness. I have decided to raise the score.", "rebuttal": "We appreciate very much your constructive comments on our paper. Please kindly find our response to your comments below. We hope that our response satisfactorily addresses the issues you raised. Please feel free to let us know if you have any additional concerns or questions.\n\n**Q1**: The notation $\\lambda$ and $\\tau_{new}$ in Sec.3.1 should be clearly described.\n>Thank you for pointing this out.\n> - $\\lambda$ is an optional scaling term, it is determined using validation sets.\n> - $\\tau_{new}$ is the sum of all task vectors.\n\n**Q2**: The capabilities of Meerkat and GeLLMO are both worse than the base model LLaMA-3. Why were they selected for merging in the experiments? Would the merged model perform better if they were excluded?\n>Thank you for the insightful question.\n> - The merged model **wouldn't perform better** if they were excluded. The result is following:\n>   | Model   | Chembench Average | Mol-Instructions Average |  Overall Average |\n>   | -------- | ------ | ------ |  ----- |\n>   | Merged model(without GeLLMO and Meerkat)   | 61.41   | 0.35   |  56.52 |\n> - Excluding Meerkat and GeLLMO results in an overall score of **56.62**, which is slightly lower than that of the original merged model. Although Meerkat and GeLLMO perform relatively poorly on the selected benchmarks, they carry complementary chemical knowledge. This additional knowledge contributes to the construction of chemical understanding in the merged model.\n> - Similarly, previous research has demonstrated that **\"strong student models can benefit from learning even from weaker teacher models\"**[1].\n>\n>[1] Hunter et al. \"Theoretical Analysis of Weak-to-Strong Generalization.\"\n\n**Q3**: The performance on ChemBench improves after removing MolInst-Molecule and Meerkat. More discussion on this observation would be beneficial.\n>Thanks for your valuable comment regarding the issue of conflicts in model merging.\n> - In model merging, it is often difficult to avoid interference with previously acquired capabilities when new ones are introduced. This is because **key parameters related to earlier capabilities may be modified during the merging process**. In our experiments, the CMM-merged model successfully acquired the ability to solve Mol-Instructions tasks, while the performance on ChemBench experienced only a slight drop. We consider this trade-off acceptable, as **it reflects a balance aimed at optimizing the model's overall performance**.\n\n**Q4**: There are some typos. For instance, \"Tab. 4.2\" in lines 170 and 284 should be corrected to \"Tab. 3.\"\n> Thank you for pointing this out. We have carefully reviewed the paper and corrected all these typos.\n\n**W1**: The proposed approach has only been evaluated on LLaMA-3 based models for chemical tasks.\n> We appreciate the reviewer’s thoughtful comment.\n> - CMM is **generalizable** and **applicable** beyond the chemical domain. We also applied CMM to **general**-purpose LLMs and still achieved the best performance, as outlined below.\n>   - We adopted the universal **Meta-Llama-3.1-8B-Instruct**[1] as the base model, and used **Hermes-3-Llama-3.1-8B**[2], **Llama-3.1-Tulu-3-8B**[3], **Llama-3.1-Storm-8B**[4], and **Llama-3.1-SuperNova-Lite**[5] as expert models.\n>   - The performance of the models was evaluated across 7 benchmarks spanning multiple domains, including **general**, **instruction following**, **mathematics**, and **coding**. For comparison, we also included models generated using the Task Arithmetic (TA) method.\n>   - The model merged using CMM **outperforms** all expert models and the task arithmetic baseline in terms of average score across the seven benchmarks.\n>   | Model                             | MMLU  | IFEval | ARC-C | DROP  | BBH   | GSM8K | Math  | Avg   |\n>|----------------------------------|-------|--------|-------|-------|-------|-------|--------|--------|\n>| Meta-Llama-3.1-8B-Instruct       | 69.35 | 68.11  | 81.69 | 82.33 | 57.61 | 84.15 | 39.76 | 69.00 |\n>| Hermes-3-Llama-3.1-8B            | 64.29 | 59.59  | 82.03 | 77.73 | 62.25 | 81.50 | 25.20 | 64.66 |\n>| Llama-3.1-Tulu-3-8B_hf           | 66.26 | 75.18  | 66.10 | 76.68 | 61.18 | 87.87 | 40.02 | 67.61 |\n>| Llama-3.1-Storm-8B               | 68.87 | 72.18  | 81.69 | 79.01 | 56.01 | 82.94 | 35.60 | 68.04 |\n>| Llama-3.1-SuperNova-Lite         | 69.29 | 69.90  | 84.07 | 81.84 | 60.10 | 83.40 | 36.58 | 69.31 |\n>| TA                               | 67.96 | 67.03  | 82.03 | 80.02 | 69.80 | 81.43 | 41.10 | 69.91 |\n>| CMM                              | 69.81 | 70.26  | 81.69 | 83.07 | 61.11 | 84.61 | 40.96 | **70.22** |\n>\n>[1] Team Llama et al. \" The llama 3 herd of models.\"\n>\n>[2] Teknium et al. \"Hermes 3 Technical Report.\"\n>\n>[3] Nathan et al. \"Tulu 3: Pushing Frontiers in Open Language Model Post-Training.\"\n>\n>[4] Ashvini Kumar Jindal. \"Llama-3.1-Storm-8B: Improved SLM with Self-Curation + Model Merging.\"\n>\n>[5] Lucas et al. \"Arcee-SuperNova: Training Pipeline and Model Composition.\"\n\n**Limitations**: It would be beneficial to include a clear discussion of the limitations in the appropriate section, such as the conclusion or a dedicated limitations section.\n>Thank you for pointing this out. We have now included a dedicated \"Limitations\" section in our revised paper. The main limitations of our current work are as follows:\n> - **Homogeneous architecture constraint**: Model merging requires all expert models to share the same architecture, which limits its applicability to broader scenarios involving heterogeneous model types.\n> - **Scalability with many expert models**: When merging a large number of expert models, performance tends to degrade. As discussed in prior work, this is a **known limitation** of model merging in general, not specific to our method.\n> - **Theoretical understanding and extensibility**: While there has been some exploration into why CMM outperforms existing merging methods, the theoretical foundations are still not fully understood. Further research is needed to deepen our understanding and to develop methods that can integrate a larger number of expert models while preserving high performance."}]

Author rebuttal:

Meta-review: This paper proposes Curriculum Model Merging (CMM), a training-free framework that sequentially merges specialized chemical LLMs from weakest to strongest using a curriculum with weighted task vectors. The goal is to consolidate domain expertise without access to original training data and with low computational overhead. 

Strength including:
- A simple, well motivated curriculum strategy tailored to challenges of chemical LLMs.
- Strong empirical results across predictive and generative tasks, with careful ablations on merge order and weights.
- Practical value: training free, privacy preserving, reproducible, and orthogonal to the choice of expert models.
- The Rebuttal additions, including SAR based analysis, added comparisons to SFT on limited data, and scalability checks,  strengthened the paper

The primary concerns including:
- Compute efficiency claims are still qualitative. The paper states that merging takes minutes, but lacks concrete wall clock time, memory footprint, and hardware details relative to both training based baselines and other merging methods.
- The method, by design, is limited to homogeneous architectures. The rebuttal explains this and contrasts with fusion, but the limitation should be made prominent.
- A single aggregated score may underweight niche capabilities. The rebuttal offers mitigation ideas but there is no experiment that explicitly protects rare skills during ranking.
- Authors note degradation when merging many experts, and cite prior work. More explicit guidance or safeguards would help.

Three reviewers lean toward acceptance (scores of 5, 4, and 4), while one maintains a borderline reject. The rebuttal and discussion were active and constructive. The contributions are practical and relevant to a growing area in domain specific LLMs. Remaining issues are addressable in the camera ready.
Decision: Accept (poster)