Title: Curriculum Model Merging: Harmonizing Chemical LLMs for Enhanced Cross-Task Generalization
Venue: NeurIPS 2025
Accepted: yes
Keywords: LLM

model merging

chemical
Abstract: The emergence of large language models (LLMs) has opened new opportunities for AI-driven chemical problem solving. However, existing chemical LLMs are typically tailored to specific task formats or narrow domains, limiting their capacity to integrate knowledge and generalize across tasks. Model merging offers a promising route for efficiently combining specialized LLMs into a unified model without access to original training data, which is urgently needed in the chemical domain where in-house data and privacy preservation are critical. However, effective model merging in the chemical domain poses unique challenges: (1) significant disparities among chemical LLMs due to task-specific specialization, and (2) a highly imbalanced distribution of chemical LLMs in targeted downstream tasks, where some are over-benchmarked while others remain underexplored. These challenges intensify model inconsistencies such as parameter interference and accumulated fine-tuning noise, which collectively hinder effective model merging. To this end, we propose Curriculum Model Merging (CMM), a curriculum-based framework that progressively merges expert chemical LLMs in a moderate and continual manner. CMM aims to harmonize their inconsistencies while meantime preserve their domain-specific expertise. Comprehensive experiments on two benchmark datasets show that CMM effectively consolidates task-specific expertise and outperforms the state-of-the-art methods by 29.03\% in terms of overall average performance. Moreover, CMM facilitates chemical knowledge generalization across prediction and generative tasks without sacrificing robustness, exhibiting promising merging performance under both expert-abundant and expert-sparse scenarios.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors introduce a novel model merging approach known as Curriculum Model Merging (CMM), which progressively integrates expert chemical Large Language Models (LLMs) in a gradual and continuous manner. This method comprises two primary steps: First, the models are ranked according to their capabilities, and a merge order is established. Second, an iterative merging procedure is executed. The experimental results on ChemBench and Mol-instructions demonstrate the effectiveness of the proposed approach.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper is well-organized, and the proposed approach is technically sound. Model merging is an interesting yet challenging task that enables the integration of multiple expert LLMs into a unified model without requiring access to the original training data or incurring additional computational costs. The proposed approach is simple but effective for practical applications.\n\nHowever, the reviewer still has some concerns. The notation in Sec. 3.1 should be clearly described, for example, $\\lambda$ and $\\tau_{new}$. It would be beneficial to provide more in-depth discussion on the experimental results. Additionally, the proposed approach has only been evaluated on LLaMA-3 based models for chemical tasks. The paper should be carefully reviewed to avoid minor errors.\n\nQUESTIONS:\n1. The notation $\\lambada$ and $\\tau_{new}$ in Sec.3.1 should be clearly described.\n2. According to Tab. 1, the capabilities of Meerkat and GeLLMO are both worse than the base model LLaMA-3. Why were they selected for merging in the experiments? Would the merged model perform better if they were excluded?\n3. It can be observed in Tab. 4 that the performance on ChemBench improves after removing MolInst-Molecule and Meerkat. More discussion on this observation would be beneficial.\n4. There are some typos. For instance, \"Tab. 4.2\" in lines 170 and 284 should be corrected to \"Tab. 3.\" The authors are advised to thoroughly review the paper for consistency and accuracy.\n\nLIMITATIONS:\nThe authors claim to have discussed the limitations in Section 5. However, the reviewer did not notice any such discussion in the conclusion section. It would be beneficial to include a clear discussion of the limitations in the appropriate section, such as the conclusion or a dedicated limitations section.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents Curriculum Model Merging (CMM), a novel framework for combining multiple, specialized Large Language Models (LLMs) in the chemical domain into a single, more versatile model. The authors identify that directly merging chemical LLMs is challenging due to significant disparities among models fine-tuned for specific tasks and an imbalanced distribution of expertise across the field.\nThe core idea of CMM is to merge models progressively, following a curriculum. The key contributions of the paper are --\n\nA Novel Merging Framework: CMM first evaluates and ranks the individual expert models based on their performance. It then iteratively merges them, starting from the weakest-performing model and gradually incorporating stronger ones. This step-by-step process aims to harmonize the models, reduce parameter interference, and preserve specialized knowledge.\n\nState-of-the-Art Performance: The paper demonstrates through extensive experiments that CMM significantly outperforms existing model merging methods. On a combined set of predictive and generative tasks, CMM achieves an overall score 29.03% higher than the next best baseline method.\n\nRobust Generalization: The proposed method proves effective in two distinct scenarios: an \"expert-abundant\" setting with many models available and an \"expert-sparse\" setting where only one model possesses the critical expertise for a task. CMM successfully consolidates knowledge for both predictive and generative tasks without significant performance degradation.\n\nPractical Application: This work is the first to systematically adapt model merging for the chemical domain, offering a practical, privacy-preserving solution that does not require access to original training data or incur substantial computational costs\n\nSTRENGTHS AND WEAKNESSES:\nStrengths -----\nQuality: The paper demonstrates high quality through its rigorous and comprehensive experimental validation. The authors use two distinct benchmarks, Chembench and Mol-Instructions, which together cover 11 different chemical tasks, spanning both prediction and generation. The inclusion of extensive ablation studies systematically validates the core components of the CMM framework; for example, the performance drops significantly from 56.62 to 41.67 when the merging order is reversed, confirming the curriculum's importance. Furthermore, the comparison against multiple relevant and high-performing baseline methods like TA, TIES, and DARE_TA provides strong evidence for CMM's superior performance.\n\nClarity: The paper is clear and well-written. The core concepts are introduced logically, and the methodology is presented with clear mathematical formalism. A major strength is the use of visualizations to aid understanding. Figure 1 provides an excellent intuitive illustration of the CMM process compared to standard methods , while Figure 2 effectively visualizes the performance gains at each step of the merging curriculum, making a compelling case for the chosen merging order.\n\nSignificance: The work addresses a problem of high practical significance: creating a single, versatile chemical LLM from multiple specialized models without needing the original data or incurring massive computational costs. The demonstrated success of CMM in both \"expert-abundant\" (Chembench) and \"expert-sparse\" (Mol-Instructions) scenarios underscores its robustness and broad utility. The finding that CMM can preserve and even generalize rare knowledge—as seen in its performance on retrosynthesis where only one expert model was competent—is a particularly significant result. The overall performance improvement of 29.03% over the best SOTA baseline is substantial and impactful.\n\nOriginality: The paper's primary originality lies in being the first to systematically adapt and tailor model merging techniques to the unique challenges of the chemical LLM domain. The authors originally identify the key problems of high model disparity and imbalanced task coverage in chemistry. The proposed solution, framing the merging process as a curriculum where models are progressively integrated based on their capability, is a novel and insightful contribution that directly addresses these challenges.\n\n\nWeaknesses -----\nQuality: A potential critique of the paper's quality could center on the homogeneity of the models used. All the expert models chosen are based on the LLaMA architecture (LLaMA-2, LLaMA-3, Mistral). An argument could be made that the method's effectiveness has not been demonstrated on a more diverse set of model architectures, leaving its generalizability beyond LLaMA-like models in question. Additionally, the entire curriculum is constructed based on a single aggregated performance score, which could be a point of weakness if a model performs poorly on average but possesses unique, valuable capabilities on a niche task not well-represented in the overall score.\n\nClarity: While the paper's results are clear, the analysis of why the merged model generalizes could be deeper. The paper makes the exciting claim that the CMM-merged model \"does not merely inherit expert knowledge\" and \"appears to generalize relevant patterns\". However, this is not substantiated with qualitative examples or deeper analysis, leaving the inner workings of the knowledge fusion somewhat opaque.\n\nSignificance:  As all experiments and validation are confined to the chemical domain, the paper does not provide evidence that the CMM approach would be beneficial for merging models in other domains like computer vision or robotics, where the nature of expert specialization might differ.\n\nOriginality: An argument could be made that the paper's originality is somewhat limited, as it combines two existing lines of research: task-vector-based model merging (like Task Arithmetic) and curriculum learning. From this perspective, the core contribution could be framed as a clever and effective application of known concepts to a new domain, rather than the invention of a fundamentally new merging paradigm.\n\nQUESTIONS:\n1. On the Generalizability of CMM to Architecturally Diverse Models --\nYour experiments compellingly demonstrate CMM's effectiveness when merging models from the LLaMA family (LLaMA-2, LLaMA-3, Mistral). This architectural homogeneity raises a question about the broader applicability of the CMM framework. Could you discuss the anticipated challenges or necessary modifications for applying CMM to a more architecturally diverse set of models? For instance, what would be the implications of merging a traditional transformer-based LLM with a model based on a graph neural network, another common architecture in the chemical domain?\n\n2. Providing Concrete Evidence for Synergistic Knowledge Generalization --\nThe paper makes the exciting claim that the CMM-merged model \"does not merely inherit expert knowledge\" but \"appears to generalize relevant patterns beyond individual model boundaries\". This suggests a synergistic effect where the whole is greater than the sum of its parts. To make this powerful claim more concrete, could you provide a few qualitative examples from your test set?\n\n3. Regarding the Robustness of the Single-Metric Ranking Curriculum --\nThe curriculum in CMM is constructed based on a single, overall performance score that is aggregated across multiple tasks. While your ablation study explores different score computation methods, it still relies on a single final metric for ranking. Have you considered scenarios where this single metric might be insufficient? For example, a model could have a low overall score but be the only expert on a rare but critical task. How would CMM ensure this vital, niche capability is not diluted or lost when it is merged early in the curriculum as a \"weaker\" model?\n\n4. Elucidating the Choice of Merging Weight Hyperparameters --\nYour ablation study on the merging weight distribution (constant, linear, exponential) is very informative. The paper states that the best-performing linear strategy used coefficients that increased from 0.3 to 0.6. Could you provide more intuition on how sensitive the model's performance is to this specific range? How was this range determined?\n\nLIMITATIONS:\n1. Discussion of Methodological and Experimental Limitations --\nA dedicated limitations section should be added to address the following points:\nArchitectural Homogeneity: The current study exclusively uses expert models from the LLaMA family. The authors should acknowledge that the effectiveness of CMM on architecturally diverse models (e.g., merging a transformer with a graph neural network) has not been tested and may present unforeseen challenges.\n\nScalability of Sequential Merging: The experiments involve merging four expert models. The paper should discuss the potential scalability of the sequential CMM approach. As the number of expert models grows into the dozens or hundreds, the linear, one-by-one merging process could become a computational bottleneck, and potential \"catastrophic forgetting\" of the earliest merged models might become more pronounced.\n\nRobustness of the Curriculum Ranking: The curriculum is based on a single aggregated performance score. The authors should discuss the potential brittleness of this approach. For example, a model that is weak overall but possesses a unique, critical capability might be merged too early, risking the dilution of its rare knowledge by subsequent, stronger models.\n\nHyperparameter Sensitivity: The paper shows a linear weighting strategy works best but could benefit from a discussion on the sensitivity and tuning process for the weight coefficients (e.g., the 0.3 to 0.6 range). This would provide a more complete picture of the practical implementation of CMM.\n\n2. Discussion of Potential Negative Societal Impact\n\nThe development of more powerful and versatile chemical language models carries potential risks that warrant discussion:\n\nDual-Use Applications: A model that effectively consolidates knowledge from disparate chemical domains could be a powerful tool for beneficial research like drug discovery. However, it could also be misused to accelerate the design of harmful substances, such as novel toxins, illicit drugs, or chemical weapons. The authors should acknowledge this dual-use potential as a significant societal risk associated with creating more powerful, generalist chemical AI.\n\nOver-Reliance and De-skilling: The availability of a highly capable, \"all-in-one\" chemical LLM could lead to over-reliance by researchers and practitioners. This could potentially de-skill human chemists or stifle creative, out-of-the-box human problem-solving. It also raises the risk of significant errors if the model's failure modes are not well-understood by its users.\n\nAccessibility and Malicious Use: The CMM method is designed to be practical and efficient. This accessibility means that malicious actors could potentially use it to merge their own privately-trained expert models to create a powerful tool for nefarious purposes, bypassing the safeguards of public models. A brief discussion on the importance of responsible model governance would be appropriate.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper first identifies some characteristics of chemical LLMs, including the prevalence of in-house training data in the chemical domain and significant disparities in parameters and downstream functionality, which highlight the necessity of novel model merging methods. Therefore, they propose Curriculum Model Merging (CMM), a method that constructs curriculum to progressively merge the expert chemical LLMs in a moderate and continual manner. Their comprehensive experiments on two benchmark datasets show that the proposed method concentrates task-specific expertise and outperforms the state-of-the-art methods by 29.03% in terms of an overall average performance score.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n(1) This paper identifies a critical problem of merging chemical LLMs due to prevalence of in-house training data in this domain. \n\n(2) The authors propose a model merging method to tackle this problem.\n\n(3) Extensive experiments are conducted to demonstrate the effectiveness of the proposed method.\n\n**Weaknesses**\n\n(1) The experimental setting, especially those of baseline models, are not clearly described. In my understanding (with some of my best guesses), the Expert Models (e.g. Table 1) are actually evaluated by their zero-shot performance on ChemBench. The SOTA merge methods are evaluated similarly on their out-of-domain performance on ChemBench, as none of the expert models used training information from Chembench before testing. On the contrary, only the proposed CMM method gets access to Chembench information through the $S_i$ score. I would like to confirm with the authors that the understanding above is correct.\n\n(2) If so, the comparison to all baselines in this paper seems unfair. The baselines in this paper such as TA and TIES are actually not designed for scenarios like this. These methods serve as lightweight methods with generalizability to multiple tasks, while the CMM is constructing task-specific models which are tailored to a “selected benchmark suite”.\n\n(3) Of course, the application scenario proposed by the authors might be useful. But in such cases, the baselines to compare should not be limited to those model merging methods shown in this paper. For example, what is the performance of directly finetuning a LLM on Chembench, and what about directly using a cutting edge LLM like GPT-4o with in-context learning? In other words, this paper proposes to use model merging to obtain high performance on chemical tasks like Chembench, then it is necessary to show advantage of CMM to all kinds of machine learning methods. Only comparing CMM to model merging methods looks insufficient to me.\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes Curriculum Model Merging (CMM), a method for integrating multiple specialized chemical large language models (LLMs) into a unified model without requiring access to original training data or significant computational resources. CMM addresses challenges such as model disparities and imbalanced task coverage by progressively merging expert models in a curriculum-based manner. The method demonstrates superior performance, outperforming state-of-the-art merging techniques by 29.03% on average across two benchmark datasets. The paper highlights CMM's ability to generalize across predictive and generative tasks, even in expert-sparse scenarios\n\nSTRENGTHS AND WEAKNESSES:\nThe paper proposes Curriculum Model Merging (CMM), a method for integrating multiple specialized chemical large language models (LLMs) into a unified model without requiring access to original training data or significant computational resources. CMM addresses challenges such as model disparities and imbalanced task coverage by progressively merging expert models in a curriculum-based manner. The method demonstrates superior performance, outperforming state-of-the-art merging techniques by 29.03% on average across two benchmark datasets. The paper highlights CMM's ability to generalize across predictive and generative tasks, even in expert-sparse scenariosStrengths:\n\n1. The curriculum-based merging approach is novel, particularly in the context of chemical LLMs. This work adapts and extends ideas from NLP to a domain with unique challenges.\n2. This work addresses a critical gap in the chemical domain by enabling the integration of specialized LLMs while preserving privacy and reducing computational costs. The results demonstrate substantial improvements over existing methods.\n3. This work presents a well-designed and thorough experimental evaluation, validating CMM's effectiveness across diverse chemical tasks. The ablation studies provide insights into the impact of merging order and weight coefficients.\n\nWeaknesses:\n\n1. The theoretical foundation for the merging process could be expanded to better justify the chosen curriculum strategy and weight assignments.\n2. Some technical details, such as the exact computation of normalized scores for ranking models, could be more explicitly described.\n3. The paper could further discuss the scalability of CMM to larger or more diverse model sets.\n\nQUESTIONS:\n1. Could the authors provide more theoretical analysis or intuition behind why the proposed curriculum merging order (weakest to strongest) is optimal? Are there scenarios where a different order might be more effective?\n2. The paper mentions reduced computational costs, but could the authors provide concrete metrics (e.g., training time or resource usage) comparing CMM to other merging methods?\n3. How might CMM perform in non-chemical domains (e.g., biology or materials science)? Are there domain-specific adjustments needed?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}