Title: Differential Fine-Tuning Large Language Models Towards Better Diverse Reasoning Abilities
Venue: NeurIPS 2025
Accepted: no
Keywords: Large Language Models

Reasoning Abilities

Supervised Fine-Tuning
Abstract: Reasoning abilities of large language models (LLMs) require explicit derivations compared to general question-answering, supervised fine-tuning (SFT) can empower multiple reasoning abilities in LLMs via learning from various datasets. However, neither training the datasets jointly (mix-up) nor continually can maintain the performance of single-dataset SFT, sometimes better while sometimes even worse, illustrating vanilla SFT can not only facilitate reasoning abilities but also introduce conflicts. In this paper, we propose a novel framework to mitigate the conflicts and preserve benefits among different reasoning tasks, and even surpass each task's single dataset SFT performance. We start by exploring the differences between reasoning fine-tuned and base LLMs by analyzing their parameter variations during model inference, and we discover that each reasoning capability has exclusive parameters that benefit itself more evidently than others. In contrast, the overlapped parameters of tasks can bring benefits or conflicts. Inspired by the findings, we propose to update the exclusive and overlapped parameters according to specific reasoning task combinations differentially, thereby avoiding unnecessary conflicts while maintaining benefits. Consistent improvements in mix-up and continual SFT experiments demonstrate that the proposed SFT strategy can achieve better performance on various LLMs (Llama3-8B, Mistral-7B, and Qwen2.5-14B) and diverse reasoning tasks with fewer conflicts, showing the superiority and generality of our analysis findings and the proposed approach.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work addresses the challenge of multi-task or continual fine-tuning of large language models on diverse reasoning tasks.\n\nThe authors introduce a method to analyze the sensitivity of each weight to each task by evaluating differences in weights before and after fine-tuning on sampled data. Specifically, the approach measures the magnitude of weight updates to identify which parameters are critical.\n\nDepending on the fine-tuning scenario, the method either freezes or selectively updates these critical weights to improve performance in multi-task (mix-up) or continual fine-tuning settings.\n\nEmpirical results demonstrate that the proposed method outperforms baselines in both scenarios, mitigating reasoning conflicts while preserving shared benefits.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n- The work is well-motivated, with clear analysis of performance across different supervised fine-tuning (SFT) datasets spanning math, code, and logic tasks.\n- The proposed method is novel: no prior work has measured weight importance in the specific way introduced here. The key question is how effective this approach is and how convincingly the paper validates it.\n\n### Weaknesses\n\n- **Cost of the proposed method.** The approach requires fine-tuning on each dataset to identify sensitive weights, introducing extra computational costs compared to baselines. These costs are not adequately discussed in the experiments or analysis. Moreover, the performance gains appear marginal relative to the overhead.\n- **Limited analysis.** To convincingly demonstrate that the selected weights effectively mitigate conflicts in mix-up or continual SFT, experiments varying the number of selected rows (e.g., ablations on how many parameters are frozen or updated) are necessary. However, such experiments do not appear in the main manuscript.\n- **Lack of justification for the method's rationale.** It is unclear why selecting parameters with large differences before and after fine-tuning should inherently reduce catastrophic forgetting or enhance shared reasoning skills. While the paper notes that these parameters (DSR) are those most updated during fine-tuning (lines 165–168), this does not guarantee that freezing or updating them produces the desired effects. In other words, I would like to understand the underlying logic supporting the claim: *The core idea is to identify and protect parameters crucial for simultaneously and previously learned tasks while allowing the model to adapt to more reasoning proficiencies* (lines 207–209). Is there any reference or theoretical justification to support this assumption?\n\nQUESTIONS:\n### Questions\n\n- Why is `model.layer.24.mlp.gate.proj` selected for visualization in Figure 2?\n- What happens if rows are randomly selected, using the same number of rows as the proposed method? While this is related to Section 5.4, it remains important to clarify whether the results in Table 3 reflect proper selection of rows or merely the number of parameters trained.\n- What does DSR stand for?\n- What is the size of the sets for DSR_union and DSR_diff?\n\n### Suggestions\n\n- Increase the font size in Figure 1 for better readability.\n- Improve the clarity of Figure 2. As presented, it is difficult to see whether the trends are consistent across seeds or whether they differ across datasets. Although Section 3.3 interprets the results, the figure and caption alone are hard to follow.\n- In Table 2, explicitly indicate that DiFT refers to the proposed method (e.g., label it as \"ours\") so the reader can immediately identify it.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis article starts with the current SFT training, whether it is mixed training or continuous training, where the model's performance either improves or deteriorates due to the type of data. It obtains the key parameters of different types of data through delta-scale row, and based on the core idea of only updating the key parameters, proposes the DiFT method, providing new possibilities for SFT.\n\nSTRENGTHS AND WEAKNESSES:\n**Quality**\n\nOverall, this article is relatively high quality. Starting from the current situation where data from different fields show both mutual promotion and conflicts in SFT, the DiFT method was proposed through the delta-scale row. In general, its effectiveness is improved compared to previous methods. \nHowever, the enhanced capabilities are limited and seem difficult to be applied on the instruct model, thus having certain limitations. And there is no link to the open-source code.\n\n**Clarity**\n\nThis article is generally written in a smooth and standard manner. \nHowever, the explanation of the method could be clearer, and the organization of the appendix and the main text could be more orderly.\n\n**Significance**\n\nThis article addresses the actual problems existing in the mixed training and continuous training methods used in SFT training, and proposes feasible solutions, which make great significance.\nHowever, for the catastrophic forgetting issue, the method proposed in the article cannot solve it, and its effect on the instruct model is limited, which restricts its significance.\n\n**Originality**\n\nThe method proposed in this article is quite novel. However, the author also mentioned that it was inspired by another article, which to some extent undermines the originality.\n\nQUESTIONS:\nIn the article, I noticed that you have compared it with some other methods. May I ask if there is a comparison with the SOTA methods that have been used for the same model on the relevant benchmarks at present?\nThe appendix of the article mentions that your analytical method is also effective for the instruct model. Are there any relevant test results? The explanation in the article seems a bit too simplistic.\nThe promoting effect of the combined training of mathematics and code has been studied extensively in previous research. But by your method, did any other potentially mutually reinforcing data emerge? I think this might be a meaningful contribution.\nWhy is there no link to the open-source code?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper observes that different datasets used in supervised fine-tuning (SFT) can have varying impacts on the inference performance of different large language models. To address this, the authors propose a method to identify which parameters are primarily updated by specific data, thereby revealing which parts of the model are more crucial for a given task. Building upon this insight, the paper introduces a differential fine-tuning strategy: for mixed SFT, only the parameters deemed important for the target task are fine-tuned; for continual SFT, different datasets are used to train disjoint subsets of model parameters.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\nThe paper presents a novel perspective by identifying which parameters are primarily affected by specific SFT data, offering a fine-grained view into how different datasets influence model's SFT behavior. The differential fine-tuning approach—targeting only task-relevant parameters or disjoint subsets for different datasets—introduces a practical method that could improve training efficiency and task specificity compared to conventional SFT.\nWeakness:\nSee in Questions.\n\nQUESTIONS:\n1.When introducing related work, avoid overusing citations as the subject (e.g., \"[X] proposes...\"). A few instances are acceptable, but excessive use may hinder readability. Consider rephrasing for better flow.\n\n2.For the proposed model in this work, whether the differential computation of effective parameters spans all layers of the model? including feed-forward networks (FFN), attention layers, and layer normalization layers.\n\n3.The baseline models used in this work are all dense models. It would be valuable to discuss how different domain datasets influence parameter importance during SFT, especially since the current SFT dataset is relatively small. For instance, if the in-domain training data is sufficiently large, could most parameters become \"important,\" making the approach resemble traditional SFT? Additional analysis on this aspect would strengthen the paper.\n\n4.Has the proposed method been tested or evaluated on Mixture-of-Experts (MoE) models? \n\n5.The current visualizations of inference processes under different SFT strategies are relatively limited. Providing more detailed visual analyses would enhance interpretability. Furthermore, the models used in the experiments are not representative of typical reasoning LLMs which “Think-Then-Answer.” Given the paper’s emphasis on reasoning tasks,could the authors provide more details on how the baseline models and the proposed method how to response under reasoning-specific challenges?  \n\n6.the data curation principles are unclear, hindering reproducibility and thorough assessment.\n\nLIMITATIONS:\nThere may be two potential limitations concerning both the data and model aspects:\nThe effectiveness of the proposed method has not been fully validated on both dense models and Mixture-of-Experts (MoE) models—particularly important given the growing adoption of MoE architectures in contemporary reasoning models.\nThe SFT data used may still rely primarily on traditional question-answer pairs, rather than incorporating chain-of-thought (CoT) data that is commonly employed in modern reasoning-focused models.\n\nWhile these limitations may exist, they do not fundamentally undermine the overall assessment of the work. As the reviewer may not be fully familiar with the latest innovations in this area, it is recommended that the Area Chair consider the perspectives of other reviewers when evaluating the contribution of this paper.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies why supervised fine-tuning (SFT) on several reasoning datasets sometimes helps but often hurts other abilities.\nAfter analysing weight-change patterns between base and task-specific models, the authors observe that only a handful of “delta-scale rows’’ in each linear layer dominate the change for a given task.\nThey then propose DiFT, which freezes or updates parameters selectively according to the union/difference of those salient rows when mixing or sequencing tasks. Also, DiFT consistently improves mix-up SFT and mitigates performance collapse in continual SFT on three open LLM families and four reasoning tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n**(1) Clear empirical motivation.** The paper first reproduces mix-up and continual conflicts and quantifies them, motivating a deeper look.\n\n**(2) Practical fine-tuning recipe (DiFT).** The union/difference masking idea is intuitive, compatible with vanilla full-parameter or LoRA updates, and needs no additional loss terms or gradient surgery. Algorithm 1 is explicit.\n\n**(3) Broad experimental coverage.** Results span 3 model families, 2 training regimes (mix-up & continual), 5 baselines (DMT, CoBa, HFT, LoTA, PEFT), ablations on number of rows, inverse-mask sanity checks, and 14 B scale. Improvements are consistent (e.g., +0.55 ATA on Mix-Math-Code 14 B).\n\n\nWeaknesses:\n\n**(1) Catastrophic forgetting largely unsolved.** DiFT mitigates conflicts but math accuracy still drops by ~50 % in Continual-Math-Logic (10.99 %), similar to vanilla SFT.  This limits practical adoption in continual settings. This paper could  consider to combine DiFT masks with existing CL regularisers (e.g., EWC/Fisher, rehearsal) or adaptively expand the union mask over time, reporting forgetting metrics.\n\n**(2) Composite ATA metric can mislead** To compute “average target accuracy” the authors multiply code pass-rate by 50 and logic accuracy by 2. Such asymmetric rescaling hides absolute regressions (e.g., −2 pp in xGLUE can look tiny). A reader cannot tell whether gains stem from one task swamping the others.\n\n**(3) Comparison set could be stronger.** Baselines mostly freeze-mask or data-schedule methods; gradient-projection approaches (e.g., PCGrad, GradVac), adapter-routing or MoE fusion are missing.\n\n**(4) Limited contrast with prior parameter-attribution work.** Related-work lists outlier dims  and task–vector arithmetic but never clarifies how delta-scale rows differ in signal captured or compute cost.\n\nQUESTIONS:\nplease refer to the weaknesses\n\nLIMITATIONS:\nplease refer to the weaknesses", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-06"}