Enough is as good as a feast: A Comprehensive Analysis of How Reinforcement Learning Mitigates Task Conflicts in LLMs
Authors: Zixuan Ren
Jinliang Lu
Junhong Wu
Yang Zhao
Dai Dai
Hua Wu
Haifeng Wang
Chengqing Zong
Venue: ICLR 2026 Poster (2026)
Keywords: Large language model
Reinforcement learning
Model merging
Primary area: foundation or frontier models, including LLMs
TL;DR: We provide a comprehensive analysis of how reinforcement learning mitigates task conflicts in LLMs

Abstract: Model merging plays a crucial role in consolidating multiple specialized models into a single, unified model, especially in the era of large language models (LLMs). Recent research has primarily focused on developing strategies to enhance merging performance with the trained models, while the impact of training paradigms, such as supervised fine-tuning (SFT) and reinforcement learning (RL), on the effectiveness of model merging remains underexplored. In this study, we systematically explore the merging behavior of RL-trained LLMs compared to those trained with traditional SFT. Through comprehensive evaluations across five representative tasks, we find that RL significantly reduces task conflicts and results in less performance degradation after merging, making RL-trained models particularly well-suited for this process.
To unearth the reasons behind the superior suitability of RL for model merging, we conduct extensive empirical experiments and theoretical analyses. Our findings highlight three key factors: (1) On-policy training data in RL control the gradient updates in a smaller magnitude, reducing the risk of overwriting existing knowledge for other tasks in the model. (2) The RL optimization objective, which favors "\textit{enough is as good as a feast}", progressively reduces the magnitude of parameter updates as the model converges, thereby alleviating inter-task conflicts. (3) Joint optimization of positive and negative examples in RL steers the model towards an unbiased task-specific parameter subspace, ensuring robust performance while further preventing parameter conflicts.

Published as a conference paper at ICLR 2026
ENOUGH IS AS GOOD AS A FEAST:
A COMPREHENSIVE ANALYSIS OF HOW REINFORCE- MENT LEARNING MITIGATES TASK CONFLICTS IN LLMS
Zixuan Ren 1.2 Jinliang Lu .Junhong Wu! Yang Zhao 1.2 Dai Dai Hua Wu², Haifeng Wang', Chengqing Zong 1.2
State Key Laboratory Multimodal Artificial Intelligence Systems, Institute of Automation, CAS, Beijing, China
SSchool of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China Baidu Inc., Beijing, China
(renzixuan2021, wujjnnhong2221, htt.://....2...aaooooogog...aa..
ccrzong/enIpr..a.cdo iaaiaaaacc..... an
[lujinliang, daidai, wu hua, wanghaai
ABSTRACT
Model merging plays crucial role in consolidating multiple specialized mod- els into a single, unified model, especially the era of large language models (LLMs). Recent research has primarily focused on developing strategies to en- hance merging performance with the trained models, while the impact train- ing paradigms, such supervised fine-tuning (SFT) and reinforcement learning (RL., on the effectiveness of model merging remains underexplored. In this study, we systematically explore the merging behavior of RL-trained L.MSs compared those trained with traditional SFT. Through comprehensive evaluations across five representative tasks, find that RL significantly reduces task conflicts and results in less performance degradation after merging, making RL-trained models partic- ularly well-suited for this process. To unearth the reasons behind the superior suitability of RL for model merging, we conduct extensive empirical experiments and theoretical analyses. Our findings highlight three key factors: (1) On-policy training data in RI. control the gradient updates in smaller magnitude, reducing the risk of overwriting existing knowledge for other tasks in the model. (2) The RL optimization objective, which favors "enough is good feast", progres- sively reduces the magnitude and the number conflict parameter updates as the model converges. (3) Joint optimization of positive and negative examples in RL steers the model towards an unbiased task-specific parameter subspace, ensuring robust performance while further preventing parameter conflicts.
INTRODUCTION
Large language models (LLMs) have fundamentally reshaped the landscape of artificial intelligence, capturing growing interest from both academia and industry (Team] 2024 Grattatiori et al. 2024 Xiang al., 2025). Recent statistics show that there are now more than 270,000 models with over 3 billion parameters available on HuggingFacd These large models often exhibit diverse capa- bilities and strengths (Shao et al.) 2024 Ahmad al. 2025) Toshniwal et al. 2025), prompting researchers to investigate ensemble techniques that integrate the specialized abilities of different models (Li et al.) 2023) Chen et al., 2025 Ruan et al. 2025). Among these techniques, model merging which directly fuses the parameters of independently fine-tuned models without requir- ing access to the original training data, expensive retraining, or the maintenance of multiple check- points has emerged as particularly promising solution (Lu al. 2024 Yang et al. 2024b).
Corresponding author.
https://hugging http://hhgggggggaaaaooo ctt.co/cooocoo co/mode11

---

Published as a conference paper at ICLR 2026
SFT-trained models
RI-trained models
0%
Performance Chainge
-20%
Code
-20%
Instruction Pollowing
40%
-60%
-00%
Purzie
-00%
100%
100%
Number merged models
Number merged models
Figure 1: Comparison of performance changes between SFT and RFT in model merging.
A central challenge in model merging is preserving performance when integrating different mod- els, as interference between parameters can lead performance degradation in the merged model, a phenomenon commonly referred to as task conflicts. To mitigate this issue, prior research has proposed a range of model-merging strategies (Ilharco al. 2023 Yadav al.) 2023) Yu et al.) 2024). typically under the assumption that the models involved have already been trained on dif- ferent tasks. Nonetheless, the approach used train these task-specific models critically impacts the ultimate effectiveness of the merged hadel-hass been largely overlooked. In the era of LLMs, training paradigms can be broadly classified into supervised fine-tuning (SFT) and rein- forcement learning (RL) (Hu et al. [222 DeepSeek-Al et al.] 2025). Most existing works focus on merging models trained via SFT. leaving differences between SFT and RL in the context of model merging largely unexplored and deserving of further investigation.
In this work, we investigate the role of fine-tuning paradigms (SFT and RL) in model merging. A series of controlled experiments conducted across five representative tasks mathematical reasoning, code generation, instruction following, logical puzzles, and ranking. And we assess the extent to which models fine-tuned with SFT or RL, respectively, preserve their original performance after model merging. Figure [ reveals that models trained with RL exhibit significantly better performance preservation after merging than those trained with SFT. Extensive experiments suggest that our finding holds regardless of different merging methods, various base models, or distinct RL algorithms. Furthermore, we demonstrate that RL alleviates parameter conflicts, thereby making it particularly well-suited for model merging.
To further investigate the reason behind the superiority of RL on model merging, we conduct a series of theoretical and empirical analyses. Our results demonstrate three primary contributing factors: (1) On-policy training data: unlike SFT, where training data are sampled from fixed datasets or human annotations, RL relies on data sampled from the model itself. This substantially reduces gradient magnitudes and thereby lowers the risk of overwriting knowledge acquired from other tasks. (2) The intrinsic dynamics of RL: the training objective of RL algorithms naturally attenuate parameter updates and avoiding the increase of parameter conflicts as the model converges, favoring a state where "enough is as good as a feast". In contrast, SFT applies update with fixed intensity regardless of convergence. This intrinsic characteristics of RL alleviates task conflict during integration. (3) The joint optimization over positive and negative samples: RL simultaneously optimizes over both positive and negative examples, whereas SFT relies only on positive samples. This leads to more unbiased updates, resulting in higher convergence stability and more robust task integration.
In summary, this work highlights a previously underappreciated advantage of RL: its natural supe- riority for model merging. Our findings offer new insights into how different fine-tuning paradigms shape the task conflicts of LLMS and point toward more robust and scalable strategies for building generalist models without retraining from scratch.
Our key contributions are as follows:
We investigate the effect of LLM post- testtrrainin paradigms model merging. Our empirical results consistently demonstrate that RL-trained models are more suitable for merging, irrespective of the employed merging methods, the selection of RI. algorithms, the choice
2

---

Published as a conference paper at ICLR 2026
of base models. Further analysis suggests that the superiority of RL-trained models stems from its ability to mitigate task conflicts.
Through both empirical and theoretical analyses, advance novel hypotheses and argue that (1) the use of on-policy data, (2) the intrinsic characteristics of RL algorithms, and (3) the joint optimization of positive and negative samples collectively contribute to the enhanced suitability of RI-trained models for model merging.
2 BACKGROUND
2.1 MODEL MERGING
Model merging aims to integrate the multiple independently trained models into a single unified model. Assume there are T" models with parameters 02, Oz. 0r, each sharing the same archi- tecture and initialized from common base model 0.- The goal of merging obtain a new model mmerge by applying merging operator:
i merge merge 02, ,0))
Recent advances merging methods are typically based on the task-relevant parameter updates Ti 0: 0b namely Task Vector (llharco et al., 2023). The direction of these Task Vectors often reveals conflicts between different tasks. mitigate such conflicts, recent studies have introduced pruning processes to remove redundant or unimportant parameters, seen in approaches like DARE (Yadav
al.
2023
Yu et al. 2024; Wang et al. 2024).
2.2 SUPERVISED PINE-TUNING AND REINFORCE LEARNING
Supervised Fine-Tuning (SFT) Given a dataset DSFT {(I, y4) " consisting of prompts x; and their ground-truth responses Yt. SFT optimizes the model by minimizing the negative condi- tional likelihood of generating the correct response:
l
(SFT -B log (a ar, yer)
where Ty denotes the model policy parameterized by O. This objective enforces strict adherence labeled data, enabling stable convergence but limiting the flexibility incorporate reward-based feedback beyond supervised signals.
Reinforce Learning (RL) provides alternative optimization paradigm that aligns model behav- ior with task-specific reward functions. Formally, the objective is to maximize the expected return:
JRL Bg--Da, yoro [r(v.))]
where r(y, .)) is a scalar function that reflects the expected reward of the output y given the input In the of large language models, widely used reinforcement learning algorithm is Proximal Policy Optimization (PPO, Schulman et al. (2017)), which can be formulated as:
To (V6 Teae (3elWer)
No (ye (yellee)) Te (Belyee)
e, 1 c,1+e) 2n)
(P) EEgW DRL
min
A,, clip
(4)
where A, denotes the estimated advantage based on the reward r and the function clip(-) con- strains the probability ratio to the interval [1 - e 1 + E] to ensure training stability. Standard PPO typically employs a critic model to estimate the advantage A To further improve the training efficiency, recent studies have proposed critic-free variants, such GRPO (Shao al.) (024)) and REINFORCE+ (Ahmadian et al. 2024), which are also widely adopted in current research.
3 REINNORCEMENT LEARNING MITIGATES TASK CONFLICTS
In this section, we experimentally compare the performance preservation of SFT-trained and RL- trained models after model merging.

---

Published as a conference paper at ICLR 2026
Math
Code
IF
Puzzle
Rank
Average
SFT
61.9
60.5
63.9
86.2
52.8
61.5
Averaging
52.0(-16%)
56.0(-7.4%)
49.2(-23%%
30.8(-65%)
51.6(-2.3%)
47.9(-22%)
TIEs
56.8(-8.3%)
58.0(-4.1%)
47.5(-25%)
35.8(-58%)
51.3(-)7%%
49.9(-19%)
Arithmetic
52.4(-15%)
56.3(-7.0%)
48.2(-25%)
44. (449())
(1.1(-3.3%)
50.6(-18%)
DARE
58.2(-6.1%%
58.0(-4.19)
46.7(-27%
38.0(-56%)
49.3(-6.7%
50.0(-19%)
RL (GRPO)
64.6
65.6
90.0
85.2
55.7
72.2
Averaging
62.1(-3.9%)
61.7(-5.9%)
84.4(-6.2%)
37.8(-56%)
54.4(-2.3%)
60.1(-17%)
TIEs
63.3(-2.0%)
64.3(-2.0%)
90.0(-0%)
64.6(-24%)
53.1(4.77)
67.1(-7.1%)
Arithmetic
62.6(-3.1%)
63.8(-2.7%)
89.2(-0.9%)
60.7(-29%)
54.6(-2.0%%
66.2(-8.3%)
DARE
63.5(-1.7%)
64.2(-2.1%)
89.9(-0.1%)
65.0(-24%)
53.1(4.7%%
67.1(-7.1%)
Table I: Performance comparison across five tasks using different merging strategies (Averaging, TIEs, Arithmetic and DARE), applied to both SFT and RL (GRPO) models. The values in paren- theses indicate the relative performance drop compared to the original unmerged model and less performance drop.
3.1 EXPERIMENT SETUP
Models and Settings We adopt the open-source models Llama-3.2-3B (Grattafiori et al. 2024). Alama-3.1-8B GGrataafiori et al. 2024), and Mistral-Smaall-32241 as the base models. For training. we employ three representative RL algorithms: PPO (Schulman et al. 2017). GRPO (Shao et al., 2024), and REINFORCE++ CaaEnEEE+AAhaaiiaaa (Ahmadian et al. 2024). Unless otherwise specified, all experiments and analyses default to Llama-3.1-8B as the base model and GRPO as the optimization algorithm.
Experiment Tasks and Data Our evaluation spans five tasks that permit automatic verification: math, code, instruction following (IF), logical puzzles (puzzle), and ranking (rank). For math, models are trained on a subset of OpenMathlnstrutts (Ahmad et al. (0255 and evaluated on GSMSK (Cobbe al. 2021) and MATH-500 (Hendrycks et al., 202 Lightman et 2023). For code, models are trained on a subset of OpenCodelnstruct (Ahmad et al. 2025) and evaluated on the HUMANEVAL (Chen al. 2021) and MBPP (Austin et al.) (021) datasets. For instruction follow- ing, models are trained on the instruction subset from Tulu-3-FFT (Lambert et al., 2025) and eval- uated on IFEVAL (Zhou et al. 2023b) and the instruction- foolowiinnnnoooowing...oo subset LIVEBENCH (White et al. 2024). For logical puzzles, models are trained on task Knights and Knaves (Johnson-Laird & Byrne 1990) with synthetic data using templates implemented by Xie et al. (2024) and evaluated on the same task. For rank, models are trained on the Rankl dataset (Weller et al. (225)) and evaluated on the pairwise ranking benchmark NEVIR. Further details are listed in appendix
Model Merging Settings We evaluate four model merging strategies: model averaging (Aver- aging for short) (Choshen et al. (022). TIEs (Yadav et al., 2023). Task-Arithmetic (Arithmetic for short) [Thhacco et al. (023) and DARE+ TTES (DARE for short) Yu al., 2024). In our experiments, time investigate the robustness focus pairwise merging, where two models are merged and compatibility of different training paradigms. For each task, we report the average performance of the saarwise-merged models from the specific task to any other tasks.
3.2 MAIN RESULTS
Table compares the effects of four parameter-merging methods on models trained via SFT and GRPO. A consistent, paradigm-dependent gap emerges: RL-trained models are substantially more suitable for merging, largely preserving the performance of individual models after merging, both specific tasks and in overall averages. SFT-trained models suffer severe degradation for instance, Puzzle drops by up 65% and the mean decline spans 18-22%. By contrast, RL-trained models limit losses to under 10% for most strategies. Even on the most fragile task (Puzzle), RL- based models degrade less sharply. This advantage holds across methods: with Averaging, SFT shows a 22% mean decline versus 17% for RL; with TIEs, RL falls only 7.1 % compared to 19% for
htttss:/miistee http://mmsttrall htt///w///mm /mi stral 11-3
4

---

Published as a conference paper at ICLR 2026
2.5
-5.5
-15
SF1
-2 68P0
PRO
Membbros+
Math
Code
IF
Average
Figure 2: Performance comparison across three tasks using TIEs merging, applied both SFT and RL models. The values in parentheses indicate the relative performance drop compared to the original unmerged model.
0.0
252007
0.0
Performence Deap
7.7-7.6
-6.
-7.1
5.5
-20
Models
Lisma-3.2-30
Methods
Liama 88
SFT
Garo
Math
Code
Average
Figure 3: Performance comparison across three tasks using TIEs merging. applied to both SFT and GRPO models with different base models. The values in parentheses indicate the relative perfor- mance drop compared to the original unmerged model.
SFT. Arithmetic and DARE follow the same pattern, with RL consistently outperforming SFT by wide margin.
Generality of RL Algorithms in Model Merging To further assess the generality of RL algo- rithms the context of model merging, we extend our analysis beyond GRPO by incorporating two additional algorithm-PPO and REINFORCE++-with TIEs merging. As shown in Figure 2 across all RL algorithms, the merged models consistently exhibit significantly lower performance degra- dation compared to those trained with SFT. After TIEs merging, SFT models exhibit substantial 28.7% decrease in IF, whereas RL models experience only negligible performance losses (GRPO: -0.3%, PPO: -2.5%, REINFORCE++: -2.6%). This reinforces our key observation: RL-trained model bases are substantially more suitable for model merging than their SFT counterparts, regardless of the specific RL algorithm employed. Detailed results with other merging methods provided in Table in the Appendix.
RL Algorithms Benefit Model Merging with Different LLMs We further conduct experiments on different base models, Llama-3.2-31 3-B Llama-3. 1-B and Sestrra-Smmall3-220 across three tasks: Math, Instruction Following, and Code. We evaluate merged models TIEs merging. As shown in Figure 3 across all base models, GRPO-trained models consistently suffer substantially less per- formance degradation than their SFT-trained counterparts. For instance, after TIEs merging, SFT models drop by -35.6% to 11.77% on IF, whereas RL-trained models decline -4.6% to +0.4%. These results reinforce central observation: RL-trained models substantially more suitable for model merging, irrespective of the base models. Detailed results with other merging methods are included Table 5 in the appendix.
3.3 PERFORMANCE LANDSCAPE OF SFT- AND RL-TRAINED MODELS
To understand why RL-trained models are more suitable for model merging, we investigate the source of this superiority from aspects: "RL-trained models more robust parameter per- mutation" and "RL mitigates the task conflict". Specifically, given two models fine-tuned indepen- dently tasks 4 and l2, let their respective parameters be denoted f,, and o.,- Model merging
5

---

Published as a conference paper at ICLR 2026
aims to obtain unified parameter set merge that integrates knowledge from both tasks. From the perspective of model , the merged model can be expressed as a perturbation in parameter space:
merge or, A0,
where A0 = merge 0, represents the update direction induced by merging with Oz.- This formu- lation naturally raises the question: Is the performance loss of the model caused by simple parameter perturbations, or by parameters that conflict with other tasks A
To investigate this, visualize the performance landscape (i et al. 2018) around 0 For each model, we compare two perturbation directions: (1) Task-induced direction: A = merge o.,.
(2) Random direction: radd ~ N I). scaled such that ||0 1 = ||00|2 We evaluate model performance over the two-dimensional surface defined by:
f( B) = L0,, + as0 30 randd),
where ((.) denotes the task-specific performance function, and (a,, E R.2 parameterize the per- turbation magnitude along each direction.
RL-Math
SFT-Math
RL-IF APtumtrek
SFT-IF
Cade
106
0.00
0.21
-50%
wappr 09
-204%
0.00
0.99
0.99
1,00
-2.5%
1.00
0.00 120
0.43
3:00
0.82
1.00
5,20
3.42
0.90
9.t 1.00
032 6/0
022
6.13
8.00
Figure 4: Performance landscape visualization around Or, for SFT and RL models. The a-axis corresponds to the task-induced update AO, and the B-axis corresponds a random perturbation of equal norm.
As illustrated in Figure f adding a random perturbation Orand has minimal impact on performance in both SFT and RL models, indicating robustness parameter noise. However, the task-induced update AØ reveals a stark contrast: for SFT-trained models, increasing a along AØ leads to a no- ticeable degradation in task performance, suggesting strong interference from task f2. In contrast, RL-trained models maintain stable performance even when perturhed in the direction of AO, im- plying that parameters learned from task 1/ do not conflict with task f1. These observations support our key finding: parameter updates induced by SFT tend to be more entangled across tasks, leading interference, while RL encourages task-orthogonal updates that are less disruptive when merged.
4 How RL MITIGATES THE TASK CONFLICT
Having established that RL-trained models exhibit significantly lower task interference than those trained with SFT, we delve deeper into the mechanisms underlying this advantage. In this section, we systematically analyze how RL can mitigate task conflicts by examining three key dif- ferences between RL and SFT, which specifically are on-policy data, different training objective and joint use of positive and negative examples.
4.1 THE EFFECT OF ON-POLICY DATA
One main difference between SFT and RL is the training data. In RL, the training data are sampled from the current model (on-policy) while in SFT, it is from a fixed dataset (off-policy). To quantify this effect, we compute the norm of the whole model updates under the three training paradigm: SFT, RL and Rejection-Fine Tuning (RFT, ?). Notably, RFT can be viewed as special case of SFT where the training data are sampled from the original model. Table[]shows that on-policy data consistently produce significantly lower parameter magnitudes than off off-policy SFT. For instance, in
Full content is shown in appendix D.1

---

Published as a conference paper at ICLR 2026
math task, the norm fo models trained with SFT is 6.5, much higher than that of RFT- and RL-trained models, which are 2.36 and 0.78. These results demonstrate that on on-policy data help to regulate the magnitude of gradient updates and reduce parameter sensitivity.
A closer examination reveals that the reduced norm of RL and SFT updates stems from a broader distribution of mwggaittudde parameter changes. Considering the proportion of parameters with update magnitudes exceeding 10-5, RL exhibits only 25.0%, 20.7%, and 24.1 % for the math, code, and IF tasks, respectively, in stark contrast to the substantially higher proportions of 79.9%, 78.0%, and 73.9% observed under SFT. This pattern is consistent with recent findings (Mukherjee et al.) 2025]. which suggests that the dominance preserve the knowledge acquired from other tasks. Therefore, the use of on-policy data reduces the likelihood of parameter conflicts and makes RI-trained models more suitable for model merging.
low-magnitude updates
RL-trained models helps
4.2 THE INTRINSIC CHARACTERISTICS OF REINFORCEMENT LEARNING
Another difference of SFT and RL is the optimization targets. Through theoretical and empirical analyses, we find that different optimization targets leading to a natural property of RL, which is "RL optimization is inherently adaptive". Specifically, it adjusts updates according to the model's current capacity and performance. As training progresses, this adaptivity reduces the effective up- date magnitude, which in turn moderates parameter sensitivity and lowers the likelihood of harmful cross-task interference.
Theorem 1. For single query DRI., the expected absolute advantage is upper bounded by
E Ba [|A(a, x)]| I /Var(r(a,z))
where A(a, x)) == r(a, x²) b(x) denotes the advantage, computed as the deviation of the observed reward r(a, x)) from baseline b(zz).
Theorem 2. Based on theorem 1, the advantage estimate A, (a, x) at the n-th step converges to zero in expectation. Since rewards bounded within fixed interval and advantages have mean for each state, we obtain
lim ((IAn(a, x))) -.
Theorem 2 implies that as training stabilizes, the expected scaling factor A diminishes. Conse- quently, RL progressively down-weights parameter updates, whereas SFT continues apply up- dates of similar magnitude even at convergence. We list the proof in the appendix B.] For recently widely used "normalized advantage", A(a,x) -((.)) the theorem 2 also works. We also std(r)
list the proof in the appendix [.2
Update Magnitudes. For sampled trajectories {s) 2=1 the cumulative parameter update magni- tudes can be expressed as
n
n
n
AO b:
Σ 7 G
|
RL 4
Σ 1 A G
Sn
A
G
where G Ve log (y(""">"")) [z*) denotes the stochastic gradient contribution of sample s. The key difference that RL scales each update by its corresponding advantage Af,. which decays over training (Theorem 2), while SFT applies uniform updates. Similar to the analysis in $4.1 the less updates could reduce the influence of conflicted parameter.
Conflict norm One way to understand how RL reduces task interference is to measure the con- sistency of parameter updates across tasks. Task conflicts occur when two tasks push parameters in opposing directions, leading to destructive interference. To capture this effect, we define the conflict indicator matrix as
C(A0r;: A0t;) 20, c A0
where denotes the Hadamard (element- wise) product. Negative entries of indicate conflicting updates, while positive entries correspond aligned updates. We then define the conffict norm as:
|CC(O,, A02) |ccontlet = (C(AOL. A0.,) 200
(10)
7

---

Published as a conference paper at ICLR 2026
30
Math
Code
IF
23
200
ratur- nar
SFT
6.50
7.75
4.83
23-
150
RFT
2.36
2.17
1.70
300
RL
0.78
0.71
0.64
00
25
120
22
07
120
Taiiin stede
7ha r2ng 47eps
Table 2: The norm of A for different tasks and settings.
Figure 5: The norm of SFT and RL
Figure 6: The conflict norm of SFT and RL
where [cs is an indicator function that selects only the conflicting entries. This measure thus quantifies the aggregate strength of parameter conflicts between two tasks.
Implications for Task Interference. Building on these results, we examine the conflict norm between two tasks & and 12. Under SFT:
||C(20;, A0ta))
SET conftict = 1 Σ
G
G t2 llcaatiic
k=
whereas under RL:
12 |IC(s,,, (01g)) = -
a G
Σ
1 G ( M...)llooniiet
A=1
k=1
the verifiable setting where r E {0, 1]. we have |A| as training stabilizes. It follows that
Var(r) S 4.4 and A expectation
[II A0..) coomtica
E[|IC(A,,, A0.,)
SFT confiic
(11)
which formalizes the intuition that RI. reduces cross-task parameter conflicts by down-scaling gra- dient magnitudes through the vanishing advantage.
We plot the norm of update parameter in Sigure5aann the conflict norm in Figure 6 As is shown in the figures, the growth trend of norm and conflict norm of RL is obviously slower than SFT, indicating that the advantages effect during the training process.
4.3 ANALYSIS OF OPTIMIZATION OVER BOTH POSITIVE AND NEGATIVE SAMPLES
The third difference between RL and SFT is that RL is optimized both positive and negative samples. In this section, we provide empirical analyses to elucidate the effects optimization over both positive and negative samples.
To further isolate the contribution of negative samples, we design controlled experiment in which their influence is selectively removed. Specifically, construct RL variant (RL-Pos) in which the advantage values for all negative samples are set to zero, thereby excluding them from gradient updates while retaining the KL regularization and on-policy sampling.
We test two hypotheses: HI (Single-task performance). Models trained with both positive and negative samples should converge higher task-specific accuracy than models trained on positive samples only. H2 (Cross-task conflict). Given the same training budget, models trained with both types of samples should exhibit lower cross-task conflict (as measured by performance degradation after model merging) compared to positive-only training.
To validate H1, we report the convergent accuracy of three Modelss-FFT, RI, and RL-Pos on Math, Code, and IF tasks in Table 3 Results show that while RL-Pos improves over SFT, it still underperforms full RL, confirming that negative samples facilitate better single-task optimization. To test H2, we select RL and RL-Pos checkpoints trained for the same number of steps and apply two merging strategies parameter averaging and Task- Task-Independen Experts (TIEs). As shown in Figure 7 RL consistently suffers less performance degradation, reinforcing the claim that jointly optimizing both positive and negative examples RL guides the model toward an unbiased, task- specific subspace and further reduces parameter conflicts.
8

---

Published as a conference paper at ICLR 2026
buriona sance Drop
Math
Code
IF
Avg.
SFT
61.9
60.5
63.9
62.1
4.
RL
64.6
65.6
90.0
73.4
Tai Methace Nofte
Herding Wethods
RI.-Pos
58.5
61.7
86.1
68.8
CCD Averaging
Math
Code
Figure 7: Performance drop in cross-task merg- ing for RL and RL-Pos models under model av- eraging and TIEs.
Table 3: Convergent performance across tasks.
5 RELATED WORKS
Model Merging of Large Language Models Model merging offers practical alternative to retraining by eliminating the need for access to raw training data computationally expensive fine-tuning procedures (Ilharco et al. 2022] Crisostomi et al., 2024; Yang et al.] (024a). In this work, we focus training-free free model merging, where the merging process relies solely the weights of pre-trained and fine-tuned models. Recent advances have improved this approach through heuristically guided merging strategies based on parameter statistics (e.g. value magnitudes sign alignment) (Yadav et al. 2023), task importance derived from the Fisher Information Matrix (Tam al. 2023), task-specific layer-wise attribution modeling (Wang et al. (024), and trust region con- straints (Sun et al. 2025). Several efforts have also aimed to explain the effectiveness of model merging from theoretical perspective, drawing on tools such as loss landscape geometry (Izmailov)
al. 20188] Gupta et al. (020), bias-variance decomposition (Arpit et al.
2022
Rame et al. 2022).
and linear mode connectivity (LMC) in neural networks (Frankle et al.
2020 Zhou et al. 2024
However, these analyses have been predominantly confined to models fine-tuned via SFT. To date, little attention has been paid to how different fine-tuning paradigms particularly reinforcement-
based methods-affee the model merging and interaction of models.
Analysis of LLM Post- Trainira Post-training has emerged as an effective approach for enhancing the task-specific capabilities of large language models (LLMs) (Grattafiori al. [024|] Team 2024
Shao et al. 2024). Among the most commonly used paradigms, SFT adapts pre-trained models to downstream tasks by training them task-specific datasets, often formatted instructions (Wei
cet al. 2021 Zhou et al., 2023a Chung et al., 2024 Li et al. 2024). In contrast, Reinforcement Learning (RL) is typically employed to align models with human preferences or to optimize per-
formance on specific target tasks (Ouyang et al. 2022: Ahmadian et al.) 2024 Guo et al. 2025
Yang et al., 2025 Zhao et al.] (022).. To deepen understanding of the post-training stage, recent research has explored how different learning paradigms influence model behavior. For instance, several studies investigate memorization and generalization dynamics across knowledge-intensive
and reasoning tasks (Allen-Zhu & Li 2023 Ye et al. 2024: Qi et al., [2024] Chu et al. 2025 Kang ctt al. 2025). Others have examined learning dynamics by contrasting SFT and RL in terms of con- vergence behavior and sample efficiency (Ren & Sutherland 2024 Zeng et al. 2025 Kang al. 2025). Recent research has also highlighted distinctive properties of RI. compared with SFT, show- ing that RL tends produce sparser parameter updates (Shenfeld et al. 2025 and help mitigate catastrophic forgetting (Mukherjee et al. 2025). In this work, we take complementary perspective by analyzing how the choice of post-training paradigm impacts task conflicts in model merging. Our findings demonstrate that models fine-tuna tune-tuned with RL exhibit significantly reduced inter-task interfer- ence compared to those trained with SFT, thereby offering a more robust foundation for multi-task integration.
6 CONCLUSION
This work provides a comprehensive investigation into the influence of post-training paradigms on model merging in LLMss. Our central finding is that RI,, opposed to standard SFT, inherently

---

Published as a conference paper at ICLR 2026
mitigates task conflicts, making it more substantially suitable for model merging. To further understand the mechanism behind this advantage, we isolate and evaluate three components of the RL training objective: (1) on-policy training data, (2) the intrinsic characteristies of RL algorithms, and (3) optimization over both positive and negative samples. Both theoretical and empirical analy- demonstrate that the three components play critical role in conflict mitigation. Taken together, our findings indicate that RL is not merely an alternative to SFT but constitutes a fundamentally more suitable paradigm for multi-task post-training in foundation models. Beyond this, the results provide new insights into how different fine-tuning paradigms shape task conflicts in LLMs and highlight RL as a robust and scalable strategy for developing generalist models.
ETHICS STATEMENT
authors have read and comply with the ICLR Code of Ethics. This study presents an analysis of the impact of different post-training
SFT and RL-on the model merging of LLMs. All experimental components, including models, datasets, and benchmarks, are drawn from publicly available sources and are well-established in the academic literature. Consequently, the research employs no proprictary or sensitive data, involves no human subjects, and presents no additional foreseeable risks beyond those associated with the broader field of LLM research.
REPRODUCIBILITY STATEMENT
Our experiments are built upon open-source models, datasets, and algorithms for both RL training and model merging. We employ greedy decoding for all evaluations. Note that minor variations in results may occur due hardware differences inference framework implementations.
ACKNOWLEDGMENT
The research work has been supported by the Natural Science Foundation of China under Grant No. 62476271
REFERENCES
Wasi Uddin Ahmad, Aleksander Ficek, Mehrzad Samadi, Jocelyn Huang, Vahid Noroozi, Somshubra Majumdar, and Boris Ginsburg. Opencodeinstruet:: scale instruction tuning dataset for code Ilms, 2025. htt.ttpp://rrrd..or///aass
04030
Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadace, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting reinforce style optimization for learn- ing from human feedback in llms. arXiv preprint 4rXiv:2402.147400 2024.
Zeyuan Allen-Zhi and Yuanzhi Li. Physics of language models: Part 3.1, knowledge storage and extraction. arXiv preprint arXiv:2309. 14316, 2023.
Devansh Arpit, Huan Wang, Yingbo Zhou, and Caiming Xiong. Ensemble of averages: Improv- ing model selection and boosting performance in domain generalization. Advances in Neural Information Processing Systems, 35:8265-8277 2022.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Temy, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2 100.07708.777. 2021.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda. Nicholas Joseph, Greg Brockman, al. Evaluating large language models trained on code. arXiv preprint 10Xiv:2107.03:: 00.003337 2021.
Zhijun Chen, Jingzheng Li, Pengpeng Chen, Zhuoran Li, Kai Sun, Yuankai Luo, Qianren Mao, Dingqi Yang. Hailong Sun, and Philip S Yu. Harnessing multiple large language models: A survey Ilm ensemble. arXiv preprint arXiv:2502. 18036, 2025.
10

---

Published as a conference paper at ICLR 2026
Leshem Choshen, Elad Venezian, Noam Slonim, and Yoav Katz. Fusing finetuned models for better pretraining, 2022. httpla::/aaaiiiiiae wtt.i.twg./a....ataoooo.oot.oott abs/ 2/.//2204.000 03044
Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V Le. Sergey Levine, and Yi Ma. SFT memorizes, RL generalizes: A comparative study of founda- tion model post-training In Forty-second International Conference on Machine Learning, 2025. URL http:tts///ppeeeeiiiice. wwt.:/nnne//rmmmme// hett://oruurriddecoeeaaaaaaaaaa
Hyung Won Chung, Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetune lan- guage models. Journal of Machine Learning Research, 25(70):1 2024.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint 10.168,,, 2021.
Donato Crisostomi, Marco Fumero, Daniele Baieri, Florian Bernard, and Emanuele Rodola. c2m3: Cycle-consistent multi-model merging. Advances in Neural Information Processing Systems, 37: 28674-28705, 2024.
DeepSeck-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao. Hui Qu, Hui Li. Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingehuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang. R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang. Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun. Tian Pei. Tianyu Sun, T. Wang, Wangding Zeng, Wanjia Zhao. Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xi- aosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q Wang. Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang. Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo. Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui L., Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforce- ment learning, 2025. URL https: //arxiaavvvv////////a org/ wtggbb///////
abs/ 2201 12948
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In Hal Daumé III and Aarti Singh (eds.), Pro- ceedings of the 37th International Conference on Machine Learning, volume 119 of Proceed- ings of Machine Learning Research, pp. 3259 3269. PMLR, 13-18 Jul 2020. URL https: //proceedings.o www.smmm....s.ss...e. wttpss/we9/////eep..e...epptwwtt wwtp:1199/raaaee20011 html
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprini arXiv: 1783, 2024.

---

Published as a conference paper at ICLR 2026
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning, arXiv preprint arXiv:2501.129.00 2025.
Vipul Gupta, Santiago Akle Serrano, and Dennis DeCoste. Stochastic weight averaging in parallel: Large-batch training that generalizes well. arXiv preprint arXiv:2001.022 02312, 2020.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint ar 038744 2021.
Jian Hu, Xibin Wu, Zilin Zhu, Xianyu, Weixun Wang, Dehao Zhang, and Yu Cao. Openrlhf: An easy-to-use, scalable and high-performance rlhf framework. arXiv preprint 24Xxv:2405.1114: 2024.
Jingeheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung- Yeung Shum. Open-reasoner-zeen:: An open source approach scaling up reinforcement learning on the base model, 2025. http:httss:///rxiv...o http://wwww..orr/////.2..22... 2203..2290) 24290
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. arXiv preprint arXiv:2212.04089, 2022.
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic, 2023. URL https:
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wil- son. Averaging weights leads to wider optima and better generalization. arXiv preprint arXiv:1803.05407, 2018. 2018.
Philip N Johnson-Laird and Ruth MJ Bymne Meta-logical problems: Knights, knaves, and rips. Cognition, 36(1):69-84, 1990.
Katie Kang, Amrith Setlur, Dibya Ghosh, Jacob Steinhardt, Claire Tomlin, Sergey Levine, and Aviral Kumar. What do learning dynamics reveal about generalization in LLM mathematical reasoning? In Forty- second International Conference on Machine Learning, 2025. URL httpsss:: //openrevi ew. net./ met//forum forum? d=bi vUBfTTDo)
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brah- man, Lester James V. Miranda, Alisa Liu. Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Ma- lik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luca Soldaini, Noah A. Smith, Yizhong Wang, Pradeep Dasigi, and Hannaneh Ha- jishirzi. Tulu 3: Pushing fronticrs in open language model post-training, 2025. URL https: 7/arxiv. org. abs/ //tp//411.1111 15124
Chong Li, Wen Yang, Jiajun Zhang, Jinliang Lu, Shaonan Wang, and Chengqing Zong. X- instruction: Aligning language model in low-resource languages with self-curated tross-lingual instructions. In Findings of the Association for Computational Linguistics: ACL 2024, pp. 546- 566, 2024.
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss land- scape of neural nets. Advances neural information processing systems, 31, 2018.
Weishi Li, Yong Peng, Miao Zhang, Liang Ding, Han Hu, and Li Shen. Deep model fusion: A survey. arXiv preprint arXiv: 2023.
Hunter Lightman, Vincet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference Learning Representations, 2023.
Jinliang Lu, Ziliang Pang, Min Xiao, Yaochen Zhu, Rui Xia, and Jiajun Zhang, Merge, ensemble, and cooperate! a survey on collaborative strategies in the era of large language models. arXiv preprint arXiv:2407.06i0 2024.
12

---

Published as a conference paper at ICLR 2026
Sagnik Mukherjee. Lifan Yuan, Dilek Hakkani-Tur, and Hao Peng. Reinforcement learning finetunes small subnetworks in large language models. arXiv preprint arXiv:2505.11711, 2025.
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models fol- low instructions with human feedback. Advances in neural information processing systems, 35: 2022.
Valentina Pyatkin, Saumya Malik, Victoria Graf, Hamish Ivison, Shengyi Huang, Pradeep Dasigi, Nathan Lambert, and Hannaneh Hajishirzi. Generalizing verifiable instruction following, 2025.
Zhenting Qi, Hongyin Luo, Xuliang Huang, Zhuokai Zhao, Yibo Jiang, Xiangjun Fan, Himabindu Lakkaraju, and James Glass. Quantifying generalization complexity for large language models. arXiv preprint arX1::2400.07699, 2024.
Alexandre Rame, Matthieu Kirchmeyer, Thibaud Rahier, Alain Rakotomamoniy, Patrick Gallinari, and Matthieu Cord. Diverse weight averaging for out-of-distribution generalization. Advances in Neural Information Processing Systems, 35: 15821-1110883 2022.
Yi Ren and Danica Sutherland. Learning dynamics of Ilm finetuning. arXiv preprint
arXiv:2407. 10490, 2024.
Wei Ruan, Tianze Yang, Yifan Zhou, Tianming Liu, and Jin Lu. From task-specific models unified systems: A review of model merging approaches. arXiv preprint arXiv:2503. 08998, 2025.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.066 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.
Idan Shenfeld, Jyothish Pari, and Pulkit Agrawal. Rl's razor: Why online reinforcement learning forgets less. arXiv preprint arXiv: 2025.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 2409. 2024.
Wenju Sun, Qingyong Li, Boyang Li, Wen Wang, and Yangliao Geng. Task arithmetic in trust region: A training-free model merging approach to navigate knowledge conflicts, 2025. URL
lttps:://peeneeviet net wow.mmmdddgatttaaaae
Derek Tam, Mohit Bansal, and Colin Raffel. Merging by matching models in task parameter sub- spaces. arXiv preprint arXiv:2312.04339, 2023.
Qwen Team. Qwen2 technical report. arXiv preprint arXiv:2407.10677 2024.
Shubham Toshniwal, Wei Du Ivan Moshkov, Branislav Kisacanin, Alexan Ayrapetyan, and Igor Gitman. Openmathinstruct-2: Accelerating AI for math with massive open-source instruction data. In The Thirteenth International Conference Learning Representations, 2025. URL
http:::///peerreveeee net. forum? wtt.:e/eoooummmmdddatoeee200000
Ke Wang, Nikolaos Dimitriadis, Guillermo Ortiz-Jimenez, François Fleuret, and Pascal Frossard. Localizing task information for improved model merging and compression. arXiv preprint arXiv:2405.07813, 2024.
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoe Le. Finetuned language models are zero-shot learners. arXiv preprint
0116022009001652, 2021.
Orion Weller, Kathryn Ricci, Eugene Yang, Andrew Yates, Dawn Lawrie, and Benjamin Van Durme. Rank1: Test-time compute for reranking in information retrieval, 2025. URL https:
13

---

Published as a conference paper at ICLR 2026
Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Ben Feuer, Siddhartha Jain, Ravid Shwartz- Ziv, Neel Jain, Khalid Saifullah, Siddartha Naidu, et al. Livebench: A challenging, contamination- free Ilm benchmark. arXiv preprint arXiv:2406. 22.1::2406.19114., 4, 2024.
Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 6(3):229-256, 1992.
Lu Xiang, Yang Zhao, Yaping Zhang, and Chengqing Zong. A survey of large language mod- els in discipline-specffi research: Challenges, methods and opportunities. arXiv preprint arXiv:2507.08425, 2025.
Chulin Xie, Yangsibo Huang, Chiyuan Zhang, Da Yu, Xinyun Chen, Bill Yuchen Lin, Bo Li, Badih Ghazi, and Ravi Kumar. On memorization of large language models in logical reasoning. arXiv preprint ar arXhv:2410.23122 2024.
Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. mies-merging: Re- solving interference when merging models. Advances in Neural Information Processing Systems, 36:7093-7115, 2023.
Enneng Yang, Li Shen, Guibing Guo, Xingwei Wang, Xiaochun Cao, Jie Zhang, and Dacheng Tao. Model merging in Ilms, milms, and beyond: Methods, theories, applications and opportunities. arXiv preprint arXiv:2408.07666 2024a.
Enneng Yang, Li Shen, Guibing Guo, Xingwei Wang. Xiaochun Cao,, Jie Zhang, and Dacheng Tao. Model merging in llms, milms, and beyond: Methods, theories, applications and opportunities, 2024b. URL http::/::/ httttt:s//axxiiii /ttpaaaaivv...//////e.eeea.aa.va org/ abs 07666
Wen Yang, Junhong Wu, Chong Li, Chengqing Zong. and Jiajun Zhang. Parallel scaling law: Unveiling reasoning generalization through cross-ling perspective. arXiv preprint arXiv:2510.02272, 2025.
Tian Ye, Zicheng Xu, Yuanzhi Li, and Zeyuan Allen-Zhu. Physics of language models: Part 2.1, grade-school math and the hidden reasoning process. arXiv preprint arXiv:2407.20311, 2024.
Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models are super mario: Absorbing abilities from homologous models a free lunch. In Forty-first International Conference on Machine Learning, 2024. URL https::/ www. foott/foummmio
EqoNaiU8Ex
Xinyue Zeng, Haohui Wang, Junhong Lin, Jun Wu, Tyler Cody, and Dawei Zhou. Lensllm: Unveil- ing fine-tuning dynamics for Ilm selection. arXiv preprint arXiv:2505.03793. 2025.
Kaiqing Zhang, Alec Koppel, Hao Zhu, and Tamer Basar. Global convergence of policy gradient methods to (almost) locally optimal policies. SIAM Journal on Control and Optimization, 58(6): 3586-3612, 2020.
Yang Zhao, Yaping Zhang, Lu Xiang, Jiajun Zhang, Yu Zhou, and Chengqing Zong. Imitate, re- ward and introspect: Enhancing neural machine translation by utilizing large language models as environments. IEEE Transactions on Audio, Speech and Language Processing, 114476-47477 2025.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36:55006-55021 2023a.
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. fnstruction-folloiing evaluation for large language models. arXiv preprint arXiv:2311.07911 2023b.
Zhanpeng Zhou, Zijun Chen, Yilan Chen, Bo Zhang, and Junchi Yan. Cross-task linearity emerges in the pretrinnin-finetuninng paradigm. CoRR, 2024.
14

---

Published as a conference paper at ICLR 2026
A THE USE OF LARGE LANGUAGE MODELS
In this study, LLMs are employed solely to assist with the refinement of written expression. Specifi- cally, they used improve grammar, enhance clarity, and ensure fluency in academic writing. The LLMs were not involved in data analysis, methodological design, or interpretation of results, but served only linguistic polishing tool improve readability and presentation.
B PROOF OF THE CONVERGENCE OF (An)
B. UPPER BOUND ESTIMATION FOR ADVANTAGE WITH BASELINE
In the standard formulation of advantage estimation with a baseline
(Williams) 1992)
A = r b,
where b = E[r,] is the expectation of the reward r. Then we obtain the following bound:
B(IA)) < VE(A2)
VE((r - B(r))3)
(B(r) /E(r2) (((r))-2E(r)BB0))) (r)
(12)
= -VE(P) -E'
Vaa(())
Recent work has established the global convergence of reinforcement learning algorithms (Zhagg et al. 2020). In particular, the expected reward at the n-th step. E(r,), converges to r", the reward of the optimal policy. Formally,
lim E(rx) =r
(13)
Variance Decay. Since the reward is bounded in the interval r.,, we can apply the Bhatia-Davis inequality:
Var(r,) (0())) (Tx) S (r* ((rr)) - -a..
(14)
As (E(rn) + rrt the right-hand side tends to zero, which implies
lim Var(ra) = 0,
which leads to
lim E(IAn]) = 0.
(15)
B.2 CONVERGENCE FOR DISCRETE ADVANTAGE WITH BASELINE AND STANDARD DEVIATION
widely adopted variant of advantage estimation, used for example GRPO and Reinforcet+, normalizes the centered reward by its standard deviation:
E(r)
A
Var(r)
Theorem 3.1. For any bounded reward distribution P, supported on [a, b]. the expectation of the absolute advantage maximized when P, a Bernoulli distribution P,, with the same expectation, i.e.,
Er, (144) S Ep., (14(),
with Er, (r) Ep,, (r)-
Proof. Define the centered random variable X = T u. where μ [E[r]. Then the normalized advantage can be written as
B(XD)
B(((())))
VE(X2)
15

---

Published as a conference paper at ICLR 2026
Consider the family of reward distributions with fixed expectation:
Sμ == Er(r) = μ)-
This set is convex and compact under the topology of weak convergence (by Prokhorov's 's theorem), since all measures are supported on the compact interval [0, 1].
Supporting-Line Reduction. Define the linear functionals
U(P) = (((()))
V(P) = Ep(2)).
Maximizing the functional
U (U, =
VV
over S₁ can be reduced maximizing linear functional. Specifically, if (u., v.) is an optimizer of vb, then set
u.
Ux
2v.
By convex analysis, & admits the supporting- inequality
cVV S u, + A(v v.),
Vo 2 0.
Hence, for any (u,v) u) Snz
u Au S 04 Av. which shows that (u., v.) also maximizes the linear functional
L(u, Au
over S,, Thus, the nonlinear problem
sup (,,,),,, V
and
(u (u-Au) Xv) (u,r)ES,
sup
share the same maximizer U.),
Existence of Optimizer. Since Sy, is compact and L is continuous, the supremum is attained some (u,, v.) Sp. By definition, there exists a distribution P. such that
(u., v.) (U(P.)
Moreover, since is linear in (u, u) and both U and V are linear functionals of P, maximizing L over S,, is equivalent to maximizing
/ (|x) Xx²) P(dx),
P
over the convex, compact set S,,, By the Krein-Milman theorem, the supremum attained at an extreme point of Sy,
Extreme Points of Sy.
The extreme points of S,, are precisely two-point distributions of the form p(z) = cō(x 21) + (1 - c) 6(x - =x2),
satisfying the expectation constraint CD1 -c) μr., in which is the dirac delta function.
Proof.
Sufficiency (two points + extreme). Let P = + (1 c) drg with 1 o x2 and E (0, 1) satisfy
c + (1 c) 222 = μ. Suppose
P XP (1
0 0<><h P2PEES,
16

---

Published as a conference paper at ICLR 2026
For any Borel set E C R we have P(E) (P((E) + +1 (1- ()) Since P(()) = 0 for every y & f21, }2). it follows that Pi((y)) = Pa((u)) = for all such y (otherwise the convex combination would be positive), hence
supp Pi), supp(Ps) S {-1, #2).
Write P. = ar 6n, (1 a;) 62n for 1, with some a; [0, 1]. Because P. E Sp,, each must satisfy the same mean constraint:
D2 (i= 1,2.
0, + (1-a))22=
32
Thus on 02 c, and then the identity P XP1 + (1 A) forces P1 P2 P. Hence P extreme. The degenerate one-point case P = 5g also yields extremality: the mean constraint implies x² μ.. and the same support argument shows any convex decomposition must be trivial.
Necessity (2 3 points
extreme). Suppose the size of p's support, [supp(p)] is greater than 2. Assume P E S₁ has at least three distinct atoms y1, y2, y3 with masses a1, Q22, 03 > 0:
3
P
8g. + Prest. a1 + 02 + 03 > 0,
.were Prest is the remainder (possibly zero). Consider the 3 matrix
M 1
1
1/1
V/
Since rank(M) (M) = 2, there exists nonzero vector 3 (3), B2, B₂) in the nullspace of M, i...
B1 + B2 + B3 = 0,
$191 + B282 + Bays = 0, B + 0.
Define the signed measure
=:= Σ B 6g.
1=1
Then f dv = and x dv = 0. Choose
LE 0, min +,,500 [.3.|
so that all coefficients a, + t3; remain nonnegative. Set
P= = P P tv,
P2 = P - tv.
By construction, P1 and P2 are probability measures (ʃ 1 dP, 1), they satisfy the constraint
df 14), and P P1 1 Ps. Since + 0, we have P1 # Pz, showing that P is not extreme.
If P has infinite support or a non-atomic part, pick three disjoint Borel sets of positive and perform the same construction after restricting to those sets (approximating each by a point via conditional expectations), which yields nontrivial with f du and f dv 0. Hence any P with supp(P) P)] 2 3 fails to be extreme.
Combining the two parts, the extreme points of S,, are precisely the probability measures supported
on most points, i.c. the two-point laws cō., (1 1 c) 82, satisfying cæi (1 c) x2 μ (with the one-point Dirac at μ as a degenerate case).
Convergence of the Discrete Advantage.
Let b = r*, from the previous proof, we have lim E(r,)
n-000
For the discrete rewards, which lives finite grid {a, i2, iix b). the extremizer with fixed mean j E j b] is supported on {ix, b} and
2/ (b
1) ix)
lim E([4,l) lim E p
0.
(16)
ik
Here μ is E(r). This result shows that, under convergence the RL algorithm, the normalized advantage vanishes asymptotically, ensuring that parameter updates diminish and training stabilizes.
17

---

Published as a conference paper at ICLR 2026
Math
Code
IF
Average
SFT
61.9
60.5
63.9
62.1
Averaging
53.8(-10%)
(5.7(-
51.7(-19.1%)
54.4(-12.4%L
TIEs
57.1(-7.6%)
57.2(-5.5%)
47.7(-28.7%)
54.0(-13.0%%
GRPO
64.6
65,6
90.0
73.4
Averaging
60.9(-5.3%)
62.0(-5.4%%
83.7(-7.0%%
68.9(-6.2%)
TIEs
62.1(-3))%)
64.0(-2.5%)
89.8(-0.3%)
72.0(-2.0%)
PPO
62.8
65.2
87.4
71.8
Averaging
60.9(-3.0%)
61.7(-5.4%)
80.0(-8.5%)
67.5(-5.9%
TIEs
60.5(-3.6%)
62.3(-4.4%)
85.2(-2.5%)
69.3(-3.4%
REINFORCE++
62.3
63.7
83.8
70.0
Averaging
61.5(-1.3%%
(1.0(-))2%)
(9.1(-5.6%)
67.2(-4.0%)
TIEs
60.4(-3.0%)
63.1(-1.0%) 81.6(-2.6%)
68.4(-2.33)
Table 4: Performance comparison across three tasks using different merging strategies (Averaging and TIEs), applied both SFT and RL models. The values in parentheses indicate the relative performance drop compared to the original unmerged model.
Math
Code
IF
Average
Based Llama-3.2-3B
SFT
42.7
42.8
42.2
42.6
Averaging
33.1(-22.5%)
38.3(-10.4%)
33.1(-21.6%)
34.8(-18.2%)
TIEs
38.3(-10.2%)
39.8(-7.1%)
27.2(-35.6%)
35.1(-17.6%)
GRPO
41.4
49.8
56.7
49.3
Averaging
40.4(-2.4%)
46.2(-7.2%)
50.0(-1188))
45.5(-7.6%)
TIEs
38.2(-7.7%%
47.7(-4.2%
54.1(-4.6%)
46.7(-5.3%%
Based Llama-3.1-8B
SFT
61.9
60.5
63.9
62.1
Averaging
55.8(-10%)
55.7(-7.8%)
51.7(-19.1%)
54.4(-12.4%%
TIEs
57.1(-7.6%)
57.2(-5.5%)
47.7(-28.7%)
54.0(-13.0%)
GRPO
64.6
65.6
90.0
73.4
Averaging
60.9(-5.3%)
62.0(-5.4%)
83.7(-7.0%)
68.9(-6.2%)
TIEs
62.1(-3.9%)
64.0(-2.5%)
89.8(-0.3%)
72.0(-2.0%%
Based Mistral--mall-244B
SFT
73.9
71.7
76.5
74.0
Averaging
71.5(-3.3%)
71.6(-0%)
66.5(-13.0%)
69.9(-5.6%)
TIEs
68.9(-68%%
70.3(-2.0%)
62.9(-17.7%)
70.4(-4.3%)
GRPO 77.9
73.9
89.6
80.5
Averaging
77.0(-1.2%)
74.0(-0%)
86.4(-3.6%)
79.1(-1.7%)
TIEs
77.9(-0%
73.4(-0.7%)
90.0(+0.4%)
80.4(-0%
Table 5: Performance comparison across three tasks using different merging strategies (Averaging and TIEs), applied to both SFT and GRPO models with different base models. The values in paren- theses indicate the relative performance drop compared to the original unmerged model.
C DETAILED EXPERIMENTS RESULTS
D PARAMETER SIGN CONFLICTS
direct way to assess task interference between large language models (LLMs) is to measure the parameter sign conflict, i.c., the proportion of parameters for which the update directions (weight differences) differ between models trained on different tasks. Prior work has shown that parameter sign conflicts substantially impact merged model performance, as inconsistent signs may lead to destructive interference during parameter fusion (Yadav et al.] (023). To quantify this, we compute the ratio of sign conflicts while varying the kept parameter rate, defined the fraction of parameters
18

---

Published as a conference paper at ICLR 2026
Nelrov
Gerve Nev.
pAoSEAfATY Jays
keyt
RL
SPF
Figure 8: Proportion of parameter sign conflicts under varying proportions of retained parameters.
Math
IF
Code
1 Math Gode
10 Math
40 Code Math
Le randomm
RL
0.50
0.30
1.00
3.00
0.0
6.50
0.50
1.00
6.20
0.40
0.60
1.00
0.50
0.20
0.40
0.60
6.80
3.50
SFT
0.50
0.00
0.50
1.00
0.00
2.00
0.40
0.03
0.30
3.50
Figure 9: The full content of performance landscape
retained after selecting those with the largest absolute values. Figure S reports the conflict rate function of the kept rate. As shown, RL-trained models consistently exhibit lower sign conflict rates than their SFT-trained counterparts, particularly in the highest-importance parameter subset. This suggests that RL optimization produces more compatible update directions across tasks, thereby reducing destructive interference during merging.
D.I PERFORMANCE LANDSCAPE
D.2 THE NUMBER OF MERGING MODEL
We examine how the number of merged models influences task conflicts under different post-training paradigms. In this experiment, we employ the TIEs merging strategy to combine varying numbers of independently fine-tuned models. As shown in Figure the performance degradation patterns differ markedly between paradigms. For RL-trained models, the average performance decreases only gradually the number of merged models increases. In contrast, SFT-trained models exhibit much steeper decline, suggesting that parameter conflicts accumulate more rapidly when merging multiple SFT-baaed models. This result further supports earlier findings that RL produces more task-
19

---

Published as a conference paper at ICLR 2026
orthogonal parameter updates, thereby mitigating cross-task interference in multi-moder merging scenarios.
E
FUTHER DETAILS
E.1 EXPERIMENT SETUP DETAILS
Model and training datasets We adopt the open-source model L.AMA-3.1-8B (Grattafiori al. 2024) as the base model for experiments and analysis. Our evaluation covers five tasks that allow for automatic verification: mathematics, code generation, logical puzzles, instruction following, and ranking. For the puzzle task, we adopt the representative logical reasoning benchmark Knights and Byrne 1990). For the SFT setting, we prepare task ssecc-fccific training data as follows. For the math task, we use subset of OpenMatthnssruutt2 (Ahmad et al.) (025). For the code generation task, we adopt a subset of OpenCodeInstrucc (Ahmad et al. 2025). For the puzzle task, we automatically generate synthetic data using templates implemented by Xie et al. (0024) with the number of people ranging from 2 to 8. For the instruction-folloinng following task, we use the instruction subset from 3-SF (Lambert et al., 2025). For the ranking task, dataset (Weller et al. 2025). For the RL setting, we reuse the same query for the math, code, puzzle, and ranking tasks. To enable verifiable supervision for the instruction-followin task, we employ the signal construction method proposed by |Pyatkin et al. (2025). Before applying reinforcement learning, we first fine-tune the base model the math, code, and instruction-folloiing datasets to equip it with basic instruction-follovii capabilities. This initialization step ensures stable and meaningful reward signals during the RL training stage.
Knaves (Johnson-Laird
the Rankl
Benchmarks and Evaluation Metrics To comprehensively assess model performance across di- verse task domains, we employ a broad set benchmarks. For mathematical reasoning, we adopt GSM8K (Cobbe et al. 2021) and MATH-500 (Hendrycks et al. 202 Lightman et al. 2023), and report the accuracy on each benchmark. For code generation, we utilize the HUMANE EVAL VAL (Chen let al. (021) and MBPP (Austin et al. (2021) datasets, including both the base and plus versions. The evaluation metric is pass @ 7, which measures the percentage of correct solutions in the first attempt. For instruction following, we adopt IFEVAL. (Zhou al.] 2023b) and the follooin subset of LIVEBENCH (White et al. 2024). We report both the loose and strict accuracies for IFE- VAL, and the overall LiveBench score as computed by the benchmark's 's official evaluation script. For the puzzle-solving task, we generate evaluation data using a templated prompt. Specifically, we construct an in-domain test set involving scenarios with to 8 people, and an out-of-domain (OOD) test set with 9 to 13 people. Accuracy is reported separately for in in-oomiin and OOD settings. Fi- nally, for ranking, we evaluate on the pairwise ranking benchmark NEVIR, and report the accuracy computed using the official MTEB evaluation protocol.
E.2 IMPLEMENTATION DETAILS
For SFT experiments, we train models using the OPENRLH Hu et al. 2024) framework. Most hyperparameters adopted from the tulu-3 configuration, including learning rate of 5 x 10-6 batch size of 128, warm-up ratio of 0.03, a learning rate decay scheduler, and a total of training epochs. For the Reinforcement Learning (RL) experiments, we employ the VERL (Sheng al., 2024) framework and GRPO as the RL algorithm. The training configuration includes a learning rate of x 10-6 rollout batch size of 512, rollout count of 8. rollout temperature and top-p both set to 1.0, and a KL-divergence coefficient of 1 x 10 For model merging, we utilize the MERGEKIT toolkit. In the case of TIEs merging and DARE, we follow the default hyperparameter settings recommended by MERGEKIT, setting the sensitivity to 0.8 and the interpolation weight to 0.5. In Trit-Arithmettic we set = 0.7, which is recomended by the original paper.
20


Reviews:
[{"review_id": "R73MQheSGY", "reviewer": "Reviewer_6WLa", "summary": "The paper discovers a phenomenon that the model trained by RL is more suitable for merging than model trained by SFT. From three aspects, the gradient magnitude, conflict parameter magnitude and effect of positive and negative samples, the paper designs experiments and explains the reasons why merging RL models has less performance degeneration.", "questions": "1. The number of merging models is up to 5. What will happen if the models is more?\n2. Whether the number of training data affects the results is not studied. What will happen if merging a model trained on two datasets with a model trained on other two datasets?\n3. The comparison between SFT and RL models’ gradient magnitude is not enough. What will happen if we keep them the same gradient magnitude?\n4. If we train SFT with normalized continuous label like the advantages of RL, will it affect the results?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well written and the motivation is clear.\n2. The experiment results are consistent with the proposed ideas.", "weaknesses": "The designed experiments in the paper can basically verify the proposed factors which affects the model merging. However, there still are some shortcomings. First, the largest number of tasks is 5. Then, the paper does not study the performance of merging model trained on Math and Coder and model trained on IF and Puzzle. It is not clear that whether the number of training data will influence the results. Furthermore, if we keep the gradient of SFT and RL the same, will it affect the results? Finally, if we train SFT in continuous labels, will it affect the results? Also, please see questions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "y9zO14zJ8a", "reviewer": "Reviewer_ojWS", "summary": "This paper addresses a crucial question in Large Language Models (LLMs): how the training paradigm, specifically Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL), impacts the efficiency of model merging. Model merging aims to merge multiple specialized models into a single model, which is often plagued by multi-task conflicts that lead to performance degradation. The paper demonstrates that RL-trained models preserve performance better after merging compared to SFT-trained models.", "questions": "Q1. Why does the Puzzle task in Table 1 show severe degradation (-24% to -56%) even for RL models? Does this imply a unique property of this task's parameter subspace that makes it resistant to the \"orthogonalization\" effect of RL? \n\nQ2. The norm for RL (0.78 for Math) is much smaller than for RFT (2.36). Since both are on-policy, is this difference entirely attributable to the RL objective from Section 4.2? Could the authors provide a clearer quantitative attribution of how much of the total norm reduction (from SFT's 6.50) comes from \"on-policy data\" (SFT to RFT) versus the \"RL objective\" (RFT to RL)? \n\nQ3. Did the authors record the average reward and/or average absolute advantage ∣A∣ over training? A plot showing reward saturation and ∣A∣ decaying to 0 would be a powerful empirical validation for the \"enough is as good as a feast\" hypothesis. \n\nQ4. Does the \"RL-Pos\" variant (learning only from positive samples) also result in a larger update norm (just like in Table 2) or a faster-growing conflict norm (like in Figure 6) compared to full RL?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+  Novel problem: The paper identifies a fundamental and significant question. The finding that RL can mitigate task conflicts is a major contribution, shifting the researcher's focus from post-hoc merging strategies to the pre-hoc training paradigm.\n\n +  Good empirical support: The problem is validated across 5 tasks , 4 merging methods (Table 1) , 3 RL algorithms (Figure 2) and 3 base models (Figure 3), almost fully proves the robustness of the core findings. \n\n+  Deep insight: It not only proves RL is better, but also explore why. The three analysis (on-policy , RL objective , pos/neg samples) is insightful and well-supported by ablations. The performance landscape visualization (Figure 4) shows SFT models are \"brittle\" in task-specific directions while RL models are ''robust``.", "weaknesses": "-  In Table 1, the \"Puzzle\" task shows a very large performance drop even for RL-trained models (e.g., -56% for Averaging, -24% for TIEs). While this is still better than SFT's degradation (-65% to -56%), it is a much starker drop than for other tasks (like Code or IF). The paper notes this but doesn't fully explore why the Puzzle task remains a sharp drop for both paradigms. Does this suggest a limitation of RL's conflict mitigation, or is there a unique property of the logical puzzle task (e.g., requiring very sparse, specific parameter activations) that makes it an outlier? \n\n- Attribution of Norm Reduction (SFT & RFT & RL):In Section 4.1 and Table 2, the update norms are SFT (6.50) > RFT (2.36) > RL (0.78) for the Math task. The author says that on-policy data helps regulate the norm (SFT vs. RFT). However, the gap between RFT (on-policy SFT) and RL (on-policy RL) is also very large (2.36 vs. 0.78). This suggests that the RL objective is also a major contributor to the small norm in Table 2. The paper presents these as separate sections (4.1 and 4.2), but the experiment in 4.1 seems to conflate both effects. The argument could be strengthened by clarifying how much of the norm reduction is due to on-policy data vs. the RL objective. \n\n- The paper theorizes in Section 4.2 that the advantage A vanishes as the model converges. It would be good to see an empirical plot of the average reward r and/or average absolute advantage ∣A∣ over training steps. This would link the theory to the experimental results, strengthening this hypothesis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UMJ44Z1Zgy", "reviewer": "Reviewer_uswu", "summary": "This paper investigates the role of reinforcement learning (RL) versus supervised fine-tuning (SFT) in mitigating task conflicts during large language model (LLM) merging. The authors demonstrate that RL-trained models significantly outperform SFT-trained models in preserving task-specific performance after merging. Through theoretical and empirical analyses, the paper highlights three key mechanisms underlying RL's superiority: (1) the use of on-policy data, which reduces gradient magnitudes and parameter conflicts; (2) RL's adaptive optimization objective, which naturally decreases update magnitudes as training converges; and (3) joint optimization over positive and negative samples, which steers models toward unbiased task-specific parameter subspaces. The results offer valuable insights into building robust, scalable multi-task LLMs without retraining from scratch.", "questions": "Could the authors provide more details about the experiments conducted on the Qwen model? Specifically, were there any unique observations regarding the performance of RL-based training on this model compared to others?\n\nHow do RL training hyperparameters, such as the coefficients for KL divergence and entropy, impact the performance of model merging?\n\nIn the GRPO algorithm, how does the choice of the parameter n (rollout number for each sample) influence the results? Were any ablation studies conducted to analyze its effect on task performance or parameter conflicts? What is the impact of batch size in the context of model merging? Does a larger batch size help stabilize training or reduce task conflicts more effectively?\n\nReferring to the recent \"LoRA Without Regret\" paper by Thinking Machines, could the improved performance of RL in model merging also be attributed to the low-rank nature of its parameter updates? For example, does RL inherently constrain updates to a lower-rank subspace, similar to LoRA, thereby reducing task conflicts and preserving task-specific knowledge better? If so, could this be explicitly analyzed or compared in the context of your experiments?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "The paper addresses a novel and important problem—how different post-training paradigms influence task conflicts in LLM merging. The focus on RL as a more suitable paradigm for model merging is both creative and impactful.\n\nThe experiments are comprehensive, covering diverse tasks, multiple RL algorithms, and various merging strategies. Theoretical insights are rigorously derived and supported by empirical evidence.\n\nThe paper is well-organized, with clear explanations of concepts, methodologies, and results. Figures and tables effectively complement the narrative.", "weaknesses": "See questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "J8TACe1kCC", "reviewer": "Reviewer_W4Vs", "summary": "This paper presents a systematic empirical and theoretical analysis of how reinforcement learning (RL) and supervised fine-tuning (SFT) affect model merging performance in large language models (LLMs). By evaluating five representative NLP and reasoning tasks, the authors demonstrate and dissect why RL-trained models are more robust to parameter/task conflicts and preserve higher performance post-merging. The study identifies three contributing mechanisms: reduced parameter update magnitude due to on-policy data in RL, diminishing update strength from RL’s adaptive optimization objectives, and the role of joint positive/negative example optimization in RL. These analyses collectively suggest RL-trained LLMs are better-suited for consolidation via model merging than their SFT-trained counterparts.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1.\tComprehensive Empirical study: This paper systematically explores the effects of SFT vs. RL across multiple LLMs, tasks (math, code, instruction following, logical puzzles, ranking), and merging algorithms (Averaging, TIEs, Arithmetic, DARE), providing robustness to the experimental findings (see Table 1, Table 4, Table 5).\n2.\tThe writing is meticulous: The paper proposes three hypothesis of why RL performs better than SFT from three different perspectives and verify these hypothesis both from theoretical analysis and experiments", "weaknesses": "1.\tThe experiments focus on pairwise merges, but the field is also interested in merging larger ensembles of specialized models. Do the observed RL advantages scale when merging 3, 4, or more models across highly distinct or hierarchical tasks? If not, are there any limiting factors?\n2.\tFor some tasks/methods, performance loss is reported as -1% to -4%. Are these differences statistically significant? Including error bars or statistical significance tests would add clarity and reliability to interpretation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)