Reinforcement Learning Finetunes Small Subnetworks in Large Language Models
Authors: Sagnik Mukherjee
Lifan Yuan
Dilek Hakkani-Tür
Hao Peng
Venue: NeurIPS 2025 poster ()
Keywords: Large Language Models
Reinforcement Learning
Primary area: deep_learning
TL;DR: RL fine-tuning in LLMs updates a small subnetwork containing 20–30% of parameters leaving rest of the parameters unchanged.

Abstract: Reinforcement learning (RL) yields substantial improvements in large language models’ (LLMs) downstream task performance and alignment with human values. Surprisingly, such large gains result from updating only a small subnetwork comprising just 5%-30% of the parameters, with the rest effectively unchanged. We refer to this phenomenon as parameter update sparsity induced by RL. It is observed across all 7 widely-used RL algorithms (e.g., PPO, GRPO, DPO) and all 10 LLMs from different families in our experiments.
This sparsity is intrinsic and occurs without any explicit sparsity-promoting regularizations or architectural constraints. Finetuning the subnetwork alone recovers the test accuracy, and, remarkably, produces a model nearly identical to the one obtained via full finetuning.
The subnetworks from different random seeds, training data, and even RL algorithms show substantially greater overlap than expected by chance.  Our analysis suggests that this sparsity is not due to updating only a subset of layers; instead, nearly all parameter matrices receive similarly sparse updates. Moreover, the updates to almost all parameter matrices are nearly full-rank,
suggesting RL updates a small subset of parameters that nevertheless span almost the full  subspaces that the parameter matrices can represent. We conjecture that the this update sparsity can be primarily attributed to training on data that is near the policy distribution; 
techniques that encourage the policy to remain close to the pretrained model, such as the KL regularization and gradient clipping, have limited impact.

Reinforcement Learning Finetunes Small Subnetworks in Large Language Models
Sagnik Mukherjee
Lifan Yuan Dilek Hakkani-Tür Hao Peng University of Illinois Urbana-Champaign {sagnikm3, lifan4, dilek, haopeng)eillle 0illinoss.
Abstract
Reinforcement learning (RL) yields substantial improvements in large language models' (LLMs) downstream task performance and alignment with human val- ues. Surprisingly, such large gains result from updating only small subnetwork comprising just 5%-30% of the parameters, with the rest effectively unchanged. We refer to this phenomenon observed across all 7 widely-used RI. algorithms (e-g., PPO, GRPO, DPO) and all 10 LLMs from different families in our experiments. This sparsity occurs without regularizations or architectural constraints. Fine-
parameter update sparsity induced by RL. It is
any explicit tuning the subnetwork alone recovers the test accuracy, and, remarkably, produces a model nearly identical to the one obtained via full finetuning. The subnetworks from different random seeds, training data, and even RL algorithms show substan- tially greater overlap than expected by chance. Our analysis suggests that this sparsity is not due to updating only subset of layers; instead, nearly all parameter matrices receive similarly sparse updates. Moreover, the updates to almost all parameter matrices nearly full-rank, suggesting RI. updates small subset of parameters that nevertheless span almost the full subspaces that the parameter matrices represent. We conjecture that the this update sparsity be primarily attributed to training on data that is near the policy distribution; techniques that encourage the policy remain close the pretrained model, such as the KL regularization and gradient clipping, have limited impact. Our code is available at https:///githbb cww/Sagn wwaignniiiuueejeeoooo htt.a//ooooeeepparaaiti........ pessppasittttinnrr
Introduction
Reinforcement learning (RL) (Sutton al. 1998 Ouyang al. 2022 Ziegler
2020 Rama-
murthy et al.) 20233 Sun et al. 2024 Zhou et al. 2025) is an important post-pretraining stage for adapting large language models (LLMs) sol solving complex reasoning problems (Lightman al.
2023; Wang et al. 2025a c Cui et al. 2025), alignment with human values (Ouyang et al. 2022
Bai et 2022: Dai al. 2023), and adherence safety protocols (Mu et al. [024] Huang et al.
2024 Zhang et al. 2024 Duan et al. 2024). Since these desired behaviors often differ significantly
from those the pretrained model (Ouyang et al. 2022 Bai al. 2022 OpenAl 2024), it is often
assumed that achieving them requires substantial changes to the model parameters and therefore full finetuning widely applied during RI. (Cui al. 2025 HuggingFace 2025 Liu et al. 20256)
Pan et al. 2025
Zeng al. 2025).
Finding
RL-induced parameter update sparsity in LLMs: RL updates only small subnetwork of pretrained large language model, leaving the rest of the parameters effectively unchanged.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Sparsily Comparison:
01.34
75.00
73.00
60
Spirsity (Ng
15.31
35.94
3 (DPO)
Tooo 3708 (DPO)
Espux 2 (PPO)
Dasponde (GAPO)
Lnnw (300
(5MPP))
Figure 1: Comparison in accumulated gradients in the SFT stage vs RL stage for popular released checkpoints. SFT stage has accumulated much denser updates, while RL is mostly sparse.
While RL with full finetuning is allowed to update all parameters, does it actually do so? This paper presents surprising findings and answers this question in the negative. Finding [ is observed in all widely-used RL algorithms studied, namely PPO (Schulman et al.) (017b), GRPO (Shao et al. 2024). ORPÒ (Hong al. 2024), KTO (Ethayarajh al., 2024). DPO (Rafailov et al. (023), SimPO (Meng et al. (2024) and PRIME (Cui et al.) 2025), as well supervised finetuning with rejection sampling (Xiong et al. 2025), and 10 models in our experiments, with the subnetworks consisting of as little as 5% of the model parameters in some cases ($ It emerges without any explicit sparsity-promotin regularization, architectural constraint, or use of parameter-efficient training or pruning methods. Moreover, observe strong consistency among the subnetworks emerged under different random seeds, training data and its order, and even different RL algorithms, suggesting that the pretrained model contains partially transferable structure that is consistent across varied training conditions ($5)
Interestingly, our experiments with PRIME suggest that approximately 20% of the parameters are consistently updated and make up the subnetwork. An additional 8% receive non-zero gradients during training that cancel out, while the remaining 70% parameters remain untouched throughout the entire training process. This observation motivates us to articulate the following conjecture:
Conjecture
Effective Subnetwork
Fine-tuning only the subnetwork identified at the end of RL training, with all other parameters frozen, produces a model that is nearly identical to the original model that has undergone full finetuning, both in test accuracy and in parameter values.
Cancalled out Uodates
5%
More formally, let Oaa denote the parameters after full RL finetuning from the initial model 6mn- Define a binary mask m E {0, 1) [0m where m, = (O Ora)); + and 0 otherwise. We finetune second model from Oini the same data with the same hyperparameters and number of gradient updates, but, at each step, mask the gradients as m O V VeL( right before the parameter update, so that only the identified subnetwork receives non-zero gradients. Let fuab denote the resulting parameters of this subnetwork finetuning process. Conjecture states that Bubb = Oun We provide supporting evidence for the conjecture PRIME and DPO in $4
Figure 2: In PRIME, 72% parameters are never updated, 8% have gradients canceling each other out, and 20% con- stitute the subnetwork that is consistently updated ($6)
find that the updates of RL finetuning do not concentrate in specific layers or components of the transformer. Instead, nearly all parameter matrices receive similarly sparse updates ($3). An exception the layer normalization layers, which receives little to no updates. Moreover, despite the sparsity in the updates, they are almost always full-rank. This suggests that, instead of forcing the

---

updates to reside in low-rank subspace as in LoRA (Hu et al. 2022), RL full finetuning updates small subset of parameters that nevertheless span almost the full subspaces that the parameter matrices can represent.
To better understand the potential reasons for this phenomenon, we conduct series of experiments in $ The results indicate that a primary factor is training on data that is near the policy distribution through, e.g., on-policy RL or performing supervised finetuning (SFT) on the same data before RL (Wang et al., 2024a Cui et al. 2025). Intuitively, it requires less change to the policy distribution when the model learns on a sequence sampled from a distribution close to itself. In contrast, SFT often involves distribution shifts (Zhang et al., 2025) and densely updates the models in our experiments ( Other factors like di-eeeereence regularization towards the reference model, gradient clipping (as used in PPO, GRPO, and PRIME), online vs. offline RL, all have limited impact on the sparsity of accumulated updates.
Our findings have important implications for the RL fine-tuning stage of LLMSs. They suggest that when RL fine-tuning is performed on data closely aligned with the current policy, is typical in practice, optimization concentrates primarily on a small, consistently active subnetwork, leaving most other parameters effectively inert. Conjecture goes beyond the lottery ticket hypothesis (LTH) (Frankle and Carbin 2019): not only can the subnetwork, finetuned in isolation, match the performance of the full model in performance as posited by LTH, we show that also converges to an effectively identical model. These results offer fresh evidence supporting recent findings that RL better preserves the pretrained capabilities compared SFT (Chu al., 2025| /etlur al. (2025). potentially by updating substantially fewer parameters. They also open up new possibilities for more efficient RI. training methods that explicitly leverage this update sparsity (Chen et al. 2022).
2 Related work and Background
2.1 Related Work
The Lottery Ticket Hypothesis (LTH: Frankle and Carbin 2019) posited that dense neural networks contain sparse subnetworks capable of matching the performance of the full model when trained
in isolation. Subsequent extensions to LLMs identified task-specific subnetworks that mitigate
catastrophic forgetting without retraining entire models (Panda et al. [022]] Panigrahi et al., 2023
Yadav al., 2023). Related efforts further discovered sparse subnetworks pretrained language
models crucial for encoding specific knowledge (Marks et al. 2025 Bayazit et al.. 2024 Liu et al..
2022). Recent works have also explored exploiting the winning lotteries improve training efficiency
[Chen et al. (2022). While our observation is closely related to LTH, it differs in three core dimensions:
LT) identifies winning tickets by pruning, while we study subnetwork that naturally emerge; (2) LTH showed that the final model's performance can be reproduced, we show that, in addition to the performance, the exact same model can almost be recovered; (3) LTH focuses on models trained from scratch, while we focus on finetuning from pretrained LLMs.
Sparse training methods exhibit notable benefits in RL efficiency (Graesser al. 2022 Sidahmed et al. 2024| Sokar et al., 2022 Tan et al. Low-Rank Adaptation (LoRA) (Hu et al.
2023] Davelouis et al. 2025). Recent studies also employ
2022 in RL and have achieved competitive performance
alongside significantly reduced computational overhead (Sidahmed et al. 2024 Wang et al. 2025b).
small number of parameters, we find that fine-tuning the naturally emerging subnetwork can match or even surpass the performance finetuning. Moreover, despite its sparsity, the updates are nearly full-rank.
In contrast approaches like LoRA that explicitly constrain updates
of
2.2 Background
We briefly introduce key concepts and notations to be used in onward discussion.
The sparsity of parameter updates. Let 80, 0 E R denote the model parameters before and after finetuning, respectively. We define the update sparsity as sparsity(8") .!') := 1 - ||o| 8%11, /n. where I1-Ilo counts the number of non-zero elements. is important to clarify that even the update 0, Og is sparse, it does not imply that the finetuned model 0, is sparse. Since no sparsity is assumed for 0g sparse update can still result in a dense 0g if 0o is dense.
3

---

Table 1: Parameter update sparsity across different RL algorithms. We report sparsity for suite of open models from Hugging Face. For all models, at least 68.5% and often much more- of the parameters remain unchanged after RI.
Algo.
Init Model
Model
Update Sparsity
On-Policy
Kl.
Online
3. --lu-llul---- 3-88
Liana T-lu-3-88 3-8B
81.4
DPO
Llama 3.1- 1al0-3-7200 10---708-30 SPT
Lana 1.1-3.111111111 DPO
95.2
x
x
GRPO
despseek math 7b instruct
DeapSeak 93 base
deepseek math
JeepSook-81-244
86.0
68.5
/
/
y
/
y
ORPO
mistral 7B v0.
nistral orpo-beta beta
76.9
x
x
Durus-7b-eft
Eurua-7b-Xto
96.0
x
/
x
KTO
Llama 3- Bane- 8B-
Llana -3 Bl--Bsc--88 88 SB----8--PPP---- KTO
81,2
x
/
x
PPO
mittra-77--aft
Sath--aheheerd-mnstttll-----
80.8
/
/
/
SimPO
-eta-Llema-3-40 Instruct
Llama -antaa---Iattuut--B83 8B--inPP
86.5
x
x
x
PRIME
Burue-2-7b-sft
Burua-2-78-PAI1x
77.0
x
Unless otherwise specified, we follow standard practice and consider two bfloat16 values as equal when their absolute difference does not exceed 10 5, account for numerical precision limits All models in our experiments are in the bfloati 16 data type. Sparsity with different tolerance values can be found in Table G the Appendices.
Learning from in-distribution data. We use "in-distiibttion" to refer to training on data drawn from distribution that closely matches the current policy. An example is on-policy RL with, e.g., PPO, GRPO, and PRIME, which sample data online from the evolving policy during training. Another way achieve in-distribution RL is to perform SFT on the same data used for subsequent RI, that the policy adapts to the data distribution before RI. This is a common practice in off-policy methods like DPO and KTO. On-policy methods inherently train on in-distribution data, and off-police methods can also do so when the training data closely matches the policy distribution. As we will show later in So training on in-distribution data is primary reason for the update sparsity in RL.
KL-divergence regularization and gradient clipping in RL. Two widely adopted techniques to keep the policy from deviating too far from the reference model are KL-divergence regularization and gradient clipping. KL regularization (Schulman et al. 2017a), formally computed as DKL ( (Tat) =
Em log [)]()] constrains policy shifts. Gradient clipping further stabilizes training by bounding
{y(a))
the update norm. Both are widely used in algorithms such PPO, GRPO, and PRIME. In $
show that, counterintuitively. both have limited impact on the update sparsity.
3
RL Induces Sparse but Full-rank Updates; SFT Induces Dense Ones
This section aims to answer the following research question
RQI: To what extent does RL induce sparse parameter updates and where in the model do these updates occur? How does SFT compare?
Setup. To answer this question, we analyze publicly released model checkpoints on Hugging Face released by the authors. With the exception of models where RL is applied dircetly to the pretrained base model (e.g., DeepSeek-R1-Zero), most models follow a conventional three-stage pipeline: pretraining, supervised fine-tuning (SFT), and RL. We analyze both the RL and SFT stages by measuring the update sparsity between model checkpoints before and after RL or SFT fine-tuning,
Our experiments cover Tulu 8B/70B (Lambert et al. 2025). Eurus 7B (Yuan et al. 2025 Cui et al.) 2025), DeepSeek Math 7B (Sha et al. 2024), and KTO/SimPO models (Meng et al. 2024).
Results. As shown in Table for all RL-finetuned models, 68.5%-96.0% of parameters remain unchanged after RL. This trend holds across different RL algorithms and model families. Particularly, Deepseek-R1- R1- Re-Zero presents a update sparsity of 86.0%, regardless of directly training from the pretrained base model, namely RL-Zero (DeepSeck-A) Al et al., (025), and large-scale training for over 8K steps. Although exact training configurations are not always available, observe that within the same model family, larger models tend to show higher sparsity. Importantly, all of these models are
E.g., Tig.,PyTorce PyTorch uses 10-5 as the default tolerance for gradient checking.

---

Layerwise Sparsity (DPO)
Layerwise Sparsity (PRIME)
Sparsty
Sparih (80
Lager Brdwe
Average Sparsity
-
MLP
Layer Normalization
Figure 3: Layerwise and perparaamete--marii update sparsity for DPO (left) and PRIME (right). All layers are similarly sparsely updated, with the only exception of the layer normalization layers. which receive little to no updates.
trained using full finetuning without any pparsity-promotee regularization techniques or constraints. This suggests that the update sparsity emerges naturally.
In contrast, Figure shows that SFT induces dense updates (only 6%-15% sparsity). These results offer fresh evidence supporting recent findings that RL better preserves the pretrained capabilities than SFT (Chu al. 2025 Setlur et al. 2025). possibly by updating substantially fewer parameters.
Takeaway
RL leads consistently sparse parameter updates (often >00% sparsity) while SFT produces dense updates. This sparsity emerges without regularizations or architectural constraints.
Almost all transformer layers receive similarly sparse updates. We next examine how parameter updates in RL are distributed across the model lay- ers and individual parameter matrices (e.g., Q, K, projections), based DPO and PRIME models. If updates were concentrated in a subset of layers or modules, one could exploit that structure for enhanc- ing the efficiency (Pan et al. 2024). Figure shows layerwise and per-matrix sparsity across the mod- els. The "Average Sparsity" is over each transformer layer, while others correspond to specific parameter matrices. We observe that parameter updates are distributed across different matrices rather than localized specific ones. Except for consistently high sparsity in layer normalization layers, most layers exhibit similar sparsity levels. Our results show that sparsity is relatively even across the model. This suggests that recovering the behavior of the fully finetuned model requires updating all layers, albeit with only subset of parameters in each.
Table 2: Mean ranks update matrices, as percentage of maximum possible rank across models after RL finetuning.
Model and Algo.
Update Rank (%)
Tulu 8B (DPO)
99.8
Furus 7B (PRIME)
99.5
Llama-3 SB (KTO)
99.2
DeepSeek Math 7B (GRPO)
99.4
Updates sparse but full-rank. Given the sparsity of RL-induced updates, natural question is whether these updates are also low-rank. This distinction between low-rank and sparse updates is important: the former would imply that finetuning operates within a subspace, while the latter implies that a small subset of parameters (that can span the full parameter space) are selected to finetune. Notably, while the updates are sparse, a closer inspection reveals that they are nearly full rank (Tab 2b. To compute rank, we calculate the average rank of individual update matrices across all layers. We further examine the rank of the update for each layer and parameter matrix, and find that most are full-rank throughout the model. These findings suggest that RL updates are localized to subset of parameters that almost span the full subspaces that the parameter matrices can represent, instead of residing in low-rank subspace.
Takeaway 2
All layers and parameter matrices receive similarly sparse but full-rank updates. While layer normalization parameters are almost never updated.
5

---

Finetuning the Subnetwork Alone Can Reproduce the Full-finetuned Model
Since RL primarily fine-tunes a small subnetwork, we investigate two research questions inspired by but extending beyond the Lottery Ticket Hypothesis (LTH);
RQ2:
Can finetuning the subnetwork in isolation recover the performance of the full-finetuned model?
RQ3: Can subnetwork-only finetuning also recover the exact parameter values produced by full
RL finetniigg This section answers both in the positive.
Setup. We follow the procedure described in 1 to obtain two models: one with full finetuning Oun and another finetuned on the same data and hyperparameters but updating only the subnet- work Sabb. We experiment on two very different algorithms DPO, an off-policy algorithm using implicit outcome rewards, and PRIME, an on-policy one with process reward models, to ensure that our conclusion can generalize. We implement DPO with Open-Instruct and PRIME with verl. The exact hyperparameter choices for both can be found in Appendix For evaluation, choose subset of tasks reported in the original papers for both. For DPO we choose the LSAT (Wang et al. 2022). LogiQA (Liu et al. 2021) splits from AGIEval (Zhong et al.] (002)). Math split of MMLU Pro WWang et al. (024b). For PRIME, report results the MATH500 2021) benchmark across difficulty levels. For evaluation in DPO we use olmes
(Hendrycks et al.
Results. In DPO, 94.0% weights
are same between Uun and Babb: it is 90.5% for PRIME. Notably, for both DPO and PRIME, Ouall and Oabb are 100% identical when using toler- ance of 10-4 instead of the default 10 indicating that the two models converge to nearly identical param- eter values. As shown in Tables Ba and 3b) Pub matches or outperforms full on all tasks across both algo- rithms. These results suggest that the parameters outside the subnetwork play little role in the optimization pro- cess, and freezing them has a negligi- ble even beneficial impact the model's performance. We further observe that the training loss consistently lower in the subnetwork finetuning setting than full finetuning (Fig. 4.. They also provide supporting evidence for our Conjecture in $ This finding opens up new possibilities for more efficient RL training methods that explicitly leverage this update sparsity (Chen et al. 2022).
Tule sDPO Bobeet Finturing
Tale DPO (-ul Finsturin)
0.6
0.55
Step
Figure 4: Training loss for training DPO with subnetwork finetuning and full finetuning. Training the subnetwork in isolation consistently causes train loss be lower.
Table 3: Test set performance of fuull and 8uh trained with DPO and PRIME. Training only the subnetwork (0sub) can achieve better performance than full finetuning (Orn). Lvl. indicates the difficulty levels of MATH500.
(a) DPO
(b) PRIME
Task
0
A
Lvl.
Oua
0ub
A
AGIEval LSAT-AR
21.3
24.8
+3.5
93.0
93.0
+0.0
AGIEval LSAT-LR
53.1
54.7
+1.6
85.6
85.6
+0.0
AGIEval LogiQA-en
43.5
45.5
+2.0
82.9
83.8
+0,9
GPQA
32.8
32.8
+0.0
4
71.1
74.2
+3.1
MMLU Pro Math
50.8
51.6
+0.8
5
40.3
45.5
+5.2
Avg
40.3
41.9
+1.6
Overall
69.8
72.2
+2.4
'Open- Instrucn volcengine/verll
verl:
https: /ttpth///ithub.... com/
olmes:

---

Table Subnetwork overlap varying random seeds, training data, and RL algorithms. Despite these changes, subnetworks show non-trivial overlap compared to random- random-guessing guessing baselines.
Variation Axis
Setting
Random
RL Subnetwork
Sparsity
I:: 42
01: 36.7
01: 60.5
63.3
Seed
I2: 123
02: 36.7
02: 60.6
63.3
11: Tulu Data
01: 14.6
01: 26.7
63.3
Data
Ig: PRIME Data
02: 36.7
02: 67.1
85.4
Σ₁: DPO
01: 23.0
01: 59.1
87.1
Seed Data + Algo.
Σ2: PRIME
02: 12.9
02: 33.2
77.0
5
Consistency of Subnetworks Across Seeds, Data, and Algorithms
This section aims to answer the following research question:
RQ4:
How consistent is the RL-updated subnetwork under varying training conditions such as random seed, training data, RL algorithm, and even all of them?
If the subnetwork remains largely consistent across these variations, it would suggest that the identified subnetwork not merely an artifact of specific training configuration but generalizable and transferable structure of the pretrained model.
Setup. To quantify the similarity between two subnetworks, define overlap metric. Let 81 and 52 denote the sparsity levels of two models, and let I and Σz be the sets of indices of the updated parameters. The size of the common subnetwork is given by [Z, n I2]. One-sided overlap is then = I n Z2l/|Zi| = [Z| n 1/1 - 81), which quantifies the proportion of Z₁ that is covered by the subnetwork Iz, i.e., how well Ig captures the parameters updated in Iz. Similarly, 02 [1 nZg /11)//(1))))))))))) quantifies how well I captures the parameters updated in Zz. We compare the observed overlaps 01 and o2 against a random guessing baseline, where subnetwork is constructed by uniformly selecting the same number of parameters as identified by RL (Appendix (
We evaluate three settings: (1) varying the random seed alone, (2) varying the training data alone, and (3) changing the seed, data, and the RL algorithm altogether as a stress test. We conduct controlled experiments and all factors not under investigation are the same. Unless otherwise mentioned, all ablations were done with a batch size of 32, trained for one epoch with the base model Tulu-3-88 8F- When varying the training data, switch between the Tulu preference datase and the PRIME rollout datase To adapt the rollout dataset to DPO format, we select only positive samples, and pair it with randomly sampled negative one. When varying the RL algorithm, we train a DPO model initialized from PRIME REEEE--RREEEuus... Batarus---BB-. 2-7B-SFT and compare it to the PRIME-RL. RL/REruE------ 7-IB-RIMME model.
Results. Table reports our observed overlap. Despite changes initialization, the resulting subnetworks show substantial overlap- well above the random baseline. For instance, varying the random seed yields overlaps of o = 60.5% and 02 = 60.6%. Similar consistency is observed when the training dataset varied. Even under stress test, where the data, seed, and RL algorithm all changed, we still observe notable overlaps of 59.1 19.1% % and 33.2%. These findings indicate the presence of subnetwork that is at least partially transferrable other different settings.
Takeaway 3
For a given base model, we observe substantially higher subnetwork overlap than random guessing across different seeds, training data, and RL algorithms. This suggests the potential of consistent and at least partially transferrable subnetwork structure across these different training settings.
While the observed subnetwork overlap across seeds, datasets, and training algorithms falls short of 100%, it suggests that partial subnetwork reuse may still offer practical utility. In particular, partial subnetwork reuse could reduce redundant computation across repeated RL runs, such as those in hyperparameter sweeps ablation studies, by partially reusing the subnetworks. In addition, one
? "allenai/1lama -.1.-mmm----tt------ -ee---utte--------eeeeeee----- www.--rreeeeeeemiittitee mixture
'PRIME-RL/ REie-RRREEurrss Str///uuuus--oolll 2-Rollout. It has model generations for math datasets alongside label for correctness
7

---

might be able to reuse part of the subnetwork identified by a cheaper algorithm like DPO and reuse it in more expensive ones like PPO, greatly reducing the training cost without sacrificing performance.
Why Do the Subnetworks Emerge?
This section answers investigates the following research question:
RQ5:
What factors contribute to the update sparsity observed in RL finetuning
We investigate the following factors: gradient clipping, KL regularization towards a reference policy, performing SFT prior to RL, and the number of RL update steps. Our investigation suggests that the dominating factor is how close the training data distribution is to the policy's, s, i.e. whether the training data is in-distribution. Another important factor the total number of gradient updates.
HMTI (S]
A.6 2324
60%
Gradient clipping and KL regularization. As discussed in $2.2 gradient clipping and KL reg- ularization are commonly used to keep the pol- icy from deviating too far away from refer- ence model. Since both mechanisms explicitly suppress large parameter updates, they are nat- ural candidates for contributing factors to the observed update sparsity. To test their impact, we train GRPO variant using comparing models with and without these regularization terms. We find that both configurations exhibit comparable sparsity levels, suggesting that neither gradient clipping nor KL regularization a primary driver of the update sparsity. our experiments, the GRPO variant trained with KL regularization achieved a sparsity of 69.8%, while the variant trained without KL regularization reached 68.8 68.8%. Further, SimPO removes the KL term by dropping the base policy normalization in DPO, and as reported in [ SimPO also produces sparse updates, providing further negative evidence for KL.
20%
Global Step
Figure 5: Update sparsity of intermediate check- points of a training run of PRIME. We observe that with more training the sparsity slowly decays.
Performing SFT before RL. A common de- sign choice is to perform SFT on the same data as the subsequent RL Ouyang et al. (2022).
3.0
However, as shown Table I our findings extend to models such as DeepSeek-RI-Zan which forgoes SFT entirely yet still exhibit
outside subnet
2.5
2.0
high update sparsity. This suggests that SFT not a main contributing factor to the update sparsity in RL finetuning.
1.5
1.0
Training duration. is intuitive that with more gradient steps, model is expected to drift more from the base model. Figure 5 shows how update sparsity changes during the training of PRIME. As training progresses, update sparsity gradually decreases but even- tually shows signs of convergence to around 80%. Notably, DeepSeek-R1 R1-rrro under-
0.5
127
191
223
255
287
319
Steps
Figure 6: Percentage of updated weights that
outside the final subnetwork across training steps
'We conjecture that, for the sake of illustration, if one were to perform backpropagation manually on paper with unlimited numerical precision, the resulting parameter updates would be dense. In practice, however, modern computers rely floating-point arithmetic with limited precision. As result, updates with very small magnitudes (e.g., absolute values below 10 (0-4)) cannot be represented and effectively discarded, hence the empirically observed sparsity the parameter updates. Importantly, such near zero updates appear to have negligible impact on model performance, as strong results have been achieved without them. The equivalent question we address in this section is: What factors contribute to these near-zero parameter updates during RL fine-tuning?

---

goes 8K training steps (numbers from Figure 2 in DeepSeek- ct al. 2025 using GRPO, over 2 more than PRIME, but shows a comparable update sparsity (86%). Therefore, we conjecture that training duration's impact update sparsity is more prominent during early training but gradually decreases as training progresses.
Figure shows the percentage of updated parameters (relative to model size) that lic outside the final subnetwork. This proportion increases during the early stages of training but steadily declines in later stages. This trend suggests that some parameters outside the final subnetwork receive non-zero gradient updates that cancel out. Overall, about 8.5% of parameters that are ever updated during training fall outside the final subnetwork.
While it is possible, though less likely, that all models study, including DeepSeek-R1-25 (86.0% update sparsity after 8K steps), are severely undertrained, and that the observed update sparsity would diminish with substantially more training. Nonetheless, question the practicality of this hypothesis since it runs counter to the RI. literature arguing against overtraining to prevent overfitting and improve generalization Fu et al. (2019).
Training on in in-distribution data. Intuitively, when gradients are computed on sequences that the pol- icy already assigns high probabilities to, little update to the parameters would be needed. We evaluated two scenarios: (1) rejection sampling. and (2) DPO on
data by not performing SFT that that DPO with in-distribution data induces sparse data) induces dense ones, this additional experiment can serve control group to isolate the factor of training in- vs. out-of-distributton data.
prior to RL. Since we' ve already learned from $
updates while SFT (with out-of-distribus
As shown in T Our experiments reveal that SFT on in-distribution data produces sparse updates, while DPO with out-of-distribution data produces dense ones. Specifically, performing SFT with Twen/ 5-Math-7B on rejection sampled in-distribution data yields around 90.0% update sparsity. This is reinforced by the examination of a previous work: RAFT++ (Xiong et al. 2024), which performs supervised finetuning with iterative rejection sampling, yields an update sparsity of 99.4%. In contrast, DPO models, with a 6.8% update sparsity. These findings suggest that training on in-distribution data could be major driver of update sparsity in not only RL, but also SFT.
data produces dense updates in Zephyrr- -e---bbbbeee beta
Takeaway 4
We conjecture that training on in-distribution data could be a reason of update sparsity; KL- divergence regularization and gradient clipping have limited impact.
Table RFT indicates eejctiin-sampliing fine-tuning (Touvron 2023 Dong et al. 2023), and RAFT++ is iterative RFT A comparative analysis across SFT and DPO as well as in- vs out-of- distribution training shows that in an-distribution consistently produces sparse updates.
Model
Method
Sparsity (%)
SFT/RL
In-Dist
Qwen2. Awen2.5-Math: 5-Math-77
RFT
91.2
SFT
N-en2.5-Math-7B
RAFT+
69.4
SFT
2lama-3.1-8B-20
SFT
6.8
SFT
X
DPO
6.8
RI.
X
Zephyr-7b-Bet
DPO
7.7
RL
X
Llama-3. 1.ama-3.1-8B-DP. 1-0
DPO
81.4
RL
7 Limitations and Future Work
Because RL is computationally demanding, we choose to vary one factor at time; yet the observed sparsity may actually result from complex interactions among many, an avenue future work should ex- amine. Further, fully controlled experiments are computationally prohibitive, and we thus sometimes resort to resort to public checkpoints. While our experiments focus on language models, it would be interesting to explore the same questions for multimodal and diffusion models. Subsequent research could investigate methods for early identification of the sparse subnetwork and ways to leverage its structure for more efficient learning. Finally, empirical findings invite deeper theoretical

---

analysis, with the goal of uncovering theoretical explanations for the update sparsity in RL. Lastly, while our observations generally hold, there are confounders (for eg. Liu et al. (2025a)). However the confounding reason is not trivial and is worth exploring part of future work. Further to the best of our observations this phenomenon largely applicable to all RI. settings.
Conclusion
Our study reveals that RL finetuning in LLMs updates only sparse subnetwork constituting ap- proximately 5%-30% of total parameters, leaving the rest unchanged. This sparsity emerges without explicit spasity promoting techniques such as regularization or structural constraints. Crucially, finetuning just this subnetwork in isolation reproduces the full model's performance, aligning closely with original parameter values. For given base model across different seeds, datasets and learning algorithms, a non-trivial portion of the subnetwork remains the same. Our findings highlight that learning from in-iistriuttion samples while training is a key driver of this phenomenon, pointing towards more efficient and effective training strategies in RL-based finetuning of LLMs.
Acknowledgments
We would like to thank Pavan Jayasinha, Cheng Wang. Abdul Waheed, Shivanshu Shekhar, Alexander Schwing, Gabriel Stanovsky, Roy Schwartz and Gokhan Tur for their valuable feedbacks and suggestions. Further we extend our gratitude to the members of ConvAl and Alta lab groups at UIUC for their thoughtful remarks our draft. This research used the Delta advanced computing and data resource which supported by the National Science Foundation (award OAC 2005572) and the State of Illinois. Delta is joint effort of the University of Illinois Jrbana-Champan and its National Center for Supercomputing Applications.
References
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. Constitutional ai: Harmlessness from ai feedback, 2022. URL httpt::tt/s:/iiii org/ at//a/bs///2/// 2211222222222
Deniz Bayazit, Negar Foroutan, Zeming Chen, Gail Weiss, and Antoine Bosselut. Discovering knowledge-critical subnetworks in pretrained language models, 2024. URL https://arxiv org/abs/2311 23100 03084
Tianlong Chen, Xuxi Chen, Xiaolong Ma, Yanzhi Wang, and Zhangyang Wang. Coarsening the granularity: Towards structurally sparse lottery tickets, 2022. /Rtp:/tps///rrii.oorr 2202.04736
Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shenghang Tong, Saining Xie, Dale Schuurmans, Quoe V. Le, Sergey Levine, and Yi Ma. Sft memorizes, generalizes: comparative study of foundation model post-training, 2025. URL https:ts://arxx org/
Ganqu Cui, Lifan Yuan, Zefan Wang. Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, Jiarui Yuan, Huayu Chen, Kaiyan Zhang, Xingtai Lv, Shuo Wang, Yuan Yao, Xu Han, Hao Peng, Yu Cheng, Zhiyuan Liu, Maosong Sun, Bowen Zhou, and Ning Ding. Process reinforcement through implicit rewards, 2025. URL htttt://taaaaiiii http:///aaii...rr/////////0/0/
Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. Safe rlhf: Safe reinforcement learning from human feedback, 2023. URL. https: //axxx/xvvvv////////////aaaa/aaa org/abs hag//bb//333 12773
10

---

Fatima Davelouis, John D. Martin, and Michael Bowling, On the interplay between sparsity and training in deep reinforcement learning, 2025. httLp::s///axxii.orgg wttp:///rggbbbb//////// 16729
DeepSeek-AI, Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang. Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng. Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang. Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang. Jingchang Chen. Jingyang Yuan, Junjic Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang. Kuai Yu, Lean Wang. Lecong Zhang. Liang Zhao, Litong Wang. Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang. Minghua Zhang, Minghui Tang. Meng Li, Miaojun Wang. Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei, Tianyu Sun, T. Wang. Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wengin Yu, Wentao Zhang, W. I.. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li. Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang. Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang. Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang. Yue Gong, Yuheng Zou, Yujia He. Yumfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou. Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma Ying Tang, Yukun Zha, Yuting Yan, 7. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning, 2025.
URL htwp::ttp//pte //apxi:////xvvv///ea org. httg//bb//5001 12948
Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. Raft: Reward ranked finetuning for generative foundation model alignment, 2023. URL https: :tap::///axxiie org/ abs/ 06767
Shitong Duan, Xiaoyuan Yi, Peng Zhang, Tun Lu, Xing Xie, and Ning Gu. Negating negatives: Alignment without human positive samples via distributional dispreference optimization. CoRR, abs/2403.03411 033199 2024.
hhttts::://ooo org/10 10. 48550/arXiv arviv. 2403. 03419
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization, 2024. URL
01306.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2019. URL https
//openreview.nette www.eeww.ee/feeee disesfoouummdddr---BBBRet
Justin Fu, Aviral Kumar, Matthew Soh, and Sergey Levine. Diagnosing bottlenecks in deep q- learning algorithms. Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 2021-2030. PMLR, 09-15 Jun 2019. URL httphttts///prceeeee
mlr. w.r.press/vvl 2t00/////...////// html
I Laura Graesser, Utku Evei, Erich Flsen, and Pablo Samuel Castro. The state of sparse training in deep reinforcement learning, 2022. https:/ http:::/wwwwwwwwwa////as/////e/e wrgg/bs/////a/////// 10369
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang. Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Thirty-fffth Conference on Neural Information Processing Systems Datasets and Benchmarks Truck (Round 2), 2021. URL

---

Jiwoo Hong, Noah Lee, and James Thome. Orpo: Monolithic preference optimization without reference model, 2024. URL.atpss//axxxx..or///aa///
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen -Zu, Yuanzhi Li, Shean Wang, Lu Wang. and Weizhu Chen. LoRA: Low-rank adaptation of large language models. International www.:neee///iiiieo
Conference on Learning Representations, 2022. URL m2eVKeePYf9
Xinmeng Huang, Shuo Li, Edgar Dobriban, Osbert Bastani, Hamed Hassani, and Dongsheng Ding. One-shot safety alignment for large language models via optimal dualization. In The Thirty- eighth Annual Conference on Neural Information Processing Systems, 2024. URL https:// http://wwww..eeoooom//ooedddddaa
HuggingFace. Open rl: A fully open reproduction of deepseek-rl, January 2025. URL https //github www.hhgggiggaae///eeeeooooooe/ee open-
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu. Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luca Soldaini, Noah A. Smith, Yizhong Wang, Pradeep Dasigi, and Hannaneh Hajishirzi. Tulu Pushing frontiers in open language model post-training, 2025. URLpttps::
URL/htts:://arxt //araii/////x////aa org/abs/
2411. 15124
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step, 2023. URL
https://arxiv org/ abs/ 2305. 20050
Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. Logiqa: challenge dataset for machine reading comprehension with logical reasoning. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAU'2, 2021. ISBN 9780999241165.
Mingjie Liu, Shizhe Diao, Ximing Lu, Jian Hu, Xin Dong, Yejin Choi, Jan Kautz, and Yi Dong. Prorl: Prolonged reinforcement learning expands reasoning boundaries in large language models, 2025a. httpsss::///axi
org/ abs abs/ 24864
Yuanxin Liu, Fandong Meng, Zheng Lin, Jiangnan Li, Peng Fu, Yanan Cao, Weiping Wang, and Jie Zhou. A win-win deal: Towards sparse and robust pre-trained language models, 2022. URL
05211
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding r1 rl-zero-like training: A critical perspective. arXiv preprint arXiv:2503.20783, 2025b.
Samuel Marks, Can Rager, Eric J. Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller. Sparse feature circuits: Discovering and editing interpretable causal graphs in language models, 2025. Utt/httss://aaxiit wwwp:///////aaaaaeooooooosss
abs abs/ 19647
Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with reference- free reward, 2024. URL/tttps //ariii:///rxxvc..o org/ abs 2405 14734
Tong Mu, Alec Helyar, Johannes Heidecke, Joshua Achiam, Andrea Vallone, Ian Kivlichan, Molly Lin, Alex Beutel, John Schulman, and Lilian Weng. Rule based rewards for language model safety, 2024. URL // arxia.iiiv..... org.orgg//.o.. ats/aab/////../
OpenAl. Openai ol system card. 2024.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models follow instructions with human feedback, 2022. URL
https: //aaxi//aii//////a org/ 02155
12

---

Jiayi Pan, Junjie Zhang, Xingyao Wang. Lifan Yuan, Hao Peng, and Alane Suhr. Tinyzero. htpps//eithub..oo/iaaiii PanrTnnyeeo,, 2025. Accessed: 2025-01-24.
Rui Pan, Xiang Liu, Shizhe Diao, Renjie Pi, Jipeng Zhang, Chi Han, and Tong Zhang. LISA:
Layerwise importance sampling for memory-efficient large language model fine-tuning. In The eiiht Annual Conference on Neural Information Processing Systems, 2024. URLhttps //openreview nwt meet/forunidd-881f1XX
Ashwinee Panda, Berivan Isik, Xiangyu Qi, Sanmi Koyejo, Tsachy Weissman, and Prateek Mittal. Lottery ticket adaptation: Mitigating destructive interference in Ilms, 2024. httph:ts:///rxt
org/aba/2400 16797
Abhishek Panigrahi, Nikunj Saunshi, Haoyu Zhao, and Sanjeev Arora. Task-specific skill localization in fine-tuned language models. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 277033 PMLR, 23 29 Jul 2023. URL BtRLhttts::/// htttps://proceedinggg .mro..gssm.......mm.. mlr press/v202/
panigrahi23a.. html
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In Thirty-sevntth Conference on Neural Information Processing Systems, 2023. URL https:// apenreview.net/Eeee
Rajkumar Ramamurthy, Prithviraj Ammanabrolu, Kianté Brantley, Jack Hessel, Rafet Sifa, Christian Bauckhage, Hannaneh Hajishirzi, and Yejin Choi. reinforcement learning (not) for natural language processing: Benchmarks, baselines, and building blocks for natural language policy optimization. In The Eleventh International Conference on Learning Representations, 2023. URL
https://openreviiet www./eooommddddaaiiiiadaoooo
John Schulman, Sergey Levine, Philipp Moritz, Michael I. Jordan, and Pieter Abbeel. Trust region policy optimization, 2017a.
http///bb/11522/ 05477
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017b.
Amrith Setlur, Nived Rajaraman, Sergey Levine, and Aviral Kumar. Scaling test-time compute org/ 2albb//252.111 12118
without verification or rl is suboptimal, 2025.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathemat- ical reasoning in open language models, 2024. URL /ttthttss//ariii org/ abs/ 2402. 03300
Hakim Sidahmed, Samrat Phatale, Alex Hutcheson, Zhuonan Lin, Zhang Chen, Zac Yu, Jarvis Jin, Simral Chaudhary, Roman Komarytsia, Christiane Ahlheim, Yonghao Zhu, Bowen Li, Saravanan Ganesh, Bill Byrne, Jessica Hoffmann, Hassan Mansoor, Wei Li, Abhinav Rastogi, and Lucas Dixon. Parameter efficient reinforcement learning from human feedback, 2024. URL. https
//arxiv. . org/ abs////3.... 10704
Ghada Sokar, Elena Mocanu, Decebal Constantin Mocanu, Mykola Pechenizkiy, and Peter Stone. Dynamic sparse training for deep reinforcement learning, 2022. Uttpstts::
org/ abs/
2106.04211 04217
Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liangyan Gui, Yu-Xiong Wang, Yiming Yang. Kurt Keutzer, and Trevor Darrell. Aligning large multimodal models with factually augmented RLHF. In Lun-Wei Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguistics: ACL 2024, pages 15088-13111 Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 18.11863/ 1/2024. fin.2244.iinnin--a..777... URL https:://cclattthhete
orge:oog////// 2rgg/2224.indinns-aa.. findings-acl
775/
Richard Sutton, Andrew G Barto, et al. Reinforcement learning: An introduction, volume MIT press Cambridge, 1998.
13

---

Yiqin Tan, Pihe Hu, Ling Pan, Jiatai Huang, and Longbo Huang. RIx2: Training a sparse deep reinforcement learning model from scratch, 2023. URL [nttps///aaxx wwwpxiii.rrg///eo hrggabs/2205... 15043
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cris- tian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan. Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, llyaan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023. URI.https::// httttp::/aaxxvvorrree
/200 09288
Hongru Wang, Cheng Qian, Wanjun Zhong, Xiusi Chen, Jiahao Qiu, Shijue Huang, Bowen Jin, Mengdi Wang, Kam-Fai Wong, and Heng Ji. Otc: Optimal tool calls via reinforcement learning, 2025a. URL httpttts:///axiiv orgg abs 2504. 14870
Peiyi Wang, Lei Li, Zhihong Shao, R. X. Xu, Damai Dai, Yifei Li, Deli Chen, Y. Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce llms step-by-step without human annotations, 2024a.
URL http:///aaxxiv..o/abbbbbs/ 2232.08935
Shangshang Wang, Julian Asilis, Ömer Faruk Akgüi, Enes Burak Bilgin, Ollie Liu, and Willie Neiswanger. Tina: Tiny reasoning models via lora, 2025b. URL httph:ttt//artte org/abrgaaa.....
2504. 15777
Siyuan Wang, Zhongkun Liu, Wanjun Zhong, Ming Zhou, Zhongyu Wei, Zhumin Chen, and Nan Duan. From Isat: The progress and challenges of complex reasoning. JEEE/ACM Trans. Audio, Speech and Lang. Proc., 30:2201 2216, April 2022. ISSN 2329-92900 doi: 10.1109/AALLP.2022. 3164218. URL https: / /ooi. org/10 10. 1109/TASLP TASLP. 2022. 3164218
Yiping Wang, Qing Yang, Zhiyuan Zeng, Liliang Ren, Lucas Liu, Baolin Peng, Hao Cheng. Xuehai He, Kuan Wang, Jianfeng Gao, Weizhu Chen, Shuohang Wang. Simon Shaolei Du, and Yelong Shen. Reinforcement learning for reasoning in large language models with one training example, 2025c. URL httph:ttts///rreeee
org/ w.t./2500 20571
Yubo Wang, Xueguang Ma. Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. MMLU-pro: A more robust and challenging multi-task language understanding benchmark. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024b. URL /ttttts:://ppeeeeeeeee wwt.:orumffoiiic.o..m y10DM6R2r3
Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang. Iterative preference learning from human feedback: Bridging theory and practice for RLHF under KL-constraint. In Forty-first International Conference on Machine Learning, 2024. URL https:://ppnrrevieu http:rewwwwwweeeeeeeeeeeeeeeeeee http://rrmmmd-cccAcaeaaae
Wei Xiong, Jiarui Yao, Yuhui Xu, Bo Pang, Lei Wang, Doyen Sahoo, Junnan Li, Nan Jiang, Tong Zhang, Caiming Xiong, and Hanze Dong. A minimalist approach to Ilm reasoning: from rejection sampling to reinforce, 2025. URL http::tttt://aatttt http//aaaxvv..../baaaa/ abs/ 2504. 11343
Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raffel, and Mohit Bansal. Ties-merging: Resolving interference when merging models, 2023. URL. htttts://axxiii wrwpiiioorg//baa. arg/abs/2//55 01708
Lifan Yuan, Ganqu Cui, Hanbin Wang, Ning Ding, Xingyao Wang, Jia Deng, Boji Shan, Huimin Chen, Ruobing Xie, Yankai Lin, Zhenghao Liu, Bowen Zhou, Hao Peng. Zhiyuan Liu, and Maosong Sun. Advancing I1m reasoning generalists with preference trees. In ICLR, 2025.
14

---

Weihao Zeng, Yuzhen Huang. Qian Liu, Wei Liu, Keqing He. Zejun Ma, and Junxian He. Simplerl- z00: Investigating and taming zero reinforcement learning for open base models in the wild, 2025.
URL https::///aaiiii org/ / 18892
Dylan Zhang, Qirun Dai, and Hao Peng. The best instruction-tuning data are those that fit, 2025.
URL https://arxii.....t// org/ at////bbb////// 04194
Ruiqi Zhang, Licong Lin, Yu Bai, and Song Mei. Negative preference optimization: From catastrophic
collapse effective unlearning. In First Conference Language Modeling, 2024.
Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang. Amin Saied, Weizhu Chen, and Nan Duan. AGIEval: A human-centric benchmark for evaluating foundation models. In Kevin Duh, Helena Gomez, and Steven Bethard, editors, Findings of the Association for Computational Linguistics: NAACL 2024, pages 2299-222222222 -314, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10.1 10.1866/v11/224 1/n.i/.inn.......ng... Andings-naacl.144 naacl.149. URL attps://aclantholooy) org 2024. wiwdinnnnnnnannnn naach 149/
Guanghao Zhou, Panjia Qiu, Cen Chen, Jie Wang. Zheming Yang, Jian Xu, and Minghui Qiu. Reinforced milm: survey on rl-based reasoning in multimodal large language models, 2025.
URL https://arxiv. org/ 2550 21277
Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences, 2020. URL
orgg ab9// 08593
15

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No]. [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justificaton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked While "[Yes] " is generally preferable to "[No]" ", is perfectly acceptable to answer "[No] provided a proper justification is given (e.g., "error bars expensive" or "we were unable to find the license for the dataset we used"). ) In general, answering "[No] "[NA]" is not grounds for rejection. While the questions are phrased in binary way, acknowledge that the true answer is often more nuanced, so please just use your best judgment and write justification elaborate, All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
the checklist one of the factors in their evaluation.
reported because it would be too computationally
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Sections 3-6 provides empirical evidence for the claims made in intro and
abstract
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Separate dedicated limiation section included
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: The paper doesnt discuss theoretical proofs, most results are empirical
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
Justification: All relevant datasets, models are listed, all hyperparameters discussed in detail
5. Open access to data and code
16

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
Guidelines:
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer:
[Yes]
Justification:
All details shared for reproducibility
Guidelines:
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Most accuracies are reported on deterministic setups
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: [TODO]
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: Most experiments performed are with publicly released models, datasets and codebases.
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: No immediate societal impacts of the work
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: No such risks
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All works are cited.
17

---

13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Doesnt release new assets
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Doesn't 't involve crowdsourcing.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: does not involve crowdsourcing nor research with human subjects
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: Does not involve LLMs usage
18

---

A
Appendix
Algorithm
Model (RL Checkpoint)
Tol 1e-8
Tol le-7
Tol 1e-6
Tol 1e-5
DPO
Ailenai/LJamma-e
76.04
76.04
76.14
81.38
DPO
87.58
87.59
87.79
95.24
GRPO
deppscckaiiddeppsekk-mm-----
68.14
68.14
68.14
68.53
ORPO
kaist almmisstall-oppo-bbttt
73.16
73.18
73.23
76.94
ORPO
kaist- huiis--iimmissralloo---pppll
50.40
50.41
50.48
53.23
KTO
openbmb//Euru-70--tte
71.78
71.79
73.14
95.98
PPO
aely:997ymaath-shhhhhrd----mmme mibtral
52.45
52.47
53.21
80.77
PPO
PRIME-RRL/Eumu---7BBPPUU
75.26
75.27
75.36
77.04
SimPO
71.00
71.00
71.10
76.42
SimPO
Llama-3- Beat-----ac----S-- Base-8B-SFT-SimmO
79.47
79.47
79.60
86.52
SimPO
Phnarall77-Immttuue--SiPPO
59.37
59.40
60.31
89.07
SimPO
Histral-7B-Base-SFF-SSim
62.58
62.60
63.56
91.44
Table Sparsity (%) of parameter updates under different thresholds across algorithms and
checkpoints.
B Hyperparameter choices for Gradient Masking experiments
DPO: For DPO, we fine-tuned the LLaMA-3. 1-Tulu-3-88 model using Direct Preference Opti- mization (DPO) with bfloat16 mixeed-precision and DeepSpeed Stage 3 for memory and compute efficiency across processes. Training uses a sequence length of 2048 tokens with an effective batch size of 128, achieved by setting the per-device batch size to I with 16 gradient accumu- lation steps. A linear learning rate schedule is applied with a peak learning rate of 10-7 and a warmup ratio of 0.1, without weight decay. The model is trained for one epoch on the allemai/1lama-3 1...1-tulln tutuuu-22 See---------re-eeeceeeee mixture dataset.
For PRIME, We fine-tune Q 5-Math-7B using on mixture of GSMSK and MATH datasets. The training batch size is set 64. The actor is optimized with learning rate of 5 10-7 while the reward model is trained with a learning rate of x 10-6, We performed four rollouts are performed per sample. We use gradient clipping of 10.0, and a temperature 3 of 0.05. Training is conducted on for 15 epochs.
PRIME:
C
Model Checkpoints: SFT RL sparsity comparison
SFT Checkpoints.
We compare the following base and SFT checkpoints:
meta-11ama/Llama-3.11 3
meta-llama/llana-3.1. 1-70B vs. 1tlllaaiiillaaaa3.--ull---------
Duen/Quen2.5-Matth vs. BRIME-AL/Burrs-2277BB5
RL Checkpoints.
We compare the following SFT and RL-finetuned checkpoints:
1lleaa//llamaa33---Tuu---------- vs. A.lenal/LLimmm33311111---------
PAIME-AL/Eurru-2277899977 vs. PRIME PRIME- http:///PP.........e/e/////rr..
D Training Dynamics
We analyzed intermediate checkpoints of the PRIME model. Notably, our goal is to observe the convergence the sparsity with training time. Does the sparsity decay with gradient steps Does asymptotically reach sparsity level of zero or is the convergence to non-zero point ?
19

---

Experimental setup We analyze 21 intermediate checkpoints from training run of the PRIME model. Let the model parameters at these checkpoints be denoted by 8,, ................................ ., 821, and let mit denote the parameters of the corresponding base model (i.e., SLAME--R/EEruu-7bbe the sparsity between the base model checkpoint as sparsitys sparsity(Ok: Omst)), and the sparsity between two checkpoints i and sporsity, sparsity(O,. 0,).
We define
Key Findings Our analysis begins by examining how the sparsitys evolve with training progress, offering insight into the update patterns.
Suadly
2024
00%
60%
01.00%
48%
784
Global Step
S0ep
(a)
(b)
Figure 7: Sparsity Analysis of intermediate checkpoints of PRIME (a) shows the sparsity
Figure 7 aa illustrates the sparsity of intermediate checkpoints alongside their accuracy the MATH500 and AIME2024 tasks. The plot clearly demonstrates that all intermediate checkpoints exhibit non-trivial sparsity. Furthermore, training progresses, the sparsity converges numeri- cally significant asymptote, suggesting that a substantial proportion of weights remain unaffecctd after prolonged period of training. In Figure bb), report the sparsity of the sparsily; for all consecutive checkpoint pairs, i.e., where j = + 1. Notably, in each successive step, on average, only 7% of the weights receive non-zero gradient update.
E
Random Guessing baseline
model has sparsity 81 and model 2 has sparsity 82, the expected overlap is given by: (1-5)):(1-2)
100
Normalizing like earlier, we get O 1rrandom (1-sw) and O22 O2,random (1-#1) 100 guessing the overlap for model is the density of model 2.
100 i.e. for random
20


Reviews:
[{"review_id": "ELo64Hxq2h", "reviewer": "Reviewer_2DPM", "summary": "This analysis paper studies how fine-tuning a pre-trained LLM via RL changes the weights of the model. Their analysis focuses particularly on parameter sparsity across 7 RL algorithms and 10 LLMs. The authors find that RL only changes a small subset of all parameters (5-30%). In contrast, SFT results in dense updates. They present a number of analyses on what parameters exhibit high sparsity, the rank of the updates, subnetwork overlaps between individual runs, the effect of update constraints (grad clipping, KL), and the effect of in-distribution data on sparsity.", "questions": "1. What are your thoughts on how the subnetworks could be identified prior to or at the beginning of the fine-tuning process?\n2. How does the optimizer choice affect the resulting sparsity?\n3. How does the downstream performance change when you initialize the RL fine-tuned checkpoint with the base model weights for the specific weights that you identified as unchanged (i.e., &lt; 1e-5)? It is possible that the small differences nevertheless impact generation.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: \n\n\n\n1. The findings in this paper are well-presented, insightful, and can be interesting for future works on RL fine-tuning. \n2. They rely on the concept of update sparsity to illustrate how much/little the parameters change, which is a simple concept but demonstrates their findings well.\n3. They evaluate several RL algorithms and models and find that the phenomenon they study persists throughout. This suggests that their findings are generally applicable, rather than an artifact.  \n\nWeaknesses: \n\n\n\n1. While the paper provides extensive empirical evidence, there is no theoretical analysis of why the subnetworks arise. However, I understand the difficulty of this.\n2. It is not investigated and therefore unclear how the choice of the optimizer affects the sparsity ratios.", "was_revised": true, "final_justification": "The additional analyses on the choice of optimizer and weight resetting are useful additions to the main message of the paper. I am happy to maintain my original score.", "rebuttal": "We thank the reviewer for their time and valuable comments. It is great to read that they found the paper well-written and the analysis insightful. Here we try to address their questions\n\n**Weakness 1:**\n\n\"While the paper provides extensive empirical evidence, there is no theoretical analysis of why the subnetworks arise. However, I understand the difficulty of this.\"\n\n**Response:**\n\nWe appreciate the reviewer acknowledging that a detailed theoretical analysis is beyond the scope here. The goal of this paper is to highlight a general phenomenon that is observable across RL trained LLMs. We do think the observation itself is  intriguing and important, and makes a valuable contribution alongside the attributes of it that we study in the form of RQs in our paper. \n\nWe conjecture that the curvature of the loss landscape, quantified by, e.g., Fisher information, can potentially reveal interesting insights. A thorough investigation is beyond the scope of this project and is interesting to conduct in future work.\n\n**Weakness 2:**\n\n\"It is not investigated and therefore unclear how the choice of the optimizer affects the sparsity ratios.\"\n\n**Response:**\n\nOur observations from the standard training recipes of popular open sourced models are that they often use the same AdamW optimizer for the SFT stage [3] as well as the alignment stage [4] (causing sparse updates in RL and dense updates in SFT). And hence the optimizer did not stand out as a candidate causing sparse updates in RL.  To validate this conjecture, we experimented with the RMSProp optimizer for DPO (keeping other training configurations the same), Our observations show that it yields 81.85% sparse updates, aligning with our hypothesis.  We will include this in the appendix section of our revision. \n\n\n**Question 1:**\n\nWhat are your thoughts on how the subnetworks could be identified prior to or at the beginning of the fine-tuning process?\n\n**Response:**\n\nThis is a great question, and we did some initial explorations here. One approach here could be to do a small proxy run to identify the subnetwork. Our initial experiments there revealed that within the first 40% of steps, 58% of the subnetwork can be recovered. This happens because the subnetwork evolves over training. Further, we conjecture that Fisher information could potentially be an effective way to identify the subnetwork prior to training [1]; techniques from the lottery ticket hypothesis [2] literature can also be promising. A thorough investigation is definitely interesting and of practical benefits, which we plan to explore in future work.   \n\n**Question 2:**\n\nHow does the optimizer choice affect the resulting sparsity?\n\n**Response:**\n\nThe response to the weakness 2 can be referred to here. \n\n**Question 3:** \n\nHow does the downstream performance change when you initialize the RL fine-tuned checkpoint with the base model weights for the specific weights that you identified as unchanged (i.e., < 1e-5)? It is possible that the small differences nevertheless impact generation.\n\n**Response:**\n\nThanks for the interesting question, and in order to do a sanity check we set the DPO model’s updates <1e-5 to 0. And observe no change in downstream performance. Indicating these are indeed noisy updates and have little impact on the performance. The downstream task performance after resetting the weights are shared below. \n\n| Task                    | Tulu 8b | Reset weights |\n|-------------------------|---------|--------------|\n| agi eval:: lsat ar      | 21.3    | 21.3         |\n| agi eval:: lsat lr      | 53.13   | 53.13        |\n| agi eval logiqa_en      | 43.47   | 43.62        |\n| GPQA                    | 32.81   | 32.14        |\n\n\n\n[1] Training Neural Networks with Fixed Sparse Masks\n\n[2] The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks\n\n[3] open-instruct GitHub repo - open-instruct/open_instruct/finetune.py\n\n[4] open-instruct Github repo - open-instruct/open_instruct/dpo_tune_cache.py\n\n(We can not link the last two references due to this year's rebuttal policy on not using links)"}, {"review_id": "McVW43ciOs", "reviewer": "Reviewer_ks1c", "summary": "This work provides an empirical study revealing that RL training for LLMs updates only a sparse subset of the full model, leaving the rest unchanged. They conducted numerical verification to compare the difference between open-source models w/ and w/o RL training. Besides, they conducted experiments to compare full-parameter RL training and subnet RL training; their results show subnet training can achieve similar results on many benchmarks and even better performance than the full-parameter training. They conduct ablation studies to investigate the cause for this phenomenon and attribute that training on in-distribution data is a major reason of sparsity.", "questions": "Please see the above section.", "limitations": "N/A", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Pros.\n1. The analysis experiments comparing the difference between LLMs w/ and w/o RL training are reasonable, e.g., comparing 'Llama-3.1-Tulu-3-8B-SFT' and 'Llama-3.1-Tulu-3-8B-DPO'. They also considered different numerical tolerances from 1e-5, 1e-6, 1e-7, 1e-8, which also makes sense.\n\n2. Then they conducted experiments with corresponding subnet of full parameters for RL training and achieved similar or even better performance.\n\n3. They conducted analysis experiments to investigate the reason behind the phenomenon. They analyze different factors: performing SFT before RL, training duration, in-distribution data, KL-divergence regularization, and gradient clipping. The ablation study covers extensive factors; the comparison results between in-distribution and out-of-distribution are significant.\n\nCons:\n\n1. As this is mainly an empirical study, there are many experimental setups that should be further clarified. e.g., What is the optimizer used in each experiment? The checklist section: Experimental setting/detail is marked 'yes', but clearly, I didn't find it in the full content or appendix.\n\n2. If the authors use adaptive optimizers, then the updates are different from gradients. For example, in line 56, the author mentioned they masked the gradient. But it is not clear what is done. \nDo authors mask it before computing the slot variables like momentum or after? Could authors specify how to mask the update for a subnetwork with an optimizer?\n\n3. Some claims lack enough support, like the full rank of updates.\nAgain, for table 2 and lines 170-179, the authors use updates. \n  -  Could you provide more details about the update rank?\n  -  How do you compute the rank, what model parameters did you consider? Which tools or methods did you use to calculate the rank? How many layers or which layers did you consider? \n  -  Did you consider the embedding layer or the final language modeling head layer (which is large in size).\n  -  Did you compare the corresponding rank of updates for SFT?\n\n4. Why don't you consider weight decay as an ablation factor? Is SFT training with weight decay like using AdamW, and your experiments don't?", "was_revised": true, "final_justification": "I adjust my score to positive as the authors provide missed experimental details and explain the effect of weight decay in their response.", "rebuttal": "We appreciate the reviewer’s insightful comments on the strengths and weaknesses of the paper. Here we try to address their concerns. \n\n**Weakness 1:** As this is mainly an empirical study, there are many experimental setups that should be further clarified. e.g., What is the optimizer used in each experiment? The checklist section: Experimental setting/detail is marked 'yes', but clearly, I didn't find it in the full content or appendix.\n\n**Response:** \n\n* Thanks for highlighting this. Since there are ablation experiments throughout the paper, it’s hard to have one consolidated Experimental Setting. We provided detailed descriptions for hyperparameters in **Appendix B**, which is followed throughout the paper, unless otherwise mentioned. Here is a more thorough description of the hyperparameters used, which we will provide in our revised version. \n* For all the experiments in the paper, the optimizer is the **AdamW optimizer** with consistent hyperparameter setting with the original Tulu models as well as the PRIME models. (details in [1] and [2]). Please note that this is the standard setting in most open models and frameworks. (Open-instruct [3], verl[4], GRPO [5] etc. )\n  * Specifically for all DPO experiments we use AdamW with weight_decay=0, learning rate = 5e-7. And for PRIME, we use AdamW with a constant learning rate of 5e-7 for the policy model and 1e-6 for the PRMs (weight decay set to 0). \n* Summarising, for section 4 the experimental details are in appendix B, where we provided details for the gradient masking experiment. For experiments in section 5 all hyperparameters are kept the same as in section 4. And other changes specific to the experiment are discussed in place. The seed and dataset experiments are done with DPO, and the settings are the same as in appendix B. The Seed + Data + Algo experiments are also done with default configurations of DPO and PRIME as discussed in appendix B. We will add a detailed section in the revision. Similarly for section 6, all ablations were done with keeping the same hyperparam settings as in 6, but only ablating the factor mentioned. \n* We will make a detailed subsection in Appendix to address each section’s hyperparameter choices separately. \n\n\n**Weakness 2:** If the authors use adaptive optimizers, then the updates are different from gradients. For example, in line 56, the author mentioned they masked the gradient. But it is not clear what is done. Do authors mask it before computing the slot variables like momentum or after? Could authors specify how to mask the update for a subnetwork with an optimizer?\n\n**Response:** Concretely, on every backward call we register a tensor‑level hook (param.register_hook) that zeroes the entries outside the sub‑network. PyTorch fires this hook immediately after autograd has produced the raw gradient and before the tensor is stored in param.grad; the masked value is therefore what AdamW uses to compute its first‑ and second‑moment estimates. Effectively, this forces that all parameters outside the subnetwork to always receive zero updates\n\n\n**Weakness 3:** Some claims lack enough support, like the full rank of updates. Again, for table 2 and lines 170-179, the authors use updates. Could you provide more details about the update rank? How do you compute the rank, what model parameters did you consider? Which tools or methods did you use to calculate the rank? How many layers or which layers did you consider? Did you consider the embedding layer or the final language modeling head layer (which is large in size).\n\n**Response:** \n\nThank you for your detailed question. Here we have detailed our approach in computing the rank as reported in the paper. \n\n*Short summary response*\n * tool used for computing rank is torch.linalg.matrix_rank\n * Layers considered - all layers except layer norm, including embedding and language modeling head\n * Method to compute rank - detailed below\n\nHere is the detailed approach to how we computed the reported ranks: In order to compute rank, \n* We first compute the element wise delta matrices for all sublayers (please note that we considered all layers and sublayers) - This includes embedding, q_proj, k_proj, v_proj, FFN, including the lm_head. Which results in a matrix containing the element wise update values for each sublayer. We don't compute ranks for the layernorm weights, since the weight vectors are 1 dimensional ( These amount to only 0.04% of all model weights ).\n* Next compute rank for all of these matrices. For each matrix we compute their matrix ranks separately with built-in function  torch.linalg.matrix_rank (this rank is computed using Singular Value Decomposition). \n* Next we compute the rank as a fraction of the maximum possible rank for the respective matrices\n* We report the mean of this fraction (across all sublayers) in the paper. \n\n\n**Weakness 4:** Did you compare the corresponding rank of updates for SFT?\n\n**Response:**\n\nThe corresponding rank of updates in SFT is captured in the table below. Despite SFT performing dense updates, the update rank is slightly lower than RL.  The fact that RL updates are much sparser makes this observation even more interesting. (Note: The following ranks for the SFT stage is computed in the same approach detailed above)\n\n\n| Model  family        | SFT rank | RL rank |\n|----------------|----------|---------|\n| tulu 8b        | 99.75    | 99.8    |\n| Eurus 7b       | 97       | 99.5    |\n| Llama 3 8b KTO | 98       | 99.2    |\n\n\n**Weakness 5**: Why don't you consider weight decay as an ablation factor? Is SFT training with weight decay like using AdamW, and your experiments don't?\n\n\n**Response:** Thank you for your thoughtful question. For the tulu series of models, both the SFT stage [6] as well as the DPO stage [7] use a  weight decay of 0; while the SFT yields dense updates and the DPO stage yields sparse ones. This provides evidence that weight decay is not a major factor in terms of the sparsity of the updates and therefore we do not think it is a  contrasting factor to ablate. Further to validate our conjecture we set weight decay to 0 for the policy model as well as the reward model in PRIME and trained for 150 steps on GSM8k and MATH (please note due to the time constraint of the review period, its infeasible to train for more steps). And the resulting model had 81.6% sparse updates. This further strengthens our conjecture that weight decay is not the root cause for sparse updates. \n\n\n\n[1] Tulu 3: Pushing Frontiers in Open Language Model Post-Training\n\n[2] Process Reinforcement through Implicit Rewards\n\n[3] open-instruct GitHub repo - open-instruct/open_instruct/dpo_tune_cache.py\n\n[4] Verl GitHub repo - verl/workers/fsdp_workers.py\n\n[5]deepseekmath: Pushing  the Limits of Mathematical Reasoning in Open Language Models\n\n[6] open-instruct GitHub repo - scripts/train/tulu3/finetune_8b.sh\n\n[7] open-instruct GitHub repo - scripts/train/tulu3/dpo_8b.sh"}, {"review_id": "t9Kk73qUsB", "reviewer": "Reviewer_j68b", "summary": "RL has become an important learning algorithm in the post-pretraining stage of LLMs. However, the precise mechanisms through which RL yields its benefits are subject to ongoing investigation. The authors address this question from the lens of parameter update sparsity. They find that in RL finetuning subnetworks emerge that are sparse (less than 30% of the parameters receive non-zero, non-cancelling gradients), but full rank. Using two diverse RL methods - off-policy DPO and on-policy PRIME - they demonstrate that freezing the remaining 70% of the weights when fine-tuning has a negligible or positive effect on benchmark accuracy. According to the authors, this stands in contrast to SFT, which produces dense updates. Emerging subnetworks show considerable consistency across variations in the training pipeline (varying random seeds, preference datasets and RL algorithms), indicating a transferable subnetwork structure. However, when SFT is trained based on rejection sampling (in-distribution data) and DPO on out-of-distribution data, they find that SFT produces sparse updates and DPO dense updates, indicating that a main reason of update sparsity is training on in-distribution data.", "questions": "In section 4 the authors answer the two questions - can finetuning the subnetwork in isolation recover the performance of the full-finetuned model? And can subnetwork-only finetuning recover identical parameters as full RL finetuning? - in the affirmative. For this they use DPO and PRIME, covering on and off-policy algorithms. However, the comparison of in vs. out-of-distribution data should be much more at the center of their paper, given the striking difference this makes, and I would like to see this comparison here. Also, given that the paper addresses the difference between SFT and RL, I would be curious to see the same approach applied to SFT. \n\n\nThe paper frames sparsity as a mechanistic hallmark of RL finetuning, but the evidence points instead to the role of distributional alignment. Experiments in §6 show that both RL and SFT can produce sparse updates when the training data is in-distribution. Thus, claims about RL-specific mechanisms should be tempered or reframed accordingly.\n\nIf these concerns are addressed, I recommend acceptance.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "*Strengths*\n\nThe paper is timely and investigates the internal mechanisms of RL algorithms and their difference to SFT from an interesting perspective: parameter update sparsity. While this lens has been studied before, the paper focuses on naturally emerging small subnetworks and links these to in- vs. out of distribution training. This, to the best of my knowledge, is a novel insight. The paper furthermore spans a considerable amount of RL algorithms and language models, is methodologically well-conceived and clearly structured.\n\n*Weakness*\n\nThe paper’s central claims occasionally overgeneralize what is otherwise a strong empirical result. The evidence suggests that sparsity in parameter updates is not an intrinsic property of RL algorithms themselves, but rather a result of training on data that closely matches the current policy distribution. The authors show that RL with out-of-distribution data does not induce sparsity, and conversely, SFT with in-distribution data does. Yet at several points in the paper, claims are made that attribute sparsity to RL itself.\n\nBelow-mentioned sentences overgeneralize and contribute to occasional lack of clarity around what the key driver of sparsity is — algorithmic properties of RL, or data distribution. Such claims should be contextualized more. The paper already provides the corresponding evidence for this contextualization. A more accurate framing would emphasize that update sparsity is primarily driven by distributional alignment, and RL finetuning often happens to operate in this regime. Recasting the finding this way would make the paper’s contribution both more accurate and more broadly applicable.\n- “It emerges intrinsically from RL finetuning, without any explicit sparsity-promoting regularization ...”\n- “Our findings have profound implications on RL finetuning of LLMs. They suggest that RL concentrates optimization entirely on a small and consistently active subnetwork, while the remaining parameters remain effectively inert.”\n- “These results offer fresh evidence supporting recent findings that RL better preserves the pretrained capabilities than SFT.”", "was_revised": true, "final_justification": "I appreciated the authors’ thorough rebuttal. I was glad to see that they have taken the concern about overgeneralization seriously and that they planned to revise the framing to clarify that update sparsity is primarily driven by training on in-distribution data, rather than being an intrinsic property of RL algorithms themselves. Their proposed rewordings were appropriate and would improve the scientific accuracy of the claims. I am generally satisfied with the direction of the planned revisions and remain supportive of accepting the paper, maintaining my original score as is.", "rebuttal": "We sincerely thank the reviewer for their kind comments on the novelty of the finding. We are glad that they found the paper timely as well as methodologically well-conceived and clearly structured. Here we try to address their concerns.\n\n**Weakness 1:** \n\nThe paper’s central claims occasionally overgeneralize what is otherwise a strong empirical result....\n\n**Response:** \n\nWe thank the reviewer for raising this concern. As the reviewer correctly points out, some statements could unintentionally be interpreted as sparsity being attributed to RL algorithms themselves, rather than RL as a post-training fine-tuning stage for LLMs, which was our intended meaning. We plan to revise relevant statements to address this confusion. Specifically, for those raised by the reviewer, we plan to revise:\n\n* “It emerges intrinsically without any explicit sparsity-promoting regularization, when fine-tuning with on-policy RL algorithms (e.g., PPO, GRPO, REINFORCE) or off-policy algorithms (e.g., DPO) that follow an initial SFT stage on preferred responses, as is common practice.”\n\n* “Our findings have important implications for the RL fine-tuning stage of LLMs. They suggest that when RL fine-tuning is performed on data closely aligned with the current policy, as is typical in practice, optimization concentrates primarily on a small, consistently active subnetwork, leaving most other parameters effectively inert.”\n\n* “These results provide fresh evidence supporting recent findings that RL fine-tuning, usually performed on data close to the current policy, better preserves pretrained capabilities compared to SFT, usually performed on off-policy data.”\n\n**Question 1:**\n\nIn section 4 the authors answer the two questions - can finetuning the subnetwork in isolation recover the performance of the full-finetuned model? And can subnetwork-only finetuning recover identical parameters as full RL finetuning? - in the affirmative. For this they use DPO and PRIME, covering on and off-policy algorithms. However, the comparison of in vs. out-of-distribution data should be much more at the center of their paper, given the striking difference this makes, and I would like to see this comparison here. Also, given that the paper addresses the difference between SFT and RL, I would be curious to see the same approach applied to SFT.\n\n**Response:** \n\nThis is a great question - and actually the distinction between the on-distribution vs out-of-distribution nature of the data is quite important. However, we would like to highlight that in our experience, an out-of-distribution training does not give rise to sparse updates. And hence finetuning the subnetwork alone induced by RL on out-of-distribution data would require finetuning almost the full model, which we believe we recover full finetuning’s performance and parameter values in a trivial way. Similarly, SFT also causes dense updates to the base model, and hence the subnetwork finetuning experiment will require us to finetune almost the entire model.  \n\n**Question 2:** \n\nThe paper frames sparsity as a mechanistic hallmark of RL finetuning, but the evidence points instead to the role of distributional alignment. Experiments in §6 show that both RL and SFT can produce sparse updates when the training data is in-distribution. Thus, claims about RL-specific mechanisms should be tempered or reframed accordingly.\n\n**Response:**\n\n\nWe appreciate this thoughtful comment. We will definitely make the necessary changes (as already highlighted in our response to weakness 1) such that these concerns are appropriately addressed."}, {"review_id": "1fH9R6Aws6", "reviewer": "Reviewer_jVpM", "summary": "This paper investigates a surprising phenomenon in reinforcement learning (RL) for large language models (LLMs): RL fine-tuning consistently results in sparse parameter updates, affecting only 5–30% of the model’s weights while achieving performance comparable to full-model tuning. This parameter update sparsity emerges naturally—without any explicit regularization—and is observed across a wide range of RL algorithms and LLM architectures.", "questions": "1. The authors attribute the observed parameter update sparsity primarily to training on in-distribution data. Could the authors validate this hypothesis by conducting ablation studies using out-of-distribution data within the RL fine-tuning process? Such experiments would help clarify whether the sparsity effect is truly tied to data distribution or influenced by other factors.\n\n2. All experiments in the paper are conducted on transformer-based architectures. Is the observed parameter update sparsity unique to transformers, or does it generalize to other neural network architectures as well? Evaluating this phenomenon beyond transformers would help assess the broader applicability of the findings.\n\n3. The paper suggests that a large portion of parameters receive little to no updates during RL fine-tuning. Could the authors elaborate on why certain gradients consistently remain close to zero across different runs? Is this behavior due to properties of the model, the data distribution, or the RL algorithms themselves? A deeper analysis of this could offer valuable insights into the mechanisms behind update sparsity.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. Novelty: A key strength of this paper lies in its novel observation of parameter update sparsity during RL fine-tuning of large language models. Unlike prior work that focuses on architectural or algorithmic sparsity, this paper uncovers an intrinsic sparsity pattern that emerges naturally across multiple RL methods and model families—without any explicit regularization. \n\n2. Broad Impact: This insight is both surprising and original, offering a new perspective on how RL updates influence LLMs and opening up potential directions for more efficient fine-tuning strategies. Potential studies on efficient learning or pruning could benefit from this study. \n\n3. Another strength of the paper is its comprehensive experimental validation, covering 7 widely-used RL algorithms (e.g., PPO, GRPO, DPO) and 10 LLMs from different model families. This broad evaluation strongly supports the generality of the observed sparsity phenomenon and adds credibility to the paper’s claims across both methods and architectures.\n\nWeakness: \n\n1. A notable weakness of the paper is that the contributions are largely observational, with limited theoretical or mechanistic analysis. While the discovery of parameter update sparsity is intriguing, the paper does not delve deeply into why this phenomenon occurs or provide formal explanations to support its findings. As a result, the work raises interesting questions but leaves many of them unanswered. \n\n2. Although the updates are sparse, they are also nearly full-rank within each affected matrix, which suggests that their structure is complex and difficult to predict in advance. As a result, it is currently impractical to design algorithms that can exploit this sparsity proactively, limiting the immediate applicability of the findings to algorithmic improvements or efficiency gains.", "was_revised": true, "final_justification": "**Review Summary**\n\nThis paper investigates a surprising phenomenon in reinforcement learning (RL) fine-tuning for large language models (LLMs): RL fine-tuning consistently produces sparse parameter updates, affecting only 5–30% of the model’s weights, while achieving performance comparable to full-model tuning. This aligns with my own intuition and experience with RL, which tends to be a lighter compared to SFT.\n\n**Concerns and Clarifications**\n\nHowever, some of the paper’s conclusions could be misleading in their current form. For example, in Line 170, the authors raise the question of whether these updates are also low-rank and then claim they are full rank. Yet, this conclusion is based solely on computing the practical rank, which may lead readers to believe that low-rank updates are impossible in this context. This is not necessarily the case—low-rank approaches could still be viable.\n\nSimilarly, while the paper correctly states that SFT updates are dense, prior work [1] has shown that SFT can also yield sparse updates under certain conditions. A more nuanced discussion of this would strengthen the paper and avoid overgeneralization.\n\n**Recommendation**\n\nDespite these issues, I find the core empirical findings compelling and valuable for the community. The contributions outweigh the identified shortcomings. I therefore recommend **Weak Accept**.\n\n[1] Interpreting and Improving Large Language Models in Arithmetic Calculation. ICML 2024.", "rebuttal": "We thank the reviewer sincerely for their time and valuable feedback. We are grateful on their comments on the originality, novelty and potential to influence future work in pruning and efficient learning.  Here we try to address their concerns. \n\n**Weakness 1:** \n\nA notable weakness of the paper is that the contributions are largely observational, with limited theoretical or mechanistic analysis. While the discovery of parameter update sparsity is intriguing, the paper does not delve deeply into why this phenomenon occurs or provide formal explanations to support its findings. As a result, the work raises interesting questions but leaves many of them unanswered.\n\n**Response:**\n\nWe thank the reviewer for acknowledging that the observation itself is quite intriguing. The goal of this paper is to highlight a general phenomenon that is observable across RL trained LLM models. While some questions are left unanswered, we believe those are out of the scope, and can be addressed in follow up research.\n\n\n**Weakness 2:**\n\nAlthough the updates are sparse, they are also nearly full-rank within each affected matrix, which suggests that their structure is complex and difficult to predict in advance. As a result, it is currently impractical to design algorithms that can exploit this sparsity proactively, limiting the immediate applicability of the findings to algorithmic improvements or efficiency gains.\n\n**Response:** \n\nWe see immediate efficiency gains in the following possible venues\n\n* One conclusion from our paper is that this subnetwork is consistent (as compared to a random baseline) across variations in seed and dataset. An immediate efficiency gain could be to do a small proxy run to identify the subnetwork. And when searching for better hyperparameters this subnetwork can be reused directly, and this bypasses the need to early detect the subnetworks. \n\n* Further, We conjecture that the curvature of the loss landscape, quantified by, e.g., Fisher information, can potentially reveal interesting insights. A thorough investigation is beyond the scope of this project and is interesting to conduct in future work.\n\n\n\n**Question 1:** \n\nThe authors attribute the observed parameter update sparsity primarily to training on in-distribution data. Could the authors validate this hypothesis by conducting ablation studies using out-of-distribution data within the RL fine-tuning process? Such experiments would help clarify whether the sparsity effect is truly tied to data distribution or influenced by other factors.\n\n**Response:** \n\nThanks for the great question. We aimed to answer this question in Table 5. Row 5 (in table 5) reports update sparsity for an off-policy DPO model, which leads to dense  updates to the base model. This strengthens our conjecture that the nature of in-distribution data causes these sparse updates. \n\n**Question 2:** \n\nAll experiments in the paper are conducted on transformer-based architectures. Is the observed parameter update sparsity unique to transformers, or does it generalize to other neural network architectures as well? Evaluating this phenomenon beyond transformers would help assess the broader applicability of the findings.\n\n**Response:**\n\nThanks for the  interesting question. We focus on transformer-based LLMs because almost all recent LLMs use the transformer architecture and our findings based on it are more practically relevant. As we acknowledged in the “Limitations and Future Work” section and correctly pointed out by the reviewer, it would be interesting to other architectures like diffusion, multimodal LLMs, state-based models; these architectures have fewer readily available open checkpoints that have undergone RL, and a thorough investigation might require more extensive training workload and is therefore left to future work. \n\n**Question 3:** \n\nThe paper suggests that a large portion of parameters receive little to no updates during RL fine-tuning. Could the authors elaborate on why certain gradients consistently remain close to zero across different runs? Is this behavior due to properties of the model, the data distribution, or the RL algorithms themselves? A deeper analysis of this could offer valuable insights into the mechanisms behind update sparsity.\n\n**Response:** \n\nOur experiments in section 6 support our conjecture that this is happening due to training data being in-distribution (or on-policy). To validate this, we considered a suite of models that were trained with in-distribution data as well as out-of distribution data. (refer to table 5). Our observations indicated that \n1.  Rejection sampling (i.e. in distribution SFT) causes sparse updates, while out of distribution DPO causes dense updates. \n\n2. SFT causes dense updates (which is typically out of distribution) and in-distribution DPO causes sparse updates. Hence we conjecture that the observed sparsity could be driven by the data being in-distribution."}]

Author rebuttal:

Meta-review: This work provides an empirical study revealing that RL training for LLMs updates only a sparse subset of the full model, leaving the rest unchanged. They conducted numerical verification to compare the difference between open-source models w/ and w/o RL training. Besides, they conducted experiments to compare full-parameter RL training and subnet RL training; their results show subnet training can achieve similar results on many benchmarks and even better performance than the full-parameter training. They conduct ablation studies to investigate the cause for this phenomenon and attribute that training on in-distribution data is a major reason of sparsity.

All reviewers appreciate the novel insights found in this paper. The AC agrees and thus recommends acceptance.
Decision: Accept (poster)