PLoP: Precise LoRA Placement for Efficient Finetuning of Large Models
Authors: Soufiane Hayou
Nikhil Ghosh
Bin Yu
Venue: ICLR 2026 Poster (2026)
Keywords: Efficient Finetuning of Large Language Models;LoRA;
Primary area: foundation or frontier models, including LLMs
TL;DR: A novel lightweight method for module selection for LoRA finetuning

Abstract: Low-Rank Adaptation is a widely used finetuning method for large models. Its small memory footprint allows practitioners to adapt large models to specific tasks at a fraction of the cost of full finetuning.  Different modifications have been proposed to enhance its efficiency by, for example, setting the learning rate, the rank, and the initialization. Another improvement axis is adapter placement strategy: when using LoRA, practitioners usually pick \emph{module types} to adapt with LoRA, such as Query and Key modules. Few works have studied the problem of adapter placement, with nonconclusive results: original LoRA paper suggested placing adapters in attention modules, while other works suggested placing them in the MLP modules. Through an intuitive theoretical analysis, we introduce PLoP (Precise LoRA Placement), a lightweight method that allows automatic identification of module types where LoRA adapters should be placed, given a pretrained model and a finetuning task. We demonstrate that PLoP consistently outperforms, and in the worst case competes, with commonly used placement strategies through comprehensive experiments on supervised finetuning and reinforcement learning for reasoning.

Published as a conference paper at ICLR 2026
PLoP: PRECISE LORA PLACEMENT FOR EFFICIENT FINETUNING OF LARGE MODELS
Soufiane Hayou* Simons Institute UC Berkeley
Nikhil Ghosh
Bin Yu
Flatiron Institute
Dept of Statistics UC Berkeley
ABSTRACT
Low-Rank Adaptation is widely used finetuning method for large models. Its small memory footprint allows practitioners to adapt large models to specific tasks at fraction of the cost of full finetuning. Different modifications have been proposed to enhance its efficiency by, for example, setting the learning rate, the rank, and the initialization. Another improvement axis is adapter placement strat- egy: when using LoRA, practitioners usually pick module types to adapt with LoRA, such as Query and Key modules. Few works have studied the problem of adapter placement, with nonconclusive results: original LoRA paper suggested placing adapters in attention modules, while other works suggested placing them in the MLP modules. Through an intuitive theoretical analysis, we introduce PLoP (Precise LoRA Placement), lightweight method that allows automatic identification of module types where LoRA adapters should be placed, given a pretrained model and a finetuning task. We demonstrate that PLoP consistently outperforms, and in the worst case competes, with commonly used placement strategies through comprehensive experiments on supervised finetuning and rein- forcement learning for reasoning.
INTRODUCTION
Low-Rank Adaptation (LoRA, Hu et al. (2022)) is widely used parameter-efficient fine-tuning (PEFT) methods for large language and vision models. LoRA significantly reduces the computational and memory requirements of finetuning by freezing the pretrained model weights and inserting low-rank matrices into the model. This approach has enabled the adaptation of production-scae models on limited hardware resources while achieving performance comparable to full finetuning.
LoRA improvements. Several works have considered improving LoRA performance by e.g. using different learning rates for LoRA modules (Hayou et al., 2024a), using normalized updates (Liu et al., 2024), setting adaptive LoRA rank (Kim et al., 2024; Lu et al., 2025), improving initialization (Hayou al., 2024b), and many other variants, e.g. (Zhang al., 2023a; Dettmers et al., 2023; Kopiczko et al., 2024; Zhang et al., 2023b; Tian et al., 2024; Jiang et al., 2024). A critical aspect of LoRA is module selection deciding which specific components of the model should receive the low-rank adaptation. In practice, instead of selecting individual modules, one selects module types such as "q proj"" (Query modules), "v "_prrjj" (Value modules), etc. In Hu al. (2022), the authors suggested that inserting LoRA in attention modules (Query, Key, and Value) generally yields the best performance among other possible placements. However, in a recent note (Fomenko et al., 2024), the same authors further explained the difficulty encountered in LoRA adapter placement, and mentioned that optimal placement depends on pretrained model and the finetuning task. Another work He et al. 1)021) found that for some models, placing LoRA adapters in MLP modules gives better performance. Faced with this confusion, practitioners generally follow one of these guidelines or insert LoRA adapters in all modules which comes at higher finetuning cost. Therefore, it is natural to ask:
Given a model and a task, how can we select target module types for LoRA reasonable cost?
Work partially done Johns Hopkins University. Corresponding author: hayout juu.ddi

---

Published as a conference paper at ICLR 2026
Inputs Model & module types
Calculate NFN(W) 1Wall
Rank & Insert
[
input module W fn. random with || ||=m.]]
I|
Subset from Task D1 CD
measures module-tase task aligrment
Ireert Adaplers Low Scores (U.G in this caree)
Figure I: Mechanism of PLoP. For each module W in the model, we calculate alignment scores called NFN (Normalized Feature Norms), rank them, and pick module types with the lowest alignment scores for LoRA insertion. The NEN scores are defined by the ratio between actual activation norms [W and random baseline W 2mm I, where &in randomly generated vector with the same cuclidean norm Zin-
Memory footprint of LoRA. In practice, LoRA used finetune large models with relatively low cost. Consider Llama3.2-3B (Llama-Team, 2024), processing sequences of 2048 tokens with batch size of With full finetuning, the memory requirements are substantial. The model parameters require 12GB in float32, while the Adam optimizer states add another 24GB. The activations for a single forward pass consume approximately 48GB of memory. This brings the total memory requirement to approximately 84GB necessitating high-end GPUs. This becomes more problematic with larger, production-scale models. With LoRA, the computational cost changes dramatically. Using rank-16 16 adapters query and value modules introduces only 10 million trainable parameters (0.33% of the model). Notably, since gradients are only computed for the adapter weights, the memory overhead for gradient computation reduced by over 99%. This enables finetuning on single 24GB GPU with the same batch size and sequence length. These low memory footprint is what makes LoRA attractive for finetuning.
Anatomy of a practical module selection method for LoRA finetuning. Based on the com- putational constraints outlined above, any practical module selection method for LoRA adapter placement must operale within these resource limitations. We identify three main pillars of a practical method: (i) the method cannot require computing gradients with respect to the full model parame- ters, as this would defeat the primary purpose of using LoRA, (ii) the selection process should not necessitate multiple forward passes through different model configuratio configurations, as this would multiply the already significant activation memory requirements by the number of candidate configurations being evaluated, (iii) the method must avoid storing large intermediate computations or maintaining extensive state across different module evaluations, which would further strain memory resources. Only methods satisfying these stringent requirements can truly serve practitioners operating in the resource-constrained environments where LoRA provides its greatest value.
In this paper, we introduce PLoP (Precise LoRA Placement), a lightweight module placement method for LoRA based on a specific measure of module-data alignment that can be calculated with few forward passes (no gradients, no extensive forward passes, and no storage of intermediate caleulations), and therefore, checks all the three points above (see the compute cost paragraph in Section 3 for more details). The mechanism of PLoP is described in Fig. 1. Specifically, our contributions are as follows:
1. We develop a theoretical framework to study module-data alignment in large neural networks, the core concept behind PLoP.
2. Based on our theoretical analysis of module-data alignment, we develop PLoP, which identifies which module types should be used for LoRA finetuning.
3. We validate our results with extensive experiments showing the benefits of PLoP with LoRA in three post-training scenarios: supervised finetuning for classification, supervised finetuning for text generation, and reinforcement learning for mathematical reasoning.
The paper is structured as follows: Section 2, we introduce the main theoretical intuition behind method. In Section 3, we present method PLoP and provide quantitative and qualitative analysis of our method. In Section 4, we report empirical results showing the benefit of PLoP two post-training scenarios: supervised finetuning and reinforcement learning.
2

---

Published as a conference paper at ICLR 2026
1.1 RELATED WORK
The effectiveness of LoRA critically depends on the placement of adapter modules. Initially, Hu et al. (2022) studied the placement of adapters in attention modules, observing strong performance in various NLP tasks. He al. (2021) showed that adapters placed in MLP modules can sometimes outperform attention- based placements. Fomenko et al. (2024) mentioned that optimal adapter placement varies significantly depending on the pretrained model architecture and the downstream task. The authors recommended the following general strategy for adapter placement: start with attention layers, then embeddings, then MLP blocks, and if further capacity required, raise the LoRA rank. In machine translation, Gheini et al. (2021) found that tuning exclusively cross-attention parameters could achieve performance comparable to mill-model tuning.
More adaptive approaches include sensitivity-basse parameter selection methods. Zhang et al. (2024) proposed gradient-based scoring approach that ranks parameters according to their importance the task, tuning only the highest scoring subset. Similarly, He et al. (2023) developed sensitivity- fine-tuning technique for vision models that dynamically assigns tunable parameters to layers based on local responsiveness. However, such methods require calculating and storing gradients of the full model which is suboptimal for LoRA finetuning (see discussion above). Another variant of LoRA Zhang al. (2023a) introduces modifications to the adapter structure adaptively distribute capacity between modules. However, our focus in this paper is on module type selection for LoRA. In our experiments, compare with two baselines: Insertion in attention modules recommended by Hu et al. (2022), and Insertion in MLP modules as recommended by He et al. (2021).
Finally, our method is based module-data alignment score. Several alignment scores exist in the literature. For instance, Baratin et al. (2021) introduced the centered tangent kernel alignment as measure of how well aligned each layer is with the task, and Lou al. (2022) provided theoretical analysis of such aligmment. He et al. (2024) studied the emergence of large feature norms in the network as a result of different training configurations. Our work introduces a new alignment metric based on feature norms.
2 FEATURE NORMS CAPTURE MODULE-DAT DATA ALIGNMENT
We provide an intuitive theoretical analysis that shows how feature norms capture information about module-data alignment. Consider a general neural network of the form
(x) Wine, Yi(x) FiW, (z-1 le [L]. (x) (oot)=WootYyn (o)
where x Rd is the input, T.2 1 is the network depth, (Fi)e(z) are mappings that define the layers, Wi e _gx= are the hidden weights, where is the network width, and Win, Wout are input and output embedding weights.
Module-Data Alignment. Fix training sample æ. To understand how modules align with the the training sample x, we track how hidden features change as we train the model on the singleton {x). For the sake of simplification, we consider the case where only single module W is trained and other modules are frozen. The trainable module has the form
Zout W 2an:
where zin. e IR" is the input, and Fout R is the output that we call feature, both evaluated at the training sample x.! For Transformer models, the module can be for instance a single Query head, a Projection module in some MLP, etc. The gradient of the loss with respect to the weight matrix W is given by dW = dzout & Zins where dzout = V sont f, the gradient the loss & with respect to feature Zout, evaluated for the sample The weights are updated with Adam (Kingma and Ba, 2017), which
'erre consider that and Zout have the same dimension However, our analysis can he extended to the case where they have different dimensions.
3

---

Published as a conference paper at ICLR 2026
normalizes gradients. Considering the momentum-less version of Adam, updates are given by
Zi Wiziin - a S(dz Zin)
(2)
-Wizin - a |zin S(dzout):
where the superscript in sinut = Winin refers to update step t,', and S refers to the sign function (+] for positive, -1 for negative). In Eq. (5), we used the fact that sign function S(.) can be expanded across outer product. This is one of the main observations behind the development of μP (Yang and Hu, 2022), which sets scaling exponents for initialization and learning rate with respect to model width We follow μPP parametrization of the learning rate and set a syn for some constant 1j > 0 (see Appendix B for more details about the mechanisms of pP).. This yields
n [ 12 n I Wie 12 n in 2nmn" 22nlll x n (Wi (dzz
The term n- (W, Zin, s(dzowt) measures the alignment between the features zoot Wazin and the "signed" gradients S(dZout). When these two terms are uncorrelated (e.g. at the initial training stages), we have
(Wizin: S(dzout) O(n))),
which yields
n [We+12m I ann=2 l2nlll 1l O(n-1/2)
Since 22n=~21a, If is positive and does not vanish asymptotically (because || I1 is of size n). then the feature norm n "1|[ increase training progresses. An in-depth analysis of this phenomenon is provided in Appendix A.
The increase in feature norms indicate increased alignment be- tween W and Zin- To verify this phenomenon in a controlled set- ting, we consider a three layers linear neural network given by
3
Layer Layer Layer
f(x) = W2W1 Wox, where æ E R. Wo Raxd W1 E RR*x,, and W2 E 1x The training data consist of N 1000 datapoints of dimension d generated from a linear model y w x: E with E ~ (0, 0.025). wi d' d (0,,)), 1), and x are generated randomly as standard Gaussian random variables. We use n = d= 100 in our experiments and train the model with Adam. See Appendix D for more details and results.
100
Train Loss
0.5
Figure 2 shows the increasing alignment pattern as measured by feature norms for the three layers we train the model. We include a baseline with no alignment (dashed lines) which shows the norms [W2,, where én is random Gaussian vector with iid coordinates, normalized such that || = || ||- The baseline does not show any significant growth over the course of the training which further confirms that feature norms grow as result of increasing alignment between module weights and module inputs.
100
200
Steps
Figure 2: Evolution of feature norms during training for the linear network described in Ap- pendix A. We train the model for 300 steps with Adam, Feature norms for different layers exhibit differential growth patterns we train the model. We shifted curves corresponding to different layers for better visualization.
Different alignment levels for different modules. Although we use the same learning rate for all modules, feature norms in the second layer (n [ (?2) grow much more significantly than those in the input layer (n |Wozing I2). This indicates different alignment levels for each module. Such varying alignment patterns between layers has been discussed in Nam et al. (2024) for different alignment metric.
Different alignment levels for different inputs/tasks. In Fig. 2, we report feature norms during training for the actual training inputs [2all 2.. When evaluating feature norms | 2/, for an out-of- distribution input L', the resulting alignment depends on how similar is x' to the training samples. The extreme case where x' is very different from the training samples should result in low to
Also known as SignSGD Bernstein et al. (2018). See Appendix for more details. Note that we do not use such superscript for Zin since does not change when we update W.
4

---

Published as a conference paper at ICLR 2026
NFN-Map for Uama-3.211B-instruucc (Math)
attn.k_prgj
3:51
3.28
2.49
3.32
3.04
2:51
2.69
2.29
2.32
2.57
2.22
2.50
2.25
2.22
2.32
2:65
attn.o_pro)
1.51
5:23
1:23
1.0%
0:83
0.85
0.74
0.71
9.77
0.73
0.75
0.85
0:83
0:81
0:95
0.82
Module Type
attn. q_ardj
2:10
2:25
2.56
atth.v_prej
0:82
1.02
1.09
0.99
1.02
0.899
3:1.0
mlp.down
1:24
1.00
2.35
1.06
1.06
1.00
1.09
1.08
1:01
1:03
1.02
1.00
0.96
0.97
0.96
3.09
mip.gate
1:24
1:31
3.40
1.47
1.57
1.54
1.41
1:30
1.22
1.20
1.30
1.66
rrlp.up-
1:14
1:06
1.05
1.05
1.79
Layer Index
Figure 3: NFN-map for LLama-3.2-1 3.----BB-BBB---- I--ntrruct on Math datasct (GSM8K). See Appendix D for NFN-maps of other models.
alignment, as the random baseline [W We provide a more in-depth analysis of this behavior in Appendix B.
This analysis suggest that feature norms can be used to measure module-data alignment in L.MMs. In the next section, refine this notion of alignment and use it to create a method for module type selection for LoRA finetuning.
3 PLOP: FINETUNING MODULE TYPES WITH THE LOWEST ALIGNMENT
Given pretrained model and finetuning dataset D, we compute feature norms for all modules on the task D by averaging across a subset of D. This captures module-data aligmment as detailed the previous section. Our methodology in based on the following intuition: The modules with the lowest alignment levels have more potential for adaptation, and therefore should be prioritized finetuning.
To obtain scale-free alignment score, we need to normalize with some baseline. Specifically, as shown in Fig. 2, if we consider a random vector 5,n with the same norm as 2mn the feature norm I 2in does not exhibit any significant change throughout training, which is expected due the to lack of correlation between W and ž.n This random baseline can act as a natural normalizer for the actual feature norms, which leads to the following intuitive definition of our aligmment metric.
Definition 1 (Normalized Feature Norm (NFN)). Given a pretrained model, module with weight W in this model, and an input x, we define the Normalized Feature Norm as
W Zin ())(x))) (x)
NFN(W, x)
I im (x)
where (x: is a vector of the same dimension and norm of Zin (2), with i.i.d coordinates distributed as centered Gaussian random variables.
By incorporating the random baseline [W ], in the definition of NFN scores, expect to see scores NFN> 1 when W is well aligned with Zin, while the NFN score should be when alignment not significant. Under some assumptions W and Zin. we can prove that when the width is large enough, the NEN score can be approximated by NFN(W, x) x
where [W]== VEn Eng W3 is the Frobenius norm of W. This approximation shows that dividing by [W essentially normalizes W and zin- While both forms are cheap compute, we prefer the expression Definition as provides an intuitive interpretation of what NFN scores are representing.
Wz..(x) [
From this analysis and the intuition above, we introduce PLoP, a method that leverages NFN scores to identify which modules should be prioritized for LoRA finetuning. Our method is described below.
5

---

Published as a conference paper at ICLR 2026
Alignement by Module Type
2.0
Alignement Module type
rendem
fandom
maxs
1.
lugic
2.
hiltory
code
15
20
0.5
0.5
0.0
gire pro)
sPS)
4,500)
gate patl
down poof
up sro
(a) LJama3.2- Baama..2-BB-Iaaee 1---Irrrnstt
(b) Gemma3-1B In-Itruuctt
Allgnament by Module Type
Alignement by Module
2.5
r2nddm
math
20
8 (ogic
hatory
code
codee
15
1.0
10
0.5
0.5
0.
0.0
gat jraj
doom Jnyj
A.
pm
smj
qale and
proj
(c) 3-3-m33.-3B-
(d) Qwen3- 1.7--11BBBBa Instruct
Figure 4: NFN scores aggregated by module type for different models. The scores are for different datasets (math, code, history, and logic).
PLoP - Module Type Selection
Inputs:
Model M, Finetuning dataset 'D.
Step1 (Scores): For each model type T e [Query, Key, Value, OutProj, GatcProj. UpProj. DownProj), compute average NFN score across W T and x = 'D. Step2 (Insertion):
Insert LoRA in module types with the lowest NFN scores.
As stated above, PLoP is based on the hypothesis that modules with the lowest alignment have higher potential for adaptation, and thus should be targeted in finetuning.
For ablation analysis, we also experiment with the reverse PLoP method where instead of choosing module types with the lowest scores, we choose the ones with the highest scores. We call this method PLoP-1 and we evaluate its performance in Section 4.
Figure 3 and Fig. 4 show NFN scores for for different models and 4 tasks: math (GSM8K, Cobbe et al. (2021)), code (HumanEval, Chen et al. (2021)), history (MMLU high school european history, Hendrycks al. (2021)), and logic (MMLU logical fallacies). The NFN-map in Fig. 3 provides the most granular level of scoring and shows the NEN scores by module. We can see that key and query attention modules are most aligned with the task in this case, while MLP modules are less aligned, suggesting the need for adaptation in those modules. To see this by module type, we aggregate by averaging over all modules of the same type (step in PLoP) and show the results in the Fig. 4 for different models. We observe significant variability of NFN scores across models, module types, and datasets. For -lama3.2-113 IB, module types with the highest scores (Query, Key) average around 2-3X the baseline (sx 1), and the lowest scores (Value, Gate, Down, Up) hovering around the baseline score of 1. In this case, PLoP indicates that adaptation should be focused on the (value, gate, down, up) modules rather than the attention query and key matrices. Note that this coincides with the recommendation of empirical work by He et al. (2021) for Llama models but is contradictory to the recommendations of Hu al. (2022) to finetune mainly attention modules.
Owen3-1.7B shows high alignment in Query, Key, and Gate modules, with lower alignment for other MLP modules, and low alignment for the Value module (s 0.75). This indicates the Value modules in Qwen3- 1.7 are "negatively" aligned with all datasets, suggesting that inputs to the Value modules are aligned with the smallest singular directions of the Value weight matrices. The same pattern can

---

Published as a conference paper at ICLR 2026
Alignment by Type (GSMBK)
Alignment by Module Type for Code (HumanEval)
2ween2.-1.30
Alignmant Ratio (NFN)
Owen2.5-1..8
0wen223-Mathh11300
5-Cdeer-15
1.5
Alinmmnt Rato
1.0
1.0
0.5
0.5
proj
V_pol
purraj
gate, proj
down proi
proj
q_praj
proj
gate pnaj
down proj
Figure 5: Module types NFN scores for general and specialized Qwen2.5 models. Specialized models (math, code) are finetuned on spec-fc data. Scores are higher with the specialized models.
be observed in Gemma3-IB IB, and we currently do not have explanation for this phenomenon. In Appendix D, we provide additional NFN scores for Qwen, Gemma, and Llama models.
NFN scores are task-sensitive. The alignment scores differ between tasks. For instance, model weights show larger alignment with history compared to math, suggesting that their training data consisted more of sequences similar to general natural language than math related tokens, which is expected. However, note that all tasks share some "base" alignment level given by the general magnitude of the NFN score for each module type. This a more fundamental phenomenon that is independent of the task, and is related to some basic level of feature learning that is required for token processing.
NFN scores consistent across different model sizes. In Fig. 4 (a) and (c), we show NEN scores for two model sizes of Llama3.2, IB and 3B. The ranking of module types based on NFN scores is roughly the same for both models, suggesting consistency of NFN scores across different model sizes. Intuitively, having similar NFN score patterns suggests similar pretraining and post-training processes for these models, which is expected for models of the same family (Llama3.2 in this case).
Specialized models show higher NFN scores. In Fig. 5, we compare NFN scores for instruction- tuned and more specialized version of the same model Qwen2.5-1. for math/code tasks. As expected, the specialized models show higher NFN scores overall which further confirms that NFN scores, while cheap to calculate, can be reliable metric for module-data alignment. The increased NFN score confirm our theoretical analysis in Section and shows that training on domain-specific data increases module-data alignment, captured by the NFN scores.
Compute cost of PLoP. To obtain the results in Fig. 3, used a single forward pass with batch size 200, with a maximum sequence length of 256. The cost of PLoP is approximately the same as performing standard batched prefill over 200 inputs, which can be merged with the first step of LoRA finetuning, i.e. roughly zero additional cost. Note that PLoP does not re- quire gradient computation any stage, making it substantially cheaper than any maadient-based selection strategy.
1.0
Kendall r
0.6
Liama 3.2 Owen-3 7.78 7B
0.4
2
2
2
2
2
2
2 2
Selection batch size
Sensitivity of PLoP to selection subset size. As mentioned above, the experiments in Fig. 3, Fig. and Fig. 5 were con- ducted with forward propagation batch size of m 200. To understand the sensitivity of PLoP the choice of m.
Figure 6:
&eesitivity analysis of PLoP in GSMSK dataset.
run the following experiment: for varying batch size m E {2* 12* 3....... 1,),, we divide GSM8K train set (7k+ samples) into pairs of batches of size m (the pair has 2m samples in total) and compute the Kendall r correlation coefficient between the module types selected by PLoPfor each pair. We function of
do this for (dataset size/2m) pairs, and show the distribution of Kendall r in Fig.
*The mechanisms of feature learning in deep neural networks are still largely misunderstood. Quantitative approaches such Nam al. (2024) offer some insights, but far from heing comprehensive.
7

---

Published as a conference paper at ICLR 2026
m. Across different model sizes, we observe that correlation is close to for batch size 2 64, indicating that PLoP is also sample efficient. We conduct a similar study for sequence length, which report in Appendix D.6.
4 EXPERIMENTS
We consider three post-training scenarios: Supervised Finetuning for sentence classification, Super- vised Finetuning for text generation, and Reinforcement Learning (GRPO, Shao al. (2024)), all with LoRA adapters. We report results with Llama (Llama-Team, 2024), Qwen (Qwen Team, 2025), and Gemma (Team, 2025) models across different sizes. Our experiments are as follows:
1. SFT for classification: we finetune classifers on ANLI (Nie al., 2020).
2. SFT for text generation: we consider three experiments: 1) Train on MetaMathQA (Yu et al., 2023) and evaluate the results on GSM8K (Cobbe et al., 2021), 2) Finetune on language datasets from AYA (Singh et al., 2024) and evaluate train/test NLL,, and 3) Finetune
CommonsenseQA (Talmor et al., 2019) and evaluate accuracy.
3. RL: we RL-tune on MetaMathQA using GRPO and evaluate on GSM8K.
We investigate the effect of different module placement strategies: our method PLoP(placing LoRA in module types with the lowest NFN scores), Attn (inserting LoRA only in attention modules, as suggested in Hu et al. (2022)), MLP (inserting LoRA only in MLP modules, as suggested He al. (2021)), ALL (inserting LoRA in all module types). Random (inserting LoRA in randomly picked module types and average over runs), and as a sanity- check PLoP-1 (the inverse of our method, i.e. placing LoRA modules types with the highest NFN scores) demonstrate that PLoP is capturing meaningful signal.
Qwen2.5-0.38
0.550
1000
2005
3000
4200
5000
Slep
LLama-3.22 3-ma--322-1
Hereafter, we use the following letters to denote specific mod- ules: Q (Query), K (Key), V (Value), O (Out projection), U (Up projection), G (Gate projection), and D (Down projection). All module type NFN scores and experimental details are provided in Appendix D.
0.52
0.60
0.58
MLP
1200
2000
3002
4200
5:000
Sh.pp
Owen3 Embediing 48
4.1 SUPERVISED FINETUNING FOR CLASSIFICATION
Aac
0.6
The Adversarial Natural Language Inference (ANLI) is lan- guage classification task that is more challenging compared to similar tasks (e.g. MNLI). Using LoRA with different placement strategies, we finetune pretrained models on ANLI and report results in Fig. For Qwen2.5-0.55 and Qwend-Embeddingg- 8B, we observe a significant difference in performance between PLoP and other strategies. For Llama3.2- 2-B PLoP and MLP yield roughly the same performance, while Attn is significantly worse. Note that for the Llama model, the MLP modules have small NFN scores, comparable to scores of modules selected by PLoP(V-O-D, see Fig. 4), which could explain why we obtain similar performance with both methods.
0.73
530
NLP
1203
2000
3002
4000
5:00
Step
Figure 7: LoRA finctuning on ANLI for different models. We use LoRA rank = 8 for MLP strategy and adapt for PLoP and Attn to match number of parameters for fair com- parison. curves are smoothened with EMA(a = 0.8) for better visual- ization. See Appendix D.3 for more details about the experimental setup.
4.2 SUPERVISED FINETUNING FOR TEXT GENERATION
SFT on MetaMathQA. For challenging tasks such as mathematical reasoning, SFT is often used to "prime" the model for subsequent RL-based finetuning. We perform SFT on the MetaMathQA dataset Yu et al. (2023). To evaluate the performance we measure the accuracy on the GSM8k benchmark Cobbe et al. (2021). All experimental details are provided in Appendix D.3.
We finetune Qwen3 0.6B and 1.7B models. The results are shown in Tables and 2 respectively. In both cases, the placement of adapters only the attention layers clearly suboptimal, demonstrating
8

---

Published as a conference paper at ICLR 2026
Table 1: Qwen3-0.6B
Table 2: Qwen3-1.7B
Module Types
#Params [M)
Train
Eval Ace
Module Types
#Params (M)
Train
Eval Ace
FT
48.1%
No
64.7%
PLoP(D-U-VV) 76)
21.8
0.118
63.8%
(DoPVD---V) 102)
43.9
0.109
75.4%
(LoP(D-U-V)CC 64)
18.4
0.116
62.0%
PLoP(D-O 0-V) (r= 64)
27.5
0.111
75.2%
PLoP (r= 64)
16.5
0.119
60.6%
PLoP PLoP- (G-K-Q) 64)
27.5
0.113
74.6%
MLPID-G-U) 61)
63.3%
MLPID-G-U)
44.0
0.108
75.0%
Atin(K-Q Qr 64)
12.8
0.130
58.6%
Atin[K Q-V) 64)
18.4
0.119
69.5%
(r 64)
40.4
0.113
62.4%
all 64)
69.7
0.105
73.9%
Random (r 61)
0.117
59.1%
Random (r 61)
0.112
70.1%
Table 3: Train/Test NLL languages from AYA
that for challenging tasks such as mathematics, adapting only the attention layers has limited ef- fect. We see that the most competitive placements for both model sizes is to place adapters according to PLoP or in the MLP layers, even outperform- ing the placement of adapters in all layers which requires between 1.5-1.8x the number of trainable parameters. When adjusting for an equal number of trainable parameters, PLoP produces the best results with a slight edge over the MLP placement. For the 1.7B parameter model PLoP has small edge over the MLP placement even with about 60% the number of parameters.
dataset. All methods have comparable trainable param- eter count which fixed by setting r for the MLP method.
Porluguese
Japanese
Arabic
Train
Test
Train
Test
Train
Text
No FT
7.74
5.06
6.82
11 3222 L.amma
Random
0.475
0.491
1.81
1.95
2.13
2.33
MLP
0.447
0.479
1.56
1.70
0.991
1.05
PLoP
0.512
0.612
1.73
1.91
1.79
2.33
No FT
7.26
5.93
8.72
Random
0.356
0.421
1.44
1.50
1.59
SFT on Language datasets. We finetune Llama-3.2 IB and 3B models on languages datasets from the AYA dataset (Singh et al., 2024) with different adapter placement strategies, while matching trainable parameters count. All experi- mental details are deferred to Appendix D.3. The
Llana-
Alln
0.383
0.451
1.57
1.69
1.98
2.05
MLP
0.292
0.418
1.25
1,50
0.917
0.955
PLoP
PLoP
0.241
0.381
1.20
1.41
0.791
0.843
results reported in Table 3. On language datasets, PLoP(V-O-D) consistently outperforms other methods, with the second best ranking method consistently MLP. (Q-K-)) shows signifi- cant gap in performance, which suggests that picking the modules with the highest NFN scores is suboptimal, and that PLoP picking meaningful signal. The Attn baseline and Random baseline (average over 4 random selections) rank lower than MIP, confirming previous empirical findings by He et al. (2021), and significantly lower than PLoP.
Table Train NLL/Test Acc on CommonsenseQA. All methods have comparable trainable parame- ter count which is fixed by setting for the MLP baseline.
SFT on QA We finetune Qwen-2.5 2.5 7B and 32B models on CommonsenseQA (Talmor et al., 2019), dataset con- sisting common sense question answering, and evaluate the train NLL and test accuracy. Similar to SFT on language datasets, we consider different adapter placement strategies and match the trainable parameter count for fair comparison. All experimental details are deferred to Appendix D.3. The results are reported in Table 4. The results show that PLoP, which selects V-O-U mod- ules for both model sizes, consistently outperforms other strategies. PLoP-1, which selects Q-K-G modules for both module sizes, un- derperforms other methods indicating that high NEN scores are associated with lower performance.
Train NLL
Test Ace
No FT
82.8%
Qwen-25 7B5
Random
0.258
86.2%
Alin
0.263
86.0%
MLP
0.247
87.3%
PLoP
PLoP
0.235
88.6%
No FT
88.2%
Qwen-2.5 3215
4.3 RLVR FOR MATHEMATICAL REASONING
Atnn
0.081
91.5%
Reinforcement Learning has emerged as promising approach for test-time scaling. Algorithms such as GRPO incentivize the model to follow pattern of "thinking" before providing the final answer. This implicit approach to reasoning (versus the more explicit approaches such MCTs (Xie et al., 2024)) showed very
MI.P
0.075
91.9%
PLoP
0.080
91.0%
PLoP
0.072
92.6%

---

Published as a conference paper at ICLR 2026
promising results, especially with the impressive performance of DeepSeek-RI (DeepSeek-Al, 2025). this section, we experimented with "GRPO on a budget" using LoRA adapters (instead of training the full weights) to enhance mathematical reasoning. We select 3 module types for LoRA adapter placement using the strategies stated above. We compare both rewards (columns Rwd/Format and Rwd/Answer) and Evals (GSM8K Sshots prompting Pass @ 1). Note that because LoRA is a lightweight finetuning method, it would not be sufficient to induce reasoning in base models with GRPO, especially with small rank r. For this purpose, apply GRPO with LoRA to instruction-tuned models instead of base models. For more implementation details, see Appendix
Table 5: GRPO results for Qwen3-1.7B trained on MetaMathQA.
Module Types
#Params (M)
Rwd/Format
Rwd/Answer
Eval/GSM8K
No GRPO
65.50%
Attn(Q-K-V) 16)
4.58
1.89
0.91
71.49%
Attn(Q-K-V) (r 25)
7.17
1.97
0,98
72.13%
MLP(U-G-D) (r= 16)
11.01
2.57
1.28
73.61%
(QoP-(Q--KGG) (r= 16)
6.88
1.71
0.86
71.41%
PLoP(V-O-D) 16)
6.88
2.67
1.32
74.52%
PLoP(V-O-D) (r =2))
10.75
2.75
1.32
75.03%
With GRPO, we define the think-then thhe--nnwwer pattern in similar way to DeepSeek-RI. The model is rewarded for placing thinking process in between <think> and </think>, and then giving This is encoded in the format reward
the answer in between <so asolutt utiion> on
and </ "solution function (R (wwd/Fomma).. The correctness of the solution rewarded as well (Rwd/Answer), We track this reward as we train the model with GRPO and show the final results (at convergence).
Table shows the results of GRPO with the Qwen3-1. 1.7B trained on MetaMathQA. PLoP yields better performance overall both in training rewards and evaluation (GSM8K), Compared to Attn, PLoP performs better even when matching the number of trainable parameters (Att((r = 25) vs PLoP(r 16)). Interestingly. PLoP performs better than MLP placement strategy when using the same rank r = 16, in which case we have 6.88M trainable parameters with PLoP and 1 11.01M parameters with MIP. For the number of parameters, the performance is further improved with PLoP(r= = 25).
We experimented with Gemma3-IB as well, and found that PLoP outperforms other alternatives, although the score on GSM8K is low due the inherent limitations of Gemma3-1B in mathematical reasoning. See Appendix D for more details.
DISCUSSION AND LIMITATIONS
We introduced PLoP, intuitive module type selection method, designed specifically for LoRA fine-tuning and based on NFN scores notion of module-date alignment supported by an intuitive theoretical analysis. PLoP meets the computational criteria needed for efficient LoRA finetuning as articulated in the introduction: PLoP's lightweight nature makes it particularly valuable in resource- constrained environments where LoRA most beneficial. PLoP based on the NFN-map, which enables more granular selection beyond module types. However, deliberately focused module type selection as it represents the most widely adopted aggregation approach among practitioners, avoiding the additional implementation complexities of more fine-grained selection. While explored using PLoP for laver- selection by inserting LoRA into target layers with low NFN scores, encountered inconsistent results and have reserved this question for future research.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENT
This work used GPU servers at Delta AI (NCSA) through allocation #CIS250193 from the Advanced Cyberinfrastruuture Coordination Ecosystem: Services Support (ACCESS) program (Boemne et al., 2023), which is supported by U.S. National Science Foundation grants #2138259, #2138286, #2138307, #2137603, and #2138296.
REFERENCES
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, Weizhu Chen, al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
Soufiane Hayou, Nikhil Ghosh, and Bin Yu. LoRA+: Efficient low rank adaptation of large models. In Forty-first International Conference on Machine Learning, 2024a. URL https: // //openrevi ew. net/ forum? d=NEv8YQBROS
Shih-Yang Liu, Chien-Yr Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang- Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation. In ICMI,, 2024. URL https: wpw.revieweee.eeeee/ep net.
Minsoo Kim, Sihwa Lee, Wonyong Sung, and Jungwook Choi. RA-LoRA: Rank-adaptive parameter- efficient fine-tuning for accurate 2-bit quantized large language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguistics: ACL 2024, pages 15773-15786. Bangkok, Thailand, August 2024. Association for Computational Linguisties. doi: 10.18653/v 1/2024. 1/tp:///////2222..nnnnn-------e 2024. wwnniiinnndanaeeeeme acl. .933/.
URL https: /tt///alaaaae /aaaaaaaannooooooooo org/
Haodong Lu, Chongyang Zhao, Jason Xue, Lina Yao, Kristen Moore, and Dong Gong, Adaptive rank, reduced forgetting: Knowledge retention in continual learning vision-language models with dynamic rank-selective lora, 2025. URL https: www.aaaivvvv....aaaaaaaaaaaaaaaa abs 0tbp:/442.00000000
Soufiane Hayou, Nikhil Ghosh, and Bin Yu. The impact of initialization on lora fine- tuning dynamics. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, Tomezak, and C. Zhang, editors. Advances Neural Information Processing Sys- tems, volume 37, pages 117015-117040. Curran Associates. Inc., 2024b. URL
iigss.iiinnn.....e www.inns....nnneeeeeeeeeeeeeeeee ps. cc httpeerrrillleeppp///22///
Qingru Zhang, Minshuo Chen, Alexander Bukharin, Nikos Karampatziakis, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adalora: Adaptive budget allocation for parameter-efficient fine- tuning, 2023a. URL https : /al//aaiiii aa//arxivvvrrg. abs 2303. 10512.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized 1lms. Advances neural information processing systems, 36:100688-001111 2023.
Dawid J. Kopiczko, Tijmen Blankevoort, and Yuki M. Asano. Vera: Vector-based random matrix adaptation, 2024. URL https: //aaviv.///ii.aa/aa///a org/ http:/bb/233....
Longteng Zhang, Lin Zhang, Shaohuai Shi, Xiaowen Chu, and Bo Li. Lora-fa: Memory-efficient low-rank adaptation for large language models fine-tuning, 2023b. URL https://arxiv org/abs/ 2308 03303.
Chunlin Tian, Zhan Shi, Zhijiang Guo, Li Li, and Chengzhong Xu. Hydralora: An asymmetric lora architecture for efficient fine-tuning, 2024. URL http::///wwwx.../a/////////oo 192454
Ting Jiang, Shaohan Huang, Shengyue Luo, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang Deqing Wang, and Fuzhen Zhuang. Mora: High-rank updating for parameter-efficient fine-tuning, 2024. URL. https: arxiv.c org/ abs /2405 12130.
Vlad Fomenko, Han Yu, Jongho Lee, Stanley Hsieh, and Weizhu Chen. A note on lora. arXiv preprint arXiv:2404. 05086, 2024.
11

---

Published as a conference paper at ICLR 2026
Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Sorg-Kirkpatrick, and Graham Neubig. Towards unified view of parameter-efficient transfer learning. arXiv preprint arXiv:2110.04566,,20000 2021.
Jlama-Team. The Ilama 3 herd of models, 2024. URL htt.sss/aaravv..orraabb///// 21783.
Mozhdeh Gheini, Xiang Ren, and Jonathan May. Cross-attention is all you need: Adapting pretrained transformers for machine translation. arXiv preprint arXiv:2104.08771, 2021.
Zhi Zhang, Qizhe Zhang, Zijun Gao, Renrui Zhang, Ekaterina Shutova, Shiji Zhou, and Shanghang Zhang. Gradient-based parameter selection for efficient fine-tuning. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 98566-28577 2024.
Haoyu He, Jianfei Cai, Jing Zhang, Dacheng Tao, and Bohan Zhuang. Sensitivity-wware visual parameter-efficient fine-tuning. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 11825-10 11835, 2023.
Aristide Baratin, Thomas George, César Laurent, R Devon Hjelm, Guillaume Lajoie, Pascal Vincent, and Simon Lacoste-Julien. Implicit regularization neural feature alignment, 2021. URL httgg/bbbs.. abs /tap///000.. 00938.
https: http///axiv..// org/
Yizhang Lou, Chris E Mingard, and Soufiane Hayou. Feature learning and signal propagation deep neural networks. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 14248-14282 PMLR, 17-23 Jul 2022. URL https: /pppoceedinggss mll. www.m.l.........mmmmmmmmmmmmm.. v 62 / 1ou22a...... html.
Bobby He, Lorenzo Noci, Daniele Paliotta, Imanol Schlag, and Thomas Hofmann. Understanding and minimising outlier features in neural network training, 2024. URL https: //tpxxxxx...r////// 2oo52405.20 11.5..9777 19279.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017. URL https://arxiv org/ org/abs//////oooooooooooo//s.o.r /11//.111112 6980.
Jeremy Bernstein, Yu-Xiang Wang. Kamyar Azizzadenesheli, and Anima Anandkumar. signsgd: org/abs/
Compressed optimisation for non-convex problems, 2018. URL 1802.04434
Greg Yang and Edward J. Hu. Feature learning in infinite-width neural networks, 2022. URL
http://...rr//a///11..1./...///. /bbs/201..14522
Yoonsoo Nam, Chris Mingard, Scok Hyeong Lee, Soufiane Hayou, and Ard Louis. Visualising feature learning in deep neural networks by diagonalizing the forward feature map, 2024. URL
https: //tpivaaaavvv///a////a//aaaaawaa orgg alss 22100 04264.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Hcewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint 10Xiv:2110.141633 2021.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint 1xXiv:2107.03374., 2021.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, 2021. URL ht htRs: //arpiii///xvv.../ org/abs/
2009.000 2009 03300.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li. Y. Wu, and Daya Guo. Deepseckmath: Pushing the limits of mathematical reasoning in open language models, 2024, URL https: : //tpxxi..voo.//// org abs/ 2402. 03300.
12

---

Published as a conference paper at ICLR 2026
Qwen Team. Qwen3 technical report, April 2025. URL https://gtttiiii httpt/hhhhhccoooooootooeoteoot wttpl//cco////////. Lww.IM/
Owen3/blebs mtt//maiin///////// mttmm/maann/nnmmmmme//m/mmm htt////eennneeee Techhicchhhccccceeeeee hetpal_eeorrt....e Released April 29, 2025.
Gemma Team. Gemma 3 technical report, 2025. URL
19786.
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. Ad- versarial nli: A new benchmark for natural language understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguisties, 2020.
Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint arXiv:2309. /2244, 2023.
Shivalika Singh, Freddie Vargus, Daniel Dsouza, Börje F. Karlsson, Abinaya Mahendiran, Wei- Yin Ko, Herumb Shandilya, Jay Patel, Deividas Mataciunas, Laura OMahony, Mike Zhang, Ramith Hettiarachchi, Joseph Wilson, Marina Machado, Luisa Souza Moura, Dominik Krzeminski, Hakimeh Fadaei, Trem Ergün, Ifeoma Okoh, Aisha Alaagib, Oshan Mudannayake, 7aid Alyafeai, Vu Minh Chien, Sebastian Ruder, Surya Guthikonda, Emad A. Alghamdi, Sebastian Gehrmann, Niklas Muennighoff, Max Bartolo, Julia Kreutzer, Ahmet Üstün, Marzich Fadace, and Sara Hooker. Aya dataset: An open-access collection for multilingual instruction tuning, 2024. URL https://arxxe httpss:/aarxi...rr//bbs/// 2402. 06619.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsensequ: A question answering challenge targeting commonsense knowledge, 2019. URL https:ts////axii.. orrg
abs/18 /1ss/1111 00937.
Yuxi Xie, Anirudh Goyal, Wenyue Zheng, Min- Yen Kan, Timothy P. Lillicrap, Kenji Kawaguchi, and Michael Shieh. Monte carlo tree search boosts reasoning via iterative preference learning, 2024. URL httts:://tpaiii org/ abs /2ts//2455 00451.
DeepSeek-AL, Deepscek-rl; Incentivizing reasoning capability in Ilms via reinforcement learning. 2025. URL https://aaxae org/ abs 12948.
Timothy J. Boerner, Stephen Deems, Thomas R. Furlani, Shelley L. Knuth, and John Towns. Access: Advancing innovation: Nsf's advanced cyberinfrastructure coordination ecosystem: Services & support. In Practice and Experience in Advanced Research Computing 2023: Computing for the Common Good, PEARC '23, page 173-176, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9781450399852. doi: 10.1145/3569551.559777 https: /t/.oiooooooo http//ooo.oo///////// 111..1115/// 01156695513599777
URL
Jordan Hoffmmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. Training compute-optimal large language models, 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification, 2015. URL https: http:://axxv...oo.e.. abs/ 1502. 0100111 1885..
Soufiane Hayou, Amnud Doucet, and Judith Rousseau. On the impact of the activation function on deep neural networks training. Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 2672-2680. PMLR, 09-15 Jun 2019. URL https: //proceedings ml press / v97 hwyyuul 9a html.
G. Yang. Scaling limits of wide neural networks with weight sharing: Gaussian process behavior, gradient independence, and neural tangent kernel derivation. arXiv preprint arXiv: 2019.
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. triviaqa: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprchension. arXiv e-prints, art. MxXiv:1055.03. 2017.
13

---

Published as a conference paper at ICLR 2026
A THEORETICAL ANALYSIS OF MADULE-DATA ALIGNMENT
Given a pretrained model and a finetuning task, our goal is to strategically place LoRA adapters in modules that would contribute most significantly to performance. In practice, we usually select module types instead of single weight matrices. For instance, for Llama3 models, we might choose insert LoRA in Query ("q_proj") and Key ("k_proj"" modules.
As we discussed in Section 1, for such a method to be useful, it should first be a lightweight method that operates efficiently in resource-constraiieed environments and ideally rely existing computation pipelines, Ideally, the method should rely exclusively on standard forward propagation, as this computational pipeline is already necessary for inference and adds no significant overhead to the existing workflow.
This section provide a more in-depth theoretical analysis, complementing that of Section
Mechanisms behind the growth in feature norms. The reason
Feature Norm
behind the growth in feature norms for certain modules non-trivial. The naive explanation to this phenomenon is that with training, weight norms grow for some modules and remain constant constant or decrease for others. However, we will see in the next analysis, the mechanisms behind this phenomenon are more subtle, and the most important factor is form of alignment that occurs between module weight and its input.
1.0
Norm
Festure Norm
0.5
Train Loss
0.5
Specifically, show that this growth in feature norms in some modules appears primarily as a result of two factors: 1) large-width in neural networks (large embedding dimension), a condition that is generally satisfied in practice, and 2) progressive alignment of modules weights with their respective inputs.
0.0
0
100
200
300
Steps
Figure Illustration of fea- ture norm growth during train- ing. This shows the feature norms n IW Znn l for a module W in the model (W R R=**), ). See Ap- (4) pendix A. for details about the model and training.
Consider general neural network of the form
(x) = W (z(x) FiW, Yi- (x)), le [L] Your (x) W (z).
where e Rd is the input, L 1 is the network depth, (Fi) are mappings that define the layers, W, e Rax= are the hidden weights, where is the network width, and Win, Want are input and output embedding weights.
Model (4) is pretrained on some data mixture D to minimize some loss function & the next-token prediction loss in the case of language models. We introduce some notation that will facilitate the presentation of our analysis.
Notation. Hereafter, n will always denote model width. As n grows, given sequences c E RR and d, Rt, we write e, O(d,) to refer to e, kd,, for some constant O. We write o,, = (dd,)) if we have kidn S C, S k2d, for some K1, 0. For vector sequences c. (c²) Isisk e * (for some > 0), we write c, = (d,,) when ci, = (d',) ) for all e [k:], and same holds for other asymptotic notation. Finally, when the sequence Cn is vector of random variables, asymptotics are defined in the sense of the second moment (L2 norm). For a vector 2 E R", we will use the following norms:
|||| (E/+=?) ?? (euclidean norm), and |z||+ |
Σ | (E, norm).
Intuitive theoretical analysis. For the sake of tractability, consider the where single weight matrix (module) in the model is trained and other modules are frozen. We further simplify
FFom the literature on infinite-width theory, when we take the width to infinity, the training dynamics converge with rate of roughly On (Yang and Hu, 2022). In practice, width of ž 10 is generally considered large enough for the theoretical predictions be good approximation of practice.
"While this is unrealistic, it provides the right intuition behind our methodology, and makes the analysis tractable.
14

---

Published as a conference paper at ICLR 2026
the analysis by assuming that the model is trained in single datapoint We later discuss the impact of batch training. The trainable module has the form
Zout
W Zin:
where Zin e IR" the input, and Sout R is the output that we call feature, both evaluated at the training datapoint x. For Transformer models, the module can be for instance a single Query head, Projection module in some MLP, etc.
The gradient of the loss with respect the weight matrix W is given by
dW = dZout @ Zin:
where dzout Suuce e, the gradient of the loss with respect feature 2out-
In general, modern LLMs are trained with Adam (Kingma and Ba, 2017), which normalizes gradients.
In momentum- form, Adam becomes SignSGD (Bernstein et al., 2018), which is defined by S(dW;) = +1 if cW,; v O, otherwise -1. SignSGD is nice simplification of Adam: it captures the property of normalization and allows tractable the theoretical analysis as we will see below. With SignSGD, feature updates are given by
W Fin Wizin S(dZout Zin Zin)
(5)
Wizin a Izin S(dz'ut):
where the superscript in 2/t Wi refers to update step t. Note that we do not use such superscript for Zin since it does not change when we update W.
The key trick used in Eq. (5) that the sign function S(.) can expanded across outer product. This is one of the main observations behind the development of μP (Yang and Hu, 2022), which sets scaling exponents for initialization and learning rate with respect to model width n. Under jaP, all weights in the model are initialized to have roughly 1/VTi magnitude (or more precisely J/Vfan_ina in), which implies that features Zout and their inputs 2in to have (I,()) norm at initialization (i.e. (1)).
n =1 1 11
Eq. (5) describes the evolution of features Zout as we update weights W. Ideally, we want both
stability (W, does not grow in magnitude with n) and non-triviality (W,zin does not converge 0 with n). These conditions are both satisfied when W,+1 Zin W. = O₂ (1) element-wise, which constant ry O. to compensate the
implies that the learning rate should scale growth in am 1. which is exactly the μP scaling rule for the learning rate. See Appendix B below for more details about the mechanisms of pP. With this parametrization of the learning rate, we obtain
ryn-1 for
W Zin 12
[W ' || 21pn 11 ||==||]
(Wizin; S(dz
We can normalize by n so terms on both sides have (1)(1) magnitude width n,
n- I n We 1
2mm 2inll 1 x n 2Wzzin, S (dzz
The term n (W S.2innSSde S(dz measures the alignment between the features 2 Wi and the "signed" gradients S(dzout). Intuitively, the initial training stages, these two terms roughly independent (as random variables) because of the randomness from the initialization weights. As result, in those initial training stages, we have
Zin S(dzout)) (n)
(6)
which yields
n I n
| / a 2 On Onn 1/2)
7 "Here we consider that Zim Zont have the same dimension n. However, our analysis can he extended to the case where they have different dimensions. Feeturee update the change of the features Sout after taking one training step.
15

---

Published as a conference paper at ICLR 2026
is positive and asymptotically non-zero, if the width is large enough, we should expect the (normalized) feature norm n n W to grow initially during training. The next results provides rigorous description of this phenomenon for linear networks.
Since 2
|in 2in =
Theorem 1 (Feature Norm Growth in Linear Networks (Informal)). Assume that the neural network is a linear MLP (see Appendix B for more details). Then, for any (0. (02) under some assumptions stated in Appendix C, there exists universal constant X 0 such that for any T and vn satisfying T Aa the following holds with probability at least 1 2n 1+28
sup 1stsT
n [Wazin]
rol f C
In other words, when the width n is large enough, [We exhibits quasi-quadratic growth at initial training stages.
where r,= + (L 3 - nn | and I' n I[ 12
Theorem characterizes the growth in feature norms n- IW 1 as training progresses. The proof is provided in Appendix C. In this case, n 1 grows in quasi-quadratic pattern, which becomes perfectly quadratic when n + oo. This is the most important takeway from this result: this phenomenon is associated with large width. With more realistic models, expect the growth property to hold, but not necessarily with the quadratic form. See next section for empirical results.
A.1 EVOLUTION OF FEATURE NORMS
Consider three layers linear neural network given by f(x) W,W, Woar, where x Red Wo Raxd W1 Rx* and W₂ Risn The training data consist of N = 1000 datapoints of dimension d generated from a linear model y = w _x + E with E (O,00.022) wi ~ d dO, 1), and & are generated randomly as standard Gaus- sian random variables. We use n and train the model with Adam. See Appendix D for results with SignSGD and more details about the experimental setup.
n- 1p02.1
Layer Layer Layer
100 in our experiments
Train Loss
6.5
Figure 9 shows the growth in feature norms for the three modules (corresponding to the three layers in this case) as we train the model. We include a baseline (dashed lines) which shows the norms W 2 where iin is a random Gaussian vector with iid coordinates, nor- malized such that | = ainlll (see next section for intuitive explanation of this baseline). The baseline does not show any signif- icant growth over the course of the training which further confirms that feature norms grow as a result of increasing alignment between module weights and module inputs, and not simply as a result of increase in weight norms.
Steps
Figure 9: Evolution of feature norms during training for the linear network described in Ap- pendix A. We train the model for 300 steps with Adam. Feature norms for different layers exhibit differential growth patterns as we train the model. We shifted the curves corresponding to diffcrent layers for better visualization.
Most of the growth occurs early in training. Interestingly. most of the growth in feature norms occurs in the first T 200 steps, which also correlates with the most significant drop in training loss. After T = 200, feature norms remain roughly stable until conver- gence. This suggest that the norm growth is associated with an initial phase where significant feature learning occurs, and remains roughly
unchanged after that initial growth phase. Intuitively, as we train the network, the dot product between Rout and S (dzout) (Eq, (6) grows from (n22/2) to roughly O (n) (in absolute value) and therefore the argument behind the feature norm growth explained in the discussion above no longer holds later in training. As a result, the growth plateaus after some number of steps T.
Different growth levels for different modules. Although modules, the norm growth in the input layer (n (n-1lWozin )|2) is much less significant than that observed in the second layer (n ([W (12) To understand this difference, we should take into account that when training all modules (layers in this case), the inputs to W1 and W2 change with training. The update in feature are given by
the same learning rate for all
W W412 Weeh. -nn- x S Wes 1 A++1
16

---

Published as a conference paper at ICLR 2026
where Azht = 2in -/-1-2ff, is the input change after one step. Under pPP scaling rule for the learning rate (nn )) the magnitude of 2 remains e, (1) for all t. however the constant in 6 (1) naturally depends on the layer. Additionally, the term W Axft introduces more complex update dynamics, and contribute in a non-trivial way to the change in the feature norms. See Nam et al. (2024) for a more detailed discussion on how feature learning changes from one layer another. Both of these aspects lead to uneven growth in the feature norms for different layers.
Different growth levels for different inputs/tasks. In the setup of Theorem 1, we consid- ered a batch size of 1, which results in feature updates of the form W+1 Pin Wi 7 x n | I S(dZoue), and we saw that with 1//n initialization, we have n n- ||zin]] (1). In the realistic setting of batch training, feature updates for an input x' é R are given by
( Σ DEB
1 [B]
1 = Wezen (x') nnn S
de (2r) @ zin((t) zin (x');
Therefore, we can no longer directly expand the sign function and obtain the >salla term that leads to the O (1) term. In this case, we need a strong correlation between S and Zan (x') to obtain the same effect. This translates to whether the datapoint has some similarity with the batch used for the update. As a result, we should expect to see higher scores for datapoints that are similar to the training dataset, and lower scores for significantly different datapoints.
[ Σxs dzout()) @ Zin (x))
B ADDITIONAL THEORETICAL DETAILS
B. INFINITE- ANALYSIS AND pP
Scaling remains the main paradigm to improve performance of language model (see e.g. Hoffmann et al. (2022)). This includes model capacity which can be increased via width (embedding dimension) or depth (number of layers) or both. training data, number of training steps etc. In our theoretical analysis in Section 2, we mentioned the infinite-width n + oo and how our results hold in this limit. This is motivated by the fact that most state-of-the-are art language and vision models have large width.
As the width grows, most hyperparameters in the model such as the initialization and the learning should be adapted avoid numerical instabilities and ensure efficient learning. For instance, the initialization variance should scale as 1/n to prevent arbitrarily large pre-activations as we increase model width n (e.g. He init He et al. (2015)). To derive such scaling rules, principled approach consist of analyzing statistical properties of key quantities in the model (e.g. pre-activations) as n grows and then adjust the initialization, the learning rate, and the architecture itself to achieve desirable properties in the limit do Hayou et al. (2019); Yang (2019).
In this context, Yang and Hu (2022) introduces the Maximal Update Parameterization (or μP), a set of scaling rules for the initialization scheme, the learning rate, and the network architecture that ensure stability and maximal feature learning in the infinite width limit. Stability is defined by Yi = 0(1) for all and i where the asymptotic notation 'O(.)' is with respect width n (see next paragraph for a formal definition), and feature learning is defined by AY, = O(1) where A refers to the feature update after taking a gradient step. jPP guarantees that these two conditions are satisfied at any training step f. Roughly speaking, μP specifies that hidden weights should be initialized with ((n21/2 random weights, and weight updates should be of order 1n= Input weights should be initialized (11) and the weights update should be O(1) as well. While the output weights should be initialized (n=)) 2) and updated with (n These rules ensure both stability and feature learning in the infinite-widie width limit, in contrast standard parameterization (exploding features the learning rate is well tuned), and kernel parameterizations (e.g. Neural Tangent Kernel parameterization where AY O(n i.e. no feature learning in the limit).
C PROOF OF THEOREM
In this section, we provide the full proof for Theorem 1. Forst, we prove a result on the sign of the derivative of the loss function with respect to =out, then proceed with the full proof.
17

---

Published as a conference paper at ICLR 2026
C.1 CONSTANT LOSS DERIVATIVE SIGN IN THE INITIAL TRAINING STAGE
Consider linear network of the form
f(x) = W Wox, I IR!
(8)
where L 1 is the network depth, Wi E IR for all é {1. 2.2.... L-1} are hidden layers parameters, W e Rixm is the projection layer weight, and Wo IRaxd is the input layer weight.
The network trained with the setup described in Section 2, namely:
Dataset: single datapoint (2. R R.
Training algorithm: SignSGD with learning rate mm-
only single hidden layer that denote W é trained, and other layers weights are fixed to their values at initialization. Without loss of generality, assume that the trainable layer is lg. i.e. W Wio
Single layer:
is
Training loss: quadratic loss given by L(W) 2- (fw 2)
In this setting, the linear network training dynamics become tractable and expressions in steps and width n. Let us use the same notation as in Section 2 and denote the input and output of the trainable layer Sout W Zin More precisely, in this case, we can express the network output as fw (I) = VT Zout V W Zin. where Zin Mæ, with M Wio-1... Wo E [ga and V WL W.... W R1*= are both non-trainable random matrices. In this case, the gradient of the loss with respect to Zout is given by
obtain closed-form
dzou (V W Zin
V.
From now on, we will abuse the notation and use the subscript denote the training step as well for the matrix W Wt, When we use the notation W,, it should interpreted as We, Taking one step with SignSGD yields
We1 We 1nnnnnn>aa (Ix:((V),
where S(.) sign(.) and X -S(VV Wizin :))
Next, we prove result that will be useful the proof of Theorem More precisely, we show that under mild ussumptions, there exists a first initial training phase in which the sign of the loss function on the training datapoint does not change. The number of steps this phase is bounded by 17" up to some constant factor. Naturally, since we initialize with random variables, it should be expected that such result could only hold with high probability.
Theorem (Constant loss derivative sign in the initial training phase). We assume that the weights Wo. Wi...., ......,,. WL initialized such that the following holds:
|+ e [Z, for all i [1 : where Z, 7 > O are constants independent of n.
Mean-field Init: [[V]]= and Var (V.( -2 (e.g. uniform distribution [-n 'D).
Further assume that y E [2,
Then, for any de 10, and T 17 (n-6- 8
7 we have with probability at least 1 - n. 1+8
2)
vt S T Xt Xo-
The assumption on the weight initialization mild and is satisfied by some standard initialized schemes, such uniform init with n variance for the hidden weights, d'1 variance for the input weights, and n -2 variance for the projection weights. The proof Theorem 2 relies on standard concentration results.
Thhis satisfied with simple adjustment of the constants 7.
18

---

Published as a conference paper at ICLR 2026
Proof. Recall the definition of X
Xt S ()
We have the following from above,
Wizi WW-12 npni -1 | Xxx1 S(V),
and therefore,
V WT W2-22nn 1pn I '1 11lV
which implies that
V Wizin = VT Wozi 1 I
Σ Xi
Bounding V W
With Chebyshev's inequality we have:
Woz
P Wozml Zn=d)<
(Zn-6)2
where
Fin 1? Var V Wozin) Var(Wo 2mn)= (ar(W)2ma))= 1
S 22
n
n
As result, we obtain:
P(IV 2 6) en -122
If Wo 5 < O, we have
Vi S V Wozin 9 + 1712
1|2nn112
V Wozsn - ZT
With probability least 1 n
we have
V" 0$ 7n - G ?T
Therefore, we have
1 (Z (
T Tsn
Vt Xt x0=
If WT Ô asymptotically this implies that -ÿ 0 (assuming On)) Similarly, obtain with probability at least 1 -1-28
V Wiein 02 V Wozin 2T,
2 2 1 2
and therefore, we have that
(- -y
Tsn
7
- n
VTS X Xu =
In summary, we have the following: Let 5 (0, 1). Then, with T s ss ( n n))). we have for all
l < T, Xt. Xo-
The assumptions in Theorem 2 can be alleviated to include more generalization initialization schemes, such as non-clipped Gaussian initialization. However, this will require additional control on the asymptotics of [| | i, and V. The result remains the same however.
19

---

Published as a conference paper at ICLR 2026
C.2 PROOF OF THEOREM 1
Theorem [[Feature Norm Growth in Linear Networks]
Assume that the neural network is linear. Then, for any å (0, 1/2), under the assumptions on the initialization stated in Theorem 2, there exists universal constant À > 0 such that for any I and such that T S Xn) the following holds with probability at least 2n 1126
sup n [W,2m -
1steT
where Dt To (1+ -(t+((t)),, 77 nn || 11 and Tu n (Woznnl other words,
n [Wain exhibits quasi-quadrutic growth at early training phase, when the width is sufficiently large.
Proof.
Recall the update with SignSGD
W W, - 17n -nnnnnxx Xx. S(V) Zin
where S(.) sign(.) and Xt S(V Wizin - -).
Denoting 04 ((V)), obtain
0:+1
.xx x 0gg -n Xt.
Therefore,
[ | 11"2
m 12 3
2nn
11 lixe
[ +.82 28x ar.
Let 8e 1/2.. From Theorem 2, it is straightforward that there exists a constant À 0 such that for any T> 1 and 7) such that Any with probability at least 1- n- 1+6 In this case, for T, have X az Xt on An Σ5- X x X x x Bn x
have for all / s T, Xt = Xo-
Xi
Therefore,
n- W 2 n (W + 23 -28xmn ag.
Using Chebyshev's s inequality, we can easily show that for any 8e (0, with probability at least
1 11 1+28 we have
Ian Yan-
aol s Zn
which yields that with at least the same probability we have
In - WWezin 12 22 - r. " 2 - 232m
where we define the sequence T = P, 38 + 2t), with To n- W Then, is straight- forward that for all / s
In | 2
With union bound, this occurs with probability least -16
Note that the probability bound can be significantly improved by considering sub-gaussian concentra- tion bounds instead Chebyshev's 's inequality. Since our aim this paper is mainly methodological, do not include it here.
20

---

Published as a conference paper at ICLR 2026
D ADDITIONAL EXPERIMENTAL DETAILS
D.1 EXPERIMENTAL. SETUP FOR THE LINEAR NETWORK
The linear network is given by
f(x) W2W1Wor
where x IR Wo Raxd Wi E R and 2 E R.....
Dimensions. We use d = 100 in our experiments.
Training Data. We generate a random vector e R with iid coordinates ~ -dd1//2(00 (0, 1) and fix for the next step. Then, we generate N 1000 samples from the following distribution:
x Rd random vector with coordinates 2, (0,1
y w x 6, where (0,00.02))
Training.
We use Adam algorithm for training, and train the model for T 300 steps with full
batch.
D.2 EXPERIMENTAL SETUP FOR SFT (CLASSIFICATION)
For ANLI experiments, we use the following training configuration
Training datasets: ANLI
Training algorithm: AdamW, no warmup, linear schedule, dropout (0.1) 1).
Max sequence length 256.
LoRA a 2r
Precision: bf16.
use 8 for MLP placement stratgy, and adapt r to match param count for other placement strategies. Specifically:
Qwen3.5-0.5B: MLP 8), Attn (r 36), PLoP(r 17)
* -llama3.2-1B: MLP (r 8), Attn (r 27), PLoP(r 15)
D.3 EXPERIMENTAL SETUP FOR SFT (TEXT GENERATION)
MetaMathQA.
For the SFT experiments on MetaMathQA we use the following training configura-
tion
Training dataset: MetaMathQA
Training algorithm: Adam
epochs: 2
warmup: 0.1 fraction
schedule: cosine
dropout
For each adapter placement sweep over the learning rate in (1,2,3,4,5) 14-
Max sequence length 1024.
LoRA a 2r
Precision: bfl6.
For evaluation GSM8k the script evaluate_ hww.cchhaaaaaaeeeee py in the official QwenLM repo. We evaluate with 8-shot examples using the Qwen chat template. We apply a strict match for evaluating the accuracy and allow 512 generation tokens.
21

---

Published as a conference paper at ICLR 2026
SFT on Languages. For the SFT experiments multilingual language modeling (AYA dataset) we use the following training configuration:
Training dataset: Aya multilingual dataset (Portuguese, Japanese, Arabic)
Models: Llama-3.2-1 11221B-Innsse Instructt and Llama-3.2-33-Imme
Training samples: 4,500 per language
Training algorithm: AdamW
epochs: 50
warmup: 0.1 fraction (10%)
schedule: linear decay
learning rate: sweep over (1,2,3, 4,53 x 10-5
weight decay: 0.01
Batch size: 128
Max sequence length: 128
LoRA configuration:
rank 16 for MLP (adjusted for other methods match param count)
32
No dropout
Precision: fp16
Evaluation metric: Negative log-likelihood (NLL) on held-out test set
CommonsenseQA Experiments. For the SFT experiments on CommonsenseQA we use the
following training configuration:
Training dataset: CommonsenseQA
Models: Dive225--B-Innstrce Qwen2.5-32B-Instre
Training samples: 9,500
Training algorithm: AdamW
epochs: 50
warmup: 0.1 fraction (5%)
schedule: linear decay
learning rate: sweep over (1,2,3,4,55 x 10-5
weight decay: 0.01
Batch size: 32 with gradient accumulation steps of 8 (effective batch size: 64)
Max sequence length: 256
LoRA configuration:
rank 8 for MLP (adjusted for other methods to match param count)
2r 16
dropout: 0.05
Precision: fp16
Evaluation metrics: Negative log-likelihood (NLL) and multiple-choice accuracy on held-out test set (500 samples)
D.4 GRPO EXPERIMENTS
For the GRPO experiments MetaMathQA we use the following training configuration:
Training dataset: MetaMathQA (50,000 samples)
Training algorithm: AdamW
22

---

Published as a conference paper at ICLR 2026
learning rate: x 10-6 (fixed, not swept due to computational constraints)
warmup: 0.1 fraction (10%)
schedule: cosine decay
weight decay: 0.01
Batch size: 64 (16 per device with gradient accumulation steps)
Number of generations per prompt:
Maximum generation length: 512
Precision: bfl6
Reward functions: Combination of correctness and format rewards
Evaluation: Custom GSM8K evaluation script using model-specific chat templates
Note: Unlike the SFT experiments, we could not perform learning rate sweeps for GRPO due to limited computational resources and the high cost GRPO training runs. We expect that learning rate tuning could further improve the results.
D.5
ADDITIONAL. EMPIRICAL. RESULTS
D.6
SENSITIVITY ANALYSIS W.R.T SEQUENCE LENGTH
1.0
0.8
Kendall
0.6
Llama-3.2 1B
0.4
Owen-3 1.78 Qwen-2.Ce
23
24
2
26
2
2
29 200
Sequence length
Figure 10: Sensitivity of the PLoP- -ick-p module types as a function sequence length.
Sensitivity of PLoP to selection sequence length. To understand the impact of sequence length on PLoP-selected module types, we run the following experiment: we randomly select batches of size 128 from TriviaQA (Joshi et al., 2017), trim the sequence length to given value and compute Kendall 7 correlation coefficient between the module types selected by PLoPfor each pair. We do this for 200 pairs of batches, and show the distribution of Kendall in Fig. 10 function of sequence length. Across different model sizes, we observe that correlation is close to for sequene length 128, indicating that PLoP can be used relatively small sequence length.
D.7 EVOLUTION OF NFN SCORE DURING LORA FINETUNING
In Fig. 11, show the evolution of the NFN scores Portuguese language dataset (see experimental setup below) for the three PLoP-selected module type: Value, Out proj, and Down proj. There roughly 1750 fientuning steps, but limit the curve to 1000 steps for better visualization. NEN scores increase with finetuning, which is agreement with theoretical predictions in Section 2.
finetuning Llama3.2 BB model on
23

---

Published as a conference paper at ICLR 2026
1.1
NFN score
V pro Out Out_proj
1.0
Down_prg
0.9
0
250
500
750
1000
Finetuning step
Figure 11: NFN evolution during finetuning.
D.8 GRPO RESULTS FOR GEMMA3
Table 6: GRPO results for Gemma3-1B trained on MetamathQA (Yu et al., 2023).
Module Types
Rwd/Forma
Rwd/Answer
Eval/GSMMK
No RL
29.10%
Attn (Q-K-V) (r = 16)
2.16
0.89
30.05%
MLP (U-D-G) (r 16)
2.11
0.88
29,81 %
PLoP-1 (O-G-D) (r 16)
1.91
0.86
28.05%
PLoP(K-V-U) (r 16)
2.36
0.92
30.52 %
Interestingly, for Gemma3 1B, we found that most of the RL rewards was accumulated in forms of format reward (placing the thinking process between <think> and </think> and the solution between <answer> and < </answer>). This is reflected in Table 6. However, for eval on GSM8K, we found that accuracy after GRPO didn't change significantly which is probably due the fact that Gemma3-1B IB is weak on such tasks. In such cases, LoRA is probably not suitable, and full finetuming needed enhance reasoning capabilities.
D.9 ADDITIONAL NFN-MAPS
D.9.1 QWEN3-1 ---IRRRUUTT
<ltn
3.52
2:6
2.70
1:20
1:3
2,42
L33
3
3:42
1.4L
1
3.26
1:3
2:22
2:24
2:07
2-42
1:14
1:12
C:44
6:77
0:72
5:01
5:74
6:71
L14
1:04
atn.g
3.54
1:45
39
1:75
L45
1:44
62
2/45
2:5
3:53
1:41
3:40
1:35
1:31
2.20
1:24
2.39
3:38
2:20
1.26
20
3:33
1:32
1.20
2:3L
3.02
2:36
No0.00
30hn K_DR
5.37
0/15
8.70
5.5T
LO
1:5:
0:57
5.77
0.35
2.55
0.12
2.57
0:38
2:54
0.57
0.5
2.55
CURE
2.55
0:30
mip down
1:14
1:12
6/44
6/4
1:66
WV
6:44
mp..at
2.37
2.7E
3.38
1.4G
2.38
3.33
3.49
152
2/43
2.27
3.0
1.00
10
11
12
16
17
18
19
20
21
22
25
27
Layer
Allgnement by Module Type
2.0
random
Aligmeet Rato (NFN)
math
lngic
history
1.0
1.pr
k.pmj
T_prjj
p,praj
gutt praj
down prej
proj
Figure 12: NEN scores for Qwen3-1.7B
24

---

Published as a conference paper at ICLR 2026
D.9.2 QWEN2.5-3B IN-INSTRUCT
HFN--Map wwn22.5 D-instruct (Math)
4ltn pTS
6:21
2:32
38tn 9.0.
240
L-0
2:30 6:52
33
1:00
9:00
000
647
630
XCE
C.22
0:30
3.99
004
0.79
2:30
0.90
L.3
6.53
6:55
L07
1.00
0.00
Wodi 5300
AtTn (
114
1:30
170
2ltn pty
L10 242
224
CU
LA
L.3
0.
080
Alignement by Module Type
2.0
random
Aligmment Ratio (MF
logic
hilsorry
code
1.0
0.5
E.pmj
P.pmj
pry
gate, proj
down proj
up, pmi
Figure 13: NEN scores for Qwen2.5-3B
D.9.3 DWEN2.5-I.SB-INSTRRR
<ltn pmpm)
6.77
1:T
1.4G
2
L:0
3.27
3.30
1:32
2
3.27
2.22
2.02
atmo
1:13
6:92
00
0U55
9.5
5/97
SDW
9:90
1707
0.93
0.57
5:07
1/34
0:44
6.14
RURE
5/72
6:47
atn.g.
3.34
1:47
32
45
2:
1:22
LA5
3:30
2:2
3
1.26
3:34
1.34
1:25
1.05
2.04
2.33
3.23
10
1:33
1:31
2:25
0:00
2.00
L/CC
2.0Z
6.36
30hn X.ng
6.75
1.00
6:00
0:73
0/75
3.7
300
5.05
3:21
0:22
1.54
L.
2.00
0.24
2.9X
0.22
2:0
6:25
80
0:75
2.05
2:55
3.27
mip down
1.19
0.86
6:
1/0N
1/2%
1:11
14
1:26
1:07
16
6.47
mp..gate
2.33
2355
25A
2.33
1:30
3.33
1.37
2.47
2.38
3.35
27
3.34
1.55
2227
2.24
mlpup
6:1
10
11
12
16
17
18
19
20
21
22
25
27
Layer
Allgnement by Module Type
2.0
random
Aligmeet Ratoo (NFN)
math
logic
code
1.0
k.pmj
p_pmjj
p.praj
gutt, praj
down prej
proj
Figure 14: NFN scores for 0wen2.5-1.5B
25

---

Published as a conference paper at ICLR 2026
D.9.4 115.22.5-1.5B-COL INNTRRIN
NEN-Map for Qwwn2.ss-eeeeeeee 202.2--oder1..... Ee---neerrct (Coda)
<ttn 5_m
2:22
2:00
2:50
1:70
L:40
L:44
L:4
2/52
2:02
1.76
2:57
1:42
2-40
L:14
2.50
2-4
3.30
1:00
1:45
1:55
L-C
CAS
3.20
3ttn k_mm
3:24
1:08
1:35
0:97
1:41
6:76
9:95
0.93
2:28
0L08
2:67
0:27
7:
0:95
2.5
3:22
2.91
2.5
atin p.m.
2:22
2.5L
3.38
1:36
1:31
2:34
1:37
1:35
2.09
6:35
2.0
0:22
1:55
8:36
0.75
D:22
2.22
3:00
5.22
0:22
2L
0:33
2.02
5:26
3:00
0:33
0.25
022
2:33
3.22
0.5
19
Alignement by Module Type
tandom
Algnmmnt Rabo INFN)
2.5
logic
history
code
2.0
0.5
0.0
pro
k.pmj
v_Proj
prproj
ghie proj
down proj
proj
Figure 15: NFN scores for Qwen2.55-1.5--Co
D.9.5 GEMMAS-IB-INSTRUCT
NEN-Map for Gemma- 5-10- 11D-Instruct (Math)
attn. pm
1.01
3:4L
2,03
1.26
202
1:11
011
2:25
1.55
2:25
1:22
2:27
1.38
2.0
Noduls Type
1
5.55
56
1:29
2.27
3:13
1.23
3:38
1:04
6O
3.75
1000
3.33
1.0
mb gate
121
3:44
1:11
1.52
1.42
2.40
1.7
3.52
1.22
2:53
3:30
1:0
2:45
1.4
1.42
1.34
20
3.25
1:30
1.33
0.5
uppup
6:41
5:48
1:40
0.9
6:
3.9
0.59
2:91
0:40
0.91
0.97
SMI
6.95
S:N
1.00
1:54
2:44
14
36
21
Läyer Index
Alignement by Module Type
random math
Alighment (NEM))
1.5
logic history
1.0
0.5
0.0
a.pm)
k pmg
V. '_png
0. prrr
gade proj
down, down_pmj pmj
up pp.
Figure 16: NFN scores for Gemma3- IIIntrcttt
26


Reviews:
[{"review_id": "ATZSBkKDcl", "reviewer": "Reviewer_vSmF", "summary": "This paper introduces PLoP, a method for dynamically assigning LoRA adapters to different module types given pretrained models and fine-tuning tasks based on the normalized feature norm (NFN) score. The authors provide theoretical and empirical justification for the NFN score as a measure of module-data alignment. PLoP is derived from the NFN score to identify modules that benefit most from SFT. Computing MFN is done via a few forward passes (without gradients etc). The authors provide results to prove the benefit of PLoP on LoRA fine tuning for classification, text generation, and mathematical reasoning using accuracy as a metric.", "questions": "I'd like to better understand the rationale behind the chosen baselines.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Authors outline the importance of adapter placement, and show a computationally tractable and architecture-agnostic way of doing so. \n\nTheoretical justification of NFN scores is compelling. \n\nExperiments shed light on the variance of MFN scores between layers, modules, and architectures (with and without reasoning), indicating that there is a need for a method that takes that variance into account.\n\nThe proposed framework and contributions are novel to my knowledge. \n\nParameter-efficient finetuning methods are very timely (especially for discussions of continual learning).", "weaknesses": "In Text:\n- Figure 1 needs more clarification on inputs and outputs, baseline feature norm. \n- In section 3, first motivate and introduce random baseline feature norm, then introduce NFN formulation to improve clarity.\n- The second and third paragraph in the Introduction can be tied together and streamlined. They also cover almost exactly the same content as related works.\n- Please elaborate more on the baseline presented in section 3/figure 4 and 5\n- Authors need to improve formatting in Appendix.\n\nEmpirical support for PLoP as a framework is lacking: \n- It is not fully clear to me why the specialized Qwen Math model has higher NFN scores on a Math dataset; more focus on this could strengthen empirical justification\n- PLoP^{-1} serves as a weak baseline, I would expect to see current SOTA selection methods (if existing), random selection, other scoring heuristics, or oracle module placement as baselines (this could be found by running a grid search over the modules of only one layer (for simplicity) of a 1B model, and comparing to the best selection).\n- The improvements offered by PLoP in SFT for text generation are not super clear in section 4 (the authors are ablating over multiple parameters in the same table, unclear which numbers to compare); it took me quite a bit of time to understand the benefit PLoP offers over the baseline; could use better highlighting or illustrations to emphasize improvements.\n- Baselines that are lacking: full finetuning, base model performance, gradient-based selection methods.  \n- Currently, the experiments section is hard to follow. Details on experimental setup and motivation are intertwined with the results, which is distracting.\n- Please provide more details on computational complexity of PLoP and how it compares to methods that take gradients.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1bM4mZBAql", "reviewer": "Reviewer_VHUP", "summary": "The paper proposes a method called PLoP for dynamically selecting LoRA training modules through precomputed metrics. Specifically, PLoP first defines an indicator, namely NFN, to measure the alignment degree between model modules and training data, and selects the modules for inserting LoRA adapters based on this indicator. The NFN score of each module is obtained by dividing the output from the module with the original data features by the output from the module with random features.", "questions": "1) PLOP is equivalent to fine-tuning one of the modules via LoRA. Why didn't the authors indicate which module corresponds to PLOP in the experiments?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "1) The proposed method exhibits strong generalization capabilities. It presents a unified training framework that can be generalized to any large model training approach. By simply providing the model and dataset, computing the NFN metrics, and then freezing modules based on these metrics, the method can be applied universally.\n\n2) The theoretical analysis is comprehensive. The design motivation of NFN is supported by the theoretical basis of \"Feature Norm Growth in Linear Networks\", making it theoretically reasonable.", "weaknesses": "1) The most unacceptable issue with this work is that the experimental results are not significant. In the main experiments, there is no substantial difference between PLOP and fine-tuning other modules, which may indicate that researching which module is more worthy of fine-tuning has little value.\n\n2) As a method for saving computational resources, what I expect from PLoP is its efficiency in the efficient fine-tuning of large-parameter models. However, the experimental part only uses small models below 7B.\n3) The datasets used are relatively single. Consistent with the first point, since NFN calculation depends on specific datasets and models, the generalization ability of PLoP may lack verification on multiple datasets.\n4) It is recommended to optimize the typesetting of Figure 8's position.\n5) Figure 2 and Figure 8 are duplicated. Should it be corrected to Figure 2 in Line 191 ?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JeGr4EKxdY", "reviewer": "Reviewer_Goqy", "summary": "The article under review proposes PLOP, an inexpensive, task-aware way to place LoRA adapters by ranking module types with a simple alignment score and inserting LoRA where alignment is lowest.  The authors use a normalized feature norm (nfn) that compares a module’s feature magnitude on real inputs to a randomized-input baseline:\n$$\n\\mathrm{NFN}(W,x)=\\frac{\\lVert W z_{\\text{in}}(x)\\rVert}{\\lVert W \\tilde z_{\\text{in}}(x)\\rVert}\n\\approx\n\\frac{\\lVert W z_{\\text{in}}(x)\\rVert}{\\lVert W\\rVert_F,\\lVert z_{\\text{in}}(x)\\rVert}\n$$\n$(\\tilde z_{\\text{in}})$ shares the norm of $(z_{\\text{in}})$ but has i.i.d. gaussian coordinates. higher nfn means stronger module–data alignment; scores near 1 mean weak alignment. plop averages nfn per module type and places LoRA adapters in the lowest-scoring types. Some theoretical justifications behind the method are also provided.", "questions": "1. How stable are nfn rankings in practice? Please report bootstrap confidence intervals and a stability curve (kendall-τ vs number of tokens/batches) for per-type nfn, plus any sensitivity to seq length. \n\n2. Can you give a targeted layer-level ablation using the nfn-map? Choose the bottom-k individual layers by nfn and compare against type-level plop under the same parameter budget; report when layer-level helps or hurts, and whether an nfn-gap threshold or per-type cap stabilizes results. this would clarify the “inconsistent results” remark and indicate how to operationalize finer placement.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The paper reframes adapter placement as a task-aware alignment problem and proposes a simple, gradient-free proxy (normalized feature norm) to rank module types. this is a clean definition that removes scale effects via a randomized baseline, making it broadly applicable across architectures and tasks. the theory–metric–procedure chain is coherent and distinct from gradient/sensitivity-based selection, which is heavier and less LoRA-friendly.\n\n2. Theoretical pieces motivate why feature norms should grow with alignment and width, then connect that to a practical nfn estimator; even if idealized, the analysis yields testable predictions that the experiments largely reflect. the empirical study spans sft classification, sft math generation, and rl (grpo) with careful parameter-matching, showing consistent wins or parity against attention-only and mlp-only baselines, sometimes beating “all modules” at lower budget. compute claims are credible and supported by explicit cost discussions and a one-batch nfn recipe.\n\n3. The mechanism is easy to follow: compute nfn per type, pick the lowest, insert lora. the paper repeatedly reminds the reader what the type codes mean (q, k, v, o, u, g, d), provides nfns as maps and per-type bar charts, and summarizes training configs in appendices. figures and tables tie back to claims (e.g., llama3.2 parity with mlp explained by low mlp nfn; qwen math gains with d–o–v). overall presentation is minimal and readable.\n\n4. Adapter placement is a common pain point with real budget constraints; a method that costs roughly one forward pass and often matches or improves over prevailing heuristics is practically valuable.", "weaknesses": "1. The finetuning tasks are confined to anli and math (metamathqa to gsm8k) for sft and grpo; there are no non-math generative, instruction-following, multilingual, or long-context evaluations. while fig. 4–5 report nfn patterns for code/history/logic, there are no finetuning experiments on those domains, so generality remains untested.\n\n2. NFN is computed from a single forward pass using hooks, but the paper does not report estimator variance, sensitivity to batch/sequence length, or across-batch stability of type rankings. without confidence intervals or stability curves, it is hard to know how often plop would pick the same types on a new sample. \n\n3. The authors note that layer-level placement using the nfn-map gave “inconsistent results” and defer it, yet per-layer or per-head selection is exactly where a cheap ranking could shine; not probing this systematically is a weakness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zDzykHdUm1", "reviewer": "Reviewer_VYDu", "summary": "This paper introduces Precise LoRA Placement, a lightweight method to determine where to place LoRA adapters. Specifically, this paper proposes a alignment metric based on feature norm that identifies the modules requiring fine-tuning by performing inference on a small number of samples from downstream datasets. Extensive experiments demonstrate the effectiveness of the proposed method.", "questions": "1. According to the definition of the proposed metric, it is possible to directly compute the score for each module in every layer. Why not adopt a more fine-grained localization strategy, instead of grouping modules such as Q, K, and V together for unified localization?\n\n2. The results presented in Figure 5 appear to contradict the intuition behind the proposed method. The paper suggests that modules with lower alignment are more suitable for fine-tuning, yet Figure 5 shows that, for task-specific models, the largest score improvements (compared to general model) occur in the modules that were initially the highest, rather than in the low-alignment ones.\n\n3. It would be helpful to show the dynamic changes of the proposed metric during the fine-tuning process.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed metric appears to be lightweight and efficient.\n\n2. The proposed method has demonstrated its effectiveness through extensive experiments.\n\n3. The availability of code enhances the reproducibility of the work.", "weaknesses": "1. The experimental section only compares the proposed method with the original LoRA, without including other LoRA variants or approaches that also explore efficient LoRA adaptations.\n\n2. The definition of the Normalized Feature Norm is confusing. The motivation for introducing a random variable here, rather than directly normalizing by the product of the norms of two vectors (or matrices), is not clearly explained.\n\n3. The results in Figure 4 show that $v_proj$ consistently exhibits lower alignment score across different models, which is an interesting observation. Providing an explanation for this phenomenon could deepen the understanding of the proposed metric and enhance the overall depth of the paper.\n\n4. Directly reporting the computation time and memory usage required for the proposed metric would provide a clearer and more intuitive demonstration of the method’s efficiency.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)