Pay Attention to Small Weights
Authors: Chao Zhou
Tom Jacobs
Advait Gadhikar
Rebekka Burkholz
Venue: NeurIPS 2025 poster ()
Keywords: fine-tuning; LLM; efficient ML; gradient compression;
Primary area: deep_learning

Abstract: Finetuning large pretrained neural networks is known to be resource-intensive, both in terms of memory and computational cost. To mitigate this, a common approach is to restrict training to a subset of the model parameters. By analyzing the relationship between gradients and weights during finetuning, we observe a notable pattern: large gradients are often associated with small-magnitude weights. This correlation is more pronounced in fine-tuning settings than in training from scratch. Motivated by this observation, we propose \textsc{NanoAdam}, which dynamically updates only the small-magnitude weights during fine-tuning and offers several practical advantages: first, the criterion is \emph{gradient-free}—the parameter subset can be determined without gradient computation; second, it preserves large-magnitude weights, which are likely to encode critical features learned during pre-training, thereby reducing the risk of catastrophic forgetting; thirdly, it permits the use of larger learning rates and consistently leads to better generalization performance in experiments. We demonstrate this for both NLP and vision tasks.

Pay Attention to Small Weights
Chao Zhou Tom Jacobs Advait Gadhikar Rebekka Burkholz CISPA Helmholtz Center for Information Security, Saarbrücken, Germany {chao. .zou, tom.jacobs, advait, gadhikar, burkholz)@cispa. de
Abstract
Finetuning large pretrained neural networks is known to be resource-intensive, both in terms of memory and computational cost. To mitigate this, common approach is restrict training to subset of the model parameters. By analyzing the relationship between gradients and weights during finetuming, we observe a notable pattern: large gradients are often associated with small-mag weights. This correlation is more pronounced in finetuning settings than in training from scratch. Motivated by this observation, we propose NANOADAM, which dynamically updates only the small-magnitude weights during finetuning and offers several practical advantages: first, the criterion gradient-fee--the parameter subset can be determined without gradient computation; second, it preserves Sarge-manittud weights, which are likely to encode critical features learned during pretraining, thereby reducing the risk of catastrophic forgetting; thirdly, it permits the use of larger learning rates and consistently leads to better generalization performance in experiments. We demonstrate this for both NLP and vision tasks.
Introduction
With the advent of Transformer-basse models like GPT [26], large models (LMs) [32, 20] excel
across domains such natural language processing (NLP) [3, 37] and computer vision (CV) [21, 41]. They enable effective knowledge transfer via finetuning (FT) on downstream tasks, facilitating the development of domain-specific models.
is well known that fully finctuning LMs requires substantial computations and memory [33, 12]. One of the main reasons is that the predominant optimizers, Adam [16] and its variants [22], maintain both first- and second-order momentum estimates for each parameter [16, 22]. For a model with N trainable parameters, this results in memory footprint equivalent to storing approximately 3N parameters, significantly limiting scalability. To address this problem, methods such as gradient checkpointing [4], quantization [11], and parameter offloading [28] have been developed. Gradient checkpointing [4], for instance, reduces memory usage by storing intermediate feature maps and gradients and recomputing them during backpropagation--a trade-off that sacrifices computational efficiency for reduced memory demand. Similarly, 8-bit Adam [7] addresses memory overhead by quantizing optimizer statistics 8-bit precision while leveraging block-wist dynamic quantization to maintain numerical stability, thereby minimizing performance degradation.
Recently, projection- based methods have emerged promising approach to reduce memory overhead. For instance, GaLore [44] enables full-parameter training by projecting gradients into low-rank subspace, applying Adam-like updates there, and projecting them back to the original space-thus reducing memory usage. However, GaLore relies on SVD decomposition of gradi- ents, which is only applicable layers satisfying the reversibility property [27]. MicroAdam [24] compresses gradients using top-k selection and mitigates performance loss via an error feedback mechanism inspired by distributed training. However, still incurs memory overhead for storing accumulated error and introduces extra computation for quantization and feedback. In contrast, our
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

method does not require gradient information to determine which parameters to update and eliminates the need for error feedback, resulting in improved efficiency.
Contributions In this paper, we investigate the relationship between gradients and parameter magnitudes in the context of finetuning LMs. Our empirical analysis reveals a strong pattern: pa- rameters with large gradients are associated with small magnitudes. While this correlation is not seffect-smalless weights do not strictly correspond to largest gradients- we find that selectively updating small-weight parameters consistently more effective than updating those with large gradi- ents. Yet, learning small weights works because of the association with large gradients that support learning. To deepen this insight, we provide theoretical analysis using a two-layer teacher-student framework, showing that updating small weights not only yields more efficient learning but also helps mitigate catastrophic forgetting during finetuning. Conceptually, our analysis reconciles two distinct sparse finetuning principles: training parameters with large gradients versus small magnitude, showing that they act in tandem. While the former is the more dominant principle, we systematically highlight the advantages of the latter.
Motivated by these findings, we propose NANOADAM, an optimizer that selectively updates pa- rameters with small absolute magnitudes. Compared to prior methods, NANOADAM offers several advantages: (1) it avoids reliance on gradient information, allowing precomputation of update masks and dynamic sparsity control; (2) it eliminates the need for error feedback, improving both memory and computational efficiency compared to microAdam [24]: (3) after finetuning, over 80% of large parameters remain untouched, depending on the sparsity level; and (4) by avoiding updates to large weights, it preserves critical features from pretraining, mitigating catastrophic forgetting [17]. Addi- tionally, by leaving large weights unchanged, NANOADAM implicitly performs weight regularization.
We evaluate NANOADAM across a range of NLP and vision tasks, demonstrating superior memory efficiency and generalization compared to baselines such as MicroAdam, AdamW-8bit, and GaLore. Notably, the efficiency benefits become more pronounced at larger scales. Furthermore, NANOADAM significantly reduces performance degradation on previously learned tasks during continual learning, effectively alleviating catastrophic forgetting.
Related work Fully finetuning a pretrained LLM is known to be resource intensive, prompting substantial work into parameter-efficiien-- -infiiiienn--iettuiinnggg (PEFT) [12, 34]. Well-known method such as Low Rank Adaptation (LoRA) and its variants [14, 38, 40, 39, 35] update only a set of trainable low-rank matrices while keeping the original model parameters frozen. After finetuning, these low- rank matrices are merged with the original parameters, thereby preserving the inference efficiency. Nonetheless, such low-rank methods can constrain the model's 's expressiveness [2], limiting its ability to capture complex patterns in new tasks. Furthermore, the introduction of auxiliary parameters leads to an increase in model size during the finetuning process [27].
Several approaches have explored the development of memory-effieien optimizers, motivated by the substantial memory overhead of standard methods like Adam, which require storing multiple optimizer states. Quantization-basee by representing optimizer states in lower precision. GaLore [44], on the other hand, maintains full parameter training while enhancing memory efficiency through low-rank gradient factorization. However, the applicability of this type of method is restricted to layers that satisfy the reversibility property, limiting its effectiveness in models lacking this characteristic. Furthermore, the need to perform singular value decomposition (SVD) [43] introduces non-trivial computational overhead. More recently, www-Dimensional Adam (LDAAdm) [30] has been proposed, incorporating low-rank compression for both gradients and optimizer states. To address projection-induced errors, an error feedback mechanism is employed; however, it still necessitates memory to store accumulated errors.
techniques, such as Adam-Sb [7], reduce the memory footprint
Another line of work focuses on optimizers that update a small subset of trainable parameters without modifying the model architecture or the overall training procedure. For example, SensZOQ [10] finetunes small, static subset of sensitive parameters, identified by selecting those with the largest squared gradient magnitudes. This approach requires computing the full gradient to determine param- eter importance, after which a static mask is applied for zeroth-order finetuning. MicroAAam [24] improves efficiency by dynamically selecting parameters with the top-k gradient values at each opti- mization step, combined with an error feedback mechanism for correction. However, it still requires maintaining a history of selected gradients and quantized errors, potentially offsetting its memory advantages. Similarly, Dynamic Subset Tuning (DST) [31] updates parameters that exhibit the largest
2

---

distance from their pretrained values. Nevertheless, DST must first compute full optimizer updates before selecting the top-k parameters, leading additional computational overhead. BlockLLM [27] observes that parameters with smaller weight magnitudes tend to be updated more frequently. Never- theless, it still relies on gradient-based criteria to determine parameter importance. While prior work has largely focused on selecting parameters based on large gradients, one notable exception is [19], which proposes updating small unimportant weights. Their method uses a static mask defined at initialization, without considering dynamic adaptation throughout training as NANOADAM (Further discussion is provided in Appendix). Another related work [29] reduces the number of trainable parameters by randomly selecting small subset of weights, without leveraging structural signals such as gradients or magnitudes.
2 Small weights can matter
Which subset of weights should we update during sparse finetuning? Two main selection principles have been proposed: weights with large gradients approximate full finetuning dynamics [24]; b) weights that are unimportant for the pretrained model to maintain parameters that are important for generic representations, as has been argued in the context of LLM finetuning [19]. Conceptually, both approaches can have pitfalls. a) Full finetuning (and its proxy) could lead to catastrophic forgetting by adapting parameters that capture relevant concepts for both the pretraining and finetuning task. b) Meanwhile, weights that are irrelevant for the pretraining task might also be unimportant for the new task and therefore not contribute meaningfully to learning.
Our observation that large magnitude gradients are associated with small magnitude weights partially reconeiles both views. Both subset selection criteria work because they work in tandem. Yet, they still select quite distinct parameter sets. In the following, we argue in favor of updating the smallest weights dynamically during finetuning,
maintains more relevant information about the pretraining task, enables learning with larger learning rates and thus boosts generalization, and has algorithmic advantages for saving memory. This insight lays the foundation for our proposed dynamic sparse finetuning method, NANOADAM. To provide theoretical motivation and gain deeper insights into the catastrophic forgetting aspect, we study a two-layer seacher-student network.
2.1 Relationship between gradients and weights
We start by investigating the relationship between weights and gradients in finetuning scenarios. Specifically, we fully finetune the BERT-base model on the CoLA dataset from the GLUE benchmark and track the evolution of this relationship for each parameter throughout training. We also conduct similar experiment in the vision domain, where a ViT-Large model pretrained ImageNet is finetuned on the CIFAR-10 dataset. To better illustrate the distinction between finetuning and training from scratch, we repeat the vision experiment using the same model architecture but initialized randomly. Details and additional visualizations for these experiments are provided in Appendix B.
0.002
0.00
0.00
2:00
-1.04
.322
-0.00
-0.00
0.6
02
.524
-250
-8.65
0.50
630
Weight
Weight
Weight
Wegtt
(a) NI.P task: FT.
(b) CV task: FT.
(c) CV task: Train from (d) CV task: Train from scratch step=0.
scratch step=78100.
Figure 1: The relationship between gradients and weights during FT and training from scratch. The x-axis represents the magnitude of the weights, while the y-axis represents the magnitude of the gradients. From left to right, the subfigures correspond to the FT NLP task, FT CV task, training CV task from scratch at early step and training CV task from scratch at later step.
As illustrated in Figure I, parameters with large gradient magnitudes tend to correspond small- magnitude weights, with the notable exception of the final classification layer (see Appendix B). This
3

---

distinct, hyperbolic relationship is consistently observed across NLP and vision finetuning tasks, indicating that mall--magiitdee parameters are more actively involved in FT. These findings align with observations of [27]. where L.LMs were found to update small-magnitude parameters more frequently during adaptation. However, the underlying causes or implications of this phenomenon were neither explored nor exploited. We hypothesize that the strong hyperbolic correlation between gradient and weight may arise from one or more of the following factors: (1) knowledge transferability; (2) overparameterization.
Knowledge transferability As shown in Figures ((b) and (c), finetuning pretrained ViT-Large model reveals stronger hyperbolic relationship between gradients and weight magnitudes compared to training from scratch. This likely arises because large weights encode important features learned during pretraining and are therefore less plastic (i.e. are less prone or able to change) particularly when the finetuning task is similar. Instead, smaller weights adapt more readily to task-specific features. In contrast, training from scratch begins with randomly initialized weights, which lack meaningful structure. Consequently, gradients are more evenly distributed across parameters, resulting in more elliptical gradient- weight pattern (Figure (c)). As training progresses, this distribution gradually shifts toward the hyperbolic form observed in the finetuning regime (Figure I(d)), reflecting transition from general feature acquisition to more focused adaptation.
Overparameterization Large models are often highly overparameterized, allowing them to adapt to tasks without significantly modifying their pretrained large weights. To investigate this, introduce a metric in the gradient-weight space to quantify the degree of hyperbolic correlation. Specifically, identify the top-k parameters with the largest absolute gradients and compute the median of their absolute weights, denoted Wm. We take the bottom-k parameters with the smallest absolute gradients and compute the maximum of their absolute weights, denoted wM. The ratio
median [al. (aaiiaan((all-100 E
I= argmax, (lg)), == argmin
(1)
max [ JJ
captures the strength of hyperbolic association: a smaller r implies that large gradient parameters tend to have smaller magnitudes, indicating stronger hyperbolic trend. The use of the median for Wm ensures robustness to outliers.
Table 1: Ratio r for QKV weight matrices at selected layers during early finetuning. Lower r indicates stronger hyperbolic correlation between gradients and weights.
Model Layer 0
Layer 3
Layer 6
Layer 9
Layer 11
Layer 17
Layer 23
ViT-Tiny 0.200
0.040
0.120
0.070
0.200
ViT-Large 0.009
0.020
0.020
0.019
0.019
0.020
0.070
We evaluate r under two model sizes study the effect of overparameterizzaton:: ViT-Tiny and ViT-Large, both pretrained on ImageNet and finetuned on CIFAR- 10 using identical training set- tings. We compute the ratio using the top 0.01 % and bottom 80% of parameters, and report results
for QKV weight matrices across layers in Table 1. Visualizations are provided in Appendix D.
As shown, ViT-Large consistently exhibits lower r values, supporting the hypothesis that overparameterized models develop a stronger hyperbolic gradient-weight structure. This insight motivates the central question of our work: Can parameter magnitule-nu tude-serve serve
Battim 20. 2% Wrights
Heak
than gradient magni- more effective criterion for selecting which subset of parameters to update during finetuning? We explore this both theoretically and empirically next.
Gradients
-01-
2.2 Small weights and large gradients select distinct parameter subsets
0.6
0.4
0.0
Weights
To further investigate the relationship between weights and gradients, we examine two parameter subsets: the top 0.001% with the largest absolute gradients and the bottom 10% with the smallest weight magnitudes,
Figure 2:
Overlap between small
weights and large gradients.

---

based on Figure 1(b). This analysis aims to determine whether small weights tend to coincide with large gradients. The overlap is visualized in Figure 2. Interestingly, even the bottom 10% of small weights fail to fully cover the top 0.001% % of large gradients. This suggests that small weights and large gradients are not simply two sides of the same coin. Accordingly, also parameters receive large gradients that are deemed important for the pretraining task and their adaptation could lead to forgetting transferable concepts.
In contrast, we argue that focusing on small weights during finetuning has several advantages: (I) Modern neural networks are often heavily overparameterized, making it sufficient to learn downstream tasks by updating only small-weight parameters; (2) While not perfectly aligned, small weights have non-trivial chance of intersecting with large gradients. When combined with a dynamic masking schedule, this ensures broader coverage over time; (3) Large weights likely encode essential features from pretraining, and modifying them risks disrupting previously learned knowledge. Focusing updates on small weights introduces minimal interference with this structure. As shown in our experiments later, updating small weights achieves superior generalization and results smaller overall parameter shifts. This suggests that finetuning small weights follows distinct and efficient training dynamic, rather than approximating the path of full or large gradient-driven updates.
2.3 Nano gradient flow: From feature learning finetuning
We provide a theoretical motivation and visual illustration explaining why updating small weights could be more beneficial than updating large gradients.
Setup Consider a student-teacher setup based on two-layer neural networks [5]. Specifically, let R" x Raxd R denote a two-layer neural network with parameters (a. W) and input æ, fa, W(z) := Σ = (wo(w where a(-) = max(0,-) is the ReLU activation. All networks in this section follow this form. We first pretrain student network Spre in the feature learning regime using the mean squared error (MSE) loss and gradient descent for T steps. The inputs ( sampled i.i.d. from a multivariate Gaussian NO, Ta), where Ia is the d dddimnnsionna identity matrix. The targets are generated by teacher network freacher. To simulate finetuning, perturb the teacher network freacher with an additional neuron, yielding fúnctune = freacher + fextr- During finetuning, we compare two strategies: updating parameters in each layer with either the largest gradients or the smallest magnitudes. A case study involving multiple additional neurons is included in Appendix E, along with the corresponding hyperparameter settings for these experiments.
Theoretical motivation One of our goals is to pre- serve the original representation during finetuning, i.., to mitigate catastrophic forgetting. In two- layer neural network, this corresponds to retaining the largest neurons- those with the largest activa- tions. These can be ordered by their effective magni- tude, |a,| || weal ||- We show that training only the small- magnitude weights preserves the representation. We formalize this idea in the following definition.
sign
sign
Init
Final
(a) Nano GD.
(b) Largest gradients.
Figure 3: Nano gradient descent provably pre- vents catastrophic forgetting. (a) Nano gra- dient descent keeps the original representa- tion while learning the extra neuron. (b) The largest gradients can correspond to weights with large magnitudes leading unlearning of the original representation and the inability of learning the new representation.
Definition 2.1 A finetuned network f is k-neuron representation preserving iff the largest neurons cor- responding to Top-k(j E [n] [;]l[e,]]) remain unchanged compared to the pre-training task.
We assume pretraining occurs in the feature learning regime [5]. Due to the implicit sparsity bias in over- parameterized training. there often exists at least one sufficiently small neuron that has minimal impact on the pretrained representation. We analyze the effect of updating only the smallest neuron:
Theorem 2.2 Assume model f2) consisting of neurons learns the teacher faecche(a)) corre- sponding to a pre-training task so that = (x) for all x. E Rdd Furthermore, let fxx consist of at least two neurons i,r r E [n]] such that max { [a./2 |a,1)] S 6 for an 6. >> and freacher + fexrr with an
=
sign(a:) # sign(a,). ). Let new task be defined based on labels fjmetune
5

---

extra neuron feara ā Let only the neuron j of f be trainable to finetune fx) to the new }i Then, the gradient flow with respect to finetuning time of the neuron j, which is parameterized as Vi.t [aj.eWW3, and initialized at the pre-trained values U5,0 = |0y|u [Wgg converges to a value Usa so that ||000 ellza < Cc, where v is the target = jālu and C > 0 a data dependent constant.
task, where j = argming
(agnmiin(allll..aaa [a]]][wll| sign(a.)
Proof. The proof follows from using Theorem 6.4 [23] for learning with one neuron a one neuron target and controlling the perturbation incurred by the difference fieacher ((6)) i.e. the small trainable neuron u. (See Theorem E.1)
Theorem 2.2 indicates that updating the smallest neuron is sufficient for learning new representations without disrupting the pretrained structure. A key mechanism underlying the nano gradient flow (learning rate ! + 0, see Appendix E) is: In the feature leurning regime [9, 1J, grudient flow satisfies: |a,, 2 = ||,,, for all E [n] and t 2 0. This observation implies that selecting the smallest weights in each layer corresponds to training the smallest neuron. This allows nano gradient flow to learn new task-specific information while preserving the original representation. thereby reducing catastrophic forgetting. In contrast, selecting toge-gradden parameters updates large, pretrained neurons and risks overwriting important features. This highlights a more general principle by training the smallest weights: Nano gradient flow or NANOADAM learns compact task-specific representation, while (partially) preserving the pretrained representation.
Catastrophic forgetting We construct a teacher network with two neurons and pretrain a two-layer student network in the feature learning regime [5]. To simulate a finetuning scenario, we define a new task by adding a randomly initialized neuron and generating labels accordingly. See Appendix E for full details. We select the parameters with either the largest gradients or smallest magnitudes.
In Figure 3, the teacher neurons represented as arrows |a| ww. pretrained finetuned neurons visualized as points. The color indicates the sign of a. According to Figure 3a, selecting the smallest weights allows the network to learn the new representation while preserving the pretrained one. In contrast, as seen in Figure 3b, large gradients can interfere with large weights and degrade the original knowledge. Table 8 Appendix E confirms that weeght updates result in better generalization and smaller &g shifts, indicating greater representation retention. The measure is motivated by Lemma E.2 in the appendix.
NANOADAM
Motivated by the empirical findings and theoretical insights discussed above, introduce NANOADAM, an optimizer that finetunes subset of parameters based solely on their weight magnitudes, as outlined in Algorithm To further enhance efficiency, we incorporate density scheduler that dynamically adjusts the fraction of updated parameters during training. A detailed memory analysis is provided in Appendix G.
Algorithm 1 NANOADAM
We adopt standard Adam-like notation: let m. and i. denote the first- and second-order mo- mentum estimates of the gradients at step t, with momentum coefficients 31 and 82, and small constant c for numerical stability. Let f be the loss function, O, the model parameters at step i, and 7 the learning rate. The full gradient is denoted by VeJ. A mask indexes the selected
Require: initial density ko, mask interval m, den- sity interval d, total steps T, B1, 88
1: mo. up, 1, h +- 0, 0,,0,00 0, ko
2: for t = 0 to T do
3:
flag, < False
4: if t%d = 0 then
5:
+ density schedule (k, t, T)
6: flag, True
subset of parameters to update. The density of this subset is denoted by k. while m and represent the update intervals for the mask and density, respectively. Finally, T denotes the total number of optimization steps.
7: end if
8: if 1%mm 0 flag, True then
9:
I + Bottom;
end if
10:
+ [+f(0:)[1]
10: m, momentum (m 31)
3.1 Algorithm details
+ momentum update( (1:-1, (2)
B t Ar 17t vuite
The core idea behind NANOADAM is to de- termine mask for selected parameters solely
11: end for

---

based on their absolute magnitudes, selecting the subset with the smallest values. Given that small- magnitude parameters tend remain small throughout optimization, is unnecessary to update the mask at every step. Instead, we introduce a mask interval m, such that the mask 7 is only updated once every mn steps. This design provides two key benefits. First, improves computational efficiency by reducing the overhead associated with frequent mask updates. Second, it enables the optimizer to preserve the momentum-like dynamics of Adam, while maintaining the first- and second-orde momentum only for the selected subset, thereby reducing memory consumption. To further enhance memory efficiency, we incorporate a density scheduler, akin to a learning rate scheduler, that dy- namically adjusts the density l throughout training. By default, we employ linear decay schedule, though this mechanism can be disabled by setting the density update interval greater than the total number of training steps T.
Importantly, NANOADAM does not incorporate any feedback mechanism to compensate for the error introduced by gradient sparsification. This exclusion is a deliberate design choice based on three key considerations: (1) Error feedback mechanisms introduce additional memory overhead for storing residuals and computational overhead for accumulation and connertioon-ontraee optimizing efficiency; (2) NANOADAM does not aim to approximate the trajectory of full-gradient updates, but instead pursues distinct and efficient optimization path; and (3) Empirically, we find that incorporating error feedback offers no performance benefit and can even degrade generalization.
to the goal of
3.2 Ablation study
To validate the effectiveness of our method, conduct ab- lation studies comparing various masking strategies: small weights vs. large or random weights, and small weights vs. large gradients. An additional study comparing static and dynamic masking strategies is included in Appendix F.3.
Evaluation Accuracy
0.8 0.7
Weights
Weights
Randem Weights
Small vs. large vs. random weights We begin by evalu- ating the impact of different weight-based masking strate- gies in an LLM finetuning setup. Specifically, we finetune the BERT-base model on the SST-2 task from the GLUE benchmark using NANOADAM under three masking strate- gies: (1) small weights, (2) large weights, and (3) random weights. Importantly, each configuration uses the same gradient density k. (See Appendix F.1 for details.)
0.5
2000
4000
6000
8000
10000
Global Sceps
(a) Small vs. large random weights.
Evaluation Accuracy
00.92
0.91
0.90
Figure 4a shows the generalization results, with training loss dynamics available in Appendix F.1. Finetuning small- magnitude weights consistently yields the lowest training loss and highest evaluation accuracy. In contrast, updat- ing large weights results in the worst performance-ever random masking performs better. This could suggest that most large weights, which likely encode critical pretrained features, are less adaptable. In comparison, small weights exhibit greater plasticity, enabling efficient adaptation while preserving core model capabilities. This also re- lates to catastrophic forgetting: updating large weights risks overwriting pretrained knowledge, whereas small weights provide safer avenue for learning.
0.89
Small Weights
0.88
Large Gradients
2000
4000
6000
80d0
10000
Global Steps
(b) Small weights vs. large gradients.
Figure 4: Generalization performance of different masking strategies in NANOADAM using the same gradient density. (a) Small vs. large vs. random weights. (b) Small weights vs. large gra- dients.
masking achieves the best generalization performance.
Small weights vs. large gradients We further investigate whether small weights offer a more effective selection criterion than large gradients. Under the same experimental setup, we finetune the BERT- base model on SST-2 using dynamic masking interval of m 131 steps. Two selection strategies are compared: (1) parameters with large gradients and (2) parameters with small weights. parameters typically require higher learning rates for effective updates, we perform hyperparameter tuning for both strategies. The optimal learning rate 1 x 10-3 for small weights and 3 x 10-1 for large gradients. Notably, applying x 10-3 to the large-gradient strategy causes divergence. Figure 4b shows the evaluation results; training losses are included in Appendix F.2. Results show that updating small weights leads to faster convergence and better
Given that small-
7

---

generalization, supporting the view that they are more plastic and better suited for finetuning. A corresponding study the vision domain, provided in Appendix F.2, confirms these findings.
Experiments
We evaluate the effectiveness of NANOADAM both NI. and CV finetuning tasks. Our experiments
compare against several baselines, including AdamW [22]. AdamW-Sbit [7]. GaLore [44], and
MicroAdam [24]. For NLP, evaluate three language models of varying scale: BERT-Base (1 (100M parameters), BERT-Large (335M) [8], and OPT-1.3B [42]. These models are finetuned across multiple tasks from the GLUE benchmark. For CV, we examine two aspects: catastrophic forgetting and parameter shift. Specifically, we finetune ViT-Large [36], ResNet101, and ResNetl 8 [13], all pretrained on ImageNet [6]. Each model is first finetuned on CIFAR- 10 [18], followed by continued finetuning on the Flowers dataset [25]. Complete training configurations, hyperparameter details, and additional inclui--i the learning rate study-are are are provided in Appendix and Appendix
Finetuning on NLP tasks We first evaluate the effectiveness of NANOADAM on NLP finetuning tasks from the GLUE benchmark. Our experiments use Transformer models from the HuggingFace library, including BERT and OPT-1.3B. For performance evaluation, matched accuracy for MNLI, Matthew's 's correlation for CoLA, Pearson correlation for STS-B, -B, and classification accuracy for the remaining tasks. All optimizers evaluated under consistent training conditions, with the exception that the learning rate is individually tuned. Experiments are conducted on compute node equipped with 4xA 40GB GPUs. Memory usage reported the average across all GPUs. The overall performance results are summarized in Table and 3, while details on peak memory usage, training time, and training dynamics deferred to Appendix H.2.
standard metrics:
Table 2: Performance (eval metric) on GLUE dataset.
Model
Method
COLA
SST2
MRPC
STSB
QQP
MNLI
QNLI
AVG.
BERT-BASE
Microadam
60.26
92.89
83.82
88.72
90.63
84.04
91.18
84.50
NANOADAM
60.87
93.46
88.48
89.98
90.67
84.30
91.76
85.65
Galore
57.90
92.20
85.54
89.90
89.91
82.81
90.87
84.16
AdamW-8b
60.41
93.01
87.26
89.68
90.70
84.16
91.40
85.23
AdamW
59.65
93.23
87.01
87.90
89.66
83.29
91.31
84.58
OPT-133 BERT-Large
Microadam
62.55
94.04
89.22
89.68
90.45
85.67
92.04
86.24
NANOADAM
66.85
94.61
90.20
90.86
91.03
86.40
92.44
87.48
Galore
61.46
94.27
87.01
89.08
89.73
84.95
91.58
85.44
AdamW-8b
63.95
94.38
88.97
90.04
91.35
86.31
92.37
86.77
AdamW
61.53
94.15
86.03
89.74
90.05
86.09
92.18
85.68
Microadam
66.80
95.99
88.24
89.66
91.51
87.94
92.73
87.55
NANOADAM
67.69
96.45
87.99
91.00
91.33
88.24
92.75
87.92
Galore
65.88
96.10
86.03
90.86
90.80
87.89
92.72
87.18
AdamW-8b
66.36
95.87
86.28
90.36
91.57
87.20
92.79
87.20
AdamW
66.50
95.64
85.29
90.28
91.34
87.86
92.93
87.12
The results show that NANOADAM achieves lower memory usage than other memory-efficient optimizers, including AdamW-Sbit, GaLore, and MicroAdam, while also delivering superior gen- eralization performance. Notably, the memory savings scale with model size, in line with our theoretical analysis. Additionally, while methods like MieroAdam and Gal.oe suffer from signifi- cantly higher training time on larger models, NANOADAM maintains comparable runtime efficiency well-optimized basclines such AdamW and AdamW-8bil
Catastrophic forgetting on CV tasks We evaluate the catastrophic forgetting behavior of Adam W. Adam, and NANOADAM across several vision models, including ViT-Large, ResNer101, and ResNet18, 8, in continual learning setting. Each model is first finetuned on CIFAR-10 (Task 1) for a fixed number of epochs, followed by continued finetuning on Flowers 102 (Task 2). To assess forgetting, measure generalization performance (1) CIFAR-10 10 after Task 1, (2) Flowers102

---

Table 3: Average memory usage (GB) on GLUE dataset.
Model
MicroAdam
NanoAdam
GaLore
AdamW-8b
Adam
BERT-Base
3.71
3.58
4.04
3.72
3.94
BERT-Large
5.54
5.18
5,91
5.64
6.48
OPT-1.3B
13.18
11.60
14.16
13.08
18.16
after Task 2, and (3) CIFAR-IO again after Task 2. Forgetting is quantified as the drop in CIFAR-I0 accuracy before and after Task 2. Note that CIFAR- 10 and Flowers 102 differ in the number of classes. Thus, to evaluate CIFAR-I0 performance after Task 2, we reload the original classification head trained on Task 1. This isolates representational drift and allows us to assess the extent to which pretrained features are preserved. The resulting generalization performances are summarized in Table 4, with experimental details and visualisations of training dynamics provided in Appendix I. As shown, while all methods perform well on CIFAR-10 after the initial finetuning stage, most suffer substantial degradation after Task 2 2-indicating significant catastrophic forgetting. In contrast, for ResNet101, NANOADAM preserves high accuracy on CIFAR- (3.777) and also adapts well to Flowers 102 (81. (81.52%), outperforming AdamW on both tasks 229.59% and 52.97% respectively). These results suggest that NANOADAM achieves a more favorable trade-off between knowledge retention and task adaptation.
Table 4: Evaluation accuracy (%) across tasks for catastrophic forgetting.
Model
Method
CIFAR-10
Flowers102
CIFAR-10
Avg.
Forgetting
(Task 1)
(Task
(after Task 2)
MicroAdam
99.12
88.23
98.36
95.23
0.76
ViT-Large
NANOADAM
99.37
98.13
99.35
98.95
0.02
Adam W
99.3
92.51
98.61
96.81
0.69
MicroAdam
95,47
14.89
12.49
40.95
82.98
ResNet101
NANOADAM
96.32
81.52
83.77
87.20
12.55
Adam
97.72
52.97
29.59
60.09
68.1 68.14
MicroAdam
93.12
22.53
25.36
47.00
67.76
ResNet18
NANOADAM
92.52
70.67
64.17
75.79
28.36
Adam
95.63
56.58
27.59
59.93
68.05
Parameter shift analysis We further analyze the extent of parameter changes during finetuning under different optimizers. Specifically, we compute the & distance between the pretrained ViT-Large parameters and those obtained after continual finetuning on CIFARIO and Flowers, with MicroAdam, AdamW, and NANOADAM. The classification head is excluded from this analysis to isolate changes in the backbone. As in our toy example, Table summarizes the average l2 distance in parameter change alongside the average evaluation accuracy. Despite using the largest learning rate. NANOADAM induces the smallest parameter shift and achieves the best generalisation. More details are provided by visualizations in Appendix I.3.
Table 5: Averaged evaluation accuracy and parameter change in & distance, alongside learning rate.
Algorithm
LR (task1)
LR (task2)
AVG. Ace
lg Distance
AdamW
le-4
1e-4
96.81%
0.83
MdcmoAdam
le-4
le-3
95.23%
0.75
NanoAdam
le-3
2e-3
98.95%
0.68
5 Conclusions
We introduce NANOADAM, memory- and compute-efficient optimizer for finetuning large models. Motivated by consistent hyperbolic correlation between gradients and small weights observed

---

during finetuning, we propose to dynamically update parameters with small magnitudes instead of large gradients. Although this relationship less evident when training from scratch, proves highly effective in finetuning scenarios, where avoiding forgetting relevant concepts is paramount. Unlike prior methods, NANOADAM selects parameters without relying on gradient information, leading to improved generalization, less catastrophic forgetting, and reduced parameter drift. Experiments on both NLP and vision tasks show that NANOADAM matches or outperforms existing methods, offering new perspective on the role of small weights in efficient finetuning.
6 Limitations and broader implications
Our proposed method NanoAdam introduces minimal computational overhead. Specifically, for each weight matrix in each layer, we first flatten the matrix and divide it into subgroups (chunks), then apply bottom-k selection within each subgroup. This process is applied uniformly across both convolutional and MLP layers. The main computational cost arises from the bottom-k operation, which has a time complexity of O(k log k).
Thanks to its layer-wise and parameter-wise design, the method is naturally scalable to larger models and remains compatible with modern hardware acceleration and parallel training frameworks. avoids conflicts with model parallclism and pipeline layers, making it practical for contemporary large-scale architectures.
However, the method has several limitations. Its effectiveness relies heavily on knowledge transfer- ability and overparameterization. When the pretraining and finetuning tasks are well-aligned, the method helps avoid catastrophic forgetting and effectively leverages the plasticity of small weights to adapt to new tasks. In contrast, when there is limited similarity between tasks, the method may underperform compared to full-update optimizers like Adam. Moreover, the method benefits sig- nificantly from model overparameterization. As demonstrated in our experiments on vision tasks, scaling from smaller model (c.g.. ResNet-18) to larger one (c.g.. ViT-Large) results in improved overall performance and reduced forgetting. This suggests that Aanm the method is particularly well-suited for large, overparameterized models.
7
Acknowledgements
The authors gratefully acknowledge the Gauss Centre for Supercomputing e. (www. www.gauss-centre.cm for funding this project by providing computing time the GCS Supercomputer JUWELS [15] at Jülich Supercomputing Centre (JSC). We are also grateful for funding from the European Research Council (ERC) under the Horizon Europe Framework Programme (HORIZON) for proposal number 101116399 SPARSE-ML.
References
[1] Sanjeev Arora, Nadav Cohen, Wei Hu, and Yuping Luo. Implicit regularization in deep matrix factorization. In H. Wallach, H. Larochelle. A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[2] Dan Biderman, Jacob Portes, Jose Javier Gonzalez Ortiz, Mansheej Paul, Philip Greengard, Connor Jennings, Daniel King, Sam Havens, Vitaliy Chiley, Jonathan Frankle, Cody Blakeney, and John Patrick Cunningham. LoRA learns less and forgets less. Transactions on Machine Learning Research, 2024. Featured Certification.
[3] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 33:1877-1901, 2020.
[4] Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost. arXiv preprint arXiv: l00.:6604.000 74, 2016.
[5] Lenaic Chizat, Edouard Oyallon, and Francis Bach. On lazy training in differentiable program- 2020.
2020.
10

---

[6] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. 2009 IEEE conference on computer vision and pattern recognition, pages 248-255 leee, 2009.
[7] Tim Dettmers, Mike Lewis, Sam Shleifer, and Luke Zettlemoyer: 8-bit optimizers via block-wise quantization. arXiv preprint arXiv:21 10.02861, 2021.
[8] Jacob Devlin, Ming-Woi Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguisties: human language technologies, volume I (long and short papers), pages 4171-41866 2019.
[9] Advait Gadhikar and Rebekka Burkholz. Masks, signs, and learning rate rewinding. In Twelfth International Conference on Learning Representations, 2024.
[10] Wentao Guo, Jikai Long, Yimeng Zeng, Zirui Liu, Xinyu Yang, Yide Ran, Jacob R. Gardner, Osbert Bastani, Christopher De Sa, Xiaodong Yu, Beidi Chen, and Zhaozhuo Xu. Zeroth-order fine-tuning of LLMs with transferable static sparsity. In The Thirteenth International Conference on Learning Representations, 2025.
[11] Song Han, Huizi Mao, and William Dally. Deep compression: Compressing deep neural net- works with pruning, trained quantization and huffman coding. arXiv preprint 1rX.0:1510.0014,, 2015.
[12] Zeyu Han, Chao Gao, Jinyang Liu, Jeff Zhang, and Sai Qian Zhang. Parameter-efffcient finettnnin for large models: A comprehensive survey. arXiv preprint arXiv:2403. 2024.
[13] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[14] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. CoRR, abs/2 102.000099000 06.09685, 2021.
[15] Jülich Supercomputing Centre. JUWELS Cluster and Booster: Exascale Pathfinder with Modular Supercomputing Architecture at Juelich Supercomputing Centre. Journal of large- scale research facilities, 7(A138), 2021.
[16] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 11X1v:1412.6980, 2014.
[17] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521-3526 2017.
et al.
[18] Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
[19] Baohao Liao, Yan Meng, and Christof Monz. earfmeter-efficient fine-tuning without introducing new latency. arXiv preprint 21777:2305.16742. 2023.
[20] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412. 19437, 2024.
[21] Yixin Liu, Kai Zhang, Yuan Li, Zhiling Yan, Chujie Gao, Ruoxi Chen, Zhengqing Yuan, Yue Huang, Hanchi Sun, Jianfeng Gao, et Sora: review on background, technology, limitations, and opportunities of large vision models. arXiv preprint
[22] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, arXiv preprint axXiv11711.51001, 2017.
11

---

[23] Eran Malach, Gilad Yehudai, Shai Shalev-Schwartz, and Ohad Shamir. Proving the lottery ticket hypothesis: Pruning is all you need. In International Conference on Machine Learning. 2020.
[24] Ionut- Vlad Modoranu, Mher Safaryan, Grigory Malinovsky, Eldar Kurtić, Thomas Robert, Peter Richtarik, and Dan Alistarh. Microadam: Accurate adaptive optimization with low space overhead and provable convergence. Advances in Neural Information Processing Systems, 37:1-43, 2024.
[25] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In Indian Conference Computer Vision, Graphics and Image Processing, Dec 2008.
[26] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018.
[27] Amrutha Varshini Ramesh, Vignesh Ganapathiraman, Issam H Laradji, and Mark Schmidt. Blockllm: Memory-efficient adaptation of Ilms by selecting and optimizing the right coordinate blocks. arXiv preprint 2tt1:2200.112996. 2024.
[28] Minsoo Rhu, Natalia Gimelshein, Jason Clemons. Arslan Zulfiqar, and Stephen W Keekler. vdnn: Virtualized deep neural networks for scalable, memory-efficient neural network design. In 2016 49th Annual IEEF/ACM International Symposium on Microarchitecture (MICRO), pages 1-13. IEEE, 2016.
[29] Jesus Rios, Pierre Dognin, Ronny Luss, and Karthikeyan N Ramamurthy. Sparsity may be all you need: Sparse random parameter adaptation. arXiv preprint arXiv:2502.15975 2025.
[30] Thomas Robert, Mher Safaryan, Ionut-Viac Modoranu, and Dan Alistarh. Ldadam: Adaptive optimization from low-dimensional gradient statistics. arXiv preprint arXiv:2410.16103, 2024.
[31] Felix Stahlberg, Jared Lichtarge, and Shankar Kumar. Dynamic subset tuning: Expanding the operational range of parameter-efficient training for large language models. arXiv preprins arXiv:2411.088 2024.
[32] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Ann Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13977, 2023.
[33] Zhongwei Wan, Xin Wang, Che Liu, Samiul Alam, Yu Zheng, Jiachen Liu, Zhongnan Qu, Shen Yan, Yi Zhu, Quanlu Zhang, et al. Efficient large language models: A survey, arXiv preprint arXiv:2312.03886 1333863, 2023.
[34] Luping Wang, Sheng Chen, Linnan Jiang, Shu Pan, Runze Cai, Sen Yang, and Fei Yang. Parameter-efficient fine-tuning in large language models: a survey of methodologies. Artificial Intelligence Review, 58(8):227, 2025.
[35] Yiming Wang. Yu Lin, Xiaodong Zeng, and Guannan Zhang. Multilora: Democratizing lora for better musk learning. ArXiv, abs/2311. 1.5501, 2023.
[36] Bichen Wu, Chenfeng Xu, Xiaoliang Dai, Alvin Wan, Peizhao Zhang, Zhicheng Yan, Masayoshi Tomizuka, Joseph Gonzalez, Kurt Keutzer, and Peter Vajda. Visual transformers: Token-based image representation and processing for computer vision, 2020.
[37] Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, et al. Autogen: Enabling next-gen 11m applications via multi-agent conversation. arXiv preprint arXiv:2308.08155 2023.
[38] Xun Wu, Shaohan Huang, and Furu Wei. Mixture of loRA experts. In The Twelfth International Conference on Learning Representations, 2024.
[39] Yuhui Xu, Lingxi Xie, Xiaotao Gu, Xin Chen, Heng Chang, Hengheng Zhang, Zhengsu Chen, XIAOPENG ZHANG, and Qi Tian. QA-IoRA: Quantization-aware low-rank adaptation of large language models. In The Twelfth International Conference on Learning Representations, 2024.
12

---

[40] Shih yang Liu, Chien-Y Yi Wang, Hongxu Yin. Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Timg Cheng, and Min- Min-Hung Chen. DoRA: Weight-decomposed low-rank adaptation. In Forty-first International Conference on Machine Learning, 2024.
[41] Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu. Vision- laiiuag models for vision tasks: A survey. IEEE Transactions on Pattern Analysis and Machine Inielligence, 2024.
[42] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li. Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022.
[43]
Zhihua Zhang. The singular value decomposition, applications and beyond, 2015.
[44] Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. Galore: Memory-effieient IIm training by gradient low-rank projection. arXiv preprini arXiv:2403.03507, 2024.
NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes. See Theorem 2.2 and Analysis in appendix G. We also provide experi- ments to validate them.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss the limitation in the Section 2.2, where we clarify that small weights can not fully recover the large graidents. We also discuss that the proposed method is designed for finetuning task, for training from scratch.
Guidelines:
The answer NA means that paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sell-specification asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
13

---

The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't It acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proof's
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification:
We provide proof under the theorem 2.2.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Yes. We provide all experiment details in appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
14

---

While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c)
If the contribution is new model (e.g., , a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification:
We provide the code in zip file.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm
cablic/guides wttps//oooSuuuisiiioooooo ooiooiic)yyy for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / //nips. cc/ccc/ppuoo wwtpcccccp////e/..c...cccc.ttttt httpllicgguiese//eee SooeeeemmoodSSmmmmmo iemiiissionii ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide full hyperparameters and experimental settings in appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
15

---

The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question:
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We provide error bars in some toy experiments, while other computationally intensive experiments conducted under the same random seed. We also test our method across different tasks, from NLP to CV.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: The full settings for compute workers, memory, time of execution are reported in experiment sections and appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: /ttpps:://///iss.../. wwtpc//cccp...cccccoocccccccccee
Answer: [Yes]
Justification: Yes.
16

---

Guidelines:
The code is followed.
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The paper is studying an efficient optimiser.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g.. deploymeet of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper does not propose new models or data.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
17

---

Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All the assets are properly credited and properly respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. If this information is available online, the authors are encouraged to reach out the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not introduce assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our research does not include human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
18

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our research does not include human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: LLMs have not been used in the methods of this paper.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our L.LM policy (https / http///rppppcccc/oeeeeeeeeeeeeee for what should or should not be described.
19

---

A Contribution related to literature
Our work is different from [19] from the following aspects:
1. While [19] hypothesizes that updating only small-magnitude weights can be effective, we provide an explanation: large gradients are usually associated with small weights during finetuning, making them more adaptable. We show that the srudient-weight correlation much stronger during fine-tuning than training from scratch, explaining why the strategy is less effective in the latter case.
2. We attribute this pattern to knowledge transferability and overparameteriiation, offering a principled understanding of when wnight finetuning is most effective.
3. We further show that updating small weights helps mitigate catastrophic forgetting, supported by both theoretical and empirical evidence.
4. Our method is novel in that: (a) mask selection and (b) sparsity are dynamic, and (c) selection is done per layer rather than globally. Unlike [18], which uses a fixed global mask, our dynamic approach improves efficiency and performance while reducing memory usage. Dynamic masking specifically improves the implicit weight regularization by discouraging over-re over-reliance on fixed subset of parameters; and mitigates catastrophic forgetting by reducing parameter shift.
5. Lastly, unlike [18] which focuses only on NLP, we evaluate our method on CV tasks as well, demonstrating broader applicability.
B Details on studying the relationship between gradients and weights in Section 2.1
The training details used are summarised in Table 6. In Figure 5, we illustrate how the relationship between weights and gradients evolves for three representative components: the positional embedding weights, the value weight matrix in layer 6, and the weight matrix in the final classification layer, corresponding to progressively deeper layers in the network.
As shown in the figure, parameters with large gradients typically correspond to those with small weight magnitudes (except in the case of the final classification layer as shown in the appendix). There are several possible explanations for the distinct behaviour observed in the final classifier layer. First, the classifier is newly initialised from scratch, rather than being inherited from pretrained weights. Second, it needs to adapt to the task-specific label space. Consequently, in our algorithm, we exclude the final layer, along with normalisation layers, from selective updates. In contrast, for the other layers, the strong correlation between gradient magnitude and weight magnitude remains significant. This finding also supports the observation made [27], where models were found to frequently update parameters with smaller weight magnitudes. However, in their work, this phenomenon was further explored; instead, they adhered prior practice by using gradients as the primary criterion for parameter importance.
Table Hyperparameters for studying the gradient-weight relationships.
Setting
LR
WD
Batch size
Epoch
Label Smooth
CoLA (BERT-base, FT)
7e-5
0.0
32
5
CIFARIO (VIT, FT)
3e-3
0.1
128
300
0.1
CIFARIO (VIT, Scratch)
3e-3
0.1
128
300
0.1
C Small weights and large updates
To better understand the actual parameter changes during fine-tuning, we conducted an additional experiment: we fine-tuned a BERT-base model on the CoLAA task from the GLUE benchmark using AdamW for epochs.
20

---

0.0010
0.03
0.020
0.015
0.0005
Gradierds
Gradlents
0.008
0. .00000
0.00
0.000
0.,0005
-0.01
5.000
-0. 0000
0.015
-1.0
-0.5
0.0
0.5
-1.0
-0.5
0.0
0.5
1.0
-05
2.0
0.5
Wrights
Weights
Weights
(a) positional embedding weight, early of FT.
(b) positional embedding weight, middle of FT.
(c) positional embedding weight, end of FT.
0. 0.003
0.020
8.010
0.010
0.005
Godderts
0.001
0.005
0.000
0.000
-0.001 000
-0.003
000
-0.010
-0.01.
-02
-011
0.0
0.1
-02
-0.1
0.0
031
0.2
-02
-03
0.0
53
02
Weighs
Weights
Weightt
(d) value weight in layer 6, early of FT.
(c) value weight in layer 6, middle of FT.
(f) value weight layer 6,
PT.
0.066
Gradiens
Gradionts
0.030
0.000
-0.022
-0.96
0.04
-0.036
-0.056
-0. 0.0252-0.025 5.000 3,025 5.050 0.075
-0150-00 0.025.3.03
0.025 3,50 3.075
33055
0.025 0.000 5,025 0.350 0.055
Weights
Weeight
Weights
(g) classifier weight, carly FT.
(h) classifier weight middle of FT.
(i) classifier weight, of FT.
Figure 5: The dynamic of the relationship between gradients and weights during finetuning Bert-base on COLA. The x-axis represents the magnitude of the weights, while the y-axis represents the magnitude of the gradients. From left to right, the subfigures correspond to the early, middle, and late stages finetuning. From top to bottom, the subfigures represent progressively deeper layers in the network.
21

---

For each layer, we partitioned the parameters into three groups based on their absolute magnitudes in the pretrained model:
1. The bottom 30% (smallest magnitudes);
2. The top 30% (largest magnitudes);
3. The remaining middle 40%.
We then measured the average change in weights within each group, calculated as the difference between the pretrained and fine-tuned weights. This analysis allows us to examine which groups of parameters receive the largest updates during fine-tuning.
The results, presented in the table 7 below, consistently show that across all layers, the smallest- magniuude weights undergo the largest updates, while the largest-magniude weights change the least. This trend holds regardless of the layer's scale or functional role in the Transformer architecture.
Table 7: Per-layer parameter shift for the 30% smallest, 30% largest, and remaining subsets.
Layer
30% Smallest
30% Largest
Remaining
Layer 0
0.000799
0.000381
0.000393
Layer 1
0.000815
0.000325
0.000363
Layer 2
0.000787
0.000331
0.000344
Layer 3
0.000812
0.000302
0.000353
Layer 4
0.000846
0.000298
0.000354
Layer 5
0.000877
0.000282
0.000351
Layer 6
0.000821
0.000294
0.000342
Layer 7
0.000798
0.000299
0.000341
Layer 8
0.000727
0.000276
0.000321
Layer 9
0.000691
0.000278
0.000308
Layer 10
0.000669
0,000280
0.000302
Layer 11
0.000742
0.000279
0.000324
D Overparameterization leads to stronger correlation
We provide additional visualisation of gradient-weights distribution in FT ViT-Tiny and ViT-Large models on CIFARIO in Figure
0.2
0.06
0.1
Gradients
0.02
Gradients
0.0
-0.02
-0.1
-0044
-0.
-0.06
-0.4
-0.2
0.0
0.2
0.4
-0.50
-0.55
0.00
0.25
0.50
Weights
Weights
vit-tiny
(b) vit-large.
Figure 6: Gradients and weights distribution of first layer in FT ViT-Tiny and ViT-Large at early stage of FT. Overparmeterisstioon leads more hyperbolic relationship.
22

---

E
Additional theoretical statements and motivation
We provide more details regarding the two layers neural network setup. For pretraining, the data is generated from a similar structured teacher network Seaaceer with n = 2 neurons and input dimension ==2 2. Furthermore, all neurons are first initialized variables and wi.j ~ (0,, 1) for all i, E 1K, d] and normalized over the input dimension. We initialize the dense network with k 20 and the so-called COB initialization based on the rich regime in [5]. All weights are initialized with A(0, 1/n) and we ensure that a; = a2+10 for i E [10]. We train for = 10000 steps with learning rate 77 = 2. This training gives us the pretrained network fre. For the fine tuning we generate additional neurons in the same way as the teacher neuron. We train for T 10000 steps with learning rate v = A slightly smaller learning rate has been chosen to ensure convergence for the large gradient setup.
Uni( -1, 13) i.e. i.idd Rademacher random
Gradient flow
The gradient flow of a two layer neural network for training one neuron is given by
da, - Eg a (wt ax) (ara (ww n.) v/i) dt, dw E, auaiiiafac (ata (wfan 21) - ys) dt,
ag
Oinit Wig Wint::
As highlighted the study of nano gradient flow can be reduced to studying the case of training one neuron where the labels are generated by teacher neuron fextra This is under the assumption that the chosen neuron is not contributing to the representation. In our main statement we simplify even further and train one neuron with a frozen as in the next statement Theorem E.
Theorem E.1 Assume a model (z) consisting of n neurons learns the teacher fieacher cor- responding a pre-truining task so that f(x)) = Smacher(a)) for all x E Rd, Furthermore, let [2] |a,)), S for an e > and Jteacher + Sexar with an extra neuron fexma= = äo (o( Let only the neuron j of f be trainable to finetune f(x) to the new task. where j = argmin [a]]|w//l sign(a.) sign(ā)). }. Then, the gradient flow with respect to finetuning time of the neuron j. which is parameterized as U₁,t = |a5,e|W5, and initialized at the pre-trained values V3,0 |ay (wy, converges to value Vsa that ||100 olle, Cc, where v is the target u = |ā|ū and C > 0 a data dependent constant.
(x)) consist of at least two neurons i.r E [n] such that max maxx(la)] sign(a.) + sign(a,). Let a new task be defined based on labels Jfinetuee
Proof. To apply Theorem 6.4 in [23], need matching signs for the parameters Otherwise, have an immediate mismatch between the two single neuron functions. Assuming matching signs, we can absorb 0min and à into the activation to simplify the analysis. This reduces the problem to optimizing a single layer neuron (wx.) with initialization layl ko to learn a target vector e |ā with some small label perturbation that is equal to e. Without loss of generality assume the sign of a is positive. Then setting is reduced to training one neuron with gradient flow:
- du, = - n2 Σ
de 2 (a (v..2)) (vx;) + Bex)) dl,
where |Be.il < Cie is a small perturbation incurred from the teacher and C > 0 is data dependent constant. We can characterize the minumum associated with using perturbation theory i.e. can linearize around v and e 0 the right hand side of the gradient flow equation:
1 n2 Σ
+ +C=
giving us v/ = H-' where H is the Hessian data covariance matrix and C e RRd depends on all Be, for i E [n]. This leads to a bound for the perturbed equilibrium v*:
||o* - =1|12 S ||D'
Denote the process ún as the gradient flow without perturbation. It follows directly from Theorem 6.4 in [23] that & It remains to be shown that é is close to v, during the gradient flow. At initialization, we have 0g = 0g. We can bound the evolution with 2, := U2- 02:
d||zalle -
2r be
21 )t
1| "|zellez
<~~l112llle + b) dt.
23

---

where A, = - Jo ((ec-(1-)(th - 2 and b, = -. TiInm S eC1, / E; ||, =: b. Note under the assumptions that the data is spherically distributed as in Theorem 6.4 in [23] there is some data depending constant À such that 2A₁² 2 >1|21|e, for all f 0 with high probability (sufficient data samples). This relies on the fact that A, is positive semi definite and that A, = iff 24 = O. Then by Gronwall's lemma we have ||ze||z, I 5 S Ce. Therefore, for sufficiently small the trajectories stay close to each other. This implies the perturbed gradient flow enters the region of e leading to convergence to the nearby stationary point v². 0
Note that the assumption max( |a,12) S in the above theorem is justified because f learns a representation of fuacher (x) when solving pre-training task where at least one of the neurons is effectively pushed to 0 (as the teacher consists also of fewer neurons than the trained model f). Another way of dealing with the perturbation to assume that there exist two neurons with the same and opposite signs. These two neurons would not contribute to the representation they cancel each other out, thus no perturbation incurred. Then we can remove the neuron that does not match the sign of à and train with the other. This allows for immediate application of Theorem 6.4 [23]. In our toy setup, we train with both as we do not assume to know the correct sign of the added neuron.
Task difficulty measure for two-layer network The theoretical measure of how difficult our post training task is is captured by the distance in the function space L2 L2( p) between reference task and final representation f. Concretely the measure defined as
|lf 115 - E aaooua x) - Eaj(wf dp(x)
(f. A
Anok
where p is probability measure the data space. The distance measure can be approximated by the use of the empirical measure and or an upper bound solely depending on the weight space. In the main text assumed that we learned the representation fer and that the finetuning task is given by fn = Fpre + fextra
Lemma E.2 Denote the weights a; E RR and wi E Rdd for i E [m]] of fextra and is a Gaussian with
mean μ and covariance matrix E Then
Arase Saa Spre) <
Σ lag 2 + ||w/|12
Proof. (1) Apply the triangle inequality neuron wise (since we have learned the teacher representation. (2) Gaussian integral calculations for ReLU activation. (3) Apply Cauchy-Schwarz, inequality to each a;} /wsl-
Lemma E.2 substantiates the use of the lg distance in our toy example. Note the bound is tight in the balanced case.
Single neuron fine turning We report in Table 8 here the test loss and the distance traveled in lz for the experiments Figure 3.
Table 8: Average test loss on finetuning task and distance from pretrained initialization over 10 seeds. Small weights leads to better generalization and move less from the original representation.
Algorithm
Test Loss
Eg Distance
Small Weights
0.0094 + 0.012
0.027 + 0.0040
Large Gradient
0.017 + 0.015
0.057 + 0.032
More neurons fine turning We repeat the same experiment as in the main text but with an additional neuron. In Table 9 we observe that the variance for the distance by selecting the large gradients becomes high. This in line what observed in Figure 7c where we learn complete new representation.
24

---

+ sign
sign
Init
Final
(a) Dense feature lcarning.
(b) Nano GD.
(c) Largest gradients.
Figure 7: nano gradient descent provably prevents catastrophic forgetting. (a) Two layer student network learns teach networks representation. (b) Nano gradient descent keeps the original represen- tation while learning the two extra neuron.
(c) The largest gradients can lead to learning completely
different representations when the task transferability is high.
Table 9:
Average test loss finetuning task and distance from pretrained initialization over 10 seeds. Small weights leads to better generalization and move less from the original representation even when more difficult representation needs to be learned i.e. less transferable tasks.
Algorithm
Test Loss
Eg Distance
Small Weights
0.038 + 0.040
0.042 0.0077
Large Gradient
0.063 + 0.033
0.097 + 0.071
F Ablation study
F1
Small vs. large vs. random weights
For all configurations, we use a learning rate of 9 x 10-5, weight decay of 0.0, batch size of 32, 5 training epochs, and a fixed random seed of 42. The mask density is initialized at ko 0.01, the mask update interval is set to m = 131, and density scheduler is disabled for this experiment. The training loss and evaluation metrics over steps are shown in Figure 8.
0.9
0.6
Evaluation Accuracy
Training Loss
0.8
Small Weights
0.4
0.7
Large Weights
Random Weights
0.2
Small Weights
Large Weights
0.6
Random Weights
0.0
0.5
2000
4000
6000
8000
10000
2000
4000
6000
8000
10000
Global Steps
Global Steps
(a) Train loss.
(b) Evaluation metrics.
Figure 8: Ablation study comparing three masking strategies in NANOADAM: small-magnitude weights, trrge-magnitude weights, and random weights. Small- masking achieves the best training loss and evaluation performance under the same gradient density.
25

---

F.2 Small weights vs. large gradients
Study in NLP domain We conduct an ablation study comparing two parameter selection strategies: (I) selecting parameters with the smallest absolute weight magnitudes. and (2) selecting parameters with the largest absolute gradient magnitudes. All experimental configurations follow the setup described in Appendix F.1,, with the exception that the learning rate is separately tuned for each strategy to their best performance. The optimal learning rate is x 10-3 for small weights and 3 x 10-4 for large gradients. The corresponding training loss and generalization performance visualized Figure 9.
Small Weights
Large Gradients
Evaluation Accuracy
0.922
Training Loss
0.3
0.90
0.89
Small Welghts
0.1
0.88
Large Gradlents
2000
4000
6000
8000
10000
2000
4000
6000
8000
10000
Global Steps
Global Steps
(a) Train loss.
(b) Evaluation metrics.
Figure 9: Ablation study comparing two selection criteria: small weights and large gradients. Small weight is better at generalization and convergence.
Study in vision domain We also provide ablation study FT CV tasks, where compare NANOADAM under two different masking strategies: (1) Large gradients: selecting parameters with the largest absolute gradient magnitudes. (2) Small weights: selecting parameters with the smallest absolute weight magnitudes. Specifically, we finetune the ViT-Large model on the Flowers 102 dataset, using the same hyperparameter settings detailed in Table 10. We initialize the mask density at Ko = 0.001, and turn the density scheduler off. We set the mask update interval to m = 100. The resulting training loss and evaluation accuracy are presented in Figure 10. Note that in this experiment, we do not perform learning rate search for both strategies. Instead, we keep the same learning for both cases. Although the final performance is similar with both strategies, small weights achieve faster convergence.
Table 10: Hyperparameters for fully finetuning ViT-Large on Flowers 102.
LR
weight decay
batch size
epoch
seed
3e 3
0.0
128
10
42
F.3 Dynamic mask vs. static mask
To evaluate the effectiveness of dynamic masking, conduct experiments similar the previous setup, with the key difference being the masking strategy used NANOADAM, Specifically, for all configurations, we use a learning rate of 9 10-5, 5, weight decay of 0.0, a batch size of 32, training epochs, and fixed random seed of 42. The mask density is initialized at ko = 0.01, and the density scheduler disabled for this experiment. We compare two approaches: (1) Dynamic masking, where the mask is updated every m = 131 steps; and (2) Static masking, where a fixed mask from the beginning applied throughout the entire training process. The resulting training loss and evaluation accuracy are shown in Figure 11. While the static mask achieves very similar evaluation performance during the initial phase training, the dynamic mask continues to improve and ultimately surpasses the static strategy in both evaluation accuracy and final training loss. These results indicate that dynamic masking allows the model adapt more effectively throughout training. leading to better convergence and generalization.
26

---

Small Weights Large Gradients
1.0
Evaluation Accuracy
Training Loss
0.8
0.6
0.4
0.2
Weights
Large Gradients
8
Global Steps
Global Steps
(a) Train loss.
(b) Evaluation metrics.
Figure 10: Ablation study on Vir-Large finetuned the Flowers 102 dataset comparing two selection criteria: small weights and large gradients. Small weight is better at generalization and convergence.
0.90
0.6
Dynamic Mask
Static Mask
Evaluation Accuracy
Training Loss
0.85
0.80
0.3
Dynamic Mask
Static Mask
0.2
0.75
2000
4000
6000
8000
10000
2000
4000
6000
8000
10000
Global Steps
Global Steps
(a) Train loss.
(b) Evaluation metrics.
Figure 11: Ablation study on BERT-base finetuned on the SST2 task comparing two masking strate- gies in NANOADAM: dynamic mask and static mask. Dynamic mask achieves the best performance.
F,4 Sensitivity Analysis
conduct sensitivity analysis by fine-tuning the BERT-base model on the QNLI task from the GLUE benchmark using batch size of 32 and learning rate of 1.le - 4.
To the impact of the mask interval parameter m, disable the density scheduler and vary m within the range [10, 1500]. The corresponding evaluation performance is presented in the table 11 below. Our results show that model performance remains largely stable across this range. with optimal results observed when m falls between 80 and 300. Given that the total number of steps in one epoch 3274 in our experiment, this tuning range corresponds approximately 0.0245 0.095.
Table 11: Evaluation accuracy (%) vs. 771.
10
50
80
100
300
500
700
900
1100
1500
Eval (%)
90.72
90.85
90.99
91.21
91.21
91.09
91.09
91.16
91.03
90.98
For the sensitivity analysis of the density interval fix the mask interval m and vary d within the range [100, 1000]. The resulting evaluation performance is presented in the table below. We observe that when exceeds 300, the performance remains largely stable, with best performance achieved at d = 400. Given that the total number of steps in one epoch is S = 3274, this corresponds to density interval greater than approximately (300/3274) 0.092S, with the optimal interval around (400/3274) 0.122S.
27

---

Table 12: Evaluation accuracy (%) vs. d.
d
100
200
300
400
500
600
700
800
900
1000
Eval (%)
90.94
90.68
91.34
91.58
91.32
91.27
91.23
91.29
91.43
91.21
G
Theoretical memory and computation overhead comparison
We now compare the theoretical memory footprint of NANOADAM with other optimizers, including AdamW, W. AdamW-aa W-8bit, and MicroAdam, focusing specifically the optimizer state memory. Fol- lowing the setup in [24], we assume the total number of model parameters is denoted by S. We use the LLaMA-2 7B model as concrete example to illustrate memory consumption.
NanoAdam. NANOADAM stores the optimizer states m and (first- and second-order momentums) only for the selected subset of parameters in bfloa116 format, with each requiring 2 Bytes. Given a gradient density k, the number of updated parameters is KS. Therefore, the total memory required for the momentums is: 2 (states) x 2 (Bytes) x AS = 4&S Bytes. Additionally, NANOADAM stores the mask / indicating the indices of selected parameters. Simply using 64-bit integers (long), each index takes 8 Bytes, resulting in: 8 x KS = 8kS Bytes. Alternatively, if we store indices in int16 format (2 Bytes), as done in MicroAdam, the total memory reduces to: 4&S 2kS 6kS Bytes. For finetuning L- 7B (S = 6.275 B) with k = 0.01, NANOADAM requires x 0.015 = 0.3765 GB.
Adam W. Stores both v and for all parameters in bfloat1, requiring 2 2 x 5 = 4,5 Bytes. For LLaMA-2 7B, this equals 4$ 25.1 GB.
--bit. Stores m and in 8-bit precision, needing 2x1x5 S 2S Bytes, which corresponds to 25 = 12.55 GB for LLaMA-2 7B.
MicroAdam. Stores 4-bit quantized error (0.55 Bytes) and a sliding window of size m for both parameter indices in int16 and values in bf Each step in the window stores k.S parameters, leading to 0.55+4mk8 Bytes. For k = 0.01, m = 10, MicroAdam requires 0.5S 44 x.00. = 5.65 GB fientune LLaMA-2 7B.
In terms of computation, the bottom-k operation does not introduce significant overhead. In our implementation, bottom-k selection is performed independently layer (excluding the final output layer), rather than applied globally. Specifically, it proceeds as follows: For each weight matrix, first flatten it and divide it into subgroups of parameters (chunks). We then apply the bottom-k selection within each subgroup. As bottom-k operation has a time complexity of Okk log k), which remains computationally feasible in practice.
each parameter matrix in each
H Details and more results for finetuning on GLUE benchmark
H.1 Hyperparameters for finetuning on GLUE benchmark
We largely follow the hyperparameter settings established by [24] for finetuning various GLUE tasks. Specifically, we finetune for 5 epochs with a per-device batch size of 8, a fixed random seed of 42, and no weight decay. Unless otherwise stated, perform grid search the learning rate values {1e-6, 6, 3e 6, 5e 6, 7 6, le 5, 3e,35,, 5, 5e-5, 7e 7- for all optimizers and models.
For GaLore, we set the low-rank approximation rank to r = 256 and vary the SVD update interval T {20, 200}. In contrast to the original GaLore implementation, which tunes both the scale and lcarning rate, we fix the scale to I and augment the learning rate search space with 4e -e- 5,-4, 7c-4).
GaLore.
MicroAdam. For MicmoAdam, we use sliding window of m = 10 gradients and a sparsity level of l: = 1%, resulting in an effective gradient sparsity of mk = 10%. The quantization bucket size set to 64.
28

---

Adam and Adam-Sbit. All general hyperparameter settings mentioned above are directly applied both Adam and Adam-8bir baselines.
NanoAdam. For NANOADAM, we the initial update density ko 10% and linearly decay to 4% by the end of training. As training small weights typically requires a larger learning rate, we
search over a wider range: [5e-6, 6, 3e-4]. Additional hyperparameters specific to NANOADAM are
summarized in Table 13.
Table 13: Hyperparameters for NANOADAM across tasks.
task
COLA
SST2
MRPC
STSB
QQP
MNLI
QNLI
mask interval
52
7
13
711
306
81
density interval
33
263
14
27
1423
1533
409
H.2 Additional results for finetuning on GLUE benchmark
The peak memory usage and running time are reported in Table. and 16. also compare with Nanoadam using the static mask strategy, which is shown in Table. 14 to illustrate the benefits of dynamic masking.
Table 14: Performance (eval metric) GLUE dataset, comparing static mask strategy with
Admo.dam.
Model
Method
COLA
SST2
MRPC
STSB
QQP
MNLI
QNLI
AVG.
BERT
NANOADAM
60.87
93.46
88.48
89.98
90.67
84.30
91.76
85,65
BASE
NanoAdam(static mask)
56.24
91.51
84.07
89.68
89.75
82.59
90.92
83.54
BERT
NANOADAM
66.85
94.61
90.20
90.86
91.03
86.40
92.44
87.48
LARGE
NanoAdam(static mask)
59.07
92.66
84.31
89.86
90.33
85.39
91.78
84.77
OPT
NANOADAM
67.69 60,38
96.45
87.99
91.33
88.24
92.75
87.92
-1.3B
NanoAdam(ssattic mask)
95.30
86.03
90.55
91.07
87.61
91.91
86.12
Table 15: Memory usage (GB) on GLUE dataset.
Model
Method
COLA
SST2
MRPC
STSB
QQP
MNLI
QNLI
AVG.
Microadam
3,64
3.63
3.64
3.64
3.81
3.75
3.79
3.70
BERT BBAE
NAMOADAM
3.58
3.60
3.59
3.57
3.60
3.60
3.59
3.59
Galore
4.06
4.05
4.06
4.06
4.05
4.05
4.05
4.06
AdamW-8b
3.72
3.72
3.72
3.72
3.72
3.72
3.72
3.72
AdamW
3.94
3.94
3.95
3.95
3.94
3.93
3.95
3.94
Microadam
5,56
5.53
5.52
5.54
5.54
5.53
5.54
5.54
BERT LARGE
NANOADAM
5,21
5.24
5.15
5.19
5.23
5.22
5.19
5.20
Galore
6.12
5.60
6.11
6.10
5.90
5.89
5.90
5,94
AdamW-8h
5.61
5.83
5.62
5.61
5.61
5.62
5.60
5.64
Adam
6.45
6.52
6.47
6.47
6.52
6.47
6.46
6.48
Microadam
13.20
13.15
13.19
13.19
13.19
13.19
13.20
13.19
OPT -1.3B
NANOADAM
11.33
11.76
11.41
12.04
11.66
11.77
11.57
11.65
Galore
14.18
14.18
14.39
14.26
14.18
14.18
14.17
14.22
AdamW-8b
13.07
13.08
13.08
13.08
13.08
13.08
13.08
13.08
AdamW
18.18
18.16
18.16
18.16
18.17
18.17
18.16
18.16
29

---

Table 16: Training time (minutes) on GLUE dataset.
Model
Method
COLA
SST2
MRPC
STSB
QQP
MNLI
QNLI
AVG.
Microadam
3.25
17.79
1.78
6.26
98.79
103.42
28.41
37.10
NANOADAM
2.99
16.57
1.46
2.09
91.35
94.05
26.72
33.60
BERT -BASE
Aamattattta mask)
2.78
14.56
1.55
2.10
88.57
84.02
23.75
31.05
Galore
2.01
12.16
0.86
72.65
69.99
19.08
25.47
AdamW-8h
1.40
8.04
0.66
1.09
57.95
45.96
15.81
18.70
AdamW
1.13
7.38
0.55
0.88
43.63
39.14
10.93
14.80
Microadam
7.26
37.30
2.85
4.47
170.67
175.73
54.80
64.73
NANOADAM
4.36
28.99
2.33
3.23
157.40
164.44
47.97
58.39
BERT LARGE
Asatttttit mask)
4.71
25.01
2.61
3.49
146.48
150.09
39.59
53.14
Galore
4,36
26.72
1.87
3.27
162.00
160.34
44.90
57.64
AdamW-8b
3.88
18.85
1.08
1.79
96.64
93,20
25.38
34.40
AdamW
2.04
12.91
0.93
1.56
78.37
78.07
21.16
27.86
Microadam
9.17
46.14
5,55
7.10
238.46
253.18
70,35
89 .9.9
NANOADAM
4.95
37.12
2.54
4.23
186.92
196.63
53.74
69.45
OPT
Atamttttiii mask)
7.10
30.72
4.73
8.52
171.66
167.67
47.67
62.58
-1.3B
Galore
12.50
84.24
4.85
8.37
451.76
463.27
129.09
164.87
AdamW-8h
3.91
29.39
1.74
2.88
160.03
158.81
44.31
57.30
AdamW
3.38
27.73
4.27
2.53
138.25
135.58
38.54
50.04
H.3 Training Dynamics for finetuning NLP task
We also present the training dynamics observed in NLP finetuning task. Specifically, we analyze the training loss and generalization performance of various optimizers when finetuning BERT-base model on the QNLI task from the GLUE benchmark. For each optimizer, we perform hyperparameter tuning to determine the optimal learning rate. The experiment settings are the same as those described in Appendix H. The best learning rates for each method are summarized in Table. 17.
The resulting training loss and evaluation accuracy over time are shown in Figure 12a and 12b. As expected, in the early training steps, full finetuning with AdamW achieves the lowest training loss, since it updates the entire parameter set. However, at later stages, NANOADAM surpasses Adam W in both training loss and generalization. This is because updating only small-magnitude weights initially has minimal impact on the model output-dominated by large weights- but gradually exerts greater influence as training progresses.
In terms of generalization performance, AdamW performs better in the initial steps but soon over- taken by NANOADAM, which consistently achieves higher accuracy in the later stages. Furthermore, across all training steps, NANOADAM outperforms AdamoAAam, demonstrating its superior learning dynamics.
also visualize dynamics of the ratio between the number parameters updated least once and the total number of parameters during finetuning the CoLA task using BERT-base. The results are shown Figure 12c. Note that optimizers such as GaLore, AdamW, and d-Shit update all parameters by design; thus, their curves omitted for clarity. As shown, MicroAdam eventually updates over 90% the parameters, whereas NANOADAM keeps more than 80% of parameters untouched throughout training.
Table 17: learning for various optimiser on finetuning Bert-base QNLI. optimiser
Adam
AdamW-8b
Galore
MicroAdam
NANOADAM
LR
7e
7e
5
1e
4e
1.1e
30

---

AdamtW
GaloreArami
2100
Praction of Per artutttee tere Siperiin
AdsmWWibit
668
NansAdam
066
Ada mW
NansAdam MicroAdam
CalorsAram
Nanokdam
-0-
Microndam
Aeddo
14000
Global Stops
Global Steps
Gobal Stops
(a) Train loss.
(b) Evaluation metrics.
(c) fraction of updated parameters.
Figure 12: Dynamics of NLP FT.
I Details and more results for experiments on CV Tasks
I.1 Details of Experiments on CV Tasks
ViT-Large The detailed training configurations for the ViT-Large model summarized in Ta- bles 18-19, including both common and optinizer-specific hyperparameters. Task1 is CIFAR 10 and task2 is F00wers102.
Table 18: Common hyperparameters used for finetuning ViT-Large.
Batch Size
Seed
Weight Decay
LR Scheduler
Label Smoothing
128
42
0.0
CosineAmeealnnLLR
0.1
Epochs Task
Epochs Task 2
8
5
(0.9, 0.999)
10-8
Table 19: Optinizer-specific hyperparameters for ViT-Large.
Optimizer
LR Taskl
LR Task2
k ko
Dynamic Density / m
Mask Interval
NANOADAM
le-3
2e-3
0.1%
off 10
100
MicroAdam
le-4
le-3
0.1%
AdamW
lc-4
le-4
The experimental settings for ResNet101 are summarized in Tables 20-21. These include common training hyperparameters and optimizer-specific configurations. Task1 is CIFARIO and task2 100
ResNet101
Table 20: Common hyperparameters for ResNet101.
Batch Size
Seed
Weight Decay
LR Scheduler
Label Smoothing
128
42
0.0
None
0.0
Epochs Taskl
Epochs Task2
30
(0.9, 0.999)
le-8
For ResNet18, we use the same common settings as in Table 20, while the optimizer- specific hyperparameters for ResNet18 are summarised in Table, 22, Task1 is CIFAR10 and task2 Flowers102.
ResNet18
31

---

Table 21: Optimizer-specific hyperparameters for ResNet101.
Optimizer
LR Task1
LR Task2
k: / kb
Dynamic Density / m
Mask Interval
NANOADAM
le-2
7e-3
1%
off m = 10
100
MicroAdam
le-3
5e-3
0.1%
AdamW
le-3
1e-3
Table 22: Optimizer-specific hyperparameters for ResNet18.
Optimizer
LR Task1
Task2
k ko
Dynamic Density / m
Mask Interval
NANOADAM
9e-3
7e-3
1%
off
100
MicroAdam
le-3
5e-3
1%
10
AdamW
le-3
le-3
1.2 Dynamics of catastrophic forgetting
The generalisation performance of various optimisers over different tasks are shown in Figure 13- 15, while the experiment settings reported in Appendix 1.1.
0.W2
Accuraey
Accurayy
0.08
Nanokdam
MicroAdim
A32.m
of Tapk
Nane MicroAdam Adamty
Tox Phase
1 Phase
Steps
(a) Generalization Task 1.
(b) Generalization Task
Figure 13: Catastrophic forgetting with ViT-Large.
10
Eval Accuracy
Eval AccuraCy
NanoAdam
croAdam
MicroAdam
AdamW
84
AdamW
63
Task
01
Task Phase
Task Phase
Steps
Steps
(a) Generalization Task 1.
(b) Generalization Task 2.
Figure 14: Catastrophic forgetting analysis on ResNet101. (a) Accuracy on Task drops significantly after switching to FT on Task for AdamW and MicroAdam, while NANOADAM retains performance. (b) NANOADAM also achieves better adaptation on Task 2.
L.3 Parameter shift visualisation
To provide finer- finer-grained insights, we visualize the layer-wise differences between the parameters of the pretrained ViT-Large model and those after continual learning CIFARIO and Flowers pre-
32

---

Accurary 83
64
Evall Accuracc
NanoAdam croAdam
NanoAdam
MicroAdam
Adami
AdamW
Task Phase
Task Phase
(a) Generalization Task 1.
(b) Generalization Task 2.
Figure 15: Catastrophic forgetting with ResNet18.
sented as heatmaps Figure 16. The experimental setup follows the details reported in Appendix I.1. Notably, NANOADAM induces minimal drift in the attention weights (QKV), highlighting its stability in preserving critical features during continual learning.
2,025
2.025
2420
2020
2020
2010
2010
2010
300
20000
(a) AdamW, & distance=2.722 72,
(b) MicroAdam, la 2istance=2.11.
(c) NanoAdam, l distance=1.57.
Figure 16: Parameter shift (qkv block) in Layer 10 after continual finetuning on CIFAR10 and Flowers102.
L.4 Larger learning rate
We further investigate whether NANOADAM enables stable optimization under larger learning rates. We finetune ViT-Large (pretrained on ImageNet) on CIFAR-10 10 using NANOADAM, MicroAdam, and AdamW, representngg finetuning small weights, large gradients, and all weights respectively.
Using the same hyperparameter settings as in Tables 18-19, we vary the learning rate in (ee-5, 1e-3), 3}. The resulting performance is shown in Figure 17. We observe that both AdamW and MicroAdam perform best at 1e-4 and degrade significantly at 1e-3. In contrast, NANOADAM benefits from the larger learning rate, achieving its best performance at le-3, This highlights its stability and effectiveness in aggressive optimization regimes.
1.5 Effective learning rate
Effective Learning Rate (ELR) quantifies the actual rescaling applied to parameter updates during optimization. In adaptive optimizers such Adam and its variants, the update rule incorporates element-wise adaptation based on the historical statistics of gradients. For a parameter w at step f, the update is given by:
m
W++1 = We - 1)
V Ut
33

---

1.0
0,994
0.993
0.8
NanoAdam 1e-3
Accuracy
NanoAdam
Eval Accuracy
0.992
Nanokdam 1e-5
0.6
AdamiV 1e-4
Adamm 1
MicroAdamm
Microndam 1e
0.989
NanoAdam
MicroAdom ie
0.2
0.988
Adamn 1e-4
1
Steps
Steps
(a) performance with best lcarning rate.
(b) performance with full learning rate grid.
Figure 17: Small weights allow large learning rate.
Here, denotes the global learning rate; m, and ig represent the bias-corrected first and second moment estimates, respectively; and small constant added for numerical stability. The effective learning rate is thus defined as:
ELR
Vde- +
Since ELR is computed per-parameter and evolves over time, its magnitude provides insight into how aggressively each parameter is being updated. For sparse optimizers such NANOADAM and MicroAdam, which selectively update subset of parameters, we compute ELR only over the actively updated parameters. The final reported metric is the average ELR across these selected parameters. We present the ELR dynamics of NANOADAM, MicroAdam, and AdamW during finetuning on both vision and language tasks.
For the computer vision task, finetune the ViT-Large model (pretrained on ImageNet) on CIFARIO using various optimizers. The experimental settings follow those described in Tables 18-19. The ELR trends for different optimizers are depicted Figure 18. For the NLP task, we finetune the BERT-base model on the SST-2 dataset from the GLUE benchmark. The experimental details are provided in Appendix H. As shown in the results, NANOADAM enables a more aggressive effective learning rate compared to other methods.
MicroAdam
MicroAdam
700
AdammW NanoAdam
600
Adamn NanoAdam
500
500
400
300
200
200
100
100
0
0
2
4
5
Epoch
Epoch
(a) ELR for FT CV task.
(b) ELR for FT NLP task.
Figure 18: Comparison of effective learning rate in FT CV and NLP task.
J Details and results for finetuning on Commonsense benchmark
We evaluate our method, NANOADAM, alongside MICROADAM and AdamW on fully fine-tuning the Llama 3.2 3B model on the Commonsense Benchmark. For these experiments, exclude the final
34

---

1m_ head and all normalization layers from partial updates; these layers are fully updated. We largely follow the experimental setup provided by the DoRA repository and utilize gradient checkpointing. The benchmark consists of eight tasks: HellaSwag, Winogrande, PIQA, ARC-Fasy, ARC-Challenge, OpenBookQA, SocialIQA, and BoolQ. All models are trained on compute node equipped with Ax1100 40GB GPUs. The training procedures are kept identical across methods, with the exception of individually tuned learning rates. We note that MICROADAM diverges when using a learning rate larger than le 5. The hyperparameters used for these experiments are summarized in Tables 23 and 24, Note that both MICROADAM and NANOADAM use an effective gradient sparsity of 90%.
It's important to note that our proposed method functions as an efficient optimizer, and is thus orthogonal to LoRA it can be combined with LoRA rather than serving as an alternative. Therefore. we also conduct the same experiments with LoRA (r=32), which is trained using Adam W and NanoAdam, respectively. For fientuning LoRA with Adam W, we use learning rate of 1e 4, and for fientuning LoRA with Admo, we use a learning rate of 7e 4, density k = 10% and mask interval of 600 and turn off the dynamic density.
In Table 25, we report the performance on each task, the average performance, as well as the training time and average memory usage across GPUs. As shown, NANOADAM achieves the best average accuracy while also reducing memory consumption compared to the other methods.
Table 23: Common hyperparameters used for finetuning Llama3.2 3B Commonsense benchmark.
Seed
Batch Size
micro batch size
epochs
cutoff length
B
42
16
2
256
(0.9, 0.999)
x 10
Table 24: Optinizer-ppecific hyperparameters for finetuning LLaMA3,2-3D on Commonsense tasks.
Optimizer
LR
k ko
Dynamic Density m Mask Interval
NANOADAM
6e-5
10%
off m 10
600
MicroAdam
le-5
1%
A
le-5
Table 25: Performance of fine-tuning the I.lama 3.2.3B model on the Commonsense Benchmark: average accuracy (%), memory usage (GB) and time (h).
Method
HellaSwag
Winogrande
PIQA
ARC
ABC ARC-Challee Challenge
OpenBookQA
SociallQA
Bool(
AVG.
memmery
time
MicroAdam
79:07
$0.65
NANGADAM
93:21
$2:48
86.07
85.86
71.91
82.20
79.99
71.71
82.05
25.30
10,11
AdamnW
77.71
74.11
63.33
83.63
68. 34
75.20
76.10
66.67
73.14
37.75
21.8
LoRA (m-32) NanoAdam
88.
80.35
83.03
84.30
70.39
79.601
77.48
69:42
79,99
11.45
3,13
LoRA (F23)) Adam
88.38
80.43
81.22
81.89
72.35
80.20
79.02
61.53
78.88
15.24
2.84
K Details and results for finetuning on GSM-8k benchmark
We now validate the effectiveness of various optimization methods on a finetuning task. Specifically, finetune LLaMA2-7B the GSM-8k dataset, challenging benchmark for grade-school-leve mathematical reasoning. Our experiments largely follow the codebase and the settings of Mi- croAdam [24].
The model trained for 3 epochs with a global batch size 32. The micro-batch size per device is set to auto, and the maximum input sequence length is 512. To ensure robustness, we experiments across four random seeds: {7, 42, 100, 512}. The hyperparameter configurations for each method are summarized in Table 26, and the corresponding results reported in Tables 27 and 28. It is worth noting that while the configured density levels or Ko vary across methods, the resulting effective gradient density remains approximately 10% for all.
As shown in the results, NANOADAM outperforms both Adam W8b and MicroAdam in terms of accuracy, while also reducing memory usage and maintaining runtime comparable AdamW8b.
35

---

Table 26: Optimizer-speeifie hyperparameters for finetuning LLaMA2-7B on GSM-8K tasks.
Optimizer
LR
k ko
Dynamic Density / m
Mask Interval
betas
NAMOADAM
4e-4
10%
off
5
(0.75, 0.999)
MicroAdam
4e-5
1%
m= = 10
(0.9, 0.999)
AdamWSt
4e-5
(0.9, 0.999)
Table 27: Accuracy comparison of finetuning LLama2-7B using various optimisers on GSM-8k tasks.
Method
seed=7
seed=42
sced=100
seed=512
Mean
Std
Adam W88
33.28
34.27
33.36
34.04
33.74
0.49
Microadam
33.43
34.42
33.59
34.80
34.06
0.66
NANOADAM
34.27
34.57
35.63
35.33
34.95
0.64
Table 28: Total memory overhead and full run time of finetuning LLama2 7B using various optimisers on GSM-8k tasks. Results are averaged over 4 seeds.
Method
run time (h)
memory (GB)
AdamW8h 0.40
43.27
Microadam
0.47
38.90
NANOADAM
0.41
36.68
L Transfer across dissimilar domains
To evaluate the potential limitations of our method when transferring across dissimilar domains, we fine-tuned ResNet-18 model pretrained on ImageNet general-domain dataset) on the PathMNIST task from the MedMNIST dataset-a medical image classification task with 9 classes. We compared the performance of full fine-tuning using Adam (learning rate 7e-4) with our method using Nano.Adam (sparsity = 1%, learning rate 1e-3). 3). AdamW achieve evaluation accuracy 90.85% while Admo achieves 90.63 60..3% Despite the domain shift, Nano achieves performance comparable to full fine-tuning, demonstrating its robustness even in cross-domain adaptation scenarios.
M Sensitivity to initial weight distribution
To evaluate sensitivity to the initial weight distribution, we conducted the following experiment: we started with ResNet-18 model pretrained ImageNet and pruned 80% of the weights based magnitude. We then fine-tuned the resulting sparse network on CIFAR- 10 using both Adam (learning rate 1e-3) 3) and NanoAdam (gradient density=1% learning rate 5e 3). Adam achieved evaluation accuracy of 85.34%, while NanoAdam reached 86. 16%, suggesting that NanoAdam is even more effective under sparse initialization. We hypothesize that this is because pruning removes 80% of the small weights, leaving the network to rely primarily on large weights during full fine-tuning- strategy have shown be inefficient in our ablation study. In this case, focusing updates on the remaining small weights, as Admm does, leads to better adaptation and generalization.
36


Reviews:
[{"review_id": "8VtxY6Kjr2", "reviewer": "Reviewer_tQEr", "summary": "The paper presents a unique finding that during fine-tuning of large models, the weights with small magnitude often get large gradients. Based on this observation, the paper proposes a modified Adam optimizer to dynamically update only the small-magnitude weights during fine-tuning in order to reduce the cost of fine-tuning large pretrained neural networks. The paper presents results to support this method on both NLP and vision tasks.", "questions": "The writing could have been better organized.\nDue to the significant reduction in memory usage and updatable parameters, is the proposed method also prone to overfitting?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\nThe paper takes an interesting approach to restrict the fine-tuning to parameters with small absolute magnitude.\nDemonstrates superior or equivalent performance with lower memory usage.\n\n\nWeaknesses:\nIt can be used for fine-tuning and not for training from scratch, since the core hypothesis will not hold true. \nNo discussion on overfitting vulnerability due to the reduction in trainable parameters.", "was_revised": true, "final_justification": "The rebuttal addresses my concerns. It is an interesting work, but considering the scope of the work is restricted to fine-tuning, I would like to retain my initial rating.", "rebuttal": "We thank the reviewer for finding our approach interesting and acknowledging the performance gains and memory reductions of our method. Below, we address their insightful comments and look forward to the discussion period.\n\nRegarding the concern about potential overfitting, we would like to clarify the following:  In fact, our proposed method is inherently less prone to overfitting. As demonstrated in Table 5 and Appendix G.3, our method results in a smaller parameter shift away from the initial weights, indicating that it learns the downstream task with smaller movement in parameter space, which also mitigates catastrophic forgetting of the pre-training task.\n\nAnother perspective is that our method implicitly performs a form of regularization, similar to weight decay. By selectively updating only a subset of parameters—specifically avoiding those with large magnitudes—we prevent dominant features from disproportionately shaping the network’s functional space. This restraint helps mitigate overfitting and promotes more robust generalization.\n\nIndeed, our method primarily concerns finetuning, which is a common and highly relevant problem in the era of large models. Due to the simplicity of our parameter subset selection criterion, its advantages, including lower memory requirements, faster convergence, and improved generalization, can be easily realized across different architectures and training setups."}, {"review_id": "ahdYvbXrH8", "reviewer": "Reviewer_S7zk", "summary": "In this paper, the authors observe an inverse association between gradient magnitude and weight magnitude. They use this to motivate a finetuning strategy that focuses on weights with small magnitude, with a dynamically updated mask selecting those smallest weights (\"NanoAdam\"). They investigate this strategy theoretically in a teacher-student setup that illustrates the merits of this criterion. In empirical experiments on large pretrained models, they then find that NanoAdam improves both finetuning generalization and catastrophic forgetting compared to alternative methods while having a smaller memory footprint.", "questions": "1. Could you provide more detail for the proof of Theorem 2.2? Moreover, I think drawing a more explicit connection between Definition 2.1 and catastrophic forgetting (see my comments above) would be helpful. As part of that, it would also be useful to make Definition 2.1 more specific (e.g. how is the set of largest neurons selectedt? Is it the k largest neurons? Is it all neurons above a certain magnitude?\n\n2. Could you clarify your contributions relative to previous work and, in particular, reference [18]? In particular, how central is the introduction of dynamic masking to your overall contributions?\n\n3. If possible, could you add a comparison for the experiments in Section 4 to static masks as used in [18]?\n\n4. Could you add standard deviations to the learning curves?\n\nI can imagine that new experiments may not be realistic to run within the period of the rebuttal. Addressing the first two questions would be sufficient for me to increase my score though I believe addressing the latter two questions would substantially improve the paper further.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\nThe paper was well-written and it was generally easy to follow the flow of logic. I also appreciated the fact that the authors integrated a theoretical and empirical investigation into the same phenomenon. Further, I thought the investigation into the association between gradient and weight magnitude was well-executed; I personally found this effect intriguing and I also appreciated the additional investigation into how this effect emerges over pretraining, as it helps shape the reader's intuition.\n\n**Weaknesses**\n\nMy primary concerns with the paper in its current form are:\n\n*1 Soundness of the theoretical investigation*\n\nAs noted above, I generally appreciated the theoretical investigation and thought it provided a useful simplified intuition (indeed, even as an empirical toy model, I think an approach like this is helpful). However, I think the authors should provide more detail on their setup. In particular, Theorem 2.2 does not provide sufficient detail: what does it mean that \"gradient flow learns f_ft\"? Is this in the limit of infinite data? How does this follow from [9] and which result are you referring to? The closest fit I could find was Theorem 2.3 in [9], but that talks about a relative comparison between iterative magnitude pruning and learning rate rewinding.\n\nTwo minor comments: I think it would be helpful to more concretely connect the \"representation preserving\" phenomenon to its resulting impact on catastrophic forgetting, i.e. how much worse does the network get at the pretrained task as a result of finetuning. At the moment, you only talk about the impact on the representation, but ultimately we care about the impact on model output. I think, with Lemma C.1, you should be able to make a statement about this, and I think that would help tie this argument together.\n\nFurthermore, I don't understand the following statement: \"Due to the implicit sparsity bias in overparameterized training, there often exists at least one sufficiently small neuron that has minimal impact on the pretrained representation.\" I agree that this is likely to happen in practice, but if I'm not missing anything, the implicit sparsity bias (if you're referring to works like [1-3]) usually refers to a sparsity in function space rather than weight space. I.e. you can cluster the neurons in a two-layer network into a sparse set of clusters but that doesn't mean the hidden layer is actually sparse. Could you elaborate on this point?\n\n*2 Contributions of the empirical investigation*\n\nIn terms of your empirical investigation, am I understanding correctly that reference [18] already proposed and evaluated the idea to update small weights? If I'm understanding the relative contributions correctly, the primary methodological contribution you are introducing is to dynamically update this mask, right? I appreciated the comparison you drew between those approaches in Figure 11 and I think it would be important to evaluate the impact of dynamical masks more broadly in Section 4 as well, by specifically comparing the dynamic mask to the static mask across the entire set of benchmarks (extending your analysis in Figure 11). More generally, I think it would be good to contrast the novel contributions made in this paper to those already known from this previous paper in more detail. For example, are your insights into the benefits of (even static) masks for catastrophic forgetting new?\n\nAdditionally, I think it would be useful to repeat the experiments in Section 3.2 several times and indicate standard deviations of the different learning curves. It is difficult to assess how noisy these observations are.\n\n**Conclusions**\n\nOverall, in its current state, I rate this paper as a borderline reject. However, I enjoyed reading it, think it is a worthwhile study, and highlight a number of questions below that would cause me to increase my score.\n\n**References**\n1. https://proceedings.mlr.press/v125/chizat20a.html\n2. https://proceedings.mlr.press/v99/savarese19a.html\n3. https://proceedings.neurips.cc/paper_files/paper/2022/hash/7eeb9af3eb1f48e29c05e8dd3342b286-Abstract-Conference.html", "was_revised": true, "final_justification": "I thank the authors for their thorough response. The further clarification of the theoretical investigation is very helpful and the additional experiments support the authors' claims further. Finally, I found the authors' clarification about their novel contributions compared to prior work very helpful. Given that the authors addressed all of my questions, and with the understanding that they will update the camera-ready, I will increase my score to a 5.", "rebuttal": "We appreciate the reviewer’s recognition of our work and that they found both our theory and empirical analysis useful. We thank them for their constructive and valuable feedback, which we address below. We look forward to a potential exchange during the discussion period.\n\n### 1. Soundness of the theoretical investigation\nWe are delighted that the reviewer found our toy model useful and are happy to answer their questions on the setup.\n\n**Details on theoretical setup**:  We consider the same setup as [1], which is a simpler setting than [9],  which considers the infinite data limit, i.e. learning over a given data distribution. We assume that the pre-training data has been generated from a teacher model $f_{\\text{teacher}}$ consisting of $k$ neurons to define labels. On this data, first, a pre-trained model $f_{\\text{pre}}$ is learned that consists of $n$ neurons, assuming $n>k$. The finetuning task is defined with labels from a target model that adds an extra neuron to the original teacher model $f_{\\text{finetune}} = f_{\\text{teacher}} + f_{\\text{extra}}$. See also Appendix C for more details.\nThis setup is equivalent to the finetuning scenario of training a single student neuron of the network $f_{\\text{pre}}$ to approximate a single target neuron $f_{\\text{extra}}$ by minimizing the mean squared error loss as average over general feature data, for instance $X \\sim N(0,I_d)$ with $X \\in \\mathbb{R}^d$ with $d > 1$ (as in the illustration). \nDenote $a_{j}$ and $w_{j}$ as parameters of the trainable student neuron from our Theorem 2.2 and similarly denote $\\tilde{a}$ and $\\tilde{w}$ as parameters of the target neuron.\nBased on this notation and clarification, we would be happy to update Theorem 2.2 to the following more detailed formulation:\n\nTheorem 2.2:\nAssume a model $f(x)$ consisting of $n$ neurons learns the teacher $f_{\\text{teacher}}(x)$ corresponding to a pre-training task so that $f(x) = f_{\\text{teacher}}(x)$ for all $x\\in \\mathbb{R}^d$. Furthermore, let $f(x)$ consist of at least two neurons $i,r \\in [n]$ such that $\\max(\\{|a_i|^2, |a_r|^2\\}) \\leq \\epsilon$ for an $\\epsilon > 0$ and $sign(a_i) \\neq sign(a_r)$. Let a new task be defined based on labels $f_{\\text{finetune}} :=  f_{\\text{teacher}} + f_{\\text{extra}}$ with an extra neuron $f_{\\text{extra}} = \\tilde{a} \\sigma( \\tilde{w} \\cdot)$. Let only the neuron $j$ of $f$ be trainable to finetune $f(x)$ to the new task, where $j=arg\\min_{i}{\\{|a_i|||w_i||: sign(a_i)=sign(\\tilde{a})\\}}$. Then, the gradient flow with respect to finetuning time $t$ of the neuron $j$ parameterized as $v_{j,t} = |a_{j,t}| w_{j,t}$ and initialized at pre-trained values $v_{j,0}=|a_j| {w_j}$, converges to a value $v_{\\infty}$ so that $|v_{\\infty} - v| <\\epsilon$, where $v$ is the target $v = |\\tilde{a}| \\tilde{w}$.\n\n\n**Details on the assumptions and proof idea**: Note that the assumption $\\max(\\{|a_i|^2, |a_r|^2\\}) \\leq \\epsilon$ in the above theorem is justified because $f$ learns a representation of  $f_{\\text{teacher}}(x)$ when solving a pre-training task where at least one of the neurons is effectively pushed to $0$ (as the teacher consists also of fewer neurons than the trained model $f$).\n\nTo apply Theorem 6.4 in [1], we need matching signs for the parameters $a$. Otherwise, we have an immediate mismatch between the two single neuron functions. Assuming matching signs, we can absorb $a_{j}$ and $\\tilde{a}$ into the activation to simplify the analysis. This reduces the problem to optimizing a single layer neuron $\\sigma(v_{j,t} \\cdot)$ with initialization $|a_{j}| w_{j}$ to learn a target vector $v = |\\tilde{a}| \\tilde{w}$ with some small label perturbation that is equal to the initialization $a_j\\sigma(w_j x_i)$. \nTo prove this, we make some assumptions:  (1) the teacher representation has been exactly learned as in Figure 3 (2) the data set fulfills the assumptions from Theorem 6.4 (which cover  i.i.d. Gaussians). \nFor now we assume there exists a neuron such that $|a|^2 \\leq \\epsilon$ after pretraining which likely exists after pretraining.\nThen, this noise term vanishes due to a concentration inequality on the parameter initialization, as we have $|a_{j}|^2 =||w_{j}||^2$. In summary, we can apply Theorem 6.4 to show that the gradient flow for $f$ with one trainable neuron learns $f_{\\text{finetune}}$. \nWe will highlight these details in the revised manuscript. Additionally, we will include the gradient flow of the one neuron problem in the main manuscript to explain the setting in more detail.\n\n[1] Yehudai, Gilad, and Shamir Ohad. \"Learning a single neuron with gradient methods.\" Conference on Learning Theory. PMLR, 2020.\n\n**Definition 2.1 and catastrophic forgetting**: We will make Definition 2.1 more explicit using the notation of Theorem 2.2, as follows.\n\n**Definition 2.1**: A finetuned network $f$ is $k$-neuron representation preserving iff the largest neurons corresponding to $\\text{Top-k}(j \\in [n] : |a_j| ||w_j||)$ remain unchanged compared to the pre-training task.\n\n\nDefinition 2.1 is directly connected to catastrophic forgetting, as the top-k neurons correspond to the part of the pre-training task that transfers to the new task. For that reason, we do not want to forget them but use them also in the finetuning task. \nThis is also reflected in our $L_2$ proxy measure of parameter shift for catastrophic forgetting (as in Lemma C.1). Not only do the parameters of the top-k neurons not change during finetuning, they therefore do not contribute to the $L_2$ distance between old and new parameters. Also the other parameters have to change less. If some of the largest neurons would change, to compensate for that change, all trainable neurons would have to move more in parameter space. As a consequence, Definition 2.1 also implies low forgetting according to Lemma C.1, which we will also demonstrate numerically in our example in the appendix.\n\n**Implicit sparsity bias**: The implicit sparsity bias refers indeed to [2]. It implies that a product of parameters (in our case $|a|||w|| < \\epsilon$) remains small. The product tends to be small if both or either one of the parameters is small. Products of parameters thus intuitively promote sparsity. There is a large line of work on this type of bias. One could potentially interpret the product as a form of function space, but this is beyond the scope of this paper and we will clarify our meaning of sparsity in our revised manuscript. We primarily want to argue why the assumption  $\\max(\\{|a_i|^2, |a_r|^2\\}) \\leq \\epsilon$ in our theorem is justified (see comment above).\n\n[2] Arora, Sanjeev, et al. \"Implicit regularization in deep matrix factorization.\" Advances in neural information processing systems 32 (2019).\n\n\n### 2. Contributions of the empirical investigation\n\nWe appreciate the opportunity to clarify our contributions relative to [18]:\n\n- While [18] hypothesizes that updating only small-magnitude weights can be effective, we provide an explanation: large gradients are usually associated with small weights during finetuning, making them more adaptable. We show that the gradient-weight correlation is much stronger during fine-tuning than training from scratch, explaining why the strategy is less effective in the latter case.\n- We attribute this pattern to knowledge transferability and overparameterization, offering a principled understanding of when small-weight finetuning is most effective.\n- We further show that updating small weights helps mitigate catastrophic forgetting, supported by both theoretical and empirical evidence.\n- Our method is novel in that: (a) mask selection and (b) sparsity are dynamic, and (c) selection is done per layer rather than globally. Unlike [18], which uses a fixed global mask, our dynamic approach improves efficiency and performance while reducing memory usage. Dynamic masking specifically improves the implicit weight regularization by discouraging over-reliance on a fixed subset of parameters; and mitigates catastrophic forgetting by reducing parameter shift.\n- Lastly, unlike [18] which focuses only on NLP, we evaluate our method on CV tasks as well, demonstrating broader applicability.\n\n### 3. Comparison with static mask\nThe additionally requested table below shows that our dynamic masking strategy (NanoAdam) leads to significant improvements compared to static masking [18].\n\n| Model       | Method              | COLA  | SST2  | MRPC  | STSB  | QQP   | MNLI  | QNLI  | AVG   |\n|-------------|---------------------|-------|-------|-------|-------|-------|-------|-------|--------|\n| bert-base   | NanoAdam            | 60.87 | 93.46 | 88.48 | 89.98 | 90.67 | 84.30 | 91.76 | 85.65  |\n| bert-base   | NanoAdam (static)   | 56.24 | 91.51 | 84.07 | 89.68 | 89.75 | 82.59 | 90.92 | 83.54  |\n| bert-large  | NanoAdam            | 66.85 | 94.61 | 90.20 | 90.86 | 91.03 | 86.40 | 92.44 | 87.48  |\n| bert-large  | NanoAdam (static)   | 59.07 | 92.66 | 84.31 | 89.86 | 90.33 | 85.39 | 91.78 | 84.77  |\n| opt-1.3B    | NanoAdam            | 67.69 | 96.45 | 87.99 | 91.00 | 91.33 | 88.24 | 92.75 | 87.92  |\n| opt-1.3B    | NanoAdam (static)   | 60.38 | 95.30 | 86.03 | 90.55 | 91.07 | 87.61 | 91.91 | 86.12  |\n\n### 4. Standard deviations\nWe will add confidence intervals to our figures and tables based on five seeds. Please find exemplary results below.\n\n| Method         | Averaged Performance | Standard Deviation |\n|----------------|----------------------|---------------------|\n| Small weights  | 89.4                 | 0.296               |\n| Large weights  | 62.25                | 0.927               |\n| Random weights | 88.88                | 0.229               |"}, {"review_id": "ofvWR5jPOn", "reviewer": "Reviewer_Vq6D", "summary": "This paper introduces NANOADAM, a new optimization technique aimed at improving the efficiency and effectiveness of fine-tuning large pre-trained neural networks. Rather than selecting parameters for update based on gradient magnitude (a common practice), NANOADAM selects parameters with small absolute weight magnitudes, arguing that these are more “plastic” and less likely to encode critical pretrained knowledge.\n\nThe authors observe a consistent hyperbolic correlation between gradients and weight magnitudes during finetuning (particularly in NLP and vision tasks), i.e., large gradients tend to correspond to small weights. They propose leveraging this correlation to select and update only small weights dynamically. NANOADAM avoids the need for gradient-based selection or error feedback mechanisms, improving memory efficiency and preserving pretrained representations. Apparently, this reduces catastrophic forgetting.\n\nExtensive experiments across NLP (GLUE) and CV (CIFAR-10 and Flowers102) tasks demonstrate that NANOADAM outperforms existing memory-efficient optimizers (MicroAdam, GaLore, AdamW-8bit) in terms of generalization performance, memory usage, and parameter shift.", "questions": "1. Often for FT we freeze a set of layers. How does the number of layers to freeze affect the framework? I am thinking simple convnets with a MLP on top. How do convolutional and MLP layers differ?\n\n2. Could the authors provide further clarification or experimental results on why NANOADAM does not perform well when training from scratch? Is it primarily due to the lack of structure in weight initialization?\n\n3. While LoRA is cited as a PEFT baseline, it was not included in the experiments. Could the authors justify this omission or include results in the appendix?\n\n4. While the paper justifies excluding error feedback mechanisms, did the authors test simpler forms of feedback (e.g., residual accumulation without reinsertion) to see if performance improves marginally without major cost?\n\n5.  How robust is NANOADAM to variations in m (mask interval) and d (density interval)? Some empirical sensitivity curves or robustness plots would be helpful.\n\n6. NANODAM was proven for small parameter model. Do we trust this holds for large 7B+ models?", "limitations": "They acknowledge that small weights do not always fully overlap with large gradients (Section 2.2), and that their method is explicitly designed for finetuning rather than training from scratch.\n\nHowever, they could further expand on:\n\nPotential pitfalls in transfer across highly dissimilar tasks (e.g., general-domain pretraining to low-resource medical tasks).\n\nSensitivity to initial weight distributions—e.g., would pretrained weights that are uniformly sparse or quantized change NANOADAM's effectiveness?", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths\nNovel Insight into Sparse Finetuning: The paper presents a compelling observation—large gradients correlate with small weights—and builds a theoretically and empirically grounded argument for exploiting this in fine-tuning.\n\nThrough a teacher-student model, the paper provides a clean and illustrative theoretical explanation for why updating small weights can avoid catastrophic forgetting (Theorem 2.2), which supports the method’s design.\n\nEfficiency Without Gradient Use: NANOADAM does not rely on gradients for mask selection, enabling precomputation and reducing memory/computation overhead compared to methods like MicroAdam or DST.\n\nClear Ablations: The ablation studies (Figure 4) directly compare small weights vs large weights vs random weights, and small weights vs large gradients, showing consistent gains for the proposed strategy.\n\nExcellent Empirical Results:\n\nTable 2 shows that NANOADAM performs best across nearly all tasks for BERT-base, BERT-large, and OPT-1.3B.\n\nTable 3 shows lower memory usage.\n\nTable 4 and 5 demonstrate its ability to reduce catastrophic forgetting and parameter drift during continual learning.\n\nBroad Applicability: Evaluated across both NLP and CV tasks, with results on transformer and CNN architectures, making the claims more generalizable.\n\nWeaknesses:\n1. No Comparison to LoRA-type PEFT Methods: While LoRA is mentioned in the related work, no head-to-head comparisons are presented. Although LoRA adds parameters, it is a strong PEFT baseline and should have been benchmarked.\n\n2. Density Scheduler Sensitivity: While the density scheduler and update intervals are mentioned, the method might be sensitive to these hyperparameters. The default choices and tuning ranges could have been discussed in more detail (e.g., effect of mask interval m on performance).", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewers for their valuable feedback and insightful comments. We are happy to address the concerns below.\n\n### 1. Layer freezing\n\nTo understand how freezing different layers affects our framework, we conducted an ablation study using ResNet-18 on CIFAR-10. We systematically froze each layer from the first to the final fully connected (FC) layer and evaluated generalization performance:\n\n| **Layer Frozen** | **Eval Acc (%)** |\n| ---------------------- | ---------------------- |\n| No freeze              | 92.58                  |\n| Freeze layer1          | 91.53                  |\n| Freeze layer2          | 91.29                  |\n| Freeze layer3          | 91.36                  |\n| Freeze layer4          | 91.22                  |\n| Freeze FC              | 91.51                  |\n\nWe observe that freezing intermediate layers results in more severe degradation, while freezing early or final layers has less impact. This suggests that selectively updating parameters—rather than completely freezing entire layers—is more effective for preserving generalization in fine-tuning.\n\n### 2. Training from scratch\n\nOur hypothesis is that the effectiveness of NanoAdam stems from: (1) Knowledge transferability and (2) Overparameterization.\n\nAs shown in Table 4 of the manuscript, moving from smaller models (e.g., ResNet-18) to larger ones (e.g., ViT-Large)—i.e., increasing overparameterization— improves performance and reduces forgetting. This supports the view that NanoAdam benefits from the overparameterized nature of modern deep networks.\n\nTo further examine the impact of Knowledge transferability, we trained ResNet-18 and ResNet-101 from scratch on CIFAR-10 and compared AdamW (LR=1e-3) with NanoAdam (90% sparsity, LR=5e-3). This setup can be considered an extreme case of transferring knowledge from a null domain (i.e., no pretraining) to a vision domain. The generalization results are presented in the table below. We observe that NanoAdam incurs about a 1% drop in accuracy compared to AdamW, suggesting that it is less effective in training from scratch or when there is a mismatch between the knowledge in pretraining and finetuning tasks. Notably, the more overparameterized ResNet-101 shows a smaller performance drop, further indicating that overparameterization contributes to the effectiveness of NanoAdam.\n\n| **Model** | **AdamW (%)** | **NanoAdam (%)** |\n| --------------- | ------------------- | ---------------------- |\n| ResNet-18       | 92.48               | 91.29                  |\n| ResNet-101      | 94.08               | 93.06                  |\n\n### 3. Comparison with LoRA\n\nWe thank the reviewer for raising the concern regarding comparisons with LoRA. In response, we have conducted experiments to directly compare our method with LoRA. It’s important to note that our proposed method functions as an efficient optimizer, and is thus orthogonal to LoRA—it can be combined with LoRA rather than serving as an alternative. Specifically, we fine-tuned a LLaMA 3.2-3B model on the Commonsense benchmark using the following setups:\n\n* Full fine-tuning with AdamW\n* Full fine-tuning with NanoAdam (90% sparsity)\n* LoRA (r=32) with AdamW\n* LoRA (r=32) with NanoAdam (90% sparsity)\n\nThe results are summarized in the table below. We observe that full fine-tuning with NanoAdam yields the highest performance, albeit with increased memory usage compared to LoRA. However, when memory efficiency is a priority, combining LoRA with NanoAdam proves highly effective—reducing memory consumption from 15.2 GB to 10.7 GB while also achieving improved generalization performance.\n\n| **Method**          | **Eval Acc (%)** | **Memory (GB)** | **Runtime (h)** |\n| ------------------------- | ---------------------- | --------------------- | --------------------- |\n| Full FT with AdamW        | 73.14                  | 37.75                 | 21.80                 |\n| Full FT with NanoAdam     | 82.05                  | 25.30                 | 10.11                 |\n| LoRA (r=32) with AdamW    | 78.88                  | 15.24                 | 2.84                  |\n| LoRA (r=32) with NanoAdam | 79.09                  | 10.73                 | 3.13                  |\n\n### 4. Error feedback mechanism\n\nWe appreciate the reviewer’s insightful comment regarding the error feedback mechanism. We experimented with incorporating a similar error feedback strategy as used in MicroAdam; however, we observed that it actually degraded performance in our setting.\n\nWe hypothesize that this is due to the nature of our parameter selection strategy, which is based on the magnitude of the weights themselves. As discussed in Section 2.2, our small-weights update rule does not aim to replicate the trajectory of large gradient updates. Instead, it follows a distinct and highly efficient learning tracetory in finetuning. Introducing gradient accumulation and feeding the residual error back into this trajectory may interfere with its dynamics, ultimately leading to reduced performance.\n\n### 5. Sensitivity analysis\n\nWe conduct a sensitivity analysis by fine-tuning the BERT-base model on the QNLI task from the GLUE benchmark using a batch size of 32 and a learning rate of 1.1e-4.\n\nTo assess the impact of the mask interval parameter $m$, we disable the density scheduler and vary $m$ within the range [10, 1500]. The corresponding evaluation performance is presented in the table below. Our results show that model performance remains largely stable across this range, with optimal results observed when m falls between 80 and 300. Given that the total number of steps in one epoch is $S = 3274$ in our experiment, this tuning range corresponds to approximately $0.024S$ to $0.09S$. The hyperparameters reported in Table 10 of the manuscript are chosen to fall within this empirically validated range.\n\n| m        | 10    | 50    | 80    | 100   | 300   | 500   | 700   | 900   | 1100  | 1500  |\n| -------- | ----- | ----- | ----- | ----- | ----- | ----- | ----- | ----- | ----- | ----- |\n| Eval (%) | 90.72 | 90.85 | 90.99 | 91.21 | 91.21 | 91.09 | 91.09 | 91.16 | 91.03 | 90.98 |\n\nFor the sensitivity analysis of the density interval $d$, we fix the mask interval at $m = 81$ and vary $d$ within the range [100, 1000]. The resulting evaluation performance is presented in the table below. We observe that when $d$ exceeds 300, the performance remains largely stable, with the best performance achieved at $d = 400$. Given that the total number of steps in one epoch is $S = 3274$, this corresponds to a density interval greater than approximately $(300/3274)S=0.092S$, with the optimal interval around $(400/3274)S=0.122S$. The hyperparameters reported in Table 10 of the manuscript follow this observation.\n\n| d        | 100   | 200   | 300   | 400   | 500   | 600   | 700   | 800   | 900   | 1000  |\n| -------- | ----- | ----- | ----- | ----- | ----- | ----- | ----- | ----- | ----- | ----- |\n| Eval (%) | 90.94 | 90.68 | 91.34 | 91.58 | 91.32 | 91.27 | 91.23 | 91.29 | 91.43 | 91.21 |\n\n### 6. Scaling to larger models\n\nWe appreciate the reviewer’s suggestion to evaluate our method on larger models. To address this, we conducted additional experiments on two larger-scale models. Specifically, we fine-tuned the LLaMA3.2-3B model on the Commonsense Benchmark, as described in our response to Question 2, and we fine-tuned the LLaMA 2-7B model on the GSM8K dataset.\n\nFor the 7B model experiments, we used 4 random seeds and report the average performance along with standard deviation in the table below. The results demonstrate that our proposed method continues to offer performance gains when scaled to larger models.\n\n| method    | Eval acc     | Memory (GB) | Runtime (h) |\n| --------- | ------------ | ----------- | ----------- |\n| AdamW8b   | 33.74 (0.49) | 43.27       | 0.4         |\n| MicroAdam | 34.06 (0.66) | 38.9        | 0.47        |\n| NanoAdam  | 34.95 (0.64) | 36.68       | 0.41        |\n\n### Limitations\n\n1. To evaluate the potential limitations of our method when transferring across dissimilar domains, we fine-tuned a ResNet-18 model pretrained on ImageNet (a general-domain dataset) on the PathMNIST task from the MedMNIST dataset[1]—a medical image classification task with 9 classes. We compared the performance of full fine-tuning using AdamW (learning rate = 7e-4) with our method using NanoAdam (sparsity = 1%, learning rate = 1e-3). AdamW achieve evaluation accuracy 90.85% while NanoAdam achieves 90.63%. Despite the domain shift, NanoAdam achieves performance comparable to full fine-tuning, demonstrating its robustness even in cross-domain adaptation scenarios.\n\n    [1] Yang, Jiancheng, Rui Shi, and Bingbing Ni. \"Medmnist classification decathlon: A lightweight automl benchmark for medical image analysis.\" 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI). IEEE, 2021.\n\n2. To evaluate sensitivity to the initial weight distribution, we conducted the following experiment: we started with a ResNet-18 model pretrained on ImageNet and pruned 80% of the weights based on magnitude. We then fine-tuned the resulting sparse network on CIFAR-10 using both Adam (learning rate 1e-3) and NanoAdam (graident density=1%, learning rate 5e-3). Adam achieved an evaluation accuracy of 85.34%, while NanoAdam reached 86.16%, suggesting that NanoAdam is even more effective under sparse initialization. We hypothesize that this is because pruning removes 80% of the small weights, leaving the network to rely primarily on large weights during full fine-tuning—a strategy we have shown to be inefficient in our ablation study. In this case, focusing updates on the remaining small weights, as NanoAdam does, leads to better adaptation and generalization."}, {"review_id": "cAJYqYyfHK", "reviewer": "Reviewer_yCVR", "summary": "The paper proposes NanoAdam, a gradient-free, memory-efficient optimizer for finetuning large pretrained models. Motivated by the empirical observation that small-magnitude weights often correspond to large gradients, the method restricts updates to low-magnitude weights. This approach is designed to reduce catastrophic forgetting, preserve pretraining features, and improve training efficiency.", "questions": "n/a", "limitations": "While the authors briefly address limitations in Section 2.2, the discussion primarily reads as an analysis of the method’s advantages rather than a critical evaluation of its shortcomings. It would strengthen the paper if the authors included a dedicated section explicitly discussing the limitations and broader implications of their approach, including aspects such as the generalizability of the optimizer, its computational efficiency, and potential trade-offs in different model architectures or training regimes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "- While the observed correlation between large gradients and small weights is interesting, the assumption that small weights are always the best candidates for updates is overly simplistic. In Adam-style optimizers, gradients are normalized by past variance; thus, even small weights with high gradients may not result in large updates. Moreover, weight magnitude is not always indicative of parameter importance, especially in transformers where scale and function can differ layer-wise.\n- NanoAdam dynamically estimates the subset of smallest weights at intervals via sorting (Bottom_k(|θ|)). Although this occurs infrequently (every m steps), it could still introduce overhead, particularly in large models with tens of billions of parameters. It is unclear whether this cost counteracts the claimed efficiency gains—especially in practice on large-scale LLMs.\n- Despite strong results on BERT and vision models, it’s unclear how NanoAdam scales to contemporary LLMs (e.g., Llama, Mistral) with model parallelism and pipeline layers. The dynamic masking may interfere with hardware-accelerated fused optimizers and distributed training frameworks, which are critical for LLM-scale deployment.\n- The paper’s writing quality is suboptimal and affects readability. For instance, in the introduction (lines 16–38), the authors discuss motivation and related challenges, but abruptly switch to “Our method…” without first introducing what the method is or how it addresses the problem. Such transitions feel disjointed, and similar structural issues appear throughout the paper, weakening the overall presentation and clarity.", "was_revised": true, "final_justification": "The authors' responses resolve most of my concerns, I would adjust my rating.", "rebuttal": "We thank the reviewer for appreciating our insights as interesting and our empirical results as strong and giving us valuable feedback to improve our manuscript. Below, we address their insightful comments and answer their questions. We look forward to a potential exchange during the discussion period.\n\n### 1. Small weights and large updates\n\nWe appreciate the reviewer’s insight regarding the behavior of Adam-like optimizers and their gradient normalization effect. To better understand the actual parameter changes during fine-tuning, we conducted an additional experiment: we fine-tuned a BERT-base model on the CoLA task from the GLUE benchmark using AdamW for 5 epochs.\n\nFor each layer, we partitioned the parameters into three groups based on their absolute magnitudes in the pretrained model:\n\n1. The bottom 30% (smallest magnitudes)\n2. The top 30% (largest magnitudes)\n3. The remaining middle 40%\n\nWe then measured the average change in weights within each group, calculated as the difference between the pretrained and fine-tuned weights. This analysis allows us to examine which groups of parameters receive the largest updates during fine-tuning.\n\nThe results, presented in the table below, consistently show that across all layers, the smallest-magnitude weights undergo the largest updates, while the largest-magnitude weights change the least. This trend holds regardless of the layer’s scale or functional role in the Transformer architecture.\n\nAdditionally, in our implementation, the bottom-k selection is performed per layer, not globally—ensuring layer-wise adaptivity.\n\n| **Layer** | **30% Smallest** | **30% Largest** | **Remaining** |\n| --------------- | ---------------------- | --------------------- | ------------------- |\n| Layer 0         | 0.000799               | 0.000381              | 0.000393            |\n| Layer 1         | 0.000815               | 0.000325              | 0.000363            |\n| Layer 2         | 0.000787               | 0.000331              | 0.000344            |\n| Layer 3         | 0.000812               | 0.000302              | 0.000353            |\n| Layer 4         | 0.000846               | 0.000298              | 0.000354            |\n| Layer 5         | 0.000877               | 0.000282              | 0.000351            |\n| Layer 6         | 0.000821               | 0.000294              | 0.000342            |\n| Layer 7         | 0.000798               | 0.000299              | 0.000341            |\n| Layer 8         | 0.000727               | 0.000276              | 0.000321            |\n| Layer 9         | 0.000691               | 0.000278              | 0.000308            |\n| Layer 10        | 0.000669               | 0.000280              | 0.000302            |\n| Layer 11        | 0.000742               | 0.000279              | 0.000324            |\n\nWe agree that parameter magnitude is not always a perfect indicator of importance. In fact, for this reason, most of our analysis has served the purpose to understand under which conditions small weight magnitude is associated with high gradients (and thus high potential for adaptability to a new task) and when we can therefore expect our method NanoAdam to work well: We both require a high alignment between the pre-training and finetuning task (i.e. good transferability) and high degree of overparameterization. We will also highlight this fact in a separate limitations section in our updated manuscript. It is also noteworthy that our objective is not to identify the most precise weight importance measure. Rather, our aim is to develop a simple yet highly efficient criterion for finetuning that significantly reduces memory overhead from gradient updates while preserving—or even improving—generalization performance.\n\n### 2. Scalability and overhead\n\n\nWe appreciate the reviewer’s observation regarding the potential overhead introduced by the bottom-k operation. In our implementation, bottom-k selection is performed independently on each parameter matrix in each layer (excluding the final output layer), rather than applied globally. Specifically, it proceeds as follows: For each weight matrix, we first flatten it and divide it into subgroups of parameters (chunks). We then apply the bottom-k selection within each subgroup. As bottom-k operation has a time complexity of O(k \\log k), which remains computationally feasible in practice.\n\nTo assess scalability, we conducted experiments on larger models such as LLaMA 2-7B. The results of fine-tuning the LLaMA 2-7B model are shown in the table below. Despite the increase in model size, our method continues to deliver better generalization performance and significant memory savings, with only a modest increase in runtime overhead.\n\n\n| **Method** | **Eval Acc (%)** | **Memory (GB)** | **Runtime (h)** |\n| ---------------- | ---------------------- | --------------------- | --------------------- |\n| AdamW-8b         | 33.74                  | 43.27                 | 0.40                  |\n| MicroAdam        | 34.06                  | 38.90                 | 0.47                  |\n| NanoAdam         | 34.95                  | 36.68                 | 0.41                  |\n\n### 3. Compatibility with modern techniques\n\nWe appreciate the reviewer’s insight regarding the scalability of our method to contemporary large language models (e.g., LLaMA, Mistral), particularly in the context of model parallelism and pipeline layers. In our implementation, the bottom-k selection is applied independently to each parameter matrix within each layer, rather than performed globally across the model. This design ensures compatibility with modern hardware acceleration techniques and parallel training frameworks, and avoids conflicts with model or pipeline parallelism.\n\n### 4. Writing\n\nWe will thoroughly review the paper and revise the relevant sections to ensure smoother and less abrupt content switches. For instance, in the Introduction (lines 16–38), after outlining the motivation and related challenges, we will revise the transition sentence as follows: “In this work, we propose a method that relies purely on parameter magnitude—without the use of gradient information or error feedback—resulting in improved computational efficiency.”\n\n\n### Limitations\n\nWe will include a dedicated section discussing the limitations and broader implications of our method. This will address scalability and overhead concerns( as noted in points 2 and 3 above), clarify that the method is designed for the fine-tuning scenario and would likely not work as well in pretraining, as its success relies on knowledge transferability between pretraining and fine-tuning tasks. We will also highlight that the method benefits from model overparameterization, which is a key factor in its effectiveness. We will include a dedicated section discussing the limitations and broader implications of our method, as follows.\n\nOur proposed method NanoAdam introduces minimal computational overhead. Specifically, for each weight matrix in each layer, we first flatten the matrix and divide it into subgroups (chunks), then apply bottom-k selection within each subgroup. This process is applied uniformly across both convolutional and MLP layers. The main computational cost arises from the bottom-k operation, which has a time complexity of O(k \\log k).\n\nThanks to its layer-wise and parameter-wise design, the method is naturally scalable to larger models and remains compatible with modern hardware acceleration and parallel training frameworks. It avoids conflicts with model parallelism and pipeline layers, making it practical for contemporary large-scale architectures.\n\nHowever, NanoAdam the method has several limitations. Its effectiveness relies heavily on knowledge transferability and overparameterization. When the pretraining and finetuning tasks are well-aligned, the method helps avoid catastrophic forgetting and effectively leverages the plasticity of small weights to adapt to new tasks. In contrast, when there is limited similarity between tasks, the method may underperform compared to full-update optimizers like Adam.\nMoreover, the method benefits significantly from model overparameterization. As demonstrated in our experiments on vision tasks, scaling from a smaller model (e.g., ResNet-18) to a larger one (e.g., ViT-Large) results in improved overall performance and reduced forgetting. This suggests that NanoAdam the method is particularly well-suited for large, overparameterized models."}]

Author rebuttal:

Meta-review: The paper analyses the correlation of small weights and gradients and identifies that they are correlated - therefore they propose an efficient adam variant which only updates small weights in stead of all. Leading to less forgetting as well as improved performance. 

Strengths: Paper provides nice empirical evidence for their claims, method is interesting, and paper is easy to follow and nicely structured. 

Weaknesses: [18] already proposes a very similar method, and this paper only proposes small but potentially important changes. Contributions are not entirely clear, and the rebuttal by the authors is not totally satisfactory in my opinion. Nevertheless, the reviews are all positive and mostly concerns were adressed during the rebuttal. 

This is definitely a borderline paper for me, although I lean towards acceptance.
Decision: Accept (poster)