Title: Pay Attention to Small Weights
Venue: NeurIPS 2025
Accepted: yes
Keywords: fine-tuning; LLM; efficient ML; gradient compression;
Abstract: Finetuning large pretrained neural networks is known to be resource-intensive, both in terms of memory and computational cost. To mitigate this, a common approach is to restrict training to a subset of the model parameters. By analyzing the relationship between gradients and weights during finetuning, we observe a notable pattern: large gradients are often associated with small-magnitude weights. This correlation is more pronounced in fine-tuning settings than in training from scratch. Motivated by this observation, we propose \textsc{NanoAdam}, which dynamically updates only the small-magnitude weights during fine-tuning and offers several practical advantages: first, the criterion is \emph{gradient-free}—the parameter subset can be determined without gradient computation; second, it preserves large-magnitude weights, which are likely to encode critical features learned during pre-training, thereby reducing the risk of catastrophic forgetting; thirdly, it permits the use of larger learning rates and consistently leads to better generalization performance in experiments. We demonstrate this for both NLP and vision tasks.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes NanoAdam, a gradient-free, memory-efficient optimizer for finetuning large pretrained models. Motivated by the empirical observation that small-magnitude weights often correspond to large gradients, the method restricts updates to low-magnitude weights. This approach is designed to reduce catastrophic forgetting, preserve pretraining features, and improve training efficiency.\n\nSTRENGTHS AND WEAKNESSES:\n- While the observed correlation between large gradients and small weights is interesting, the assumption that small weights are always the best candidates for updates is overly simplistic. In Adam-style optimizers, gradients are normalized by past variance; thus, even small weights with high gradients may not result in large updates. Moreover, weight magnitude is not always indicative of parameter importance, especially in transformers where scale and function can differ layer-wise.\n- NanoAdam dynamically estimates the subset of smallest weights at intervals via sorting (Bottom_k(|θ|)). Although this occurs infrequently (every m steps), it could still introduce overhead, particularly in large models with tens of billions of parameters. It is unclear whether this cost counteracts the claimed efficiency gains—especially in practice on large-scale LLMs.\n- Despite strong results on BERT and vision models, it’s unclear how NanoAdam scales to contemporary LLMs (e.g., Llama, Mistral) with model parallelism and pipeline layers. The dynamic masking may interfere with hardware-accelerated fused optimizers and distributed training frameworks, which are critical for LLM-scale deployment.\n- The paper’s writing quality is suboptimal and affects readability. For instance, in the introduction (lines 16–38), the authors discuss motivation and related challenges, but abruptly switch to “Our method…” without first introducing what the method is or how it addresses the problem. Such transitions feel disjointed, and similar structural issues appear throughout the paper, weakening the overall presentation and clarity.\n\nQUESTIONS:\nn/a\n\nLIMITATIONS:\nWhile the authors briefly address limitations in Section 2.2, the discussion primarily reads as an analysis of the method’s advantages rather than a critical evaluation of its shortcomings. It would strengthen the paper if the authors included a dedicated section explicitly discussing the limitations and broader implications of their approach, including aspects such as the generalizability of the optimizer, its computational efficiency, and potential trade-offs in different model architectures or training regimes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces NANOADAM, a new optimization technique aimed at improving the efficiency and effectiveness of fine-tuning large pre-trained neural networks. Rather than selecting parameters for update based on gradient magnitude (a common practice), NANOADAM selects parameters with small absolute weight magnitudes, arguing that these are more “plastic” and less likely to encode critical pretrained knowledge.\n\nThe authors observe a consistent hyperbolic correlation between gradients and weight magnitudes during finetuning (particularly in NLP and vision tasks), i.e., large gradients tend to correspond to small weights. They propose leveraging this correlation to select and update only small weights dynamically. NANOADAM avoids the need for gradient-based selection or error feedback mechanisms, improving memory efficiency and preserving pretrained representations. Apparently, this reduces catastrophic forgetting.\n\nExtensive experiments across NLP (GLUE) and CV (CIFAR-10 and Flowers102) tasks demonstrate that NANOADAM outperforms existing memory-efficient optimizers (MicroAdam, GaLore, AdamW-8bit) in terms of generalization performance, memory usage, and parameter shift.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\nNovel Insight into Sparse Finetuning: The paper presents a compelling observation—large gradients correlate with small weights—and builds a theoretically and empirically grounded argument for exploiting this in fine-tuning.\n\nThrough a teacher-student model, the paper provides a clean and illustrative theoretical explanation for why updating small weights can avoid catastrophic forgetting (Theorem 2.2), which supports the method’s design.\n\nEfficiency Without Gradient Use: NANOADAM does not rely on gradients for mask selection, enabling precomputation and reducing memory/computation overhead compared to methods like MicroAdam or DST.\n\nClear Ablations: The ablation studies (Figure 4) directly compare small weights vs large weights vs random weights, and small weights vs large gradients, showing consistent gains for the proposed strategy.\n\nExcellent Empirical Results:\n\nTable 2 shows that NANOADAM performs best across nearly all tasks for BERT-base, BERT-large, and OPT-1.3B.\n\nTable 3 shows lower memory usage.\n\nTable 4 and 5 demonstrate its ability to reduce catastrophic forgetting and parameter drift during continual learning.\n\nBroad Applicability: Evaluated across both NLP and CV tasks, with results on transformer and CNN architectures, making the claims more generalizable.\n\nWeaknesses:\n1. No Comparison to LoRA-type PEFT Methods: While LoRA is mentioned in the related work, no head-to-head comparisons are presented. Although LoRA adds parameters, it is a strong PEFT baseline and should have been benchmarked.\n\n2. Density Scheduler Sensitivity: While the density scheduler and update intervals are mentioned, the method might be sensitive to these hyperparameters. The default choices and tuning ranges could have been discussed in more detail (e.g., effect of mask interval m on performance).\n\nQUESTIONS:\n1. Often for FT we freeze a set of layers. How does the number of layers to freeze affect the framework? I am thinking simple convnets with a MLP on top. How do convolutional and MLP layers differ?\n\n2. Could the authors provide further clarification or experimental results on why NANOADAM does not perform well when training from scratch? Is it primarily due to the lack of structure in weight initialization?\n\n3. While LoRA is cited as a PEFT baseline, it was not included in the experiments. Could the authors justify this omission or include results in the appendix?\n\n4. While the paper justifies excluding error feedback mechanisms, did the authors test simpler forms of feedback (e.g., residual accumulation without reinsertion) to see if performance improves marginally without major cost?\n\n5.  How robust is NANOADAM to variations in m (mask interval) and d (density interval)? Some empirical sensitivity curves or robustness plots would be helpful.\n\n6. NANODAM was proven for small parameter model. Do we trust this holds for large 7B+ models?\n\nLIMITATIONS:\nThey acknowledge that small weights do not always fully overlap with large gradients (Section 2.2), and that their method is explicitly designed for finetuning rather than training from scratch.\n\nHowever, they could further expand on:\n\nPotential pitfalls in transfer across highly dissimilar tasks (e.g., general-domain pretraining to low-resource medical tasks).\n\nSensitivity to initial weight distributions—e.g., would pretrained weights that are uniformly sparse or quantized change NANOADAM's effectiveness?", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors observe an inverse association between gradient magnitude and weight magnitude. They use this to motivate a finetuning strategy that focuses on weights with small magnitude, with a dynamically updated mask selecting those smallest weights (\"NanoAdam\"). They investigate this strategy theoretically in a teacher-student setup that illustrates the merits of this criterion. In empirical experiments on large pretrained models, they then find that NanoAdam improves both finetuning generalization and catastrophic forgetting compared to alternative methods while having a smaller memory footprint.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\nThe paper was well-written and it was generally easy to follow the flow of logic. I also appreciated the fact that the authors integrated a theoretical and empirical investigation into the same phenomenon. Further, I thought the investigation into the association between gradient and weight magnitude was well-executed; I personally found this effect intriguing and I also appreciated the additional investigation into how this effect emerges over pretraining, as it helps shape the reader's intuition.\n\n**Weaknesses**\n\nMy primary concerns with the paper in its current form are:\n\n*1 Soundness of the theoretical investigation*\n\nAs noted above, I generally appreciated the theoretical investigation and thought it provided a useful simplified intuition (indeed, even as an empirical toy model, I think an approach like this is helpful). However, I think the authors should provide more detail on their setup. In particular, Theorem 2.2 does not provide sufficient detail: what does it mean that \"gradient flow learns f_ft\"? Is this in the limit of infinite data? How does this follow from [9] and which result are you referring to? The closest fit I could find was Theorem 2.3 in [9], but that talks about a relative comparison between iterative magnitude pruning and learning rate rewinding.\n\nTwo minor comments: I think it would be helpful to more concretely connect the \"representation preserving\" phenomenon to its resulting impact on catastrophic forgetting, i.e. how much worse does the network get at the pretrained task as a result of finetuning. At the moment, you only talk about the impact on the representation, but ultimately we care about the impact on model output. I think, with Lemma C.1, you should be able to make a statement about this, and I think that would help tie this argument together.\n\nFurthermore, I don't understand the following statement: \"Due to the implicit sparsity bias in overparameterized training, there often exists at least one sufficiently small neuron that has minimal impact on the pretrained representation.\" I agree that this is likely to happen in practice, but if I'm not missing anything, the implicit sparsity bias (if you're referring to works like [1-3]) usually refers to a sparsity in function space rather than weight space. I.e. you can cluster the neurons in a two-layer network into a sparse set of clusters but that doesn't mean the hidden layer is actually sparse. Could you elaborate on this point?\n\n*2 Contributions of the empirical investigation*\n\nIn terms of your empirical investigation, am I understanding correctly that reference [18] already proposed and evaluated the idea to update small weights? If I'm understanding the relative contributions correctly, the primary methodological contribution you are introducing is to dynamically update this mask, right? I appreciated the comparison you drew between those approaches in Figure 11 and I think it would be important to evaluate the impact of dynamical masks more broadly in Section 4 as well, by specifically comparing the dynamic mask to the static mask across the entire set of benchmarks (extending your analysis in Figure 11). More generally, I think it would be good to contrast the novel contributions made in this paper to those already known from this previous paper in more detail. For example, are your insights into the benefits of (even static) masks for catastrophic forgetting new?\n\nAdditionally, I think it would be useful to repeat the experiments in Section 3.2 several times and indicate standard deviations of the different learning curves. It is difficult to assess how noisy these observations are.\n\n**Conclusions**\n\nOverall, in its current state, I rate this paper as a borderline reject. However, I enjoyed reading it, think it is a worthwhile study, and highlight a number of questions below that would cause me to increase my score.\n\n**References**\n1. https://proceedings.mlr.press/v125/chizat20a.html\n2. https://proceedings.mlr.press/v99/savarese19a.html\n3. https://proceedings.neurips.cc/paper_files/paper/2022/hash/7eeb9af3eb1f48e29c05e8dd3342b286-Abstract-Conference.html\n\nQUESTIONS:\n1. Could you provide more detail for the proof of Theorem 2.2? Moreover, I think drawing a more explicit connection between Definition 2.1 and catastrophic forgetting (see my comments above) would be helpful. As part of that, it would also be useful to make Definition 2.1 more specific (e.g. how is the set of largest neurons selectedt? Is it the k largest neurons? Is it all neurons above a certain magnitude?\n\n2. Could you clarify your contributions relative to previous work and, in particular, reference [18]? In particular, how central is the introduction of dynamic masking to your overall contributions?\n\n3. If possible, could you add a comparison for the experiments in Section 4 to static masks as used in [18]?\n\n4. Could you add standard deviations to the learning curves?\n\nI can imagine that new experiments may not be realistic to run within the period of the rebuttal. Addressing the first two questions would be sufficient for me to increase my score though I believe addressing the latter two questions would substantially improve the paper further.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

comments: SUMMARY:
The paper presents a unique finding that during fine-tuning of large models, the weights with small magnitude often get large gradients. Based on this observation, the paper proposes a modified Adam optimizer to dynamically update only the small-magnitude weights during fine-tuning in order to reduce the cost of fine-tuning large pretrained neural networks. The paper presents results to support this method on both NLP and vision tasks.

STRENGTHS AND WEAKNESSES:
Strengths:
The paper takes an interesting approach to restrict the fine-tuning to parameters with small absolute magnitude.
Demonstrates superior or equivalent performance with lower memory usage.


Weaknesses:
It can be used for fine-tuning and not for training from scratch, since the core hypothesis will not hold true. 
No discussion on overfitting vulnerability due to the reduction in trainable parameters.

QUESTIONS:
The writing could have been better organized.
Due to the significant reduction in memory usage and updatable parameters, is the proposed method also prone to overfitting?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 2
DATE: 2025-07-05