Title: Optimization Inspired Few-Shot Adaptation for Large Language Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Large Language Models

Opitmizer learning

Few-shot adaptation
Abstract: Large Language Models (LLMs) have demonstrated remarkable performance in real-world applications. However, adapting LLMs to novel tasks via fine-tuning often requires substantial training data and computational resources that are impractical in few-shot scenarios. Existing approaches, such as In-context learning and Parameter-Efficient Fine-Tuning (PEFT), face key limitations: In-context learning introduces additional inference computational overhead with limited performance gains, while PEFT models are prone to overfitting on the few demonstration examples. In this work, we reinterpret the forward pass of LLMs as an optimization process, a sequence of preconditioned gradient descent steps refining internal representations. Based on this connection, we propose Optimization-Inspired Few-Shot Adaptation (OFA), integrating a parameterization that learns preconditioners without introducing additional trainable parameters, and an objective that improves optimization efficiency by learning preconditioners based on a convergence bound, while simultaneously steering the optimization path toward the flat local minimum. Our method overcomes both issues of ICL-based and PEFT-based methods, and demonstrates superior performance over the existing methods on a variety of few-shot adaptation tasks in experiments.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Optimization-Inspired Few-Shot Adaptation (OFA), an approach for adapting LLMs to new tasks with limited data. The paper reinterpret the forward pass of LLMs as a sequence of preconditioned gradient descent steps and leverage this insight to propose a parameter-efficient adaptation method that modifies only the LayerNorm parameters. OFA combines three objectives: cross-entropy loss, a step-ratio objective that improves optimization efficiency by promoting smoother convergence paths, and a sharpness minimization objective that enhances generalization by steering optimization toward flat minima. The method is theoretically grounded with convergence and generalization bounds, while being extremely parameter-efficient (using only 0.27M parameters). Extensive experiments across multiple datasets and model architectures demonstrate that OFA consistently outperforms existing ICL and PEFT methods, achieving 4-10% improvements over baseline approaches without introducing additional inference overhead.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n- The paper provides an interesting theoretical interpretation of transformer layers as implementing gradient descent steps, creating a good foundation for their method.\n- OFA achieves impressive results while updating only LayerNorm parameters (0.27M parameters), making it significantly more efficient than competing approaches.\n- The experiments cover multiple datasets and model architectures (Llama2-7B, Llama3-8B, GPT2-XL), demonstrating consistent performance improvements.\n- The paper provides theoretical analysis connecting their objectives to convergence speed and generalization bounds.\n\n### Weaknesses\n- citation [35] on line 27 doesn't adequately support the claim that PEFT methods suffer from overfitting in few-shot scenarios. This weakens one of their motivating arguments.\n- The Hutchinson approximation for sharpness estimation adds implementation complexity that isn't fully addressed in terms of computational overhead during training.\n- Some of the theoretical results rely on assumptions like locally Lipschitz gradients and quadratic approximations that may not always hold in practice for complex LLMs.\n\nQUESTIONS:\n- In your experimental evaluation, you use IA3 as one of the baseline methods for comparison. However, the original IA3 paper (citation [35]) introduces additional training components beyond the parameter-efficient architecture, specifically Unlikelihood Training and Length Normalization, which are designed to address challenges in few-shot learning scenarios. Did you implement these additional loss terms when training your IA3 baseline? If not, do you believe including these components might narrow the performance gap between IA3 and your proposed OFA method?\n- The paper introduces novel loss terms (step-ratio objective and sharpness minimization) that are theoretically motivated to improve optimization efficiency and generalization. Did you consider applying these loss terms to other parameter-efficient fine-tuning methods such as IA3 or LoRA to isolate their effect?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new few-shot adaptation algorithm that adjusts the parameters of layer-norm on few-shot data. This approach is motivated by the gradient descent properties of transformer layers to encourage smooth gradient behaviors across layers (step ratio regularization). The proposed approach also aims to learn flat minima using the Hutchinson approximation method. Empirical results show the proposed approach outperforms other baselines: ICL, soft prompt, label anchor, task vector, IA3, I2CL, and LoRA. In particular, it outperforms LoRA by a large margin while using less parameters. Ablation study through probe classifier analysis shows the importance of step ratio and sharpness regularization.\n\nSTRENGTHS AND WEAKNESSES:\nStrength\n\n1. The paper is well-motivated. The proposed regularization terms on step ratio and loss sharpness are interesting and technically sound.\n2. The proposed approach demonstrates strong empirical results, outperforming LoRA by a large margin while using less parameters.\n3. The paper carries out comprehensive ablation studies and visualization to demonstrate the value of step ratio and sharpness regularization.\n\nWeakness\n\n1. Loose connection between regularization and parameter efficient fine-tuning. The proposed approach is comprised of cross entropy loss on the layer norm parameters, and two regularization terms. It seems these regularization terms could be applied just as easily to other PEFT (or any training) methods, such as soft prompt or LoRA.\n2. Scaling with exemplar size. It is unclear how the proposed approach would work if we increase the number of exemplars, and at what point it would become limited by the representation of the layer-norm parameters.\n\nQUESTIONS:\n1. Are the proposed terms applicable to any SFT or PEFT training algorithm?\n\n2. How many exemplars are used in the test sets? How the proposed approach would work if we increase the number of exemplars, and at what point it would become limited by the representation of the layer-norm parameters.\n\n3. How to set the hyperparameters of different loss terms and is the model sensitive to these parameters?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Optimization-Inspired Few-Shot Adaptation (OFA), an approach for adapting LLMs to new tasks using only a few demonstration examples, addressing key limitations in existing few-shot adaptation methods. The paper builds upon previous finding which view the forward pass of LLMs as gradient descent optimization. The key insight is that LayerNorm parameters can be finetuned per task to learn preconditioning matrices for each gradient descent step, which happens during the forward pass. The method introduces two objective terms: one for optimization efficiency (step-ratio minimization) and another for generalization (sharpness minimization via Hutchinson approximation). The authors evaluate OFA across multiple classification tasks and LLM architectures, demonstrating consistent improvements over strong baselines.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\nOverall, the contribution of this work is solid.\n\n(Novelty): To the best of my knowledge, this line of research is novel. The authors rely on optimization and in-context learning theory to guide their design choices. Importantly, the paper also includes compelling empirical evidence that is consistent with what theory predicts.\n\n(Strong empirical performance): OFA demonstrates clear improvements over baselines across multiple classification datasets and LLM architectures.\n\n### Weaknesses\n\nThe practical usability of the approach is still unclear to me since the paper lacks some details about the experimental validation.\n\n(Lack of hyperparameter ablation): The method introduces two additional hyperparameters (λ₁, λ₂) requiring grid search across a substantial range (6 different values each), potentially making the method less practical and adding tuning burden. An ablation study over the two hyperparameters seems crucial to me. It would be informative to clarify whether the values of these hyperparameters are specific for each experiment or shared across all.\n\n(Lack of details about the training setup / overhead): Another missing detail is the runtime during training/finetuning. It would be interesting to get insights on the difficulty of the optimization when considering the two loss terms.\n\nQUESTIONS:\n- Could the authors comment on the sensitivity of the OFA hyperparameters?\n- What is the overall cost of finetuning an LLM with OFA? How does it compare to the baselines?\n- In the transition from equation L144 to the preconditioning matrix formulation (equation L147), a bias term $\\frac{\\mu_t}{\\sigma_t}$ appears to be dropped. Could the authors elaborate on this step?\n\nLIMITATIONS:\nYes, but only in Appendix.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 4, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes Optimization-Inspired Few-shot Adaptation (OFA), a lightweight way to dapat a language model with only a small portion of labeled examples. The paper views the transformer block as one step of gradient descent and uses the gradient descent to adapt the model. OFA uniquely adapts an LLM by learning the LayerNorm scales as diagonal pre-conditioners that rescale each layer’s gradient, which is different from the previous optimization-inspired method. Only the scale parameters are fine-tuned. After training, the adapted model runs with zero extra tokens. The experimental results showed that OFA outperforms ICL-based and PEFT-based methods while requiring only fewer parameters and matching zero-shot latency.\n\nSTRENGTHS AND WEAKNESSES:\nFor quality, the paper conducted thorough experiments comparing OFA with ICL-based and PEFT-based methods and other optimization-inspired methods. The tasks however are limited to single sentence classification, no generation, or name entity recognition. For clarity, the paper explains clearly how OFA is compared to ICL and PEFT, and the algorithm and figures are easy to follow. The work showed that tweaking only LayerNorm scales can beat ICL and PEFT. Offer practical insights for the community. It is also the first to turn the LayerNorm scales into a preconditioner and bridges the optimization-inspired methods with PEFT. Other weaknesses are discussed in the Questions section.\n\nQUESTIONS:\n1. Theorem 3.1 requires $\\rho_t < 1$ for local contraction. However the paper does not provide and proof or intuition for why we can assume this. Can you provide a theoretical argument or a lemma with conditions on $\\eta$ and $P_t$ that help to guarantee this? Or is this possible to supply some empirical evidence on demonstrating $\\rho_t < 1$ in most of the cases.\n\n2. Please give a precise definition of $Z^*$ and state where the symbol first appears.\n\n3. Did the paper look at tasks other than single-sentence classification? Does the method work for other tasks, such as generation, or name entity recognition? \n\n4. When do the authors plan to release the implementation and pre trained checkpoints for reproducibility?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-03"}