Title: Curvature Tuning: Provable Training-free Model Steering From a Single Parameter
Venue: NeurIPS 2025
Accepted: yes
Keywords: pretrained models

transfer learning

fine-tuning

decision boundary

robustness

generalization
Abstract: The scaling of model and data sizes has reshaped the AI landscape, establishing finetuning pretrained models as the standard paradigm for solving downstream tasks. However, dominant finetuning methods typically rely on weight adaptation, often lack interpretability, and depend on heuristically chosen hyperparameters.  In this paper, we take a different perspective and shift the focus from weights to activation functions, viewing them through the lens of spline operators. We propose Curvature Tuning (CT), an interpretable and principled steering method that modulates a model's decision boundary by injecting a single hyperparameter into its activation functions. We show that CT provably adjusts model decision boundary curvature and, more fundamentally, projects a model onto a space of smooth functions---thereby complementing current finetuning methods, whose effect lies primarily in feature adaptation. Making this hyperparameter trainable gives rise to a novel and highly parameter-efficient finetuning method. Empirically, CT improves both generalization and robustness. For example, it boosts downstream accuracy of ResNet-50/152 by 8.59\%/8.34\% over linear probing and 4.64\%/1.70\% over LoRA across 12 datasets, and improves robust accuracy on the $\ell_{\infty}$ benchmark from RobustBench by 1032.64\%/1494.46\%. Our code is available at https://github.com/Leon-Leyang/curvature-tuning.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a method for adapting pre-trained neural networks to\ndownstream tasks through the modification of activation functions. The main\nadvantage of their method, in comparison to existing popular techniques such as\nLoRA, is its parameter efficiency, introducing at most two additional parameters\nper activation layer. The method is motivated by its ability to preserve\nnon-zero curvature when applied to neural network blocks that conform to the\n'max-affine spline function' class. The method is evaluated on two distinct\narchitectures: ResNets and Swin Vision Transformers. The authors compared two\nvariants of their proposed method (CT and Train CT) against LoRA. Train CT\nexhibited superior performance in the ResNet experiments, whereas LoRA was more\neffective when applied to Swin Vision Transformers.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- The paper is well written, and the CT method is elegantly explained. I enjoyed reading Sections 2 and 3 in particular!\n- The method is very relevant to the community, as the ability to efficiently\n  fine-tune models is a key requirement for many applications.\n- Both proposed methods are interesting, with CT method potentially allowing\n  incredibly cheap fine-tuning, and Train CT being orders of magnitude more\n  parameter efficient vs. existing methods.\n\nWeaknesses:\n\n- Theorem 3.1 might benefit from extra clarity on the exact meaning of\n  'non-vanishing curvature'. Is it lower-bounded by something sensible? If so,\n  how does the bound behave as we vary $\\beta$/$c$/model size?\n- Interpretability is highlighted extensively in the abstract/introduction, but\n  in what exact sense, nor how this is useful in practice is not really discussed.\n- Experiments are only conducted on image tasks, e.g. there are no language\n  benchmarks.\n- Results are disappointing: while the method shows improvement vs. LoRA on the ResNet architecture (Table\n  1), it does not show an improvement on the Swin ViT (Table 3).\n- Invalid baselines: the benefit of Train CT over LoRA is deceptive, since LoRA\n  is only rank 1, and $\\alpha$ has not been tuned.\n- Authors mention that CT performs poorly with Swin, potentially because the\n  pretraining dataset they used, Imagenette, is small. This hypothesis can be\n  tested by e.g. pretraining Swin on a larger dataset (ImageNet perhaps). Or\n  alternatively by finding an pretrained image ViT that uses SiLU or ReLU.\n\nOverall, while the Methods were well presented, I unfortunately found the experimental results unconvincing, and that they hinder the quality of paper.\n\nOther smaller points:\n\n- The paper introduces aconyms without defining them (e.g. DNs (line 90), MASO (line 141), VQ (line 126))\n- typo line 253: \"achieves better performance then LoRA\"\n- missing error bars in Table 1 and 3.\n\nQUESTIONS:\n- The paper emphasizes interpretability in the abstract and introduction as a\n  key advantage of the CT method. However, it was not clear to me in the\n  experiments how interpretability was demonstrated. Could you give a specific\n  example of how the claimed interpretability is helpful when fine-tuning on\n  realistic problems?\n- Can CT be used e.g. for GRPO? Can CT be used for language model fine-tuning?\n  It would be great to see some experiments in this context.\n- You mentioned that you had to pretrain Swin from scratch because it used GELU\n  and not ReLU. You also mention that SiLU is compatible with CTU, so can you\n  perform some experiments fine-tuning on a SiLU model? (E.g. Qwen3 and Llama\n  3.2 use SiLU)\n- You replace ReLUs with CTUs with $c=0.5$. Why this value in particular? Is this\n  value in some sense the closest to ReLU? It is not clear to me despite looking\n  at Figure 3.\n- Can you add results to Table 1 for LoRA, where you have tuned $r$ and\n  $\\alpha$? This would allow us to frame the performance of CT correctly.\n\nLIMITATIONS:\nAuthors are missing a few limitations. Around their experiments, in particular\nthey are missing experiments with other modalities (e.g. text), and larger scale\nexperiments. Their method appears limited in that it is only better than (untuned) LoRA on\nthe ResNet.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel perspective on fine-tuning by shifting the focus from conventional weight adaptation to activation function modulation—an often overlooked yet critical component in determining the model's expressivity and decision boundary.\nTo this end, the authors propose Curvature Tuning (CT), a method designed to finetune the curvature of the activation functions. CT theoretically ensures smoother decision boundaries and empirically demonstrates improved generalization and robustness across various downstream tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n1.\tNovel Perspective on Model Steering\n\nThe paper proposes a fundamentally new fine-tuning paradigm by modulating activation functions, as opposed to the conventional weight-centric approaches. This enables direct control over the curvature of decision boundaries and, by extension, the model's hypothesis space.\n\n2.\tSound Theoretical Grounding\n\nThe proposed method is underpinned by a rigorous theoretical framework. Specifically, it interprets deep neural networks as compositions of max-affine spline operators (MASO) and formulates Curvature Tuning (CT) as a projection onto a smooth function space, thereby providing analytical clarity and mathematical justification.\n\n3.\tParameter Efficiency\n\nThe method introduces minimal overhead: the Steering CT variant introduces only a single hyperparameter, while the Trainable CT variant requires significantly fewer parameters than rank-1 LoRA, yet achieves superior performance.\n\n4.\tInterpretability\n\nThe impact of curvature control is both mathematically tractable and visually interpretable. The adjustment of the 棺 parameter provides an intuitive handle for regulating the non-linearity and expressive capacity of the model.\n\n5.\tStrong Empirical Performance\n\nCT consistently outperforms established baselines such as linear probing and LoRA across 12 diverse downstream classification tasks. It also demonstrates marked gains in robustness, achieving state-of-the-art results on robustness benchmarks such as RobustBench.\n\n\nWeaknesses\n\n1.\tLimited Architectural Diversity in Evaluation\n\nWhile the theoretical foundation of Curvature Tuning (CT) leverages Max-Affine Spline Operators (MASOs), which generalize a broad class of activation functions including ReLU and Leaky ReLU, the empirical validation is confined to ResNet and Swin Transformer backbones. To substantiate the architectural generality of CT, further evaluation on diverse model families—such as VGG, GoogLeNet, MobileNet, DenseNet, or WideResNet—would be beneficial, even if not complete.\n\n2.\tIncomplete Robustness Evaluation\n\nWhile Table 1 provides a comprehensive comparison across baseline methods, the robustness experiments lack inclusion of key variants such as Trainable CT and LoRA. This omission makes it difficult to fully assess the robustness advantage of Curvature Tuning (CT) under distributional shifts. Including these comparisons would strengthen the empirical claim of CT’s superior robustness.\n\nQUESTIONS:\n1.\tLimited Architectural Diversity in Evaluation\n\nTo what extent does Curvature Tuning (CT) generalize across diverse neural network architectures beyond ResNet and Swin Transformers? Have the authors considered evaluating CT on a broader range of model families (e.g., VGG, MobileNet, GoogLeNet, DenseNet, or WideResNet) to substantiate its architectural robustness and flexibility?\n\n2.\tIncomplete Robustness Evaluation\n\nWhile Table 1 presents a broad comparison, why are Trainable CT and LoRA omitted from the robustness experiments? Would including these baselines help more rigorously validate CT’s claimed advantage under distributional shifts?\n\n3.\tLimited Experimental Scope\n\nCan the authors clarify whether Curvature Tuning (CT) is applicable beyond vision tasks? Have they considered evaluating its generalization performance in other domains such as NLP, multimodal learning, or reinforcement learning to demonstrate broader applicability?\n\n4.\tAdditional Clarification on Class Imbalance\n\nHow does CT perform under class-imbalanced conditions? Has its robustness to skewed label distributions been empirically evaluated, and if not, would this be a worthwhile direction to explore given real-world dataset imbalances?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Curvature Tuning (CT), a novel method for steering pretrained models by modifying their activation functions instead of their weights. Grounded in spline theory, CT uses a parameter, $\\beta$, to modulate the curvature of the model's decision boundary. The paper presents a training-free \"steering\" version and a highly parameter-efficient \"Trainable CT\" for finetuning, demonstrating improved generalization and robustness\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n*  The approach of tuning activations is a fresh perspective in model adaptation. Its foundation in spline theory provides a clear, interpretable mechanism for controlling model curvature, unlike more heuristic methods.\n\n* Trainable CT achieves impressive results with significantly fewer parameters than competing methods. For instance, on ResNets, it outperformed LoRA while using only 11% to 59% of the trainable parameters.\n\n* The method delivers substantial gains in both generalization and robustness on ResNet architectures. The improvements on downstream accuracy and against L infinity adversarial attacks are particularly noteworthy.\n\nWeakness:\n\n* The method's theoretical guarantees are tied to ReLU-based networks. Applying it to models with other common activations, like GELU, required modifying the base architecture for the experiments, which may limit its off-the-shelf utility.\n\n* While effective on CNNs, Trainable CT underperformed LoRA in the transformer-based experiments. This raises questions about its effectiveness in more complex, attention-based architectures.\n\n* The non-trainable steering version of CT requires a grid search to find the optimal $\\beta$ for each task, adding a tuning step that finetuning methods avoid.\n\nQUESTIONS:\n* $\\beta$ and c are often U-shaped. Is this an emergent property you find desirable? Have you tried regularizing the training to explicitly encourage or discourage this distribution?\n\n* Your motivation for the CTU was to mitigate a mean shift from simpler formulations. Could you provide an ablation study in the appendix that shows the negative impact of this shift in deep networks?\n\n* Given that Trainable CT used far fewer parameters than LoRA on transformers, do you think the performance gap could be closed by allocating a moderately higher parameter budget to CT?\n\nLIMITATIONS:\nIn the conclusion, the authors identify their work's primary focus on ReLU-based networks as a limitation, stating that while CT is compatible with activations like Softplus or SiLU, this area \"warrants further study\". They also note that a detailed, layer-wise analysis of the learned $\\beta$ and c parameters in Trainable CT was not performed and remains an avenue for future research to understand representation and abstraction.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces and studies curvature tuning (CT) as a novel paradigm to parameter-efficiently finetune pretrained models. Instead of finding direct modifiers over model weights, the authors suggest (learnable) modifications on the utilized activation functions. In controlled scenarios, the authors showcase provable adjustments to model decision boundaries to facilitate generalization capabilities, highlight complementarity with existing PEFT techniques, and showcase high parameter efficiency even when compared to rank 1 LoRA.\n\nSTRENGTHS AND WEAKNESSES:\n__Strengths:__\n* The paper is really well written and structured.\n* To the best of my knowledge, attacking parameter finetuning as a function of activation function tuning is novel, and incredibly interesting.\n* Beyond the novelty, this paper offers a very principled perspective into the efficacy of curvature tuning, and highlight the complementary/orthogonality to existing finetuning methods.\n* On its own, CT is also really parameter efficient (in parts significantly below even rank 1 LoRA).\n* The experiments provided are sufficient to convince of the importance and potential general efficacy of curvature tuning for model finetuning.\n\n__Weaknesses:__\n\nMy only gripe with this paper is the depth of comparative experiments testing CT on more models, activation functions, datasets and domains and comparing against finetuning techniques beyond LoRA (and even here, ideally also contrasting against higher rank variants). Other than that,, I am rather hard pressed to find any other major issues with this work - I find it a really interesting read, and potentially a highly impactful contribution to the model finetuning domain.\n\nQUESTIONS:\nSee Weaknesses.\n\nLIMITATIONS:\nExplicitly addressed in Discussions, offering sensible avenues for future research.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-06"}