Curvature Tuning: Provable Training-free Model Steering From a Single Parameter
Authors: Leyang Hu
Matteo Gamba
Randall Balestriero
Venue: NeurIPS 2025 poster ()
Keywords: pretrained models
transfer learning
fine-tuning
decision boundary
robustness
generalization
Primary area: deep_learning
TL;DR: We introduce Curvature Tuning (CT), a model steering method that improves generalization and robustness of pretrained models by provably modulating model decision boundary curvature.

Abstract: The scaling of model and data sizes has reshaped the AI landscape, establishing finetuning pretrained models as the standard paradigm for solving downstream tasks. However, dominant finetuning methods typically rely on weight adaptation, often lack interpretability, and depend on heuristically chosen hyperparameters.  In this paper, we take a different perspective and shift the focus from weights to activation functions, viewing them through the lens of spline operators. We propose Curvature Tuning (CT), an interpretable and principled steering method that modulates a model's decision boundary by injecting a single hyperparameter into its activation functions. We show that CT provably adjusts model decision boundary curvature and, more fundamentally, projects a model onto a space of smooth functions---thereby complementing current finetuning methods, whose effect lies primarily in feature adaptation. Making this hyperparameter trainable gives rise to a novel and highly parameter-efficient finetuning method. Empirically, CT improves both generalization and robustness. For example, it boosts downstream accuracy of ResNet-50/152 by 8.59\%/8.34\% over linear probing and 4.64\%/1.70\% over LoRA across 12 datasets, and improves robust accuracy on the $\ell_{\infty}$ benchmark from RobustBench by 1032.64\%/1494.46\%. Our code is available at https://github.com/Leon-Leyang/curvature-tuning.

Curvature Tuning: Provable Training-free Model Steering From a Single Parameter
Leyang Hu" Brown University edu
Matteo Gamba*t
Randall Balestriero Brown University
KTH mgambaOkth.se
eendillbbalettrrrroooroo.m
Abstract
The scaling of model and data sizes has reshaped the Al landscape, establishing finetuning pretrained models as the standard paradigm for solving downstream tasks. However, dominant finetuning methods typically rely on weight adaptation, often lack interpretability, and depend on heuristically chosen hyperparameters. In this paper, we take a different perspective and shift the focus from weights to actívation functions, viewing them through the lens of spline operators. We propose Curvature Tuning (CT), interpretable and principled steering method that modulates model's decision boundary by injecting single hyperparameter into its activation functions. We show that CT provably adjusts model decision boundary curvature and, more fundamentally, projects model onto a space of smooth ceteiionssthereee
complementing current finetuning methods, whose effect lies primarily in feature adaptation. Making this hyperparameter trainable gives rise to a novel and highly parameter-efficien finetuning method. Empiri- cally, CT improves both generalization and robustness. For example, it boosts downstream accuracy of ResNet-50/152 by 8.59% over linear probing and 4.64%/1. 1.70% over LoRA across 12 datasets, and improves robust accuracy the foo benchmark from RobustBench by 103...44//1444.... Our code available
at http:///gthooocccooocoom///////r
Introduction
The scaling of model and data sizes has given rise foundation models, such Llama3 [1] for natural language processing (NLP), DINOv2 2 for computer vision (CV), CLIP 3 and SigLIP for multimodal tasks, and OLe S for embodied agents. These models have shown remarkable capabilities, uccelerating a paradigm shift artificial intelligence: transitioning from training task- specific models from scratch to leveraging models pretrained on large datasets and finetuning them for downstream applications.
Full finetuning, the process of steering pretrained model by adapting all its parameters to down- stream datasets, was once the primary approach for transferring knowledge. While it effectively enhances generalization [6] and robustness (7), it is computationally expensive at large model scales. To mitigate this, parameter-efficient finetuning (PEFT) methods such as Serial Adapter [8] and LoRA 99 have been introduced, which finetune only small subset of parameters. However, these approaches usually lack interpretability and principled design. For instance, they treat the model as black box. making it unclear how the model is steered for downstream tasks. Consequently. they usually rely on heuristic choices- as LoRA's rank, placement, and initialization with minimal
'Equal contribution.
Work done at Brown University.
'We use steering general term for tuning model, including training- and non-training- based methods. We use finetuning to refer specifically to steering methods that adapt the model's parameters training.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Initial Model
Pretrained Model
B=1.0 (ReLU)
f=0.9
A=0.5
Pretrain
Supervired
Sel-sspeeviiee
Regression
p-0.5
0.45
0-1.0 (BeLU))
0.30
0.15
0.50
Figure I: Illustration Curvature Tuning (CT) on classification (top) and regression (bottom) tasks. The pretrained model for classification is 3-layer MLP with hidden width 20 trained for 2000 steps: for regression, it a 9-layer MLP with hidden width 64 trained for 20000 steps. CT steers pretrained model by replacing ReLUs with 8-parameterized activation function and tuning from to 0, effectively modulating the model's decision boundary curvature.
theoretical guidance. This leads to a natural question: how can we construct principled steering solutions addressing both efficiency and interpretability?
In this work, we answer the question by introducing novel model steering perspective. We observe that despite differences in specific forms, existing finetuning methods all share a focus on adapting model weights-either by introducing new ones or updating existing ones. However, one critical component of the model has been largely overlooked: the activation functions (e.g., ReLU), which are responsible for the model's nonlinearity and, ultimately, expressivity [TO
Grounded in the spline interpretation of deep networks [2
we propose Curvature Tuning (CT), steering method (denoted as S-CT) that provably modulates model's decision boundary curvature by injecting single hyperparameter / into the activation function, as shown in Fig. [ Additionally, allowing / to be trained leads novel finetuning method (denoted as T-CT). We highlight four key advantages of CT below:
13
CT is interpretable and principled. We show that CT provably modulates the curvature of the model's decision boundary with as few as only one hyperparameter.
CT complements current finetuning methods. More essentially, while current finetuning methods adapt features, CT projects the model to space smooth functions.
CT is highly parameter-efficient As a steering method, S-CT introduces only one paraaeerrr per network. As finetuning method, T-CT still uses significantly fewer parameters than LoRA with rank one, requiring only 0.58% to 59.09% of the parameters used by LoRA in our experiments.
CT improves both generalization and robustness. For example. T-CT boosts transfer accu- racy of ResNet-50/152 by 8.59% over linear probing and 4.64% over LoRA with rank across 12 downstream datasets. S-CT improves robust accuracy of ResNet-50/152 by 1032.64%//1444.666 on the Eoo benchmark from RobustBench.
In summary, our key contributions are both theoretical and empirical. Theoretically. propose CT and show that it provably modulates the model's decision boundary curvature. by projecting the model onto space of smooth functions. Empirically, and T-CT as a finetuning method, demonstrating improved generalization across six models and 12 downstream datasets, well as robustness gains the RobustBench benchmark [14].
introduce S-CT as a steering method
The remainder of this paper is organized as follows: Section 2 reviews current finetuning techniques and introduces relevant spline concepts, the foundation for our method. Section 3 details our proposed method and theoretical guarantees. Section 4 presents experimental results, and Section summarizes our findings and potential future directions.

---

2
Background
This section first reviews current finetuning techniques and their limitations (Section 2.1 then introduces spline theory and its connections to deep networks as the foundation for CT (Section 2.2
2.1 Current finetuning techniques and limitations
Finetuning refers to steering pretrained model improve its performance downstream tasks through training. Initially. the common practice was to continue training all model parameters process known as full finetuning. Notable examples include GPT [6] and DINO [15]. However, as model sizes have grown, full finetuning has become increasingly costly and often impractical, partic- ularly when downstream datasets small. Given these challenges, parameter- efficcien finetuning (PEFT) methods were developed to mitigate the cost while maintaining effectiveness.
We follow the categorization of Han et al. [6]. which groups PEFT methods into four main categories. Additive PEFT introduces additional trainable parameters to the pretrained model, training only these new parameters during finetuning. Examples include Serial Adapter [8]. Prefix-tuning [17). (IA)² 18]. and RoAd [19]. Selective PEFT identifies a subset existing parameters for finetuning, with examples such as U-Diff pruning and S-Diff pruning (20]. Reparameterized PEFT decomposes pretrained weights into low-rank matrices, finetuning only the low-rank components, which are converted back during inference; examples include LoRA [9] and DyLoRA (1]. Hybrid PEFT
combines multiple PEFT strategies, such as UniPELT 22 and S4 (23].
While PEFT methods differ in the parameters they update, they all adapt model weights and operate on learned features-an an approach that often relies on heuristic tuning. For example, LoRA requires decisions about adapter placement (24), rank [2], scaling [26). and initialization [27]. In contrast, as described Section CT introduces only a single hyperparameter into the activation functions that provably modulates the decision boundary curvature, offering more interpretable alternative that instead operates on the model's underlying function space without changing model weights.
The problem of learning low-curvature activation functions has been explored in the context of pretraining, where Srinivas al. (28 exploit a parametric version of Softplus. Our paper differs in two substantial aspects. First, we introduce more general parametric activation function (CTU) encompassing a wider family of non-linearities, which provides direct applicability to current network architectures and particularly transformers. Second, we are the first to propose modulating activation functions as novel PEFT direction, allowing to control downstream generalization and robustness without changing the pre-trained model's learned features.
2.2 The spline formulation of deep networks
In this subsection, we review relevant concepts splines, which provide a mathematical framework for understanding the relationship between piecewise-affine functions and deep networks (DN).
A spline function is continuous function IR," - R defined piecewise by polynomials. An affine spline function is a special case where each piece is defined by an affine mapping. Such a function be parameterized by three components: matrix A E RRxx representing the slopes of the affine mappings, vector b R representing the offsets, and partition Ω (wi....,4u wn) of the input space RD into R regions. For an input x E DR,,, the affine spline function is defined as:
R b(A,b, = Σ ((Ar,,, x) + b,) [
r=1
where 1 (xew.) is an indicator function that equals if x belongs to region w, and 0 otherwise.
max-affine spline function is special case of an affine spline function that does not need explicit knowledge of Q. Instead, its output is computed the maximum value over the affine mappings:
s s(A, b)(x) = max ((Ar,, x) + b,).
r=I... R
The key result underpinning our study that many DN layers-such as fully connected and convolu- tional layers, and convex piecewise-affing activations (e.g.. ReLU, max pooling, and maxout) can
3

---

(a) Pretrained
LoRA
(c) CT
Figure 2: Toy example illustrating how modulating model's activation functions steers decision boundaries curvature. The model is a 2-layer MLP with hidden width 7; (a) baseline pretrained for 4000 steps, then finetuned for another 4000 steps using (b) LoRA (r 1, 1) and (c)
Truinable CT. CT achieves near-optimal approximation by smoothing the decision boundary of the pretrained model, whereas LoRA only operates the model parameters, without changing the model's underlying geometry.
be exactly represented as max-afline spline operators and that DNs composed of such layers can be represented affine spline operators [13] (further details in Appendix (A).
Now that we have reviewed existing finetuning methods and their limitations, and introduced the necessary spline-based foundations, proceed to present proposed method in Section B
3 Curvature Tuning (CT): a provable method for model steering
In this section, we introduce our proposed method, Curvature Tuning (CT). We begin by motivating the benefits of modulating decision boundaries as model steering technique. Then, we dive into CT's construction Section 3.1 with implementation details Section 3.2 Additional theoretical intuition is provided in Section Extensive experimental validation is presented in Section J
Motivating example. Consider toy binary classification problem in R2a, whereby the optimal decision boundary separating two classes is given by the unit circle 5 {x E R², ||x||2 1}.
parameterized by the curve "
(cos2nt, sin 2xct), for E [o, 1]. Let (2) [+exp(=)
exp(e)
be the sigmoid. By definition of decision boundary, an optimal network f R22 + R should predict both classes with equal probability ((((rtt)) 0.5, Vt E [0, < f(r(t)) 0, Vt.
Focusing on the decision boundary, the approximation error e is given by the line integral = S. [f(x) fo [f(r((t)llll((()) For ReLU network (Eq. (1) computing the error
the regions 22 Ω n St yields 2n Σ ((-1)au(ta) (9t, (1)) + (-1)ex(tt=>)) (gr, (t)) for
12y1, where tx are the spline breakpoints pulled back from IR2 [0, 1]. r denotes which spline region the k-th segment [/x, [x+1] falls into, z (t) = (Arg. br))+br, <0}- 8k [4,, 4+1] is cos 25t + b., (full derivations in 2x
value flip point of (1) and gr, (1) = A 2 Ar,.
2nt
Appendix C.2). Assuming the network considered attains optimal approximation error > 0, then clear that 1x+1 -> lx = - The converse implication holds when the task changed from classification to regressing the unit circle. In this case, reducing the approximation error by adapting the model's weights (cither through PEFT or training larger model from scratch) will still result in an affine spline operator, for which > 0 and further reducing to O would require infinitely many neurons. This paper explores an orthogonal approach: by modulating the model's 's activation functions, one efficiently control its curvature and, in turn, that of its decision boundaries. thereby steering them toward optimality without modifying the model's weights. Fig. 2 iillustrates this phenomenon: methods such as LoRA implicitly tune the spline slopes and breakpoints, whereas modulating model nonlinearities changes the model's underlying geometry.
-withut
Here, operator simply refers to mapping between vector spaces (i.e., function RRD + R² obtained by concatenating K functions like

---

3.1 The 3i- --Veetor-Quantizattion (VQ) inference framework
This section builds upon the max-affine spline formulation from Eq. to construct model steering method operating on the model's 's activation functions. By inspecting Eq. (2) we observe that the mapping remains affine within each (implicitly defined) region where the pointwise maximum does not change. Specifically, for any input x where arg max,=1...R ((Ar,, x) + b,) remains constant, all such inputs belong to the same region, as they share the same affine mapping. The nonlinearity of the function arises when transitioning between these regions.
Smoothing the nonlinearity by smoothing the spline region assignment process. Instead of going from one affine mapping to another in an abrupt fashion (whenever crossing that hyperplane), one may consider smoother transition. There are two common practices achieve that goal.
We know that each unit a layer is a max- spline. The inference process of each unit can thus be decomposed into two steps:
1. VQ Inference Step (region selection): Determine the affine transformation that maximizes the VQ process. The decision is encoded in one-hot selection variable t E RR, where R is the number of input region partitions the max-affine spline function, and the r"-th entry is set to 1, where:
output, which can be viewed
arg max (AAr,, b,)) (E.
2. Computation Step (affine transformation): Compute the output of the neuron based on the selection variable t:
R
f(x) = Σ tr ((Ar,: x)xxx/00 b,). r=1
(4)
As discussed, the affine transformation selected in a hard manner, where only the transformation that maximizes the output is chosen. Alternatively, seft selection can be employed. in which the selection variable is no longer a one-hot vector but is inferred probabilistically. To formalize this,
we follow the probabilistic formulation from [29] and introduce the following regularized region selection problem, where the new selection variable & is computed as below:
R
t arg max
B
Σ tr ((Ar,,,) + b,) (1 B)H(()
(5)
EEAR
where H (t) denotes the Shannon entropy of the selection variable, and AR is the probability simplex in IR R The optimal solution 8 has the closed-form
(((AA...))+bb
8
exp
1-9
for r I....,R. R.
(6)
R 1==1 exp
xx)+b.))+bb))) 1-8
Using the computation step in Eq. and a ReLU activation function, switching from 1 to 3 = 0.5 is provably equivalent to replacing ReLU with the Sigmoid Linear Unit (SiLU). In the limit as 3 + O, the activation function becomes linear-thus making the entire input-output mapping of the network linear as well. 299
Smoothing the nonlinearity by smoothing the max. As previously mentioned, there is an alternative way to smooth the max-affine spline mapping from Eq. (2). Instead of relying on a soft region assignment, we can instead directly smooth the maximum function. already well known that smoothing the maximum operator leads to the log-sum-exp operator (i.e. Softplus). Hence, the mapping from Eq. (2) now becomes
(An.. x)),,))bb,) )
(1 - 3) In
Σ exp
(7)
1 1-
where we parameterized the mapping so that its behavior is akin to Eq. (5) a value of - 1 recovers the original affine spline activation, e.g., ReLU.
The crucial observation we make is that both parameterizations tend to shift the mean of the output of the unit either by negative factor (for Eq. (5) or positive factor (for Eq. (7)). This means
5

---

that in very deep models, varying 8 with either parameterization produces shift in the decision boundary or regression that cannot be recovered unless the parameters are trained once again. which we are trying to avoid. As result, as detailed in Section 3.2 our implementation combines the two parameterizations in a weighted manner to mitigate this bias, as Fig. ST illusttrates.
3.2 Implementation of CT
We begin by presenting the core activation that gives CT its expressive power-referred to as CT Unit (CTU). The activation is obtained by combining the two parameterizations discussed in Section 3.1 in order to mitigate the mean shift introduced by each parameterization individually:
5x
(1 c) c) In 1 exp ( [ ] (1 -1-
(1-3)
(x)c(x)= = c c.
(8)
I-
where 3 E (0, IF modulates the curvature, c C [o, 1] is the mixing coefficient, and (.) denotes the sigmoid function. This is essentially convex combination of reparameterized SiLU and Softplus:
B SiLU(x)= (1x)) x, 1 B
Softplus(x)
In [1 + exp (7x)]; 7
B
It is worth noting that CTU naturally encompasses broad family of functions. Particularly, it recovers SiLU (c = 1), Softplus (c = 0), as well as closely approximates GELU (c = and 3 = 0.64).
Steering vs Trainable CT. We provide two inplementations of CT differing in how CTU is applied. The first, denoted Steering CT (S-CT), replaces all ReLUs in the network with CTUs using fixed only single
c= 0.5 and a shared 3 E [0, 1]. This version is highly prammetr--ffcient-innddddee hypperparaeterr-ane does not require backpropagation, making it suitable as a steering method.
The second, referred to as Trainable CT (T-CT), also replaces all ReLUs with CTUs but assigns each output neuron its own trainable pair (3,c), c). optimized via backpropagation. This version serves as a finetuning method: while it introduces additional parameters, the increase is modest compared to methods like LoRA and yields competitive performance, as shown in Section 4.2 Code for both implementations is provided in Appendix D
3.3 Curvature Tuning operates as a projection
This section provides characterization of CT, by casting it projection of a ReLU network to space of smooth functions. All proofs are deferred to Appendix C.I
Theorem 3.1 (Informal). For a ReLU U network f RDD R with parameter W (collecting all weights and biases), for fixed e e [0, 1] and B3 E [0, 1), replacing every instance of ReLU with a CTU (Eq- { with hyperparameters B, is equivalent to projecting f to a smooth function fa,e with bounded gradients and curvature, while keeping W fixed. Importantly, for 0 < B < 1, fa.e enjoys higher local expressivity than f for the same parameter W, due to non con-vanishing local curvature.
To conclude, we observe how varying 3 modulates the curvature of the whole model function f and, in turn, of the model's decision boundaries. We begin by noting that for deep network f IR. R the decision boundary between any class i and j is given by {x E IRD gxx) == f((x) fx( = =]. for any j 1,..., K with * j. Particularly, g is itself deep network, sharing the same parameters as f up until the penultimate layer, after which the parameters are the vector W - W and the bias b - -bj. Importantly, when varying / while keeping all model parameters fixed, the Jacobian (x and the Hessian (²)( are respectively given by the gradients and Hessian of -x correpponiig to the post-activation output of the L- 1-th laier-weighted by W.i - W. Hence, modulating the nonlinearity of activation functions via 3 directly controls the curvature of both model function and its decision boundaries
Particularly, for (Eq. ( ). 3 O. the activation becomes linear. Since modern DNs (c.g. MLP, CNN, RNN) are composed of activation functions interleaved with affine layers, it follows
II practice, for numerical stability we use 1j = I-3+ and = -- where e = 10 allows the method remain well-defined 3
'In the following. unless specified, we will thus refer interchangeably to the curvature of a DN mapping and that of its decision boundaries whenever modulating nonlinearities via CT.

---

directly that the entire outuut mapping becomes affine when 3 - 0. In this setting, the curvature of the mapping- -asping-defined defined as the norm of its the original DN mapping (3 = 1) to the linear setting effectively modulates the network decision boundary curvature, reducing it continuously to zero the limit. For e < 1, / O, the model retains non-vanishing local curvature, while the mapping becomes smooth.
zero. As result, transitioning from
Enhancing model generalization and robustness with CT
In this section, we empirically validate the effectiveness of CT across multiple settings. We first demonstrate that S-CT improves generalization (Section 4.1p. while T-CT achieves improvement comparable LoRA with substantially fewer parameters (Section 4.2). Then we show that CT can improve the robustness of models through its implicit bias (Section 4.3). Finally, we demonstrate CT"s effectiveness on transformers despite only partial guarantees (Section 4.4). GPU and seed
details are provided in Appendix B
4.1 Improving generalization on downstream datasets with S-CT
In this subsection, we evaluate the effectiveness of S-CT in improving model generalization variety of downstream datasets. Specifically, we transfer ImageNet-pretrainee ResNet-18. ResNet-50, ResNet- 152 and VGG-11 models to 12 downstream tasks, including Arabic Characters 30 Arabic
Digits [3 Beans [32], CUB-200-2011 [33, DTD [34), FashionMNIST [35), FGVC-Aircraft [36),
Flowers 102 37 Food101 [38), and three subsets from McdMNNSSTPPatMMNN OCTMNIST, and DermaMNIST 39]. Each dataset split into
sets (details in Appendix B.T).
To apply S-CT. we replace all ReLUs in the backbone with CTUs, freeze the model, and train linear classifier on the penultimate layer. The optimal / selected via grid search over [0.7, 1] (step size 0.01) using validation accuracy, and the corresponding test accuracy is reported. For the bascline. we train a linear classifier on the frozen original model and report the test accuracy of the checkpoint that performs best on the validation set. All linear classifiers use the same training setup detailed in Appendix B.I
Table 1: Mcan accuracy (%) over three runs of ImageNet-pretrenn to 12 downstream datasets. The second row under each method indicates the number of trainable parameters (excluding the linear classifier). S-CT outperforms linear probing the frozen backbone, and T-CT surpasses LoRA (rank 1). Full results (# std) in Table ST
ResNet- 18/50 when transferred
ResNet-18
ResNet-50
Dataset
Frozen (0)
S-CT
LoRA (35923)
T-CT (3968)
Frozen (0)
S-CT (1)
LoRA
T-CT
(1)
(79443)
(45440)
Arabic Characters
81.91
87.65
93.37
93.76
80.65
83.66
94.21
95.67
Arabic Digits
97.93
98.77
99.08
99.03
98.33
98.37
99.08
99.16
Beans
87.76
90.36
93.23
94.01
89.58
91.93
94.79
95.57
CUB-200
62.84
63.18
54.83
64.30
65.23
64.62
66.17
71.03
DTD
62.80
62.66
54.36
63.62
67.34
66.91
64.70
65.07
FashionMNIST
88.63
88.70
91.65
91.07
90.05
90.34
92.19
92.78
#GVC-Aircrale
36,80
38.68
29.19
46.44
38.03
41.16
41.99
55.70
Flowers
80.86
81.97
67.53
86.55
84.00
83.84
82.58
87.62
Food101
61.41
62.27
64.40
66.04
68.06
68.02
71.42
73.60
DermaMNIST
74.83
75.05
74.21
77.66
75.94
75.89
75.73
78.02
OCTMNIST
65.03
67.27
74.27
69.53
67.53
68.00
75.90
74.13
PathMNIST
86.77
87.51
87.62
87.17
90.08
90.26
85.43
87.33
Average
73.96
75.34
73.64
78.26
76.24
76.92
78.68
81.31
Tables [ ST show mean accuracy over three runs for linear probing with and without S-CT on 12 downstream datasets using ResNet-18/50/152 and VGG-1 backbones. S-CT consistently improves generalization, with average relative gains of 1.97%, 1.16%, 0.02%, and 0.71% % respectively. We also report the average optimal /3 across datasets: 0.84 ResNet-18 18, 0.94 for ResNet-50, 0.96 for
ResNet- 152 and 0.90 for VGG-11 (full results in Table S2
These values are consistently close to 1,

---

suggesting the search range can be narrowed for efficiency. Example accuracy curves in Fig. S22 show that accuracy varies smoothly with B and typically peaks in the middle of the search range.
Additionally, we conduct ablation experiments on the choice of c = 0.5 for S-CT under the same settings. As shown in Table S3 setting e 0.5 yields better performance than c (SiLU) or 0 (Softplus) by 1.99% and 0.56% respectively on ResNet-18, and by 0.94% and 0.57% on ResNet-50, while performing slightly worse by 0.05% and 0.73% on ResNet-152.
4.2 T-CT is comparable to LoRA with fewer parameters
In this subsection, we show that T-CT further improves generalization, achieving performance comparable to LoRA with fewer parameters. We conduct experiments using the same setup as in Section |4.] evaluating both T-CT and LoRA. For both methods, we replace the original linear classifier in each model with an appropriate one, and apply the respective method to the pretrained backbone. In T-CT, we initialize all / parameters to 0.8 and all e parameters to 0.5. For LoRA, we apply it to all convolutional and linear layers in the backbone (implementation details in Appendix [F). And we set the rank r- = and scale a = 1, so that has a comparable number of parameters to T-CT. Refer to Appendix B.2 for training configurations applied to the two methods. Here we report the test accuracy of the best checkpoint on the validation set.
The results, summarized in Tables T and ST show that T-CT achieves the best performance across all methods, with average relative improvements on 18/50/11 and VGG-11 of 6. 6.75%, 8.59%, 8.34% and 5.53% over the baseline; 4.62%, 7.13%, 8.35% and 4.73% over S-CT; and 10.20%, 4.64%, 1.70% and 4.05% over LoRA. Importantly. T-CT achieves better performance than LoRA with fewer parameters. As reported in Tables and [ST the number of trainable parameters (excluding the classifier) used by T-CT amounts to only 11.05%. 57.20%, 59.09% and 36.39% of that used by LoRA on the four models respectively- even with LoRA operating at its lowest-rank setting (r = 1). This highlights the parameter efficiency of our approach. We further compare T-CT and LoRA with varying ranks (r E {1, 2, 4)) and scaling factors (a {r, 2r, 4r)) in Table S4 For each dataset and LoRA rank, we report the best test accuracy achieved among the candidate scaling factors. As shown, T-CT can still outperform LoRA under this more challenging setting, achieving relative improvements of 9.65%, 6.56%, and 3.34% over the three ranks on ResNet- 18; 4.18%, 2.27%, and 0.19% ResNet-50; and 1. 1.22%, -0.91% %. and -1.84% on ResNet-151 further demonstrating the effectiveness and parameter efficiency of T-CT.
To better understand how T-TT behaves during training, we analyze the distributions of learned B and e values (full statisties provided in Tables[S5Jande (S)). We observe a high degree of within-model variation, with standard deviations ranging from 0.31 to 0.38, while the means remain remarkably stable across architectures: 0.69 to 0.79 for and 0.57 to 0.61 for c. These mean values are close to those used in S-CT, though the learned /3 values tend to be smaller than the optimal shared / found in S-CT (0.84 to 0.96), while the learned e values are larger than the fixed e = 0.5. We further visualize the distributions in Figs. $3 and S4 In most datasets, shown in Fig. S3 (OCTMNIST), both B and c exhibit a sharp U-shaped distributionnconccne concentrating near 0 and with a flat middle. This suggests that T-CT leverages its parameter flexibility assign values the extremes, producing an effective average close to the manually chosen settings in S-CT, rather than concentrating around the mean values themselves F In few datasets, observe deviations from this trend, as exemplified in Fig. SS4 (DTD). Nonetheless, consistent pattern is that for any given dataset, the distributions remain visually similar across models.
4.3 Improving model robustness through CT's implicit bias
In this subsection, we demonstrate that CT exhibits an implicit bias toward enhancing model robust- ness, using benchmarks from RobustBench [14]. We evaluate the robustness of ResNet-18/50/ on
CIFAR-10/100 and ImageNet using the official &g and lo adversarial benchmarks from RobustBench
14
and the common corruption benchmark [40] See Appendix B.3 for more details.
More specifically, first show that S-CT can improve robustness without any robustness oriented objective, by applying it to pretrained models, performing a grid search over , E [0.7, 1] with a step
'This behavior may in part be influenced by the sigmoid diseedd-based parameterization used in our implementation T-CT to constrain 8 and e during training.
8

---

size of 0.01, and reporting the best robust accuracy achieved. We then show that T-CT, when used for finetuning (from ImageNet to CIFAR-10/100 following the same training configurations in Sections 4.1 a (.2), also enhances robustness despite the absence of robustness objective- the finetuned models achieve stronger robustness than those trained with linear probing or LoRA (r 1).
Table 2: Mean robust accuracy (%) over three runs of ImageNet- ResNet- 18/00/152 under Eyles attacks and corruptions on CIFAR-10/100 and ImageNet. S-CT yields substantial improvements under to attacks, with the selected 3 values close to 1. Full results (# std) in
S7
lou
Corruption
Model
Dataset
Frozen
S-CT
Frozen
S-CT
Frozen
S-CT
3
CIFAR 10
53.67
53.67
1.00
11.17
14.93
0.90
77.73
77.73
1.00
R8sNe118
CIFAR 100
24.30
25.50
0.92
4.47
6.90
0.92
51.81
51.95
0.94
ImageNet
23.37
23.37
1.00
0.00
7.00
0.89
33.11
33.32
0.92
Average
33.78
34.18
0.97
5.21
9.61
0.90
54.22
54.33
0.95
CIFAR 10
55.10
56.53
0.97
10.10
12.08
0.90
77.26
77.26
1.00
ResNet50
CIFAR 100
23.83
25.80
0.96
4.43
7.90
0.93
53.91
53.93
0.98
ImageNet
31.90
31.90
1.00
0.30
9.30
0.93
39.64
39.64
1.00
Average
36.94
38.08
0.98
4.94
9.76
0.94
56.94
56.94
0.99
CIFAR10
56.27
56.27
1.00
11.47
15.00
0.99
78.82
78.83
0.99
ResNet152
CIFARI00
27.90
28.23
0.98
5.40
7.70
0.99
56.12
56.12
1.00
ImageNet
42.50
42.50
1.00
0.30
13.53
0.97
45.47
45.47
0.99
Average
42.22
42.33
0.99
5.72
12.08
0.98
60.14
60.14
0.99
Table 3: Mean robust accuracy (%) over three runs of ImageNet--rettrinee 1 transferred to CIFAR-10/100 under lg. fo attacks, and corruptions. T-CT improves lx robustness significantly compared to linear probing and LoRA. Full results (# std) in Table S8
to0
Corruption
Model
Dataset
Frozen
LoRA
T-CT
Frozen
LoRA
T-CT
Frozen
LoRA
T-CT
CIFARIO
8.47
5,93
8.93
0.30
0.70
1.57
21.34
13.59
16.83
R8sNet18
CIFAR100
1.57
0.77
1.10
0.03
0.07
0.17
2.96
4.62
Average
5.02
3.35
5.01
0.16
0.38
0.87
13.22
8.28
10.72
CIFARIO
6.23
4.57
6.83
0.20
0.33
2.43
16.23
11.69
12.68
ResNet50
CIFAR100
0.70
0.37
0.47
0.00
0.03
0.07
3.47
2.04
1.61
Average
3.47
2.47
3.65
0.10
0.18
1.25
9.85
6.86
7.14
CIFARIO
8.03
4.63
8.00
0.43
0.20
5.10
13,82
11.33
9.83
R5sNet152
CIFART00
0.90
0.47
0.50
0.17
0.00
0.00
2.07
2.13
1.72
Average
4,46
2.55
4.25
0.30
0.10
2.55
7.94
6.73
5.78
For S-CT, summarized in Table 2 it is particularly effective against Eo attacks, achieving sub- stantial relative improvements of 44.01 %, 1032.64%, and 1494.46 for 188 ResNet-50, and ResNet- 152 Improvements under lg attacks and common corruptions comparatively moderate. The corresponding optimal 3 values, also shown in Table are consistently close to 1, suggesting that modest curvature modulation suffices improve fobrsteess-further highlighting the practical efficiency of S-CT. For T-CT, as shown in Table 3 it likewise enhances f.x. robustness significantly, achieving average relative improvements over linear probing of 445.00% 1 1115.00% and 493.029442. 0.0%. and over LoRA of 133.57%, 384.85%, and 2450.00%. These results collectively demonstrate CT's implicit bias toward robustness enhancement, further validating its effectiveness beyond standard generalization improvements.
We exclude from
computation any where the baseline robust accuracy is O.

---

4.4 CT shows promise on transformers
In this subsection, we investigate the effectiveness of T-CT in improving the generalization of transformer architectures. Unlike ResNets, transformers include attention layers that fall outside the max-affine spline framework and typically employ non-ReLU activation functions (e.g., GELU), which weakens our theoretical guarantees.
Concretely, we apply T-CT to -mretrnetpretraineed Swin-'T and Swin-S models, both of which use GELU activations in their feed-forward blocks. As before, we transfer these models to the same 12 downstream datasets and compare T-CT against linear probing on the frozen backbone and LoRA (r = 1, = 1). Additional training details are provided in Appendix B.4
Table 4: Mean accuracy (%) three of MmageeNet-pretraiine Swin-T/S when transferred to 12 downstream datasets. The second row under each method indicates the number of trainable parameters (excluding the linear classifier). T-CT improves over linear probing but underperforms
LoRA. Full results (+ std) in Table S9
Swin-T
Swin-S
Dataset
Frozen (0)
LoRA (74832)
TCT
Frozen (0)
LoRA (148560)
TCT
(532)
(868)
Arabic Characters
83.48
93.24
85.02
83.83
94.38
86.65
Arabic Digits
98.
99.19
98.47
98.28
99.19
98.39
Beans
88.28
94.01
89.06
90.89
95.05
91.41
CUB-200
73.42
78.73
74.33
72.66
79.45
73.40
DTD
70.66
70.99
71.45
69.77
71.56
72.43
FashionMNIST
89.89
93.15
90.23
89.75
93.52
89.85
FGVC-Aircraft
48.06
48.29
47.58
44.36
51.94
45.72
Flowers
86.66
90.22
85.35
83.24
87.67
85.08
Food101
77.05
83.69
78.90
77.59
85.17
79.45
DermaMNIST
75.83
76.71
75.86
76.64
78.15
77.14
OCTMNIST
69.97
76.30
67.97
66.90
76.97
69.07
PathMNIST
89.14
92.26
91.73
89.74
92.79
92.13
Average
79.22
83.06
79.66
78.64
83.82
80.06
The results, presented in Table show that T-CT yields average relative improvements over linear probing of 0.48% and 1.94% Swin-T and Swin-S, respectively. However, it underperforms LoRA in this setting, trailing by 4.01% and 4.76% on the two models. Nonetheless, as shown in T-CT requires only 0.71% and 0.58% as many trainable parameters LoRA on Swin- T and
Table 4 Swin-S-a much lower ratio than in the ResNet experiments. Thus, despite its lower performance, the results still highlight its potential for transformer architectures. It is also worth noting that in the current implementation, CT is applied only to the feed-forward blocks, which constitute a relatively small portion of the transformer, while the attention layers make up the majority. Extending CT to modulate the curvature within attention mechanisms-suce as by tuning the temperature in the softmax of the attention block-is interesting future work.
5 Conclusion
In this paper, we propose Curvature Tuning (CT), an interpretable and principled model steering method that provably modulates network's decision boundary through single parameter injected into its activation adjusts model's nonlinearities and effectively projects it onto space of smoother functions, offering a complementary perspective to existing PEFT methods. Practically, we introduce two variants: a steering form with fixed parameters (S-CT) and finetuning form with learnble ones (T-CT). Both improve model generalization and exhibit an implicit bias toward enhancing model robustness.
cnactions-withon altering the model weights. Theoretically, we show that CT
While promising, CT also presents open questions for future work. In particular, exploring how to integrate the method more effectively with popular components such as softmax-base based attention as well as kernelized attention remains an exciting direction for future work.
10

---

Broader impacts
This paper presents work whose goal advance the field of deep learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
Acknowledgments and disclosure of funding
Computations were in part enabled by the Berzelius resource provided by the Knut and Alice Wallenberg Foundation the National Supercomputer Centre.
References
[1] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The Ilama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
[2] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07793, 2023.
[3] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021. URL
htttp://arxix>> http://///axxio..aaaab////// http:/ab//11/000/
[4] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1 1975-11986, 2023.
[5] Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source vision-language-action model. arXiv preprint arXiv:2406.09: 09246, 2024.
[6] Alec Radford. Improving language understanding by generative pre-training. 2018.
[7] Ahmadreza Jeddi, Mohammad Javad Shafice, and Alexander Wong. A simple fine-tuning is all you need: Towards robust deep learning via adversarial fine-tuning. arXiv preprint arXiv:20/2. 13628, 2020.
[8] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nip, 2019. URL htttt:ttt///xiiiii org/
00751
[9] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021. URL attps://arxin www.oog///a...ee http:/bb///.009666//
[10] Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are universal approximators. Neural networks, 2(5):359-366, 1989.
[11] George Cybenko. Approximation by superpositions of sigmoidal function. Mathematics of control, signals and systems, 2(4):303-314, 1989.
[12] Guido F Montufar, Razvan Pascanu, Kyunghyun Cho, and Yoshua Bengio. On the number of linear regions of deep neural networks. Advances in neural information processing systems, 27, 2014.
[13] Randall Balestriero and Richard Baraniuk. A spline theory deep learning. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 374-383, PMLR, 10-15 Jul 2018. URL
mlr wttpss//vw//aalstteeiiiieeeeeeee
html

---

[14] Francesco Croce, Maksym Andriushchenko, Vikash Sehwag, Edoardo Debenedetti, Nicolas Flammarion, Mung Chiang. Prateck Mittal, and Matthias Hein. Robustbench: standardized adversarial robustness benchmark. arXiv preprint arXiv:2010.09670, 2020.
[15] Mathilde Caron, Hugo Touvron, Ishan Mísra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in supf-supe vision transformers. In Proceedings of the IEEE/CVF international conference computer vision, pages 9650-96600 2021.
[16] Zeyu Han, Chao Gao, Jinyang Liu, Jeff Zhang, and Sai Qian Zhang. Parameter-efficient fine-tuning for large models: A comprehensive survey. arXiv preprint arXiv:2403. 14608, 2024.
[17] Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00/1 2021.
[18] Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel. Few-shot parameter-cfficient fine-tuning is better and cheaper than in-context learning. Advances in Neural Information Processing Systems, 35:1950-1965, 2022.
[19] Baohao Liao and Christof Monz. 3-in-1: 2d rotary adaptation for efficient finetuning, efficient batching and composability. arXiv preprint (iXiv:24099001//1 2024.
[20] Demi Guo, Alexander M Rush, and Yoon Kim. Parameter-efficient transfer learning with diff pruning. arXiv preprint arXiv:20/2.07463, 2020.
[21] Mojtaba Valipour, Mehdi Rezagholizadeh, Ivan Kobyzev, and Ali Ghodsi. Dylora: Parameter efficient tuning of pre-trained models using dynamic search-free low-rank adaptation. arXiv preprint arXiv:22 10.05558, 2022.
[22] Yuning Mao, Lambert Mathias, Rui Hou, Amjad Almahairi, Hao Ma, Jiawei Han, Wen-tau Yih, and Madian Khabsa. Unipelt: A unified framework for parameter-efficient language model tuning. arXiv preprint arXiv:2110.07577, 2021.
[23] Jiaao Chen, Aston Zhang, Xingjian Shi, Mu Li, Alex Smola, and Diyi Yang. Parameter-effiiient fine-tuning design spaces. arXiv preprint arXiv:2301.018) 2023.
[24] Chongyang Gao, Kezhen Chen, Jinmeng Rao, Baochen Sun, Ruibo Liu, Daiyi Peng, Yawen Zhang, Xiaoyuan Guo, Jie Yang. and VS Subrahmanian. Higher layers need more lora experts. arXiv preprint 2xXiv:2402.08562. 2024.
[25] Shuaijun Chen, Omid Tavallaie, Niousha Nazemi, Xin Chen, and Albert Y Zomaya. Au- torank: Mcda based rank personalization for lora-enabled distributed learning. arXiv preprint arXiv:241..15553, 2024.
[26] Damjan Kalajdzievski. A rank stabilization scaling factor for fine-tuning with lora. arXiv preprint arXiv:2312.0377: 2023.
[27] Soufiane Hayou, Nikhil Ghosh, and Bin Yu. The impact of initialization on lora finetuning dynamics. arXiv preprint arXiv:2406.0844 2024.
[28] Suraj Srinivas, Kyle Matoba, Himabindu Lakkaraju, and François Fleuret. Efficient training of low-curvature neural networks. Advances in Neural Information Processing Systems, 35: 25951 25951-25964, 2022.
[29] Randall Balestriero and Richard G. Baraniuk. From hard to soft: Understanding deep network nonlinearities via vector quantization and statistical inference, 2018. URL https://a/sxx
htg/aas/18110.00
[30] Ahmed El-Sawy, Mohamed Loey, and Hazem El-Bakry. Arabic handwritten characters recogni- tion using convolutional neural network. WSEAS Transactions on Computer Research, 1:11-19, 2017.
[31] Ahmed El-Sawy, EL--Baryy Hazem, and Mohamed Loey. Cnn for handwritten arabic digits recognition based on lenet-5. In International conference on advanced intelligent systems and informatics, pages 566-575. Springer, 2016.
12

---

[32] Makerere Al Lab. Bean Disease Dataset, January 2020. URL
com/
AI-Lab- Makkerere/ibean/
[33] C. Wah, S. Branson, Welinder, Perona, and Belongie. The caltech-uesd birds-200-2011 dataset. Technical Report CNS-TR-2011-001. California Institute of Technology. 2011.
[34] M. Cimpoi, S. Maji. I. Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In Proceedings of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2014.
[35] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv: (.XXiv:100.00700 00100000777,,77 2017.
[36] Subhransu Maji. Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine- grained visual classification of aircraft. arXiv preprint arXiv:1306.5152 2013.
[37] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over large number of classes. In 2008 Sixth Indian conference on computer vision, graphics & image processing, pages 729..22..0 IEEE, 2008
[38] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 mining discriminative components with random forests. In European Conference on Computer Vision, 2014.
[39] Jiancheng Yang, Rui Shi, Donglai Wei, Zequan Liu, Lin Zhao, Bilian Ke, Hanspeter Pfister, and Bingbing Ni. Medmnist v2-a large-scale lightweight benchmark for 2d and 3d biomedical image classification. Scientific Data, 10(1):41, 2023.
[40] Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint arXiv: /903. /2261, 2019.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
The last paragraph of the conclusion section discusses the limitations of the work.
Justification:
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
14

---

Answer: [Yes]
The main text clearly states the assumptions and the complete proof is in appendix.
Justification:
Guidelines:
The answer NA means that the paper does include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
The paper describe all experiment details needed to reproduce the results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., .., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path reproducing verifying the results. 5. Open access to data and code
15

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
The paper releases the code in the supplementary.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ ccipps.c/puulli//guu /tte:///uuddde/oooSuuubbmiibia ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper clearly specifies all training and test details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: The paper plots mean and standard deviations in the figures clearly.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
16

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [No]
Justification: The paper currently only reports the type of GPUs used in the experiments and will add further details after acceptance.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported paper (e.g., preliminary or failed experiments that didn't It make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics httpo//sshttee//ooopcpo////////h
Answer: [Yes]
Justification:
The research is consistent with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no clear societal impact of the work performed as the work is about general deep learning.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The paper cites each dataset properly.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
18

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
19

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
20

---

Appendix
The remainder of the paper presents complementary background, experimental validation, and theoretical derivations that support our main results. The appendix is organized as follows.
1. Appendix
briefly connects several deep network architectures to affine spline operators.
2. Appendix details our experimental setup and results.
3. Appendix provides theoretical intuition behind CT.
4. Appendix provides pseudocode for S-CT and T-CT.
5. Appendix provides pseudocode for LoRA, describing how the method was applied throughout our experiments.
A Spline theory
The spline theory of deep learning establishes that large class of deep network (DN) layers can be modeled as Max Affine Spline Operators (MASO), More precisely:
Theorem A.1. (Propositions 1-4 in (I3]) Any DN layer comprising linear operator (e.g. fully connected or convolutional layer) followed by convex and piecewise affine nonlinear operator (e.g.. ReLU, lel,,-ReLUU, absolute value activation, mux/average/channel pooling, maxoul; with without skip connections) is MASO.
Consequently, deep network (e.g.. MLP, CNN, RNN, ResNet) composed of such linear operators and convex, piecewise affine nonlinear operators is a composition of MASOs. However, it is important to note that the network as whole is not a MASO but Affine Spline Operator (ASO). In other words, conditioned on the input, such deep networks are equivalent to an affine operator, but globally, the induced mapping is not convex.
Smoothing the Region Assignment
Smopthing the Max
Combined
3
1.0
0.9
0.8
D.7
0.6
g
0.4
0
0.2
-1
0.1
0.0
Figure S1: Visualization of nonlinearity smoothing through region assignment smoothing, max smoothing, and their combination. For ReLU network, the combined approach mitigates the opposing biases introduced by the individual methods.
Building on the MASO interpretation, curvature tuning proposes to smoothen nonlinearities (e.g. ReLU) of DN novel form of model steering that avoids retraining finetuning the learned layers. By recalling Section 3.1 when smoothing is performed by applying Eq. (5) Eq. 7 to DN layer (interpreted as MASO), the layer's output is statistically biased by either negative or positive factor, respectively. In order to counter the bias without retraining, convex combination of the equations is used, as shown in Fig. for different values of (..
B Supplementary experimental details
This section provides additional experimental setup details and results, organized to correspond with the subsections in Section 4
All experiments were conducted using 8 RTX 3090 GPUs and one I.40 GPU, with runs performed under random seeds 42, 43, and 44.
21

---

B.1 Improving generalization on downstream datasets with S-CT (Section 4.1
For each of the 12 downstream datasets, we split the data into training, validation, and sets. a dataset does not include a validation set, we hold out 20% of the training data using stratified sampling. Otherwise, we use the original validation split provided.
All linear classifiers are trained for 20 epochs using the Adam optimizer with a learning rate of 10-3 We apply linear warm-up during the first epoch and decay the learning rate by factor of 10 after epoch 10.
Additional results are provided follows:
Table ST mean accuracy (# std) over three runs peretrrii-e ResNet- BesNet-18/50//11 18/50/152 and VGG-11 when transferred to 12 downstream datasets, comparing linear probing with and without S-CT.
Table
S2
mean optimal values (# std) of S-CT across three runs.
Table
ablation experiments on the choice e = 0.5 for S-CT.
Fig.
S2
example validation accuracy vs. ,3 curves over three runs for S-CT.
Mean Acturaty
72-
Mean
70-
VS Accurscy
84-
Jolidatonn
70
Accuracy
21
0.75
0.35
0.00
0.35
(a) ResNet-18 on Arabic Characters (b) ResNet-50 on PGVC-Aincraft (c) ReReNet--SS on DermaMNIST
Figure S2: Validation accuracy (%) of S-CT during the 3 search, averaged over three runs. The accuracy curve varies smoothly and typically peaks in the middle of the 3 range.
B.2 T-CT is comparable to LoRA with fewer parameters (Section
Both T-CT and LoRA are trained for 20 epochs using the Adam optimizer. To ensure proper convergence,
different learning rates: for T-CT, learning rate of 10 is applied to the (8, c) parameters and 10 3 to the linear classifier; for LoRA, learning rate of 10 used for both the adapter parameters and the classifier. As before, apply linear warm-up during the first epoch and decay the learning rate by a factor 10 after epoch 10.
Additional results are provided follows:
Table mean accuracy (# std) over three runs
18/Nett-1//51122
and VGG-11 11 when transferred to 12 downstream datasets, comparing LoRA and T-CT.
Table $4 additional experiments on LoRA with rank E 21,2,4} and scaling factors
a E {r, 4r).
Tables S5
and 56
mean (+ std) of the learned 3 and values of T-CT across three runs.
Figs. S3 and S4 example distributions and values in T-CT, illustrating commonly
and uncommonly observed patterns.
B.3 Improving model robustness through CT's implicit bias (Section 4.3
Due to computational constraints, we evaluate each benchmark using 1,000 samples. For adversarial evaluations, follow the official RobustBench settings: e2 0.5 for & attacks and 8 for loo attacks.
Additional results are provided follows:
22

---

Mean Dersty
Mean Density
Meon Deriity
50
std
=1 std
90
16-
0.0
0.2
0.0
0.6
0.0
1.0
0.0
0.2
0.0
0.6
0.0
1.0
0.0
0.1
(a) ResNet- 18 9
(b) ResNet-50 3
(c) ResNet- 152 f
Mean Density
Mean Dersity
Mean Dersity
std
23-
std
10-
0-
0-
0.0
87
0.1
0/5
1.
6b
67
1.
0.
(d) ResNet-18
(c) ResNet-50
(f) ResNet-152
Figure S3: Common distributions 3 (top) and e (bottom) in T-CT across 1esNeet18/50/152
averaged over three runs (OCTMNIST shown representative dataset). Both 3 and consistently exhibit sharp U-shaped distributions that appear similar across all models.
Mean Density
Mean Density std
Meen Dersity #1 std
=1 std
Density
6-
2-
0.0
0.2
1.2
0.0
0.2
0.6
1.0
(a) ResNet-18 8
(b) ResNet-50 8
(c) ResNet- 152 B
3.5-
Mean Dersity
Mean Density
Mean Density
3.0
std
2.0
0.5
0.5
0.0
0.0
(d) Re
(c) ResNet-50 e
(f) ResNNet-152
Figure S4: Uncommon distributions of 8 (top) and (bottom) in T-CT across 1osNett18/50/152. averaged over three runs (DTD shown as example dataset). While the overall shape is dataset- specific, the distributions of both B and remain consistent across models.
23

---

Table S7
mean robust accuracy (# std) over three runs of ImageNet-pretrainee ResNet- under /x/2 attacks and corruptions on CIFAR-10/100 10/100 and ImageNet.
18/50/152
Table S8
mean robust accuracy (# std) over three of MmageNee-pretraiine ResNet-
18/50/152 transferred to CIFAR-10/100 under foo, l2 attacks, and corruptions.
B.4 CT shows promise on transformers (Section 4.4
In this experiment, LoRA applied all QKV projection layers. For all three methods-linear probing, LoRA, and T-CT-w- we perform a grid search over the learning rate and select the model achieving the best validation accuracy for testing. The learning rate selected from 10-2 10-3 10-4 44 for linear probing, from 10-3, 10-4 10 for LoRA, and for T-CT, we fix the learning rate of the linear classifier to 10-3 while searching over 10-1, 10-2 2 10 10 for (3, c) parameters. All other training configurations follow those described in Appendices B. and B.2
Additional results are provided follows:
Table S9 mean accuracy (# std) over three runs of ImageNett-reerrannd Swin-Thh when transferred to 12 downstream datasets.
24

---

Table S1: Mean accuracy (%) standard deviation over three runs
ResNet-
18/50/152 and VGG-11 when transferred to 12 downstream datasets. The second row under each method indicates the number of trainable parameters (excluding the linear classifier). S-CT outper- forms linear probing on the frozen backbone, and T-CT surpasses LoRA (rank 1).
(a) ResNet-18
Dataset
Frozen (0)
S-CT
LoRA (35923)
T-CT (3968)
(1)
Arabic Characters
81.91 0.15
87.65 0.18
93.37 # 0.31
93.76 + 0.22
Arabic Digits
97.93 0.08
98.77
99.08
0.05
99.03 = 0.01
Beans
87.76
9.13
93.23
94.01 + 0.37
CUB-200
0.29
0.21
5.38
+ 64.30
DTD
62.80
6.26
54.3
63.62 0.67
FaShionMNIST
0.13
88.70
91.65
9.10
FGVC-Aircraft
36.80
38.68
46.44 29.19
Flowers
80.86
0.29
86.55 0.76
Food101
61.41
6.227
6.104 6..40
DermaMNIST
74.83
75.05
74.21 0.50
77.66 = 0.29
OCTMNIST
65.03 0.69
67.27
74.27 = 0.49
69.53 # 1.11
PathMNIST
86.77 0.04
$7.5120.00
87.62 + 0.12
87.17 = 0.66
Average
73.96
75.34
73.64
78.26
(b) ResNet-50
Dataset
Frozen (0)
S-CT (1)
LoRA (79443)
T-CT (45440)
Arabic Characters
80.65 + 0.07
83.66 + 0.41
94.21 + 0.28
95.67 = 0.03
Arabic Digits
98.33 # 0.02
9.00
99.08 0.00
99.16 # 0.03
Beans
89.58 0.74
9.99
9.77
95.57 # 0.74
CUB-200
65.23
6.32
6..17
71.03 # 0.64
DTD
0.13
66.91
64.70
65.07 # 0.37
FashionMNIST
90.05
9..34
92.19
92.78 + 0.06
FGVC-Aircraft
0.30
41.16
0.09
55.70 # 0.76
Flowers 102
0.00
0.18
8.44
87.62 # 0.28
Food101
0.10
0.10
7.14
73.60 # 0.13
DermaMNIST
75.89 0.112
75.73 0.72
78.02 + 0.50
OCTMNIST
0.17 6.25
75.90 # 0.33
74.13 + 1.65
PathMNIST
90.08 0.22
90.26 + 0.20
85.43 + 1.99
87.33 + 0.74
Average
76.24
76.92
78.68
81.31
(c) ResNet- 152
Dataset
Frozen (0)
S-CT (1)
LoRA (243283)
T-CT (143744)
Arabic Characters
79.86 0.12
79.21 0.55
95.96 + 0.21
96.47 + 0.39
Arabic Digits
98.07 0.05
0.115
99.15
99.10 0.05
Beans
87.50
93.75 8.780
96.35 + 1.33
CUB-200
0.54
6..15
0.72
73.04 0.19
DTD
66.95
66.97
66.63+0.00
63.39 + 0.34
FashionMNIST
90.37
00.44
9..07
93.39 + 0.12
FGVC-Aircraft
0.164
0.15
40.54
58.16 + 0.31
Flowers
82.98
0.22
8..40
83.43 + 1.01
Food101
0.009
7.08
74.66+000
76.08 # 0.15
DermaMNIST
75.68 + 0.447
76.23 0.14
76.91 0.79
77.94 # 0.60
OCTMNIST
69.27 0.98
69.10 1.47
76.43 # 0.54
75.17 # 2.10
PathMNIST
89.91 0.12
89.82 + 0.09
84.94 + 1.27
83.60 + 0.42
Average
76.51
76.54
80.42
81.34
25

---

(d) VGG-11
Dataset
Frozen
S-CT
LoRA (60315)
T-CT
(0)
(1)
(21888)
Arabic Characters
81.19
8
88.88 0.377
9..04
Arabic Digits
97.97 0.03
9.01
9.07
Beans
91.67=0.99
93.75 0.37
92.45 1.95
CUB-200
61.04
6.69
59.22
63.13+0.3
DTD
65.07 = 0.45
6.16
0.76
65.25 # 0.56
PashionMNIST
0.11
89.35
90.52 0.15
90.49 + 0.16
FGVC-Aircraft
39.48
39.29 4..4
47.79 # 0.23
Flowers 102
80.32
78.38 0.59 0.340
84.31 # 0.20
Food101
61.03
64.22 0.09
66.43 # 0.05
DermaMNIST
0.29
7.00
74.06 + 0.22
77.97 + 0.38
OCTMNIST
0.11 0.92
72.13 1.18
70.77 + 0.45
PathMNIST
86.85 0.14
87.55 0.03
89.23 + 0.43
88.90 + 0.53
Average
74.78
75.18
75.99
78.30
Table S2: Mean 8 standard deviation of S-CT over three runs of Image.Net- pretrained ResNet- 18/50/152 and VGG-11 when transferred 12 downstream datasets. The average optimal values are consistently high, ranging from 0.84 to 0.96 across models.
Dataset
ResNet-18 18
ResNet-50
ResNet-152
VGG-11
Arabic Characters
0.77 + 0.01
0.89 + 0.01
0.96 + 0.00
0.88 + 0.02
Arabic Digits
0.75 + 0.01
0.04
0.95 0.01
0.85
Beans
0.06
0.98 0.94
0.76
CUB-200
0.91
0.03
0.04
0.82
DTD
0.88 0.02
0.01
0.93 1.00
FaSH
0.01
0.00
0.02
0.02
FGVC-Aircraft
0.02
0.90
0.03
0.82
Flowers 102
0.02
0.96
0.00
0.02
Food101
0.01
0.98
0.99
0.02
DermaMNIST
0.94 0.08
0.05
0.00
0.02
OCTMNIST
0.80 0.00
0.94
0.09
0.90
PathMNIST
0.83 + 0.00
0.00
0.00
0.91
Average
0.84
0.94
0.96
0.90
26

---

Table S3: Mean accuracy (%) standard deviation over three runs TraageNet-pretraiie ResNet- 18/50/152 and VGG-11 when transferred to 12 downstream datasets. The second row under each method indicates the number of trainable parameters (excluding the linear classifier). By setting 0.5, S-CT performs better than SiLU (c = 1) and Softplus (c 0) on ResNet-18 18 and ResNet-50, but slightly worse on ResNet-152.
(a) ResNet-18
Dataset
Frozen (0)
S-CT (1)
SiLU (1)
Softplus (1)
Arabic Characters
81.91 0.15
87.65 0.18
84.89 0.191
Arabic Digits
97.93 17.9300000 0.08
08.77
97.95
98.64
Beans
87.76
90.36
8.19
88.02
CUB-200
0.29
63.18
6..9
63.08
DTD
62.80
6.26
0.38
63.24
MaSSIonMNIST
88.63 0.13
88.70
88.63
FGVC-Aircraft
36.80
38.68
37.44 36.29
Flowers
80.86
0.29
83.14 0.35
Food101
6.001
0.22
6.03
62.76 = 0.06
DermaMNIST
74.83 #.23
75.05 #
0.119
75.13 # 0.22
OCTMNIST
65.03 2 0.69
67.27
65.10
66.57 = 0.75
PathMNIST
86.77 - 0.04
87.51 0.05
86.77 = 0.05
87.84 + 0.25
Average
73.96
75.34
74.01
74.97
(b) ResNet-50
Dataset
Frozen (0)
S-CT (1)
SiLU (1)
Softplus (1)
Arabic Characters
80.65 0.07
83.66 0.41
8121 0.03
Arabic Digits
98.33 0.02
98.37
9.15
9.15
Beans
89.58
91.93
86.46
0.90
CUB-200
65.23 2 0.43
6.36
6..12
65.61
DTD
6.13
66.91
67.06 6.02
FashionMNIST
90.05
0.23
0.11 0.01
FGVC-Aircraft
0.30
4.316
38.03
40.09 0.02
Flowers102
0.06
0.18
0.114 8.119
Food101
6.11
0.11
6.19 0.06
DermaMNIST
0.12
2.89
75.79 7.11
OCTMNIST
0.21
68.00 # 0.17
67.33 0.15
67.00 + 0,20
PathMNIST
90.08 - 0.22
90.26 0.20
89.95 0.13
89.99 + 0.14
Average
76.24
76.92
76.43
76.53
(c) ResNet-152
Dataset
Frozen (0)
S-CT (1)
SiLU (1)
Softplus
(1)
Arabic Characters
79.86 0.12
79.21 0.55
80.24 0.44
7
Arabic Digits
98.07 0.05
98.15
98.20 + 0.18
97.98 + 0.01
Beans
87.50
0.780
88.80+1.99
88.80 + 2.39
CUB-200
0.54
6..15
67.68 #.66
69.80 # 0.32
DTD
66.95
6..97
66.8640.23
66.95 + 0.03
FashionMNIST
90.37
90.44
0..2
90.60 # 0.03
FGrC-Aireraft
0.116
38.51
3..24
39.20 + 0.38
Flowers 102
0.16
0.22
8..97
0.13 83.54 # 0.13
Food101
7.09
71.11 0.01
71.19 # 0.08
DermaMNIST
75.68 0.47
+.14
74.76 0.69
76. # 0.10
OCTMNIST
69.27 +.98
69.10 1.47
69.47 + 1.29
69.90 + 0.50
PathMNIST
89.91 1 0.12
89.82 + 0.09
89.91 + 0.15
90.75 # 0.06
Average
76.51
76.54
76.59
77.06

---

Table S4: Mean accuracy (%) standard deviation over three runs Imageeee--pretraiiee ResNet- 18/50/152 when transferred to 12 downstream datasets. The second row under each method indicates the number of trainable parameters (excluding the linear classifier). For each dataset and LoRA rank we report the best test accuracy achieved among the candidate scaling factors a E {r, 2r, 4r). T-CT can still outperform LoRA under this more challenging setting.
(a) ResNet-18
Dataset
T-CT (3968)
LoRA (r= = (35923)
LoRA (r = 2) (71846)
LoRA (r = 4) (143692)
Arabic Characters
93.76 0.22
94.23 0.13
95.30
96.26 0.12
Arabic Digits
99.03 + 0.01
99.12
99.21 +
99.23 + 0.03
Beans
94.01
9.74
9.38
95,83 1.33
CUB-200
64.30
0.37
0.26
57.47 # 0.51
DTD
6.662
5..3
56.17
57.93 + 0.43
FashionMNIST
0.16
0.11
92.83
93.50 + 0.05
FGVC-Aircraft
0.49
29.5
32.94
39.13 + 0.32
Flowers102
0.21
0.76
69.68
73.30 + 0.46
Food101
0.17
6..40
66.97 + 0.35
DermaMNIST
77.66 0.29
0.200
76.79 # 0.77
76.72 + 0.51
OCTMNIST
69.53 1.11
0.17
76.37 + 0.66
76.47 + 0.26
PathMNIST
87.17 0.66
87.78 0.18
88.08 + 1.15
88.85 + 0.48
Average
78.26
74.01
75.44
76.80
(b) ResNet-50
Dataset
T-CT (45440)
LoRA (r = 1) (79443)
LoRA (r 2) (158886)
LoRA (r 4) (317772)
Arabic Characters Arabic Digits
95.67 0.03
94.38
95.67 # 0.34
96.30 + 0.04
99.16 + 0.03
9.00
99.22 + 0.09
99.22 # 0.02
Beans
95.57 0.74
9.33
96.09 1.09
95.31 # 0.64
CUB-200
71.03 0.64
6..17
0..51
68.93 + 0.23
DTD
65.07 0.37
6.379
64.91
67.07 0.31
FashionMNIST
92.78 #.06
92.19
92.90
93.59 0.13
FGVC-Aircraft
55.70
42.12+0.17
4..46
52.64 0.47
Flowers
87.62 + 0.28
0.47
83.39
+ 84.63
Food101
73.60 0.13
7.14
73.01 +
74.89 + 0.05
DermaMNIST
78.02 0.50
0.27
77.26 # 0.36
77.39 + 0.47
OCTMNIST
74.13 1.65
7.009
76.07 + 1.19
77.83 + 0.82
PathMNIST
87.33 0.74
87.29 0.67
86.04 + 0.23
87.44 + 0.16
Average
81.31
79.07
79.99
81.27
(c) R55Net-152
Dataset
T-CT (143744)
LoRA (r = 1) (243283)
LoRA (r = 2) (486566)
LoRA (r = 4) (973132)
Arabic Characters
96.47 + 0.39
96.00 0.16
# 96.43
96.87 # 0.11
Arabic Digits
99.10 + 0.05
99.16
99.21 0.03
99.25 0.01
Beans
96.35 1.33
94.5
9.392
97.14 0.74
CUB-200
73.04 + 0.19
7..75
7.114
71.72 0.43
DTD
63.39 0.34
6.06
0.56
68.28 + 0.51
FashionMNIST
#3.39
92.77
93.49
93.98 + 0.14
FGVC-Aircraft
08.16
0..06
5..82
59.98 + 0.26
Flowers
83.43 1.01
8.551
84.83 0..17
86.24 0.04
Food101
76.08 0.15
7.066
76.00+0.J0
76.86 + 0.10
DermaMNIST
77.94 + 0.60
7..02
77.31 + 0.75
77.46 + 0.16
OCTMNIST
75.17 + 2.10
77.90
78.23 + 1.32
78.63 + 0.21
PathMNIST
83.60 0.42
86.75 + 0.86
88.33 + 0.33
86.81 + 1.95
Average
81.34
80.81
82.18
82.77
28

---

Table S5: Distribution of 3 values in T-CT, computed over all 3 parameters across all three runs of ImageNet-pretrained ResNet-18/50/152 and VGG-11 when transferred to 12 downstream datasets. The mean and standard deviation of 3 are similar across models (means between 0.69-0.79, stds between 0.31-0.37), suggesting consistent tuning behavior the model level, while the relatively large standard deviations indicate substantial variation of 8 within each network.
Dataset
ResNet- 18
ResNet-50
ResNet- 152
VGG-11
Arabic Characters
0.72 # 0.34
0.63 + 0.41
0.68 = 0.39
0.73 + 0.39
Arabic Digits
0.70 = 0.43
0.62 0.48
0.62 0.47
0.73 0.43
Beans
0.72 = 0.26
0.23
0.77 = 0.19
0.79 + 0.20
CUB-200
0.81
0.29
0
0.75
DTD
0.19
0.77
0.24
0.80 #80:0.2. 0.24
FaShionMNIST
0.41
0.46
0.46
0.81 = 0.37
FGVC-Aireraft
0.23
0.33
0.74 0.32
Flowers102
0.16
0.25
0.74 0.17
Food101
0.30
0.74
0.40
0.88 +
DermaMNIST
0.34
0.79
0.37
0.81 = 0.33
OCTMNIST
0.67
0.48
0.63 0.47
0.83 + 0.35
PathMNIST
0.69
0.65 0.47
0.61 = 0.48
0.82 + 0.37
Average
0.74
0.69 0.37
0.71 = 0.35
0.79 + 0.32
Table S6: Distribution of e values in T-CT, computed over all e parameters across all three runs of Imaeeeett-retrrineed ResNet-18/50/155 and VGG-11 when transferred to 12 downstream datasets. The mean and standard deviation of are similar across models (means between 0.57-0.61, stds between 0.32-0.38), suggesting consistent tuning behavior the model level, while the relatively large standard deviations indicate substantial variation of within each network.
Dataset
ResNet-18
ResNet-50
ResNet-152
VGG-11
Arabic Characters
0.63 + 0.39
0.61 + 0.39
0.57 # 0.37
0.65 + 0.33
Arabic Digits
0.59 = 0.43
0.57 0.42
0.55 0.41
0.62 = 0.38
Beans
0.61 + 0.29
0.53 0.25
0.55 + 0.21
CUB-200
0.60 0.37
0.60 0.33
0.67
DTD
0.59 0.31
0.32
0.57 = 0.30
0.59
FashionMNIST
0.44
0.40
0.61 0.42
FGVC-Aireras
0.61
0.36 0.35 0.33
Flowers 102
0.26
0.24
0.20 0.23
Food101
0.47
0.44
0.60
0.58
DermaMNIST
0.58
0.39
0.36
+ 0.30
OCTMNIST
0.55
0.40
0.57 0.42
0.62 + 0.39
PathMNIST
0.51 0045
0.58
0.57 0.42
0.58 + 0.40
Average
0.57 + 0.38
0.59 + 0.37
0.57 + 0.36
0.61 + 0.32
29

---

Table S7: Mean robust accuracy (%) standard deviation over three runs of ImageNet- ResNet- 18/50/152 under Ey/ attacks and corruptions on CIFAR-10/100 and ImageNet. S-CT
yields substantial improvements under lo aftacks, with the selected / values close to 1.
Robustness Model
Dataset
Frozen
S-CT
B
CIFAR-10
53.67 0.32
53.67 = 0.32
1.00 + 0.00
CIFAR-100
24.30 0.10
25.50 # 0.00
0.92 # 0.00
ResNet-18
ImageNet
23.37
23.37 = 0.06
1.00 + 0.00
Average
33.78
34.18
0.97
CIFAR-10
0.10
56.53 # 0.21
0.97 0.00
CIFAR-I00
23.83
25.80 0.20
0.96 0.00
ResNet-50
ImageNet
31.90
31.90 = 0.00 38.08
1.00 + 0.00
Average
36.94
0.98
CIFAR-10
56.27
56.27 0.23
1.00 0.00
CIFAR-100
2.
28.23 # 0.12
0.98 0.00
ResNet-152
ImageNet
4
42.50 # 0.00
1.00 + 0.00
Average
42.22
42.33
0.99
CIFAR-10
1.06
14.93 0.06
0.90 0.00
CIFAR-100
4.47 # 0.06
6.90 = 0.00
0.92 + 0.00
ResNet-18
ImageNet
0.00 0.00
7.00 + 0.10
0.89 0.00
Average
5.21
9.61
0.90
CIFAR-10
10.10
14.83 # 0.06
0.95 + 0.00
loo
CIFAR-100
4.43 # 0.06
7.90 + 0.00
0.93 0.00
ResNet-50
ImageNet
0.30 0.00
9.30 # 0.17
0.93 = 0.00
Average
4.94
9.76
0.94
CIFAR-10
11.47
15.00 # 0.20
0.99 0.00
CIFAR-100
5.40 # 0.00
7.70 # 0.17
0.99 0.00
ResNet- 152
ImageNet
0.00
13.53 + 0.06
0.97 + 0.01
Average
5.72
12.08
0.98
CIFAR-10
77.73
77.73 0.00
1.00 0.00
CIFAR-100 100
51.81 +.00
51.95 # 0.00
0.94 + 0.00
ResNet- 18
ImageNet
33.11
33.32 # 0.00
0.92 0.00
Average
54.22
54.33
0.95
CIFAR-10
77.26
77.26 0.00
1.00 + 0.00
Corruptions
CIFAR-100
5.000
53.93 # 0.00
0.98 0.00
ResNet-50
ImageNet
3.00
39.64 # 0.00 56.94
1.00 0.00
Average
56.94
0.99
CIFAR-10
78.82 + 0.00
78.83 + 0.00
0.99 + 0.00
CIFAR-100
56.12 0.00
56. 0.00
1.00 0.00
ResNet- 152
ImageNet
45.47 + 0.00
45.47 # 0.00
0.99 + 0.00
Average
60.14
60.14
0.99
30

---

Table $8: Mean robust accuracy (%) + standard deviation over three runs of ImageNettpretrainel ResNet-18/50/15 transferred CIFAR-10/100 under f2, fxc attacks, corruptions. T-CT im- proves lo robustness significantly compared to linear probing and LoRA.
Robustness
Model
Dataset
Frozen
LoRA
T-CT
10
8.47 + 0.26
5.93 # 1.65
8.93 + 0.37
ResNet18
CIFAR100
1.57 + 0.21
0.77 + 0.33
1.10 0.45
Average
5.02
3.35
5.01
CIFAR
6.23 = 0.34
4.57 # 1.32
6.83 + 1.48
l
ResNet50
CIFAR100
0.70 + 0.08
0.37 + 0.26 2.47
0.47 = 0.31
Average
3.47
3.65
CIFARIO
8.03 0.52
4.63 # 2.01
8.00 1.22
ResNet152
CIFAR100
0.90 = 0.08
0.47 + 0.26
0.50 = 0.08
Average
4.46
2.55
4.25
CIFARIO
0.30 + 0.00
0.70 + 0.71
1.57 + 0.74
ResNet18
CIFARI00
0.03 = 0.05
0.07 = 0.05
0.17 + 0.12
Average
0.16
0.38
0.87
CIFAR10
0.20 + 0.08
0.33 + 0.29
2.43 + 1.54
600
ResNet50
CIFARI00
0.00 + 0.00
0.03 = 0.05
0.07 0.09 1.25
Average
0.10
0.18
CIFAR10
0.43 + 0.09
0.20 + 0.14
5.10 + 2.97 0.00 0.00
ResNet152
CIFAR100
0.17 + 0.05
0.00 + 0.00
Average
0.30
0.10
2.55
CIFAR10
21.34 + 0.29
13.59 + 0.30
16.83 # 2.36
ResNet18
CIFAR100
5.10 + 0.15
2.96 + 1.05
4.62 + 0.68 10.72
Average
13.22
8.28
CIFARIO
16.23 0.21
11,69 0.90
12.68 + 2.06
Corruptions
ResNet50
CIFAR100
3.47 + 0.09
2.04 + 0.36 6.86
1.61 + 0.13 7.14
Average
9.85
CIFARIO
13.82 + 0.49
11.33 + 1.22
9.83 + 2.07
ResNet152
CIFARIO0
2.07 + 0.12
2.13 # 0.22
1.72 # 0.51
Average
7.94
6.73
5.78
31

---

Table S9: Mean accuracy (%) + standard deviation over three runs of Inaeanee--retrained Swin-T/S when transferred to 12 downstream datasets. The second row under each method indicates the number of trainable parameters (excluding the linear classifier). T-CT improves over linear probing but underperforms LoRA.
(a) Swin-T
Dataset
Frozen (0)
LoRA (74832)
T-CT (532)
Arabic Characters
83.48 + 0.15
93.24 = 0.13
85.02 # 0.30
Arabic Digits
9.014
99.19
0.01
98.47
Beans
88.28 + 1.10
+
89.06
CUB-200
78.73 0.17
7.13
DTD
0.13
70.99 +
71,45 0.31
FaShionMNIST
9.11 0.08
90.23 # 0.03
FGVC-Aircraft
0.30
47.58 0.44
Flowers 102
86.66
9..22
8.22
Food101
0.00
8..69
7.11
DermaMNIST
76.71 0.28
75.86
OCTMNIST
69.97
76.30 # 1.66
67.97 # 1.01
PathMNIST
89.14 0.23
92.26 + 0.12
91.73 0.19
Average
77.69
82.23
78.53
(b) Swin-S
Dataset
Frozen (0)
LoRA (148560)
T-CT
(868)
Arabic Characters
83.83 0.05
94.38 = 0.34
86.65
Arabic Digits
98.28 +: 0.03
99.19 0.05
9.04
Beans
90.89 1.95
0.64
CUB-200
0.56
79.45
73.40 +.110
DTD
69.77
7..56
7
MNSST
0.03
0.05
89.85 # 0.08
FGVC-Aircraft
0.23
51.94 0.60
45.72
Flowers 102
87.67 0.05
85.08 0.25
Food101
0.06
35.17
7.14
DermaMNIST
7.222
78.15 # 0.67
77.14
OCTMNIST
6.290
76.97 # 0.45
69.07 # 0.60
PathMNIST
89.74 + 0.38
92.79 # 0.33
92.13 + 0.15
Average
78.06
82.90
79.24
32

---

C Theoretical intuition
This section provides theoretical intuition behind Curvature Tuning. Section C.1 projection over a space of smooth functions, while Section C.2 provides a toy example illustrating how CT can improve approximation of target function non con-vanishing curvature, upon an ideal baseline ReLU network.
CT
C.1 CT operates as projection
At its core, Curvature Tuning operates by modulating the nonlinearity of the activation functions of trained model, providing a novel approach to model steering. order to formalize the effect of CT, the following briefly introduces the notion of spaces of smooth functions.
Sobolev spaces Let f R + R be function and Ω S Id bounded domain. For 1 S < oc, define (9t) the space of functions f Ω IR such that the IP norm is finite, i.e.
(L/ [xx
[Iflle(())
<000
(10)
Let a (a1,..., Ca) denote a E N" For la] > 0, define the Sobolev semi-norm
with |a| = Σi oz and a, N, Vi 1....,d d. Let
|/ I/laora)) =
Σ | (ll ol
(11)
glaly with D = dx;" atft...00 00.
denoting |a|-th order partial derivatives of Define the Sobolev norm
[[ = (1S1 + +IPra))))
(12)
and the Sobolev space W = (ʃ : - R.t. ( + 1flwor(s) < 20).
For finite set D {xx / the Sobolev semi-norm becomes
IflWor(D)=
Σ
D fx,,
(13)
lo|Sa
t=
Finally, for x Rd, let ||x||p denote the p-norm, corresponding to the Euclidean norm for 2.
Curvature Tuning acts as a Sobolev Projection characterize Curvature Tuning, we are inter- ested in the space W2 (S2), equipped with the Sobolev semi-norm
If Ifliaa(()) V +
V 1 fl2
(14)
We begin by considering the Sobolev semi-norm of ReLU network (cquivalent to the case of Eq.
8
with 3 + 1). For each E Edd the gradient of ReLU network
f(x) (W
o Wa) (x)
(15)
with [p(a)] == max(0, a;) for E m and i E [1, m), is given by
vxf(x) WL II (')x) (=--1
(16)
where (x(x) is diagonal matrix with (x) (xt20)) with z Wfp(((-11 by denoting the
pre-activation of the {-th layer, for 1 1, L, with == x.
We make the following observations:
33

---

OI Since ReLU networks are differentiable a. the gradients Vxf(x) are bounded in norm by the network's Lipschitz constant, which can be defined as supxen [x)l2>) Hence, for f = D, the Lipschitz constant provides an upper bound on the first-order term of the Sobolev semi-norm in Equation [4
02 Finally, observe that since ReLU networks express piece-wise affine functions, the Hessian norm vanishes a.c. (i.c. wherever the Hessian is well defined), providing a bound on the second-order term of Equation [4
Equipped with the above observations, in the following we characterize CT by formally restating and proving Theorem 3.1
Theorem C.1. Let f I2 - IR denote ReLU network, with model parameter W collecting all weights and biases. For c E [0,1] and fixed B E [0,1), replacing every instance of ReLU with CTU (Equation 8 with hyperparameters B, is equivalent to projecting fa,c E WZ)( in the Sobolev space 1y2.2(()), with bounded Sobolev semi-norm.
smooth function
Particularly, it holds (x)]( S ff expressivity (non-vanishing curvature), while retaining the same model parameter W.
(2(0)) from which fB.c enjoys higher local
Before proving Theorem we state the following Lemma, bounding the derivative of CTU.
Lemma C.2. Let (z) be defined according Eq. (). for 3 E [0,1) and [0, 1]. Then
bx
(/) c (bx(xx) + boc ( ) a(br))()))))))))))))))))))))[[ c)a
(17)
3
where b. = I-B and o (xx)
1 x is the sigmoid activation.
exp
Furthermore, 3 ha E IR+ such that
-cho S (l),(()) < + chb
Ya ER, E
(18)
Proof. We recall that, since Vz E R. (x) is defined as the convex combination of the SiLU
activation function (c= 1) and the Softplus activation (c = 0), we can bound (x), by the convex combination of individual bounds obtained for the cases 0 and c 1.
Softplus. If 0, then (22,0(z) a ( and (x)) s 1 Vz, since the derivative is defined as a sigmoid.
SILU. If 1, (x) o(bx) + boo (bx)(bb))11 (1=))()(() The first term in the sum is bounded by definition of sigmoid. For the second term, note that (bbr) (1 a(br) is also bounded, and achieves maximum at æ = 0, for which I a (bz)) - (bx)) < 1.4 Furthermore, in the limit x + +0c, holds sp's,, (x) + 1, while (x) + for + -c.
0. and so the maximum value of bzo (bx)(b - (br)) also depends on br. To bound B, in this case, let us first consider æ > 0. By defining Tú maxba20 (bx o (bz)), then we finally obtain S (2)1(2)) < 1 1 hg.
In the non-asympototic regime, (bbx)(1 or (bx))
For the case x < 0, by using the identity (xx = (-=x), we have that -Tis S (/)1(()) < 1. By combining the results, we have
Thp & 4 (x) < Tub
Ve E R,
B E [0,
(19)
In conclusion, by convex combination of cases O and
1. Eq. 19 holds uniformly in
We can now prove Theorem C.1
To do so, for f3.c we have to show that
1. f3,e is smooth in x, for x E Ω
2. /fa.cllwa.2000 6o
for a network f3,e obtained by replacing every ReLU yp with a CTU &8..o while keeping learned parameters W fixed.
34

---

Proof. We provide a proof for Ω = D = {x: under the common i..d. assumption on D.
To prove the first point, we observe that for /3 [0, 1), the CTU activation function is smooth, i.e. PP, E - (IR), thus making the whole network Sa,e smooth.
We now consider the Sobolev semi-norm ( Starting with the first-order gradient, by recalling that CT replaces each occurrence ReLU with the CTU activation function (Equation the input gradient of CT is given by
Vx/a,((x)= Wt.
II
D& (a.((')
(20)
6=L-1
where D (z() diag( (z))) with (z)) (//,
according to Eq.
17
To bound the Jacobian norm, we observe that
VV ||
II E=L-1
D (2))())
(21)
s <| t=L-1
II
(22)
< ||
II (=L-1
dd + cha)ll <0
(Lemma C.2
(23)
independent of x, for W E Redexde11, with do := d.
We now bound the second order term. By recalling that, for every x E Rdd the Hessian H(x)
V (x)x,(()) symmetric positive-definite then for S D it holds
fa n
H )12
max X (H(x:))d oc SSSaa
(24)
with Amax (H())) denoting the largest singular value of H(x:). Importantly, since ReLU network has vanishing curvature a.., then for 0 S 3 < 1, we have
[+) S [[))).
Lastly, we note that, whenever is a finite discrete set D, J3,e is measurable, ensuring that [ < oc, concluding the proof.
Theorem shows that CT operates by projecting a ReLU network f to smooth function fa.c in a restricted Sobolev space. Crucially, La.. enjoys bounded gradients (and so is well behaved), and
non-vanishing local-curvature for O < B < 1, making it locally more expressive than the affine spline f, for fixed W.
Furthermore, for fixed (B,c), CT indeed operates as projection, since replacing every ReLU with Pe, is idempotent. Importantly, while for the original ReLU network f E W/2.2 the derivatives Da f are understood in a weak-sense, for E [0, 1] and B E [0, 1), fø.e belongs to a Sobolev space (I2) of smooth functions, whereby the derivative D fa.e understood in the strong (i.e. classical) sense.
w (Ω) (
We leave for future work extending our result to T-TT, which is associated with non-convex optimization problem of finding optimal (3,c) for every neuron in the network. An additional important direction is to more closely compare V> and |/V=fa,ell, which may reveal more precise Lipschitz behaviour for CT, potentially better guiding the search for and c.
C.2 Toy example
We conclude the discussion by providing the full derivation for the motivating example in Section 3
35

---

Consider a binary classification problem in R². whereby one is given two classes fx E R²; ||x||a $ 2 and fx E IR?
| S 2}. The decision boundary maximizing the mar- gin between the two classes is given by s [x E R | = 1}.
For a ReLU network f : R + R. the maximum margin boundary is recovered by assigning f(x) - 0 E St, for which (f(()) = 0.5. To measure the approximation error e, the boundary is parameterized by 7(t) = (cos 2T sin 2nl), for / E [0,
Then, the enror is expressed by the line integral = J, |f|dx = fo ( Since is an Affine Spline Operator, and each linear region in 52 is convex, then the integral along y can be broken down into the integral along the intersection of 7 with the spline partition Ω, i.c. Bay 2n St. Importantly, this allows us to pull back the affine spline breakpoints from Gty to [0, 1]. so that 0 v h S S by S 1, where r' = [Syl- And we augment the breakpoints with the end points so that 0 to S h ... to tr Then,
e = [ ((7(1))1)7
(25)
fx+1
2m
7(1) + bre dt
(26)
k=0
sk+1
= 2x
(-1)sa(0) (() + by,) dt
(2
k=0
with (t) = 1 g (t) <0}, where denotes which spline region the k-th segment [fx, tx+1]
falls into. Then,
1kt1
2mt
(-1)sa(t) (Arx,1 cos 2nt + Are. sin 2xt + b dt
(28)
k=0
(-1)au(0) (1)d ( pat+ (-1)za(0) g'z (D)
= 2m
(29)
k=
(-1)ze((a) (y,, (1)]ez + (1))s [gru (1))
2m
(30)
where
Ir (t) = -Ar,,1 COS 2nt + 2 sin 2tt + bru
sin 2nt
cos 2nt
Dr, (1) A
Arx.2
+ brg
2x
2π
and sh [tx, tx+1] 1] is defined so zu (t) holds the same value for [1,, sk] and the opposite for t E (sk, 1.41]. If for ł E [t,, 4++1 (t) holds the same value, then simply set 8n = tu.
Then since both (-1)au(()) [9r, (1)] and (-1)sx(tura) [9r, (4)) are non-negative, is clear 4-11- + th Vk = -
Hence, assuming the ReLU network considered attained optimal approximation error e > 0, reducing the error further requires increasing the number of breakpoints of the ASO, in turn requiring degree of retraining (either through PEFT or training from scratch). With this view, Curvature Tuning opens an additional avenue for model adaptation: steering the model's decision boundaries by modulating the nonlinearity of the activation function, allowing to tune model towards optimality without expensive retraining. To this end, important to note that modulating decision boundaries is orthogonal to feature adaptation and finetuning, since it allows to change the shape of decision boundaries while keeping the model parameter W fixed.
36

---

D
Curvature Tuning (CT) implementation
The following code provides the Python implementation for S-CT and T-CT:
SCTU & TCTU: classes that define the CTU module used in S-CT and T-CT, respectively.
replace_ module & replace @pplace_module mwdulo dynamic: functions that apply the appropriate module replacement to integrate S-CT or T-CT into model.
import torch
from torch import mn
import torch. nn. functional as F
class SCTU (nn. Kodule):
CTU for Steering CT.
def __init__ (self, shared_rawe bettba, =20)
threshold
super ii.it self threshold threshold _ol__rrawbbeta shared_raw_bets
self salf._raw_ceste shared_rav_coeff
self raw beta. requires_grad False
coeff requires_grad False
eproperty
def beta(self))
return torch sigmoid (aaeelfa_ra_bbtta
@property def coeff (self);
return torch.. iigoi .ra_ccoff_r
def forward x):
beta = torch. sigmoid raw beta) coeff torch. torch. sigmoid (selff_rav_cooe_eeeee______eeeee one_minus_beta
beta 1e-6
x_scaled x one one_minus_beta
return (coeff torch sigmoid x_scaled) + coeff) softtpuu (x_ccale, threshold-self. threshold) one_ming one_minus_beta
class TCTU (nTUule)oddue)): ***
CTU for Trainable CT.
def initi_ (self, jum_input_diin _ims,, channela,,,, raw berabeta=1 11.3. 386, raw_coeff00.0, threshold=20):
super () ..._innt___
self. threshold threshold
# Decide channel dim based input shape
if num ium_input_dime L, D)
or num_input_dime 3: (B, C) or
channel_ dim = -1
elif hom_input_ddiee dims = 4: (B, c, H, W)
channel_dim
else:
raise NotImplementedErre (f" Unsupported input dimension num imm_inut__dims>>
37

---

param_shape = [1] num_input_dims
param_shape [channel_dim] out_channels
# Init beta
self. _alf__raw_ette beta un Parametee (torch... full (paran_shape, float (
raw_beta)))
Init coeff self._raw_coet nn Parameter (torch.. full (param_shapa float raw_coeff)))
Sproperty
def beta (self): return torch. (sel.. __raw
oproperty
def coeff(seff)
return torch sigmoid _ra_ccoffaa.ooof
def forward x):
beta torch. ba.. coeff torch... torch.sigmoid sigmoid (ea..ossse........oofff.... one one_minus_bert beta me__inns_beta
beta 1e-63
x_scaled
return (coeff torch sigmoid x_scaled) x
(1 coeff) (.saat threshold) one_minus_bes beta)
(x_scaled, threshold=self.
def replace_module (model, (model, old_module=nn ReLU, new_module=SCTU, ** kvargs):
Replace all instances of old_module in the model with new_module.
device next (model parameters torch, tensor (()) device Handle models with parameters
Replace modules
for name, module in model. named_modules (): if ianastanceemoduu old_module); ct nev_module (**kwargs) (device)
# Replace module in the model names sepit((")) parent model for in names[:-1]]
if n. ()diiiitt() parent parent (in] # for Sequential/ ModuleList
else:
parent getattr n)
last _ame_aame_-e names [-1] [-m]
if last_name isdigit
parent [int (last_name)] et for Sequential/
ModuleList
else:
setattr (parent, last_name, ct)
return model
def mopluce_nodule..m dynamic (model, input_shape, old old_module-ni un. ReLU, new TCTU **bwargs):
38

---

Replace all instances of old_module in the model with new_module that's dynamically created based the number of output channels.
device next (model parameters torch tensor (())) device dummy_input torch randn (+ (winput_shape) (device) (
module aidule_metadate 0 hooks []
(num_input_dinn) out_channels)
def make_ _ook (name):
def hook (module, imput, output):
num_input_
input dim ()
if num jum_input_diaee dime in (2, 3)
(B, C) or (B, L, D)
out out_chhanels output shape [-1]
elif num num_input_dimm == 4:
(B, C, H, W)
out_channels output. shape [1]
else:
raise cotImplementeddErre (f "Ussupported output shape {
Srrput.shappe in (name}")
module_metadata [name] (num_input_dims out_channels)
return hook
Register hooks all modules of the target type
for name, module in model. momuleeod if isinstance (module, old_module): hooks. append (module..
(make_hoop hook (name))
Run dummy forward pass model (dummy_input)
Clean up hooks for hook in hooks: remove))
Replace modules
for name, module in model [omeud_odules(( ():
if isinstance (module, old_module) and module, metadata:
nun_input_dims out_channels codule_metadee [name]
ct = nev_module (num (wt___inputdiiissaaaaaaaaaeee _iputttipttt_t simpuu_dimm,, out oat_chaanelsroottcchamee
**wargg) to (device)
Replace module the model names name. samee.p""((""") parent model for in [:-]]: i.isdigit
parent parent [i)](n)] for Sequential/
ModuleList
else:
parent getattr(parent n)
last last_name name names [-1] if last_name. name. isdigit
parent [int (last_name) ct for Sequential/ ModuleList
else:
setattr Sptaetr(parent, last_name, ct)
return model
39

---

E LoRA implementation
The following code provides the Python implementation of LoRA used in Section
LoRALinear & LoRAConv2d: classes that define LoRA-enhanced versions the Linear and Conv2d modules.
get_lora_modaee a function that replaces all Linear and Conv2d modules model with their corresponding LoRA versions.
import torch
from torch import un as nn from torch... nn import functional as F
class LoRALinear (nn. Module)
A Linear layer that applies LoRA to frozen, pretrained Linear.
def -_iniiiitt__.. in (self, original_layer: nn Linear, Ii int 4, alpha: float 1.0):
super (n..))init..f self. in_features original_ in features self. out_features triginal_layer self.r =I self.alpha = alpha
Freeze the original layer's parameters self. self. weight nn Parameter (original weight data, requires.grad=Falee
if original_layer bias is not None self.bias Parameter (original. layer bias data, requires_grad-Palls
else:
self bias None
LoRA parameters and
# B: [out_features r]
# A: [r, in_features]
self B Parameter (torch. zeros ((self, out_features r)))
self A Parameter (torch zeroa ((r, self in features)))
# Initialize LoRA weights
nn init. mamning_unifrrann (self.B B, a-5 . 0.5)
init. wwwiiiiersss............mm.mssst (self
def forward x):
# Normal forward with the frozen weight result = F.linear (x, self veight, self bias)
# LORA path: BOA
# shape of BA [out_featured featuree # Then F.linear with BA fora_updat = F.linear (self alpha self (self. e self.A))
return result lora_update
class LoRAConv2d (nn. Module)
A Conv2d layer that applies LoRA to a frozen, pretrained Conv2d.
40

---

def - (self, original_layer: nn Conv2d, int 4, alpha: float 1.0): super initt..
self out out_channels original_layer out out_channels
self colfiin_channess - Filginal_layrr.in_chl in_chenells
self kei.keenel_size original_layer kernel_size
wtw.eeyyyee.eeeeeeiieee.e self.stride original,layer.stride
self. = original,layer.
self self.groups original.layer.dilat
self self.dilation
self aaf.bias_available (original.layer bias not None)
self.r self.alphn alpha = alpha
Freeze original parameters
self weight = un nn. Parameter Paraastte laye weigg data, requires.grad-False)
if silf.baas_availes
self.bias nn. Parameter (ormm bias data,
Pequires_grad-Fals.
else:
self bias None
# Flattened shape for weight is [out_channels, channells k_h k_v]
k_h, k_w self ternel_size fan_in self channels k_h k_w # Flattened input dim
# Define LORí parameters: and
# B: [out_channels, r]
# A: [r, fan_in]
self.B Parameeer (torch zeros ((self out_ out_chhanels, r))) self. A nn. Parameter (torch zeros ((r, fan_in))
# Initialize LoRA weights nn init kaiming. uniform (self.8 B, a=5 ** 0.5)
init.iiitii zeros, (self.A)
def forvard x):
# Standard (frozen) convolution
result = (.conv
x,
self seightiight
bias-self.biee
boaddesself.stridee
beadingsself.padiiee
dilation=self. stt.esaelllddllaio..... groups=self groups
# Compute LoRA update
# 1) Flatten conv kernel the same manner as above
# 2) Multiply and -> shape [out_channeles ih_nhellels k_h k_v]
back to [out_channels, channeln,, k_h, k_vl BA self.B 0 self.A shape [out_channels, fan_in]
# 3) Reshape
# Reshape to conv kernel k_h, k_w self kernel_size lora_weight BA. aa.view( self oolf.out_chamaee out_ccannee,,,,, self inffiincchaneel. k_h,
41

---

k_w (self alpha / self.r)
# Perform conv2d with the LoRA weight (no extra bias term for LoRA)
lora_update - cocovd
x,
lerah
bias=None,
stride=seif.strie stride,
padding=self, padding,
dilation=self.din dilation,
groups-self. groups
result lora_update
def get_lora_moder (model: nn Module, I: int 4, alpha: float 1.0):
Recursively replace all Conv2d and Linear modules in model with LoRA- versions. Freezes original weights and adds LoRA parameters.
for name, child in list (model. named_children m ()):
# child is Conv2d, replace it with LoRAConv2d if (ehsaanccechhll un Conv2d)
lora_module LoRAConv2d (child, r-r, alpharalpha) setattr (model, name, lora_module)
child is Linear, replace with LoRALinear elif isinstance (child, Linear): lora_module LoRALinear (child, r=r, alpha=alpha) setattr (model, name, lora_module)
else:
Recursively traverse children met_lora_model (child, r-r, alpha-alpha)
model
42


Reviews:
[{"review_id": "9JGcEK7Eqa", "reviewer": "Reviewer_uDax", "summary": "This paper introduces and studies curvature tuning (CT) as a novel paradigm to parameter-efficiently finetune pretrained models. Instead of finding direct modifiers over model weights, the authors suggest (learnable) modifications on the utilized activation functions. In controlled scenarios, the authors showcase provable adjustments to model decision boundaries to facilitate generalization capabilities, highlight complementarity with existing PEFT techniques, and showcase high parameter efficiency even when compared to rank 1 LoRA.", "questions": "See Weaknesses.", "limitations": "Explicitly addressed in Discussions, offering sensible avenues for future research.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "__Strengths:__\n* The paper is really well written and structured.\n* To the best of my knowledge, attacking parameter finetuning as a function of activation function tuning is novel, and incredibly interesting.\n* Beyond the novelty, this paper offers a very principled perspective into the efficacy of curvature tuning, and highlight the complementary/orthogonality to existing finetuning methods.\n* On its own, CT is also really parameter efficient (in parts significantly below even rank 1 LoRA).\n* The experiments provided are sufficient to convince of the importance and potential general efficacy of curvature tuning for model finetuning.\n\n__Weaknesses:__\n\nMy only gripe with this paper is the depth of comparative experiments testing CT on more models, activation functions, datasets and domains and comparing against finetuning techniques beyond LoRA (and even here, ideally also contrasting against higher rank variants). Other than that,, I am rather hard pressed to find any other major issues with this work - I find it a really interesting read, and potentially a highly impactful contribution to the model finetuning domain.", "was_revised": true, "final_justification": "The authors provided additional experimental support to an already great paper, including VGG11, Swin & additional LoRA experiments. Consequently, I maintain my strong recommendation for acceptance of the paper.", "rebuttal": "We thank the reviewer for their thorough and encouraging feedback! As suggested, we have conducted three additional experiments to further demonstrate the effectiveness and generality of Curvature Tuning (CT). These experiments address broader model families, activation functions, and comparisons with additional LoRA configurations.\n\nWe also take the opportunity to emphasize that the aim of the paper is to establish CT as a highly efficient approach to parameter tuning, which is orthogonal and complementary to established parameter-centric PEFT methods. In our experimental evaluation, we achieve results that are comparable to other PEFT methods, but with considerable gains in terms of parameter-efficiency. \n\n## Experiment 1: CT on VGG11\nTo evaluate the effectiveness of CT on a different convolutional architecture, we apply Trainable CT to an ImageNet-pretrained VGG11 model (with batch normalization). We compare it against linear probing and LoRA with $r=1$ and $\\alpha=1$.\n|Dataset|Frozen|LoRA|Trainable CT|\n|------------------|--------|------|---------------|\n|Arabic Characters|81.07|89.20|**93.07**|\n|Arabic Digits|98.00|98.83|**99.06**|\n|Beans|91.41|**92.97**|**92.97**|\n|CUB-200|61.44|59.30|**63.22**|\n|DTD|**64.89**|64.36|64.47|\n|FashionMNIST|89.49|**90.63**|90.33|\n|FGVC-Aircraft|39.51|40.14|**47.49**|\n|Flowers102|80.08|77.59|**84.57**|\n|Food101|60.95|64.51|**66.44**|\n|DermaMNIST|76.31|73.87|**78.50**|\n|OCTMNIST|66.10|**73.80**|70.20|\n|PathMNIST|86.67|**89.64**|88.38|\n|**Average**|74.66|76.24|**78.23**|\n\n**Results:** *Trainable CT* achieves an average relative improvement of 5.55% over linear probing and 3.44% over LoRA, showing its strong performance even on models not originally considered in the paper.\n\n## Experiment 2: CT on GELU-based Swin-T/S\nTo test CT’s compatibility with other activation functions, we begin with GELU as a representative case. Since GELU can be approximated by SiLU, which is parameterized by CTU with $\\beta=0.6403$ and $c=1$, we can apply *Trainable CT* to GELU-based models by starting with these values. \nMore specifically, we apply *Trainable CT* to ImageNet-pretrained Swin-T and Swin-S, using the original GELU activations (unlike our earlier experiment in the paper, which replaced GELU with ReLU). And we compare it with linear probing and LoRA ($r=1$, $\\alpha=1$).\n|Dataset|Swin-T|||Swin-S|||\n|------------------|--------|------|---------------|--------|------|---------------|\n|                 |Frozen|LoRA|Trainable CT|Frozen|LoRA|Trainable CT|\n|Arabic Characters|77.62|**93.57**|86.10|75.30|**94.58**|85.06|\n|Arabic Digits|98.06|**99.12**|98.04|98.10|**99.18**|97.88|\n|Beans|88.28|**95.31**|92.19|87.50|**92.97**|89.84|\n|CUB-200|73.59|**76.82**|72.30|72.16|**77.68**|73.04|\n|DTD|**71.17**|69.73|70.85|70.16|70.48|**72.82**|\n|FashionMNIST|89.75|**92.95**|89.10|89.81|**93.45**|86.68|\n|FGVC-Aircraft|39.48|47.16|**47.73**|34.47|**52.42**|45.09|\n|Flowers102|84.52|83.25|**85.41**|81.36|81.43|**85.04**|\n|Food101|77.05|**82.30**|78.02|77.72|**84.47**|75.89|\n|DermaMNIST|76.51|**77.11**|75.21|76.66|**77.41**|75.46|\n|OCTMNIST|67.00|**76.70**|67.30|66.80|**77.00**|64.30|\n|PathMNIST|90.65|**92.56**|91.24|89.89|**91.91**|90.33|\n|**Average**|77.81|**82.22**|79.46|76.66|**82.75**|78.45|\n\n**Results:** *Trainable CT* achieves an average relative improvement over the baseline of 2.92% (Swin-T) and 3.76% (Swin-S). While *Trainable CT* still trails LoRA by 3.19% and 5.42% respectively, the performance gap is notably smaller than in the original paper. This supports our hypothesis in the original paper that the larger gap in the earlier ReLU-based Swin experiments (pretrained on the smaller Imagenette) stemmed from underpowered representations, which gave LoRA’s greater parameter count a stronger advantage. On ImageNet-pretrained models now, *Trainable CT* achieves more comparable results **despite using just 0.71% and 0.58% of LoRA’s trainable parameters**.\n\nWe also emphasize: CT is complementary to LoRA, not a replacement—beating LoRA is not our goal, but rather demonstrating CT's efficiency and compatibility.\n\n## Experiment 3: CT vs. Higher-rank and Tuned-$\\alpha$ LoRA\nWe extend our LoRA comparison by evaluating higher ranks and tuned $\\alpha$ values on ImageNet-pretrained ResNet-18 and Imagenette-pretrained Swin-T (ReLU-based). More specifically, we test LoRA with $r \\in {1, 2, 4}$. For each rank, we use $\\alpha \\in \\{r, 2r, 4r\\}$ and report the best performance.\n|Dataset|ResNet-18||||Swin-T||||\n|------------------|------------|-------------|-------------|-------------|------------|-------------|-------------|-------------|\n|                 |Trainable CT|LoRA $r=1$|LoRA $r=2$|LoRA $r=4$|Trainable CT|LoRA $r=1$|LoRA $r=2$|LoRA $r=4$|\n|Arabic Characters|93.76|93.37|95.06|**96.19**|41.95|56.32|68.36|**77.86**|\n|Arabic Digits|99.03|99.08|**99.22**|**99.22**|90.82|97.54|98.28|**98.62**|\n|Beans|94.01|93.23|96.09|**97.66**|68.49|75.52|80.47|**82.81**|\n|CUB-200|**64.30**|54.83|56.45|58.15|6.09|7.42|9.03|**11.36**|\n|DTD|**63.62**|54.36|56.17|57.77|17.04|16.99|20.64|**20.74**|\n|FashionMNIST|91.07|91.65|92.79|**93.42**|77.07|83.90|86.83|**88.73**|\n|FGVC-Aircraft|**46.44**|29.19|32.43|38.76|6.14|5.59|5.91|**7.41**|\n|Flowers102|**86.55**|67.53|68.92|72.89|16.53|16.66|**20.36**|19.91|\n|Food101|66.04|64.40|66.00|**66.75**|15.20|18.17|21.54|**25.80**|\n|DermaMNIST|**77.66**|74.21|76.46|77.36|71.37|74.08|75.41|**76.16**|\n|OCTMNIST|69.53|74.27|**77.30**|76.10|53.23|63.53|66.80|**71.40**|\n|PathMNIST|87.17|87.62|**89.36**|89.28|77.35|**81.31**|79.97|76.16|\n|**Average**|**78.26**|73.64|75.52|76.96|45.11|49.75|52.80|**54.75**|\n\n**Results:** On ResNet-18, *Trainable CT* outperforms LoRA with rank 1/2/4 by 10.20%, 6.63%, and 3.22%, respectively, indicating that CT remains highly effective even when compared to stronger LoRA baselines. While for Swin-T, *Trainable CT* trails LoRA (rank 1/2/4) by 8.29%, 16.30% and 21.17%, respectively.\n\n## Future Work: CT on Language Model\nAs a promising direction for future work, we plan to evaluate *Trainable CT* on language models and benchmarks. This would allow us to examine CT's effectiveness in non-vision domains.\n\n## Conclusion\nThese new experiments collectively reinforce the generality and practicality of Curvature Tuning across several key aspects suggested by the reviewer:\n\n- **More Models**: Demonstrated in Experiment 1 (VGG11) and Experiment 2 (GELU-based Swin-T/S), showing that CT applies to broader model families.\n- **More Activation Functions**: Experiment 2 illustrates CT's applicability to GELU-based networks, beyond the original ReLU-based settings.\n- **More Finetuning Techniques**: Experiment 3 highlights CT’s competitiveness even against stronger LoRA configurations with higher rank and tuned alpha.\n\nWe appreciate the reviewer’s insightful suggestion and hope these additions further validate CT as a compelling and complementary approach to parameter-efficient finetuning."}, {"review_id": "hynAtY1zpi", "reviewer": "Reviewer_cxvD", "summary": "This paper introduces Curvature Tuning (CT), a novel method for steering pretrained models by modifying their activation functions instead of their weights. Grounded in spline theory, CT uses a parameter, $\\beta$, to modulate the curvature of the model's decision boundary. The paper presents a training-free \"steering\" version and a highly parameter-efficient \"Trainable CT\" for finetuning, demonstrating improved generalization and robustness", "questions": "* $\\beta$ and c are often U-shaped. Is this an emergent property you find desirable? Have you tried regularizing the training to explicitly encourage or discourage this distribution?\n\n* Your motivation for the CTU was to mitigate a mean shift from simpler formulations. Could you provide an ablation study in the appendix that shows the negative impact of this shift in deep networks?\n\n* Given that Trainable CT used far fewer parameters than LoRA on transformers, do you think the performance gap could be closed by allocating a moderately higher parameter budget to CT?", "limitations": "In the conclusion, the authors identify their work's primary focus on ReLU-based networks as a limitation, stating that while CT is compatible with activations like Softplus or SiLU, this area \"warrants further study\". They also note that a detailed, layer-wise analysis of the learned $\\beta$ and c parameters in Trainable CT was not performed and remains an avenue for future research to understand representation and abstraction.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strength:\n*  The approach of tuning activations is a fresh perspective in model adaptation. Its foundation in spline theory provides a clear, interpretable mechanism for controlling model curvature, unlike more heuristic methods.\n\n* Trainable CT achieves impressive results with significantly fewer parameters than competing methods. For instance, on ResNets, it outperformed LoRA while using only 11% to 59% of the trainable parameters.\n\n* The method delivers substantial gains in both generalization and robustness on ResNet architectures. The improvements on downstream accuracy and against L infinity adversarial attacks are particularly noteworthy.\n\nWeakness:\n\n* The method's theoretical guarantees are tied to ReLU-based networks. Applying it to models with other common activations, like GELU, required modifying the base architecture for the experiments, which may limit its off-the-shelf utility.\n\n* While effective on CNNs, Trainable CT underperformed LoRA in the transformer-based experiments. This raises questions about its effectiveness in more complex, attention-based architectures.\n\n* The non-trainable steering version of CT requires a grid search to find the optimal $\\beta$ for each task, adding a tuning step that finetuning methods avoid.", "was_revised": true, "final_justification": "My \"Borderline Accept\" score reflects a balance. The core idea of tuning activations is really cool and a valuable contribution, and the authors did a great job in their rebuttal. They successfully fixed a critical experimental flaw with the transformer results by correcting the learning rate, which makes the paper much stronger. However, a few things hold it back from a higher score. Even with the fix, the method still lags behind a standard LoRA baseline on transformers, and its theory is pretty focused on ReLU-based models. Ultimately, I think the paper's novelty and the successful resolution of key issues outweigh these remaining concerns, justifying its publication.", "rebuttal": "We thank the reviewer for their thorough and encouraging feedback. In response, we conducted two additional experiments and provide below detailed clarifications to address the reviewer’s concerns. \n\nBefore addressing the raised concerns, we begin by reiterating the central aim of our paper: to establish Curvature Tuning (CT) as a highly efficient approach to parameter tuning, which is orthogonal and complementary to established parameter-centric Parameter-Efficient Fine-Tuning (PEFT) methods. In our experimental evaluation, we achieve results that are comparable to other PEFT methods, but with considerable gains in terms of parameter-efficiency and novel applications to downstream robustness. \n\n## Weakness 1: Limited Utility on Other Activations\nWe agree that our theoretical guarantees are currently tied to ReLU-based networks. However, this does not preclude the application of CT to other activations. In fact, SiLU and SoftPlus are special cases of our Curvature Tuning Unit (CTU), and *Trainable CT* can be directly applied to models using these activations.\nMoreover, for GELU—commonly used in transformer models—it can be approximated by SiLU with a CTU parameterization of $\\beta=0.6403$ and $c=1$. This allows us to apply *Trainable CT* to GELU-based models by initializing with these values.\nTo address this point concretely, we conducted new experiments applying *Trainable CT* to ImageNet-pretrained Swin-T and Swin-S models using their original GELU activations (unlike our earlier experiments, which replaced GELU with ReLU). We compare against linear probing and LoRA ($r=1$, $\\alpha=1$):\n|Dataset|Swin-T|||Swin-S|||\n|------------------|--------|------|---------------|--------|------|---------------|\n|                 |Frozen|LoRA|Trainable CT|Frozen|LoRA|Trainable CT|\n|Arabic Characters|77.62|**93.57**|86.10|75.30|**94.58**|85.06|\n|Arabic Digits|98.06|**99.12**|98.04|98.10|**99.18**|97.88|\n|Beans|88.28|**95.31**|92.19|87.50|**92.97**|89.84|\n|CUB-200|73.59|**76.82**|72.30|72.16|**77.68**|73.04|\n|DTD|**71.17**|69.73|70.85|70.16|70.48|**72.82**|\n|FashionMNIST|89.75|**92.95**|89.10|89.81|**93.45**|86.68|\n|FGVC-Aircraft|39.48|47.16|**47.73**|34.47|**52.42**|45.09|\n|Flowers102|84.52|83.25|**85.41**|81.36|81.43|**85.04**|\n|Food101|77.05|**82.30**|78.02|77.72|**84.47**|75.89|\n|DermaMNIST|76.51|**77.11**|75.21|76.66|**77.41**|75.46|\n|OCTMNIST|67.00|**76.70**|67.30|66.80|**77.00**|64.30|\n|PathMNIST|90.65|**92.56**|91.24|89.89|**91.91**|90.33|\n|**Average**|77.81|**82.22**|79.46|76.66|**82.75**|78.45|\n\n**Results:** *Trainable CT* improves over the frozen baseline by 2.92% (Swin-T) and 3.76% (Swin-S) on average, while trailing LoRA by only 3.19% and 5.42%, respectively—despite using only 0.71% and 0.58% of LoRA's trainable parameters.\n\n## Weakness 2: Underperformance in Transformers\nIn the original submission, we hypothesized two potential reasons for the underperformance of *Trainable CT* compared to LoRA on ReLU-based Swin-T/S models pretrained on Imagenette:\n1. *Trainable CT* uses significantly fewer parameters than LoRA (0.71% and 0.58% on Swin-T/S).\n2. The small size of the Imagenette pretraining dataset results in weaker representations, amplifying the benefit of high-parameter methods like LoRA.\n\nIn our new experiment on ImageNet-pretrained, GELU-based Swin-T/S models as shown above, the number of trainable parameters for both *Trainable CT* and LoRA remains identical to the original experiment. Thus, the first factor still holds—CT continues to operate with a significantly smaller parameter budget.\nImportantly, the new results support the second hypothesis: when models are pretrained on a much larger dataset (ImageNet), the performance gap between *Trainable CT* and LoRA narrows. This suggests that the earlier underperformance was at least partly due to the limited representation capacity of models pretrained on the smaller Imagenette dataset.\nIn summary, as demonstrated by the new experiment, while *Trainable CT* still trails LoRA by 3.19% (Swin-T) and 5.42% (Swin-S), it achieves this competitive performance with only 0.71% and 0.58% of LoRA's trainable parameters—highlighting its strong parameter efficiency and practical value.\n## Weakness 3: Tuning Overhead of Steering CT\nWe acknowledge that the non-trainable version of CT involves a grid search over $\\beta$. However, we observe that the optimal $\\beta$ values tend to be close to 1, suggesting that the search range can be substantially narrowed in practice (e.g., to [0.9, 1]) for efficiency.\nMoreover, popular PEFT methods such as LoRA also require hyperparameter tuning—for instance, selecting appropriate values for rank and $\\alpha$—and typically involve grid searches as well. Thus, tuning overhead is not unique to our method.\n\n## Question 1: U-shaped Distributions of $\\beta$ and $c$\nAs noted in Footnote 5 of the paper, the U-shaped distributions may be partially attributed to the sigmoid-based parameterization of $\\beta$ and $c$ (which constrains them to [0,1]). Interestingly, the average values of these parameters are close to the optimal settings discovered in the steering version of CT, which we consider a desirable outcome.\nWe have not explicitly regularized these distributions, as it's unclear whether such U-shapes are beneficial or detrimental. However, we agree this is an intriguing direction and plan to explore regularization effects in future work.\n## Question 2: Negative Impact of the Shift\nTo demonstrate the value of our current CTU formulation in mitigating the shift caused by simpler formulations, we compare the steering version of CT against the simpler formulation of just using SiLU or SoftPlus (both reparameterized as in CTU) on ResNet-18 and ResNet-50:\n|Dataset|ResNet-18|||ResNet-50|||  \n|-------|---------|----|--------|---------|----|--------|  \n||CT|SiLU|SoftPlus|CT|SiLU|SoftPlus|  \n|Arabic Characters|**87.65**|81.96|84.89|83.66|**86.46**|81.10|  \n|Arabic Digits|**98.77**|97.99|98.64|98.37|**98.56**|98.41|  \n|Beans|**90.36**|88.80|88.02|**91.93**|86.46|91.15|  \n|CUB-200|**63.18**|62.92|63.08|64.62|65.06|**65.61**|  \n|DTD|62.66|62.89|**63.24**|66.91|67.18|**67.20**|  \n|FashionMNIST|88.70|88.52|**88.93**|90.34|89.92|**90.35**|  \n|FGVC-Aircraft|**38.68**|36.31|37.44|**41.16**|37.91|40.09|  \n|Flowers102|81.97|80.37|**83.14**|**83.84**|83.77|83.69|  \n|Food101|62.27|61.34|**62.76**|68.02|**68.04**|67.90|  \n|DermaMNIST|75.05|74.63|**75.11**|**75.89**|75.74|75.79|  \n|OCTMNIST|**67.27**|64.90|66.57|**68.00**|67.30|67.00|  \n|PathMNIST|87.51|86.81|**87.84**|**90.26**|90.03|**90.26**|  \n|**Average**|**75.34**|73.95|74.97|**76.92**|76.37|76.55|\n\n**Results:** The SiLU-only version trails steering CT by 1.98% and 0.94% on ResNet-18 and ResNet-50, respectively. And the SoftPlus-only version trails steering CT by 0.53% and 0.54%, respectively. This performance gap illustrates the negative impact caused by using simpler formulations. We also observed that the average optimal $\\beta$ for CT on ResNet-18/50 is 0.84/0.94. In contrast, for the other two formulations, the values are closer to 1—0.99/0.95 for SiLU and 0.94/0.98 for SoftPlus—indicating a smaller room for model steering. This reflects the negative impact of the shift from another perspective.\n\n## Question 3: CT Performance with Higher Parameter Budget \nIn our current setup, *Trainable CT* already uses the maximum number of parameters allowed under its formulation—specifically, by assigning one pair of parameters ($\\beta$, $c$) to each output neuron of the activation functions throughout the network. Therefore, it is not possible to increase the parameter count within the current formulation.\nHowever, since *Trainable CT* is orthogonal to other PEFT methods, any additional parameter budget can be leveraged by combining it with methods like LoRA. This opens up a promising direction for achieving even stronger performance while maintaining modularity and efficiency.\n\nWe sincerely appreciate the reviewer’s thoughtful and constructive comments. We hope that our responses and additional experiments have sufficiently addressed the concerns raised and clarified the strengths and contributions of our work."}, {"review_id": "0sOEp284yk", "reviewer": "Reviewer_W2ME", "summary": "This paper introduces a novel perspective on fine-tuning by shifting the focus from conventional weight adaptation to activation function modulation—an often overlooked yet critical component in determining the model's expressivity and decision boundary.\nTo this end, the authors propose Curvature Tuning (CT), a method designed to finetune the curvature of the activation functions. CT theoretically ensures smoother decision boundaries and empirically demonstrates improved generalization and robustness across various downstream tasks.", "questions": "1.\tLimited Architectural Diversity in Evaluation\n\nTo what extent does Curvature Tuning (CT) generalize across diverse neural network architectures beyond ResNet and Swin Transformers? Have the authors considered evaluating CT on a broader range of model families (e.g., VGG, MobileNet, GoogLeNet, DenseNet, or WideResNet) to substantiate its architectural robustness and flexibility?\n\n2.\tIncomplete Robustness Evaluation\n\nWhile Table 1 presents a broad comparison, why are Trainable CT and LoRA omitted from the robustness experiments? Would including these baselines help more rigorously validate CT’s claimed advantage under distributional shifts?\n\n3.\tLimited Experimental Scope\n\nCan the authors clarify whether Curvature Tuning (CT) is applicable beyond vision tasks? Have they considered evaluating its generalization performance in other domains such as NLP, multimodal learning, or reinforcement learning to demonstrate broader applicability?\n\n4.\tAdditional Clarification on Class Imbalance\n\nHow does CT perform under class-imbalanced conditions? Has its robustness to skewed label distributions been empirically evaluated, and if not, would this be a worthwhile direction to explore given real-world dataset imbalances?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths\n\n1.\tNovel Perspective on Model Steering\n\nThe paper proposes a fundamentally new fine-tuning paradigm by modulating activation functions, as opposed to the conventional weight-centric approaches. This enables direct control over the curvature of decision boundaries and, by extension, the model's hypothesis space.\n\n2.\tSound Theoretical Grounding\n\nThe proposed method is underpinned by a rigorous theoretical framework. Specifically, it interprets deep neural networks as compositions of max-affine spline operators (MASO) and formulates Curvature Tuning (CT) as a projection onto a smooth function space, thereby providing analytical clarity and mathematical justification.\n\n3.\tParameter Efficiency\n\nThe method introduces minimal overhead: the Steering CT variant introduces only a single hyperparameter, while the Trainable CT variant requires significantly fewer parameters than rank-1 LoRA, yet achieves superior performance.\n\n4.\tInterpretability\n\nThe impact of curvature control is both mathematically tractable and visually interpretable. The adjustment of the 棺 parameter provides an intuitive handle for regulating the non-linearity and expressive capacity of the model.\n\n5.\tStrong Empirical Performance\n\nCT consistently outperforms established baselines such as linear probing and LoRA across 12 diverse downstream classification tasks. It also demonstrates marked gains in robustness, achieving state-of-the-art results on robustness benchmarks such as RobustBench.\n\n\nWeaknesses\n\n1.\tLimited Architectural Diversity in Evaluation\n\nWhile the theoretical foundation of Curvature Tuning (CT) leverages Max-Affine Spline Operators (MASOs), which generalize a broad class of activation functions including ReLU and Leaky ReLU, the empirical validation is confined to ResNet and Swin Transformer backbones. To substantiate the architectural generality of CT, further evaluation on diverse model families—such as VGG, GoogLeNet, MobileNet, DenseNet, or WideResNet—would be beneficial, even if not complete.\n\n2.\tIncomplete Robustness Evaluation\n\nWhile Table 1 provides a comprehensive comparison across baseline methods, the robustness experiments lack inclusion of key variants such as Trainable CT and LoRA. This omission makes it difficult to fully assess the robustness advantage of Curvature Tuning (CT) under distributional shifts. Including these comparisons would strengthen the empirical claim of CT’s superior robustness.", "was_revised": true, "final_justification": "I have reviewed the submitted rebuttal, and many of my concerns have been resolved through the extensive experiments provided. I recommend acceptance of the paper.", "rebuttal": "We thank the reviewer for their thorough and encouraging feedback! In response, we conducted an additional experiment and provided detailed clarifications addressing the reviewer’s concerns. Before delving into specific points, we would like to reiterate the central aim of our paper: to establish CT as a highly efficient approach to parameter tuning, which is orthogonal and complementary to established parameter-centric PEFT methods. In our experiments, CT achieves competitive performance with significantly greater parameter efficiency.\n\n## Weakness 1: Limited Architectural Diversity in Evaluation\nTo demonstrate CT's applicability to broader model families, we conducted an additional experiment applying *Trainable CT* to an ImageNet-pretrained VGG11 (with batch normalization). We compared it with linear probing and LoRA ($r=1$, $\\alpha=1$):\n\n|Dataset|Frozen|LoRA|Trainable CT|\n|------------------|--------|------|---------------|\n|Arabic Characters|81.07|89.20|**93.07**|\n|Arabic Digits|98.00|98.83|**99.06**|\n|Beans|91.41|**92.97**|**92.97**|\n|CUB-200|61.44|59.30|**63.22**|\n|DTD|**64.89**|64.36|64.47|\n|FashionMNIST|89.49|**90.63**|90.33|\n|FGVC-Aircraft|39.51|40.14|**47.49**|\n|Flowers102|80.08|77.59|**84.57**|\n|Food101|60.95|64.51|**66.44**|\n|DermaMNIST|76.31|73.87|**78.50**|\n|OCTMNIST|66.10|**73.80**|70.20|\n|PathMNIST|86.67|**89.64**|88.38|\n|**Average**|74.66|76.24|**78.23**|\n\n**Results:** *Trainable CT* achieves an average relative improvement of 5.55% over linear probing and 3.44% over LoRA, demonstrating strong performance even on architectures not originally considered in the paper.\n\n## Weakness 2: Incomplete Robustness Evaluation\nWe omitted LoRA from the robustness experiments because applying it to improve robustness would introduce confounders compared to Steering CT. Indeed, methods such as LoRA require an explicit optimization target—such as minimizing cross-entropy loss for classification tasks. In contrast, Steering CT can improve robustness by simply adjusting the curvature of the model decision boundary, without relying on labeled data or explicit loss functions. Similarly, while Trainable CT directly modulates the curvature of decision boundaries (and thus bears a direct effect on adversarial robustness [1]), a direct and fair comparison with LoRA would require incorporating adversarial training techniques, such as generating adversarial examples using PGD, which would somewhat obfuscate the isolated effect of Trainable CT.\n\nBeyond the above experiment, deriving methods specifically tailored around CT that enable more targeted defenses against distributional shifts or adversarial threats is an exciting direction for future work.\n\n## Question 1: Limited Architectural Diversity in Evaluation\nSee Weakness 1.\n\n## Question 2: Incomplete Robustness Evaluation\nSee Weakness 2.\n\n## Question 3: Limited Experimental Scope\nCT is not limited to vision tasks. The steering CT can be applied directly in any domain via a lightweight grid search procedure, while *Trainable CT* only requires a task-specific objective, similar to other PEFT methods such as LoRA. Therefore, we anticipate that CT is applicable to other domains, including NLP, multimodal learning, and reinforcement learning. Exploring these directions would be an intriguing avenue for future research.\n\n## Question 4: Additional Clarification on Class Imbalance\nWe do include class-imbalanced datasets in our experiments—most notably DermaMNIST, where the most frequent class accounts for 67% and the least for only 1% of training samples. As shown in Table 1 and Appendix Table 4 of our paper, *Trainable CT* consistently outperforms all considered baselines on this dataset across ResNet-18/50/152. This suggests that CT maintains robustness under skewed label distributions.\n\nWe sincerely appreciate the reviewer’s thoughtful and constructive comments. We hope that our responses and the additional experiment have adequately addressed the raised concerns and further highlighted the strengths and generality of our method.\n\n## References\n[1] Robustness via Curvature Regularization, and Vice Versa. Moosavi-Dezfooli, et al; CVPR 2019."}, {"review_id": "S3qvyxsGJQ", "reviewer": "Reviewer_Njg7", "summary": "This paper introduces a method for adapting pre-trained neural networks to\ndownstream tasks through the modification of activation functions. The main\nadvantage of their method, in comparison to existing popular techniques such as\nLoRA, is its parameter efficiency, introducing at most two additional parameters\nper activation layer. The method is motivated by its ability to preserve\nnon-zero curvature when applied to neural network blocks that conform to the\n'max-affine spline function' class. The method is evaluated on two distinct\narchitectures: ResNets and Swin Vision Transformers. The authors compared two\nvariants of their proposed method (CT and Train CT) against LoRA. Train CT\nexhibited superior performance in the ResNet experiments, whereas LoRA was more\neffective when applied to Swin Vision Transformers.", "questions": "- The paper emphasizes interpretability in the abstract and introduction as a\n  key advantage of the CT method. However, it was not clear to me in the\n  experiments how interpretability was demonstrated. Could you give a specific\n  example of how the claimed interpretability is helpful when fine-tuning on\n  realistic problems?\n- Can CT be used e.g. for GRPO? Can CT be used for language model fine-tuning?\n  It would be great to see some experiments in this context.\n- You mentioned that you had to pretrain Swin from scratch because it used GELU\n  and not ReLU. You also mention that SiLU is compatible with CTU, so can you\n  perform some experiments fine-tuning on a SiLU model? (E.g. Qwen3 and Llama\n  3.2 use SiLU)\n- You replace ReLUs with CTUs with $c=0.5$. Why this value in particular? Is this\n  value in some sense the closest to ReLU? It is not clear to me despite looking\n  at Figure 3.\n- Can you add results to Table 1 for LoRA, where you have tuned $r$ and\n  $\\alpha$? This would allow us to frame the performance of CT correctly.", "limitations": "Authors are missing a few limitations. Around their experiments, in particular\nthey are missing experiments with other modalities (e.g. text), and larger scale\nexperiments. Their method appears limited in that it is only better than (untuned) LoRA on\nthe ResNet.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 4, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n- The paper is well written, and the CT method is elegantly explained. I enjoyed reading Sections 2 and 3 in particular!\n- The method is very relevant to the community, as the ability to efficiently\n  fine-tune models is a key requirement for many applications.\n- Both proposed methods are interesting, with CT method potentially allowing\n  incredibly cheap fine-tuning, and Train CT being orders of magnitude more\n  parameter efficient vs. existing methods.\n\nWeaknesses:\n\n- Theorem 3.1 might benefit from extra clarity on the exact meaning of\n  'non-vanishing curvature'. Is it lower-bounded by something sensible? If so,\n  how does the bound behave as we vary $\\beta$/$c$/model size?\n- Interpretability is highlighted extensively in the abstract/introduction, but\n  in what exact sense, nor how this is useful in practice is not really discussed.\n- Experiments are only conducted on image tasks, e.g. there are no language\n  benchmarks.\n- Results are disappointing: while the method shows improvement vs. LoRA on the ResNet architecture (Table\n  1), it does not show an improvement on the Swin ViT (Table 3).\n- Invalid baselines: the benefit of Train CT over LoRA is deceptive, since LoRA\n  is only rank 1, and $\\alpha$ has not been tuned.\n- Authors mention that CT performs poorly with Swin, potentially because the\n  pretraining dataset they used, Imagenette, is small. This hypothesis can be\n  tested by e.g. pretraining Swin on a larger dataset (ImageNet perhaps). Or\n  alternatively by finding an pretrained image ViT that uses SiLU or ReLU.\n\nOverall, while the Methods were well presented, I unfortunately found the experimental results unconvincing, and that they hinder the quality of paper.\n\nOther smaller points:\n\n- The paper introduces aconyms without defining them (e.g. DNs (line 90), MASO (line 141), VQ (line 126))\n- typo line 253: \"achieves better performance then LoRA\"\n- missing error bars in Table 1 and 3.", "was_revised": true, "final_justification": "I would like to increase my score, as I believe I was previously not taking into account the relevance of the LoRA r=1 baseline presented by the authors. I think this new framing presents their method as a viable and cheap alternative to LoRA, thus deserving of acceptance.\n\nThere are still some minor improvements to make. I would still like to see more ablations/baselines (e.g. tune the activation function with very obvious parameters) to further justify the presented theory as strictly necessary from a practical point of view. I would also like to see some experiments combining LoRA with Trainable CT.", "rebuttal": "We thank the reviewer for the constructive feedback!  Before addressing specific concerns, we briefly restate the central aim of our paper: to establish CT as a highly efficient approach to parameter tuning, which is orthogonal and complementary to established parameter-centric PEFT methods. In our experimental evaluation, we achieve results that are comparable to other PEFT methods, but with considerably less parameters. \n## Local Curvature\nWe thank the reviewer for suggesting an interesting theoretical comparison: understanding the impact on the approximation error of increasing model size vs tuning curvature of a fixed-size model. In the following, we discuss the implications of Theorem 3.1 for fixed-size models, and leave further theoretical analysis to future work.\nThe proposed CT method is grounded in the theory of Affine Spline Operators (ASOs). By tuning the curvature of each activation function, CT allows to increase local expressivity of a model (i.e. local curvature). \nImportantly, while the hypothesis space of ASOs parameterized by sufficiently deep ReLU networks enjoys universal approximation ability [1], Th 3.1 shows that once a model size is fixed, affine spline operators are characterized by zero (i.e. vanishing) local curvature a.e. in their domain. \nImportantly, to approximate functions of non-zero local curvature, increasing model size of ASOs provably yields non-zero error in the non-asymptotic limit, due to their local affine nature. This is further discussed in the motivating example in Section 3, as well as Appendix C.2.\n## Interpretability of CT\nAs acknowledged by reviewers cxvD and W2ME, CT is interpretable in that it exposes two learnable parameters per neuron, which provably modulate the activation function’s curvature. In turn, this controls the curvature of decision boundaries, which is intimately tied to both generalization and robustness [2,3].\nTo see this, we note that for a deep network $f$ , the decision boundary of any class $i$ and $j$ is given by the set of $ \n\\{ x \\in R^d : g(x) := f_i(x) − f_j (x) = 0 \\} $, for $i \\ne j$. Particularly, $g$ is itself a deep network, sharing the same parameters as $f$ up until the penultimate layer. Hence, modulating non-linearity of activation functions via $\\beta$ directly controls the curvature of both model function and its decision boundaries, which is provably connected to generalization and robustness [2,3]. We are happy to add a paragraph to the main paper clarifying this relationship and its connection to interpretable curvature tuning.\n## Performance on Transformers\nThe paper posits two potential reasons for the underperformance of Trainable CT vs LoRA on ReLU Swin-T/S models pretrained on Imagenette:\n1. Trainable CT uses significantly fewer parameters than LoRA (0.71% and 0.58% on Swin-T/S).\n2. The small size of the Imagenette pretraining dataset results in weaker representations.\n\nTo investigate the second factor, we conducted an additional experiment using ImageNet-pretrained Swin-T/S models, with GELU activations (equivalent to CTU with $\\beta=0.6403$ and $c=1$).\n\n|Dataset|Swin-T|||Swin-S|||\n|------------------|--------|------|---------------|--------|------|---------------|\n|                 |Frozen|LoRA|Trainable CT|Frozen|LoRA|Trainable CT|\n|Arabic Characters|77.62|**93.57**|86.10|75.30|**94.58**|85.06|\n|Arabic Digits|98.06|**99.12**|98.04|98.10|**99.18**|97.88|\n|Beans|88.28|**95.31**|92.19|87.50|**92.97**|89.84|\n|CUB-200|73.59|**76.82**|72.30|72.16|**77.68**|73.04|\n|DTD|**71.17**|69.73|70.85|70.16|70.48|**72.82**|\n|FashionMNIST|89.75|**92.95**|89.10|89.81|**93.45**|86.68|\n|FGVC-Aircraft|39.48|47.16|**47.73**|34.47|**52.42**|45.09|\n|Flowers102|84.52|83.25|**85.41**|81.36|81.43|**85.04**|\n|Food101|77.05|**82.30**|78.02|77.72|**84.47**|75.89|\n|DermaMNIST|76.51|**77.11**|75.21|76.66|**77.41**|75.46|\n|OCTMNIST|67.00|**76.70**|67.30|66.80|**77.00**|64.30|\n|PathMNIST|90.65|**92.56**|91.24|89.89|**91.91**|90.33|\n|**Average**|77.81|**82.22**|79.46|76.66|**82.75**|78.45|\n\n**Results:** *Trainable CT* achieves an average relative improvement over the frozen baseline of 2.92% (Swin-T) and 3.76% (Swin-S). The gap with LoRA is considerably narrower (3.19% and 5.42%) than that observed in the ReLU experiment on Imagenette. This supports our second hypothesis: the earlier performance gap was partly due to weaker representations from small-scale pretraining, which amplified the advantage of LoRA’s higher parameter count.\nThese new results suggest that *Trainable* CT can achieve competitive performance while using only 0.71% and 0.58% of LoRA’s trainable parameters, underscoring its strong parameter efficiency. Importantly, CT and LoRA operate from orthogonal perspectives and can be combined for improved performance—our goal is not to outperform LoRA directly, but to offer a complementary PEFT method. Additionally, this experiment showcases the generality of Train CT: which covers GELU-based models.\n\n## CT vs. Higher-rank and Tuned-$\\alpha$ LoRA\nWe extend our LoRA comparison to higher ranks and tuned $\\alpha$ values on ImageNet-pretrained ResNet-18 and Imagenette-pretrained Swin-T (ReLU-based). More specifically, we test LoRA with $r \\in {1, 2, 4}$ and $\\alpha \\in {r, 2r, 4r}$ and report the best performance.\n\n| Dataset | ResNet-18 |  |  |  | Swin-T |  |  |  |\n|---|---|---|---|---|---|---|---|---|\n|  | Trainable CT | LoRA $r=1$ | LoRA $r=2$ | LoRA $r=4$ | Trainable CT | LoRA $r=1$ | LoRA $r=2$ | LoRA $r=4$ |\n| Arabic Characters | 93.76 | 93.37 | 95.06 | **96.19** | 41.95 | 56.32 | 68.36 | **77.86** |\n| Arabic Digits | 99.03 | 99.08 | **99.22** | **99.22** | 90.82 | 97.54 | 98.28 | **98.62** |\n| Beans | 94.01 | 93.23 | 96.09 | **97.66** | 68.49 | 75.52 | 80.47 | **82.81** |\n| CUB-200 | **64.30** | 54.83 | 56.45 | 58.15 | 6.09 | 7.42 | 9.03 | **11.36** |\n| DTD | **63.62** | 54.36 | 56.17 | 57.77 | 17.04 | 16.99 | 20.64 | **20.74** |\n| FashionMNIST | 91.07 | 91.65 | 92.79 | **93.42** | 77.07 | 83.90 | 86.83 | **88.73** |\n| FGVC-Aircraft | **46.44** | 29.19 | 32.43 | 38.76 | 6.14 | 5.59 | 5.91 | **7.41** |\n| Flowers102 | **86.55** | 67.53 | 68.92 | 72.89 | 16.53 | 16.66 | **20.36** | 19.91 |\n| Food101 | 66.04 | 64.40 | 66.00 | **66.75** | 15.20 | 18.17 | 21.54 | **25.80** |\n| DermaMNIST | **77.66** | 74.21 | 76.46 | 77.36 | 71.37 | 74.08 | 75.41 | **76.16** |\n| OCTMNIST | 69.53 | 74.27 | **77.30** | 76.10 | 53.23 | 63.53 | 66.80 | **71.40** |\n| PathMNIST | 87.17 | 87.62 | **89.36** | 89.28 | 77.35 | **81.31** | 79.97 | 76.16 |\n| **Average** | **78.26** | 73.64 | 75.52 | 76.96 | 45.11 | 49.75 | 52.80 | **54.75** |\n\n**Results:** On ResNet-18, *Train CT* outperforms LoRA with rank 1/2/4 by 10.20%, 6.63%, and 3.22%, respectively, indicating that CT remains highly effective even when compared to stronger LoRA baselines. While for Swin-T, *Trainable CT* trails LoRA (rank 1/2/4) by 8.29%, 16.30% and 21.17%, respectively.\n\n## Setting $c=0.5$ for CTUs\nThe rationale for choosing $c = 0.5$ is explained in the last paragraph of Sec 3.1. Since the SiLU and SoftPlus tend to shift the mean of the activations in opposite directions (negative for SiLU and positive for SoftPlus), setting $c = 0.5$ balances these effects and helps mitigate the shift, making the mean output closer to ReLU.\n\nTo demonstrate why this is beneficial, we conducted an additional ablation comparing Steering CT vs SiLU- and SoftPlus-only variants (both reparameterized by CTU) on ResNet-18 and ResNet-50:\n|Dataset|ResNet-18|||ResNet-50|||  \n|-------|---------|----|--------|---------|----|--------|  \n||CT|SiLU|SoftPlus|CT|SiLU|SoftPlus|  \n|Arabic Characters|**87.65**|81.96|84.89|83.66|**86.46**|81.10|  \n|Arabic Digits|**98.77**|97.99|98.64|98.37|**98.56**|98.41|  \n|Beans|**90.36**|88.80|88.02|**91.93**|86.46|91.15|  \n|CUB-200|**63.18**|62.92|63.08|64.62|65.06|**65.61**|  \n|DTD|62.66|62.89|**63.24**|66.91|67.18|**67.20**|  \n|FashionMNIST|88.70|88.52|**88.93**|90.34|89.92|**90.35**|  \n|FGVC-Aircraft|**38.68**|36.31|37.44|**41.16**|37.91|40.09|  \n|Flowers102|81.97|80.37|**83.14**|**83.84**|83.77|83.69|  \n|Food101|62.27|61.34|**62.76**|68.02|**68.04**|67.90|  \n|DermaMNIST|75.05|74.63|**75.11**|**75.89**|75.74|75.79|  \n|OCTMNIST|**67.27**|64.90|66.57|**68.00**|67.30|67.00|  \n|PathMNIST|87.51|86.81|**87.84**|**90.26**|90.03|**90.26**|  \n|**Average**|**75.34**|73.95|74.97|**76.92**|76.37|76.55|\n\n**Results:** The SiLU version trails steering CT by 1.98% and 0.94% on ResNet-18 and ResNet-50, respectively. And the SoftPlus version trails steering CT by 0.53% and 0.54%, respectively. These results highlight the performance degradation caused by the output shift in simpler formulations, and the effectiveness of balancing the two.\nAdditionally, we observed that the average optimal $\\beta$ for CT is 0.84 (ResNet-18) and 0.94 (ResNet-50), while for SiLU and SoftPlus, the values are closer to 1—0.99/0.95 (SiLU) and 0.94/0.98 (SoftPlus)—suggesting more limited flexibility for steering. This provides further support for the necessity of setting $c = 0.5$, enabling broader adaptability.\n\n## Applicability of CT in More Contexts\nAt its core, CT provides a mechanism for steering model behavior and is not tied to any specific modality. The steering CT can be applied directly in any domain via a lightweight grid search procedure, while *Trainable CT* only requires a task-specific objective, similar to other PEFT methods such as LoRA. Given this flexibility, we believe CT can be readily extended to domains beyond vision. While we focused on image classification in this paper, we agree that applying CT to language model finetuning (e.g., for GRPO) is an exciting direction for future work.\n\n\n## References\n[1] Universal Function Approximation by Deep Neural Nets with Bounded Width and ReLU Activations. Hanin. Mathematics. 2019.\n\n[2] Spectrally-normalized margin bounds for neural networks. Bartlett, et al. NeurIPS 2017.\n\n[3] Robustness via Curvature Regularization, and Vice Versa. Moosavi-Dezfooli, et al; CVPR, 2019"}]

Author rebuttal:

Meta-review: This paper introduces a novel steering method that manipulates activation functions through a single-parameter injection, presenting an underexplored pathway in the PEFT domain. All reviewers recognize its significant contribution to the community, citing: (1) a refreshing idea, (2) strong theoretical grounding, (3) strong empirical performance (albeit under a limited scope), and (4) a well-organized manuscript with thoughtful insights.

From this AC’s perspective, the primary concerns centered on the evaluation scope and the method’s applicability, which was limited to ReLU (e.g., with all GeLUs replaced by ReLUs). The authors successfully addressed these points during the rebuttal, resolving them during the discussion.

Although some reviewers noted extra minor weaknesses, this AC believes the paper’s strengths clearly outweigh these concerns, which can be addressed in revision. This AC concurs with the reviewers’ positive assessments and recommends acceptance.
Decision: Accept (poster)