Improving Deep Learning Optimization through Constrained Parameter Regularization
Authors: Jörg K.H. Franke
Michael Hefenbrock
Gregor Koehler
Frank Hutter
Venue: NeurIPS 2024 poster (2024)
Keywords: Deep Learning
Regularization
Augmented Lagrangian
Primary area: optimization_for_deep_networks

Abstract: Regularization is a critical component in deep learning. The most commonly used approach, weight decay, applies a constant penalty coefficient uniformly across all parameters. This may be overly restrictive for some parameters, while insufficient for others. To address this, we present Constrained Parameter Regularization (CPR) as an alternative to traditional weight decay. Unlike the uniform application of a single penalty, CPR enforces an upper bound on a statistical measure, such as the L$_2$-norm, of individual parameter matrices. Consequently, learning becomes a constraint optimization problem, which we tackle using an adaptation of the augmented Lagrangian method. CPR introduces only a minor runtime overhead and only requires setting an upper bound. We propose simple yet efficient mechanisms for initializing this bound, making CPR rely on no hyperparameter or one, akin to weight decay. Our empirical studies on computer vision and language modeling tasks demonstrate CPR's effectiveness. The results show that CPR can outperform traditional weight decay and increase performance in pre-training and fine-tuning.

Improving Deep Learning Optimization through Constrained Parameter Regularization
Jörg K.H. Franke University of Freiburg, Germany
Michael Hefenbrock
RevoAL, Karlsruhe, Germany
Gregor Koehler
Frank Hutter
German Cancer Research Center (DKFZ) Heidelberg, Germany
ELLIS Institute Tübingen, Germany University of Freiburg, Germany
Abstract
Regularization is critical component in deep learning. The most commonly used approach, weight decay, applies a constant penalty coefficient uniformly across all parameters. This may be overly restrictive for some parameters, while insufficient for others. To address this, we present Constrained Parameter Regularization (CPR) alternative to traditional weight decay. Unlike the uniform application of a single penalty, CPR enforces an upper bound on statistical measure, such as the L2-norm, of individual parameter matrices. Consequently, learning becomes a constraint optimization problem, which we tackle using an adaptation of the augmented Lagrangian method. CPR introduces only a minor runtime overhead and only requires setting an upper bound. We propose simple yet efficient mechanisms for initializing this bound, making CPR rely on no hyperparameter weight decay. Our empirical studies on computer vision and language modeling tasks demonstrate CPR's effectiveness. The results show that CPR can outperform traditional weight decay and increase performance in pre-training and fine-tuning.
one, akin
Introduction
Deep neural networks are the bedrock of many state-of-the-art machine learning appli- cations [I] While these models have exhib- ited unparalleled expressivity, they also possess millions, sometimes trillions, of parameters (2). This massive capacity makes them susceptible to overfitting, where models memorize nuances of the training data but underperform on unseen examples. To mitigate this, many different reg- ularization techniques have been adopted, with weight decay and L2 regularization [ 5 be- ing the most popular. L2 regularization penal- izes the squared magnitude of model parameters and (decoupled) weight decay (which equiv- alent to L2 regularization for non-adaptive gra- weights with constant at every step. This seemingly simple act offers numerous benefits by curbing the growth of individual weights, reducing the risk relying any partieular feature excessively, and thus promoting model generalization.
AdamCPR 200k
18.8
Adam W 200k
Perplexity
18.6
AdamW 300k
18.4
18.2
18.0
100k
200k
300k
Optimization ion Steps
Figure 1: GPT2s training using Adam with weight decay or CPR (Kappa-IP). AdamCPR outper- forms Adam with the same budget and only re- quires 2/3 of the budget to reach the same score.
dient algorithms (6) multiplics
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

However, not all parameters in neural network have the same role or importance and different weights could benefit from different regularization. Similarly, is unclear if a single weight decay value is optimal for the entire duration of optimization, especially for large-scale training. Indeed, Ishii and Sato 7 showed that small deep learning model could benefit from layer-wise weight decay values, and various works showed that scheduling weight decay could improve final performance [8] 2000 This indicates that a dynamic penalty for each individual parameter matrix (c.g.. a weight matrix in linear layer) could be beneficial for neural network training. Since both scheduling and parameter-wise weight decay require additional hyperparameters that are often sensitive to the task, propose different approach to obtain customized, dynamic parameter regularization. Instead of uniformly penalizing weights, we propose to keep them in a certain range, thus ensuring stability without imposing regularization where it is unnecessary. Constraining parameters, especially based on statistical measures like the 12 norm, provide a flexible and adaptive form of regularization that accounts for the heterogeneity of parameters.
In this paper, we propose Constrained Parameter Regularization (CPR), which enforces an upper bound on statistical measure of individual parameter matrices. Consequently, regularization is expressed as a constrained optimization problem, which we address by an adaptation of the augmented Lagrangian method. The regularization of each parameter matrix is handled by a separate constraint and Lagrange multiplier, resulting in an individual regularization strength that adapts over time. The method requires the selection of desired constraint values well as update rate for the Lagrange multipliers. We found that the update rate can be fixed to 1.0. For choosing constraint values, introduce four strategies, three of which require single hyperparameter, while the last one is hyperparameter-ffee. We show in our experiments performance improvements over weight decay when pre-training or finetuning models for image classification (CIFAR 100 and ImageNet), language modeling Wepee,,,eext,, and medical image segmentation. For example, when training a GPT2s model, we achieved the same performance AdamW but only require 2/3 of the budget, see Figure Applying our method for fine-tuning, we find performance improvements and less catastrophic forgetting. In the following, and after discussing related work (Section 2 and background on weight decay and the augmented Lagrangian method (Section we make the following contributions:
Introducing CPR for individualized and dynamic weight regularization Specifically, formu- lating regularization as a constraint optimization problem and proposing CPR as a solution (Section 4.1).
Identifying four different strategies for initializing this constraint (Section 3 Kappa-WS, has a strong default that outperforms tuned AdamW; and another one, Kappa-IP, is entirely hyperparameter-free and yields even better performance in pre- pre-training.
One of them,
Showing improved performance over weight decay in image classification, medical image segmentation, and pretraining and fine-tuning language models (Section(5).
2 Related Work
Weight decay is an effective regularization technique to improve the generalization and model performance [12], and the idea of adapting parameter regularization during training is not new. Lewkowycz and Gur-Ari [8] investigated the effect of L2 regularization on overparameterized networks and found the time it takes the network reach peak performance is proportional to the 122 regularization parameter. They proposed an initialization scheme for L2 regularization and an annealing schedule for the L2 parameter. Yun al. ( use combination of weight decay scheduling and knowledge distillation to improve performance on computer vision tasks. More recent works on self-supervised vision transformers also use weight decay schedule [IO] [ In contrast to our work, none of these proposes a dynamic and individual adaptation of each regularized parameter matrix. Also, a schedule comes with varying hyperparameter choices while CPR adapts arbitrarily many parameter matrices with only two hyperparameters (out of which one is fixed in all our experiments). Instead of using a schedule, Nakamura and Hong [33 proposes AdaDecay. where the La penalty is scaled by standardized gradient norms and sigmoid function. Ghiasi et al. [14] propose another gradient-base
approach, Adaptive Weight Decay (AWD), which dynamically adjusts the weight decay based on the ratio of weight norms to gradient norms to balance the contributions from the cross-entropy and regularization losses aiming to improve the robustness. AMOS (15
Please find our implementation under http://twww...coo/a/toott//e//e
com

---

leverages model-specific information for initialization and gradients to adapt L2 regularization during the training. Another way to regularize parameters fix the norm of individual parameter matrices [16]. to schedule the weight norm 17 or to limit the total norm of all parameters [18] to
fixed value. This fixed value is a more sensitive hyperparameter than the hyperparameter our work.
Our proposed method is not the first to use Lagrangian methods in machine learning 19] Its application in deep learning so far focuses on variational methods and generative models: Rezende and Viola (20] introduced the Generalized ELBO with Constrained Optimization algorithm to optimize VAEs using Lagrange multipliers optimized by the min-mat max scheme, and Kohl al. 22 and Franke et al. [22] adapted the Lagrangian method from Rezende and Viola [20] to train probabilistic U-nets and probabilistic Transformer models. While these works adopt Lagrangian methods handle several losses in joint optimization problems, our work leverages them to enable individual regularization strengths.
Background
3.1 Lg Regularization and Weight Decay
Regularization methods, such as Le-regularizztion or weight decay, are commonly used to restrict parameter updates and enhance generalization by reducing unnecessary complexity 445 Both can be motivated by introducing "cost" to weight magnitudes. Specifically, in 1 -g-reuularization, instead of minimizing only the loss function L(O, X, y) with parameters 0 and data D {(X., y/ ) weighted penalty (regularization) term R()) is added to the loss, resulting in the training objective
min L(O, x y) + R(O),
where R(0)- 1 1111 denotes the regularization function and E IR the strength of the penalty. On the other hand, weight decay directly modifies the update rule of the parameters to
O +- O, + Opt(L, n) 1 7 71-7.00 O.,
where Opt(L, 7) denotes an optimizer providing the gradient-based update at iteration and == T.O,,, Xx, yr) the loss. For example, Opt (),, = = VoI.(O,, X,, yt) with learning rate E IR+ in case of gradient descent. Thus, the main difference between weight decay and L2- regularization is that the gradients of the regularization accumulate in momentum terms in the case of Le-regularisation, while they are treated separately in (decoupled) weight decay. This has also been extensively discussed by Loshchilov and Hutter (6] with the introduction of the AdamW optimizer.
3.2
The augmented Lagrangian method
We briefly review the augmented Lagrangian method for constrained optimization, see e.g. Bertsekas 23 which our method is based on. For the derivation, we follow the motivation of Nocedal and optimization problem
Wright 24 pp. 523-524). Consider the following
minimize f() s.t. c(x) O,
with fxx : Rd + R and a constraint c(x) R" + E. One way address the constraint is to find an equivalent, unconstrained problem with the same optimal solution. For example,
minimize F(x) with F(x) = max f(x) + A c(x). X20
(I)
22
Unfortunately, if f (ax) and c(x)) differentiable, F(x:) is not differentiable. This is due the maximization over X in F(x). where in case of c(x) > 0, F(w) + co. Consequently, we cannot run gradient- optimization on this objective.
To alleviate this problem, we consider a smooth approximation of F(z), namely
1 (A 2,)
F(x, Ae, 1)) max f(x) + x-(x)) X
(2)
2μ
where d E I may be seen as point we wish to remain proximal to and μ I as a factor determining the strength with which this proximity is enforced. For μ. co, F(x, de, H) F(x).
3

---

The maximization in F(x, Ae, μ() has a closed form solution with X = (Ac + e(x))², where ()+ max(0, J) see Appendix A for the derivation. Consequentl, F(x, Ae, 11) can be written
(1, f(x) + haar,(,,,1,1
with
cl,)) {c(x)), if X μ c(x) 2 2p X
h(x, Ao. p4)
else.
The constraint thus only interferes with the minimization (gradient) f()) if X + μ (x-c(()) 2 0. can now try to solve the unconstrained problem minimize F(x At, with familiar methods, such as gradient descent, and obtain an approximate solution to the original problem. Specifically, the gradient of F(x de: p)) with respect to æ is given by
Va (ax, At, 11) = Vaf(x) X v
The quality of the approximation, and thus the solution, clearly depends l and X To improve this approximation we can refine the choice X via an iterative procedure and repeat the optimization with Ac+1 + X = (A, + c(x))++ Intuitively, if the previous minimization of F(x, Ae, 14) resulted in an infeasible solution with c(x) > 0, A++1 > Aς- Hence, the minimization of F(x, Ae+1. p)) likely results in solution with less constraint violation. On the other hand, c(x) I O, At+1 S X Subsequently, the influence of the constraint is decreased. This loop of alternating minimization of F(x, Ao μ)) and updating Å can be repeated until sufficiently good solution is found the procedure converges if XS does not receive updates anymore. For multiple constraints (x), j = 1, -. .J, the above can be readily extended with multiplier X for each constraint. Since the maximization in the smooth approximation is separable in the A/. the same update rule may be applied for each X separately using the respective constraint (æ)).
Constrained Parameter Regularization
In this section, introduce Constrained Parameter Regularization (CPR), where adapt the augmented Lagrangian method to enforce upper bounds on regularization terms. Compared to classical regularization, with a fixed regularization coefficient y, the proposed approach will allow for variable regularization coefficients X (Lagrange multipliers) for j 1.... , parameter matrices 0 C 0 that should be regularized. These regularization coefficients are updated alongside the network parameters O.
4.1 Regularization through constraints
Classical weight decay, as introduced earlier, is used as a means to restrict the freedom parameter adaptation. This restriction is applied with scaling factor 7 (hyperparameter) and applies uniformly to all parameters. However, we conjecture that applying an individual adaptation pressure instead may be beneficial. Unfortunately, this would require separate coefficient for each parameter matrix where a separate weight decay should be applied. To avoid the need for separate scaling coefficients, formulate regularization
constrained problem. Here, the loss function L(0, X, y), with network parameters 0, takes the place of the objective. Consequently, the learning problem becomes
minimize L(0, X, y)
Cy (0') R(O') x²¹ < O, for j 1,.....
where R(0i) a regularization function (e.g., the squared L2-norm in case of weight decay) for parameter matrix 0 C 0, 1, J, and k² E R denotes a chosen bound.
To solve equation 3 follow the augmented Lagrangian method with slight modifications. First, instead of performing full optimization of the loss before updating Aç, we perform updates in every step. This motivated by the fact that full optimization is generally infeasible in a deep learning setting. Moreover, similar to the difference between weight decay and Lz-regularization, we treat the update between the loss boss-dependent and the munstrannt-eeennder part separately. Hence, instead of introducing i..,,, >e, ju) analogously to equation 2 and performing optimization on this objective, we independently apply updates for both steps. Consequently, the constraint violations do not accumulate

---

Algorithm Optimization with constrained parameter regularization (CPR)
Require:
Loss Function L(0, X, y) with parameters 0, and data D {(X,, yn J n=
Require:
Hyperparameters: Learning rate E R Lagrange multiplier update rate E (= 1.0)
Require: Optimizer Opt(-) for minimization, Regularization function R(0) (e.g. L2-norm)
X 0 for -,
2: xf Initialize (%) for j 1, 1,...
Initializing the upper bound K, see Section 4.3
3: for Xx, yt ~ D do
4: 0++1 0. + (p0, X v1),
Classic parameter update using, e.g., Adam.
5:
for each regularized parameter group 0% in 0, do
6:
1 + (\} + (R(O)) - x²))
7:
0/41 0'+1 V R(O,)) N2+1
8: end for
9: end for
in momentum terms. We also remove the influence of the learning rate on the regularization, From practical perspective, our modification does not interfere with gradient- based optimization algorithms and can be readily combined with any such optimizer. The full algorithm is given by Algorithm
Conceptually, the method can be understood as the XL accumulating constraint function values (weighted with μ) over the iterations These then increase (or decrease) the influence of the constraint (via its gradient) on the search direction. When points in the feasible domain are found for which C (0) < 0, X decreases until it eventually reaches O. If, on the other hand, the optimal solution lies the boundary, where ey (0) 0, X should converge the optimizer and the gradient the constraints cancel each other. However, this situation is unlikely to occur in deep learning setting due the stochasticity of minibatches.
value where the update direction of
4.2 How is CPR different from weight decay?
The optimality conditions of the CPR problem and an Læ-regularized training objective reveal connection between the two approaches. To see this, consider the training objective of L2 regu- larization with given 7, assuming it has minimum at 0*. Consequently, at this point, have 0 = VL + VR(0"), and the value of the regularization function is R((*).
If set K* R(0*), the Karush- Kuhssh-KKhn--aa Kuhn--uckerr Tucker (KKT) (optimality) conditions for CPR = (L(0*) + À VR(O*) and R(0*) x* S 0 (which holds with equality), with the Lagrange multiplier O. We can that for X y. the solution pair (0*, satisfies the KKT condi- tions. Hence, there is a choice of к (namely k²*) for which the CPR problem has the same optimal solution candidates the Læ-regularized training objective for given y. CPR could therefore be seen as a different approach to searching for the same solution candidates but is parameterized with different hyperparameters (œ instead of 7). Unlike Le-regularization (or weight decay), CPR can mimic the behavior of different values for different parameter matrices. This behavior changes over time as the X values are updated and thus leads to different training dynamies compared to weight decay. Additionally, focusing on bound on the regularization function x instead of a penalty coefficient ? may allow us identify better indicators for the selection of (default) values for these hyperparameters.
4.3 Initialization of Upper Bounds
The upper bound is the most crucial hyperparameter for CPR, and we identify four ways to initialize it. (1) Kappa-K: Set xd +- the same value k for all parameter matrices. (2) Kappa-kIg: Set vd based on the initial parameter matrices' regularization function value: şu <- R(O())) with
k E RR+ as the factor of the initial measure. (3) Kappa-Ws, Train the model parameters f for
specific number of warm start (WS) steps N+ and then set rd < R(0 (see algorithm for
5

---

Adam W
AdamCPR
AdamCPR
1e:4.
70
70
71
7D
71
71
T1
70
70
80
1e-3.5
74
75
75
75
75
76
76
76
76
76
75
74
73
Learning Rate
Currect Label
1e-3.0
75
75
75
75
76
77
76
76
13
1e-2.5
74
74
744
76
75
76
71
74
76
75
75
74
755
70
1e-2.0
73
73
73
76
73
73
65
1o-1.5
68
68
70
72
54
72
58
70
7:3
72
70
69
60
1o-1.0
65
67
64
60
0.0
le-4
1e-3
1e-2
1e-1
250
300
1k
2k
8k 16k
Weight Decay
(Kappa-IP)
Warm start steps (Kappa-Wa WS)
Figure 2: Percentage of correct labels (1) of a ResNet18 trained on CIFAR 100 with Adam and AdamCPR with Kappa-IP or Kappa-WS. We use learning rate warm-up of 500 steps and the best Kappa-WS value 2x the warm-up steps. We report the mean of three runs with random seeds. We see that both CPR versions outperform weight decay
CPR with Kappa-WS in Appendix B While the previous strategies all require a hyperparameter, our last strategy is essentially hyperparameter-fnce (4) Kappa-IP: Use the first inflection point (IP) of the regularization function at step i (change of curvature
the training steps) warm start
each parameter matrix individually. Specifically, x² R(O,..) where is the first iteration where A,A, R(0)) < O. The intuition behind this choice comes from the fact that the rate of change decreases at the inflection point. This hints at a saturation of the improvement expected through raising the value of the regularization function further. The position of the inflection point thus indicates a good choice for K, as demonstrated healthy training dynamics while still restricting the model from over-adapting (see Section 5). Consequently, this method leverages the natural progression of the model's training rather than relying on an external hyperparameter, aiming to adaptively find a suitable upper bound.
5 Experiments
We now describe a set of experiments to understand CPR and its parametrization. Preliminary experiments showed that μ not a sensitive hyperparameter and we chose μ 1.0 for all our experiments. We provide a detailed analysis of in Appendix C Similar to weight decay, we choose the squared L2 norm as a default regularization function for CPR. We also tested an adaptive bound, where we adjusted kappa during training but found it not to be beneficial; details are reported in Appendix D In the following experiments, we regularize all parameters a network except for bias terms and normalization weights. Since CPR does not require additional gradient calculations or parameter updates, we find only small runtime overhead with our CPR implementation (in Py Torch, no CUDA optimization, 0.4%-5.8%% for GPT2) which is mentioned in each experiment individually and analyzed Appendix[]
5.1 Train Image Classification Model (CIFAR100)
To evaluate CPR's effectiveness and design choices, we tested AdamW and Adam with CPR (Adam-
CPR) in image classification using ResNet18 on the CIFAR 100 dataset 25
26 We compared
AdamW to AdamCPR with the four initializations described in Section 4.3 The initialization
Kappa-WS after warm steps performed best, see Figure 2 We base our choice of the warm start on the 500 steps learning rate warmup out of 20k total training steps and found a large range of hyperparameters that consistently outperform weight decay. Also, the hyperparameter-ffcee method Kappa-IP outperforms weight decay. To detect the infection point, we found it sufficient to sweep the statistical measure an interval of 10% of the learning rate warmup. We also apply this in all further experiments. The superior performance of Kappa-WS and Kappa-IP may be due to its general flexibility, as warm-started bounds may considered "learned," reflecting the actual magnitudes and distributions of the parameter matrices during training. Appendix E contains training details and plot with all initializations and standard deviation across three random seeds in Figure H.T ResNet18 training took 15-20 minutes consumer GPU, with no significant runtime difference between

---

Table 1: Comparison of AdamW and AdamCPR in a DeiT (2 pertaining on ImageNet. We train small (22M parameters) and base model (86M) with different regularization parameters.
ImageNet Pretraining
A
AdamCPR
weight decay
Kappa WS (x Ir-warmup)
Kappa IP
0.005
0.051
0.5
1x
2x
4x
DeiT-Small (22M)
Top-1 Acc. (%)
76.97
79.03
79.16
79.81
79.33
78.04 TBA
79.84
DeiT-Base (86M) Top-1 1 Acc. (%)
76.19
78.59
80.56
81.19
79.61
80.95
AdamW and AdamCPR. We also tested the standard deviation as a choice for the regularization function, which performed well but not better than the squared L2 norm (see Figure E.2
To investigate the relationship between the learning rate warm-up and the number of warm start steps of Kappa-WS or Kappa-IP, we experimented with varying warm-up steps. We found that setting the CPR warm start steps s twice the warm-up steps is good initial choice. For very low warm-up steps, the best was four times the warm-up count. Conversely, with long warm-up phase, shorter CPR warm start (x1) is preferable. Notably, the optimal choice of is almost independent of the learning rate, as shown in Figure E.3] The optimal warm start steps are consistent across a wide range of learning rates. A simple baseline representing a similar regularization approach a weight decay schedule. We evaluated a cosine schedule for decreasing and increasing weight decay values, similar to 10 [II The results, shown in Figure E.4 indicate that the decreasing schedule outperforms a fixed weight decay but not CPR. We tested il CPR is particularly good for noisy data and perfomed experiments on the noisy CIFAR100-C dataset (27]. The results, in Figure E.5 show that AdamCPR outperforms AdamW slightly. However none of the optimizer and hyperparameter configurations lead to an outstanding performance on this task, we wouldn't claim that CPR is particularly good for noisy data. We also used CPR with SGD. We found, as shown in Figure E.6 that SGD with CPR outperforms SGD with weight decay when using the Kappa-WS initialization. However, IIppa-IP seems not to work with SGD, probably due to the changed convergence behavior in contrast Adam.
Additionally, we compared our method to related work. We implemented AdaDecay [13 and evaluated the method for different alpha values, as seen in Figure E.7 We also compared AdamW and AdamCPR to adaptive Weight Decay (AWD) [14] and AMOS 1SI. Furthermore, we used Adam with parameter rescaling from Liu et al. [8]. We found AdaDecay superior to AdamW, while AMOS and Rescaling performed less well. However, CPR outperforms all related approaches. We report all results across multiple learning rates and weight decay values in Figure E.8
5.2 Train an Image Classification Model (ImageNet)
We compare AdamW and AdamCPR in vision transformer 22 training on ImageNet 30 We choose to train the DeiT 28 model with 22M (small) and with 86M (base) parameters. We make use of the PyTorch Image Models library 3 and train with the configuration given in 22 for 300 epochs. To explore the impact of weight decay, we also train with 10x and 0.1 x the weight decay value. For CPR, we initialize with Kappa-WS (x Ir-warmup) and IP. We observed a minor runtime increase when using CPR. For example, training the small model on 4 A100 GPUs took 14.85h for AdamW and 14,89h for AdamCPR. All relevant hyperparameters can be found in Appendix F As seen in Table AdamCPR outperforms AdamW V for small and base DeiT training with both kappa initialization methods. Most notably, the hyperparameter free regularization with Kappa-IP outperforms AdamW in both cases. However, the base model training, Kappa-Wa S surpasses Kappa-IP. IP.
5.3 Fine-tuning CLIP model
We conducted fine-tuning experiments using CLIP model 33 on the ImageNet dataset. We used the ViT-B/32 model pre-trained by Radford et al. 33 The model was fine-tuned for 10 epochs following the hyperparameter choices of Wortsman et al. R2 (learning rate of 3 x 10-5, cosine- annealing learning rate schedule with 500 warm-up steps) but without the special classification head initialization and the training was performed on a single GPU with batch size of 512. We compare
7

---

Table Comparison of AdamW and AdamCPR for CLIP finetuning on ImageNet. We report the top-1 accuracy and follow the hyperparameters and schedule from WiSE-FT 32].
ImageNet Finetuning
AdamW
AdamCPR
weight decay
Kappa WS
Kappa IP
0.0001
0.001
0.01
0.1
1.0
2x
4x
Top-1 Acc. (%)
75.24
75.39
75.32
75.17
74.4
75.27
75.52
75.41
75.40
Adam W
AdamCPR
Ada
Learning Rate
1e-3.0
=0.02
18.56
+0.03
18.46
+:0.02
18.32
+:002
17.08
=0.04
18.03
+0.01
18.14
#0.02
18.35
GPT2s
1o:2.5
=0000
18.45
+0.01
18:23
+0.02
18. 86
+0004
17.97
20.03
18.02
+0002
18:03
+0.03
18.24
Perplexity
18.00
18:34
20.51
18
18,
10.17
+0.03
1:0001
1.005
0.00
1:005
1:0008
0.001
0.01
0.1
5k
10k
20k
GPT2m
1e-2.5
16.37
16.04
16.52
15. 58
15.65
15.72
16. 10
=0.01
10.01
+:0.00
+:004
10.01
+:0.2
+0.03
0.001
Weight. Decay
(Kappa-1P)
Warm start steps (Keppa-WS)
Figure 3: Perplexity (1) + std across three random seeds of GPT2s and GPT2m trained on OpenWeb- Text with A (left) and AdamCPR with Kappa-IP (middle) and AdamCPR with Kappa-WS (right). We a learning rate warm-up of 5 steps. The CPR with the hyperparameter-Ine strategy Kappa-IP IP outperforms weight decay but also CPR with warm start.
AdamW W with different weight decay values to AdamCPR in different configurations, where we report the top-1 accuracy after finetuning. The results in Table show that the Kappa-WS initialization also leads to better results in this finetuning setting, comparing favorably to traditional weight decay. CPR with Kappa- IS performs similarly to the best weight decay values, but again, without the need for finding a regularization hyperparameter.
5.4 Pretraining a Large Language Model (Open WebText)
We performed experiments training a GPT2 language model 334 on Openwebtext [35]. We compared AdamW on different weight decay values to AdamCPR using Kappa-WS with different warm start steps and Kappa-IP. We use a learning rate warmup for 5k steps (2.5% of total training steps) followed by cosine annealing. Again, we select the warm start steps of based the warmup steps of the learning rate and evaluate s E (5k, 10, 20k) steps. We train the model sizes GPT2s and GPT2m with 124M and 354M parameters for 200k steps. The results are shown in Figure CPR outperforms weight decay at all learning rates, in both model sizes and with both kappa initialization strategies. We also see that the Kappa-IP IP initialized CPR are less sensitive the learning rate than weight decay Remarkably, CPR with the hyperparameter-fiee initialization Kappa-IP performs best, achieving 0.2 0.3 better perplexity than weight decay. To illustrate the performance difference, we trained a model with weight decay for longer schedule to get the same performance as with CPR, the result is shown Figure CPR saves up 33% training budget on that scale. Figure shows the difference in training dynamics with CPR. We find that Kappa- IP is close to the optimal warm start step for Kappa-WS but find individual starting points for different layers, see Figure G. We provide details of the training and hyperparameters in Appendix H. We found no runtime overhead of CPR in contrast to AdamW training GPT2s but about 2.5% for GPT2m (see runtime analysis in Appendix Π]. We also evaluated AdaDecay [3), Adaptive Weight Decay (AWD) [14] and AMOS [S] the GPT2s training setting but neither of the related methods outperforms AdamW nor AdamCPR, results in Table H.I

---

MistraliB PubMediQA
Adam
Adam (Kappa-W5:
% Aocuracy Improviieut
Learning Rate
1o.4.5
#0.46
3.8
+0
3.8
=0.
+0.57
4.0
+0.22
4.2
#0.55
4.2
10-4
#0.57
3.7
+1.07
3.8
#0.78
3.4
#1.21
4.0
+0.59
4.0
+0.45
3.8
1o-3.5
1.0.6
3.1
10.922
20.70
1.039
10.37
11.15
0.001
(4x) Warm start steps (x Ir warmup)
[1x)
(2x)
Weight Decay
Figure 4: Percentage of performance change before and after fineuning Mistral 7B with pubmedQA artificial data (t) with the use of Adam (left) and AdamCPR with Kappa-W5 (right). We use learning rate warm-up of 50 steps. We see that CPR outperforms weight decay for each learning rate.
5.5 Fine-tuning a Large Language Model
Probably a more common task than pre-training large language model (LLM) to fine-tune one. Hence, we evaluate CPR in the fine-tuning of the Mistral7B large language model [36]. incorporating low-rank adaptation (LoRA) (37] Specifically, we fine-tune artificially generated biomedical question- answering (QA) pairs from the PubMedQA dataset (38). We fine-tune all attention and feed-forward weights using either AdamW or AdamCPR with learning rate warm-up of 50 steps, followed by cosine annealing. We experiment with different values of weight decay and warm start steps for Kappa-WS, set at 1 x, 2x, and 4x the learning rate warm-up steps. The fine-tuning was performed on four GPUs for about lh. Each configuration is trained across three random seeds. We evaluate the LLM before and after the fine-tuning on the expert-annotated PubMedQA QA instances and report the change in answer accuracy (means and standard deviations across three random seeds) in Figure The fine-tuning enhances the performance on the PubMedQA benchmark and CPR outperforms A for each learning rate. As in both the ImageNet and GPT2 experiments, the best Kappa-WS value was 2x the warm-up steps (here, 50 x 2). We also tested Kappa-IP IP but it performed worse due to the lack of an inflection point for some parameters, short learning rate warmup, and different training dynamics with LoRA. We also found that CPR helps to mitigate catastrophic forgetting, therefore we evaluate before and after finetuning some learning rates helps to reduce a performance drop e.g. on the TruthfulQA benchmark, which evaluates models' abilities mimic human falsehoods 399 on up 3% (see results in Figure K.1 Detailed hyperparameters and plots including standard deviations are available in Appendix K
set of benchmarks and found that CPR with
5.6 Medical Image Segmentation
Aside from image classification, also applied CPR to (medical) image segmentation using the nnU- Net framework [40] and training with the SGD optimizer in combination with CPR with Kappa-WS For this, we considered the tasks of Multi-Atlas Labeling Beyond the Cranial Vault (BTCV) 41 where we improve the Dice score from 83.99 to 84.23, the Heart Segmentation task of the Medical Segmentation Decathlon 42 where improve the Dice score from 92.92 to 93. 03.18 and the 2020 version of the Brain Tumor Segmentation challenge (BraTSS) task [43 where we improve the Dice score from 76.22 to 76.65. These results show that CPR also works in combination with SGD where we replace weight decay. Training details for the task and all results are in Appendix ]
Discussion
Our extensive evaluation of Constrained Parameter Regularization (CPR) across multiple tasks underscores its effectiveness robust alternative to traditional weight decay. A critical aspect of CPR's success its initialization strategy. To this end, we propose four strategies to initialize the upper bound к. With our findings, we identify two strategies, Kappa-WS and Kappa-IP prime candidates showing strong performance, consistent across multiple tasks. The good performance of the warm-started bound Kappa-WS can be attributed to the fact that even carefully chosen initialization of parameters does not consider the training task and data. Therefore, the actual parameter weights during training better reflected in a warm-started bound, which also takes into

---

GPTT2s Training Dynamies of AdamW (blue) and AdanCPR (green) with Kappa-11
00 19
Kappa initialiaaioo as step 7:00 =0.002
Layer Weight
0.0
5.0e-06
0.0e++00
1.0c-04
A
0.0e4-00
Validation
3.25
Loss
3.00
0
25000 50000
75000
100000
125000
150000
175000
200000
Optimization Stepe
Figure The training dynamics of Adam (blue) and AdamCPR with Kappa- IP (green) GPT2s training run. The upper plot shows the squared L2 norm of the first fully connected weight in the fifth layer. Below we see the gradient of the squared 1.2 norm regarding the training steps. After the inflection point (7400), Kappa-IP initializes kappa xO + R(O(=;) and starts the regularization. The third plot shows CPR's lambda enforcing the constraint. At the bottom, we see the validation loss. Adam converges faster in the beginning of the training but CPR leads to a more linear improvement and better final performance.
account the network's depth and the varying gradient updates deeper layers. We found that setting the CPR warm start steps to twice the learning rate warm-up steps serves as an effective initial configuration for any training setup. However a pre-training setting, setting the upper bound based on the first inflection point of the regularization function (Kappa-IP) yields an additional advantage: removes even the one hyperparameter present in the warm start strategy, bringing the regularization capabilities of CPR without any additional hyperparameters. Simultaneously. this strategy shows best-in-class performance in GPT2 training, seemingly even extending the range usable learning rates given task. This reduces the effort in hyperparameter optimization not only for the optimal regularization but also for the optimal learning rate. CPR also changes the training dynamics, as shown in Figure and Figure G.I While both weight decay and CPR can achieve similar final L2 regularization, the path to this norm different. Weight decay allows for intermediate overadaptation with high L2 norms, whereas CPR controls the L2 norm throughout the entire training process. This results in slower initial loss drop but more consistent decay, leading to a better final performance.
A noted limitation of CPR is an increase in runtime by up to 6% for larger models (1.1B parameters), as detailed in Appendix However, for smaller models or larger batch sizes, this overhead is negligible. The benefit of CPR diminishes in scenarios where weight regularization has minimal impact, such as when training small models on large datasets with a high ratio of training samples to parameters. Future research could explore the application of CPR to even larger models and broader range tasks.
Conclusion
Constrained Parameter Regularization (CPR) offers significant advancement in regularization techniques, providing robust and efficient altermative to traditional methods. By enforcing an upper bound on the regularization function, CPR integrates seamlessly with gradient- based optimizers and incurs minimal runtime overhead. Its dynamic tailoring of regularization to individual parameter matrices and reduces hyperparameter optimization by eliminating the need for a weight regularization hyperparameter in pre-training. Our four experiments demonstrate that neural networks trained using CPR outperform those with traditional weight decay. These findings highlight CPR's potential as versatile and powerful tool for improving model performance and open promising future research.
10

---

Acknowledgements
This research was funded by the Deutsche Forschungsgemeinschati (DFG, German Research Foun- dation) under grant number 417962828, We acknowledge funding by the European Union (via ERC Consolidator Grant DeepLearning 2.0, grant no. 101045765). Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Research Council. Neither the European Union nor the granting authority can be held responsible for them.
Funded by the European Union
The authors gratefully acknowledge the Gauss Centre for Supercomputing e. e.V. (wwwggauss-center for funding this project by providing computing time on the GCS Supercomputer JUWELS [44 at Jülich Supercomputing Centre (JSC). We acknowledge the financial support of the Hector Foundation.
References
[1] T. Goodfellow, Y. Bengio, and A. Courville. Deep Learning. MIT Press, 2016.
[2] William Fedus. Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(1)0):1-39, 2022.
[3] Stephen Hanson and Lorien Pratt. Comparing biases for minimal network construction with back- propagation. In Advances in Neural Information Processing Systems, volume 1. Morgan-Kaufmann, 1988.
[4] Anders Krogh and John Hertz. A simple weight docay can improve generalization. Advances in Neural Information Processing Systems, 4, 1991.
[5] S. Bos and Chug. Using weight decay to optimize generalization ability of a perceptron. In Proceedings of International Conference on Neural Networks (ICNN'96), volume 1, pages 241 2-246 vol.1, 1996.
[6] Loshchilov and F. Hutter. Decoupled weight decay regularization. In Proceedings of the International Conference on Learning Representations (ICLR 19). 2019.
[7] Masato Ishii and Atsushi Sato. Layer-wist weight decay for deep neural networks. In Image and Video Technology, pages 276 289, Cham, 2018. Springer International Publishing.
[8] Aitor Lewkowycz and Guy Gur-Ari. On training dynamics of deep networks with / regularization. In Advances in Neural Information Processing Systems, volume 33, pages 4790-4799, 2020.
[9] Juseung Yun. Byungjoo Kim, and Junmo Kim. Weight decay scheduling and knowledge distillation for active learning. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXVI 16, pages 431-447. Springer, 2020.
[10] Mathilde Caron. Hugo Touvron. Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEF/CVF international conference computer vision, pages 9650-9660, 2021.
[11] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07199 2023.
[12] Guodong Zhang, Chaoqi Wang, Bowen Xu, and Roger Grosse. Three mechanisms weight decay regularization. In International Conference on Learning Representations, 2018.
[13] Kensuke Nakamura and Byung- Woo Hong. Adaptive weight decay for deep neural networks. IEEE Access, 7:118857 118865, 2019.
[14] Mohammad Amin Ghiasi, Ali Shafahi, and Reza Ardekani. Improving robustness with adaptive weight
decay. Advances in Neural Information Processing Systems, 36. 2024.
[15] Ran Tian and Ankur Parikh. Amos: style optimizer with adaptive weight decay towards model-oriented scale. arXiv preprint arXiv:22 /666, 2022.
[16] Tim Salimans and Durk P Kingma. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. volume 29, 2016.
[17] Ilya Loshchilov. Weight norm control, arXiv preprint arXiv:2311.11444 2023.
[18] Ziming Liu, Eric J Michaud, and Max Tegmark. Omnigrok: Grokking beyond algorithmic data, In The Eleventh International Conference on Learning Representations, 2023.
11

---

[19]
John Platt and Alan Barr. Constrained differential optimization. In Advances in Neural Information Processing Systems, volume 0, 1987.
[00] Danilo Jimenez Rezende and Fabio Viola. Taming vaes. arXiv preprint a10.00/810000599, 2018.
[21] Simon Kohl, Bernardino Romera-Paredess Clemens Meyer, Jeffrey De Fauw. Joseph R Ledsam, Klaus Maier-Hein, SM Eslami, Danilo Jimenez Rezende, and Olaf Ronneberger. A probabilistic u-net for segmentation of ambiguous images. Advances in Neural Information Processing Systems, 31, 2018.
[22] Jörg Franke, Frederic Runge, and Frank Hutter. Probabilistic transformer: Modelling ambiguities and distributions for ma folding and molecule design. Advances in Neural Information Processing Systems, 35:
21553, 2022.
[23] Dimitri P Bertsekas. Constrained Optimization and Lagrange Multiplier Methods, Athena Scientitic, 1996. ISBN 1886529043.
[24] Jorge Nocedal and Stephen J. Wright. Numerical Optinization. Springer, New York, NY, USA, 2c edition, 2006.
[25] K. Hc, X. Zhang. S. Ren, and Sun. Deep residual learning for image recognition. In Proceedings of the International Conference on Computer Vision and Pattern Recognition (CVPR' 16), pages 770-778, 2016.
[26] A. Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
[27] Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In International Conference Learning Representations, 2018
[28] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning, pages 10347-10355 PMLR, 2021.
[29] Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2020.
[30] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In /EEE/CVF Conference Computer Vision and Pattern Recognition, pages 248-255. IEEE, 2009.
[31] Ross Wightman. Pytorch image models. htttp://ggitaab..mm/rraaaggaaiie wiwpaa///pppoooooonimmeeeeeeeee models 2019.
[32] Mitchell Wortsman, Gabricl Ilharco, Jong Wook Kim, Mike Li, Simon Komblith, Rebecca Roclofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, et al. Robust fine-tuning of zero-shot models. In IEEF/CVF Conference on Computer Vision and Puttern Recognition, pages 7959-7971, 2022.
[33] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages $748-8763 PMLR, 2021.
[34] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever. Language models are unsupervised multitask learners. OpenAl blog, 1(8)99 2019.
[35] Aaron Gokaslan and Vanya Cohen. Openwebtext corpus.
http:///9kylion007gggte 10/
OpenNebTextCorpus 2019.
[36] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06883 2023.
[37] Edward Hu, Phillip Wallis, Zeyuan Allen Zhuu Yuanzhi Li, Shean Wang, Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2021.
[38] Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua La. PubMedQA: dataset for biomedical research question answering. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)) pages 2567-2577. Association for Computational Linguistics, Nov 2019.
[39] Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 32552, 2022.
12

---

[40] Fabian Isensee, Paul F Jaeger, Simon AA Kohl, Jens Petersen, and Klaus H Maier-Hein. nnu-net: self-configuring method for deep learning-based biomedical image segmentation. Nature methods, 18(2): 203-211, 2021.
[41] Bennett Landman, Zhoubing Xu, J Igelsias, Martin Styner, T Langerak, and Amo Klein. Miccai multi-atlas labeling beyond the cranial worksho and challenge. In Proc. MICCAI Multi-Ailas Labeling Beyond Cranial Vault-Workshon Challenge, volume 5. page 12, 2015.
[42] Michela Antonelli, Annika Reinke, Spyridon Bakas, Keyvan Farahami, Armetteaop-Schneeder,,, Bennett A Landman, Geert Litjens, Bjoem Menze, Olaf Ronneberger, Ronald M Summers, Bram van Ginneken, Michel Bilello, Patrick Bilic, Patrick F Christ, Richard K G Do, Mare Gollub, Stephan H Heckers, Henkjan Huisman, William Jarnagin, Maureen McHugo, Sandy Napel, Jennifer S Goli Pernicka, Kawal Rhode, Catalina Tobon-Gomez, Eugene Vorontsov, Henkjan Huisman, James A Meakin, Sebastien Ourselin, Manuel Wiesenfarth, Pablo Arbelaez, Byeonguk Bae, Sihong Chen, Laura Daza, Jianjiang Feng. Baochun He, Fabian Isensce, Yuanfeng Ji. Fucang Jia, Namkug Kim, Ildoo Kim, Dorit Merhof, Akshay Pai, Beomhee Park, Mathias Perslev, Ramin Rezaifar, Oliver Rippel, Ignacio Sarasua, Wei Shen, Jaemin Son, Christian Wachinger, Liansheng Wang. Yan Wang, Yingda Xia, Daguang Xu, Zhanwei Xu, Yefeng Zheng, Amber I. Simpson, Lena Maier-Hein, and M Jorge Cardoso. The Medical Segmentation Decathlon. Nature Communications, 13(1 13128, 2022.
[43] Bjoem H. Menze, Andras Jakab, Stefan Bauer, Jayashree Kalpathy-Crama Keyvan Farahani, Justin Kirby, Yuliya Burren, Nicole Porz, Johannes Slothoom, Roland Wiest, Levente Lanczi, Elizabeth Gerstner, Marc-André Weber, Tal Arbel, Brian B. Avants, Nicholas Ayache, Patricia Buendia, D. Louis Collins, Nicolas Cordier, Jason J. Corso, Antonio Criminisi, Tilak Das, Hervé Delingette, Çağatay Demiralp. Christopher R. Durst, Michel Dojat, Senan Doyle, Joana Festa, Florence Forbes, Ezequiel Geremia, Ben Glocker, Polina Golland, Xiaotao Guo, Andac Hamamci, Khan M. Iftckharuddin, Raj Jena, Nigel M. John, Ender Konukoglu, Danial Lashkari, José António Mariz. Raphael Meier, Sérgio Pereira, Doina Precup. Stephen J. Price. Tammy Riklin Raviv, Syed M. S. Reza, Michael Ryan, Duygu Sarikaya, Lawrence Schwartz, Hoo-Chang Shin, Jamie Shotton, Carlos A. Silva, Nuno Sousa, Nagesh K. Subbanna, Gahor Szekely, Thomas J. Taylor, Owen M. Thomas, Nicholas J. Tustison, Gozde Unal, Flor Vasseur, Max Wintermark, Dong Hye Ye, Liang Zhao, Binsheng Zhao, Darko Zikic, Marcel Prastawa, Mauricio Reyes, and Koen Van Leemput. The multimodal brain tumor image segmentation benchmark (brats). IEEE Transactions on Medical Imaging, 34(10):1993-22) 1993-2024, 2015.
[44] Jülich Supercomputing Centre. JUWELS Cluster and Booster: Exascale Pathfinder with Modular Super- computing Architecture at Juelich Supercomputing Centre. Journal of large-scale research facilities, 7 (A138), 2021.
[45] Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré, FlashAttention: Fast and memory- efficient exact attention with 1O-awareness. In Advances in Neural Information Processing Systems, 2022.
[46] Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding, 2021.
[47]
Mykola Novik. torch-optimizer-calle collection of optimization algorithms for PyTorch., 1 2020.
[48] Léón Bottou and Olivier Bousquet. The tradeoffs of large scale learning. Advances in Neural Information Processing Systems, 20, 2007.
[49] Yuanfeng Ji, Haotian Bai, Jie Yang, Chongjian Ge, Ye Zhu, Ruimao Zhang, Zhen Li, Lingyan Zhang, Wanling Ma. Xiang Wan, et al. Amos: A large-scale abdominal multi-organ benchmark for versatile medical image segmentation. arXiv preprint arXiv:2206. 08023, 2022.
[50] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhiní Agarwal, Ariel Herbert- Voss,, Gretchen Krueger. Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler. Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are sho-shot leamners. 2020.
[51] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2020.
[52] Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical common- sense in natural language. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 7432-7439, 2020.
13

---

Appendix
A Derivation of the Lagrange multiplier update
For simplicity, we consider a single constraint. Note that multiple constraints can be addressed separately the optimization problem would be separable in the respective X. We need solve
1 (A Xx)². 2μ
maximize f((x) + x c(x)) A20
The optimal point of this problem is equivalent to the optimal point of
minimize - ((x) c(xr)= + 1 (A xx)
s.t. - -
2/
To find candidates for optimal points, we need to solve the Karush-Kuhn-Ta Kuhn-Tuckerr (KKT) system with the Lagrange function [(X, vb) and the Lagrange multiplier v
C(A, = --(() c(xx) (A xx)2 4.A
24
Which leads to the KKT system
VAC(A, w) =0 = 0= c(x) - 2e) 10 /
((,,)) <0 02
-
x===0
According the complementary conditions in equation the constraint is either active, hence A and vb 2 0 or inactive, such that À > 0, and consequently. /b = 0.
Case: x 0 and & 0
Here, (by assumption), and & is given by
1
Vx w) 0 = 0= 0= -c(x) + (0 A1) +
/
X
c(x)
Since we require ab 2 0 for a KKT point, (note that > 0)
X
0Σ -c(x)
1
0 cxx) X
2 μ c(x)
Consequently, À = 0 is candidate for the optimal point only when 0 02 A + p. c(x).
Case: À > 0 and &b 0 (inactive constraint)
For this case we get
VxL v) = -0= + - () - -x)) -0
- c(x) + + d
À = X c(x)
Due to the geometry of the problem (quadratic with bound constraint), is the optimal solution if the constraint is active, i.e., if 1/2 2 0, which is the case if 0 2 A c(x).. Consequently, the optimal solution given by x = (he + p c(
(4)
14

---

Plugging this into F(x, )e, pt), we get
[(x) + c(x:(Ax + [c(x)), if
d + p c(ar) 2 0
F(x, 14)
f(z) 1 Ao.
else
2μ
And the gradient with respect to æ
[=f(x) + vac())():
c(x)), if - clx) 20
Val de, (1)
V f=)
else
Or more compactly by using equation
V Fax V=]( + Vac(x) X
B
The CPR Algorithm with Kappa-WS
Algorithm 2 Optimization with constrained parameter regularization (CPR) and Kappa-
Require: Loss Function L(O, X, y) with parameters 0, and data D {(X,,, )n)) }
Require:
Hyperparameters: Learning rate r) E RR+,, Lagrange multiplier update rate μ E R R starting
step for CBR.
Require: Optimizer Opt(-) for minimization, Regularization function R((0) (e.g. L2-norm)
1: # Initialization
2: 4-00 0
3: 0, Initialize(L(-))
4: X 0 for j 1.... J
5: R² +- 00 1, JJ
6: # Training
7: for Xx. Yt ~ D do
8: 0 0, + (p), v1),
Classic parameter update using, e.g., Adam.
9: for each regularized parameter group 0 in 0, do
10:
1 + (N + f (R(0)) x²))
11:
0/ + 01+1 Vo: R(O)) 1441
if then
Kappa-kI, initialization, see Section 4.3
13:
xJ <- R(0))
14:
end if
end for
16:
17: end for
15

---

C Experiments on the Sensitivity of the Update Rate
We analyze the sensitivity of the update rate μ in CPR with experiments on & trained on the CIFARIO 100 and GPT2s trained on Open WebText. For the ResNet18 experiments, we consider update rates from μز 0.01 to μ 10 and apply two kappa initialization methods,Kappa-kI, and Kappa-WS. As shown in Figure C.I we see no significant impact of on the performance. We report mean percentage of correct labels across three random seeds. We also performed short-runtime experiments with GPT2s and update rates of E (0.01, 0.1, 1, 10}. and observe very similar results, see Table C.T To get an impression of how ls impacts À and therefore the squared 1.2 norm the weight matrices with the use of CPR, plotted the squared L2 norm and À for three weight matrices during the training in Figure C.2 We found no impact on the stability of the squared L2 norm despite the difference in the magnitude of the for different μ values.
AdamCPI (Kappa-Ve WS)
AdamCPI (Kappa-kly)
1e-10
#0.10
20:14
20.20 20
$0.51
20.16
20 12
20.14
In-10
$0.34
$0.05
20
20 23
20.01
20.3 34
$0 06
80
1e3.5
$4.54
20.30
20. 1.8
2:0.1 15
220 18
1e-3.
40.02
20.14
20.00
20. 34
2.020
31
210
Learning Rate
75
1e3.0
$3.00
200
20. 110
2:1. 12
220 10
$5.25
1e-300
$10.4
=0
#0 21
$1.10
18
1:0 27
1e-2.5
$0..44
2011
210 31
2010
2:11 21
220 211
$1.21
1e.2.5
1:10 49
20 U8
#0 35
$11.1
20 16
1:0
70
1e-2.0
#0..0
2.1.60
20. 11
$10 33
2:1 336
220 08
40.00 50
1e.2.0
$10.53
20
2:0 20
$01.9
13
210
65
1e-1.5
#3..9
20.15
88
$3.00
20.40
20 41
1m.1.
$0.31
$1..09
20 14
0 43
$0.34
30
$0.33
60
1e:1.0
#1.02
201.05
20.00
14
$1.00
20.00
20 43
1c-100
$9.70
$1.59
10
#1
11.84
20 5b
$1:15
1c-1.6
1c.0.5
1e0.0
1e1.0
1c-2.0
16-10
1e:0.6
1.0.0
1c0.6
1e1.0
Update rate 1
Update rate j
Figure C.I: The Figure shows the percentage of correct labels of the ResNet18 trained on the CIFAR 100 with the use of IIp (left), AdamCPR (Kappa-WS) (right) with different update rates μ. The elements in the heat map are experiments with different learning rates and each element is colored according to the mean accuracy of three random seeds and the numbers are the mean accuracy and standard deviation of the experiments. The experiment shows that the AdamCPR regularization is not sensitive to the choice of the parameter.
Table C.I: Comparison of different values for the update rate of AdamCPR. We run experiments with GPT2s with 50k total steps, learning rate warmup of 2.5k steps, and kappa warm start of 5k steps.
Method (Ir value)
Accuracy t
PPL +
GPT2s
AdamCPR μ 10
0.422
20.91
AdamCPR
0.423
20.90
AdamCPR μ 0.1
0.423
20.90
AdamCPR 0.01
0.423
20.90
16

---

GPT2s with AdamCPR with different
Layer 1 Attn Weight
0.0075
1100
0.0025
==0
0.0004
Layer Arta Weight
0.0002
0.0000
Layer 5 Fel Weight
0.010
10
0.005
0.000
Layer 5 Fel Weight
0.0002
0.0000
Fe22 Weight
Layer 10
0.010
[|00 11
0.000
Layer Fe2 Weightt >
0.0002
0.0000
Trahinng Loss
Validation
3.4
10000
20000
30000
40000
50000
Optimization Steps
Figure C.2: A comparison of different update rates μ in the training GPT2s model. We see three weight matrices during the training with AdamCPR. We also how À regulates the constraint of the bound on the squared L2 norm. The bottom two plots show the training and validation loss.
17

---

D Adaptive Bounds
With fixed bounds k², some parameter matrices 03, for which X 0 will not be regularized. While this can be beneficial, CPR can also be used to apply continuous pressure similar to weight decay. For this, the bounds x² of parameter matrices O with X can be set to the current value of the constraint function K c(O(). ). Such an adaption guarantees that each parameter matrix is always exposed to some regularization. This should result in gradual reduction of the bounds k throughout training without exerting excessive pressure on the optimization process. In our experiments, we refer the usage of adaptive bounds as AdaCPR.
This contrasts with weight decay, where continuous pressure is applied to enhance generalization throughout the training. To emulate the continuous pressure of weight decay, propose adaptive mechanism to adjust the upper regularization bound during training. This can be achieved by leveraging existing states. Specifically, the value of X offers insights into constraint violations. When X = 0, the constraint (0) can be regarded as inactive. In this case, we may consider adjusting its bound x² to align with the current constraint value of c(O,). To implement these adaptive bounds, we add conditional update rule for w² after our CPR update. It updates the upper bound for each parameter matrix 0g individually by
R(O))
if and X otherwise,
where X/-1 O indicates that the upper bound previously violated and c0 was active. Consequently, this enables gradual reduction of the bounds s² throughout training without exerting excessive pressure on the optimization process. Please find AdaCPR in Algorithm 3 below.
Algorithm 3 Optimization with adaptive bound constrained parameter regularization
Ada CPR
Require: Loss Function L(O, X, y) with parameters 0, and data D = {(X,,, fn) )n))
Require: Hyperparameters: Learning rate 7) E R+ Lagrange multiplier update rate jr R+
Require: Optimizer Opt(-) for minimization, Regularization function R(0) (e.g. L2-norm) Initialization
2: t ---0 0
0, +- (ittializeeL(-))
4: X + 0 for j = 1, 1,... JJ
5: w + 0: Initializo 0%) for 1,...,
6: # Training
7: for X,, V: D do
8: 0c+1 0, + Opt(L(O,, X,, vt),
Classic parameter update using, e.g., Adam.
9: for each regularized parameter group 0 in 0 do
10:
X + (> (R(0?) - kU))
11:
0:41 < 0/+1 V R(O() N441
if X and X then
Update xe if the constraints are not active.
13:
x³ < R((:)
14:
end if
15:
end for
16:
++1
17:
end for
The experimental results in Figure E. training is not beneficial. While it does not harm the performance, it also does not lead substantial improvement. We therefore do not use it to keep our method as simple as possible.
also show that the adaptation of the upper bound during the
18

---

E Experiments on Image Classification (CIFARI00)
For the initialization Kappa-K, we use a range of |0.005, 0.16]. for Kappa-kIg range of l: = [4..... .... .266), and for Kappa-W5 a range of [250.-. 4000] steps. We use learning rate warmup of 500 steps followed by a closing annealing. This is 2.5% of the total training steps (20k).
For detailed list of training hyperparameters, we refer the reader to Table E.
We found that initializing with Kappa-kIn performs better than selecting uniform in Kappa-K. K. This may be explained by the value of the regularization function depending on the size the jointly regularized parameter matrix and initialization method. The warm start initialization method, Kappa-Wa WS. performed the best. The best configuration with CPR outperforms weight decay and the choice of hyperparameters seems be more robust.
Table E.1: Hyperparameters of the ResNet18 on CIFAR100 experiment.
Parameter
Value
Seed
1,2,3
Dataset
CIFARI00
Batch size
128
Training Steps
20000
Model
ResNet18
Optimizer
AdamW / Adam+Rescaling / AdamCPR
Learning Rate
0.001
Betal
0.9
Beta2
0.98
Weight Decay
0.1
Lr Schedule
Cosine with warmup
Lr Warmup Steps
500
Lr Decay Factor
0.1
Rescale Alpha
0.8. ......15 16
CPR-#
1.0
CPR-K
0.8 16
CPR-k
256
CPR-K warm-start steps
250 16000
Adaptive Bounds
False / True
19

---

AdamCPR (Kappa- K)
Adam Adam+AdaCPPH (Kappa K)
1c.4.0
20. 16
29.23
20:0
70
70.4 20 12
20010
70.4
20
1o-4.0
20
20
17
2:0
70.5
20
27
10 15
2.023
16:3.5
20 12
10.12 12
20 60
100 23
2- 60
20.35
16:3.5
96
10 15
10011 1.
20.55
200 04
10.31
Learning Rate
4
75.3
75
70
76.3
1.0
1.0
10.30 36
32
1e.2.5
28.3
75.8
75.0 8
74.2
74.8
24.0
1e-2.5
75.0
70-7
76-2
74 74-7
74.6
74.0
1e-2.0
F2.1
75.4
+0000
75.8
24.9
1e-2.0
71.7
74.
73
Yb.1
16.1.
#0.22
$9.23
+00 4.
+10 1.5
+-3.10
#0.50
1o-1.5
+1.07
+0.61
+-0.24
#0. 46
+1 05
+0.08
1c-1.0
+1.62
#1.00
52.5
#33.0
20.60
1c-1.0
11.16
+2 15
4-2.02
+2 74
+:0.1
0.005
0.01
0.02
0.04
0.08
0.16
0.003
0.01
0.02
0.04
0.08
0.16
Kapps
Kappu
AdamCPR (Kappa klo)
Adamm AdaCPR (Kappa-kla kly)
1e-4.0
+1.022
25
+0.24
10
+0.20
76.2
1e-4.0
70.2
+0.05
70.5
73.2
+1..0
-0.37
+0.23
$0.0
1e.3.5
74.8
74.5
74-7
74.4
74.2
74.6
74-7
1e-3.5
75.0
T4-4
24.6
74-7
74-4
74-3
74.4
Leaning [tate
1e-3.0
76.9
76.4
+0.23
79.1
75.2
74.9
75.
1e-3.0
75-7
16.3
+0.05
9
75.8
+1.00
70
T6.0
75.0
72.5
16.2.5
#1..37
-0.50
+1.22
+-0 4.5
+3.10
+4.24
-0.02
1c-2.5
+0.14
+00008
+0.42
+-220
+10 14
-0.11
+-0.2
70.0
1c-2.0
+0.39
57
+10.27
07
+0
=0.37
1c-2.0
#0.60
+:0.22
+-0
+-0.0 0/
06
=0
+90.1
55.5
20
633-
50
76.6
T1 66
1o-1.5
52.3
11:22
6R
255
70-1
65.0
62.6
28.5
33.08
38:-1
44
50.7
56.0
50-11
1e-1.0
20:5
25.5
28 2
37.
411
40.7
56.4
60.0
16
82
64
128
256
16
32
64
128
256
Factor k
Factoo
AdamCPR (Kappe-WS)
Adam + AdaCPR (Kappa-
1s-4.0
71-2
-0.26
2
+0.17
21.a
70
+0.13
70.9
20-3
-0.07
70.1
1e-4.0
21.0
+0.06
71.1
70.8
+3.39
70.7
+0.
20.5
-0.00
70-a
+0.25
70.3
16-3.5
10.19
10.08
75.7
10.31
76.1
10
19.21
10
74.4
20
1e-3.5
25.7
10:39
10
10.35
75.6
0.22
74.3
10.05
70.0
Learting Itate
1e-3.0
10:36
10.05
10.39
10. 18
10.14
11.01
0..67
1e-3.0
10.52
10.31
10.21
10.26
10. 40
0..28
10.17
73.8
75.6
7-4-
74. 1
70.5
74.6
10.34
27
19.00
1..66
1.0.
10:21
10.47
1e-2.0
64.4
73.9
75.7
75.0
74.1
73.-4
73.
1e-2.0
6:3-7
72.11
75 4
75.3
74.3
73.88
73..
1e11.
11.08
97.7
69.1
1:0.44
72.7
10.05
70.3
11..3
00.0
20.18
1e-1.5
1.0.01
53.7
10.44
88.8
1.0
23.41
72.1
110.00
78.6
-0.16
70.3
110.00
89.4
1w.100
+:3.33
-2.76
1.0.09
4-2 05
+0.81
+11.22
=1 -..07
1e-1.0
+:2.01
4-3.32
04.54
19.57
410.05
-1.12
111.0
4000
16000
1000
4000
5000
start steps
steps
Figure E.1: Percentage of correct labels of the ResNet18 trained on the CIFAR100 with use of Adam with CPR (left) and AdaCPR (right) with use of the three different initialization techniques from Section 4.3 from top to bottom: Kappa-K, Kappa-kIo. and Kappa-ws. The elements in the heat map are experiments with different learning rates and regularization hyperparameters. Each element is colored according to the mean accuracy of three random seeds and the numbers are the mean accuracy and standard deviation of the experiments.
20

---

AdamCPR L2 regularization
AdamCPR std regularization
1e-10
10.25
71.2
10:17
13.30
70.5
1.0.
10.00
1e-1.0
10:00
70.6
10.31
20.00
10.17
70.6
1.
00
1.0
80.0
1m3.5
76.0
76.7
76.1
40.33
75.5
74.6
1e-.5
75.0
76.7
1m3.0
76.7
76.6
400.6
76.2
1e-3.0 0.
76.5
26.5
75
Learatag
1e-2.5
73.8
23
74.6
1e-9.5
21.83
74.6
-0.11
4:0.2:
70.0
1e-2.0
70.0
75.0
73.3
1e-2.0
64.7
01.11
-0.
23.3
72.
05.0
1e.1.5
4:141
+1..5
19.15
68.6
1m:1.5
22.42
2man
73.3
Enan
20.1
2ma0
62.5
1e-1.0
72.76
60.2
+2.00
65.7
+1.83
=0.21
63.00
$1.07
63.0
1e-1.0
=2
2ann
26.5
Sman
$10.77
64..4
63.3
250
500
1000
2000
4000
8000
16000
250
600
1000
2000
4000
8000
16000
stape
Figure E.2: Percentage of correct labels of the ResNet18 trained the CIFAR100 with the of AdamCPR using 1.2 regularization measure (left) and standard deviation as regularization measure (right). The elements in the heat map experiments with different learning rates and warm start steps (s of KS)pa-WS). Each element colored according to the mean accuracy of three random seeds and the numbers the mean accuracy and standard deviation of the experiments.
21

---

Adamm W wr-warmup 250 steps
AdamCPR
AdamCPR wa-warmup 250 steps
1c.4.0
20.36
16
27
20 24
2:0 20
20.34
20.00
20.31
20 16
06.06
20.05
74.5
76.0
74-3
74.5
Learaing Rate
1e:3.0
24.7
76.0
74-4
74-T
78.
78
74.5
76.0
76.4
75.2
70.0
7440
75.0
1e-2.5
78.3
+1..8
74.0
-0.02
T5.8
-0.35
78.0
11.44
72.00
+-0.33
75.0
76.0
744.5
T-42
+00400
75.9
1e.2.0
+0.40
+4
+-022
32
01 55
+-0 63
+10.04
00021
+0.20
+14.45
+-0.443
+0.011
16-1.5
+0.45
08.
+-0 66
#1124
00.70
=0.15
+:0
+1.00
+0.40
01:2
02.4
62.0
044
38.
44.1
500
96.8
02.0
01.0
01.7
0.0
0.0001
0.001
0.01
0.1
250
300
1000
2000
4000
8000
16000
Weight Decay
(Kappa-1P)
stops (Kappa-WS: WS)
Adam warmuu 500 sleps
AdamCPR
AdamCPR lewarrup 500 sleps
70.2
70-4
70.5
70.3
70.5
71-0
71-2
71-2
71.3
70.8
70.3
70.3
70.1
$0.0
1e-3.5
+0.22
-0.13
74.8
-0.35
74.
76.0
+1.00
75.7
76.1
+1..22
74.4
+0.20
74.0
77.6
Learning (tate
1e-3.0
+0.20
+10.23
-0.21
-0.15
-0
+1.0
15
011 36
+1.00
0.222
+0.15
+1.14
78.6
+1.00
+0.07
1e-2.5
+0.44
+-0.43
+00.40
035
4.7
25
34
+-0.2
-0.20
+0.00
+-0.4
+0.13
70.0
3-
73.33
1..11
74
5.
21
30
7:4-3
73.4
10.58
07.5
68.2
68.1
19.-8
71. 14
24.5
72
57.
65A-7
72.7
71-7
70.3
60.00
66.6
62.5
1e-1.0
11:00
63
1.0.7
2.0
.05
16.05
1:0
1:3.33
1.2.76
83.
97.
2.00
1.0
64.4
1.0.23
63.9
11.00
63.9
60.0
16000
Weight Decay
(Kappa-IP)
wteps (Kappa-WWS
Adem W (r-warmup Ik steps
AdamCPR
AdamCPR lr-warmup 1k steps
1c.4.0
10.30
70.1 10:13
10.14
70.3
20:47
70
20
1:0 20
2 1:0. 10
20.35
20034
40 10
70.5
40
70.5
1e:3.5
10.00 00
10.30
10.35
-0.00 22
2n
10 27
10. 1A
10.11
0.11
10.24
10 16
10.30
10.00
Learting Rate
75-5
75-4
75.4
75.5
76-0
75.4
3.
0.23
10.31
1e-2.5
74.9
75.1
74..8
70.
75.5
75.
60.
72.2
75.4
75.05
25.2
74.99
74.8
1e-2.0
1.0.00
74.3
1...0
74.-1
7:4
2m
1.0.38
75.7
2.0.35
1.11
7.4.5
1.0.
T-8.0
10.66
+4.4
1e.1.
14..2
1.0.2
2.0.2
-0.00 m1
21.0
1:0
34
1.135
2:0.11
0..17
2.0.43
1.0.7
1.0..00
1.0.20
1e-1.0
14.54
1:1.000
1.0.44
42
36
50.8
95.1
2.0.38
06.0
1:0.23
1.0.4
0.0
0.0001
0.000
0.01
0.1
250
500
1000
2000
4000
8000
16000
Weight Decay
(Kappe-IP)
Wirth start steps (Kaopp-WW5)
Figure E.3: Comparison of AdamW and AdamCPR with different learning rate warm-up steps. The Figure shows the percentage of correct labels of the ResNet18 trained on the CIFAR 100 with the use of AdamW (left side), AdamCPR (Kappa-II) IP) (middle), and AdamCPR (Kappa-WS) wW)) (right side) with learning rate warm-up steps between 250 and 1000 steps. The elements in the heat map are experiments with different learning rates and regularization hyperparameters. Each element is colored according to the mean accuracy of three random seeds and the numbers are the mean accuracy and standard deviation of the experiments.
22

---

Adam W
AdamCPR (Kappa- WS)
1e-1.0
10.35
19:10
10.32
10:20
10.20
1e-40
10.57
10:20
10:17
19:30
18:33
0..10
10.07
1e-3.5
10.39
74-7
39
76.7
73.3
1e-355
20:10
10008
100
1
19.33
70
24
74.4
3:0.20
Learaing Itate
1e-3.0
70.2
75.0
74.0
7e.d
75.4
1e-300
74.8
T0.7
76.6
76.2
78.6
75.8
76.2
1e-2.5
+10.3
74.2
+3.43
74.3
75.1
+0.47
74.8
1e-2.5
70.9
+0.20
T5.8
70. 5
+.3.2
74.6
+:0.33
T4.6
1e-2.0
+1..3
73.2
+9.63
73.3
+-0.3 15
+0.19
+0.53
1e-2.0
+10.1
64.4
+0.27
72.0
+10.
75
+0.17
75.0
+-0.30
74
-0.31
7A-4
+10.9
73.1
16.1.
10.64
19.01
10. 17
10.17
22.44
1o-1.5
11.00
10.64
1.0
29.41
10. 25
00-33
10.14
63.3
62.4
16.8
41:4
63.33
67.0
65.7
63.0
63..9
10:11
79
K3
0.0
0.0001
0.001 Welght Decay
0.01
0-1
250
500
1000
2000
4000
8000
18000
Warm stars steps
AdamW WD schedule (decreasing x0.1)
Adam WD schedule (decreasing x0.01)
20 8
70-2
a
20.5
1e4.0
70-3
0
20.2
80.0
1c:3.5
24.3
74.5
20.0
74.5
74.6
1o-3.5
76.5
20.1
74.00
76-3
74 4
77.5
Learung [tate
1c:3,0
20.0 05
20.30
76.
20.26
75.0
20.07
1o:3.0
40.00
76.3
20 89
75
20.29
75-3
20 20
72.5
1e.2.5
20.00 09
10.30
20.00
1.002
1e/2.5
10:47
20.61
10.34
51
70.0
16
73.3
74.0
1.0.0
1e-2.0
10.00
73.3
76-a
87.5
1e-1.5
09.5
68.8
73.4
70.-1
Ie-1.5
68.6
69.7
71.5
70.2
62.5
1e-1.0
21.10
10:de
64.7
06.4
...44
11.57
1e-1.0
11:30
64.1
10:47
64.3
10.75
67.3
12.20
46.0
60.0
0.0001
lattial Weight Decay
Initial Weight Decay
Adam WD schedule (increasing x10)
Adam WD schedule (increasing x100)
1c.4.0
20.00
20.67
20.00
20. 26
1o-4.0
70.9 10:18
70 20:30 04
10.18
20:
16:3.5
04
10.35
-0.24
10
16:3.5
10.93
04
20.17
20. A3
Learaing Rate
75 1
75-1
73
+0.31
+0.00
+1..33
+0.30
+0.92
24.4
74.4
74.0
09.8
1e-2.5
74.3
74.0
00.3
38
1e-2.0
23.1 1
+0.31
19.3
-0.40
73.4
+0.13
01:7
1e-2.0
75.5
72.5 8
+1.33
01.00
13:
16.1.5
=0.25
#0.12
70.0
=0.96
+1.0
1o-1.5
#00.00
=0.71
#1.86
4141
1c-1.0
20.21
#1.04
=0.84
+2 14
1c-1.0
#0.05
#0.00
0.0001
0.001
0.01
0.1
0.0001
0.001
0.00
0.1
Initial Weight Decay
Initial Weight Decay
Figure E.4: Comparison of AdamW, AdamCPR, and weight decay scheduling similar to 10 I1. Figure shows the percentage of correct labels of the ResNet1 8 trained on the CIFAR 100 with the use of AdamW (top left), AdamCPR (Kappa-WS) (top right), and Adam with weight decay scheduling. We evaluated the task with cosine decreasing weight decay to 0.1 and 0.01 times the initial weight decay value and with cosine increasing weight decay to 10 and 100 times the initial weight decay value. The elements in the heat map are experiments with different learning rates and regularization hyperparameters. Each element is colored according to the mean accuracy of three random seeds and the numbers are the mean accuracy and standard deviation of the experiments. It should be mentioned that Yun et [9] also performed weight decay scheduling on CIFAR100 with the use of ResNet18. 8. Since their code was not published, we point to Figure 3 of their experimental results, where an accuracy of around 60% was reported, which is below our Adam W baseline.
The
23

---

Adam W
AdamCPR
AdamCPR
0.000316
62.0
611.4
62.3
62
62.5
62.3
63.0
63.
62.7
08.5
G2
80
Conrecl Labe
0.001
62:
6339
10.54
1.042
10.34
1.0.0
1-3.0
16.37
10.25
10.74
+3.00
70
0.00346
10.83
11.00
1.:47
1..11
11:00
13.000
23.75
1:1.43
10.96
1:1.22
1:033 30
65
0.01
09.55
50
D7
94.0
20.0
50 35
60
0.0
0.0001
0.001
0.01
0.1
500
1k
Weight Decay
(Kappa-IP)
Warm start steps (Kappa- (hppaa-WW)
Figure E.5: Percentage of correct labels of the ResNet18 trained on the CIFAR 100-C with use of AdamW (left), AdamCPR with Kappa-IP (middle) and AdamCPR with Kappa-WS (right). The elements in the heat map are experiments with different learning rates and regularization hyperpa- rameters. Each element colored according the mean accuracy of three random seeds and the numbers are the mean accuracy and standard deviation of the experiments. We see that AdamCPR outperforms AdamW which could indicate that CPR leads that AdamCPR performs better than AdamW with Kappa-Wa WS but not with Kappa-II. Kappa-IP does fail and performs better than the average weight decay performance. None of the optimizer and hyperparameter configurations lead to an outstanding performance on this task, we wouldn's 't claim that CPR is particularly good for noisy data.
more robust optimization. We see
SGD
SGDCPR
SGDCPR
Learnnng Rate
16.2.5
10.19
10:34
20.24
25.6
1..26
10.44
-0.00
70.
80
Corredt Labbe
75
1o-2.0
76-2
T5.7
53.7
71.
74.5
74
74
73.88
75.
70
1o-1.5
10:43
19.00
10.10
10.07
15.00
1.320
10.30
1.
16
-0 30
20.04
66
16-1.0
10.11
1.0
11.19
11.1
10.51
77.5
30
110
47
1.4
00
20.00
1e-1
1e-3
Weight Decay
(Kappa-IP)
Warm
steps (Kappa WS)
Figure E.6: Percentage of correct labels the ResNet18 trained on the CIFAR 100 with use of SGD with weight decay (left), SGD with CPR and Kappa-IP (middle) and SGD with CPR and Kappa-WS (right). The elements in the heat map are experiments with different learning rates and regularization hyperparameters. Each element is colored according to the mean accuracy of three random seeds and the numbers are the mean accuracy and standard deviation of the experiments.
24

---

Adam W
Adam CPR (Kappa WS)
1c.4.0
10.35
20.10
20 52
20.20
20:20
1c:4.0
20:87
20.26
20:30
=0
10
20.07
74-3
76-7
23-3
75.6
70
74.6
Learaing Rate
1e:3.0
7b.4
75-0
74.3
74.0
75.4
10:3.0
74.8
T0.7
76.6
75.2
70.4
75.8
76.2
1e-2.5
74.3
74.8
76.1
74.5
1e-2.5
70.9
78
74-0
74.4
T4.6
1e-2.0
+1.23
+9.63
+10 15
+10.19
+0.83
1e-2.0
+0.10
+-0.27
+10
+0.11
+-0 30
-0.81
+10.00
16-1.5
+0.64
#3.91
17
71.8
#2.44
1o-1.5
+1.00
+0000
+0441
+10.80 8-
-0.95
#10.1
08.2
03.0
02.
16.0
41.4
08-2
67
65.1
0.4
65
08.0
0.0
0.0001
0.001
0.01
0.1
250
500
1000
2000
4000
8000
16000
Welght Decay
Warm wcars stagp
Adam + Deray (alpha 1.0)
Adam Addam-AAdan Decay (alpha 2.0)
70.4
70..
70.5
70.6
70.3
70.3
70-4
70.0
$0.0
1e-3.5
74.8
+0.15
74.9
-0.49
74.5
76.1
1e-5.5
+1.20
-0.00
74.88
+0.32
74.3
74.9
77.6
Learning (tate
1e-3.0
+0.35
+1.45
-0.13
+0.26
1e-3.0
+0.11
-0.27
+0.18
+0.06
1e-2.5
74
39
+0.23
-0.14
+0.45
1e-2.5
+1..88
74
50.50
+0.40
-0.17
70.0
73
2.
73.6
74.1
73.8
10:39
73-3
10:51
10.30
07.5
09.4
60.99
72.3
65.9
1e-1.5
60.0
69.2
73.2
66.7
62.5
1e-1.0
21:23
04.4
1:36.82
63.5
44.3
1:21
13.5
1e-1.0
135.90
62.5
147
1.00.7
65.4
2.4.0
60.0
0.0001
0.01
Inttial Weight Decay
Initial Weight Decay
Adam + Aecay (alpha 4.0)
Adam + AdaDecay (alpha 8.0)
1c-4.0
70.4 20.10
10:17
70.3
70.2 -0.06
100
70.3
1o-4.0
40.10
70.9
70-3 20-36
70-4 10.
20
Te:3.5
20.12
10:33
-0.24
10. 10
1e:3.5
10.31
20.00
10.10
10:37
Learting Rate
74 8
75.3
75-1
75-3
75-7
20.27
10:15
1e-2.5
74-3
74.6
74.4
74.4
1e-2.5
76.3
74.4
74.9
75.2
1e-2.0
10.40
10:47
T3.4
-0.3
73.8
14
74.1
1e=2.0
10.73
73.1
10.67
73.5
10:21
T3.9
100 20.22
74.1
1e.1.5
20:40
14:40
20.06
110.30
1e-1.5
10.31
20:16
10.47
20:15
65.4
1356.7
11.00
+221.47
+1..80
20.00
110.1
13.01
0.0001
0.001
0.01
0.1
0.8001
0.001
0.01
0.1
Inttial Weight
Initial Weight
Figure E.7: Comparison of AdamW, AdamCPR, and Adam with AdaDecay [3] The Figure shows the percentage correct labels of the ResNet18 trained on the CIFAR 100 with the use Adam (top left), AdamCPR (Kappa-WS) (top right), and Adam with AdaDecay with different (1.0, 2.0, 4.0, 8.0) values for the alpha hyperparameter in AdaDecay. The elements in the heat map are experiments with different learning rates and regularization hyperparameters. Each element is colored according to the mean accuracy of three random seeds and the numbers are the mean accuracy and standard deviation of the experiments.
25

---

Adam W
Adam CPR (Kappa WS)
WSP
1c.4.
20.10
20.42
20:30
20.20
1c:4.0
20:87
20.26
20:30
=0 10
20.07
74-7
74.4
74-7
76-3
75.6
70
74.6
Learaing Rate
1e:3.0
75 1
74.9
74.0
75.4
10:3.0
74.8
T6.7
76.6
75.2
70.4
75.8
76.2
1e-2.5
+0.45
14.4
75.1
+1047
74.8
1e-2.5
70.9
78 0
74-0
74.4
T4.6
1e-2.0
=0.03
+0.13
-0.19
+1..3
1e-2.0
+0.10
+-0.27
+10
+0.11
+-0 30
-0.81
+10.00
16-1.5
=0.91
+0.17
=0.17
+2.44
1o-1.5
+1.00
+0000
+0441
+10.8 8.
-0.95
#10.1
63.0
06.1
62.4
16.8
41.4
08.2
67
65.1
0.4
65 0
08.0
0.0001
0.001
0.01
0.1
250
500
1000
2000
4003
8000
16000
Welght Decay
Warm wtars stagp
Adam + Decay (alpha 1.0)
+alam--WWW
70.4
70..
70.5
70.6
70.3
70.1
70.2
70-4
$0.0
1e-3.5
74.8
+0.15
74.9
-0.49
74.5
+0.00
76.1
1e-5.5
+1.30
74.3
-0.40
74.8
+0.27
74.4
74.3
77.6
Learning (tate
1e-3.0
+0.35
+1.45
-0.13
+0.26
1e-3.0
+0.21
-0.30
+0.10
+0.22
1e-2.5
74
39
+0.23
-0.14
+0.45
1e-2.5
+1..2
=0.41
+0.20
+0.57
70.0
73
2.
73.6
74.1
73.8
23-4
0
10.63
73.1
07.5
09
60.99
72.3
65.9
1e-1.5
60.6
69.3
69.1
69.1
62.5
1e-1.0
21:21
64.4
1:36.82
63.5
41.3
1:21
13.5
1e-1.0
11:44
63.7
20:32
63
1:0.3
63.3
2.0
63.3
60.0
0.0001
0.001
0.01
Inttial Weight Decay
Weight Decay
AMOS
Adam Rescaling
1c-4.0
10.33
47
47.3 10.34
10 95
10.96
20 00
1o-4.0
20
10
10.31
20.85
10 86
40-47
1e:3.5
18 2.
19.16
10. 46
11.30
10.00
1e:3.5
10.12
10. 17
19.34
20.61
10.63
11:97
Learning Rate
68.
70.0
21.4
75.3
73.4
22
11.00
1e-2.5
70.5
71.2
73.3 9
16.0
3.0
1e-2.5
65.01
68.
76.1
75.0
71.2
62.5
1e-2.0
75.
75.1
10.00
10.00
1e.2.0
10.74
45.
1.1.15
11:03
65.6
10.14
70.88
73.0
10.51
68.0
1e-1.5
10.32
13.14
1.0 20
10.50
20.00
1e-1.5
23.54
1.2.07
1.9.20
1:3.16
1.0.4
10.32
11.22
4:0.03
50
71.4
25.00
1e-1.0
24.20
1.1
22:
2.333
11..00
11.00
1w-06
1u-05
0.0001
0.001
0.01
0.8
1.0
2.0
4.0
8.0
16.0
Weight Decay
Factor of Initial Total Wesglat Norm
Figure E.8: Percentage of correct labels of the ResNet18 trained the CIFAR100 with AdamW AdamCPR, AdaDecay 133, AWD 14], AMOS [15), and Rescaling. We use different values of weight decay for AdamW, AdaDecay. AWD, and AMOS. For Adam with Rescaling, we use different factors of the initial total weight norm. AdamCPR uses Kappa-WS. ws. We use a learning rate warm-up of 500 steps and the best Kappa-W5 value is 2x the warm-up steps. Each element colored according to the mean accuracy of three random seeds and the numbers are the mean accuracy and standard deviation of the experiments.
26

---

Experiments on Image Classification (ImageNet)
Table F.1: Hyperparameters for the DeiT small experiments on ImageNet.
ImageNet Pretraining
AdamW
AdamCPR
weight decay
Kappa WS
Kappa IP
(x Ir-warmup)
0.005
0.05
0.5
1x
2x
4x
Model Architecture
DeiT-Small (patch size 16, image size 224)
Learning Rate
le-3
Warmup Epochs
5
Epochs
300
Batch Size
256
Optimizer
AdamW
AdamCPR
Weight Decay
0.005
0.05
0.5
Init Param
6280
12560
25120
K Init Method
warm_start
Scheduler
cosine
Auto-augment
mund-m9-mstdh
Mixup Alpha
0.8
CutMix Alpha
1.0
Random Erase Prob
0.25
AMP
Yes
TorchScript
Yes
Pin Memory
Yes
Data Parallel Jobs
Table F2: Hyperparameters for the DeiT base experiments ImageNet.
ImageNet
AdamW
AdamCPR
Pretraining
weight decay
Kappa WS
Kappa IP
(x Ir- (1-warmp))
0.005
0.05
0.5
1x
2x
4x
Model Architecture
DeiT-Base (patch size 16, image size 224)
Learning Rate
1e-3
Warmup LR
1e-6
Min LR
le-5
Warmup Epochs
5
Epochs
300
Batch Size
256
Optimizer
AdamW
AdamCPR
Weight Decay
0.005
0.05
0.5
K Init Param
6280
12560
25120
Drop Path Rate
0.1
Mixup Alpha
0.8
CutMix Alpha
Color Jitter Factor
0.3
Random Erase Prob
0.25
Train Interpolation
Bicubic
27

---

G Training Dynamics of GPT2
Training Tynamicx of Arbon (blun) AdamCPPR (gresn] Koppo-
11P13
Lawer Atte Weightt
0.0
0.0001
0.0000
1013
Layer 0 Weggh 42/WP
0.0
0.0001
16 18
0.1
Loyor 10
0.0001
0.0000
Validaioon
L,06
25000
50000
75000
100000
125000
150000
125000
200000
Optimicntion
Figure G.1: The training dynamics of AdamW and AdamCPR with Kappa- of one layer GPT2s training run. The upper plot shows the squared L2 norm of the attention weight in the first layer. Below we see gradient of the squared 1.2 norm regarding the training steps, after the first inflection point Kappa-IP initializes kappa and starts the regularization. The third plot shows CPR's lambda enforcing the constraint on kappa. The six plots below show the dynamics for the first weight matrix of the feed-forward block in the 5th layer and the second weight matrix of the feed-forward block the 10th layer. At the bottom, we see the validation loss. We see that Kappa- IP initializes different layers at different time steps, e.g. layer 5 FCI before layer attention weights. While weight decay leads to a steady increase of the squared L2 norm for the first quarter of the training, CPR regularizes much earlier and avoids over-adaption. Adam W converges faster in the beginning of the training but CPR leads to a more linear improvement and a better final performance.
28

---

H Experiments on Language Modelling
For an efficient implementation. we use flash attention 45 and rotary position embedding 446] The complete hyperparameters can be found in Appendix H The GPT2s and GPT2m models are trained on 8 A100 GPUs up 28h. detailed runtime analysis can be found in Appendix !
Table H.1: Comparison of Adam AdamCPR, AdaDecay, AWD, and AMOS on GPT2s trained on Open WebText. For A and AdamCPR report the mean across three random seeds. For the other methods, only a single seed reported. The number next to the optimizer name is the weight decay coefficient y except for AdamCPR, here is the number of warm start steps for Kappa-WS.
Method
Perplexity 4
1e-3
18.45 + 0.0039
A
le-2
18.23 + 0.0113
le-1
18.86 + 0.0169
5k (Ix)
18.02 0.0258
AdamCPR (Kappa-WS)
10k (2x)
18.03 + 0.0178
20k (4x)
18.24 + 0.0320
AdamCPR (Kappa-IP)
17.94
le-3
18.42
Adam Adadecay
le-2
18.24
le-1
18.87
le-3
18.42
Adam AWD
le-2
18.47
le-1
18.49
le-3
NaN
AMOS
le-2
NaN
le-1
NaN
29

---

Table H.2: Hyperparameters of the language modeling task (GPT2 and Openwebtext).
Parameter
GPT2s
GPT2m
GPUs
8x A100 40GB
Gradient Clip Val
1.0
Max Steps
200k
Precision
bf16-mixee
Seed
1234
Betal
0.9
Beta2
0.99
Eps
1.0 x 10-9
Bias Weight Decay
False
Normalization Weight Decay
False
Lr Num Warmup Steps
5000
Lr Decay Factor
0.1
Lr Schedule
Cosine
Model Dimension
768
1024
Number of Layers
12
24
Number of Heads
12
Fed Forward Dim
3072
4048
Attn Dropout
0.1
Resi Dropout
0.1
Embed Dropout
0.1
Rotary Pos Embed
True
Rotary Emb Fraction
0.5
Softmax Scale
True True True 0.02 Uniform Openwebtext
Use Bias
Flash Attn
Initializer
Dataset Name
Max Sample Len
1024
Batch Size
32
24
Val Ratio
0.0005
30

---

I Runtime Analysis on LLM training
To analyze the runtime in more detail, measured the runtime per step of different regularization techniques on different GPT2 model sizes (see Table [.Tp. For Adam we use the PyTorch 2.1 default implementation, for AdamCPR we adapt the AdmW implementation of PyTorch with the imple- mentation described in Algorithm ] for AWD and AdaDecay exists no open source implementation and we implemented it based on the PyTorch Adam class but without "for_each" optimization, and for AMOS we used the implementation form the pytract-optimizer package [47. We compare the runtime on a node with 4 A A100 GPUs and report the mean time per training step across two random seeds and 3000 steps per experiment. In Table 1.2 we compare the runtime with a batch size of and in Table [.3] we repost the runtime with the maximal possible batch size on a 40GB A100 (in samples steps of 4).
Table L1:: GPT-2 Model Sizes and Parameter Counts
Model
Parameters
Model Dimension
Layers
Heads
GPT2s
124M
768
12
12
GPT2m
354M
1024
24
16
GPT21
773M
1280
36
20
GPT2xl
1.19B
1600
36
25
Table I.2: Comparison of optimizer and regularizer runtime per step (batch size=1) across different GPT2 model sizes. Percentages indicate the increase in runtime compared to AdamW. The time is calculated as the mean time per training step across two random seeds and 3000 steps per experiment.
Method
GPT2s
GPT22
GPT21
GPT2xl
AdamW
0.069s
0.152s
0.273s
0.341s
AdamCPR
0.073s (+5.76%)
0.162s (+6.45%)
0.289s (+6.09%)
0.36s (+5.83%)
Adam AdaDecay
0.111s (+60.94%)
0.231s (+51.72%)
0.421s (+54.51%)
0.531s (+55.91%)
Adam AWD
0.089s (+30.04%)
0.18s (+18.55%)
0.318s (+16.64%)
0.385s (+13.05%)
AMOS
0.146s (+113.25%)
0.295s (+93.95%)
0.471s (+72.61%)
0.537s (+57.68%)
Table L.3: Comparison of optimizer runtime per step at maximum batch size across different GPT2 model sizes. Percentages indicate the increase in runtime compared to AdamW. The time is calculated as the mean time per training step across two random seeds and 3000 steps per experiment.
Method
GPT2s
GPT2m
GPT21
GPT2xl
AdamW
0.25s
0.493s
0.473s
0.382s
AdamCPR
0,249s (-0.40%)
0.505s (+2.44%)
0.49s (+3.59%)
0.404s (+5.76%)
Adam AdaDecay
0.309s (+23.60%)
0.577s (+17.05%)
0.617s (+30.44%)
0.573s (+50.00%)
Adam AWD
0.269s (+7.60%)
0.528s (+7. (07)
0.517s (+9.30%)
0.431s (+12.83%)
AMOS
0.302s (+20.80%)
0.614s (+24.54%)
0.703s (+48.62%)
0.581s (+52.09%)
The runtime comparison across various GPT2 models shows that AdamCPR closely matches AdamW's 's efficiency, particularly at larger batch sizes where its runtime increase becomes min- imal even slightly better. In contrast, Adam AdaDecay, AWD, and AMOS significantly increase runtime, particularly in larger models and batch sizes.
However, since not all operations for CPR are implemented in a "for saaheaach optimized manner, CPR's runtime could benefit from an additional CUDA coptiipt implementation.
31

---

Experiments on Medical Image Segmentation
To demonstrate the effectiveness of the proposed CPR approach where using SGD, we also evaluate it in the context of medical image segmentation. We test CPR on four segmentation benchmarks. First, with the Adam optimizer on the Multi-Atlas Labeling Beyond the Cranial Vault (BTCV) 4 task, the Heart Segmentation task of the Medical Segmentation Decathlon [2 and the 2020 version of the Brain Tumor Segmentation challenge (BraTS) task [43].
Here, we make use of the data pipeline and network architectures following the nnU-Net frame- work 44), which is regarded the state-of-the-art framework for medical image segmentation. We implement a training schedule with total of 25k steps (for the Heart and BraTS tasks) and 125k steps for BTCV. We introduce learning rate warmup of 2k steps (8%), followed by a polynomial annealing, see all hyperparameters in Appendix JT] We run each experiment on one consumer GPU for up to 2 days. We present the results in Table J.T. where different weight decay configurations in AdamW W are evaluated to AdamCPR with Kappa-Ws initialization. We report the commonly used Dice scores, averaged across cross- cross-validanee validation folds. These results indicate that CPR surpasses even the best AdamW results. We note that applying Kappa-WS initialization too late can cause instabilities due to weak regularization.
Since nnU-Net by default uses the SGD optimizer 488] we also test CPR to constrain optimization with the SGD optimizer in this context. As more recent benchmark of segmentation performance. we report experiments on the Multi-Modality Abdominal Multi-Organ Segmentation Challenge 2022 49 This benchmark represents a very competitive segmentation challenge environment where differences as small as 0.1 in Dice score can decide on challenge winners. As the experiments in warm start steps, after the learning rate warmup leads the best results, we resort to using 1k warm start steps for CPR since no learning rate warmup is present in the case of SGD in nnU-Net. As the weight decay value, we employ nnU-Net's default value of 3e-5. We show strong performance out of the box in this context, improving baseline (89.45 Dice score) by a margin of 0.13 Dice points to a Dice score of 89.59. We note that hyperparameter tuning would most likely yield further performance improvements in this regard.
Table J.I suggest that on average
the very competitive nnU-Net
Table J.1: Results of medical image segmentation training the BTCV, Heart, and BraTS datasets. We show the mean Dice score across folds (3 for BTCV) for a range weight decay values (?) for AdamW and different warm start steps for CPR. The learning rate warmup is 2k.
SGD
SGD+CPR
le-5
le-4
1e-3
le-2
le-1
1k
2k
3k
4k
BTCV
83.04
83.1
83.17
83.99
73.92
81.17
84.14
84.23
55,41
Heart
92.92
92.75
92.88
92.9
92.85
92.77
93.18
93.16
74.44
BraTS
75.85
76.01
76.22
76.12
75.42
75.29
76.46
76.65
75.63
32

---

Table J.2: Hyperparameters of the medical image segmentation experiments.
Parameter
Value
Fold
0,1,2,3,4
Dataset
BTCV, Heart, BraTS
Preprocessing
Default nnU-Net preprocessing 440
Batch size
2 (following 40]
Patch size
(48x192x192 BTCV, (80x (xxx16xx16) Heart, (128x128x128) BraTS
Training Steps
125k (BTCV), 25k (Heart & BBraTS)
Model
3d fullres U-Net (following
40
Optimizer
AdamW / AdamCPR
Learning Rate
0.01
Betal
0.9
Beta2
0.99
Weight Decay
-e- lE 1 (AdamW)
Schedule
Polynomial decay with warmup
Lr Warmup Steps
2000
LA Polynomial exponent
0.9
CPR-M
1.0
CPR-K
1.0
CPR-k
False
CPR-R warm-start steps
4000
Adaptive Bounds
False
33

---

K Experiments on Fine-tuning a Large Language Model
Table K.1: Hyperparameters for the fine-tuning an LLM experiment.
Parameter
Value
Model Name
mistralai/Mistral 7B-Instructt-O.. q_proj. y_projj k proj. o_proj.
Replace Layer
gate prop__proo.. up upp_proo, down down_prooj
Learning Rate
0.0005
Warmup Steps
50
Pubmedqa Artificial Samples
100000
Epochs
Batch Size
128
Lora R
128
Lora Alpha
1.0
34

---

Adam
AdamCPR (Kappa- WS)
Leorning Rate
1e-4.
10.46
3.8
10.34
3.8
10.13
3.9
10.57
10.22
4.2
10.55
4.2
PubMelQA
1o-4
3.7
4.0 10.39
1.0.7
1.1.07
1.0.78
1.121
1.0.55
1o-3.5
3.1
2.6
3.3
3.2
3.4
3.1
::0.66
Learning Hate
1o4.5-
1:0.6
-1.1
-0.9
-0.3
-0.9
-1.0
-1.3
Arithmetic
1o-4
-0.8
-1.2
-0.6
-0.3
-0.4
-1.1
+0.31
+:092
20.71
#0.68
#0.91
+0.99
1e-3.5
+0.80
-0.6
#2.11
-1.3
#1.13
-0.33
#0.15
#0.38
1.0
+0.61
Learning Rate
1o4.55
#0.10
0.9
#0.40
0.8
=0.58
1.0
#0.48
1.1
#0.24
0.9
+0.13
-
MMLU
1o-4
#0.41
#0.55
+0.60
#0.11
+0.55
0.5
+0.39
10
1e:3.5
+1.25
-1.9
+0.41
-2.6
#2.09
-1.0
#0.79
1.5
#1.13
1.2
+1.07
-0.8
-3
%
Learning Hate
1o-45
4.3
-1.2
1.1
1.2
10.25
4.2
1.3
PIQA
lo-4
#0.22
3.7
+0.41
3.8
20.27
3.8
#:0.1
3.8
#0.12
3.8
+:026
3.8
25
3.1
4.2
1o-3.5
#0.26
#0.49
20.73
#0.21
#0.25
+0.12
lo4.5
-13.1
-13.1
-13.6
-13.1
-13.2
Learning Hate
#0.67
10.77
=0.31
20.37
#0.22
+0.56
TruthfiIQA
1o-4
-19.5
-19.4
-18.9
-19.7
-19.3
-19.5
+0.91
+0.28
#1.19
#1.28
=0.75
+0.38
1e-3.5
-22.1
-21.6
-22.8
-10.9
-1.00
-20.8
+1.82
+1.53
+.20
=0.05
#0.67
+1
0.001
0.01
0.1
50 (1x)
100 (2x)
200
(4x)
Weight Decay
Warm start steps Ir warmup)
Figure K.I: Percentage of performance change before and after fineuning Mistral 7B with pubmedQA artificial data with the of AdamW (left) and AdamCPR (right). AdamCPR as a regularization function and Kappa-wS. We use a learning rate warm-up of 50 steps. The heatmap shows the mean performance and standard deviation across three random seeds. We use the Arithmetic dataset with 10 tests that involve simple arithmetic problems in natural language (0), the comprehensive MMLUU benchmark [51), the PiQA benchmark on reasoning about physical commonsense in natural language [52]. and the TruthfulQA benchmark, which evaluates models' abilities mimic human falsehoods 39
the L2 norm
35

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The paper discusses limitations in the experiments in Section 5 and in the
discussion Section 6 The runtime overhead discussed in detail in Appendix
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sellsspeciiiationn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
36

---

Answer:
[NA]
Justification:
The paper does not include theoretical results that require formal proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The paper provides comprehensive details about the experimental setup, including all hyperparameters and datasets. We provide training code for experiments and the implementation of our method in the supplemental materials.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
37

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We provide training code for experiments and the implementation of method in the supplemental materials. All used libraries and datasets are publicly available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cc.
cablic//uuddes/Coeeeubbiis wtt.eeeuwwwbiiiioooooooooooogg for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips. wwtp:/cccppplgguuuiiaeeeaec
Stteee//CoSSSSemsssS.Si. www.ssssiicgcyo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper specifies all necessary training and evaluation details, including hyperparameters and data splits.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The authors run multiple seeds for all experiments in the main paper and report the mean and standard deviation of the corresponding metrics.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoo bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
38

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The paper reports information about the compute resources required for each experiment.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc pt http://wwwwpppcooooi/icccccccaaa
Answer: [Yes]
Justification: The research conducted in the paper conforms the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our paper covers only foundational research and develops generic algorithm for optimizing neural networks.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
39

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer:[Na. NA]
The paper does not involve the release of models or data with a high risk for misuse, and thus, safeguards not applicable this context.
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
The creators and original owners of assets used in the paper are properly credited and referenced.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
40

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thcode wwwccom/aaaoo//c datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The only assets the paper releases code and is well documented in the supplemental material to ensure transparency and reproducibility.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing experiments research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing or research with human subjects, so IRB approvals are not applicable.
Guidelines:
41

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
42


Reviews:
[{"review_id": "eE6jrMmTx1", "reviewer": "Reviewer_gmoY", "summary": "The paper proposes a new regularization technique, Constrained Parameter Regression (CPR), to replace weight decay for training deep learning models. Conventional weight decay penalizes significant weight deviation. However, this can be too restrictive since some layers may need a larger deviation. Instead of applying the same regularization strength for all layers as in weight decay, the paper uses different hyperparameters for each layer, resulting in a more flexible regularization strength. Intuitively, a model with a larger weight $||\\theta||$ is allowed more deviation under CPR. The method is theoretically motivated by the augmented Lagrangian method and tested on multiple tasks and benchmarks.", "questions": "* Why does the inflection point of the regularization function $\\Delta\\Delta R$ reflect a saturation of performance? Shouldn't the inflection point of the loss function $L$ be a better fit for this? \n\n* Could the authors report results on ImageNet? If time is constrained, fine-tuning a pre-trained model, such as CLIP's pre-trained ViT-B [1], to ImageNet is a good choice. \n\n* Is there any empirical evidence supporting the design choice $k\\leftarrow R(\\theta)$? It would be great if the authors could discuss this choice in detail. \n\n* Prior works in hyper-optimization [2,3] show how to optimize hyper-parameters in optimizers differetiably. This could be a valid future direction for optimizing the upper bound $k$. A new work uses hyper-optimization in an optimizer for fine-tuning [4], which shares a similar spirit to this paper at a high level. \n\n[1] Radford, Alec, et al. \"Learning transferable visual models from natural language supervision.\" International conference on machine learning. PMLR, 2021.\n\n[2] Baydin, Atilim Gunes, et al. \"Online learning rate adaptation with hypergradient descent.\" arXiv preprint arXiv:1703.04782 (2017).\n\n[3] Chandra, Kartik, et al. \"Gradient descent: The ultimate optimizer.\" Advances in Neural Information Processing Systems 35 (2022): 8214-8225. \n\n[4] Tian, Junjiao, et al. \"Fast trainable projection for robust fine-tuning.\" Advances in Neural Information Processing Systems 36 (2023).", "limitations": "There is no potential negative societal impact.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* The method is principledly motivated and intuitively explained. Overall, the paper is very well-written.\n\n* The paper proposes multiple alternatives for setting the method's hyper-parameters, taking into account efficiency and tuning flexibility. \n\n* The paper demonstrates superior performance on multiple tasks with modern deep learning models, including large language models.", "weaknesses": "* **The regularization strength (upper bound $k$) relies on empirical observations and intuition**. While the overall method is theoretically motivated, the upper bound $k$ setting relies mainly on intuition. Specifically, the paper does not explain why $k\\leftarrow R(\\theta)$ is a good strategy. Intuitively, this strategy indicates that a larger weight $||\\theta||$ should have less regularization. \n\n* **Computer vision tasks are limited**. For classification, computer vision experiments are conducted on relatively small-scale datasets, such as CIFAR100. The paper should consider using moderate-to-large-scale datasets with well-established benchmarks, such as ImageNet, to make the empirical evidence more convincing. For segmentation, popular benchmarks, such as MSCOCO, could be a better choice to establish the superiority of this regularization.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review of our paper on Constrained Parameter Regularization (CPR). We appreciate your positive feedback on the method's principled motivation, clear presentation, and strong empirical results across multiple tasks. \n\nRegarding your summary, we would like to point out that we do not use “different hyperparameters for each layer”. We introduce only one hyperparameter for the regularization, namely the kappa initialization (or without any hyperparameter when using Kappa IP) but our method regularizes each layer individually with an additional scalar variable for each layer. We also would disagree that “a model with a larger weight $||𝜃||$ is allowed more deviation under CPR” but CPR enforces a norm of $||𝜃|| \\le \\kappa$ earlier in model training.\n\nIn the following, we address your questions:\n\n> Why does the inflection point of the regularization function ΔΔ𝑅 reflect a saturation of performance? Shouldn't the inflection point of the loss function 𝐿 be a better fit for this?\n\nThe inflection point tries to identify a point at which increasing the model complexity (measured through $R$) starts becoming less relevant for reducing the loss than in previous iterations. In other words, the benefit for increasing the model complexity further, starts saturating. Hence, we try to capture the point where $\\Delta R$ (the change in $R$ over successive iterations) starts to decrease. Also, note that each parameter group (weight matrix) has its own inflection point.\n\n> Could the authors report results on ImageNet? If time is constrained, fine-tuning a pre-trained model, such as CLIP's pre-trained ViT-B [1], to ImageNet is a good choice.\n\nWe performed both, new experiments on pretraining a vision transformer on ImageNet as well as finetuning CLIP, as mentioned in the general response. \nWe performed ImageNet pretraining experiments on a vision transformer (DeiT [1]) with two sizes, small with 22M parameters and base with 86M. Unfortunately, we only got the small experiments done until the rebuttal deadline but will provide the base results in a comment on the weekend. The results can be found in Table 1 in the Rebuttal PDF.  \nIn contrast to [1], we found that a higher weight decay works better. However, AdamCPR outperforms AdamW with both kappa initializations, a tuned warm start kappa initialization (Kappa-WS), and the hyperparameter-free Kappa-IP. In the small model training, we outperform weight decay by 0.84% without the need for tuning a regularization hyperparameter.  We measured a very minor runtime increase of 0.02% when using CPR in comparison to AdamW.\nAdditionally, we performed finetuning experiments, as you mentioned, with a CLIP ViT-B to ImagenNet. The results can be found in Table 2 in the Rebuttal PDF.  AdamCPR outperforms AdamW with the use of a tuned Kappa-WS initialization and the Kappa-IP initialization is on par with the best weight decay hyperparametrization. \n\n> Is there any empirical evidence supporting the design choice 𝑘←𝑅(𝜃)? It would be great if the authors could discuss this choice in detail.\n\nNote that $\\kappa$ is a bound for the value of $R(\\theta)$ (Section 4.3). Choosing $\\kappa \\gets R(\\theta)$ for a specific $R(\\theta)$ observed in the training run (as done in Kappa-WS and Kappa-IP), ensures that the bound is active, while also not restricting the training so much that it fails to reduce the loss. In particular, it is ensured that when the bound becomes active, a value of $R(\\theta)$ is enforced for which healthy training dynamics are expected.\n\nWe also performed an empirical evaluation of the different initialization methods which can be found in Appendix E, Figure E.1. While Kappa-kI$_0$ (kappa initialization depending on the initial $R(\\theta)$) also leads to good performance, we found Kappa-WS ( $\\kappa \\gets R(\\theta)$ after $x$ warm start steps) to perform better over a larger span of different learning rates. \n\n\n> Prior works in hyper-optimization show how to optimize hyper-parameters in optimizers differentiable. This could be a valid future direction for optimizing the upper bound 𝑘. A new work uses hyper-optimization in an optimizer for fine-tuning, which shares a similar spirit to this paper at a high level.\n\nWe thank you for this suggestion. Optimizing for $\\kappa$ could be an interesting direction. One way to get gradients for $\\kappa$ could be to use implicit differentiation using the KKT conditions of the CPR problem (see, e.g. [2]). However, this would likely not be straightforward, and while this is an interesting avenue for future extensions of this work, we hope to have demonstrated in the paper that our heuristics, in particular choosing the inflection point for setting $\\kappa$, already provide strong results for negligible computational overhead. \n\nThanks again for your review and useful thoughts.  We believe that we have addressed your concerns and the additional positive results on ImageNet training strengthen our paper. Might we kindly ask you to increase your score in case we address your named concerns?  We would be happy to provide any additional information or clarifications if needed.\n\n____\n\n[1] Touvron, Hugo, et al. \"Training data-efficient image transformers & distillation through attention.\" International conference on machine learning. PMLR, 2021.\n\n[2]  Blondel, M., “Efficient and Modular Implicit Differentiation”, 2021. doi:10.48550/arXiv.2105.15183.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CB6pCXOhJB", "reviewer": "Reviewer_TAzm", "summary": "The paper introduces Constrained Parameter Regularization (CPR), a regularization technique for deep learning by dynamically tailoring regularization to individual parameters. CPR sets upper bounds on statistical measures of parameter matrices and reduces the learning into a constraint optimization problem. The authors conduct a series of experiments to evaluate the performance of the proposed method.", "questions": "Please refer to the comments on weaknesses.", "limitations": "The authors have adequately addressed the limitations", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The article provides an accurate formal description of the background and derivation process of the algorithm, which is clear and straightforward, making it easy to understand.\n\n- The article theoretically explains the differences and connections between the CPR algorithm and weight decay algorithms, providing a good summary of existing optimization methods. It designed multiple initialization methods for the constraint upper bound κ, and extensively verified its effectiveness through experiments in various deep learning domains.\n\n- The experimental results verify that the proposed method is techinically correct.", "weaknesses": "- The optimization algorithm lacks theoretical analysis on aspects such as convergence, which needs to be further analyzed using more rigorous mathematical language. Additionally, the initialization algorithm designed lacks theoretical support.\n\n- The authors conduct experiments on CIFAR100, which is too small to show the superiority of a training method for deep learning. The authors need to give the results on ImageNET or larger-sized datasets. \n\n- The proposed methods have more learnable parameters $\\lambda$ and $\\kappa$. The authoes need to report the addtional computational and memory costs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your time and effort in reviewing our paper. We appreciate your positive feedback on the clarity of our presentation and the extensive empirical validation across multiple deep learning domains. In the following we carefully considered your concerns and questions:\n\n> The optimization algorithm lacks theoretical analysis on aspects such as convergence, which needs to be further analyzed using more rigorous mathematical language. \n\nWe acknowledge this limitation and agree that a rigorous theoretical analysis would strengthen the method. While we focused primarily on empirical validation in this initial work, we recognize the importance of theoretical foundations.\nHowever, a theoretical analysis of CPR exceeds the scope of this work. In particular, providing a fair analysis of the interplay between momentum methods and the decoupled updates (in the spirit of weight decay) likely requires a different treatment than the analysis of the standard augmented lagrangian methods. \nWhat is more, the incomplete optimization (single update step) of the loss, before $\\lambda$ is updated, further complicates the analysis as we cannot rely on the classic interpretation of the update of lambda as a step of dual gradient ascent. Also, since the update of the parameters with $\\nabla R(x) * \\lambda$ is not multiplied with the learning rate, we cannot expect lambda to converge while the learning rate is still decreasing, as stationarity is only reached when the update direction of the optimizer and $\\lambda * R(\\theta)$ cancel. \n\nTherefore, we focused in this work on the empirical evaluation of the method to provide deep learning practitioners with a powerful alternative to weight decay and in the case of Kappa-IP, without the need for tuning a regularization hyperparameter (like the weight decay $\\gamma$).\n\n\n> Additionally, the initialization algorithm designed lacks theoretical support.\n\nBoth, gamma in weight decay and kappa in CPR are hyperparameters to be determined experimentally. \nThis is because any measure of model complexity can only be estimated through the validation data. Without this, the correct model complexity, measured, e.g., through a regularization function, can only be assumed. \nSuch assumptions are implicitly made through the choice of the regularization parameters. \nFor this choice, we propose an initialization heuristic (Kappa-IP) and show that it works well across different tasks, datasets, and architectures. From this, we conclude based on experimental evidence, that the assumption underlying it generalizes well. We provide empirical evidence for the Kappa-IP initialization in experiments on LLMs (Paper Figure 3, Figure 5) and ImageNet (Rebuttal PDF Table 1). \n\n> The authors conduct experiments on CIFAR100, which is too small to show the superiority of a training method for deep learning. The authors need to give the results on ImageNET or larger-sized datasets.\n\n\nWe performed ImageNet pretraining experiments on a vision transformer (DeiT [1]) with two sizes, small with 22M parameters and base with 86M. Unfortunately, we only got the small experiments done until the rebuttal deadline but will provide the base results in a comment on the weekend. The results can be found in Table 1 in the Rebuttal PDF.  \nIn contrast to [1], we found that a higher weight decay works better. However, AdamCPR outperforms AdamW with both kappa initializations, a tuned warm start kappa initialization (Kappa-WS), and the hyperparameter-free Kappa-IP. In the small model training, we outperform weight decay by 0.84% without the need for tuning a regularization hyperparameter. We measured a very minor runtime increase of 0.02% when using CPR in comparison to AdamW.\n\nWe also want to stress that the GPT2s (124M parameter) and GPT2m (345M) are trained on the OpenWebText dataset with ~9B token, which is not small and a common experimental setting in LLM-pretraining. \n\n\n> The proposed methods have more learnable parameters 𝜆 and 𝜅. The authoes need to report the addtional computational and memory costs.\n\nNote that \\lambda and \\kappa are only scalars (one per parameter group / weight matrix). These parameters are not learnable but are used by the Lagrangian optimization. Their updates do not require gradient computations but only the computation of $R(\\theta)$. \n\nFor the computational costs, we would like to refer you to Appendix H “Runtime Analysis on LLM training” of the original submission. We have conducted a detailed analysis of the computational overhead introduced by CPR. For GPT2-small (124M params), CPR introduces only a 0.4% runtime increase compared to AdamW. For GPT2-medium (354M params), the overhead is 2.4%. Importantly, this small increase in compute time translates to significant improvements in model performance and reduced total training time. For example, with GPT2s, we achieved the same performance as AdamW while requiring only 2/3 of the training budget (Figure 1).\n\n\nWe thank you again for your constructive feedback and the opportunity to improve our work. Might we kindly ask you to reevaluate your scoring and increase the score in case we address your concerns? If any concerns remain we are more than happy to discuss these via comments in OpenReview.\n___\n\n[1] Touvron, Hugo, et al. \"Training data-efficient image transformers & distillation through attention.\" International conference on machine learning. PMLR, 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ROssbab0EF", "reviewer": "Reviewer_M5LW", "summary": "This paper illustrates a new training algorithm to improve the weight decay strategy. Instead of giving the same strength to all weights in weight decay, the proposed method penalizes only the elements that are larger than the threshold. Based on extensive evaluation, the proposed method performs better than AdamW/SGD+weight decay.", "questions": "See weakness.", "limitations": "N/A", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The experiment is strong enough to support the proposed method.\n2. The intuition is easy to understand; that is, instead of using the same weight decay strength for all weights, the strength should be different for different weights since the importance is not the same.", "weaknesses": "1. As mentioned in the paper, the computational cost is higher.\n2. The evaluation of SGD can be improved since it is well known that the best test accuracy can usually be achieved when using SGD with weight decay on image classification tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review and feedback on our paper. We appreciate your positive assessment of the strength of our experiments and the intuitive nature of our method. \nWe would like to address the weaknesses you identified as follows:  \n> As mentioned in the paper, the computational cost is higher.\n\nWe acknowledge that there is a small computational overhead with our method depending on the model size as we analyze in Appendix H “Runtime Analysis on LLM training” of the original submission. However, the benefits in performance clearly outweigh this modest increase in cost as detailed in the following.\nFor smaller models and larger batch sizes, the overhead is negligible (less than 1%). Even for our largest tested model (GPT2-XL with 1.5B parameters), the overhead was only 5.76% with a batch size of 1 and decreased to 2.44% at the maximum possible batch size. Importantly, this small increase in compute time translates to significant improvements in model performance and reduced total training time. For example, with GPT2s, we achieved the same performance as AdamW in only 2/3 of the training budget (Figure 1).\n\n\n> The evaluation of SGD can be improved since it is well known that the best test accuracy can usually be achieved when using SGD with weight decay on image classification tasks.\n\nWe thank you for this point and, as mentioned in the general response, we performed additional experiments with SGD and SGD with CPR on the CIFAR100/ResNet18 task. We trained each configuration three times with random seeds and report the mean percentage of correct labels and standard deviation of the experiments. The results are in the Rebuttal PDF in Figure 1. We find that SGD with CPR outperforms SGD with weight decay when using the Kappa-WS initialization. However, the IP initialization seems not to work with the use of SGD, probably due to the changed convergence behavior in contrast to Adam. We will add this experiment and findings to our paper.\n\nThanks again for your constructive feedback and the opportunity to improve our work. Might we kindly ask you to increase your score in case we address your concerns? We would be happy to provide any additional information or clarifications if needed.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "He38gmihzu", "reviewer": "Reviewer_4Jjm", "summary": "This paper presents Constrained Parameter Regularization (CPR) as an alternative to traditional weight decay.\nCPR enforces an upper bound on the L2-norm of individual parameter matrices. It frames learning as a constraint optimization problem solved with the augmented Lagrangian method and can be integrated seamlessly with gradient-based optimizers. \nDuring training, CPR dynamically tailors regularization and reduces the need for hyperparameter selection. \nEmpirical results on computer vision and language modeling tasks demonstrate its effectiveness.", "questions": "1. CPR constrains parameters with an L2 norm upper bound. Will this method impede the model’s learning ability under complex tasks, e.g., very slow convergence or loss of training accuracy? \n2. It would be interesting to investigate CPR’s robustness to noise and distributional shifts. How does CPR perform when the training data is noisy or comes from a different distribution than the test data?", "limitations": "N/A", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The proposed CPR is straightforward to implement and can be integrated with existing gradient-based optimization methods.\n- Experiments demonstrate CPR’s effectiveness in various tasks compared to traditional weight decay. Results also show that the performance of CPR is robust to hyperparameter selection and needs less training budget.\n- The paper is well-written and easy to understand, making the concepts accessible.", "weaknesses": "The general idea of using adaptive regularization does not seem very novel. However, the reviewer does not specialize in this area and won't view this as a significant flaw given CPR’s experimental results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thoughtful review. We appreciate your positive comments on CPR's straightforward implementation, effectiveness across tasks, and clear presentation. We'd like to address your questions and concerns:\n\n> The general idea of using adaptive regularization does not seem very novel. However, the reviewer does not specialize in this area and won't view this as a significant flaw given CPR’s experimental results.\n\nWe agree that the idea of adaptive regularization is not new, however, the idea of applying an upper bound to the $L_2$ norm (or any other statistical measurement) and enforcing this bound with the use of an augmented Lagrangian is novel.\nFurthermore, it gives access to a different hyperparameter ($\\kappa$) for which more suitable initialization heuristics, see Kappa-WS/IP, can be developed than for AdamW’s static weight decay hyperparameter. CPR with the Kappa IP initialization is a hyperparameter-free regularization and the usage of the inflection point of the $L_2$ norm is novel too. Thereby, CPR outperforms AdamW on training with 350M parameter LLM experiments as well as ImageNet vision transformer training. No other regularization alternative to weight decay showed such successful and extensive evaluation.   \n\n> CPR constrains parameters with an L2 norm upper bound. Will this method impede the model’s learning ability under complex tasks, e.g., very slow convergence or loss of training accuracy?\n\nSimilar to weight decay, CPR limits the effective capacity of the model. This naturally interferes with training loss minimization. Nevertheless, regularization, in the form of constraints or penalties (such as weight decay) is usually required to avoid overfitting and increase generalization.\nWe trained multiple complex tasks in our experiments, such as language models (GPT2s/m on OpenWebText) and image classification (ViT on ImageNet) and CPR surpassed traditional weight decay. \n\n\n> It would be interesting to investigate CPR’s robustness to noise and distributional shifts. How does CPR perform when the training data is noisy or comes from a different distribution than the test data?\n\nTo test the performance of CPR on noisy data, we performed an additional experiment with the use of the noisy CIFAR100-C dataset [1] as mentioned in the general response. The results can be found in the Rebuttal PDF Figure 2. We see that AdamCPR outperforms AdamW which could indicate that CPR leads to a more robust optimization. We see that AdamCPR performs better than AdamW with Kappa-WS but not with Kappa-IP. None of the optimizer and hyperparameter configurations lead to an outstanding performance on this task, we wouldn’t claim that CPR is particularly good for noisy data. However, a qualified analysis of the robustness will have the scope of a separate paper and could be an interesting follow-up work. \n\nWe appreciate your openness about your expertise, your fair assessment of CPR's contributions, and your questions. We believe that we have addressed them and that adding the results on noisy data (and ImageNet) will make the paper stronger. If you agree, might we kindly ask you to increase your score? We would be happy to provide any additional information or clarifications if needed.\n\n___\n\n[1] Hendrycks, Dan, and Thomas Dietterich. \"Benchmarking Neural Network Robustness to Common Corruptions and Perturbations.\" International Conference on Learning Representations. 2018.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SKV8KapsRv", "reviewer": "Reviewer_dfuS", "summary": "The authors propose a form of regularization which adjusts the regularization strength based on the weights being inside or outside of a certain norm bound. A violation of the bound results in an increasing penalty and coefficient, while conforming to the bound results in a decreasing or zero regularization penalty.", "questions": "Can you add the OOD or adversarial tests mentioned above?", "limitations": "The limitations are discussed", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The method proposes a way to adaptively control the strength of regularization for weigth groups within a model\n- The model intuitively makes sense, as the regularization term is meant to constrain weights to within a certain region. However, current optimizers leave the regularization in place regardless of how well the weights are conforming to it.", "weaknesses": "- The statement on L123-124 is hard to accept, and I think it needs more explanation. Why if $F(x)$ not suitable for gradient based optimization? It seems that if $F$ and $c$ are both differentiable, then it indeed would provide iseful information to restore the feasibility of an infeasible $x$. Am I missing something here?\n\n- It would be interesting to know what effect the training has on the rubustness to OOD inputs or noise perturbed inputs. Is it possible to run some tests on the trained models related to CIFAR100-C and AdvSQUAD or AdvGLUE (or other suitable language dataset to measure robustness)? I ask this because I suspect that the smaller weight norm maintained throughout training would have the effect of not being overly reliant on particular weights in the model, which would likely result in a model which is more robust to OOD and adversarial inputs.\n\n- The minor runtime overhead mentioned in the paper is actually on the order of 5-6%. The definition of minor is subjective, and I would not consider a 5% increase in runtime minor. I think it would be better to state the actual runtime increase within the main text of the paper to give the reader a better idea of the cost. \n\n## Minor\n\n- L70: pertaining --> pretraining", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review and valuable feedback. We appreciate your positive assessment of our method's intuition and potential for adaptive regularization. We have carefully considered your comments and would like to address them as follows:\n\n> The statement on L123-124 is hard to accept, and I think it needs more explanation. Why if 𝐹(𝑥) not suitable for gradient based optimization? It seems that if 𝐹 and 𝑐 are both differentiable, then it indeed would provide iseful information to restore the feasibility of an infeasible 𝑥. Am I missing something here?\n\nEven if $f(x)$ and $c(x)$ are differentiable, $F(x)$ is not differentiable for $c(x) > 0$. This is because of the maximization over $\\lambda$ in F(x).\n\n$\\underset{x}{\\operatorname{minimize}} \\; F(x) \\quad \\text{with} \\quad F(x) = \\max_{\\lambda \\ge 0} \\; f(x) + \\lambda \\cdot c(x)$.\n\nThe function F(x) (that is to minimize with respect to x) has an inner maximization with respect to $\\lambda$. If $c(x) > 0$, in other words, if x is infeasible, $\\lambda \\to \\infty$ “maximizes” $F(x)$ with respect to $\\lambda$. Hence, for $c(x) >0, F(x)$ jumps to $\\infty$ due to the maximization over \\lambda. On the other hand, for $c(x) < 0$, when x is feasible, the maximization over \\lambda yields $\\lambda = 0$.\nWe can thus alternatively write $F(x)$ as\n$F(x) = f(x)   if c(x) >= 0$     and    $F(x) =  \\infty   if c(x) < 0$\nFrom this formulation, it is evident that we cannot run gradient based optimization on this objective. As soon as we encounter an infeasible x, we receive an input of $\\infty$. \nThe smoothed approximation $\\hat{F}$ in Equation 1 addresses this effect by adding a quadratic term for $\\lambda$ which prevents the maximization from returning $\\infty$. Does this clarify our statement in L123-124?\nHowever, we agree that this point was not well-articulated. We will revise this section to provide a more precise and accurate explanation of the optimization challenges and how our approach addresses them.\n\n\n> It would be interesting to know what effect the training has on the rubustness to OOD inputs or noise perturbed inputs. Is it possible to run some tests on the trained models related to CIFAR100-C and AdvSQUAD or AdvGLUE (or other suitable language dataset to measure robustness)? I ask this because I suspect that the smaller weight norm maintained throughout training would have the effect of not being overly reliant on particular weights in the model, which would likely result in a model which is more robust to OOD and adversarial inputs. Can you add the OOD or adversarial tests mentioned above?\n\nWe performed an additional experiment in CIFAR100-C, see the general response and Rebuttal PDF Figure 2. We found that AdamCPR outperforms AdamW which could indicate that CPR leads to a more robust optimization. We see that AdamCPR performs better than AdamW with Kappa-WS but not with Kappa-IP. None of the optimizer and hyperparameter configurations lead to an outstanding performance on this task, we wouldn’t claim that CPR is particularly good for noisy data. However, a qualified analysis of the robustness will have the scope of a separate paper and could be an interesting follow-up work. \n\n\n> The minor runtime overhead mentioned in the paper is actually on the order of 5-6%. The definition of minor is subjective, and I would not consider a 5% increase in runtime minor. I think it would be better to state the actual runtime increase within the main text of the paper to give the reader a better idea of the cost.\n\n\nWe appreciate your perspective on the runtime increase. Upon reflection, we agree that characterizing a 5-6% increase as \"minor\" may be subjective. In the revised version, we will explicitly state the runtime increase in the main text. \n\nMinor correction: Thank you for catching the typo on L70. We will correct \"pertaining\" to \"pretraining\" in the revised version.\n\nThanks again for your review and useful thoughts. Might we kindly ask you to increase your score in case we address your named concerns?", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewer,

Thank you for your thorough and constructive reviews of our paper on Constrained Parameter Regularization (CPR). We greatly appreciate your thoughtful comments and the opportunity to address your concerns.

We have prepared a 1-page PDF with additional experimental results that directly address several of the points you raised:

- **ImageNet pre-training**: We compare AdamW and AdamCPR in a vision transformer [1] pertaining on ImageNet. We train a small DeiT[1] model with 22M parameters and a base model with 86M parameters with the use of the PyTorch Image Models library [2] for 300 epochs and with the configuration for the DeiT Paper but also with a 10x and 0.1x weight decay value.
Unfortunately, we only managed to run the smaller model experiments within the rebuttal week. The base model experiments are still running and we will announce the results in a comment on the weekend. 
As seen in Table 1 of the Rebuttal PDF AdamCPR outperforms AdamW in the small DeiT training with both kappa initialization methods. Especially the hyperparameter free regularization with Kappa-IP performs best by outperforming the best AdamW run by 0.86%. In the case of this small model, we measured a very minor runtime increase of 0.02% when using CPR in comparison to AdamW (14.85h for AdamW and 14.89h for AdamCPR on 4xA100). 

- **ImageNet finetuning**:  We appreciate the reviewer’s suggestion regarding the evaluation on ImageNet. Following the reviewer's recommendation, we conducted fine-tuning of CLIP’s ViT-B/32 model on the ImageNet dataset. We used the ViT-B/32 model pre-trained by CLIP. The model was fine-tuned for 10 epochs following the hyperparameter choices in [3], with the exception of the special classification head initialization. As in [3], we employ a learning rate of $3 \times 10^{-5}$, default PyTorch AdamW hyperparameters $\beta_1=0.9$, $\beta_2=0.999$ $\epsilon=10^{-8}$ and a cosine-annealing learning rate schedule with 500 warm-up steps. Due to time and compute constraints, the training was performed on a single GPU with a batch size of 512, compared to the original setup of 8 GPUs with a batch size of 512 each. In Table 2 of the Rebuttal PDF, we compare AdamW with different weight decay values to the proposed AdamCPR in different configurations, where we report the top-1 accuracy after finetuning. From these results, we see that the Kappa-WS initialization also leads to better results in this finetuning setting, comparing favorably to traditional weight decay. 

- **SGD CPR experiments**: As mentioned by Reviewer M5LW, the best accuracy can be achieved in image classification (probably on CNNs, on ViTs is Adam used) with SGD and weight decay, So we performed additional experiments with SGD and SGDCPR on the CIFAR100/ResNet18 task. We used SGD with Nesterov momentum of 0.9 and configured the training similarly to the CIFAR100 experiments described in the paper. The results can be found in Figure 1 in the Rebuttal PDF. We trained each configuration three times with random seeds and report the mean percentage of correct labels and standard deviation of the experiments. We find that SGD with CPR outperforms SGD with weight decay when using the Kappa-WS initialization. However, the IP initialization seems not to work with the use of SGD, probably due to the changed convergence behavior in contrast to Adam.

- **CIFAR100-C experiments**: To evaluate CPR's robustness according to data noise, we've included experiments on training a ResNet18 on the noisy CIFAR100-C dataset [4]. The training setup and configuration are similar to the CIFAR100 experiments described in the paper. The results are visualized in Figure 2 in the Rebuttal PDF. We see that AdamCPR performs better than AdamW with Kappa-WS but not with Kappa-IP. None of the optimizer and hyperparameter configurations lead to an outstanding performance on this task, we wouldn’t claim that CPR is particularly good for noisy data. We trained each configuration three times with random seeds and reported the mean percentage of correct labels and standard deviation of the experiments.

These additional experiments significantly strengthen our empirical evaluation and address points you raised. We believe they underscore CPR's effectiveness across a broader range of scenarios and dataset scales. We will add all experiments to our paper. We answer individual questions and address further concerns in each response below.

___

[1] Touvron, Hugo, et al. "Training data-efficient image transformers & distillation through attention." International conference on machine learning. PMLR, 2021.


[2] Wightman, Ross, "PyTorch Image Models.", github.com/rwightman/pytorch-image-models, GitHub repository 2019.


[3] Wortsman, Mitchell, et al. "Robust fine-tuning of zero-shot models." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.


[4] Hendrycks, Dan, and Thomas Dietterich. "Benchmarking Neural Network Robustness to Common Corruptions and Perturbations." International Conference on Learning Representations. 2018.

Meta-review: The submission received mixed ratings after the post-rebuttal discussion period. The main concern appears to be a lack of theoretical analysis. However, while a theoretical analysis would make the submission more appealing, the empirical results appear to be sufficient to merit acceptance. The authors are advised to take into account the detailed reviews as well as the clarifications provided in the rebuttal while revising the paper.
Decision: Accept (poster)