Improving Deep Learning Optimization through Constrained Parameter Regularization

Authors: jörg k h franke
michael  hefenbrock
gregor  koehler
frank  hutter
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-10 15:01:15

Abstract: Regularization is a critical component in deep learning. The most commonly used approach, weight decay, applies a constant penalty coefficient uniformly across all parameters. This may be overly restrictive for some parameters, while insufficient for others. To address this, we present Constrained Parameter Regularization (CPR) as an alternative to traditional weight decay. Unlike the uniform application of a single penalty, CPR enforces an upper bound on a statistical measure, such as the L$_2$-norm, of individual parameter matrices. Consequently, learning becomes a constraint optimization problem, which we tackle using an adaptation of the augmented Lagrangian method. CPR introduces only a minor runtime overhead and only requires setting an upper bound. We propose simple yet efficient mechanisms for initializing this bound, making CPR rely on no hyperparameter or one, akin to weight decay. Our empirical studies on computer vision and language modeling tasks demonstrate CPR's effectiveness. The results show that CPR can outperform traditional weight decay and increase performance in pre-training and fine-tuning.

Main paper: 
 related work: 
 Weight decay is an effective regularization technique to improve the generalization and model performance [12] , and the idea of adapting parameter regularization during training is not new. Lewkowycz and Gur-Ari [8] investigated the effect of L 2 regularization on overparameterized networks and found the time it takes the network to reach peak performance is proportional to the L 2 regularization parameter. They proposed an initialization scheme for L 2 regularization and an annealing schedule for the L 2 parameter. Yun et al. [9] use a combination of weight decay scheduling and knowledge distillation to improve performance on computer vision tasks. More recent works on self-supervised vision transformers also use a weight decay schedule [10, 11] . In contrast to our work, none of these proposes a dynamic and individual adaptation of each regularized parameter matrix. Also, a schedule comes with varying hyperparameter choices while CPR adapts arbitrarily many parameter matrices with only two hyperparameters (out of which one is fixed in all our experiments). Instead of using a schedule, Nakamura and Hong [13] proposes AdaDecay, where the L 2 penalty is scaled by standardized gradient norms and a sigmoid function. Ghiasi et al. [14] propose another gradient-based approach, Adaptive Weight Decay (AWD), which dynamically adjusts the weight decay based on the ratio of weight norms to gradient norms to balance the contributions from the cross-entropy and regularization losses aiming to improve the robustness. AMOS [15] leverages model-specific information for initialization and gradients to adapt L2 regularization during the training. Another way to regularize parameters is to fix the norm of individual parameter matrices [16] , to schedule the weight norm [17] , or to limit the total norm of all parameters [18] to a fixed value. This fixed value is a more sensitive hyperparameter than the hyperparameter in our work.Our proposed method is not the first to use Lagrangian methods in machine learning [19] . Its application in deep learning so far focuses on variational methods and generative models: Rezende and Viola [20] introduced the Generalized ELBO with Constrained Optimization algorithm to optimize VAEs using Lagrange multipliers optimized by the min-max scheme, and Kohl et al. [21] and Franke et al. [22] adapted the Lagrangian method from Rezende and Viola [20] to train probabilistic U-nets and probabilistic Transformer models. While these works adopt Lagrangian methods to handle several losses in joint optimization problems, our work leverages them to enable individual regularization strengths.3 Background 
l 2 regularization and weight decay: 
 Regularization methods, such as L 2 -regularization or weight decay, are commonly used to restrict parameter updates and enhance generalization by reducing unnecessary complexity [3, 4, 5] . Both can be motivated by introducing a "cost" to weight magnitudes. Specifically, in L 2 -regularization, instead of minimizing only the loss function L(θ, X, y) with parameters θ and data D = {(X n , y n )} N n=0 , a weighted penalty (regularization) term R(θ) is added to the loss, resulting in the training objectivemin θ L(θ, X, y) + γ • R(θ),where R(θ) = 1 2 ∥θ∥ 2 2 denotes the regularization function and γ ∈ R + the strength of the penalty. On the other hand, weight decay directly modifies the update rule of the parameters toθ t+1 ← θ t + Opt(L, η) -η • γ • θ t ,where Opt(L, η) denotes an optimizer providing the gradient-based update at iteration t and L = L(θ t , X t , y t ) the loss. For example, Opt(L, η) = -η • ∇ θ L(θ t , X t , y t ) with learning rate η ∈ R + in case of gradient descent. Thus, the main difference between weight decay and L 2regularization is that the gradients of the regularization accumulate in momentum terms in the case of L 2 -regularisation, while they are treated separately in (decoupled) weight decay. This has also been extensively discussed by Loshchilov and Hutter [6] with the introduction of the AdamW optimizer.We briefly review the augmented Lagrangian method for constrained optimization, see e.g. Bertsekas [23] , which our method is based on. For the derivation, we follow the motivation of Nocedal and Wright [24, pp. 523-524] . Consider the following inequality-constrained optimization problemminimize x f (x) s.t. c(x) ≤ 0,with f (x) : R n → R and a constraint c(x) : R n → R. One way to address the constraint is to find an equivalent, unconstrained problem with the same optimal solution. For example,EQUATIONUnfortunately, even if f (x) and c(x) are differentiable, F (x) is not differentiable. This is due to the maximization over λ in F (x), where in case of c(x) > 0, F (x) → ∞. Consequently, we cannot run gradient-based optimization on this objective.To alleviate this problem, we consider a smooth approximation of F (x), namelyEQUATIONwhere λ t ∈ R may be seen as a point we wish to remain proximal to and µ ∈ R + as a factor determining the strength with which this proximity is enforced. For µ → ∞, F (x, λ t , µ) → F (x).The maximization in F (x, λ t , µ) has a closed form solution with λ ⋆ = (λ t + µ • c(x)) + , where (•) + = max{0, •}, see Appendix A for the derivation. Consequently, F (x, λ t , µ) can be written asF (x, λ t , µ) = f (x) + h(x, λ t , µ) with h(x, λ t , µ) = c(x)(λ t + µ 2 c(x)), if λ t + µ • c(x) ≥ 0 -1 2µ λ 2 t else.The constraint thus only interferes with the minimization (gradient) of f (x) if λ t + µ • c(x) ≥ 0. We can now try to solve the unconstrained problem minimizex F (x, λ t , µ) with familiar methods, such as gradient descent, and obtain an approximate solution to the original problem. Specifically, the gradient of F (x, λ t , µ) with respect to x is given by∇ x F (x, λ t , µ) = ∇ x f (x) + λ ⋆ • ∇ x c(x).The quality of the approximation, and thus the solution, clearly depends on µ and λ t . To improve this approximation we can refine the choice of λ t via an iterative procedure and repeat the optimization with λ t+1 ← λ ⋆ = (λ t + µ • c(x)) + . Intuitively, if the previous minimization of F (x, λ t , µ) resulted in an infeasible solution with c(x) > 0, λ t+1 > λ t . Hence, the minimization of F (x, λ t+1 , µ) likely results in a solution with less constraint violation. On the other hand, if c(x) ≤ 0, λ t+1 ≤ λ t . Subsequently, the influence of the constraint is decreased. This loop of alternating minimization of F (x, λ t , µ) and updating λ t can be repeated until a sufficiently good solution is found or the procedure converges if λ t does not receive updates anymore. For multiple constraints c j (x), j = 1, • • • , J, the above can be readily extended with a multiplier λ j t for each constraint. Since the maximization in the smooth approximation is separable in the λ j t , the same update rule may be applied for each λ j t separately using the respective constraint c j (x). 
constrained parameter regularization: 
 In this section, we introduce Constrained Parameter Regularization (CPR), where we adapt the augmented Lagrangian method to enforce upper bounds on regularization terms. Compared to classical regularization, with a fixed regularization coefficient γ, the proposed approach will allow for variable regularization coefficients λ j t (Lagrange multipliers) for j = 1, • • • , J parameter matrices θ j ⊆ θ that should be regularized. These regularization coefficients are updated alongside the network parameters θ.Classical weight decay, as introduced earlier, is used as a means to restrict the freedom of parameter adaptation. This restriction is applied with a scaling factor γ (hyperparameter) and applies uniformly to all parameters. However, we conjecture that applying an individual adaptation pressure instead may be beneficial. Unfortunately, this would require a separate coefficient for each parameter matrix where a separate weight decay should be applied. To avoid the need for separate scaling coefficients, we formulate regularization as a constrained problem. Here, the loss function L(θ, X, y), with network parameters θ, takes the place of the objective. Consequently, the learning problem becomesminimize θ L(θ, X, y) s.t. c j θ j = R θ j -κ j ≤ 0, for j = 1, • • • , J,where R(θ j ) is a regularization function (e.g., the squared L 2 -norm in case of weight decay) for a parameter matrix θ j ⊆ θ, j = 1, • • • , J, and κ j ∈ R denotes a chosen bound.To solve equation 3, we follow the augmented Lagrangian method with slight modifications. First, instead of performing a full optimization of the loss before updating λ t , we perform updates in every step. This is motivated by the fact that full optimization is generally infeasible in a deep learning setting. Moreover, similar to the difference between weight decay and L 2 -regularization, we treat the update between the loss-dependent and the constraint-dependent part separately. Hence, instead of introducing L(x, λ t , µ) analogously to equation 2, and performing optimization on this objective, we independently apply updates for both steps. Consequently, the constraint violations do not accumulate Algorithm 1 Optimization with constrained parameter regularization (CPR) .Require: Loss Function L(θ, X, y) with parameters θ, and data D = {(X n , y n )} N n=0 Require: Hyperparameters: Learning rate η ∈ R + , Lagrange multiplier update rate µ ∈ R + (= 1.0) Require: Optimizer Opt(•) for minimization, Regularization function R(θ) (e.g. L2-norm) 1: λ j t ← 0 for j = 1, • • • , J 2: κ j ← Initialize(θ j 0 ) for j = 1θ t+1 ← θ t + Opt(L(θ t , X t , y t ), η)▷ Classic parameter update using, e.g., Adam.for each regularized parameter group θ j t in θ t do 6:λ j t+1 ← λ j t + µ • (R(θ j t ) -κ j ) + 7: θ j t+1 ← θ j t+1 -∇ θ j R(θ j t ) • λ j t+1 8:end for 9: end for in momentum terms. We also remove the influence of the learning rate on the regularization. From a practical perspective, our modification does not interfere with gradient-based optimization algorithms and can be readily combined with any such optimizer. The full algorithm is given by Algorithm 1.Conceptually, the method can be understood as the λ j t accumulating constraint function values (weighted with µ) over the iterations t. These then increase (or decrease) the influence of the constraint (via its gradient) on the search direction. When points in the feasible domain are found for which c j (θ) ≤ 0, λ j t decreases until it eventually reaches 0. If, on the other hand, the optimal solution lies on the boundary, where c j (θ) = 0, λ j t should converge to a value where the update direction of the optimizer and the gradient of the constraints cancel each other. However, this situation is unlikely to occur in a deep learning setting due to the stochasticity of minibatches.The optimality conditions of the CPR problem and an L 2 -regularized training objective reveal a connection between the two approaches. To see this, consider the training objective of L 2 regularization with a given γ, assuming it has a minimum at θ ⋆ . Consequently, at this point, we have 0 = ∇L(θ ⋆ ) + γ • ∇R(θ ⋆ ), and the value of the regularization function is R(θ ⋆ ).If we set κ ⋆ = R(θ ⋆ ), the Karush-Kuhn-Tucker (KKT) (optimality) conditions for CPR are 0 = ∇L(θ ⋆ ) + λ • ∇R(θ ⋆ ) and R(θ ⋆ ) -κ ⋆ ≤ 0 (which holds with equality), with the Lagrange multiplier λ ≥ 0. We can see that for λ ⋆ = γ, the solution pair (θ ⋆ , λ ⋆ ) satisfies the KKT conditions. Hence, there is a choice of κ (namely κ ⋆ ) for which the CPR problem has the same optimal solution candidates as the L 2 -regularized training objective for a given γ. CPR could therefore be seen as a different approach to searching for the same solution candidates but is parameterized with different hyperparameters (κ instead of γ). Unlike L 2 -regularization (or weight decay), CPR can mimic the behavior of different γ values for different parameter matrices. This behavior changes over time as the λ j values are updated and thus leads to different training dynamics compared to weight decay. Additionally, focusing on a bound on the regularization function κ instead of a penalty coefficient γ may allow us to identify better indicators for the selection of (default) values for these hyperparameters.The upper bound κ is the most crucial hyperparameter for CPR, and we identify four ways to initialize it. (1) Kappa-K: Set κ j ← κ to the same value κ for all parameter matrices. (2) Kappa-kI 0 : Set κ j based on the initial parameter matrices' regularization function value: κ j ← k • R(θ j t=0 ), with k ∈ R + as the factor of the initial measure. (3) Kappa-WS: Train the model parameters θ for a specific number of warm start (WS) steps s ∈ N + and then set κ j ← R(θ j t=s ). (see algorithm for We use a learning rate warm-up of 500 steps and the best Kappa-WS value is 2× the warm-up steps. We report the mean of three runs with random seeds. We see that both CPR versions outperform weight decay CPR with Kappa-WS in Appendix B). While the previous strategies all require a hyperparameter, our last strategy is essentially hyperparameter-free. (4) Kappa-IP: Use the first inflection point (IP) of the regularization function at step i (change of curvature over the training steps) to warm start each parameter matrix individually. Specifically, κ j ← R(θ j t=i ) where i is the first iteration where ∆ t ∆ t R(θ j ) < 0. The intuition behind this choice comes from the fact that the rate of change decreases at the inflection point. This hints at a saturation of the improvement expected through raising the value of the regularization function further. The position of the inflection point thus indicates a good choice for κ, as it demonstrated healthy training dynamics while still restricting the model from over-adapting (see Section 5) . Consequently, this method leverages the natural progression of the model's training rather than relying on an external hyperparameter, aiming to adaptively find a suitable upper bound. 
experiments: 
 We now describe a set of experiments to understand CPR and its parametrization. Preliminary experiments showed that µ is not a sensitive hyperparameter and we chose µ = 1.0 for all our experiments. We provide a detailed analysis of µ in Appendix C. Similar to weight decay, we choose the squared L 2 norm as a default regularization function for CPR. We also tested an adaptive bound, where we adjusted kappa during training but found it not to be beneficial; details are reported in Appendix D. In the following experiments, we regularize all parameters in a network except for bias terms and normalization weights. Since CPR does not require additional gradient calculations or parameter updates, we find only a small runtime overhead with our CPR implementation (in PyTorch, no CUDA optimization, 0.4%-5.8% for GPT2) which is mentioned in each experiment individually and analyzed in Appendix I.To evaluate CPR's effectiveness and design choices, we tested AdamW and Adam with CPR (Adam-CPR) in image classification using a ResNet18 on the CIFAR100 dataset [25, 26] . We compared AdamW to AdamCPR with the four initializations described in Section 4.3. The initialization Kappa-WS after s warm steps performed best, see Figure 2 . We base our choice of the warm start on the 500 steps learning rate warmup out of 20k total training steps and found a large range of hyperparameters that consistently outperform weight decay. Also, the hyperparameter-free method Kappa-IP outperforms weight decay. To detect the infection point, we found it sufficient to sweep the statistical measure in an interval of 10% of the learning rate warmup. We also apply this in all further experiments. The superior performance of Kappa-WS and Kappa-IP may be due to its general flexibility, as warm-started bounds may be considered "learned," reflecting the actual magnitudes and distributions of the parameter matrices during training. Appendix E contains training details and a plot with all initializations and standard deviation across three random seeds in To investigate the relationship between the learning rate warm-up and the number of warm start steps s of Kappa-WS or Kappa-IP, we experimented with varying warm-up steps. We found that setting the CPR warm start steps s to twice the warm-up steps is a good initial choice. For very low warm-up steps, the best s was four times the warm-up count. Conversely, with a long warm-up phase, a shorter CPR warm start (×1) is preferable. Notably, the optimal choice of s is almost independent of the learning rate, as shown in Figure E.3. The optimal warm start steps are consistent across a wide range of learning rates. A simple baseline representing a similar regularization approach is a weight decay schedule. We evaluated a cosine schedule for decreasing and increasing weight decay values, similar to [10, 11] . The results, shown in Figure E.4, indicate that the decreasing schedule outperforms a fixed weight decay but not CPR. We tested if CPR is particularly good for noisy data and perfomed experiments on the noisy CIFAR100-C dataset [27] . The results, in Figure E.5, show that AdamCPR outperforms AdamW slightly. However none of the optimizer and hyperparameter configurations lead to an outstanding performance on this task, we wouldn't claim that CPR is particularly good for noisy data. We also used CPR with SGD. We found, as shown in Figure E.6, that SGD with CPR outperforms SGD with weight decay when using the Kappa-WS initialization. However, Kappa-IP seems not to work with SGD, probably due to the changed convergence behavior in contrast to Adam.Additionally, we compared our method to related work. We implemented AdaDecay [13] and evaluated the method for different alpha values, as seen in Figure E.7. We also compared AdamW and AdamCPR to adaptive Weight Decay (AWD) [14] and AMOS [15] . Furthermore, we used Adam with parameter rescaling from Liu et al. [18] . We found AdaDecay superior to AdamW, while AMOS and Rescaling performed less well. However, CPR outperforms all related approaches. We report all results across multiple learning rates and weight decay values in Figure E.8.We compare AdamW and AdamCPR in vision transformer [29] training on ImageNet [30] . We choose to train the DeiT [28] model with 22M (small) and with 86M (base) parameters. We make use of the PyTorch Image Models library [31] and train with the configuration given in [28] for 300 epochs. To explore the impact of weight decay, we also train with a 10× and 0.1× the weight decay value. For CPR, we initialize with Kappa-WS (× lr-warmup) and Kappa-IP. We observed a minor runtime increase when using CPR. For example, training the small model on 4 A100 GPUs took 14.85h for AdamW and 14.89h for AdamCPR. All relevant hyperparameters can be found in Appendix F. As seen in Table 1 , AdamCPR outperforms AdamW for small and base DeiT training with both kappa initialization methods. Most notably, the hyperparameter-free regularization with Kappa-IP outperforms AdamW in both cases. However, in the base model training, Kappa-WS surpasses Kappa-IP.We conducted fine-tuning experiments using a CLIP model [33] on the ImageNet dataset. We used the ViT-B/32 model pre-trained by Radford et al. [33] . The model was fine-tuned for 10 epochs following the hyperparameter choices of Wortsman et al. [32] (learning rate of 3 × 10 -5 , cosineannealing learning rate schedule with 500 warm-up steps) but without the special classification head initialization and the training was performed on a single GPU with a batch size of 512. We compare AdamW with different weight decay values to AdamCPR in different configurations, where we report the top-1 accuracy after finetuning. The results in Table 2 show that the Kappa-WS initialization also leads to better results in this finetuning setting, comparing favorably to traditional weight decay. CPR with Kappa-IS performs similarly to the best weight decay values, but again, without the need for finding a regularization hyperparameter.We performed experiments training a GPT2 language model [34] on Openwebtext [35] . We compared AdamW on different weight decay values to AdamCPR using Kappa-WS with different warm start steps and Kappa-IP. We use a learning rate warmup for 5k steps (2.5% of total training steps) followed by cosine annealing. Again, we select the warm start steps of κ based on the warmup steps of the learning rate and evaluate s ∈ (5k, 10, 20k) steps. We train the model sizes GPT2s and GPT2m with 124M and 354M parameters for 200k steps. The results are shown in Figure 3 . CPR outperforms weight decay at all learning rates, in both model sizes and with both kappa initialization strategies. We also see that the Kappa-IP initialized CPR runs are less sensitive to the learning rate than weight decay γ. Remarkably, CPR with the hyperparameter-free initialization Kappa-IP performs best, achieving 0.2 to 0.3 better perplexity than weight decay. To illustrate the performance difference, we trained a model with weight decay for a longer schedule to get the same performance as with CPR, the result is shown in Figure 1 . CPR saves up to 33% training budget on that scale. Figure 5 shows the difference in training dynamics with CPR. We find that Kappa-IP is close to the optimal warm start step for Kappa-WS but find individual starting points for different layers, see Figure G.1. We provide details of the training and hyperparameters in Appendix H. We found no runtime overhead of CPR in contrast to AdamW training GPT2s but about 2.5% for GPT2m (see runtime analysis in Appendix I). We also evaluated AdaDecay [13] , Adaptive Weight Decay (AWD) [14] and AMOS [15] in the GPT2s training setting but neither of the related methods outperforms AdamW nor AdamCPR, see results in Table H We use a learning rate warm-up of 50 steps. We see that CPR outperforms weight decay for each learning rate.Probably a more common task than pre-training a large language model (LLM) is to fine-tune one. Hence, we evaluate CPR in the fine-tuning of the Mistral7B large language model [36] , incorporating low-rank adaptation (LoRA) [37] . Specifically, we fine-tune artificially generated biomedical questionanswering (QA) pairs from the PubMedQA dataset [38] . We fine-tune all attention and feed-forward weights using either AdamW or AdamCPR with a learning rate warm-up of 50 steps, followed by cosine annealing. We experiment with different values of weight decay and warm start steps for Kappa-WS, set at 1×, 2×, and 4× the learning rate warm-up steps. The fine-tuning was performed on four GPUs for about 1h. Each configuration is trained across three random seeds. We evaluate the LLM before and after the fine-tuning on the expert-annotated PubMedQA QA instances and report the change in answer accuracy (means and standard deviations across three random seeds) in Figure 4 . The fine-tuning enhances the performance on the PubMedQA benchmark and CPR outperforms AdamW for each learning rate. As in both the ImageNet and GPT2 experiments, the best Kappa-WS value was 2× the warm-up steps (here, 50 × 2). We also tested Kappa-IP but it performed worse due to the lack of an inflection point for some parameters, short learning rate warmup, and different training dynamics with LoRA. We also found that CPR helps to mitigate catastrophic forgetting, therefore we evaluate before and after finetuning on a set of benchmarks and found that CPR with some learning rates helps to reduce a performance drop e.g. on the TruthfulQA benchmark, which evaluates models' abilities to mimic human falsehoods [39] , on up to 3% (see results in Figure K.1). Detailed hyperparameters and plots including standard deviations are available in Appendix K.Aside from image classification, we also applied CPR to (medical) image segmentation using the nnU-Net framework [40] and training with the SGD optimizer in combination with CPR with Kappa-WS. For this, we considered the tasks of Multi-Atlas Labeling Beyond the Cranial Vault (BTCV) [41] where we improve the Dice score from 83.99 to 84.23, the Heart Segmentation task of the Medical Segmentation Decathlon [42] where we improve the Dice score from 92.92 to 93.18 and the 2020 version of the Brain Tumor Segmentation challenge (BraTS) task [43] where we improve the Dice score from 76.22 to 76.65. These results show that CPR also works in combination with SGD where we replace weight decay. Training details for the task and all results are in Appendix J. 
discussion: 
 Our extensive evaluation of Constrained Parameter Regularization (CPR) across multiple tasks underscores its effectiveness as a robust alternative to traditional weight decay. A critical aspect of CPR's success is its initialization strategy. To this end, we propose four strategies to initialize the upper bound κ. With our findings, we identify two strategies, Kappa-WS and Kappa-IP as prime candidates showing a strong performance, consistent across multiple tasks. The good performance of the warm-started bound Kappa-WS can be attributed to the fact that even a carefully chosen initialization of parameters does not consider the training task and data. Therefore, the actual parameter weights during training are better reflected in a warm-started bound, which also takes into account the network's depth and the varying gradient updates in deeper layers. We found that setting the CPR warm start steps s to twice the learning rate warm-up steps serves as an effective initial configuration for any training setup. However in a pre-training setting, setting the upper bound based on the first inflection point of the regularization function (Kappa-IP) yields an additional advantage: It removes even the one hyperparameter present in the warm start strategy, bringing the regularization capabilities of CPR without any additional hyperparameters. Simultaneously, this strategy shows best-in-class performance in GPT2 training, seemingly even extending the range of usable learning rates on a given task. This reduces the effort in hyperparameter optimization not only for the optimal regularization but also for the optimal learning rate. CPR also changes the training dynamics, as shown in Figure 5 and Figure G.1. While both weight decay and CPR can achieve a similar final L2 regularization, the path to this norm is different. Weight decay allows for intermediate overadaptation with high L2 norms, whereas CPR controls the L2 norm throughout the entire training process. This results in a slower initial loss drop but a more consistent decay, leading to a better final performance.A noted limitation of CPR is an increase in runtime by up to 6% for larger models (1.1B parameters), as detailed in Appendix I. However, for smaller models or larger batch sizes, this overhead is negligible. The benefit of CPR diminishes in scenarios where weight regularization has minimal impact, such as when training small models on large datasets with a high ratio of training samples to parameters. Future research could explore the application of CPR to even larger models and a broader range of tasks.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: The paper specifies all necessary training and evaluation details, including hyperparameters and data splits.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
conclusion: 
 Constrained Parameter Regularization (CPR) offers a significant advancement in regularization techniques, providing a robust and efficient alternative to traditional methods. By enforcing an upper bound on the regularization function, CPR integrates seamlessly with gradient-based optimizers and incurs minimal runtime overhead. Its dynamic tailoring of regularization to individual parameter matrices and reduces hyperparameter optimization by eliminating the need for a weight regularization hyperparameter in pre-training. Our four experiments demonstrate that neural networks trained using CPR outperform those with traditional weight decay. These findings highlight CPR's potential as a versatile and powerful tool for improving model performance and open promising future research.Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: The authors run multiple seeds for all experiments in the main paper and report the mean and standard deviation of the corresponding metrics.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer:[NA]Justification: The paper does not involve the release of models or data with a high risk for misuse, and thus, safeguards are not applicable in this context.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: The creators and original owners of assets used in the paper are properly credited and referenced.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes] Justification: The only assets the paper releases are code and is well documented in the supplemental material to ensure transparency and reproducibility.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA]Justification: The paper does not involve crowdsourcing experiments or research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: The paper does not involve crowdsourcing or research with human subjects, so IRB approvals are not applicable.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
: 
 Deep neural networks are the bedrock of many state-of-the-art machine learning applications [1] . While these models have exhibited unparalleled expressivity, they also possess millions, sometimes trillions, of parameters [2] . This massive capacity makes them susceptible to overfitting, where models memorize nuances of the training data but underperform on unseen examples. To mitigate this, many different regularization techniques have been adopted, with weight decay and L 2 regularization [3, 4, 5] being the most popular. L 2 regularization penalizes the squared magnitude of model parameters and (decoupled) weight decay (which is equivalent to L 2 regularization for non-adaptive gradient algorithms [6] ) multiplies all weights with a constant at every step. This seemingly simple act offers numerous benefits by curbing the growth of individual weights, reducing the risk of relying on any particular feature excessively, and thus promoting model generalization.However, not all parameters in a neural network have the same role or importance and different weights could benefit from different regularization. Similarly, it is unclear if a single weight decay value is optimal for the entire duration of optimization, especially for large-scale training. Indeed, Ishii and Sato [7] showed that a small deep learning model could benefit from layer-wise weight decay values, and various works showed that scheduling weight decay could improve final performance [8, 9, 10, 11] . This indicates that a dynamic penalty for each individual parameter matrix (e.g., a weight matrix in a linear layer) could be beneficial for neural network training. Since both scheduling and parameter-wise weight decay require additional hyperparameters that are often sensitive to the task, we propose a different approach to obtain customized, dynamic parameter regularization. Instead of uniformly penalizing weights, we propose to keep them in a certain range, thus ensuring stability without imposing regularization where it is unnecessary. Constraining parameters, especially based on statistical measures like the L 2 norm, provide a flexible and adaptive form of regularization that accounts for the heterogeneity of parameters.In this paper, we propose Constrained Parameter Regularization (CPR), which enforces an upper bound on a statistical measure of individual parameter matrices. Consequently, regularization is expressed as a constrained optimization problem, which we address by an adaptation of the augmented Lagrangian method. The regularization of each parameter matrix is handled by a separate constraint and Lagrange multiplier, resulting in an individual regularization strength that adapts over time. The method requires the selection of desired constraint values as well as an update rate for the Lagrange multipliers. We found that the update rate can be fixed to 1.0. For choosing constraint values, we introduce four strategies, three of which require a single hyperparameter, while the last one is hyperparameter-free. We show in our experiments performance improvements over weight decay when pre-training or finetuning models for image classification (CIFAR100 and ImageNet), language modeling (OpenWebText), and medical image segmentation. For example, when training a GPT2s model, we achieved the same performance as AdamW but only require 2/3 of the budget, see Figure 1 . Applying our method for fine-tuning, we find performance improvements and less catastrophic forgetting. In the following, and after discussing related work (Section 2) and background on weight decay and the augmented Lagrangian method (Section 3), we make the following contributions:• Introducing CPR for individualized and dynamic weight regularization 1 . Specifically, formulating regularization as a constraint optimization problem and proposing CPR as a solution (Section 4.1). • Identifying four different strategies for initializing this constraint (Section 4.3). One of them, Kappa-WS, has a strong default that outperforms tuned AdamW; and another one, Kappa-IP, is entirely hyperparameter-free and yields even better performance in pre-training. • Showing improved performance over weight decay in image classification, medical image segmentation, and pretraining and fine-tuning language models (Section 5). 
 
 Appendix: For simplicity, we consider a single constraint. Note that multiple constraints can be addressed separately as the optimization problem would be separable in the respective λ j . We need to solvemaximize λ≥0 f (x) + λ • c(x) - 1 2µ (λ -λ t ) 2 .The optimal point of this problem is equivalent to the optimal point ofminimize λ -f (x) -λ • c(x) + 1 2µ (λ -λ t ) 2 s.t. -λ ≤ 0.To find candidates for optimal points, we need to solve the Karush-Kuhn-Tucker (KKT) system with the Lagrange function L(λ, ψ) and the Lagrange multiplier ψL(λ, ψ) = -f (x) -λ • c(x) + 1 2µ (λ -λ t ) 2 -ψ • λWhich leads to the KKT systemEQUATIONAccording to the complementary conditions in equation 3, the constraint is either active, hence λ = 0 and ψ ≥ 0 or inactive, such that λ > 0, and consequently, ψ = 0.Case: λ = 0 and ψ ≥ 0Here, λ = 0 (by assumption), and ψ is given by∇ λ L(λ, ψ) = 0 ⇐⇒ 0 = -c(x) + 1 µ (0 -λ t ) -ψ ψ = -c(x) - λ t µSince we require ψ ≥ 0 for a KKT point, (note that µ > 0)0 ≤ ψ = -c(x) - λ t µ ⇐⇒ 0 ≤ -µ • c(x) -λ t ⇐⇒ 0 ≥ λ t + µ • c(x)Consequently, λ = 0 is a candidate for the optimal point only when 0 ≥ λ t + µ • c(x).Case: λ > 0 and ψ = 0 (inactive constraint) For this case we get∇ λ L(λ, ψ) = 0 = -c(x) + 1 µ (λ -λ t ) -0 0 = -µ • c(x) + λ -λ t λ = λ t + µ • c(x)Due to the geometry of the problem (quadratic with bound constraint), λ = 0 is the optimal solution if the constraint is active, i.e., if ψ ≥ 0, which is the case if 0 ≥ λ t + µ • c(x). Consequently, the optimal solution is given byEQUATIONPlugging this into F (x, λ t , µ), we getF (x, λ t , µ) = f (x) + c(x)(λ t + µ 2 c(x)), if λ t + µ • c(x) ≥ 0 f (x) -1 2µ λ 2 t , elseAnd the gradient with respect to x is∇ x F (x, λ t , µ) = ∇ x f (x) + ∇ x c(x)(λ t + µ • c(x)), if λ t + µ • c(x) ≥ 0 ∇ x f (x) -0 elseOr more compactly by using equation 4∇ x F (x, λ t , µ) = ∇ x f (x) + ∇ x c(x) • λ ⋆ .B The CPR Algorithm with Kappa-WS Algorithm 2 Optimization with constrained parameter regularization (CPR) and Kappa-WS .Require: Loss Function L(θ, X, y) with parameters θ, and dataD = {(X n , y n )} N n=0Require: Hyperparameters: Learning rate η ∈ R + , Lagrange multiplier update rate µ ∈ R + , starting step s for CBR.Require: Optimizer Opt(•) for minimization, Regularization function R(θ) (e.g. L2-norm) 1: # Initialization 2: t ← 0 3: θ t ← Initialize(L(•)) 4: λ j t ← 0 for j = 1, • • • , J 5: κ j ← ∞ j = 1, • • • , J 6: # Training 7: for X t , y t ∼ D do 8: θ t+1 ← θ t + Opt(L(θ t , X t , y t ), η)▷ Classic parameter update using, e.g., Adam.for each regularized parameter group θ j t in θ t do 10:λ j t+1 ← λ j t + µ • (R(θ j t ) -κ j ) + 11: θ j t+1 ← θ j t+1 -∇ θ j R(θ j t ) • λ j t+1 12:if t = s then ▷ Kappa-kI s initialization, see Section 4.3.13:κ j ← R(θ j t ) 14:end if We analyze the sensitivity of the update rate µ in CPR with experiments on ResNet18 trained on the CIFAR100 and GPT2s trained on OpenWebText. For the ResNet18 experiments, we consider update rates from µ = 0.01 to µ = 10 and apply two kappa initialization methods,Kappa-kI 0 and Kappa-WS. As shown in Figure C .1 we see no significant impact of µ on the performance. We report the mean percentage of correct labels across three random seeds. We also performed short-runtime experiments with GPT2s and update rates of µ ∈ {0.01, 0.1, 1, 10}. and observe very similar results, see Table C With fixed bounds κ j , some parameter matrices θ j , for which λ j t = 0 will not be regularized. While this can be beneficial, CPR can also be used to apply continuous pressure similar to weight decay. For this, the bounds κ j of parameter matrices θ j with λ j = 0 can be set to the current value of the constraint function κ j t+1 ← c(θ j t ). Such an adaption guarantees that each parameter matrix is always exposed to some regularization. This should result in a gradual reduction of the bounds κ j throughout training without exerting excessive pressure on the optimization process. In our experiments, we refer to the usage of adaptive bounds as AdaCPR.This contrasts with weight decay, where continuous pressure is applied to enhance generalization throughout the training. To emulate the continuous pressure of weight decay, we propose an adaptive mechanism to adjust the upper regularization bound during training. This can be achieved by leveraging existing states. Specifically, the value of λ j offers insights into constraint violations. When λ j = 0, the constraint c j (θ) can be regarded as inactive. In this case, we may consider adjusting its bound κ j to align with the current constraint value of c(θ j ). To implement these adaptive bounds, we add a conditional update rule for κ j after our CPR update. It updates the upper bound for each parameter matrix θ j t individually byκ j t+1 ← R(θ j t ) if λ j t = 0 and λ j t-1 > 0 κ j t otherwise,where λ j t-1 > 0 indicates that the upper bound was previously violated and c j (θ j ) was active. Consequently, this enables a gradual reduction of the bounds κ j throughout training without exerting excessive pressure on the optimization process. Please find AdaCPR in Algorithm 3 below. 4: λ j t ← 0 for j = 1, • • • , J 5: κ j ← θ j t -Initialize(θ j 0 ) for j = 1, • • • , J 6: # Training 7: for X t , y t ∼ D do 8: θ t+1 ← θ t + Opt(L(θ t , X t , y t ), η)▷ Classic parameter update using, e.g., Adam.for each regularized parameter group θ j t in θ t do 10:λ j t+1 ← λ j t + µ • (R(θ j t ) -κ j ) + 11: θ j t+1 ← θ j t+1 -∇ θ j R(θ j t ) • λ j t+1 12:if λ j t = 0 and λ j t-1 > 0 then ▷ Update κ j if the constraints are not active.13:κ j ← R(θ j t ) 14:end if15:end for 16:t ← t + 1 17: end forThe experimental results in Figure E.1 also show that the adaptation of the upper bound during the training is not beneficial. While it does not harm the performance, it also does not lead to a substantial improvement. We therefore do not use it to keep our method as simple as possible.For the κ initialization Kappa-K, we use a range of κ = [0.005, . . . , 0.16], for Kappa-kI 0 a range of k = [4, . . . , 256], and for Kappa-WS a range of s = [250, . . . , 4000] steps. We use a learning rate warmup of 500 steps followed by a closing annealing. This is 2.5% of the total training steps (20k). For a detailed list of training hyperparameters, we refer the reader to Table E.1.We found that initializing with Kappa-kI 0 performs better than selecting a uniform κ in Kappa-K. This may be explained by the value of the regularization function depending on the size of the jointly regularized parameter matrix and initialization method. The warm start κ initialization method, Kappa-WS, performed the best. The best configuration with CPR outperforms weight decay and the choice of hyperparameters seems to be more robust. Adam+AdaCPR (Kappa-K) AdamCPR lr-warmup 1k steps We evaluated the task with cosine decreasing weight decay to 0.1 and 0.01 times the initial weight decay value and with cosine increasing weight decay to 10 and 100 times the initial weight decay value. The elements in the heat map are experiments with different learning rates and regularization hyperparameters. Each element is colored according to the mean accuracy of three random seeds and the numbers are the mean accuracy and standard deviation of the experiments. It should be mentioned that Yun et al. [9] also performed weight decay scheduling on CIFAR100 with the use of a ResNet18. Since their code was not published, we point to Figure 3 of their experimental results, where an accuracy of around 60% was reported, which is below our AdamW baseline. Each element is colored according to the mean accuracy of three random seeds and the numbers are the mean accuracy and standard deviation of the experiments. We see that AdamCPR outperforms AdamW which could indicate that CPR leads to a more robust optimization. We see that AdamCPR performs better than AdamW with Kappa-WS but not with Kappa-IP. Kappa-IP does not fail and performs better than the average weight decay performance. None of the optimizer and hyperparameter configurations lead to an outstanding performance on this task, we wouldn't claim that CPR is particularly good for noisy data. For an efficient implementation, we use flash attention [45] and rotary position embedding [46] . The complete hyperparameters can be found in Appendix H. The GPT2s and GPT2m models are trained on 8 A100 GPUs up to 28h. A detailed runtime analysis can be found in Appendix I To analyze the runtime in more detail, we measured the runtime per step of different regularization techniques on different GPT2 model sizes (see Table I .1). For AdamW we use the PyTorch 2.1 default implementation, for AdamCPR we adapt the AdmW implementation of PyTorch with the implementation described in Algorithm 1, for AWD and AdaDecay exists no open source implementation and we implemented it based on the PyTorch Adam class but without "for_each" optimization, and for AMOS we used the implementation form the pytroch-optimizer package [47] . We compare the runtime on a node with 4 A100 GPUs and report the mean time per training step across two random seeds and 3000 steps per experiment. In Table I .2 we compare the runtime with a batch size of 1 and in Table I .3 we repost the runtime with the maximal possible batch size on a 40GB A100 (in samples steps of 4). The runtime comparison across various GPT2 models shows that AdamCPR closely matches AdamW's efficiency, particularly at larger batch sizes where its runtime increase becomes minimal or even slightly better. In contrast, Adam AdaDecay, AWD, and AMOS significantly increase runtime, particularly in larger models and batch sizes.However, since not all operations for CPR are implemented in a "for_each" optimized manner, CPR's runtime could benefit from an additional CUDA-optimized implementation.To demonstrate the effectiveness of the proposed CPR approach where using SGD, we also evaluate it in the context of medical image segmentation. We test CPR on four segmentation benchmarks. First, with the Adam optimizer on the Multi-Atlas Labeling Beyond the Cranial Vault (BTCV) [41] task, the Heart Segmentation task of the Medical Segmentation Decathlon [42] and the 2020 version of the Brain Tumor Segmentation challenge (BraTS) task [43] .Here, we make use of the data pipeline and network architectures following the nnU-Net framework [40] , which is regarded as the state-of-the-art framework for medical image segmentation. We implement a training schedule with a total of 25k steps (for the Heart and BraTS tasks) and 125k steps for BTCV. We introduce a learning rate warmup of 2k steps (8%), followed by a polynomial annealing, see all hyperparameters in Appendix J. We run each experiment on one consumer GPU for up to 2 days. We present the results in Table J We use a learning rate warm-up of 50 steps. The heatmap shows the mean performance and standard deviation across three random seeds. We use the Arithmetic dataset with 10 tests that involve simple arithmetic problems in natural language [50] , the comprehensive MMLU benchmark [51] , the PiQA benchmark on reasoning about physical commonsense in natural language [52] , and the TruthfulQA benchmark, which evaluates models' abilities to mimic human falsehoods [39] .Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes]Justification: We provide training code for experiments and the implementation of our method in the supplemental materials. All used libraries and datasets are publicly available.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.38th Conference on Neural Information Processing Systems (NeurIPS 2024).Please find our implementation under https://github.com/automl/CPR. 


Decision: The submission received mixed ratings after the post-rebuttal discussion period. The main concern appears to be a lack of theoretical analysis. However, while a theoretical analysis would make the submission more appealing, the empirical results appear to be sufficient to merit acceptance. The authors are advised to take into account the detailed reviews as well as the clarifications provided in the rebuttal while revising the paper.

Reviews: review: paper_summary: paper_summary: The paper proposes a new regularization technique, Constrained Parameter Regression (CPR), to replace weight decay for training deep learning models. Conventional weight decay penalizes significant weight deviation. However, this can be too restrictive since some layers may need a larger deviation. Instead of applying the same regularization strength for all layers as in weight decay, the paper uses different hyperparameters for each layer, resulting in a more flexible regularization strength. Intuitively, a model with a larger weight $||\theta||$ is allowed more deviation under CPR. The method is theoretically motivated by the augmented Lagrangian method and tested on multiple tasks and benchmarks.
strength_weakness: strength_weakness: * **The regularization strength (upper bound $k$) relies on empirical observations and intuition**. While the overall method is theoretically motivated, the upper bound $k$ setting relies mainly on intuition. Specifically, the paper does not explain why $k\leftarrow R(\theta)$ is a good strategy. Intuitively, this strategy indicates that a larger weight $||\theta||$ should have less regularization. 

* **Computer vision tasks are limited**. For classification, computer vision experiments are conducted on relatively small-scale datasets, such as CIFAR100. The paper should consider using moderate-to-large-scale datasets with well-established benchmarks, such as ImageNet, to make the empirical evidence more convincing. For segmentation, popular benchmarks, such as MSCOCO, could be a better choice to establish the superiority of this regularization.
questions: questions: * Why does the inflection point of the regularization function $\Delta\Delta R$ reflect a saturation of performance? Shouldn't the inflection point of the loss function $L$ be a better fit for this? 

* Could the authors report results on ImageNet? If time is constrained, fine-tuning a pre-trained model, such as CLIP's pre-trained ViT-B [1], to ImageNet is a good choice. 

* Is there any empirical evidence supporting the design choice $k\leftarrow R(\theta)$? It would be great if the authors could discuss this choice in detail. 

* Prior works in hyper-optimization [2,3] show how to optimize hyper-parameters in optimizers differetiably. This could be a valid future direction for optimizing the upper bound $k$. A new work uses hyper-optimization in an optimizer for fine-tuning [4], which shares a similar spirit to this paper at a high level. 

[1] Radford, Alec, et al. "Learning transferable visual models from natural language supervision." International conference on machine learning. PMLR, 2021.

[2] Baydin, Atilim Gunes, et al. "Online learning rate adaptation with hypergradient descent." arXiv preprint arXiv:1703.04782 (2017).

[3] Chandra, Kartik, et al. "Gradient descent: The ultimate optimizer." Advances in Neural Information Processing Systems 35 (2022): 8214-8225. 

[4] Tian, Junjiao, et al. "Fast trainable projection for robust fine-tuning." Advances in Neural Information Processing Systems 36 (2023).
limitations: limitations: There is no potential negative societal impact.
paper_summary: paper_summary: The paper introduces Constrained Parameter Regularization (CPR), a regularization technique for deep learning by dynamically tailoring regularization to individual parameters. CPR sets upper bounds on statistical measures of parameter matrices and reduces the learning into a constraint optimization problem. The authors conduct a series of experiments to evaluate the performance of the proposed method.
strength_weakness: strength_weakness: - The optimization algorithm lacks theoretical analysis on aspects such as convergence, which needs to be further analyzed using more rigorous mathematical language. Additionally, the initialization algorithm designed lacks theoretical support.

- The authors conduct experiments on CIFAR100, which is too small to show the superiority of a training method for deep learning. The authors need to give the results on ImageNET or larger-sized datasets. 

- The proposed methods have more learnable parameters $\lambda$ and $\kappa$. The authoes need to report the addtional computational and memory costs.
questions: questions: Please refer to the comments on weaknesses.
limitations: limitations: The authors have adequately addressed the limitations
paper_summary: paper_summary: This paper illustrates a new training algorithm to improve the weight decay strategy. Instead of giving the same strength to all weights in weight decay, the proposed method penalizes only the elements that are larger than the threshold. Based on extensive evaluation, the proposed method performs better than AdamW/SGD+weight decay.
strength_weakness: strength_weakness: 1. As mentioned in the paper, the computational cost is higher.
2. The evaluation of SGD can be improved since it is well known that the best test accuracy can usually be achieved when using SGD with weight decay on image classification tasks.
questions: questions: See weakness.
limitations: limitations: N/A
paper_summary: paper_summary: This paper presents Constrained Parameter Regularization (CPR) as an alternative to traditional weight decay.
CPR enforces an upper bound on the L2-norm of individual parameter matrices. It frames learning as a constraint optimization problem solved with the augmented Lagrangian method and can be integrated seamlessly with gradient-based optimizers. 
During training, CPR dynamically tailors regularization and reduces the need for hyperparameter selection. 
Empirical results on computer vision and language modeling tasks demonstrate its effectiveness.
strength_weakness: strength_weakness: The general idea of using adaptive regularization does not seem very novel. However, the reviewer does not specialize in this area and won't view this as a significant flaw given CPR’s experimental results.
questions: questions: 1. CPR constrains parameters with an L2 norm upper bound. Will this method impede the model’s learning ability under complex tasks, e.g., very slow convergence or loss of training accuracy? 
2. It would be interesting to investigate CPR’s robustness to noise and distributional shifts. How does CPR perform when the training data is noisy or comes from a different distribution than the test data?
limitations: limitations: N/A
paper_summary: paper_summary: The authors propose a form of regularization which adjusts the regularization strength based on the weights being inside or outside of a certain norm bound. A violation of the bound results in an increasing penalty and coefficient, while conforming to the bound results in a decreasing or zero regularization penalty.
strength_weakness: strength_weakness: - The statement on L123-124 is hard to accept, and I think it needs more explanation. Why if $F(x)$ not suitable for gradient based optimization? It seems that if $F$ and $c$ are both differentiable, then it indeed would provide iseful information to restore the feasibility of an infeasible $x$. Am I missing something here?

- It would be interesting to know what effect the training has on the rubustness to OOD inputs or noise perturbed inputs. Is it possible to run some tests on the trained models related to CIFAR100-C and AdvSQUAD or AdvGLUE (or other suitable language dataset to measure robustness)? I ask this because I suspect that the smaller weight norm maintained throughout training would have the effect of not being overly reliant on particular weights in the model, which would likely result in a model which is more robust to OOD and adversarial inputs.

- The minor runtime overhead mentioned in the paper is actually on the order of 5-6%. The definition of minor is subjective, and I would not consider a 5% increase in runtime minor. I think it would be better to state the actual runtime increase within the main text of the paper to give the reader a better idea of the cost. 

## Minor

- L70: pertaining --> pretraining
questions: questions: Can you add the OOD or adversarial tests mentioned above?
limitations: limitations: The limitations are discussed

Comments: title: Additional ViT/ImageNet results
Official Comment by Reviewer 4Jjm
Thank you for the responses.
comment: Dear Reviewer,

Please find below the additional DeiT[1]/ImageNet results on the DeiT base model that we promised in our rebuttal (since they were still running):

| ImageNet Pretraining | | AdamW | | | AdamCPR | | | |
|----------------------|-|-------|-------|-------|---------|---------|---------|-----------|
| | | weight decay | | | Kappa WS (x lr-warmup) | | | Kappa IP |
| | | 0.005 | 0.05¹ | 0.5 | 1x | 2x | 4x | |
| DeiT-Small (22M) | Top-1 Accuracy (%) | 76.97 | 79.03 | 79.16 | 79.81 | 79.33 | 78.04 | **79.84** |
| DeiT-Base (86M) | Top-1 Accuracy (%) | 76.19 | 78.59 | 80.56 | **81.19** | 79.61 | TBA | 80.95 |

Each experiment runs on 4 A100, the small model for ~15h, and the base model for ~48h. We used the Timm library [2] for all ImageNet experiments and used the configuration described in the DeiT paper:

`train.py --seed 0 --dataset wds/imagenet --aa rand-m9-mstd0.5-inc1 --mixup 0.8 --cutmix 1.0 --reprob 0.25  --model deit_base_patch16_224 --color-jitter 0.3 --train-interpolation bicubic --lr 1e-3 --warmup-lr 1e-6 --min-lr 1e-5 --warmup-epochs 5 --drop-path 0.1 --opt-eps 1e-8 --epochs 300 --opt adamw --weight-decay 0.05 --sched cosine -b 256 --amp --torchscript --pin-mem -j 8 --sched-on-updates`

Unfortunately, even though we directly used this configuration described in the DeiT paper, it did not achieve the same results for the base model as reported in the DeiT paper. We do not know the reason for this, but differences in PyTorch version, etc can make a difference. Also, in our own experiments, we found the optimal weight decay value for AdamW to be higher and not 0.05 as described in the DeiT paper. However, we used the same Timm configuration for CPR and see again that CPR with both kappa initialization, warm start and inflection point, outperforms weight decay. Thanks again for your review and useful thoughts. 
______
[1] Touvron, Hugo, et al. "Training data-efficient image transformers \& distillation through attention." International conference on machine learning. PMLR, 2021.

[2] Wightman, Ross, "PyTorch Image Models.", github.com/rwightman/pytorch-image-models, GitHub repository 2019.
We thank the reviewer for their response and clarification.  
 -  When we read “hyperparameters”, we think of some parameters to tune (beforehand). Since this is not the case we criticized it. We just misunderstood the wording.
 - In principle, the degree of allowed deviation in our method corresponds to the value of $\kappa$ and $\kappa$ depends on the $\kappa$ initialization method. For example, if one uses the $\kappa$ initialization method that has fixed $\kappa$ for all weight matrices (Kappa-K) then it does not depend on the weights $\theta$ and $\kappa \not\propto R(\theta_t)$. If one uses a $\kappa$ initialization method which is dependent on $R(\theta_t)$, like Kappa-WS, then $\kappa \propto R(\theta_t)$ and we agree that the entries of weight matrices with larger $R(\theta_t)$ are allowed more deviation. Or in other words, a model with a larger weight $||\theta||$ is allowed more deviation under CPR when initializing $\kappa$ with a $\kappa$ initialization method which is $\kappa \propto R(\theta_t)$.
Thanks for the detailed response. I have no further questions and I have raised my score to 6.
Thank you for the responses. The authors have adequately answered my questions. As I was already quite positive of the work, I will maintain my current score.
Thanks for the authors' response. 

Most of mu concerns have been addressed. But as the poposed  method is actually a training algorithm/technique for deep neural networks, I think it would be better and necessary to provide some theoretical analysis for it. I increased my rating to 4.
Thank the authors for their new experiments, and thank you for clarifying my misunderstandings of the paper. 

* I want to mention that ''additional scalar variables for each layer'' is what I meant by ''different hyperparameters for each layer''.   

* In the summary, I summarized the method as  " a larger weight is allowed more deviation". This is my logic. The goal is to keep $\||\theta\|| < \kappa$ (line 160), where $\kappa\propto R(\theta)$ (sec.4.3) and $R(\theta) = 1/2\||\theta\||^2_2$ (line 107). In other words, a weight with larger norm $\||\theta\||^2_2$ will have smaller regularization because $\kappa$ is large (a larger upper bound). Hence, it is allowed more deviation. Please let me know where I misunderstood the method. Thanks.
Thank you for your response! I don't have other concerns so I keep my score.