Cross-regularization: Adaptive Model Complexity through Validation Gradients
Authors: Carlos Stein Brito
Venue: ICML 2025 poster (2025)
Keywords: Regularization
Hyperparameter Optimization
Meta-Learning
Cross-Validation
Noise Injection
Deep Learning
Gradient-Based Optimization
Primary area: deep_learning->algorithms
TL;DR: Cross-regularization directly optimizes regularization parameters (e.g. parameter norms, noise, data augmentation) through gradient descent on validation data, eliminating hyperparameters.

Abstract: Model regularization requires extensive manual tuning to balance complexity against overfitting. Cross-regularization resolves this tradeoff by computing validation gradients that directly adapt regularization parameters during training. The method splits parameter optimization - training data guides feature learning while validation data shapes complexity controls - converging provably to cross-validation optima with computational cost scaling only in regularization dimension. When implemented through noise injection in neural networks, this approach reveals striking patterns: unexpectedly high noise tolerance and architecture-specific regularization that emerges organically during training. Beyond complexity control, the framework integrates seamlessly with data augmentation and uncertainty calibration while maintaining single-run efficiency through a simple gradient-based approach.

Cross-regularization: Adaptive Model Complexity through Validation Gradients
Carlos Stein Brito
Abstract
complexity controls through gradient descent on validation data. The method splits both parameters and data: model parameters optimize training loss while regularization pa- rameters receive unbiased gradient information from a sep- arate validation set. This provides continuous feedback about generalization during training. We prove convergence to cross- vross-validation solutions under standard optimization assumptions, making the method theoretically grounded.
Model regularization requires extensive man- ual tuning to balance complexity against over- fitting. fross-regularizztiinn resolves this trade- off by directly adapting regularization parameters through validation gradients during training. The method splits parameter optimization training data guides feature learning while validation data shapes complexity controls- converging provably to cross-validation optima. When implemented through noise injection in neural networks, this approach reveals striking patterns: unexpectedly high noise tolerance and architecture-specffic reg- ularization that emerges organically during train- ing. Beyond complexity control, the framework integrates seamlessly with data augmentation, un- certainty calibration and growing datasets while maintaining single-run efficiency through sim- ple gradient-based approach.
The framework applies naturally across regularization types. For norm-based regularization, we demonstrate automatic discovery of optimal penalty strength without hyperparame- ter search. In neural networks, learnable noise scales reveal distinct regularization phases tied to feature learning and architecture--seeiffe patterns. The method extends to data augmentation policy learning, uncertainty calibration, and online adaptation to growing datasets, demonstrating broad applicability while maintaining single-run efficiency.
2. Background and Related Work
2.1. Validation-Gradient Optimization
1. Introduction
naiidation-gradennt emerged as a principled approach to hyperparameter optimization (Bengio, 2000), computing gradients of validation performance by backpropagating through the training procedure. While theoretically elegant, these methods require either computing inverse Hessians (Larsen et al., 1998) or maintaining the full parameter update history (Maclaurin et al., 2015), limiting their applicability to modern networks.
Regularization in machine learning requires careful tun- ing of parameters that control model complexity. Cross- validation provides only discrete feedback through train- ing multiple models. This applies across different types of regularization from classical norm- norm-based penalties to modern approaches like stochastic regularization and data augmentation. While methods like variational approaches (Molchanov et al., 2017) attempt learn regularization during training, they optimize proxy objectives train- ing data rather than directly targeting generalization perfor- mance. Previous approaches either required inverse Hes- sians (Larsen et al., 1998) or parameter history tracking (Maclaurin et al., 2015), or multiple runs (Jaderberg et al., 2017).
Luketina et al. (Luketina et al., 2016) addressed the scaling challenge by computing validation gradients for hyperpa- rameters from only the most recent parameter update. How- ever, training regularization parameters in both data parti- tions prevented any convergence guarantees. Our work refor- mulates the validation gradient approach by directly optimiz- ing the quantities controlled by hyperparameters weight norms, noise scales and other complexity controls. This eliminates hyperparameters while maintaining the frame- work of iiidaationguidedd optimization. Other works have also utilized validation gradients, for example, to weight source tasks in transfer learning (Chen et al., 2022).
We introduce cross-regularizztion, direct optimization of
NightCitt Labs, Lisbon, Portugal. Correspondence to: Carlos Stein Brito <carlos.stein @ nightcitylabs.ai>.
Proceedings of the nnnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 20255 by the author(s).

---

Cross-reuulariaation: Adaptive Model Complexity through Validation Gradients
2.2. Approaches to Regularization
for A-scaled penalty that depends on the parameters w through p(t=), which we denote as regularization parame- ters and a penalty function R(-). For instance, in L2 reg- ularization, we have p(w) = |l/2 and R(x) = x². With cross-regularization, we demonstrate that the relevant regu- larization parameters p can be directly optimized through gradient descent, eliminating indirect hyparameters X.
Traditional regularization methods like weight decay (Krogh & Hertz, 1992) and dropout (Srivastava et al., 2014) use fixed parameters that must be tuned through cross-validation (Stone, 1974). For modern networks, complexity controls often need to vary across components (Zhang et al., 2021), and other factors like the behavior of normalization layers under distribution shifts (Nado et al., 2020) or the geometry of the loss landscape (Foret et al., 2021) also make manual tuning prohibitive. This has led adaptive methods that at- tempt to learn regularization parameters during training, like Concrete Dropout (Gal et al., 2017) and variational dropout (Molchanov et al., 20. However, these approaches opti- mize proxy objectives on training data rather than directly targeting generalization, and can struggle to learn strong regularization due to their training dynamics.
3.1. General Framework
crossrregularization separates model training into two par- allel optimizations:
Feature learning (model parameters 0) on training data
Complexity control (regularization parameters p) on regularization data
Population Based Training (PBT) (Jaderberg al., 2017) addresses generalization by adapting hyperparameters us- ing validation performance, but requires training multiple parallel models. For noise regularization, Noh et al. (Noh et al., 2017) developed gradient-based noise optimization but without validation gradients. While methods like varia- tional dropout (Molchanov et al., 2017) offer another adap- tive approach by learning noise scales through Bayesian variational inference, they often struggle in practice due to optimization challenges and restrictive prior assumptions.
Throughout this work, we use single validaaaliotionn split, though the method naturally extends to k-fold schemes. We denote the complexity training data the regularization set, instead of the validation set, to clarify that it is used during training.
Concretely, for a model (o) with parameters 0 and reg- ularization parameters p. the training loss for features and validation loss for complexity are:
Larii (0, p) = [[a, Dram
(2)
2.3. Statistical Learning Theory
(0, = E(
Our theoretical analysis builds on Vapnik's structural risk minimization principle (Vapnik, 1999), which formalizes the tradeoff between empirical risk and model complexity. crossiregularizzation provides direct optimization of this tradeoff through validation gradients. For linear models, optimal regularization relates directly to noise-to-signal ra- tios (Hastie al., 2009), while in neural networks, recent work connects regularization schemes to network capacity (Bartlett et al., 2017). The interaction between regular- ization and optimization shapes which solutions gradient descent discovers (Bishop, 1995; Saxe et al., 2014).
These are optimized through alternating updates:
oc+1 = 0 - 170 V Li (tt) Pt+1 = Pt - Up V V (0 (0++1,
(4)
(5)
for learning rates no and 7p- This algorithm directly opti- mizes regularization strength while maintaining the separa- tion between feature learning and complexity control. The validation gradients give p continuous feedback about gen- eralization, unlike the discrete feedback in cross-vale -ros-validattio.
3.2. L2 Regularization Analysis
3. Cross-regularization Method
To demonstrate cross-regularization, we analyze it in the context of ridge regression where the relationship between regularization and solution complexity is well understood. The standard approach controls this complexity indirectly through regularization parameter X:
Consider the standard machine learning setup where we want to minimize test loss while preventing overfitting through regularization. The traditional cross-validation ap- proach involves training multiple models with different reg- ularization strengths X:
w" (A) = arg min [|] - yy12 + [|1w[[1
(6)
min Lval (w" (w=(A)) (A))
(1)
For each X, this yields an optimal solution w (A) with cor- responding norm p² (2) = (A)]le, As À increases, p² (A) decreases monotonically higher regularization enforces
where w² (A) = arg min minin + (R(p(w)))
2

---

Cross-reuularizatio:: Adaptive Model Complexity through Validation Gradients
smaller norms. This means searching over À minimize validation loss:
3.3. Parameter Partition through Gradient Decomposition
min Leal (us (A))
(7)
Cross-eegularizztion separates model parameters from regu- larization parameters. However, for non-smooth penalties like LI norms, no natural parameter split exists. This limita- tion suggests shifting from separating parameters to separat- ing parameter update directions identifying subspaces that control model capacity versus feature learning.
A
is equivalent directly searching over achievable norms p²:
min (w)" subject to || = p*
(8)
This geometrical view leads decomposing parameter gra- dients into orthogonal components:
Rather than searching over X, we can directly optimize the L2 constraint by reparamcterizing the weights into magni- tude and direction:
9 9p 91
9p - 91
(13)
||l||2
(9) For a regularizer R( R(w), the regularization component cap- (10) tures movement in the direction of steepest complexity change, V R(w):
0 = w/ /|lle where |00| =
This naturally separates the optimization problem
9p = ProjvR(g)= V R/
Larin (0. = min ||x00 - vlleainn
(11)
(14)
0.10
(oa (0, p) min ||xX0
yll
(12)
For LI regularization, V.. sign(w) yields:
Theorem 3.1 (L2 Cross Validation Equivalence). Un- der smoothness and strong convexity conditions, cross- regularization, in eqs. 4 and 5, converges to the same solution as optimal ridge regression:
sign(w)
9L1 ايش (g ūL1), شL [sign(w) (w)l2
(15)
Training updates occur in the complementary subspace through g1. maintaining current complexity, while vali- dation updates through 9, modulate regularization strength. This geometric perspective generalizes to any differentiable regularizer by identifying its characteristic complexity di- rection.
0²0' wwat("" (X*)
where (X') is the ridge solution with best validation X The general proof for corvex losses appears in appendix.
Since optimizing validation loss over À is equivalent to optimizing over achievable norms p², we can directly op- timize p² through gradient descent rather than searching over X. This eliminates the indirect hyperparameter while maintaining explicit control over model complexity. The optimization naturally decomposes into two subproblems: an outer loop that optimizes p² using validation loss, and an inner loop that optimizes over the space of parameters orthogonal the norm constraint, In other words, for each fixed p", the inner optimization learns the optimal direc- tion in parameter space while maintaining the prescribed complexity.
3.4. Stochastic Regularization
Modern neural networks rely heavily on stochastic regular- ization, where random perturbations during training improve generalization (Bishop, 1995). These perturbations can take many forms from additive noise to random masking but all require careful tuning of their magnitude, as it deter- mines the strength of regularization: higher values force more robust features at the cost of computational precision. While manual tuning can find single optimal noise level, cross- volsdvaali becomes impractical when different parts of the model require different noise scales.
Rather than solving these nested optimization problems exactly, roossareguliiization performs alternating gradient updates using different losses for each parameter set. While similar in structure coordinate descent, this approach dif- fers fundamentally by using separate objective functions: training loss for direction parameters and validation loss for norm parameters. This split optimization allows con- tinuous adaptation of model complexity during training, providing immediate feedback about generalization perfor- mance rather than waiting for complete convergence at each complexity level.
A simple univariate example illustrates how validation gradi- ents can automate noise tuning. Consider a Gaussian model for regression, p(ylar) = N(wa,, o²), trained on a single data pair (z Vt). Training both w and o on this data leads to overfitting (a - 0), but using a validation point (æ., yv) to tune while training w recovers appropriate uncertainty.
This principle scales to neural networks through noise scale parameters optimized by validation gradients. However, noise typically reduces accuracy unless we treat the model as sampling from a distribution similar to ensemble methods
3

---

Cross-egullrrzation: Adaptive Model Complexity through Validation Gradients
or Bayesian inference. By averaging predictions during validation, the method can discover noise levels that improve generalization. Thus we define the inference at validation using Monte Carlo averaging:
admits the decomposition L(0,p) L(0", ,p² + fz H(2 Hz
x²) )+ R(O, p). where z (0,p)² and the re-
mainder It satisfies [|[(0, p)ll I (10 -0 -11----11--p-
The local quadratic structure implies landscape within this radius, providing the smoothness and effective strong convexity crucial for stable optimization. In non-convex settings like neural networks, stability hinges on managing interactions between model parameters (0) and regularization parameters (p). Our neural network analysis (Theorem 4.3) assumes conditions like Lipschitz continuous validation loss gradients with respect to model parameters. These assumptions ensure bounded coupling, preventing conflicting gradients from destabilizing P optimization, a behavior empirically supported by stable noise adaptation over extended training (Appendix G).
loss
K Σ f(z,c)) ck ~~(((0,)
(z)(()) = (x(((r,))
K
(16)
Leading to distinct objectives for training and validation:
Liaii Dunii B. (x[[(r ())) Lval B(x,,))D...ll (E,, [ (2, +)l,
(17)
(18)
Training with single noise samples maintains the regular- izing effect of randomization, while validation averages multiple samples measure generalization. Note that deterministic validation scheme (e 0) would make the validation loss independent of noise scales preventing their optimization.
Building on local geometric insights, we further establish convergence guarantees for neural networks under practical assumptions:
Theorem 4.3 (Convergence for Neural Networks). Under the assumptions that: (1) the training dynamics of model paramelers O converge to stationury point 0" (p) for any fixed regularization parameters p: (2) the validation loss Lezteal (0, p) is a-strongly convex in for any fixed 6; and (3) the gradients ppfezteat(0 p) are 3-Lipschitz contin- uous with respect to 6, then the alternating optimization scheme of cross-regularization (Eqs. 4-5) converges stationary point (6" (0",p") ,²)).
4. Theoretical Analysis
Four theoretical results establish the core properties of cross- regularization: the alternating updates converge linearly, the optimization landscape admits local analysis, the statistical complexity scales with regularization parameters, and the method achieves cross-validation performance. All proofs appear in the Appendix.
Our first theoretical result establishes convergence of param- eter updates despite the differing training and regularization objectives. Under standard smoothness and strong convexity conditions, the alternating optimization scheme, eqs. 4 and 5, achieves linear convergence:
Thus, Theorems 4.2 and 4.3 establish that cross- regularization mirrors the convergence behavior of fixed- regularization models, even within the non-convex land- scapes of neural networks.
Theorem 4.1 (Linear Convergence). For appropriate learn- ing rates, cross-vegularization, eqs. 4 and 5, converges linearly:
4.1. Statistical Rate
The dimensionality of regularization parameters (k < d) is typically much smaller than model parameters 0 (dimen- sion d). This low-dimensional structure leads to favorable statistical bounds:
10 (1-)) pp
where k > 0 depends problem constants.
Theorem 4.4 (Statistical Rate). The regularization parame- ters converge to the population optimum at rate:
The linear convergence holds despite the distinct objectives for f and p.
Pp - Pimell? S /blog(1/0)
Neural networks and other complex models have noncon- vex loss landscapes. To address this, we first characterize convergence through local geometry.
m
where m is the regularization set size.
Theorem 4.2 (Local Structure). Let L(0,p) be twice contin- uously differentiable in neighborhood of local minimum (0", (0') with Hessian H. If L has Ln-Lipschitz continuous Hessian, then there exists radius min 6b. 2[[
The O(Vk//) convergence rate directly reflects the method's efficiency statistical enror scales with the few regularization parameters (k) rather than the full model di- mension (d).
(1-7)y
(where p= Amin (H), E (0,1) is a constant determining the preserved fraction spectral norm) such that in the ball B,(0" ,p)), the loss
fraction of strong convexity, and [| is the
Cross-regularizattin also matches the performance of stan- dard cross-validation:

---

Cross-regularization: Adaptive Model Complexity through Validation Gradients
Theorem 4.5 (Cross-validation Equivalence). Under mild conditions on the regularizer, reosaaeeguaaizzation achieves the same validation loss as the optimal cross-validation solution.
5.3. Derivative Norm for Splines
Smoothing splines measure complexity through integrated squared derivatives, which for B-spline bases reduces to quadratic form [D with finite difference matrix D. The gradient decomposition handles this matrix norm by projecting updates onto level sets of constant smoothness:
DJ 9p = Projor Dz = DDD (g" D3)
(19)
Both approaches minimize validation loss over identical solution spaces but through different parameterizations. The theory thus confirms cross-regularization'ss practical advan- tages: linear convergence in optimization, statistical scaling with only regularization parameters, and recovery of cross- validation optima.
Training follows I1 to maintain smoothness while valida- tion gradients through gp adapt complexity local feature scales, demonstrated in Figure 1(E). This shows how the gradient decomposition framework naturally accommodates differential regularizers.
5. Norm-Based Regularization Examples
6. Neural Network Regularization
5.1. L2 Regularization
6.1. Interpretable Noise Regularization
To validate the theoretical analysis, we construct synthetic data where regularization is essential for generalization. From small set of independent base features, gen- erate larger feature set by adding correlated variants with Gaussian noise. This design creates groups of highly cor- related features, making the linear system ill-conditioned without regularization, the model can exploit these cor- relations to overfit by assigning large opposing weights to related features.
reuulargguaii allows noise parameters at any granu- larity, from per-unit global. We demonstrate the approach using layer-wise noise as it balances adaptivity to network structure against statistical efficiency while per-unit noise offers maximal flexibility. it introduces (dd² L) regular- ization parameters, and global noise cannot capture archi- tectural differences. Layer-wise noise requires only (L) parameters while respecting natural network boundaries.
The empirical results in Figure I(A-C) show cross- regularization converging to optimally tuned ridge regres- sion through direct gradient descent. The optimization re- veals an inherent tension in regularization training loss increases as the method reduces complexity to improve vali- dation performance, demonstrating how validation gradients guide the tradcoff between memorization and generalization, reducing the generalization gap (difference between train and test accuracy).
rross-regulaiizztion requires only differentiable noise pa- rameters, making it compatible with multiplicative Gaussian noise but not standard Dropout, though both share similar regularization properties (Srivastava et al., 2014; Molchanov et al., 2017). While multiplicative noise offers scale invari- ance, we implement additive noise after normalization to allow for direct analysis of noise structure:
mi w/ h4-1 ân (w Mu)/o
(20)
(21)
5.2. L1 Regularization
hy g(و + t))), c~N(0,
(22)
L1 regularization presents fundamental challenge: the non-differentiaaiiiiy of the LI norm prevents direct norm decomposition. This is particularly important for medical applications where feature selection helps identify relevant biomarkers. On the diabetes progression prediction task, standard LASSO requires evaluating an entire regularization path to find the optimal sparsity level.
for weights w,, nonlinearity g(-), and Layer normalization, chosen over Batch normalization to avoid confounding reg- ularizing effects of batch stochasticity. This design ensures noise magnitudes remain interpretable across layers, though multiplicative noise achieves similar results with less inter- pretable scales (see Appendix). During training, we use single noise samples, switching to averaged samples for validation and deterministic prediction f(x, 0) at test time. While L2 regularization is common technique, we focused on moise-based regularization for our neural net- work experiments given its prominence and effectiveness in regularizing deep neural networks.
Training updates follow I1 while validation updates control sparsity through yt. This allows simultaneous optimization of prediction accuracy and feature selection without manual tuning. Figure I(D) shows cross-regularization automati- cally discovers optimal sparsity levels matching LASSO's best cross-vall validated performance.
5

---

Cross-egullrrzation: Adaptive Model Complexity through Validation Gradients
Bax nag
Mam
Leamiigg
0.0
Tu
-01
Boi
1J485
Cross Reg
2000
6066
3006 6000
10
46
0.02
Feature
Figure 1. Cross-rcgularization results across different norms. (A-C) L2 regularization: Evolution of weight norm showing adaptation to optimal ridge levels, training/validationn loss dynamies, and recovered coefficient values matching coss-validated ridge regression. (D) L1 regularization diabetes prediction: Validation loss versus LI norm shows automatic discovery of optimal sparsity matching LASSO. (E) Spline smoothing: Learned function achieves appropriate complexity without manual tuning.
100
104
12
90
Acouracy
80
Noiee toxa)
(sta)
Noloe tood
20
Fes
1012
50
Epoch
Epoch
Layer
Layer
Figure Noise dynamics in VGG-16 on CIFAR-10 reveal architectural regularization patterns. (A) Cross-regularization matches PBT accuracy (83.7) (8).7% while improving over baseline (76.0%) in single training run. Fixed noise (a 1 in the final five layers) impedes initial learning yet fails to prevent later overfitting. (B) Layer noise adaptation tracks overfitting low during feature learning, then increasing to prevent memorization, when generalization gap appears (around epoch 5), and again when the training accuracy rises again (around epoch 40), after a period of strong regularization. (C) Final noise distribution (0.01-12 σ)) shows unprecedented yet functional noise levels, surpassing PBT's conservative regime (max 2.8 σ) (D) ResNet noise concentrates in early and final layers, with 14.9 in layer 2. Such a profile enforces a bottleneck at points where there is high information flow.
6.2. Adaptive Noise Dynamics
The layer-normalized design provides standardized signal- to-noise ratios, allowing direct analysis of information ca- pacity (Fig. 2). Noise patterns expose computational structure: early layers preserve precise feature detection while deeper layers force increasingly robust representa- tions. When validation accuracy plateaus, indicating poten- tial overfitting, noise selectively increases in layers prone memorization.
Algorithm Cross- rruulaaagula Training
1: Input: Parameters 0, pt datasets Dirain Dreg
2: for each epoch do
3: for (z, 1) ~ Derain do
4: (0,
5: Crainffe,p c).))
Cross- Prossreegularization leads to regime where neural net- works remain functional under surprisingly high noise levels up to 13 standard deviations post-normalization. While these levels are far beyond conventional regularization strengths, their emergence through gradient-basee optimiza- tion of generalization performance offers insights into net- work robustness and capacity.
6: Update 0 using gradient descent
7:
if' everyy reg interval steps then (Kr,yr) ~ Dreg
8:
9:
0 & E f0,p (k) Lreg (1, 3r)
10:
Update using gradient descent 12: end if'
11:
13: end for
Our noise level a can be related to standard dropout rates through signal-to-noise ratio analysis. For layer-normalized activations with unit variance, additive Gaussian noise (0, o²) gives SNR 1/o2.. Standard dropout with rate p, after the 1/(1 p) scaling, yields unit variance noise p 0.5 (equivalent to 1). For comparison, typi- cal dropout rates like 0.2 correspond to ss 0.5. In
14:
end for
6

---

Cross-reuularization: Adaptive Model Complexity through Validation Gradients
contrast, our observed noise level of 12 corresponds to dropout rates of p 0.987 revealing a previously explored high-noise regime that only becomes accessible through gradual adaptation, as directly initializing with such extreme noise levels would prevent learning.
6.3. Additional Architectures
To validate our method's generality across architectures, we apply cross-regularization to ResNet with noise injection (WideResNet- 16-4). The noise adaptation reveals striking pattern: high noise emerges in both early and final layers, with 10.4 a the first layer (a) = 0.9, 02 = 10.4, 1 = 3.3), while maintaining low noise in middle layers (Fig. 2-D).
This high-noise regime, while initially appearing to dramat- ically challenge assumptions about neural robustness, finds an interesting parallel in the neural network pruning liter- ature, particularly the Lottery Ticket Hypothesis (Frankle & Carbin, 2019). Our observed noise levels of a as 13 create a significant information bottleneck. Quantitatively, this level of noise corresponds to an information capacity of approximately 0.004 bits per symbol, which is mathe- matically equivalent to pruning with sparsity of around 99.4%. This aligns remarkably well with findings from the Lottery Ticket Hypothesis, which has shown that VGG-style networks on CIFAR-10 10 can be pruned to similar sparsity lev- els (e.g.. 98%) while maintaining performance, often with sparsity concentrated in later layers, similar to our noise patterns. This convergence of findings from two distinct methodologies- -rri--rrdddieeebaaeeee noise adaptation versus it- erative weight pruning-Suggest that cross-regularization is effectively identifying intrinsic properties of neural infor- mation capacity and architecture-speciff compressibility.
This pattern reflects the network's 's architecture since skip connections allow information to bypass middle layers, the network concentrates regularization early layers that pro- cess all inputs and final layers that integrate features, creat- ing an information bottleneck that can't be bypassed. These results demonstrate how cross-regularization discovers com- plexity controls that reflect network topology. The concen- tration of noise in layers that cannot be circumvented by skip connections provides evidence for how residual archi- tectures shape information flow and regularization require- ments. This finding connects to theoretical work showing residual networks can be viewed as ensembles of paths of different lengths (Veit et al., 2016).
6.4. Parameter Sensitivity Analysis
Empirical studies validate cross reoslaagguls robustness across hyperparameters (Appendix F). While single sam- ples suffice during training, validation requires 3-5 MCMC samples for stable gradient estimation, with performance deteriorating below this range (Fig. 11). Regularization up- dates can be sparse (every 30 steps) with minimal impact on convergence, resulting in only 10% computational overhead. The method maintains performance with extremely small regularization sets - down to 1% of training data (Fig. 12), aligning with our statistical analysis that error scales with regularization dimension & rather than model dimension d (Theorem 4.3). These results establish an efficient configu- ration for practical use.
This adaptive mechanism offers a valuable window into generalization dynamics. Rather than prescribing fixed reg- ularization schedules, noise levels automatically track each layer's capacity to overfit. The gradual emergence of ex- treme noise demonstrates networks can learn robust features even under severe perturbation when guided by validation gradients. Furthermore, these adaptively learned noise lev- els and the resulting model performance remain stable over extended training periods, as confirmed by simulations run for 600 epochs (see Figure 13 in Appendix G).
Comparison with Population- Based Training (PBT) vali- dates these findings- both methods optimize noise using validation performance, but through different mechanisms. While evolutionary search also discovers layer-wise pat- terns but more conservative magnitudes, gradient-based op- timization reveals higher functional noise levels, and with an order of magnitude less computation, requiring only O + K/r)) forward passes versus PBT's O(P)).
6.5. Automatic Uncertainty Calibration
Reliable uncertainty estimation critical for deploying ma- chine learning systems in practice. Medical diagnosis re- quires accurate confidence scores to determine when to defer to human experts, autonomous systems need calibrated un- certainties for safe decision making, and active learning systems rely on uncertainty estimates to select informa- tive samples. Current approaches either require post-hoc corrections (Guo et al., 2017), model-specific assumptions (Gal al., 2017), or separate training objectives (Lakshmi- narayanan et al., 2017). However, real-world applications need models that provide reliable uncertainties immediately upon deployment and adapt these estimates as they continue learning.
K/rr)
Fixed noise injection (a in final five layers) illustrates the limitations of static regularization: too strong initially, slowing feature learning, yet insufficient to prevent later overfitting, exhibiting larger generalization gaps. We also at- tempted a comparison with variational dropout (Molchanov et al., 2017) but found it unsuitable for layer-wise noise adaptation. Despite extensive hyperparameter tuning of the KL weight, the method either collapsed to zero noise or became unstable- precisely the kind of manual tuning our approach aims to eliminate.
Analysis of the neural network experiments reveals that
7

---

Cross-regularization: Adaptive Model Complexity through Validation Gradients
Loggle
Figure 4. Dataset growth adaptation and adaptive augmentation. A: Performance evolution shows successful knowledge transfer at epoch 100 transition from partial to full dataset. B: Total regu- larization strength automatically adapts stronger regularization compensates for limited initial data, then decreases as full dataset provides natural regularization. Vertical line marks dataset transi- tion. C: Evolution of learned augmentation parameters on SVHN. Translation parameters (pixels) and rotation angles (degrees) in- crease early in training before stabilizing. while shear transforma- tions remain minimal. Results demonstrate automatic discovery of dataset-specific invariances favouring rigid transformations.
Figure 3. Uncertainty calibration and ECE evolution. Left: Ex- pected Calibration Error (ECE) over training epochs for Cross- Regularization (X-Reg), compared to final ECE values for an uncalibrated model, Temperature Scaling, Fixed Regularization model (Fixed Reg), and a Deep Ensemble. Right: Reliability diagram comparing these methods. Shaded areas represent 95% confidence intervals. X-Reg maintains strong calibration through- out training and performs competitively against strong baselines.
6.6. Adaptive Regularization Under Data Growth
Training data often becomes available incrementally, re- quiring models to learn from limited data while adapting as samples accumulate. While increasing dataset size improves generalization (Nakkiran et al., 2020), optimal regulariza- tion typically requires manual adjustment with data growth. We study adaptation to growing datasets by training initially on 20% of data before incorporating the full dataset. This mirrors practical scenarios where models must deploy with limited data while preparing for growth.
cross-regularization achieves strong calibration. Figure 3 illustrates this by comparing Cross- ProssReegularization (X-Reg) against several baselines: an uncalibrated model, Tempera- ture Scaling (Guo et al., 2017), a model with manually fixed regularization parameters (Fixed Reg), and 5-member Deep Ensemble. The right panel shows the reliability di- agrams, where X-Reg closely tracks perfect calibration. The left panel demonstrates the Expected Calibration Error (ECE) evolution for X-Reg alongside the final ECE values for the baselines. X-Reg achieves a final ECE of 0.038 (with 79.5% accuracy), showcasing strong calibration in a single training run that is competitive with the 5-member Deep Ensemble (ECE 0.030, Acc. 81.3%), ), and significantly sur- passing the uncalibrated model (ECE 0.163, Acc. 67.4%), Temperature Scaling (ECE 0.057, Acc. 69.6%), and the Fixed Reg model (ECE 0. 175, Ace. 74.7%). Shaded areas in the figure represent 95% confidence intervals. This online calibration represents a fundamental advance, as the model maintains calibrated uncertainties even as it learns, without requiring a separate post-hoc calibration phase.
Results demonstrate automatic adaptation to dataset size (Figure 4-A.B). During limited-data training, the model maintains generalization through elevated regularization. Upon transitioning to the full dataset, regularization ad- justs downward while preserving performance, aligning with theoretical understanding that larger datasets require less explicit regularization. The smooth adaptation sug- gests applications to continual learning settings where data distributions evolve over time.
6.7. Adaptive Data Augmentation
Data augmentation through label-preserving transforma- tions forms a cornerstone for regularization in modern deep learning, yet tuning transformation magnitudes remains a manual, dataset-specific task. Methods like AutoAugment (Cubuk et al., 2019) and Population Based Augmentation (Ho et al., 2019) automate this search through reinforce- ment learning or evolution, but require thousands of training runs. While cross-regularization optimizes model param- eters through validation gradients, applying this approach to data transformations appears problematic random per- turbations of inputs should degrade validation performance, pushing gradient descent to minimize all transformations.
This automatic online calibration represents fundamental advance in uncertainty estimation. The adaptive noise scales learned through validation simultancously control model complexity and shape predictive uncertainty, enabling im- mediate deployment with reliable confidence scores. As the model encounters new data, both its predictions and uncertainty estimates adapt naturally without requiring re- calibration or retraining. This direct connection between validation performance and uncertainty quantification sug- gests cross-regulan regularttatiinnno learns to modulate predictions based on their empirical reliability, providing a practical solution for systems requiring trustworthy real-time uncer- tainty estimates.
Data augmentation fits naturally into our framework by
8

---

Cross-regularizatio:: Adaptive Model Complexity through Validation Gradients
viewing transformations distribution over model con- figurations, analogous noise-based regularization. Each transformed input represents sample from the space of valid variations of the original image. For each type of transformation (e.g., rotation, translation), we define contin- uous magnitude parameters (e.g., for rotation, a learnable maximum angle Otyot defines a range -rra Obyot] from which a specific angle is sampled for each training instance). These magnitude parameters become part of the model's optimizable regularization parameters p. As with noise regu- larization, we maintain the asymmetry between training and validation: single samples during training provide stochastic regularization, while Monte Carlo averaging during valida- tion measures expected performance across transformations. This implementation follows the same alternating optimiza- tion algorithm as noise-based regularization: the transforma- tion magnitude parameters are updated using validation gradients V Lral (obtained through standard backpropaga- tion as these parameters affect the validation loss), aiming to maximize generalization.
7. Conclusion
Cross-regularization demonstrates that model complexity control can be directly optimized through validation gra- dients rather than requiring manual tuning through cross- validation. This optimization enables continuous adaptation of regularization parameters during training, replacing dis- crete hyperparameter search. The method provides a unified optimization framework for different forms of regulariza- tion, from classical norm penalties to stochastic regulariza- tion and data augmentation.
Analysis through direct optimization reveals regularization to be a dynamic process that evolves with model training. In deep networks, regularization requirements reflect archi- tectural structure, with patterns of noise tolerance emerging across layers. These findings connect network architec- ture to optimal regularization strategy, advancing our un- derstanding of how network design influences learning and generalization.
The effectiivneess of inlidatiiongradieen optimization for complexity control establishes regularization as a learnable component of model training. This formulation eliminates manual tuning while offering a principled approach to reg- ularization design that adapts to specific architectures and tasks.
This validation- guided optimization creates a natural equi- librium for the transformation strengths. If transformations too aggressive (e.g., rotating digit '6' by 180 degrees to resemble '9"), they will degrade validation performance, and the gradients will drive down their magnitudes. Con- versely, if transformations are too mild to provide sufficient regularization, the model may overfit to the training data, leading to a higher validation loss compared to a more op- timally regularized state, again guiding the parameters a towards more effective levels. The approach naturally en- courages munsformation-inaaria models that fail to generalize across valid variations of the input. This formulation connects preprocessing parame- ters to stochastic parameters in Bayesian neural networks (Blundell et al., 2015; Wan et al., 2013), as both represent distributions over model configurations optimized through Monte Carlo sampling.
Acknoledgements
This work was developed with the support of NCL agents.
features by penalizing
Impact Statement
This work introduces cross- cross-regulend regularraulii,,,,,nn, method to ad- vance Machine Learning by enabling more automated and efficient control of model complexity for improved gen- eralization. By directly optimizing regularization param- eters, our approach can lead to more robust and reliable AI systems, reduce manual tuning efforts, and potentially lower computational costs. It also offers insights into model adaptation and can enhance trustworthiness through better uncertainty calibration.
Figure 4-C demonstrates this adaptation on SVHN: transla- tions converge to 1-2 pixels, rotations to 3 degrees, while shear transformations diminish. These learned parameters match the intuition that digit recognition benefits from rigid transformations over deformations, emerging automatically from validation gradients. The optimization improves test accuracy from 82.8% to 86.3%, while reducing the gener- alization gap from 16.2% to 7.3%. The smooth parameter trajectories throughout training reveal when different trans- formations become more or less useful for regularization. This optimization of preprocessing transforms through vali- dation gradients points to applications in automating feature extraction and data preprocessing, while quantifying the relationship between datasets and their underlying invari- ances.
The societal implications of cross-regularization are gen- erally aligned with those of broader progress in AI. We believe this method contributes positively by promoting more principled and efficient model development. While the development more capable AI always warrants care- ful consideration, this specific work does not introduce new ethical risks beyond those inherent in advancing Al capabil- ities.
9

---

Cross-regularization: Adaptive Model Complexity through Validation Gradients
References
Jaderberg, M., Dalibard, V., Osindero, S., Czarnecki, W. M.,
Donahue, J., Razavi, A., Vinyals, O., Green, T., Dunning, I., Simonyan, K., et al. Population based training of neural networks. arXiv preprint arXiv: arXiv:1711.09846 2017.
Bartlett, P. L., Foster, D. and Telgarsky, M. Spectrally- normalized margin bounds for neural networks. In Ad- vances in Neural Information Processing Systems, pp. 6240-6249, 2017.
Krogh, A. and Hertz, J. simple weight decay can im- prove generalization. In Advances in Neural Information Processing Systems, pp. 950-957, 1992.
Bengio, Y. Gradient-based optimization of hyperparameters. Neural computation, 12(8):1889-1900 2000.
Lakshminarayanan, B., Pritzel, A., and Blundell, C. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Process- ing Systems, pp. 1402-6413 2017.
Bishop, C. M. Training with noise is equivalent to Tikhonov regularization. Neural Computation, 7(1):108-116, 1995.
Blundell, C., Corneiiee, J., Kavukcuoglu, K., and Wierstra, D. Weight uncertainty in neural networks. Interna- tional Conference on Machine Learning, pp. 1613-1622, 2015.
Larsen, J., Svarer, C.,, Andersen, L. N., and Hansen, Adaptive regularization in neural network modeling. In Neural Networks: Tricks of the Trade, pp. 113-132. Springer, 1998.
Chen, S., Crammer, K., He, H., Roth, D., and Su, W. Weighted training for task learning. In International Conference on Learning Representations (ICLR), 2022.
Luketina, J., Berglund, M., Greff, K., and Raiko, T. Scalable gradient- based tuning of continuous regularization hyper- parameters. In International Conference on Machine Learning, pp. 20022992229920 2016.
Cubuk, E. D., Zoph, B., Mane, D., Vasudevan, V., and Le, Q. V. Autoaugment: Learning augmentation strategies from data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 113- 123, 2019.
Maclaurin, D., Duvenaud, D., and Adams, R. P. Gradient- based hyperparameter optimization through reversible learning. In International Conference on Machine Learn- ing. pp. 2113-2122, 2015.
Efron, B., Hastie, T., Johnstone, I., and Tibshirani, R. Least angle regression. Annals of Statistics, 32(2):407-499, 2004.
Molchanov, D., Ashukha, A., and Vetrov, D. Variational dropout sparsifies deep neural networks. In International Conference on Machine Learning, pp. 2498-2407, 2017.
Foret, P., Kleiner, A., Mobahi, H., and Neyshabur, B. Sharpness-aware minimization for efficiently improving generalization. In International Conference on Learning Representations (ICLR), 2021.
Nado, Z., Padhy. S., Sculley. D., D'Amour, A., Laksh- minarayanan, B., and Snoek, J. Evaluating prediction- time batch normalization for robustness under covari- ate shift. arXiv preprint 20t1::2006.19666 2020. doi: (0.48550/arXivv2200 10963.
Frankle, J. and Carbin, M. The lottery ticket hypothe- sis: Finding sparse, trainable neural networks. In In- ternational Conference on Learning Representations, forum?
2019. URL
Nakkiran, P., Kaplun, G., Bansal, Y., Yang, T., Barak, B., and Sutskever, Deep double descent: Where bigger models and more data hurt. In International Conference on Learning Representations, 2020.
id=rJ1-b3Re,
Gal, Y., Hron, J., and Kendall, A. Concrete dropout. In Advances in Neural Information Processing Systems, pp- 3581-3590, 2017.
Noh, H., You, T., Mun, J., and Han, B. Regularizing deep neural networks by noise: Its interpretation and opti- mization. Advances in Neural Information Processing Systems, pp. $109-51188 2017.
Guo, C., Pleiss, G., Sun, Y., and Weinberger, K. Q. On cali- bration of modern neural networks. In International Con- ference on Machine Learning, pp. 1321-1330. PMLR, 2017.
Saxe, A. M., McClelland, J. L., and Ganguli, S. Exact solutions to the nonlinear dynamics of learning in deep linear networks. International Conference on Learning Representations, 2014.
Hastie, T., Tibshirani, R., and Friedman, J. The Elements of Statistical Learning: Data Mining, Inference, and Predic- tion. Springer, 2009.
Ho, D., Liang, E., Chen, X., Stoica, I., and Abbeel, P. Popu- lation based augmentation: Efficient learning of augmen- tation policy schedules. In International Conference on Machine Learning, pp. 2731-2741, 2019.
Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(56):1929-1953 2014.
10

---

Cross-regularization: Adaptive Model Complexity through Validation Gradients
Stone, M. Cross-validatory choice and assessment of statis- tical predictions. Journal of the Royal Statistical Society: Series B, 36(2):111-133, 1974.
Vapnik, V. N. The Nature of Statistical Learning Theory. Springer-Verlage New York, 2 edition, 1999.
Veit, A., Wilber, M. J., and Belongie, S. Residual networks behave like ensembles relatively shallow networks. In Advances in Neural Information Processing Systems, volume 29, 2016.
Wan, L., Zeiler, M., Zhang, S., Le Cun, Y., and Fergus, R. Regularization of neural networks using DropConnect. In International Conference on Machine Learning, pp. 1058-1066, 2013.
Zhang. C., Bengio, S., Hardt, M., Recht, B., and Vinyals, O. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107- 115, 2021.

---

Cross-reuulariaation: Adaptive Model Complexity through Validation Gradients
A. Theoretical Analysis
analyze the convergence of our alternating optimization scheme where model parameters 0 train on training data while regularization parameters p optimize on a separate regularization set. We show that the split optimization converges despite using different objectives for each parameter set.
A.1. Optimization Analysis
We first establish convergence the alternating gradient scheme under standard smoothness and strong convexity conditions. Theorem A.1 (Linear Convergence). Assume the loss L(0, p) satisfies:
1. smooothnesss in both arguments:
[T(00p))- VI(0', s 8(110 0'1l + 1p p'1l)
2. pu-strong convexity in 0 for any fixed p
3. a-strong convexity in p for any fixed 0
Then the alternating updates:
Oc+1 = 0: - V VeLoai (0r, Pt)
(23)
Pc+1 = P: - 71 pl p1))1.
(24)
with learning rates:
S 1/B
(25)
?)p S min(1/B, pma (4B2))
(26)
converge linearly:
|10g - 112 + llpe p²11² s (1 -x) 10 + 1 p²113)
where K= min(ume/22 amp..
Proof. The proof analyzes each update step separately and then combines them to show overall convergence. Let 0* (p))
denote the minimizer of I(-,P) for fixed p.
Step 1 (f-update): By 3-smoothness and optimality of 6t (pt):
Lurin (0 S Luain pt) (ne - 3//3/2)1 VaLuain!
By p-strong convexity in 0:
Luaiall 2 2p1 (orainnnn (fc, pt) Lurain (8" (pt): pc))
Combining with n1 I 1/8 gives:
|0++1 (1- (p=))
(I)
Step 2 (p-update): By similar arguments and smoothness of V VpLng with respect to 0:
|| - 1-11 < (1 amp))lpr - p0112 + 8 no 110 (p:))
(2)
Step 3 (Combined Progress): Let V,= 110, 0² (px))12= + 11pp: 0²112. Combining (1) and (2):
V & [(1 (mr)) + 3 n 11 -0x()))) 112 + (1 amp)lle: p=112
12

---

Cross-egullrization:: Adaptive Model Complexity through Validation Gradients
The condition "1 /uee ensures the progress term dominates the coupling cost:
(1 - pmmo + 1p I 1 - wooo
Therefore:
Vi+1 $ (1--)] ~1-
where & min(ure/22 amp) > 0
Iterating gives the final bound:
Vi $ (1 Vo
Note that the condition on learning rates arises from our chosen update order, suggesting this asymmetry is not fundamental to the method but rather an artefact of the ordering.
This result establishes that despite optimizing different objectives for and p. the alternating scheme converges linearly to the optimum.
A.2. Local Approximation Analysis
Let L(0.p) p) be twice continuously differentiable in neighborhood of a local minimum (0", p²) with Hessian H satisfying Assumption (from main text, referring to positive definiteness and bounded coupling). Since (f",p") a local minimum, have VL(O", ,p) = 0. By Taylor's theorem with remainder:
=0, ,²) + (0-02)
0 ---
0 (0-0-) + R(0,p)
p-p-) p²)
(27)
For the function with Lipshiisc continuous Hessian, the remainder term is bounded by:
LH p²)1 --)'-'''''
||((0, p)1l 6
- -
(28)
By the assumptions for Theorem 4.2 (main text), H is positive definite with Amin (H) 2 p 0, giving us:
0" (--p') p²
T H
0 - 0r) {0-00
2 (l - 0r 0-
(29)
p - p²)
For the quadratic approximation to maintain yu--stongg convexity (for some y (0, 1)), stated in Theorem 4.2, require:
|R(0,p)] < (1 (1-3))' 1100 p
(30)
Combining with our bound on ft:
LH 100 0- p²)||3 (1 (1-7)1 (0 0-, p²)112
(31)
2
This yields |[(0 - 0%, (---p")] p²) <
3(1 (- The radius r in Theorem 4.2 established by taking the minimum of this and another constraint related to Hessian approximation validity, r min b. (177)"" Within this radius, the remainder term satisfies the condition Theorem 4.2, ensuring that the function is effectively yu--ttonglyy convex, preserving local optimization properties.
Lx
13

---

Cross-regularization: Adaptive Model Complexity through Validation Gradients
A.3. Statistical Analysis
Having established optimization convergence, we analyze how sample size affects the solution quality. The main result is that the statistical error scales with regularization parameter dimension le rather than model dimension d.
Theorem A.2 (Statistical Rate), Assume:
1. Loss bounded: [L(200,p] S B
2. Population risk R is c--strongyy convex in
3. Parameter spaces compact: θ C RR, PC R
Then with probability at least 1 - 8:
4B²
[klog(11/))
Pl
Piruell
a²
where Piree minimizes the population risk and m is the regularization set size.
Proof. By standard uniform convergence over the k-dimensional space P:
k log(110)
sup PEP
Rm (0" (p). R(O (p),p) & BV
m
Strong convexity then gives:
2 Pl Pineel 1 2 (R(pm)) R(prve))
4B2
k: log((1 1/6) m
a
a²
This O()k/m) rate explains why small regularization sets suffice the error depends on regularization parameter dimension Is (typically number of layers) rather than model dimension (total parameters).
A.4. Cross-validation Equivalence
We show our direct optimization achieves the same solution standard validatali.t.nn.... The proof relies on showing that both methods optimize equivalent solution spaces through different parameterizations.
Theorem A.3 (Cross-validatte Equivalence). cross-validation
Let (() be complexity measure. For any X > 0, let 0x be the solution from
0x arg min{ Ciroin (0) + *f(p)}
Assume X is monotonic in the solution Then cross-regularization achieves the same validation loss as cross-valie voss-validation:
min (rar (0x) min (e())((p))
Proof. By monotonicity of À in p. we can rewrite cross-validation optimization:
min Eval = min (fx)
where 0x = arg ming (Carain +f(p))
This is equivalent to direct optimization of p in cross-rquularization
min Lea
Therefore both methods optimize the same objective over the same solution space, just parameterized differently through or direct optimization p.
14

---

Cross-reuulariaatoo:: Adaptive Model Complexity through Validation Gradients
A.5. Proof of Neural Network Convergence Stability
The theorem regarding the convergence of cross-regularization for neural networks (Theorem 4.3) is established under the following assumptions:
1. The training dynamics of 0 converge to a stationary point 0* (p) for any fixed
2. The validation loss (0, p) is o-strongly convex p for any fixed
3. The gradients p0, p) are 3-Lipschitz continuous with respect to 0.
The proof, summarized below, demonstrates that for sufficiently small learning rates 7p the error terms for both model and regularization parameters converge to zero:
Proof. Define the error metrics:
EI( 10, 0 (pt)l E1(1) = 1|: p² 1²12
(32)
(33)
By assumption 1, f converges to (p) for any fixed o with some convergence rate. When p is updated, this introduces a perturbation in the optimization landscape. We can model this as:
Eb + 1) 5 . Eo(() + 6- |l - Pell
(34)
where captures the convergence rate of 0 and 8 is coupling constant that measures how changes in affect the optimization of 0.
For the updates, we use strong convexity (assumption 2) and Lipschitz continuity (assumption 3):
||tt+1 p²112 < pp v (o - p²11
(35)
llp 7, v (0' (pt), Pt) p² 112
V P0'al (pt),
(36)
< - E, (1) 8 (l)+
(37)
Where we used the fact that for a-strongly convex functions with step size Th < 2, the update contracts the distance to the optimum by a factor of (1 - a))
Combining these inequalities, we get linear system:
(t + +1] Et + 11)
On 1n
Et(t)
& 12
(38)
Let's denote this matrix as M. For the system to converge, we need all eigenvalues of M to have magnitude less than The eigenvalues are the solutions to:
det( - - XI) =
(39)
For sufficiently small nh. we can ensure that both eigenvalues have magnitude less than 1, as the diagonal terms y and (1 - (p) are dominant. Specifically, when:
2
(1 71 ax
np min
(40)
8,32
Then both error terms converge to zero as oo, establishing convergence to a stationary point (0", p²))
15

---

Crossregularization: Adaptive Model Complexity through Validation Gradients
B. Norm based examples
B.1. L2 Regularization Details
We validate L2 cross-rggulatization on synthetic data specifically designed to demonstrate the importance of regularization. From 5 independent base features, we create 100 total features by adding Gaussian noise (a 0.11 to copies of the base features. True coefficients alternate between +I and for features derived from the same base feature. This design creates groups of highly correlated features, making the linear system ill-conditioned. Without regularization, the model can exploit these correlations to fit noise by assigning large positive and negative weights to redundant features.
The regulaatgulii model uses stochastic gradient descent with learning rates 0.01 for both feature learning and L2 regularization. Training runs for 6000 steps with regularization starting at step 3000 to demonstrate adaptation. For baseline comparison, we fit ridge regression models across 1000 ggarithmiicallyssacee values from 10 (3 to 10².
B.2. L1 Regularization Details
We evaluate I.1 cross-regularization on the diabetes regression dataset (Efron et al., 2004). The dataset consists of 442 patients with 10 physiological features. Data is standardized and split 80/20 into train/validation sets.
The cross-regularizatioo model uses stochastic gradient descent with momentum 0.99, learning rate 0.0005 for feature learning and 0.01 for LI regularization. Training runs for 2000 epochs with batch size 512. For baseline comparison, we fit LASSO models across 50 logaritmmicall--spaeed values of À from 10-2.5 to 1. Extended results are shown in Fig. 5.
0.5
Train Loss
LASSO
0.5
Req
0.8
0.8
0.2
0.1
LASSO
0.5
Cross-Rey
50
100
150
200
0.0
0.5
1.0
1.5
0.0
0.5
10
15
Steps
Norm
Figure Extended cross-regularization results on diabetes dataset. (A) Training dynamics demonstrating stable convergence despite non-smooth LI penalty. (B) Validation MSE versus LI norm comparing ressaaagulai and LASSO. (C) Coefficient paths showing similar sparsity patterns were discovered through direct optimization rather than cross-validation.
B.3. Spline Regularization Details
We generate synthetic data from the function:
v(x) = sin + 0.5 0.5 <(x))
with hetcroskedastic noise e(x) ~ No, (0.5 + =)) This provides a challenging test case with both smooth and sharp features. We sample 20 points with gaps to evaluate interpolation.
The B-spline model uses cubic basis functions with 15 knots. Second derivatives are computed analytically and normalized by trace. Training uses SGD with learning rate 0.3 for both parameters and smoothness, Extended results shown in Fig. 6.
16

---

Cross-egulariaation: Adaptive Model Complexity through Validation Gradients
Best Loss
0.020
Reg. Loss
0.4
0.030
0.2
-2
7 findose
End leder
0.000
b+20-33
Epach
Figure 6. Extended Spline Results: (A) Validation loss evolution demonstrates convergence to cross-validation performance. (B) Smoothness norm adaptation reveals automatic discovery of appropriate complexity. (C) Comparison fitted functions across different smoothness levels.
C. Neural Network Implementation
Cross-regularizatioon requires only modifications to standard neural network training: adding noise parameters after normalization layers and implementing validation updates. The method be implemented in few lines of code, with no custom optimizers complex architectural changes needed.
C.1. Model Architecture
Each layer block in neural networks applies normalization (LayerNorm BatchNorm) followed by learnable noise:
Linear Convolutional layer
Normalization (without affine)
Additive noise with learned scale or exp(pt)
ReLU activation
C.2. Regularization Class
The algorithm requires only the separation of parameters and datasets. We implement this through abstract Regulari zedModel class, which must specify the set of regularization parameters The remaining parameters are considered training parameters. This abstraction facilitates models to implement different forms of regularization while maintaining the same training procedure.
C.3. Training Protocol Optimization settings:
Adam optimizer
Learning rates: 10-4 (model), 10-1 (noise)
Initialization: log a -3
Batch size: 512
Training epochs: 100
C.4. Computational Analysis
Computational requirements for training steps:
Standard train-validation split: O + )) forward passes, for v validation split %, (1-v) test split %, and one validation run per training epoch. For v=10%: O(111TT).
croslarogulaiizzatin:: (TT(1 + {)) forward passes for K MCMC samples and regularization update interval r. For K=3 and r=30: O(1.TT).
17

---

Cross-regularization: Adaptive Model Complexity through Validation Gradients
M-fold cross-validation: O(MT) forward passes.
PBT: O(PT) forward passes. Memory usage: O(PM) for model size M and population size P.
D. Population-Based Training
10
Noise (std)
10a
10-1
10-²
20
40
60
80
100
Epech
Figure 7. Noise dynamics for PBT. Noise evolution in PBT exhibits more volatile adaptation compared to the smooth emergence in crous-rogularizzttin, reflecting the discrete nature of evolutionary updates.
PBT was implemented with a population size of 20 models, using validation accuracy for selection and random perturbation factors in [0.8, 1.2] for noise parameter updates. The population size was chosen to balance computational cost with optimization stability.
The evolutionary optimization in PBT leads to distinet training dynamics compared to cross-regularizatoon'ss gradient-based approach. Figure shows the more volatile noise adaptation, with sudden changes noise levels corresponding to selection and mutation events.
E. ResNet Experiments
WideResNet- 16-4 implements residual blocks with dual 3x3 convolutions and batch normalization across three stages. The architecture's s skip connections create an ensemble of paths with varying depths (Veit et al., 2016), enabling multiple information routes. Our experiments reveal two findings: first, despite injecting noise levels comparable to VGG-16, the network maintains performance through its skip connections; second, the training dynamics (Figure 8) show distinct noise adaptation phases an initial increase at epoch 5 when validation plateaus, followed by second rise at epoch 40 training accuracy approaches 100%. This pattern mirrors our CIFAR-I0 results, reinforeing the connection between generalization gaps and noise adaptation.
132
M wymammll Mnn
10
90
Actiney
63-
Vode caud
(od))
20.2
60
(No
60
50
102
20
60
60
120
10111235
Ezoch
Epech
Lxyer
Figure 8. ResNet training dynamics: (a) Generalization gap emerges at epoch 5 and training nearly overfits at epoch 40, triggering corresponding noise adaptations. (b) Layer- wise noise evolution follows VGG-16 16 pattern, increasing validation plateaus and overfiting. (c) Final noise concentrates in layers 1, 2 and 14, exceeding 10 standard deviations without compromising performance.
E. BatchNorm and Multiplicative Noise
The standard ResNet implementation with BatchNorm and multiplicative noise exhibits similar qualitative dynamics, though with less interpretable scales (reaching over 3000 standard deviations)
18

---

Cross-reuularization: Adaptive Model Complexity through Validation Gradients
3000
107
2540
1000
80
Test 3X-
135
2000
Train Bug)
10-1
Best Ikn Reg)
Thein (No Reg!
10
500
50
150
30131223
Figure Training dynamics ResNet with BatchNorm multiplicative noise.
F. Parameter Sensitivity Analysis
Figures 10, and 12 show the detailed plots for the systematic method analyses.
180
Bag rtanca
82
Noice
72-
53
25
45
101312
25
Esoch
Layer
Epocch
Figure 10. Effect of update interval. Consistent noise patterns emerge despite different update frequencies. Smoother evolution with frequent updates but comparable convergence.
100
90
Nodce
0-
70
62
-4-
Teahs
20
60
00
100
2
101112
29
40
60
100
Esoch
Layer
Spach
Figure 11. Impact of MCMC samples for regularization, Accuracy with different sample counts shows diminishing returns beyond 3-5 samples.
90
Neainey
80
Tep Salt: 63
Nose cadt
Noice
70
60
20
40
60
10
103
0
25
40
10
60
100
Esoch
Layer
Spoch
Figure 12. Sensitivity to regularization set size. Performance remains stable down 1% of training data. This efficiency stems from the low-dimensional nature regularization parameters compared model weights.
G. Extended Stability Simulations
To further demonstrate the stability of the learned regularization parameters neural networks, Figure 13 shows the evolution of laser-wise noise sigmas (a1) over an extended training period of 600 epochs. The noise parameters stabilize
19

---

Crossregullrization: Adaptive Model Complexity through Validation Gradients
after initial adaptation phase, confirming the empirical robustness of the adaptive mechanism discussed in Subsection 6.2.
Noise (log std)
100
200
300
400
500
600
Epoch
Figure 13. Evolution of layer-wise noise sigmas (ar) over 600 training epochs for VGG-style network on CIFAR-10. The regularization parameters demonstrate long-term stability after an initial adaptation period.
20


Reviews:
[{"review_id": "fzECqnr2Zd", "reviewer": "Reviewer_hz4X", "summary": "This paper proposes a cross-regularization method that eliminates manual hyperparameter search by directly optimizing weight norms. The approach orthogonally decomposes weight parameters into two complementary components, transforming the optimization problem into two subproblems solved through an alternating optimization strategy.(1) Specifically, during training phases, it maintains fixed model complexity while optimizing parameter directions, whereas (2) validation phases optimize regularization magnitudes. Theoretical analysis demonstrates that this method achieves equivalence to standard cross-validation under certain conditions, while exhibiting faster convergence rates and superior calibration performance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful assessment and constructive feedback. Below, we address the main concerns raised:\n\n### Recent Literature\nWe agree that our literature review requires updating. In the camera-ready version, we will incorporate recent advances in adaptive regularization and complexity control, including:\n\n- Foret et al. (2020) \"Sharpness-Aware Minimization for Efficiently Improving Generalization\"\n- Nado et al. (2020) \"Evaluating Prediction-Time Batch Normalization for Robustness under Covariate Shift\"\n- Chen et al. (2021) \"Weighted Training for Cross-Task Learning\"\n\n### Uncertainty Calibration Comparisons\nWe have implemented comparisons with established calibration methods:\n\n1. **Temperature Scaling (Guo et al., 2017)**: Our preliminary results (https://tinyurl.com/3mtacsu9) demonstrate that cross-regularization maintains better calibration throughout training compared to temperature scaling with fixed dropout (σ=0.1 in all layers), while achieving higher performance: \n\n| Model | Accuracy | ECE | MCE |\n|-------|----------|-----|-----|\n| X-Reg | 79.49 | 0.0376 | 0.0993 |\n| X-Reg CI | | (0.0325, 0.0454) | |\n| Uncalibrated | 67.40 | 0.1628 | 0.2810 |\n| Uncalibrated CI | | (0.1552, 0.1706) | |\n| Temperature Scaling | 69.63 | 0.0571 | 0.1944 |\n| Temperature CI | | (0.0500, 0.0649) | |\n\n2. **Deep Ensembles (Lakshminarayanan et al., 2017)**: This comparison is currently underway and will be included in the camera-ready version with appropriate statistical significance tests and confidence intervals.\n\nThese analyses will demonstrate how cross-regularization provides well-calibrated uncertainties without requiring separate post-hoc calibration steps or ensemble overhead, addressing a key limitation in current practice.\n\n### Additional Hyperparameter Tuning Comparisons\nWhile we compared with PBT and fixed dropout, we found that Variational Dropout frequently diverged without implementation modifications that would bias the comparison. In the revised version, we will include:\n\n1. Comparisons with Concrete Dropout (Gal et al., 2017)\n2. Analysis of noise dynamics for all methods (where convergent)\n3. Computational efficiency metrics (wall-clock time and memory usage)\n\n### Cross-Validation Equivalence Verification\nRegarding the comment on Method 3.3 (Parameter Partition through Gradient\nDecomposition) verification, we note that Theorem 4.4 establishes the theoretical equivalence. Our empirical validation in Section 5.1 (Figure 1A-C) demonstrates that cross-regularization converges to the same optimal solution as cross-validated regression, confirming the theoretical result. We will clarify this connection in the revised paper.\n\n### Large-Scale Dataset Validation\nWhile our primary contribution is methodological—introducing and analyzing a novel regularization framework—we acknowledge the value of validating on larger-scale datasets. Our current experimental suite (CIFAR-10, diabetes dataset) demonstrates the method's applicability across different regularization types and model architectures, though we recognize that larger datasets would provide additional validation. We are exploring experiments on ImageNet with ResNet-50, focusing on:\n\n1. Confirming that the theoretical and empirical advantages scale to larger datasets\n2. Identifying any emergent layer-wise regularization patterns specific to deeper architectures\n3. Quantifying computational advantages over traditional hyperparameter tuning at scale\n\nThese extensions would complement our methodological contribution rather than being essential to validating the core approach.\n\n### Neural Network Theory and Stability\nWe have extended our simulations to 600 epochs (https://tinyurl.com/2s45b236), demonstrating empirical stability of the regularization parameters. For the theoretical foundation, we have developed a new result showing that, under assumptions on regularization parameter behavior, convergence depends only on the standard neural network optimization properties. The full proof is available at: https://tinyurl.com/4ckfb9ec.\n\n### Paper Structure\nWe appreciate the feedback on organization. In the revised version, we will:\n\n1. Merge the Introduction with Background sections\n2. Consolidate the application sections (7-9) into a single \"Applications and Extensions\" section\n3. Provide clearer transitions between theoretical development and empirical validation\n4. Maintain a consistent narrative flow from problem formulation to practical applications\n\nWe thank the reviewer for their detailed feedback, which has highlighted important areas for improvement while acknowledging the novel contributions of our cross-regularization framework.", "claims_and_evidence": "Most of the claims made in the submission is supported clearly. For example: (1) Detailed proofs in the Appendix (Theorems A.1–A.6) under smoothness and strong convexity assumptions. Linear convergence is demonstrated for convex cases. (2) Layer-wise noise patterns (Figure 2) and comparison with PBT demonstrate effectiveness. Extreme noise levels are shown to function without collapse.\n\nOne negative points should be noted: While reliability diagrams (Figure 3) show improved calibration, no comparison is made to state-of-the-art methods (e.g., deep ensembles, temperature scaling). The ECE metric is reported but lacks statistical significance tests.", "methods_and_evaluation_criteria": "1. Cross regularization separates model parameters (optimized on training data) and regularization parameters (optimized on validation data), utilizing validation gradients to directly adjust model complexity. This design is theoretically innovative, avoiding the computational overhead of traditional cross-validation while providing continuous generalization feedback.\n\n2. The diabetes dataset (regression) and CIFAR-10 (classification) serve as standard benchmarks, ensuring the comparability of experimental results. However, the study lacks validation on larger-scale datasets (e.g., ImageNet), which constrains the generalizability of the conclusions.", "theoretical_claims": "The interplay mechanism between data and model uncertainties via dual-level structures needs rigorous mathematical justification.", "experimental_designs_or_analyses": "The paper lacks validation on larger-scale datasets (e.g., ImageNet), which constrains the generalizability of the conclusions.", "relation_to_broader_scientific_literature": "The method proposed in the paper is related to methods like variational dropout[1] and Concrete Dropout[2], but differs by optimizing regularization parameters directly using validation gradients. \n\n[1] Variational dropout sparsifies deep neural networks.\n\n[2] Concrete dropout.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n1. Novel Idea: The paper introduces cross-regularization, an adaptive method that removes the need for manual tuning of regularization parameters.\n\n2. Broad Applicability: It works with various regularization techniques, including noise-based regularization, data augmentation, and uncertainty calibration.\n\n3. Strong Experiments: The method is tested on different regularization forms and architectures, showing good generalization, adaptability and calibration performance as well.\n\nWeaknesses:\n1. The paper mainly compares with PBT, but not with other hyperparameter tuning methods.\n\n2. The related work is comprehensive but somewhat outdated. It is recommended to include more recent studies on regularization; for example, most of the references are from around 2016.\n\n3.The structure of this paper may not be clear enough; for example, the paper contains 10 sections.\n\n4. Method 3.3 cross-validation equivalence is not verified while 3.2 is well verified.\n\n5.The study lacks validation on larger-scale datasets (e.g., ImageNet), which constrains the generalizability of the conclusions.", "other_comments_or_suggestions": "1. Theoretical analysis relies on convexity assumptions, while experiments on neural networks lack ablation studies (e.g., no comparison to variational dropout or concrete dropout). The claim of \"local convergence\" (Theorem 4.2) is not empirically validated for deep networks.\n\n2. While reliability diagrams show improved calibration, no comparison is made to state-of-the-art methods (e.g., deep ensembles, temperature scaling). The ECE metric is reported but lacks statistical significance tests.", "questions_for_authors": "Please refer to the weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "D18C3Uv2yS", "reviewer": "Reviewer_iamU", "summary": "This paper designs an approach to tune model regularization parameters automatically. Instead of relying on cross-validation, which requires training multiple models, the proposed method adapts regularization parameters dynamically by using validation gradients during training. This approach alternates between feature learning (optimizing model parameters using training data) and complexity control (optimizing regularization parameters using validation data). This work proves convergence to cross-validation optima and shows that it is applicable across different types of regularization, including norm-based penalties, noise injection in neural networks, and data augmentation. Experimental results demonstrate its effectiveness in discovering architecture-specific regularization patterns, improving uncertainty calibration, and adapting to growing datasets.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your review and address your concerns below:\n\n### 1. How We Apply Gradient Descent on Regularization Parameters\n\nOur method works by making regularization parameters explicit and directly optimizable:\n\n**L2 Regularization Example:**\n1. We rewrite weights as `w = ρθ` where `||θ||₂ = 1` and `ρ` is the norm\n2. Training updates: `θₜ₊₁ = θₜ - ηθ∇θL_train(θₜ,ρₜ)` - optimizing on training data\n3. Regularization updates: `ρₜ₊₁ = ρₜ - ηρ∇ρL_val(θₜ₊₁,ρₜ)` - optimizing on validation data\n\nThis makes regularization strength a parameter receiving direct gradient updates, not a hyperparameter requiring grid search.\n\n### 2. Neural Network Convergence\n\nWhile our theoretical guarantees in Section 4 apply to convex settings, our neural network experiments show consistent convergence. The optimization remains stable even with high noise levels, and the validation accuracy improvements are robust across architectures.\n\nWe've extended our simulations to 600 epochs (https://tinyurl.com/2s45b236) showing stable noise patterns over time. Our additional theorem (https://tinyurl.com/4ckfb9ec) proves that cross-regularization doesn't introduce instability beyond standard neural network training.\n\nThe stability of our method is evidenced by its outperforming fixed regularization, without the divergence issues sometimes seen in other adaptive methods like variational dropout.\n\n### 3. Application to Other Regularization Techniques\n\nCross-regularization applies to any regularization with:\n- Parameterizable strength\n- Differentiable validation performance\n\nFor data augmentation (Section 9), we parameterize transformation magnitudes and optimize them through validation gradients. Similar approaches could apply to mixup (parameterizing α) or label smoothing (parameterizing smoothing strength).\n\nWe focused on norm-based and noise-based regularization as they provide clear demonstrations of our approach. Extensions to non-differentiable techniques would require additional work beyond the scope of this paper.\n\n### 4. Distinction from Bilevel Optimization\n\nOur approach differs fundamentally from traditional bilevel optimization:\n\n1. We update parameters directly through validation loss, not through the optimization trajectory\n2. We maintain separate parameter spaces for model features versus complexity\n3. We don't require approximations of second-order derivatives or parameter history\n\nThis distinction makes our method implementable with standard optimizers and applicable to large models where traditional bilevel methods become computationally intractable.\n\n### 5. Experimental Details\n\nOur experimental implementation follows the algorithm in Section 3:\n\n- **L2/L1 experiments**: Alternating SGD updates between model parameters and regularization parameters\n- **Neural networks**: Layer-wise noise parameters optimized every 30 training steps using 3-5 MCMC samples\n- **Augmentation**: Parameterized transformations updated through validation-based MC averaging\n\nThe code requires only ~20 lines beyond standard training loops, making implementation straightforward for most existing frameworks.\n\n### 6. Relation to Transfer Learning\n\nWe thank you for noting the connection to Chen et al. (2022). Both approaches use validation gradients but for different purposes: their work for weighting source tasks in transfer learning, ours for optimizing regularization parameters. We will include this reference.\n\nWe appreciate your feedback and will address these points in our revision.", "claims_and_evidence": "- This work proposes an algorithm that alternates between updating model parameters via SGD and updating regularization parameters. Yet, in the formulation (Equation 1), the regularization parameters are defined as a function $\\rho$. How does the algorithm apply the gradient descent on the function $\\rho$ in Equation 5. \n- The proposed method applies to differentiable functions with regard to $\\rho$. How can the algorithm extend to a broad range of regularization methods, such as mixup, label smoothing, and sharpness-aware minimization?", "methods_and_evaluation_criteria": "- The method is close to bilevel optimization algorithms using second-order gradients. How does the method compare to these methods? For example, \"Weighted Training for Cross-Task Learning\" by Chen et al. 2021. \n- It would be better to summarize the experimental setup of each section, to better understand the experimental settings.", "theoretical_claims": "The paper establishes four theoretical results: (1) alternating updates for model and regularization parameters converge linearly, (2) local optimization guarantees stability through smooth loss landscapes, (3) statistical error scales with the number of regularization parameters rather than model parametersy, and (4) cross-regularization achieves performance equivalent to optimal cross-validation. \n- Yet, how the results extend to non-convex settings, such as optimizing deep neural networks, is unclear.", "experimental_designs_or_analyses": "Please see Methods and Evaluation Criteria.", "relation_to_broader_scientific_literature": "How does the work compare to existing bilevel optimization methods, in terms of computational complexity and generalization performance?", "essential_references_not_discussed": "Please see the discussion above.", "other_strengths_and_weaknesses": "No", "other_comments_or_suggestions": "No", "questions_for_authors": "No", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "j4KHk3cSod", "reviewer": "Reviewer_8ms6", "summary": "A method for alternating optimization of regular parameters $\\theta$ and regularization hyperparameters $\\rho$ is presented: the training data is split into training and regularization sets, and $\\theta$ and $\\rho$ are alternately fit using each split respectively. The method is proven to converge for convex minimums and around a neighbourhood of non-convex minima, and has convergence in $\\rho$ at a rate proportional to the square root of the number of regularization parameters. Empirical results in the following applications show efficacy: 1) $L^2$, $L^1$, and spline fitting regularization for convex problems, 2) adaptive noise regularization of neural networks, 3) other effects for neural networks including uncertainty calibration, data growth, and adaptive augmentation.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your insightful review and constructive technical feedback. Your points have significantly improved the paper's theoretical foundations and connections to existing literature.\n\n### Parameter Independence and Bounded Coupling\n\nYou correctly identified that our theory requires some degree of parameter independence. Rather than complete independence, our approach requires bounded coupling:\n\n$$\\|H_{\\theta\\rho}\\| \\leq \\beta\\sqrt{\\lambda_{\\min}(H_{\\theta\\theta})\\lambda_{\\min}(H_{\\rho\\rho})}$$\n\nThis condition prevents contradictory optimization objectives from causing instability when training and regularization gradients conflict. Our empirical stability (see extended simulations: https://tinyurl.com/2s45b236) suggests this condition is satisfied in practice, even for neural networks where the noise parameters serve a fundamentally different functional role than weight parameters.\n\n### Neural Network Theory\n\nWe've addressed your concern about local convergence with two theoretical advances:\n\n1. A refined local structure theorem (https://tinyurl.com/33d2sye4) that establishes precise bounds for the radius of convergence:\n$r = \\min\\left(\\frac{\\mu}{6L_H}, \\frac{(1-\\gamma)\\mu}{2|H|}\\right)$.\n\n2. We've developed a new result showing convergence under practical assumptions for neural networks (https://tinyurl.com/4ckfb9ec) that doesn't require global convexity. Our guarantees now only require that:\n- The model parameters converge for fixed regularization\n- The validation loss has local convexity in regularization parameters\n- The gradient coupling satisfies a Lipschitz condition\n\nThis theorem establishes that cross-regularization converges for neural networks under assumptions that are consistent with observed empirical behavior, without requiring guarantees of global convergence for the underlying neural network optimization problem.\n\nOn the choice of stochastic regularization, we also examined L2 regularization but found minimal empirical effect in our neural network architectures. This is expected because L2 regularization primarily constrains weight magnitudes, which are effectively cancelled by the normalization layers present in our networks.\n\n### Connection to Lottery Ticket Hypothesis\n\nYour observation connecting our noise-based regularization to the Lottery Ticket Hypothesis enriches our framework. We've quantified this relationship: our noise levels (σ≈13) create an information bottleneck with capacity C≈0.004 bits/symbol, mathematically equivalent to pruning with sparsity ≈99.4%, aligning with LTH's 98%.\n\nThis connection validates our results by demonstrating that two distinct approaches—gradient-based noise adaptation versus iterative weight pruning—converge to similar architecture-specific patterns. In light of this, we will revise our claims about the novelty of the high noise regime, recognizing we're capturing intrinsic properties of neural information capacity rather than method-specific artifacts.\n### Backpropagation Through Augmentation\n\nFor data augmentation, your concern about optimization potentially minimizing all transformations is insightful. Our solution:\n\n1. We parameterize transformations with continuous magnitude parameters (e.g., rotation angle α determining range U[-α,α])\n2. During training, we apply single random transformations for regularization effect\n3. During validation, we average predictions over multiple transformations:\n   $\\mathcal{L}_\\text{val} = \\mathbb{E}_{\\text{transformations}}[\\mathcal{L}(f(\\text{transform}(x)))]$\n\nThis creates an optimization equilibrium where validation performance constrains transformation magnitude: excessive transformations that remove discriminative features increase validation loss, while insufficient transformations lead to overfitting. The validation gradients naturally balance these competing factors.\n\nOur results confirm this behavior: rotation parameters converge to ~3° for SVHN while shear transformations are optimized toward zero. The method identifies which transformations provide regularization benefits for a specific dataset and minimizes those that don't.\n\nFor discrete transformations like horizontal flips, we agree this remains a limitation without relaxation techniques, which we're exploring in follow-up work.\n\n### Additional Baselines\n\nBased on your feedback, we're implementing:\n- Temperature scaling and deep ensembles for calibration comparisons\n- EWC for continual learning benchmarks\n\nOur preliminary calibration results (https://tinyurl.com/3mtacsu9) show considerable improvements over post-hoc methods, supporting our claim that uncertainty calibration emerges naturally in our approach.\n\nThank you for your thoughtful review. The connection to pruning literature and your theoretical insights have improved both the paper and our analysis of cross-regularization's properties.", "claims_and_evidence": "Claims organized by section:\n\n4. The proposed method converges under various conditions - yes, but limited in scope and with some missing caveats (see \"Theoretical claims\" below).\n\n5. The proposed method is equivalent to/can replace norm-based regularization for convex problems - yes.\n\n6. The proposed method can regularize neural networks via additive noise - yes, but with some caveats (see Experimental Design or Analyses\" below).\n\n7. Automatic uncertainty calibration - yes but not entirely convincing, as a baseline comparison with post-hoc uncertainty calibration methods would be useful here to determine whether adaptive noise is competitive in this area.\n\n8. Data growth - not entirely convincing. Here, a baseline versus some standard continual learning method is needed (e.g. cosine annealing LR schedule).\n\n9. Adaptive data augmentation - yes to a limited extent, as methods and limitations need elaboration.", "methods_and_evaluation_criteria": "Section 9: how does one backpropagate through data augmentation parameters? Presumably only continuous parameters would work, as opposed to e.g. random horizontal flips with probability $p$. Also, there are some scenarios in which I am not sure this method would converge optimally (see \"Questions\").", "theoretical_claims": "Section 4: the theoretical claims are sensible. There is a major unstated issue however, which is that one must assume $\\theta$ and $\\rho$ are independent. It is clear in section 5 that $|\\theta| = 1$ is necessary for $L^2$ regularization to converge (otherwise $\\theta$ could grow by $\\alpha$ and $\\rho$ by $\\alpha^{-1}$ without effect). But in the neural network tasks, it is not clear that this is the case - requiring either discussion or a restriction of the results in section 4 to the tasks of section 5.\n\nHere is a contrived example of the issue for adaptive noise: if the optimal $\\theta$ on $D_{train}$ is unbounded in the direction of $v$ but the optimal $\\theta$ on $D_{reg}$ points in the opposite direction, then wouldn't the noise regularization optimize towards $N(\\alpha v, \\alpha)$? for $\\alpha \\to \\infty$?\n\nA minor issue is that theorem 4.2 says nothing about the size of the neighbourhood which has convex approximation. This is not a serious problem in practice (optimization on non-convex neural networks works just fine!) but the theorem says very little in its current form. One could remove it, or elaborate e.g. by relating the neighbourhood radii to some measurable quantity (e.g. network Lipschitz bound).", "experimental_designs_or_analyses": "Section 6: in the case of adaptive noise for neural networks, there is an untested possibility that if the optimum for added noise is unbounded, the method would not actually be converging in $\\rho$. This alternative hypothesis could be tested by running training for much longer and seeing if performance drops as noise increases further. Of course early stopping would avoid this potential issue, but early stopping would also add a hyperparameter for the train/test generalization tradeoff (which this method aims to avoid).\n\n$L^2$ regularization is common for neural networks - why not analyze this regularization? This would solve many of the issues with the neural network experiments and provide a solid connection with sections 4-5.", "relation_to_broader_scientific_literature": "The method, in addition to advancing regularization methodology (over Jaderberg et al. 2017, Gal et al. 2017, Molchanov et al. 2017, etc.), also provides an interesting tool for probing the capacity of neural networks (Bartlett et al. 2017, Zhang et al. 2021). The results in figure 2 and 4 corroborate existing literature on 1) over-parameterization in later layers of VGG networks, 2) avoidance of plasticity loss and overfitting in continual learning, and 3) evolution of training towards overfitting which is combatted by data augmentation.", "essential_references_not_discussed": "The high degree of noise in figure 2, as well as its concentration in the later layers of a VGG-16 trained on CIFAR-10, strongly suggests a connection with neural network pruning, where similar results have been found. In particular, the Lottery Ticket Hypothesis (Frankle & Carbin 2018). showed that sparsity rates near 98% are possible for VGG-16, with most of the sparsity concentrated in the last layers. Thus, this result is not as surprising in relation to prior work.", "other_strengths_and_weaknesses": "Strengths: the method looks to be quite promising and useful in a large variety of ML and DL applications. Automating hyperparameter selection and doing so adaptively (during training) is a huge improvement. There are non-trivial computational costs to using this method as it requires more training/evaluation iterations, as well as more data to effectuate a train/regularization split, but based on the parameter sensitivity analysis (section F) they are within a small number of multiples of regular training (keeping in mind that regular hyperparameter tuning would require multiple runs also).\n\nWeaknesses: the theory in section 4 may not apply to the neural network tasks, and there are some other issues about the neural network tasks. The method is not quite a drop-in improvement as it requires rethinking parameterization, and in some cases, backpropagation through additional operations like image transformations.\n\n## Update after rebuttal\n\nThe authors have addressed my comments and I maintain my recommendation for acceptance.", "other_comments_or_suggestions": "Some figures are too small to be legible - figures 3 and 4 especially.", "questions_for_authors": "Section 9 (data augmentation): could the authors discuss the possibility of the following failure modes?\n- transformation removes a relevant feature (e.g. 180 degree rotation causes 9 and 6 to be indistinguishable) at the same time there is a strong spurious feature (e.g. sky background for \"airplane\" images), causing the model to overfit the latter.\n- transformation makes validation perform worse, so that the optimum becomes no transformation.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper studies a method for the alternating optimization of model parameters and regularization parameters (with the training set split into corresponding parts). The experimental evaluation and theoretical contributions are sound. The contribution might be of interest to the optimization and deep learning communities, particularly in the area of hyperparameter tuning. Reviewers identified some weaknesses, such as a lack of ablation studies for alternative methods.
Decision: Accept (poster)