Hyperparameter Loss Surfaces Are Simple Near their Optima
Authors: Nicholas Lourie
He He
Kyunghyun Cho
Venue: COLM 2025 (2025)
Keywords: experimental design
hyperparameters
scaling laws
TL;DR: We derive a theory describing the hyperparameter loss surface and yielding new statistical tools for understanding it.

Abstract: Hyperparameters greatly impact models' capabilities; however, modern models are too large for extensive search. Instead, researchers design recipes that train well across scales based on their understanding of the hyperparameters. Despite this importance, few tools exist for understanding the hyperparameter loss surface. We discover novel structure in it and propose a new theory yielding such tools. The loss surface is complex, but as you approach the optimum simple structure emerges. It becomes characterized by a few basic features, like its effective dimension and the best possible loss. To uncover this *asymptotic regime*, we develop a novel technique based on random search. Within this regime, the best scores from random search take on a new distribution we discover. Its parameters are exactly the features defining the loss surface in the asymptotic regime. From these features, we derive a new asymptotic law for random search that can explain and extrapolate its convergence. These new tools enable new analyses, such as confidence intervals for the best possible performance or determining the effective number of hyperparameters. We make these tools available at: https://github.com/nicholaslourie/opda.

Published as a conference paper at COLM 2025
Hyperparameter Loss Surfaces Are Simple Near their Optima
Nicholas Lourie New York University nick. lourie@nyu.ede edu
He He
Kyunghyun Cho New York University & Genentech kyunghyun.choenyu.ee
New York University hheênyu.
Abstract
Hyperparameters greatly impact models' capabilities; however, modern models are too large for extensive search. Instead, researchers design recipes that train well across scales based on their understanding of the hyperparameters. Despite this importance, few tools exist for understand- ing the hyperparameter loss surface. We discover novel structure in and propose new theory yielding such tools. The loss surface is complex, but as you approach the optimum simple structure emerges. It becomes characterized by few basic features, like its effective dimension and the best possible loss. To uncover this asymptotic regime, we develop a novel technique based on random search. Within this regime, the best scores from random search take on new distribution we discover. Its parameters are exactly the features defining the loss surface the asymptotic regime. From these features, can explain and extrapolate its convergence. These new tools enable new analyses, such confidence intervals for the best possible performance determining the effective number of hyperparameters. We make these tools available at hpt:s: / /gi thub.
derive new asymptotic law for random search that
Liama 33M on SlimPajama
DeBERTaV3 on MultiNLi-m
ResNet18 on ImageNet
3.75
0.1050
0.34
3.70
0.1025
valication
3.65
3.60
0.30
0.0950
3.55
0.0925
0.28
3.50
0.0900
0.26
20
10
30
search Iterations
search Iterations
search iterations
stymptotic regima
theoratical
ampitical (95% cant
Figure 1: The hyperparameter loss surface has simple structure near the optimum. Using this structure, we can reason about how the validation score will improve as we run an algorithm like random search. The plots compare the theoretical functional form against the empirical rate of progress using 1,024 training runs in each. The ground truth (dashed blue) closely adheres the theoretical form (solid yellow), with that form remaining fully inside its 95% confidence bands. Across all three Leenarios-laguuae loss), supervised finetuning (error rate), and image classification (error rate)-thh tte simple structure near the optimum drives the practical outcomes of hyperparameter search after just 1 or 2 iterations.
model pretraining (log
1 Introduction
Hyperparameters affect every aspect of foundation models, from efficiency to generalization. Unfortunately, extensive hyperparameter search becomes impossible at scale. Instead, researchers must design recipes that work across scales based on their understanding of the hyperparameters. Still, few tools exist for understanding the hyperparameter loss surface.
1

---

Published as a conference paper at COLM 2025
We want to understand the surface, but is too expensive to explore it-instead we must exploit its structure. We need an empirical theory of the hyperparameter loss surface. A similar theory, scaling laws, has had tremendous success at balancing data and compute without search (Rosenfeld et al., 2020; Kaplan et al., 2020; Hoffmann et al., 2022); however, scale is just one aspect of model's design. Researchers must also consider things like the data mix, regularization, and architecture. j4 Transfer offers insights on optimization hyper- parameters such as the initialization and the learning rate (Yang et al., 2021); nonetheless, does not offer general theory of the loss surface or reveal whether it has deeper structure.
We discover such structure exists and propose a new theory describing it. While the hyperparameter loss surface is complex, as you approach the optimum simple structure emerges. In a large area around it, the surface's structure is dominated by few basic features: the effective dimension, the noise due to random seeds, and the best possible loss. Here, in the asymptotic regime, the surface looks like a quadratic polynomial with additive, normally distributed noise.
To find the asymptotic regime, we look for the shadow it casts on random search. In it, the scores from random search take on specific distribution: the noisy quadratic. So, we look for a threshold after which the scores follow this distribution. Imagine you run random search, obtain validation scores, and plot their distribution, then you will see the distribution's tail matches a noisy quadratic. The point at which they match defines the asymptotic regime.
The noisy quadratic comes from new limit theorem we prove for random search. Under regularity conditions, the best validation scores will follow a noisy quadratic. This theorem explains and extrapolates how random search Sonverges-unovverte
its asymptotic law.
Beyond search, the noisy quadratic captures properties of the hyperparameter loss surface. Each of the distribution's parameters corresponds to a different one: the effective number of hyperparameters, the variance due to random seeds, the concentration of probability near the optimum, and the best hyperparameters' loss. By fitting the distribution, we can estimate these properties. Thus, the noisy quadratic lets us find both where simple structure emerges and what it looks like. Better still, since the noisy quadratic a parametric distribution, we can construct confidence intervals for these properties using maximum likelihood theory.
Of course, a good theory must reflect the data. We validate our theory in three practical scenarios: language model pretraining, supervised finetuning, and image classification (64). Training 1,024 models in each, we test that our theoretical form fits the empirical distribution from random search ($4.1). In all three scenarios, the theoretical form adheres closely to the ground truth. Moreover, the asymptotic regime is always sarge-occupyinng
a third to half of the search space. Beyond fit, we test assumptions: is the noise actually normal with constant variance (64.2)? In fact, it converges to normality long before the asymptotic regime, and while the variance begins inflated it quickly converges to a constant. Last, we test our theory's application. In each scenario, we extrapolate how random search will converge based on the first 48 search iterations ($4.3). While our point estimates mostly smooth their nonparametric baselines, the confidence bands show a dramatic improvement.
With our theory, researchers can identify the asymptotic regime, understand its structure, extrapolate how random search will converge, and infer properties of the hyperparameter loss surface-all all while quantifying their uncertainty. So that others may use these tools in their own research, we make them available at: https://ge thub. com wwwp:/ihhoolloorieee/ /odaa.
Simple Structure Emerges Around the Optimum
Simple Structure. In a large area around the optimum, the hyperparameter loss surface is approximately quadratic polynomial plus noise which is normally distributed with
Around the optimum. This structure emerges for all the points whose loss is better
threshold theoretical empir cal
density
than some threshold. Beyond this threshold, the scores from random search follow noisy quadratic distribution (right).
constant variance.
score
2

---

Published as a conference paper at COLM 2025
2 A Theory Based on Simple Structure
Our theory starts with a simple claim: near the best hyperparameters, the loss surface is approximately quadratic with additive normal noise. This structure emerges for those hyperparameters (x) whose expected loss (Y) is better than some threshold: IB[Y x s 0, and that threshold defines the asymptotic regime. But, how do we find it?
Finding the asymptotic regime in a high dimensional search space would be challenging; instead, we look for the effect it leaves upon random search. Within the asymptotic regime, the scores found by random search follow a new parametric family: the noisy quadratic distribution. By finding where these scores converge to it, we find the threshold defining the asymptotic regime. The noisy quadratic's s parameters then let us infer other properties of the loss surface, such as the best possible performance and its effective dimension.
Here, we derive these results informally, for proofs see $F.
2.1 Formalizing Hyperparameter Search
Imagine fitting neural network-perhape pretraining a language model.' 1 Many choices must be made: Which architecture? What learning rate? Each one a hyperparameter, and each hyperparameter takes value from the hyperparameter search space, x= [x E X. Evaluating the network then produces score, y E Y C R, such as cross-entropy. This score is random variable that depends on the initialization and the data order as well as the hyperparameters: Y x. Its conditional distribution defines the hyperparameter loss surface, and in particular we will be interested in its mean: gxx) [|([x)]
To optimize it, we use a hyperparameter tuning algorithm. Each round, the algorithm (randomly) selects a hyperparameter configuration, X,, evaluates it to obtain score, Y, then keeps the best one found so far:
T min Y; i=
(1)
We can think of k - Tg random function, which we call the funing process.
The tuning process captures the whole distribution of outcomes from hyperparameter search; however, it is complex, high dimensional object. Instead of the whole distribution, convenient to consider a summary like the median. Letting M[X] be the median of X,
the tuning curve is the function, 7: R>0 + Y.2
T(k)= == [
(2)
The tuning curve concisely describes how hyperparameter search might progress. In general, depends on the model, the search space, hyperpara fixed, we can compare the efficiency of different tuning algorithms; holding the algorithm fixed, we can compare the difficulty of tuning different models.
the search a algorithm. Holding the model
A simple standard for com comparing models is random search. In it, we sample hyperparameter configurations from a fixed search distribution, X; 1~ X. Since configurations are sampled independently, their scores are independent as well. In essence, the scores come from fixed score distribution, Y; ~ Y. By analyzing this distribution, we can understand the whole tuning process. For example, the best loss after k rounds is just the minimum of k draws, and the minimum's 's cumulative distribution function (CDF), Fe(y)= = P(T, S y), has a simple relationship to the CDF of one sample, F(y)= = P(Y S y):
(v) = 1 - P
min Y; > y il..k
II P(Y > y) 1 (1 F il...
(3)
Thus, the distribution from one round of random search defines the distribution from k rounds. If we can estimate the distribution's tail, then we can extrapolate infinitely into the future-captuuing the entire tuning process. To accomplish this, we need to find an appropriate parametric form.
Thii section closely follows our formalization in Lourie et al. (2024) ($3.1).
Following Lourie al. (2024), we extend the definition of the tuning curve from k E NN to all k R by defining Tr as the random variable with CDF F(y)*
3

---

Published as a conference paper at COLM 2025
The Taylor Expansion Determines Random Search Asymptotically
4
t=
10
0.0
0.5
-1.0
3.0
0.5
-0.5
0.5
-0.3
0.5
X
x
curren: best
go3d best
y gIX)
Taylor approximation
Figure 2:
With more iterations, random search finds better hyperparameters. As the best score improves, the region of better hyperparameters shrinks around the optimum; thereby, the Taylor polynomial gives better approximations at the hyperparannters that improve the score.
2.2 The Deterministic Case
Imagine the loss surface has no noise- that evaluating hyperparameters is deterministic. How would random search behave? At any point in time, the only hyperparameters that matter are the ones that might improve the loss. As search continues, you find better hyperparameters, and the region of even better ones shrinks about the optimum. As this region shrinks, the Taylor polynomial becomes a better approximation within it. Figure illustrates this idea.
At the optimum, y+), the gradient is 0 so the Taylor series is given by the Hessian, Hz.:
1 =x x.)" Hx. (x x.)
gxx) y. +
(4)
At the same time, any continuous probability density is roughly constant small enough interval; thus, near the optimum, the search distribution is approximately uniform.
Putting these facts together, we derive the tail of the score distribution via a geometric argument. Consider the event = (XX) S y. Rearranging the Taylor approximation gives:
xx) T (x xx) S y- y/
Since x, is minimum, the Hessian is positive semi-definite. If we marginalize out the null dimensions, then this equation defines an ellipsoid. P(Y S y) is proportional to its volume, which is proportional to (y y. where d. is the rank of the Hessian. Thus, as + y
F(y) P(Y S y) yy-
y
(6)
Motivated by this analysis, we define the comvex quadratic distribution, 2min (w. 7), by F(y, r) == wly 2/2 Usually, we prefer an alternative parametrization. Let & be the
maximum of the distribution's support. F(B) (( a) 1, w (B a) -1/2 thus:
& 7/2
F(y; p,r) = -
We can differentiate the CDF obtain the probability density function (PDF):
f(y;a,,,)) 2(B a)
y a B-a
(8)
Each parameter relates to a different aspect of the hyperparameter loss surface: a the best possible score, measures how concentrated the distribution is near the minimum, and y is the effective number of hyperparameters--we which is always less than the nominal number.
In summary, we derive new parametric family: the quadratic distribution. This family describes the score distribution's tail when optimizing deterministic function via random search. For minimization, the left tail approaches the convex quadratic distribution; for maximization the right tail approaches the concave quadratic distribution. We give formulas for both in SA. For more details the derivation, see SE.

---

Published as a conference paper at COLM 2025
2.3 The Stochastic Case
The validation score is rarely a deterministic function of the hyperparameters. More often, the score varies due to random factors such the initialization and data order.
Still, even when the scores are random, their conditional mean, E X is not. We could apply our previous analysis the mean, but we would need to bridge the gap between
and the actual observations. Taking inspiration from classic regression analysis, we might
consider whether Y varies about the mean with additive noise. Let g(X) Y then we
assume Y = g(X) ) + E where E ~ (N, o)). This simple assumption seems too good to be true, but in fact it gives a great fit to the data ($4.1). Even more surprisingly, if you retrain the same hyperparameters many times, the scores do in fact become normally distributed with constant variance as you enter the asymptotic regime (54.2). Thus, additive noise offers realistic model for this random variation.
Assuming Y = 8(X) + E, the tail of gXX) converges to a quadratic distribution, so:
Y Y= E, Q ~ Qmin
E (0,,0))
(9)
This sum defines a new family: the noisy quadratic distribution, Q(a,B, comes in two variants: the convex (Qmin) and concave (Qmax) noisy quadratic distributions. Moreover, when o 0, we recover the (noiseless) quadratic distribution special case.
Let & be the CDF the standard normal distribution. The CDF of the noisy quadratic is:
(v-B))
V V~ N~N y
Kyyaa8,,.0 +
E
(10)
0
B - a B p-a
The noisy quadratic's PDF is:
V [
y
Fly;a,B.7.0) 2(B a)
V~
(11)
B
Thus, we can express the noisy quadratic distribution's CDF and PDF in terms of properties of the normal distribution. For mathematical details, see SF.2. Equations 10 and 11 require numerical methods to evaluate, so we provide robust implementations in our library opda.
In summary, we extend the quadratic distribution to more general family: the noisy quadratic distribution. This family describes the score distribution-and thus the outcomes- of random search in typical deep learning scenarios. When minimizing, the score distribution's left tail converges to a convex noisy quadratic; when maximizing, its right tail converges to a concave noisy quadratic. See SB for formulas for both.
2.4 Applying the Theory in Practice
The noisy quadratic distribution is powerful tool for studying neural networks. With it, we can answer two types of questions: we can use it to reason about random search, and we can use random search to understand the hyperparameter loss surface.
For random search: its convergence is described by the tuning curve, and the tuning curve is determined by the noisy quadratic. In particular, random search is fast when the effective number of hyperparameters (7) low. For given model, we can find how easy to tune by estimating its tuning curve using the noisy quadratic (e.g,, see 54.3).
For the hyperparameter loss surface: its most important properties are captured by the noisy quadratic's parameters. When minimizing (maximizing), (6) is the average score of the best hyperparameters and 7 is the effective number of them. We can estimate these quantities by fitting the noisy quadratic. Moreover, we use maximum likelihood theory to test hypotheses and create confidence intervals for them (e.g,, via a likelihood ratio test). For example, you could test if adding new hyperparameter increases the effective number.
When the variant is clear from context, we write the distribution unadorned: Q(a, 0,)) We differentiate the quadratic, (aa, 7), and noisy quadratic, Q(a, 5, 7.6),, by the presence of 07. In our derivation (62.2), the effective number of hyperparameters is the rank of the Hessian.
5

---

Published as a conference paper at COLM 2025
To apply our theory, you must pick search space and search distribution. A few tips. With a small number of discrete hyperparameters, you can evenly stratify over them to ensure good coverage. More generally, the noisy quadratic emerges when there exist coordinates in which the hyperparameters are uniform and the loss surface adheres to its 2nd order Taylor polynomial. Sampling on the right scales really helps. Start bounded hyperparameters on a logit, positive ones on log, and real-valued ones on a linear scale. Afterwards, adjustt as necessary. Sample uniformly (on that scale) between the bounds you would normally choose for grid search. Tighter bounds speed up convergence, but they still must contain the optimum. To find the asymptotic regime, fit the noisy quadratic to the empirical CDF (eCDF) with several thresholds until you find the loosest one that gives a good visual fit.5
3 Experimental Setup
To test our theory, we run random search on representative scenarios: Llama 33M (Touvron et al., 2023) on SlimPajama (Soboleva al., 2023) for language modeling, DeBERTaV3 (He et al., 2023) on MultiNLI (Williams et al., 2018) for supervised finetuning, and ResNet18 (He et al., 2016) on ImageNet (Russakovsky et al., 2015) for vision pretraining. To fully capture the score distribution, we train 1,024 separate hyperparameter configurations in each.
Llama 33M is a smaller variant of Llama, a causal transformer (Vaswani et al., 2017). We pretrain it on Slim ajjama--, subset of the SlimP'ajama web corpus. We tune the learning rate, 31 and $2 for Adam (Kingma & Ba, 2015), warmup steps, weight decay, and dropout.
DeBERTaV3 V3 is pretrained BERT-like model (Devlin et al., 2019). In Lourie et al. (2024), finetuned it on MultiNLI, a natural language inference benchmark. We tuned the learning rate, fraction of first epuch for warmup, batch size, number of epochs, and dropout.
ResNet18 is a convolutional network with residual connections. We train it with momentum SGD and a 1-cycle policy (Smith & Topin, 2018) on ImageNet, image classification benchmark used for vision pretraining. We tune the learning rate, peak epoch, momentum, batch size, epochs, weight decay, label smoothing, and blurpool (an architectural parameter).
For the models' search distributions see SC. Our analysis and random search results (includ- ing sampled configurations and full learning curves) are available in opda (v0.8.0).
We validate our theory in three parts.
Assessing Goodness of Fit. We confirm the noisy quadratic matches the score distribution from random search. To do so, we estimate the score distribution using the eCDF and the nonparametric confidence bands from Lourie et al. (2024). We then fit the noisy quadratic to its tail via censored maximum spacing estimation (Cheng & Amin, 1983; Ranneby, 1984). We select the threshold for the asymptotic regime via visual diagnostics.
Testing Additive Normal Errors. We verify our strongest assumptions: normality and homoskedasticity of the variation due random seeds. For this, we take the ResNet18 results, pick the configurations scoring at the 12.5th, 25th, up to 100th percentile, and retrain each 128 times to obtain large samples from score distributions with fixed hyperparameters.
Estimating and Extrapolating the Tuning Curve. To demonstrate how to use our results to evaluate a model, we subsample 48 iterations of random search without replacement from the full 1,024 for each model. For the ground truth eCDF, we use all 1,024 iterations. For nonparametric estimates, we construct the eCDF and Lourie et al.'s (2024) confidence bands from the subsample. For parametric estimates, we select the asymptotic regime via visual diagnostics using only the subsample, fit the noisy quadratic distribution to the tail via censored maximum spacing estimation (Cheng & Amin, 1983; Ranneby, 1984), and compute parametric confidence bands from the nonparametric ones as consonance regions (Easterling, 1976). We compute these via brute-force search with a grid of 64 log-spaced values for 0, 128 linearly spaced values for a, and 256 linearly spaced values for B.
For an example, see the Extrapolating Random Search section of the Validating the Parametric Analysis in Practice notebook in opda (v0.8.0). httpps/ http:///ghhgggggececco//o /attlceeecc//tttttts//a/ datasets/ DatasetssDDKoon Dotoooooo/ooiiiiioo /SlimPajama-6B

---

Published as a conference paper at COLM 2025
Liama 33M on SilmPajama-66
DeBERTaV3 on MultiNLI-m
ResNet18 on ImageNet
0.4
Score Cotrrballii
0.3
044
0.3
0.3
0.2
0.2
02
0.1
0.0
0.0
0.0
3.55
3.00
3.05
3.70
3.55
0.095
0.100
0.105
0.110
0.115
0.28
0.30
0.32
0.34
y
3.75
0.105
3.65
0.100
0.32
Tuming Curve
3.00
validation
0.085
3.55
0.20
1.50
3.45
3.40
0.24
3.55
40
BG
100
20
40
80
80
100
20
60
80
100
search iterations
search iterations
search iterations
asymptotic regime
theoretical
empriiaa (93% confidence)
Figure 3: A comparison of the score distribution (empirical) and noisy quadratic (Iheoretical). The top row depicts CDFs, the bottom row depicts tuning curves. Each column corresponds to different scenario: pretraining Llama 33M on Slim aajama-ja (cross-entropy), finetuning DeBERTaVE V3 on MultiNLI (error rate), and training ResNet18 on ImageNet (error rate). The asymptotic regime is the performance threshold beyond which the theoretical approximations apply. All estimates use the scenarios' full 1,024 iterations of random search. In the asymptotic regine, the noisy quadratic distribution matches the score distribution from random search.
4 Testing the Theory
4.1 Assessing Goodness of Fit
Our main claim is that simple structure in the hyperparameter loss surface determines practical outcomes from search. Using this structure, we derived theoretical forms for the score distribution and tuning curve. Figure 3 compares these forms to what you actually observe.
Across three scenarios, the empirical and theoretical distributions show an excellent fit. In each, both the noisy quadratic's CDF and its median tuning curve closely adhere the ground truth. They remain within the 95% confidence bands at all times. And, as theory predicts, the point estimates fit the ground truth almost perfectly in the asymptotic regime.
Besides showing our assumptions are satisfied, these results show the theory is practically relevant. It not enough for the hyperparameter loss surface to have structure the optimum; that structure must occupy enough space around it to be useful. Figure 3 confirms this. In each scenario, a large fraction of the score distribution falls within the asymptotic regime: 44% for Llama 33M, 57% for DeBERTaV3, and 34% for ResNet18. These search spaces are broad, typical of what practitioners might use when tuning a new model. Still, almost the entire tuning curve falls in the asymptotic regime, from the first few iterations. These results also generalize when you use the same search space on new architectures (see SD). Empirically, the asymptotic regime explains much of the behavior you see in practice.
7

---

Published as a conference paper at COLM 2025
Normality of Residualss s across Percentiles for ResNet18 on ImageNet
12.5%
25.0%
37.5%
50.00 0%
62.5%
75.0%
87.5%
100.0%
[0=00879]
(0=0.463 0.463
[ 003000
( (ύ= 0.348
| 0.330
( 0.315
(ū= 0.302
(#=0.277)
Histogra
40
percentile
99
Normal Plot
90
90
error rate
Figure 4: Diagnostics for the score distribution's normality given fixed hyperparameters.
The top row shows histograms with kernel density estimates; the bottom shows Q-Q plots. Columns represent configurations across error rate percentiles for ResNet18 on ImageNet. All except the worsf performing hyperparaneters demonstrate a very high degree of normality.
Homoskedasticity over Error Rate Percentiles for ResNet18 on ImageNe:
G 1.2e-00 03
10-2
(95% contidrcca
12.5%
25.0%
37.5%
50.0%
62.5%
75.0%
87.5%
0%
(A-0.403
(j- 0.380
(A (A-0..3 0.348)
(d300.30)
(A-
percentile
Figure 5: A comparison of standard deviations for the score given fixed hyperparameters. The x-axis gives configurations at different error rate percentiles for ResNet18 on ImageNet. Points are standard deviations at those percentiles. Confidence intervals are simultaneous. The standard deviation quickly converges to constant long before the asymptotic regime.
4.2 Testing Additive Normal Errors
you fix the hyperparameters, are the scores really normal with constant variance? Often, bad hyperparameters unstable some threshold (the asymptotic regime). We take search results from ResNet18, pick the configurations the 12.5%, 25%, to 100% best error rates, and retrain each 128 times. The claim has two parts: the scores are normally distributed, and their variances are constant.
only expect this structure after the scores exceed
Testing Normality. We test normality using the venerable normal probability plot. In addition, we show histograms and kernel density estimates (KDEs) to offer more intuitive visualization. Figure 4 displays the results. On the top, besides the worst hyperparameters, the distributions exhibit familiar bell curve. The normal probability plots are even more decisive. On the bottom, the sample quantiles almost all fall on the y = x line, corresponding to the quantiles of the normal. Each plot from 25% on up displays high degree of normality.
Testing Constant Variance. We test constant variance by plotting simultaneous confidence intervals for the standard deviation across the error rates. As we have confirmed normality, we use the x2 confidence interval for the standard deviation of normal distribution; as the intervals are independent, make them simultaneous via Šidák correction. Figure 5 shows the result. The standard deviation drops to a constant around 37.5%. From then on, all confidence intervals contain common value (1.2e-3, the mean of the last three points). The intervals are small, so it is unlikely any large differences exist. Thus, the standard deviation begins inflated, but converges to constant long before the asymptotic regime.
Both analyses suggest the same conclusion: bad hyperparameters exhibit bad structure, but as hyperparameters improve-aa as you approach the cactimuumsimppee structure emerges.
8

---

Published as a conference paper at COLM 2025
Liama 33M on SilmPajama-66
DeBERTaV3 on Mum
ResNet16 on ImageNet
0.7
0.5
0.5
So0 DD Sonpuuatio
0.4
0.5
0.4
0.3
0.3
0.2
0.1
0.1
0.0
0.0
353
3,00
3.05
3.70
0.095
0.100
0.205
0.29
0.29
0.30
0.31
y
y
y
0.105
0.34
3.7
0.100
0.32
validation Score
3.6
0.30
Tuning Curve
0.095
0.28
3.5
0.085
0.22
3.3
0.080
20
40
60
20
40
60
40
60
search iterations
search iterations
search iterations
ground truth
styinptccic regme
theoratical (80% confidence)
empincal 180% confidence)
Figure 6: Examples of nonparametric (empirical) and noisy quadratic (theoretical) estimates. The top shows CDFs, the bottom shows tuning curves. Columns are different scenarios: pre- MultiNLI
training Llama 33M on SlimPajama-66 (cross-entropy) finetuning DeBERTaV3 (error rate), and training ResNet18 on ImageNet (error rate). The asymptotic regime is the threshold beyond which the noisy quadratic was fit. Estimates use 48 iterations of search. The noisy quadratic distribution gives tighter bands for the tuning curve at the same confidence.
4.3 Estimating and Extrapolating the Tuning Curve
We found simple structure in the hyperparameter loss surface. But, how can we use it? We want better tools for designing experiments for answering questions like: did a new method actually improve the model, or did we just improve its hyperparameters? To answer this, we can estimate the model's tuning curve, or performance as a function of tuning effort (Dodge et al., 2019; Lourie et al., 2024); however, such estimates can be unreliable. With the noisy quadratic, we can construct better estimates and confidence bands. particular, we can extrapolate how the score might improve as we continue hyperparameter search.
To explore this use case, we subsample 48 search iterations in each of the three scenarios. For each subsample, we plotted the eCDF to visually determine threshold for the asymptotic regime. We chose thresholds based on where the eCDF began to show the expected structure. We estimated thresholds of 3.66 for Llama, 0.104 for DeBERTaV3, and 0.302 for ResNet18. Then, we fit the noisy quadratic distribution to these subsamples using the thresholds.
Figure compares the parametric estimates against nonparametric baselines from Lourie et al. (2024). The parametric point estimates smooth out their nonparametric counterparts. This makes sense as both attempt to the same data: when the sample is not representative, they err in similar way. This variation underscores the need for confidence bands. There, the approaches give dramatically different results. The parametric bands enclose the ground truth when the nonparametric ones do," however, while the nonparametric bands become trivial after 8 iterations, the parametric bands extrapolate past all 48 used to construct them.
7 At 80% confidence, least of the 3 bands will fail to contain the ground truth 48.8% of the time.

---

Published as a conference paper at COLM 2025
5 Related Work
Hyperparameters have always been an essential part of deep learning (Orr & Müller, 1998; Montavon al., 2012); however, foundation models pose new challenges due to their cost.
Some researchers have sought theoretical solutions, such as ş TTransfer (Yang et al., 2021). TTransfer reparametrizes several important hyperparameters, such the learning rate or initialization variance, so that their optimal values stay constant across scales. When successful, this lets you find hyperparameters at small scales and transfer to large ones.
To balance resources at scale, researchers turn to empirical scaling laws (Hestness et al., 2017; Rosenfeld et al., 2020; Kaplan et al., 2020; Hoffmann et al., 2022). Scaling laws predict how loss improves as more resources become available. The first scaling laws revealed that loss has a power law relationship with parameters and data (Rosenfeld et al., 2020; Kaplan et al., 2020). Since then, researchers have discovered scaling laws for many other quantities (Liu et al., 2024; Ludziejewski et al., 2024; Kumar et al., 2025). Like our work, scaling laws find structure in the loss surface- most often a power law, which is equivalent to linear structure on Log scales. Unlike our work, scaling
focus on specific inputs. Aside from scale, there are many hyperparameters that impact performance.
Several authors have explored structure in the hyperparameter loss surface (Pushak & Hoos, 2018). Pushak & Hoos (2022) look at AutoML pipelines and find that, while researchers typically use complex optimization algorithms, the loss surface is often unimodal or even convex. Conversely, Sohl-Dickstein (2024) discovers intricate, fractal-like structure at the boundary of where training succeeds for neural networks. In contrast to prior work, we focus exclusively on the hyperparameters around the optimum-the area of most practical interest. By focusing there, we are able to uncover the quadratic structure with normal noise, a structure that is far more regular than those previously found.
We uncover this structure via limit theorem for random search. Similar limits are explored in extreme value theory (Coles, 2001; de Haan Ferreira, 2006). For example, the Pickands- Balkema-De Haan theorem gives conditions under which the tail of distribution converges to a generalized Pareto distribution (Pickands, 1975; Balkema & de Haan, 1974), which relates closely to the (noiseless) quadratic. Rather than the general theorems of extreme value theory, we analyze a specific mechanism in order to build an empirical theory of the hyperparameter loss surface.
Conclusion
The hyperparameter loss surface has simple structure near the optimum: it is approximately quadratic with additive normal noise. Surprisingly, this structure describes the surface quite well and holds over a large region about the optimum-up to 57% of the search space.
Using this structure, we derived a theory of random search in deep learning. We developed parametric family for the score distribution's tail. This family comes in two forms: the quadratic in the deterministic case, and the noisy quadratic in the stochastic one.
The noisy quadratic distribution generalizes the first, and has four interpretable parameters: a, the average performance of the best possible hyperparameters, B, a measure of the probability in the asymptotic regime, Y, the effective number of hyperparameters, and 0r, the scale of the noise due to random seeds. These parameters correspond to characteristics of the loss surface. Thus, our theory lets you reason about random search based on properties of the loss surface, but also to reason about the loss surface based on the behavior of random search.
We studied the loss surface to design better tools for deep learning experiments. While hyperparameter tuning an orthogonal goal, our discoveries might suggest more efficient algorithms, e.g. Bayesian optimization kernels that exploit the quadratic-normal structure.
Our theory offers new tools for deep learning and foundation model research. So that others may use them, we make them available at: https: //gi www..tt.....o.....eeoeee.t http.///cccollllloore///od..dddd
When maximizing instead of minimizing, the roles of and are reversed.
10

---

Published as a conference paper at COLM 2025
Acknowledgments
We thank the anonymous reviewers for their valuable feedback. In addition, we thank Wanmo Kang for helpful suggestions while developing this work. This work was supported by the Institute of Information & Communications Technology Planning & Evaluation (IITP) with a grant funded by the Ministry of Science and ICT (MSIT) the Republic Korea in connection with the Global AI Frontier Lab International Collaborative Research. This work was also supported by the Samsung Advanced Institute of Technology (under the project Next Generation Deep Learning: From Pattern Recognition to Al) and the National Science Foundation (under NSF Award 1922658). This work was supported in part through the NYU IT High Performance Computing resources, services, and staff expertise.
References
A. A. Balkema and L. de Haan. Residual Life Time at Great Age. The Annals of Probability, (5):792 - 804, 1974. doi: 10. a/p/////////////////////// 1176996548.
11766965488. /1t.///o//11199944//44 URL https://doi, org/10. 1214/aop//
R. C. H. Cheng and N. A. K. Amin. Estimating Parameters in Continuous Univariate Distri- butions with Shifted Origin. Journal of the Royal Stafistical Society: Series B (Methodological) 45(3):394-403, 7 1983. ISSN 0035-9246. doi: 10.1111 1..1111j2217-61111188.60000.001. https://doi. org/ 1111/jijjaj..... 2517-61611 1983. tb01 268. x.
URL
Coles. An Introduction to Statistical Modeling of Extreme Values. Springer Series in Statistics. Springer, 2001. ISBN 9781852334598.
L. de Haan and A. Ferreira. Extreme Value Theory: An Introduction. Springer Series in Opera- tions Research and Financial Engineering. Springer New York, 2006. ISBN 9780387239460.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techmologies, Volume 1 (Long and Short Papers), pp. 4171-4186, Minneapolis, Minnesota, June 2019, Association for Computational Linguistics. doi: 10.18653/v1 101855//111N111423 org/N19-1423.
URL https: hatts:s//aclanhhhoogyy
Jesse Dodge, Suchin Gururangan, Dallas Card, Roy Schwartz, and Noah A. Smith. Show your work: Improved reporting of experimental results. In Kentaro Inui, Jing Jiang, Vincent Ng. and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Meth- ods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 2185-2194 Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1 v1/D19-122D URL hwwp:/hooooooy.oo/111//11
Robert G. Easterling, Goodness of fit and parameter estimation. Technometrics, 18(1):1-9, 1976. doi: 10.1660/04401776.996660.oo 104899994. URL https://www tandfonline.eeeom tandfoonline.com//oo// com/doi/ abs/10. 1880/00401706. 1976. 10489394.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 JEEÉ Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016. doi: 10.1109/CVPR.22116
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding- enhanced bert with disentangled attention. In International Conference on Learning Repre- sentations, 2021. URL https://openrevieer
Pengcheng He, Jianfeng Gao, and Weizhu Chen. DeBERTav3: Improving DeBERTa us- ing ELÉCTRA-style pre-training with gradient-disentanglee embedding sharing. In The Eleventh International Conference on Learning Representations, 2023. URL https: bebpenreveew.nt//oouiiiiiie XHHHAA
11

---

Published as a conference paper at COLM 2025
Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md. Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically, 2017. URL
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannne Welbl, Aidan Clark, Thomas Hennigan, Eric Noland, Katherine Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karén Simonyan, Erich Elsen, Oriol Vinyals, Jack Rae, and Laurent Sifre. An empirical analysis of compute-optimal large language model training. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 30016-30030. Curran Associates, Inc., 2022. URL httpss:/ neur neurips wc/pppper.ilee//peeeeeeeeeeeeeee wwt.rre/peeeee/////// pdf.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020. URL https://arxiv http:////aarl..oogb/////////// org/abs/2001 08361.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015. URL https:://axxw..rrg/// 1412.6980.
Alex Krizhevsky. One weird trick for parallelizing convolutional neural networks, 2014.
URL https: www.voorg//aoooo org/abs/144 5997.
Alex Krizhevsky, Tlya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convolutional neural networks. In F. Pereira, C.J. Burges, L. Bottou, and K.Q. Weinberger (eds.), Advances in Neural Information Processing Systems, volume 25, Curran Associates, Inc., 2012. URL httpss//ppocceeeddnngg ieerrrrrrsdinnnnnnnnipsssp.s httpppaarfflle/ppappeeeeeeeeep ptllss/apper//112/ file::3998ccbb000064444464446655
pdf..
Tanishq Kumar, Zachary Ankner, Benjamin Frederick Spector, Blake Bordelon, Niklas Muen- nighoff, Mansheej Paul, Cengiz Pehlevan, Christopher Re, and Aditi Raghunathan. Scal- ing laws for precision. In The T'hirteenth International Conference on Learning Representations, 2025. URL https: httt::://opereviieee
wett//orummiddiie 1PC33CUP
Xiaoran Liu, Hang Yan, Chenxin An, Xipeng Qiu, and Dahua Lin. Scaling laws of roPE- based extrapolation. In The Twelfth International Conference on Learning Representations, 2024. URL https: /ttp:///ppeeeeieeee/eeee/ee wwtpvviww.neefffoooo www.:neeeeotteneee (o-rrmmmmd-00005555
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11966-1197,, 2022. doi: 10.1109/ CVVP52668.2022.000
Nicholas Lourie, Kyunghyun Cho, and He He. Show your work with confidence: Confi- dence bands for tuning curves. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Compu- tational Linguistics: Human Language Technologies (Volume Long Papers), pp. 3455-3442, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10.18653/ 11/222A.maacc-long..155 URL https: /aclaatthlllaaaooooooooooo org/2024, naacl-long 189.
Jan Ludziejewski, Jakub Krajewski, Kamil Adamczewski, Maciej Pióro, Michal Krutul, Szymon Antoniak, Kamil Ciebiera, Krystian Król, Tomasz Odrzygóždz, Piotr Sankowski, Marek Cygan, and Sebastian Jaszczur. Scaling laws for fine grainer mixture of ex- perts. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp- 33270-33288. PMLR, 21-27 Jul 2024. URL https://proceeding mmriinnsss........ wttp:/ng........ss/seeeeee www.s//sssss5 ludziejewski24a htm...
Grégoire Montavon, Geneviève Orr, and Klaus-Robert Müller. Neural Networks: Tricks of the Trade. Lecture Notes in Computer Science. Springer Berlin, Heidelberg, 2nd edition, 2012.
ISBN 9783642352881.
12

---

Published as a conference paper at COLM 2025
Geneviève Orr and Klaus-Robert Müller. Neural Networks: Tricks of the Trade, Lecture Notes in Computer Science. Springer, 1998. ISBN 9783540653110.
James Pickands, III. Statistical Inference Using Extreme Order Statistics. The Annals of Statistics, 3(1):119 131, 1975. doi: 10.1214/a0//117643003 URL https://doi org/10. 1214/aos/1176333002
Yasha Pushak and Holger Hoos. Algorithm configuration landscapes: More benign than expected? Anne Auger, Carlos M. Fonseca, Nuno Lourenço, Penousal Machado, Luís Paquete, and Darrell Whitley (eds.), Parallel Problem Solving from Nature - PPSN XV, pP. 271-283, Cham, 2018. Springer International Publishing. ISBN 978-3-319-99259-44
Yasha Pushak and Holger Hoos. Automl loss landscapes. ACM Trans. Evol. Learn. Optim., 2 (3), November 2022. doi: 10.1145 URL https://doi org/10 1145/3558774.
Bo Ranneby. The maximum spacing method. an estimation method related to the maximum likelihood method. Scandinavian Journal of Statistics, 11(2):93-112, 1984. ISSN 03036898, 14679469. URL http::/www// jstor. .org/ stts/e/bbbe/14/// 59444466.
Jonathan S. Rosenfeld, Amir Rosenfeld, Yonatan Belinkov, and Nir Shavit. A constructive prediction of the generalization error across scales. In International Conference Learning Representations, 2020. URL https://opemte
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge. Inf. J. Comput. Vision, 115 (3):211-25,, dec 2015. ISSN 0920-5691. doi: 10.1007 /s000//s11220 00.1007/11166--01500666-6 URL https:
Leslie N. Smith and Nicholay Topin. Super-convergence: Very fast training of neural networks using large learning rates, 2018. URL https: orgg/bs//// /708. 07120.
Daria Soboleva, Faisal Al-Khateeb, Robert Myers, Jacob R Steeves, Joel 627B token cleaned and
Hestness, and Nolan Dey. SlimPajama: https://www cerebras. net/blog/ deduplicated version of RedPajama.
htap://ieeeetttttotooommmmmmaaee URL http::///gggteteeeeee/ee///t//es
redpajjma,, 2023.
Jascha Sohl-Dickstein. The boundary of neural network trainability is fractal, 2024. URL
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023. URL http:////axxxi.oo/a///////// 13971.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran As- sociates, Inc., 2017. URL https: //ppoeeec/occc//s/ssoooo nwwrins..ncccm file/3f5ee243547deee (eed053ccc48445a0 Paper pdf.
Adina Williams, Nikita Nangia, and Samuel Bowman. broad-coverage challenge corpus for sentence understanding through inference. In Marilyn Walker, Heng Ji, and Amanda Stent (eds.), Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 1112-1122, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 111865/vv1/118 URL https: https////aaaaaalaaaaeeee org. /N18-18- 1-1101
13

---

Published as a conference paper at COLM 2025
Ge Yang, Edward Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tuning large neu- ral networks via zero-shot hyperparameter transfer. In M. Ranzato, A. Beygelz- imer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neu- ral Information Processing Systems, volume 34, Pp. 17084-17097. Curran Associates, Inc., 2021. URL httpss//pproceeiings http:///rrrreeddinenerriiio.... hwwp:/ceeeddnns.niipii...... aff72e28accbbe000//222222272aaa2
14

---

Published as a conference paper at COLM 2025
A The Quadratic Distribution
As derived in $2.2, the formulas for the convex quadratic distribution are:
7/2
F(y;a,B,r) =
a
(12)
y a 2
f(y;a,B,n)= 2(B-al a)
(13)
B -
The equivalent formulas for the concave quadralic distribution are:
(
((y;a,B.7)==1 =
(14)
B
f(y;a, 2[B 133-at a)
9
[B-11 y B - &
(15)
The quadratic distribution is supported only on the interval a S y S B. These formulas are valid within that interval. Outside of it, the density O; below it, the CDF is O; above it, the CDF is 1. The quadratic distribution family is a special case of the four parameter beta distribution, and closely relates to its other special case, the power function distribution.
For more derivations, properties, and discussion of the quadratic distribution, The Quadratic Distribution section of the Parametric Analysis notebook in opda (v0.8.0).
B The Noisy Quadratic Distribution
As derived in $2.3, the formulas for the convex noisy quadratic distribution are:
Fly:a,8,7.0)) = (y-B)
E
V~ N y
(16)
0
B- a B-
y
f(y;a, 0,) 2(B - a)
E
V~N
(17)
B a- a
The equivalent formulas for the concave noisy quadratic distribution are:
Fly;a, = -0V=- E
V N~N B
(B-y y
(18)
B a B
(,y,,,,,,,0) 2(B 2(B- a) Eb [
B o
V~N
(19)
B a' B
Unlike the quadratic distribution, the noisy quadratic is supported the entire real line.
For more derivations, properties, and discussion of the noisy quadratic distribution, see The Noisy Quadratic Distribution section of the Parametric Analysis notebook in opda (v0.8.0).
15

---

Published as a conference paper at COLM 2025
C Models and Search Distributions
As described in $3, we use random search results for Llama 33M, DeBERTaV3, and ResNet18.
For Llama 33M, we used the following search distribution:
1r LogUniformmle- le-
betal Uniform
beta2 Uniform(0.8,1
warmup steps ~ DiscreteUnniform(0.300000
weight weight_decay ~ LogUniform(1e (1e- 4, le0)
dropout ~ Uniform(0,011
For DeBERTaV3, we used the following search distribution in Lourie et al. (2024):
batch_size
DiscreteUniform (16, (16,64) 64)
num.epochs DiscreteUniform/!!
warmup proportion Uniform 0.,6
learning rate LogUniform(Ie-6 le-3)
dropout Uniform(0,0.33
Note that warmup_praaaeee caammppprropottion is the proportion of the first epoch only.
For ResNet18, we used the following search distribution:
epochs ~ DiscreteUniffrm(21) 100)
batch size DiscreteUniform/ (1128, 2128,256, 512, 1024))
Ir 0ggunifor(5ee3,5555 3e1)
proportion Uniform (0, 0.8)
1r epoch prroportion x epocha]
momentum (01,)rm00.7,1100)
veight_decay ~ LogUniform 60et:foormmee611111 61e-6,
label smoothing ~ Uniform(0.0,0.5))
use_blurpool ~ DiscreteUniform((0
In SD, we present additional results that validate our theory's generality and how applies across architectures. For it, we compare DeBERTaV3 to DeBERTa (He et al., 2021), both tuned using DeBERTaV3's search distribution above. We also run random search on AlexNet (Krizhevsky et al., 2012; Krizhevsky, 2014), ResNet18 (He et al., 2016), and ConvNext Tiny (Liu al., 2022) using ResNet18's search distribution above, except fixing use blurpool to 0 because ConvNext does not use maxpool (or blurpool) layers and thus we can not consistently apply the hyperparameter to all three.
D Generalization Across Architectures
While our theory is general, it is also asymptotic; thus, it is natural to ask: how quickly does the asymptotic approximation apply in practice? For Llama 33M, DeBERTaV3, and ResNet18 we saw the asymptotic regime covered 44%, 57% and 34% the score distribution- applying from the first or second iteration of random search. Still, perhaps the asymptotic
16

---

Published as a conference paper at COLM 2025
AlexNet on ImageNet
ResNet18 on ImagaNet
ConvNext Tiny on ImaceNet
10
1.00
0.9
Score Coniibutti
0.90
0.82
0.85
0.80
0.6
0.75
0.5
0.40
0.50
0.52
0.54
0.56
0.50
0.66
0.60
0.73
0.60
0.65
0.70
0.75
10
0.76
1.0
0.9
0.8
0.74
Tuning
axcurac
0.72
0.70
0.5
0.4
0.2
0.66
20
40
80
80
100
20
40
60
80
100
20
40
80
80
100
search iterations
search iterations
search iterations
asymptotic ragimee
theoretical
empirical (95%, confidence)
Figure 7: A comparison of the noisy quadratic (theoretical) and the score distribution (empirical) across different architectures trained on ImageNet. Each column corresponds to a different model: AlexNet, ResNet18, and ConvNext. All models use the same search distribution, and the estimates use 512, 495, and 512 iterations of random search, respectively. Empirical estimates are from the empirical distribution, while theoretical estimates use the noisy quadratic fitted to the tail via censored maximum spacing estimation. For all three models funed with the same search distribution, the asymptotic regime covers large fraction of the search space and the noisy quadratic demonstrates an excellent fit to the score distribution.
regime applies only because these architectures are so advanced, or the search spaces match them particularly well.
To investigate such questions, we compare ResNet18 with two other architectures: AlexNet (Krizhevsky et al., 2012; Krizhevsky, 2014) and ConvNext Tiny (Liu et al., 2022). AlexNet goes from ResNet into the past: many consider it the first major architecture of the current deep learning renaissance and, such, it is considerably less advanced than ResNet- missing later innovations such as batch normalization or residual connections. On the other hand, ConvNext goes from ResNet into the future: it starts with the ResNet architecture and applies lessons learned from transformer-based models. We obtain 512, 495, and 512 iterations of random search for AlexNet, ResNet18, and ConvNext, using the same search distribution across all three (see ŞC). By using the same search distribution across all three models, we guarantee
not unusually well-suited to any specific one.
Similarly, we use the same search space across DeBERTa (He et al., 2021) and DeBERTaV3, for which we obtain 1,024 search iterations each. Figures 7 and 8 present the results.
The atymptotic regime is large in practice. Across all architectures, the asymptotic regime more than large enough be practically relevant. Of the search distributions, covers 18% for AlexNet, 36% for ResNet18, 9 and 20% for ConvNext Tiny as well as 54% for DeBERTa and 57% for DeBERTaVe V3. In other words, it characterizes the tuning curve after 1-4 iterations
In this experiment blurpool is fixed which changes the asymptotic regime for ResNet18.
17

---

Published as a conference paper at COLM 2025
DeBERTa on MuitiNLi-m
DeBERTaV3 on MuitiNLI-m
10
Score ostribution
0.8
P(Ys
0.7
0.5
0.4
0.865
0.870
0.875
0.880
0.885
0.890
0.385
0.890
0.895
0.900
0.905
0.905
0.900
0.895
Tuning Curve
accurscy
0.890
0.905
0.885
0.900
0.875
0.895
20
40
100
20
80
100
search iterations
search iterations
asymptotic rogime
enerrctical
conficence))
Figure 8: A comparison the noisy quadratic (fheoretical) and the score distribution (empirical) across different models finetuned MultiNLL Each column corresponds to different one: DeBERTa and DeBERTaV3. Both models use the same search distribution, and the estimates each use 1,024 iterations of random search. Empirical estimates from the empirical distribution, while theoretical estimates use the noisy quadratic fitted to the tail via censored maximum spacing estimation. For both models tuned with the same search distribution, the asymptotic regime covers a large fraction of the search space and the noisy quadratic demonstrates an excellent fit to the score distribution.
of random search. Thus, our theory describes random search with a realistic budget. The search distribution can not be the driving factor behind this result because we use the same one across each set of architectures. Moreover, while the better architectures display larger asymptotic regimes (e.g., ResNet18 and ConvNext), our theory even describes older less advanced architecture like AlexNet after just handful of search iterations.
The effective number of hyperparameters is stable across architectures. An interesting thing happens when we use the same search space across the different architectures: the effective number of hyperparameters (7) remains constant. For AlexNet, ResNet, and Con- vNext Tiny, the estimate of y is DeBERTa and DeBERTaV3 exhibit a similar phenomenon: both models have = 1. This result suggests an intuitive conclusion: the effective number of hyperparameters seems to be more a property the search space, i.e. the hyperparameters themselves. Thus, it exhibits some stability across models.
modern deep learning, training is often limited by compute. As result, our theory must apply even when the network is not trained to convergence, Fortunately, the ConvNext Tiny results demonstrate this to be true. Despite its name, ConvNext Tiny is significantly larger than ResNet18 (29M vs 12M parameters)- instead, it is more comparable ResNet50. As our training recipe was chosen for ResNet18, it does not enough compute (epochs) for ConvNext Tiny to fully converge. This fact is evident in the best accuracy achieved: 74.9% as opposed to 92.1% Liu et al. (2022). Even in this compute-limited regime, the theory still obtains excellent fit.
Convergence not necessary.
18

---

Published as a conference paper at COLM 2025
E Deriving the Quadratic Distribution
Recall $2.2, we wish to derive a limiting form for the score distribution's tail. Near the minimum, we approximate the hyperparameter loss surface, gxx), by Taylor polynomial and we approximate the search distribution by uniform.
At the minimum, (x+,y-), ) the gradient is 0 and the Hessian is positive semi-definite with eigenvalues A; 2 O. Without loss of generality, let the nonzero eigenvalues be Ad, Since the Hessian is real and symmetric, it is diagonalizable: Hx. QAQ where Q is
orthogonal and is the diagonal matrix of eigenvalues.
The second order Taylor polynomial can be written as:
g(x) ~ y+
2 x x.)
Hz. (x - xx)
x x.) xx TQAQ" (x - x.)
= y. +
2
(x' xx)² (x' x.)
= y+ 2
Σ A (z)- - xx1)
y+ +
2 -=1
where x' = Q²xx, x, and we think of Q" as a change of coordinates. Since Q is orthogonal, X is (approximately) uniform then so is X' = Q"xx. X.
Consider the event Y gXX) S y. Rearranging the Taylor approximation, obtain:
A
x
x
<1
2
y.
This formula is the equation for an ellipsoid along the nonzero eigenvectors. Effectively, we marginalize over the null eigenvectors, along which we assume the loss surface is approximately constant.
The volume this ellipsoid is proportional to vy- raised to the dimension, or (V-y.) y Since the search distribution is approximately uniform, the probability of the event = xX) S is then proportional to this volume:
Fly) P(Y y) & y.)
For an even more detailed discussion, simulations, and visualizations of the convergence, see the Approximating the Tail section of the Parametric Analysis notebook in opda (v0.8.0).
Proofs & Theorems
In $2 and SE, we derived our results without emphasizing formality. We did this for two reasons. First, there are many ways to formalize the theorem theorem-withhuut without particular goal, any specific choice is arbitrary. Second, whether the limit applies in practice is ultimately an empirical question. Consider the normal distribution: numerous versions of the central limit theorem exist, each applying in its own context. What is important is not that one set of conditions produces the normal distribution, but that many do. Therefore, expect might appear and, accordingly, use diagnostics like normal probability plots to determine if has. That in mind, we now illustrate one way to formalize things.
E1
The Deterministic Case
We prove limit theorem for minimization via random search in the deterministic case.
19

---

Published as a conference paper at COLM 2025
First, we need the following proposition, which gives kind of inverse continuity near the minimum:
Proposition F.1. Let C Rd be compact, Y R, 8 X + Y continuous, and y,- g(xx) its unique mininum. Then V8 > 0, Be such that [g(x) - y-1 < e implies || x x.1 <6
Proof. For contradiction, assume > 0 is such that the conclusion is false. Let e; be any sequence such that e; O. For each e₁2 there exists some x; such that lg(x;) yel < e; but ||x- - x=|| > 6, otherwise the conclusion would be true.
Consider the sequence x, Since X is compact, it has convergent subsequence: X x By construction, (x(x;,) ysl Ciz- As C; 0, we have g(x;,) - yr, and because g is continuous:
8(Xcu)
=8 =
lim Xix 140000
lim 8xx,, = yy 11-000
However, ||xi xx|| > 8 so X # Xs, contradicting uniqueness of the minimum.
Theorem F.2. Let X C Rd be compact, C X + Y thrice continuously differentiable, y+ gxxx) its unique minimum in the interior of X, Hx. the Hessian at x., having full rank, and
X ~ X a distribution over with continuous PDF, μ(x). = (X) is a random variable with
CDF Fly), there exists a quadratic distribution with CDF Q(y) such that limy vy. F(y)/(())) =.
Proof. Write the 2nd order Taylor approximation of g at X, as ((x) y + 1/2(x
x.) x.) Hz,, (x (x-x)), x.). Consider some neighborhood of ||x- - -xxl| < By Proposition F.1, we can require y be sufficiently close to y+ to guarantee is in it. Throughout the neighborhood, let e the Taylor approximation's worst case error:
((x) < 8(x) < ((x) +
(20)
Consider Fly) = P(Y < y). By Equation 20, P(((x) + S y) S P(g(x) S S P(t(x) e S y). We can write this equivalently as:
P(((x) y en) - ((g()) < y) s P(t(x) +e4
(21)
Let us analyze (P)( S y)
We will need the fact that X' is approximately uniform near X.. Let c = =(xx). As μ is continuous, μ(x) + c as + x.. Let 1 be the maximum difference in the neighborhood:
c- 11 < μ(x) < c
(22)
In this sense, we can think of X as approximately uniform with density between c 17.
Returning to the Taylor approximation, g is thrice continuously differentiable so the Hessian is real symmetric thus diagonalizable: Hx. -TT AU, with U orthonormal matrix and A = biag(A],... ..,)) the eigenvalues. Think of U as a change of coordinates, u = Ux. Since U is orthonormal with det U| 1, by the change of variables theorem the density of X in these new coordinates is still approximately + 1).
Finally, consider the event: P(t(x) S y). In the coordinates u, Hz, is a diagonal matrix and t(u) = y. + 1/2 A,(4; - )2 therefore, t(u) S defines an ellipse:
(n: unn) 2 S y y.
The volume of this ellipse is:
1//2
d
(y )²1
A
Take all the terms that do not depend on y as a constant, C. The volume is then: Cly - y,)² The probability P(I(x) < y) is the density integrated over this volume. The density is
20

---

Published as a conference paper at COLM 2025
between c- 1/ and + "/, thus the probability is between products of these values and the volume: Cly y 17) < P(((x) v) < Cly <cly-y.)
(23)
Combining Equations 21 and 23, we have:
Cly y.) 7) <g((()) S y) cly+e-y.) 1)
Using the parametrization of the (convex) quadratic distribution's CDF as Q(y) = w(y a)'/ let w Cc, Yv,, and y d. Then dividing by Q(y) we have:
(c (- 11 (1 (1-
F(y) Q
(c+
(
( _
1+ e
(24)
y y
y-
Consider what happens as y y.- By Proposition F.1 the neighborhood about x, shrinks. As result, f + 0 and since g is thrice differentiable the Taylor approximation's error goes to 0 at 3rd order while ys goes to at 2nd order, thus e/ e/(y .y) + 0. Therefore, the upper and lower bounds in Equation 24 go to and thus F(y)/Q(y) - as well. other words:
lim Fly) 1 ky. Q
Thus, we obtain a limit theorem for random search under minimization, maximization being similar.
A few remarks are order. We have shown convergence under one set of conditions; however, convergence can happen under other conditions well. For example, we used uniqueness of the minimum to ensure that as y approaches y+, the corresponding x also approaches x,, the center of our Taylor approximation. If finite number of distinct minima exist, this condition still holds as we approach the global minimum. Even with multiple global minima, they can be added together without issue." 10 For example, the volume of their ellipses will be: Σ Cy y. (y y.) Ej= Cj. As this example shows, many variants of the theorem are possible.
One assumption in particular merits deeper discussion: that the Hessian is full rank. Em pirically, this assumption rarely true. In all our experiments, the effective number of hyperparameters was fewer than the nominal number- in other words, the Hessian was rank deficient. Here is one way to close this gap: if g is constant along the kernel of the Hessian, then you can marginalize over the kernel and consider as function of the quotient space, in which the Hessian will have full rank.
In the end, we just need the hyperparameter loss to be approximately quadratic in some coordinates for which the search distribution is approximately uniform. Designing the search space these assumptions are better satisfied will speed up convergence. For example, you can search for each hyperparameter using a uniform distribution on the appropriate scale (e.g,, log scale for the learning rate). Similarly, you can tighten the search space around the optimum so the Taylor approximation is better fit.
E2 The Stochastic Case
For the stochastic case, the noisy quadratic distribution defined as the sum of a quadratic and normal random variable. If the conditional mean xXX) E Y satisfies the con- ditions of Theorem F2, then it will converge to a quadratic distribution. If addition Y= g (XX) + E, E ~ (0, o) then one just needs to be small enough, otherwise the noise (E) will contaminate points where the quadratic distribution is a good approximation with the points where it is bad one.
We derive the formulas for the CDF and PDF. We will show them for minimization.
10 reasoning also applies discrete hyperparameters, which essentially multiply local minima.
21

---

Published as a conference paper at COLM 2025
First, will need the definition of the partial expectation from to b:
E = (a s 2 E S s b] J." =fz(z)dz
(25)
Now, we will prove the formulas. Proposition F.3. Let Y Q + with Q 2min (a, B, 7) and E. N((,,)). IfF is the CDF
of Y then:
Fr(1)= Φ
Vite
V~N y y- a B- a B a
Proof. Let FV) denote the CDF of Q. Since Y is the sum of two independent random
variables, we can apply the convolution formula for the CDF of sum: Fr(y) = E(Fo(y E)].
Note that this expectation is taken over the normal variable, E. Recall:
y < a
7/2
Foly)
y S B
y> B
Then, using properties of expectations, we have:
Fr(y) = EF[o(y - -E)]
= EE B [1]] + EEY y-
( (y
(W--E)-
E) - a
1/2
+ E y-a00 [0]
B - a
y s 0
By- y-
[(y - a)
7/2
where ④ the standard normal distribution's CDF. Applying the change of variables:
y a) E
B &
and noting that V is normally distributed with mean (y (y--a) aB((B a) and standard deviation o/ / - a), we obtain the desired formula:
Frly) ②
y -
B-BB
E
Vi
V~N(
y
(26)
B a B - a
Proposition E44 Let Y Q E, with Q ~ Qmin (a, B,7) and ~ ~N(00,)). If fy
is the PDF
Y then:
7 E V fr(y) 2(B a)
y y-a 0
N~Wi
B
a B
Proof. Let fo(y) denote the PDF of Q. By the convolution formula for the PDF of
have: fy()) = Eylo(y E)). Note that this expectation is taken over the normal variable, E.
Recall:
0
y
fo(v)
a y S
2(p-a]
B-a
0
y> B
22

---

Published as a conference paper at COLM 2025
Then, using properties of expectations, we have: fr(y) = Ey E]]
4
E _10) + +E B
7
(y E) a
+
E 00
0
2(B a)
y-a
122
7 B.y- 2(B a)
(y - a)
B-B
a
Applying the change of variables:
a) E
B - a
and noting that normally distributed with mean a a) and standard deviation a/ (B- a), obtain the desired formula:
fr( 2(6 a)
7
E
V V~N
y--
(27)
B
B
23


Reviews:
[{"review_id": "ueOJABEE9F", "reviewer": "Reviewer_rVAh", "summary": "This submission seems to propose an analytical model of how the best (validation) score for a model is distributed in the neighbourhood of the optimal hyperparameters. Experimental results seem to produce reasonably accurate fit to the theoretical predictions.\n\nThe main concern with the manuscript is that it is very unclear regarding not only what is done, but even what the actual goal is. The starting point is a study of the hyperparameter loss surface. In a typical learning setup, model parameters are estimated by fitting parameters for a given set of hyperparameters, the suitability of which is assessed by computing some empirical estimate of the expected risk, such as a validation loss, hence the \"hyperparameter loss surface\", or high-dimensional surface of the validation loss over the space spanned by the hyperparameters. (If this is not what is intended here, then a central point has been completely lost on this reviewer.) Yet most of the remainder of the paper seems concerned with characterizing the (one-dimensional) distribution of the loss values near their extremum, rather than the high-dimensional loss surface.\n\nThe setup relies on (i.i.d.) random hyperparameter search, which provides a convenient theoretical framework for deriving extremum distributions. However this does not seem a very realistic setting for h.p. tuning, where one would typically expect prior hyperparameter scores to inform h.p. sampling, breaking the i.i.d. assumption. Similarly to \"random search\", many concepts and techniques are not described and not referenced, and left for the reader to figure out. To give one example: the \"censored maximum spacing estimation\" used to fit one of the distributions (Fig. 3, caption).\n\nThis may be novel and original work, but the significance is entirely unclear. It would of course be highly relevant and useful if it provided an improved method for tuning hyperparameters, or at least help produce better hyperparameters. This seems to be what Section 4.3 is promising; unfortunately that section shows more comparison between empirical estimates and theoretical predictions.", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This submission seems reasonably successful at modelling whatever it is they are attempting to model.", "reasons_to_reject": "Although this submission is well written overall, the main body of the paper lacks clarity and specificity to the point that it is unclear what is done exactly, and whether and how this can have any practical use.", "questions_to_authors": "l. 129, \"retrain the same hyperparameters\" : presumably you mean retrain the model using the same hyperparameters?", "extra_scores": {}}, {"review_id": "9LWQFbC5TH", "reviewer": "Reviewer_tyYN", "summary": "This paper describes experimental and theoretical results pertaining to hyperparameter optimization. They find that the loss curve is well approximated by a quadratic in the vicinity of the maximum value. Based on these results, they propose methods for estimating confidence intervals for the loss and a \"scaling law\" for predicting the best possible loss on the basis of relatively few (possibly very expensive) steps of random search. \n\nThis is (in my opinion) a fairly small result with limited applications, and it's not surprising to anyone with experience in training neural nets. However, their empirical evaluation and theoretical framing put that intuition on a more solid footing and may be something that later work can build on.", "questions": "", "limitations": "", "rating": 7, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Sensitivity to hyperparameter settings is an ongoing challenge for evaluating large neural nets. Anything that can help quantify the variation introduced by hyperparameter search is potentially very useful.", "reasons_to_reject": "A small point: I'm not sure I'd describe the proposal here as a scaling law, since there's not really any actual scaling involved. Maybe \"quadratic bound\" or \"predicted maximum\" or something like that would be a better term for it?\n\nMore generally: there's a gigantic literature on hyperparameter optimization and loss surfaces in deep learning context and in non-local optimization in general. It would be beyond the scope of a paper like this to try to survey it, but it would be helpful if the connections and overlap with tthose other results in the \"related work\" section.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "w7cmoWl0eF", "reviewer": "Reviewer_rHdb", "summary": "The paper introduces a noisy‑quadratic parametric model that captures how random search improves validation loss once hyper‑parameters are near the optimum. Experiments on three deep‑learning setups show that this simple model fits the tail distribution well and extrapolates tuning‑curve performance.", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper proposed a principled model for tuning‑curve forecasts with random search, with support of empirical validations.\n- The paper is well structured and clearly written.", "reasons_to_reject": "In order to confirm the paper’s findings, additional experiments should be conducted to cover more diverse search spaces. Because the conference focuses on language modeling, extensive evaluations with recent language models of various sizes and types would be more appreciated.\n\nThe paper investigates only random search and claims that the proposed model can be used to build tools for analysis. However, because the analysis is limited to random search, other common HPO methods (e.g., Bayesian optimization, BOHB [1]) cannot benefit from it.\n\n[1] Falkner, Stefan, Aaron Klein, and Frank Hutter. \"BOHB: Robust and efficient hyperparameter optimization at scale.\" International conference on machine learning. PMLR, 2018.", "questions_to_authors": "See Reasons to reject.", "extra_scores": {}}, {"review_id": "by5N8U4Ow3", "reviewer": "Reviewer_DSqh", "summary": "This paper proposes a new theory about hyperparameter tuning, that is, near the optimal point, the loss landscape of the hyperparameters are noisy quadratic, therefore one may apply Taylor approximation or other similar techniques to fit the hyperparameter space, which may help find better hyperparameters. \n\nI think the paper has many problems. I have two main questions:\n1. When does the theory hold? Does it hold universally?\n2. If the theory holds, is the method proposed in this paper really optimal?\n\nSpecifically, I think the theory does not hold in general:\n1(a). Generally hyperparameters can be arbitrary, so we are really dealing with a blackbox function. It's hard for me to believe all blackbox functions are noisy quadratic near optimal. \n1(b). I think the paper mainly discussed about continuous hyperparameters. Indeed, in the experiment, most parameters they pick are continuous. What if there are discrete parameters? How do you do Taylor expansion and fit it? Or, how is Taylor expansion defined?\n1(c). I think the authors mainly discussed some training task like training LLM/Resnet. However, it is not clear to me that, why this will hold for other ML tasks. I am asking about this because years ago, there were a crazy trend about using RL to find hyperparameters, but it turns out the RL algorithm will overfit to the training scenarios easily. Now, the authors did not convince me that this paper is not a similar story: maybe the noisy quadratic case only holds for the parameters they pick on the task they chose, not for other more general cases. \n\n2(a). Even if the theory is indeed true for most cases in AI, I still doubt the noisy quadratic theory is useful. If the landscape is noisy quadratic, it seems that Bayesian methods will be very useful here, because Bayesian methods will assume the function is close to Gaussian, where noisy quadratic is indeed close to Gaussian prior. Therefore, I believe using Bayesian will be as effective. So some comparison is necessary. \n2(b). Similarly, since most parameters in the experiment are continuous, I am thinking something like stochastic gradient descent on hyperparameters may help. See e.g., [1] Maybe the authors should compare with that work as well?\n\nQuality: Low. I think the paper is not complete right now. It still has many important discussions missing. \nClarity: Good, the paper is easy to follow. \nOriginality: Low. I do not think noisy quadratic theory for hyperpameter tuning is specifically novel or useful. \nSignificance: Low. I think more comparison with existing work, and discussion of generality is requried. \n\n\n[1] Gradient-based Hyperparameter Optimization through Reversible Learning, Maclaurin et al.", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "I think the paper cannot be accepted at the current form.", "reasons_to_reject": "Please check my summary above. I think there are mainly two reasons:\n1. The theory does not seem to be general. And the paper did not provide a nice characterization of when this theory holds. \n2. If the theory holds, it is not clear to me that the method provided in the paper is the optimal one. I think there are many classical methods can do well, and more comparisons are needed.", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents a theory about the structure of hyperparameter loss surfaces near their optima, proposing that they follow a noisy quadratic distribution. The authors validate this across multiple architectures and tasks, and develop statistical tools for uncertainty quantification in hyperparameter experiments.

The reviews split in terms of enthysiasm especially wrt understanding the paper's contributions. Reviewer tyYN (score: 7) appreciated that this work is about understanding loss surface structure for experimental design, not proposing a new opt algorithm. They note it's "a fairly small result" but one that "put[s] that intuition on a more solid footing." In contrast, three other reviewers (scores: 4, 4, 4) appreciated that this was not the main goal of the paper, critiquing it for not offering a better (and more cocnrete) hyperparameter/optimization method and that it is incomplete (i.e., lacking enough empirical evidence to support some of the hypotheses)

Strenghts:

New (according to some revs) theoretical contribution characterizing hyperparameter loss surfaces as noisy quadratic near optima
Solid empirical validation across 3,000+ training runs spanning multiple architectures, modalities, and tasks
Provides practical guidance for uncertainty quantification
Well-written
Weaknesses:

The paper fails to clearly communicate its goals early on, leading to widespread reviewer confusion
Limited discussion of when the theory might break down. Some reviewers are unconvinced about generalizability across arbitrary setups
Unclear practical impact
Some terminology choices (e.g., "scaling law") may be misleading
The core issue here is not the technical merit but perhaps communication. As tyYN notes: "this paper seems to be about one thing... but it's actually about something completely different... The fact that it confused all the reviewers so much is a sign that it could be making its point in a better way."

The authors' rebuttal clarifies to a degree their actual contributions, but it's unclear if the reviewers agree even in light of it. This is challenging work to evaluate because it addresses an important but perhaps not super familiar problem. The empirical work is thorough, the theory appears interesting and sound, and the tools could be useful for the community. However, it is *extremely critical* that the authors make their goals extremely clear from the beginning, taking into account the reviews and the meta-review, and significantly revise the presentation.
Decision: Accept