Thumb on the Scale: Optimal Loss Weighting in Last Layer Retraining
Authors: Nathan Stromberg
Christos Thrampoulidis
Lalitha Sankar
Venue: NeurIPS 2025 poster ()
Keywords: last layer retraining
imbalance
upweighting
downsampling
overparameterization
Primary area: general_machine_learning

Abstract: While machine learning models become more capable in discriminative tasks at scale, their ability to overcome biases introduced by training data has come under increasing scrutiny. Previous results suggest that there are two extremes of parameterization with very different behaviors: the population (underparameterized) setting where loss weighting is optimal and the separable overparameterized setting where loss weighting is ineffective at ensuring equal performance across classes. This work explores the regime of last layer retraining (LLR) in which the unseen limited (retraining) data is frequently inseparable and the model proportionately sized, falling between the two aforementioned extremes. We show, in theory and practice, that loss weighting is still effective in this regime, but that these weights _must_ take into account the relative overparameterization of the model.

Thumb on the Scale: Optimal Loss Weighting in Last Layer Retraining
Nathan Stromberg Arizona State University nstrombe@asu. edu
Christos Thrampoulidis University of British Columbia cthrampo@ece ubc.
Lalitha Sankar Arizona State University 1sankar@asu. edu
Abstract
While machine learning models become more capable in discriminative tasks scale, their ability to overcome biases introduced by training data has come under increasing scrutiny. Previous results suggest that there are two extremes of param- eterization with very different behaviors: the population (underparameterized) set- ting where loss weighting is optimal and the separable overparameterized setting where loss weighting is ineffective at ensuring equal performance across classes. This work explores the regime of last layer retraining (LLR) in which the unseen limited (retraining) data is frequently inseparable and the model proportionately sized, falling between the two aforementioned extremes. We show, in theory and practice. that loss weighting is still effective in this regime, but that these weights must take into account the relative overparameterization of the model.
Introduction
While discriminative machine learning has produced exceptional predictive power in many settings, even the most expressive models have struggled to balance the accuracy of common (majority) classes and rare (minority) classes. This comes the most critical tasks, such disease predic- tion, fraud detection, and rare event monitoring are fundamentally extremely imbalanced. It has, therefore, become critical correct the existing majority bias even in the age of highly expressive, overparameterized models.
Substantial research has gone into studying methods for correcting models for their majority bias, and the most successful lines of research have been driven by the idea of last layer retraining (LLR) [I] 2 By updating only the final linear layer of large model, the number of parameters to train is greatly reduced while still providing the flexibility to balance the accuracy for minor- ity classes. In this setting, a variety of cost-sensitive methods have been investigated, including weighted empirical risk minimization (WERM) sometimes called importance weighting (3) down- sampling 4 3 and other corrected losses 5.10 Each of these methods has shown compelling empirical success in the LLR setting, improving the accuracy of minority classes or groups to match that of majority groups. Despite this, theoretical and empirical evidence has shown that, for over- parameterized models, WERM has no effect on the learned model whatsoever | 12]. How do we reconcile this seeming contradiction when are correcting large models?
Last layer retraining is often characterized by two quantities, the latent data dimension (in- put to the last layer) and the number of retraining samples The overparameterizzttoo ratio 8 d/n e (0, oo) has been used breakdown the performance of the above-mentioned meth- ods into two critical regimes: the populations setting (8 -> 0) and the overparameterized setting (6 > 1). For example, in the population setting, the methods described above for LLR have demon- strated both empirical and theoretical success in the population regime [ On the other hand, only downsampling and CS-SVM have been successful in improving minority class accuracy when the model is highly overparameterized, i.e., 8 > 1, and can separate the data 9 10 In this work, we
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

consider the ratio ò in the understudied but highly relevant regime of 6 E (0, 1). This underparam- eterized regime is critical because, in practice, the number of samples is often on the same order as the number of parameters in the last layer retraining scenario.
In this underparameterized regime, we study the problem of obtaining loss weights which minimize worst-class error (WCE) for binary classification. Our key contributions in this setting are as follows:
We simplify the wERM optimization with any loss on sample from a class-conditional Gaussian distribution to general system of four scalar equations for the setting where both d, oo
For the setting above and the choice of square loss, we derive optimal asymptotic weighting as a function of the overparameterization ratio d.
We compare this optimal weighting scheme to downsampling and show that optimal wERM out- performs downsampling, especially when data is limited.
Finally, we show that the trends described for simple Gaussian data in theory appear in real-world image classification problems and that our optimal weighting scheme outperform the classical ratio of priors by leveraging the notion of an effective (latent) dimension.
1.1 Related Work
Importance Weighting in Separable Settings While wERM has been successful in practice, its applicability in high-dimensional problems has been called into question. Soudry et al. [I3]. Ji and Telgarsky [14), Gunasekar et al. [1S] explore implicit bias of gradient descent-type methods on linear models and characterize the limiting model for a variety of loss families, including square and logistic losses. Xu al. [16] explicitly consider loss weighting in the implicit bias and find that importance weighting on common losses does not alter the learned model, but can improve the convergence rate in imbalanced settings. Byrd and Lipton (II), Sagawa al. [T7] explore the problem importance weighting with a focus on deep models and find that larger models are less affected by loss weighting than smaller models.
Overparameterized Learning Kini al. [9, Behnia al. [8] study CS-SVM in the overparam- eterized regime and show that in this setting the ineffectiveness of loss weighting can be overcome by considering an altered loss, namely the vector scaling loss. Their theoretical results help to ex- plain important behaviors for 5 >> 1. Lyu et al. g study a similar loss for å > O and find that even when 8 < 1, the margin-scaled loss can be helpful for mitigating model biases. Lai and Muthukumar 10 study margin-ade adjjited minimum norm interpolation in the overparameterized regime and show that this can help to mitigate poor out-of-distribution generalization. Our work instead considers loss weighting (as opposed to margin weighting) in the underparameterized regime.
Downsampling Chaudhuri et al. study downsampling in the context of separable SVM and work with fixed d while scaling up the class means as n - co. Their work utilizes extreme value theory and finds that downsampling can improve WCE for several data distributions. Several works empirically utilize downsampling in the LLR setting to improve subgroup fairesss including Kirichenko et al. [1. LaBonte et al. [19]. Stromberg et al. (20). Our work focuses weighted learning on inseparable dala and the proportional regime of 8 E (0, 1).
Population Risk Minimization In the population setting, Welfert al. 3 study wERM, down- sampling. and Mp. They show that the first two are equivalent problems, and that all three result We instead focus both d, n - at fixed ratio which
in the same solution given can better explain practical settings like LLR where the number of data points and the number of trainable parameters
the same order.
2 Problem Setup
We denote deterministic vectors as and their random counterparts X. Matrices are denoted X and their randomness clear by context. We begin by defining the generative model and key functions of interest.

---

2.1 Data and Metrics
We assume the following generative model for the latent data and label Y:
w.p. x
Y
w.p. #
X = v) ~ (yp,
with /A E I6 and I E Rddd the identity matrix. Without loss of generality, assume that 7 so +1 is the minority class, Assume we get n samples as (X, y) E RRaxd x I following the above distribution.
The key metric studied in this work is worst-class error (WCE) which provides notion of fairness for a given classifier. We can calculate the expected risk (error) on each class for a given linear model (Q, b) follows:
R. Q
73 O
6
R_
-
Q
78
b a
T E R is the energy of @ along 14, a & ||@|| E the total energy of 0, and Q(-) the standard Q-function. More details can be found in Appendix A.I These three scalar quantities and the bias will be the focus of our analysis.
where: - |ليا|| E R is the signal strength,
Tgl
We can then calculate WCE as follows:
WCE(0, & maxxRR+ R-}
(2)
the sequel, we will make use the following function class: the Moreau envelope of a function f is parameterized by À and defined as
My A) a inf f(v) + 1|u x
x11
(3)
2A
and can be thought of locally smoothed version of f. always continuously differentiable
with respect to and X. The that achieves the infimum above is given by the prox operator. We will also make use of the partial derivatives of M which we denote as (1,1x;; A) and (2z X)
for the partials with respect the first and second arguments respectively. The second derivatives are denoted similarly. See Appendix f.r for a full definition.
Asymptotic Analysis of Cost-Sensitive Last Layer Methods
We will now discuss our key theoretical results for both weighted and downsampled ERM in the asymptotic setting where both d, n + oo such that ê = d/n E (0, 1). Furthermore, we specialize in the analysis for square loss for which we can derive closed form solutions.
3.1 Weighted ERM
We consider the weighted ERM problem
min 0,
Σ wil (y.(z, 0 b))
(4)
where é the margin- based loss function of interest and only required to be convex in its argu- ment. We consider the special case of class weighting where, recalling that class + 1 is the minority class, we have
Vi -1
W P. Yi = +1.
(5)
Although weighting of groups or other subsets of the data fits into our framework, we focus on class weighting for clarity. We denote the solution to this problem by (0(p), d(p)). Note that we can reparameterize this solution (á(p), (p)), b(p)) with a and y defined in section 2.1
We begin by showing how this ddimennaiisi optimization can be reduced the solution of system of scalar equations.

---

Theorem 1 (Weighted ERM Solution), For 8 E 10. and proper. convex loss & effectively reduced a four dimensional system of equations in a, V. X, b given by
4
can be
8(a² - 7) + 2X 2 ( + b; X)]
+2X² -G p/p]
(6a)
87p
X n ps ( ( b; A)]
t SEIM D0 6; A/p) 0
(6b)
dap - x 000 ks + b; A)]
+ ( ( b; A/ =0
(6c)
A/p)) =
(6d)
with G (0, Concretely, when the solution (a","" , %_,xx,b,, b*) to this system of equations is unique, then it satisfies (å, 5, b) (a", ,,","""")))))
) in probability as n, - oo with d/n = 8 fixed.
The proof is presented in Appendix A.4 and relies on the convex Gaussian minimax theorem further in (23].
(CGMT) 21 [2) which is discussed in Appendix
The expected Moreau envelope of can be estimated from samples, allowing the system in 66 to be efficiently solved using numerical methods like fixed-point iteration. For square loss, however, the Moreau envelope admits closed form
(xr 1)
MEquam (I; A)
2 1 A
allowing us to reduce the above system and remove the randomness.
A { - 1)? we have the following set of Corollary (Square Loss). Letting ((2) = l squuare (2)
equations as a simplification of 6
2 + (1 + 2)2 (187 + (18
X²
1)2 + a²
+ π
(187 - b- 1)2 + a²) 8(a² - -22)
(8a)
(p Aj2
X
847 + b- 1 1)
+ n 5(87 - b 1)
= - 8y
(8b)
1 + x
X
X
T+ + X
(8c)
0 + X
87 + b
87
0
(8d)
1+
p p X
and additionally this system has unique solution.
Proof Sketch. We the closed form for the Moreau envelope of the square loss and evaluate all of the expectations utilizing the known data distribution. The uniqueness of the solution follows from the joint convexity of the combined expected Moreau envelopes. A full proof provided in Appendix A.5
While this system of equations is significantly simpler than that of 66 it still does not admit closed-form solution for general weighting. However, certain special cases including unweighted ERM allow for closed-form solution. Substituting in (8), we obtain the following corollary:

---

1, + 0.2,5 =
p p= 1, 1, 0.3,, -
0.5-
0.4
0.4
Class
+1
Enror
0.3
Error
0.3
Method Simulation
Class
0.2
Theory
0.2
-1
Method
0.1
Simulation Theory
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
(a) 44
0.2
(b) π
0.3
Figure 1: Plot of per-class error function of 5 on class-conditional Gaussian dataset. We see that simulation matches the theoretical closed form across all 5 values and different imbalances т.. Both classes perform more poorly for growing overparameterivatton. The majority always outperforms the minority, however.
Corollary (Unweighted Solution). Letting p = 1, we get the following solution to (
>- s(1 (T -T )2)
(9a)
1 (2 (T- πT )2)
X* 8
(9b)
1
8" = (T_ - (4)(7"s-1)
(9c)
a" (T 6 1)2 1 6- 1-))) 7=2
(9d)
Similar results have appeared in the literature 24 and we include it here for completeness.
The solution in Corollary 2 captures the effect of overparameterizztion through d. We visualize this effect in Figure where we also that the small regime strongly favors the majority whereas as + 1, the difference between classes dissipates (at the cost of increasing WCE). Its behavior as function of T4 is shown Figure
An interesting case is when we want to equalize the error of each class. From we obtain this by setting b 0; this, in turn, corresponds to choosing specific which yields b* 0. Starting from Corollary we obtain the following result for the equal error settings.
Theorem (Equal Error Solution). Assuming 2 p = p defined as
π
+
1
(10)
2T
Then, the solution given by
6"
(11a)
(11b)
I 1 s2
A
(7* 1)2 x²
(1lc)
where we define A = using p and square loss as
5 for convenience. We write the worst-class error wERM
4x_
52 V
Q
(12)
V
5

---

6 0.2, R. =02,s=1
1.0
Class
.1
0.8
Method
Simulation Theory
Enor
0.4
0.2
0.0
0.0
2.5
5.0
7.5
10.0
12.5
15.0
17.5
20.0
Figure 2: Per-class test error plotted against the weight ratio P a synthetic dataset. We see that only does our theoretical formulation match the simulation results, f (red) strongly outperforms the conventional ratio of priors (black). Note that the worst-class error for each weighting scheme is marked with a horizontal line.
Proof. Substituting 7 into g we can solve for y, o. To get the WCE we substitute into
We see that the form of p is the classical ratio of priors plus an offset that depends only on the overparameterization ratio ó and the minority prior πạ. Note that this offiet is 0 for balanced classes regardless of parameterization.
We see in Figure that aligns with the crossover of the per-class errors in theory and in simulation of a caass-conditinnl Gaussian with 2, 0.2, and 7 0.2. Additionally, we see the monotonicity required by Theorem 3 This results in (red) strongly outperforming the population- optimal weighting marked black. This suggests that using a stronger weight should helpful in practical settings where and are similar scales. We explore this further in section
worth noting that this f for WCE does not align with the conventional wisdom of the ratio of the priors except as + 0, which is the case that n - oo for a fixed d. This aligns with previous population results such [3. We compare the ratio of the priors f in Figure 2 and indeed see that the ratio of the priors is suboptimal for this 8 > 0.
We additionally see in Theorem 2 that the solution is only valid in the setting that 8 < 2n.. This an interesting restriction, as it suggests that overparameteriizaion affects the ability of weighting correct for class imbalances. Indeed risks never intersect as function of and thus the WCE is dominated only by the minority class. In this setting, the optimal choice of weighting is co.
in Figure 3b that when 5 2 the per-class
With the assumption that the per-class risks are monotonic in p (which holds in our simulations), this choice of is optimal in terms of WCE. While the assumption above may seem strong, it is intuitive that increasing the weight on class should decrease the risk for that class and increase the risk for the opposite class.
Theorem (Optimality of p). Assume that R4 and R_ are monotonically decreasing and increas- ing respectively in the weight parameter p. Then P minimizes WCE over all choices of p for given and 8 < 25
Proof. For convenience, overload WCE to be function of weight p directly. If both per-class risks are monotonic in p. WCE quasiconvex function in p as it the maximum of two monotonic functions. The choice of ő is a local minima of WCE for any >> 0, WCE(p E) 2 WCE( as R_ is increasing and WCE( <) 2 WCE(p) because R4 is decreasing. Thus ý is a global minima of WCE by quasiconvexity.
3.2 Downsampled ERM
While upweighting is one common form of cost-sensitive correction during I.I.R, another is down- sampling. We specifically consider the variant of downsampling where the majority class is down-

---

7+ =0.2, =02,
R =02,5 0.
1.0
1.0
Class
0.8
0.8
Emor
0.6
Error
0.6
0.4
0.4
0.2
0.2
2
8
10
P
2T
(b) 8 2T
Figure 3: Effect of on the per-class test error a synthetic dataset. We see the restriction for 8 2 for ý (red) to be defined has an operational meaning regarding the per-class errors. Indeed if 5 > 2T,, the per-class errors never meet and thus cannot be balanced.
p= 1, 0=1,5=11
17 5
Class
+1
0.45
0.8
-1
Method
0.40
Method
0.6
Simulation Theary
0.35
WERM
Emor
Emror
DS
0.30
0.4
0.25
0.2
0.20
0.0
0.15
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.3
0.4
Figure 4: Per-class errors for unweighted ERM. We first note that simulation using SGD matches the theoretical risk across all क val- ues. The effect of imbalance is quite extreme and the minority error quickly skyrockets.
Figure 5: A comparison of wERM using the op- timal weight ý and downsampling. We see that for fixed 8, the error for wERM much lower than that of downsampling but that both achieve balanced per-class errors.
sampled to the size of the minority class. We can adapt our existing results to capture the downsam- pling problem change of (therefore d) and priors.
Corollary 3 (Downsampling). The solution the downsampled problem is given by taking 5 & 5
2
for 8 in Theorem and setting x = T 1/ and 1. For square loss, the closed form formfollows from Corollary
We see an illustration of the relative strengths of these methods in Figure 5 which shows that the increased effective overparameterization for downsampling results in much higher error for both classes. Note that as 8 approaches O, the two methods perform equally. This aligns with the the- ory of cost-sensitive population risk minimization B]. The increasing gap between wERM and downsampling is indicative of the strength of wERM as overparameterization increases. This is
particularly important for highly imbalanced datasets as the effective overparameterrizaioo used in downsampling grows as 1.
7

---

Backbone: resnet34, Latent Dim: 256, n: 1000
Backbone: resnet34, Latent Dim: 256, n: 20
C Class
Class
0.8
-1
0.6
Emrr
0.6
Emor
0.4
0.2
0.0
2
8
2
4
10
(a) 1000
(b) 20
Figure 6: Per-class accuracies for binary hair texture classification on the CelebA dataset, with the classical ratio of the priors P marked in black and our á in red. We that the number of samples (thus d) affects the empirically optimal weighting, with larger requiring a larger weight on the minority class. This aligns with the theoretical results of Theorem 2
Application to Imbalanced Image Classification
We have presented a theoretical framework for understanding how overparameterization affects the cost-sensitive learning methodologies, and next explore this effect in practice for binary image classification tasks. Additional plots can be found in Appendix B
All relevant hyperparameters are
discussed in Appendix C
We consider the CelebA [25] dataset which consists of images of celebrity faces, each marked with 40 binary attributes. We select Straight Hair the attribute of interest with takes the value 1 in 21% samples and the value 79% of samples. We also consider binary version of CIFARIO 26 with artificial imbalance, where class - is truck with 91 % of the data and +1 is airplane with 9% of the data. We consider other classes and imbalance ratios within CIFAR1O in Appendix and find qualitatively similar behavior.
We finetune ResNet34 model on the training split each dataset using cross entropy loss before retraining the final layer with varying p on the validation split with square loss (aligning with theory). The focus on square loss may seem like major restriction, but practice, square loss is often as performant cross entropy loss, especially in the low-data fine-tuning setting (27 28). To simulate different å values, since the model size fixed for LLR, we subsample the validation data uniformly to size For each n, this retraining is repeated 10 times with different subsamples to get confidence intervals on the captured metrics. We note that CIFARIO does not provide validation split, so we create one from fixed 10% split of the test data. For this reason, only consider up to 90 for CIFAR10.
We note that while we choose the size of the latent space, this is not the d that is used for calculating We observe practice that the majority of the features in the latent space are irrelevant (see Figure 9 Appendix [ for PCA spectra). As such, we perform PCA to quantify the number of "effective" dimensions in the data as the number of features which capture 99% of the variance. This is dimensions for CelebA and 2 for CIFAR In general, the number of effective dimensions will be a function of task difficulty and model architecture. The relatively poor usage of the latent space is a core observation of Kirichenko et al. [ and motivates sparse LLR. Here, quantify the level of sparsity to better select importance weights.
We see in titt while the أ calculated using the effective dimension does not perfectly predict optimal empirical p. captures the correct behavior as n shrinks (and therefore increases). Note that for 1000, the population-optimal weights are not empirically optimal, likely due to a small shift from the validation data to the test data or inadequate model generalization. For n 20, distinctly larger than the ratio of the priors and achieves much better WCE.

---

Backbone: resnet34, Latent Dim: 512, n: 90
Backbone: resnet34, Latent Dim: 512, n: 20
Class
Class
+1
0.35
+1
0.30
-1
0.25
Error
0.3
Eroo
0.20
0.15
0.2
0.10
0.05
0.0
0.00
10
20
30
40
50
10
20
30
40
50
P
P
(a) 90
(b) 20
Figure 7: Per-class accuracies for binary classification on the CIFAR10 dataset (planes vs trucks), with the classical ratio of the priors p marked in black and our ý in red. We that the number of samples (thus 5) affects the empirically optimal weighting, with larger 5 requiring a larger weight on the minority class. This aligns with the theoretical results of Theorem
In Figure 7 for the CIFARIO dataset, we see that أ calculated from the effective latent dimension (red) predicts the empirical optimal p quite well, and is certainly better in terms WCE than the classical ratio of the priors (black). While there is significant noise with only n 20 retraining samples (so only or minority examples), weighting with أ allows us to recover a more balanced classifier.
that the optimal weighting scheme is effective across different datasets and over- parameterization levels, including other choices class for CIFAR 10 (see Appendix [B. This sug- gests that our method is quite general and could be useful in practice, acting as default weighting rather than the classical ratio of the priors.
Ultimately,
5 Limitations and Broader Impacts
Limitations While common to assume that the latent space of deep model is Gaussian, more sophisticated mixture could be useful in explaining class-imbalanced learning under spurious correlation related settings. Our analysis is also limited isotropic noise, but we hope to extend this in future works. Despite these assumptions in the analysis, we see that in real datasets, where neither of these hold, the derived أ can still useful in selecting an appropriate weighting during last layer retraining.
While our provided analysis is able to effectively capture the behavior of real models under last layer retraining with square loss, providing clear prescription for weighting depends on the "effective" dimension of the latent data. We utilize PCA, but this is rather heuristic. A weighting methodol- ogy which learns this effective dimension in more principled manner could increase the practical application of our findings and is major focus our future work.
Multiclass Classification While we limit our study to binary classification for the theoretical con- tributions, there has been recent work [29 30 exploring similar technical tools in the multiclass setting. This is beyond the technical scope of our work, but is a promising direction for future work. We conjecture that the intuition developed with our theory will prove useful even in the more complex multiclass setting. Specifically, we expect that for small n, the weights needed to reach balanced accuracy will be more extreme (favoring the smallest class) than the naive ratio of priors suggests.
Broader impacts There ongoing discussion about whether improving the performance of mi- nority classes at the expense of majority classes is desirable. However, in critical applications such as disease or rare event detection, our weighting scheme could result in significant gains on rare classes which often have outsized real-world impact.

---

Conclusion
In this work, have demonstrated the efficacy of loss weighting in the last layer retraining set- ting and derived an optimal weighting scheme which takes into account overparameterizatton. Our results close wide gap in the literature between the population (many samples) setting and the overparameterized (few samples) setting and provide new insights into this highly relevant regime.
In practice, show that this novel weighting scheme outperforms the classical ratio of priors in vi- sion tasks, but that different notion of dimension needs to be taken into account. This compensates for the fact that many dimensions of the last layer of large models go unutilized, core observation of previous works. Our prescription for loss weighting is general in that it outperforms the ratio of the priors across datasets, latent dimensions, and imbalance ratios in real-world settings. Our find- ings not only challenge the assumption that the classical ratio of priors is optimal but also provides a pathway towards more balanced model retraining by leveraging this effective dimension. We be- lieve these insights will extend the benefits of optimal loss weighting to more complex real-world fine-tuning applications.
10

---

References
[1] Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson, Last Layer Re-Training is Sufficient for Robustness to Spurious Correlations. In The Eleventh International Conference on Learning Representa- tions, 2023, URL htt.ht:tee///eeeeeeiee
[2] Pavel Izmailov, Polina Kirichenko, Nate Gruver, and Andrew Gordon Wilson On Feature Learning in the Presence of Spurious Correlations, October 2022. URL htttp://arxiv..rg//// arXiv:2210. [es].
11369
[3] Monica Welfert, Nathan Stromberg, and Lalitha Sankar. Theoretical guarantees of data augmented last layer retraining methods. In 2024 IEEE International Symposiun on Information Theory (ISIT), pages 581 586. IEEE, 2024.
[4] Kamalika Chaudhuri, Kartik Ahuja, Martin Arjovsky, and David Lopez-Paz. Why does throwing away data improve worst-group error? In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara En- gelhardt. Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 4144-4188. PMLR, 23-29 Jul 2023.
[5] Aditya Krishna Menon. Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and San- jiv Kumar. Long-tail learning via logit adjustment, July 2021. UBLL//ttp///axxii.orr//bbb////
07314 arXiv:2007.073 [es, stat].
[6] Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning Imbalanced Datasets with Sabel-Distribution-Anee Aware Margin Loss, Octoher 2019. URI. httt::/arxivv http://axiivvorrraaa//aaaa/
07413 arXiv:1906.07413 [es, stat].
[7] Haotian Ye, Chuanlong Xie, Tianle Cai, Ruichen Li, Zhenguo Li, and Liwei Wang. Towards a Theoretical Framework of Dfitt--Distributiin Generalization. November 2021. /ttppa://operreview net/
forum?id /zuDVV
[8] Jingyang Lyu, Kangjie Zhou, and Yigiao Zhong. A statistical theory of overfitting for imbalanced classi- fication, February 2025. URL http: org at////bs////2// 11323 arXiv:2502. 11323 [math].
[9] Ganesh Ramachandra Kini, Orestis Paraskevas, Samet Oymak, and Christos Thrampoulidis, Label- imbalanced and group-sensitive classification under overparameterizatton. Advances in Neural Infor- mation Processing Systems, 34:18970-18983, 2021.
[10] Kuo-Wei Lai and Vidya Muthukumar. Sharp analysis of out-of-distributtin error for "importance- weighted" estimators in the overparameterized regime. In 2024 IEEE International Symposium In- formation Theory (ISIT), pages 3701-3706, 2024. doi: 10.1109//11777644 80.88644.22222..2 1061922......
[1I] Jonathon Byrd and Zachary Lipton. What is the effect of importance weighting in deep learning? In Ka- malika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 872 881. PMLR, 09 15 Jun 2019.
[12] Da Xu, Yuting Ye, and Chuanwei Ruan. Understanding the role of importance weighting for deep learn- ing. International Conference on Learning Representations. 2021.
[13] Daniel Soudry, Flad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The implicit bias gradient descent on separable data. The Journal of Machine Learning Research, 19(1):2822-285 2878, January 2018 ISSN 1532-4435.
[14] Ziwei and Matus Telgarsky. The implicit bias of gradient descent on nonseparable data. In Proceedings of the Thiryy-Sccone Conference on Learning Theory, pages 1772 1798. PMLR, June 2019. URL/tttps: //proceedings.. mlr
html ISSN: 2640-34400 3498.
[15] Suriya Gunasekar, Jason Lec, Daniel Soudry, and Nathan Srebro. Characterizing Implicit Bias in Terms of Optimization Geometry. In Proceedings of the 35th International Conference on Machine Learning, pages 1832-1841. PMLR, July 2018. WRt. httttt:::/ppoeeeeeeeee ISSN: 2640-3498.
mlr press/v80 g/00/uunseeaar8aa
html
[16] Da Xu. Yuting Ye, and Chuanwei Ruan. Understanding the role of importance weighting for deep learn- ing, March 2021. URL http:////xxxiv.og/aaa/////// 15209 arXiv:2103.15200 [cs].
[17] Shiori Sagawa, Aditi Raghunathan, Pang Wei Koh, and Percy Liang, An Investigation of Why Overpa- rameterization Exacerbates Spurious Correlations, 2020. htt.:httt///attt harggabs/2005 04345
11

---

[18] Tina Behnia, Ke Wang, and Christos Thrampoulidis. On how to avoid exacerbating spurious correla- tions when models are overparameterized, June 2022. URL httt:://aaaiv.org/bbb////// 12739 arXiv:2206. 12739 [es].
[19] Tyler LaBonte. Vidya Muthukumar, and Abhishek Kumar. Towards last-layer retraining for group robust- ness with fewer annotations. Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[20] Nathan Stromberg, Rohan Ayyagari, Monica Welfert, Sanmi Koyejo, Richard Nock, and Lalitha Sankar, For robust worst-group accuracy, ignore group annotations. Transactions on Machine Learning Research, 2024. ISSN 2835-8856.
[21] Mihailo Stojnic. A framework to characterize performance of LASSO algorithms, March 2013. URL http: org/ aaaaab///111 1303.
7291 arXiv: [es].
[22] Christos Thrampoulidis, Samet Oymak, and Babak Hassibi. Regularized Linear Regression: A Precise Analysis of the Estimation Error. In Proceedings of The 28th Conference on Learning Theory, pages 1683-1709. PMLR, June 2015. URI htttps://proceeeingaas wtw.anng........s///00.. wwt.:peee//////eeeeeeeeee
idis15.
html
ISSN: 1938-7228.
[23] Christos Thrampoulidis, Ehsan Abbasi, and Babak Hassibi. Precise emrrr analysis of regularized estimators high dimensions. IEEE Transactions on Information Theory, 64(8):5592 5628, 2018.
[24] Francesca Mignacco, Florent Krzakala, Yuc Lu, Pierfrancesco Urbani, and Lenka Zdeborova. The Role of Regularization in Classification of High-dimensional Noisy Gaussian Mixture. Proceedings of the 37th International Conference on Machine Learning, pages 0874-6883, PMLR, November 2020. URL
https: //ppe//d///cccingga mlr, www.ppresaavvee v119/ mwgmmccoonaa...mmmmmmmmmmmmmm..m html ISSN: 2640-3498.
[25] Ziwei Liu, Ping Luo, Xiangang Wang, and Xiaoou Tang. Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Deep learning face attributes in the wild. In
[26] Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Techni- cal Report 0, University of Toronto, Toronto, Ontario, 2009. URL https: httppp:://ww... cs. cs. toronto. coronto.edu/
20.9-TR. pdf
[27] L Hui, Evaluation of neural architectures trained with square loss vs cross-entropy in classification tasks. In The Ninth International Conference Learning Representations (ICLR 2021), J, 2020.
[28] Alessandro Achille, Aditya Golatkar, Avinash Ravichandran, Marzia Polito, and Stefano Soatto. LQQF: Linear Quadratic Fine-Tuning In 202/ IEEF/CVF Conference on Computer Vision and Partern Recog- nition (CVPR), pages 15724-15734, Los Alamitos, CA, USA, June 2021, IEEE Computer Society. doi: 10.1 10.11099474444437.200 211.PP4463772221.01555 2021001477
[29] David Bosch and Ashkan Panahi. Novel Convex Gaussian Min Max Theorem for Repeated Features. In Proceedings of The 28th International Conference on Artificial Intelligence and Statistics, pages 3673- html
3681. PMLR, April 2025.
mlr ctress//258/boaciie
[30] Bruno Loureiro. Gabriele Sicuro, Cédric Gerbelot, Alessandro Pacco, Florent Krzakala, and Lenka Zde- borová. Learning Gaussian Mixtures with Generalised Linear Models: Precise Asymptotics in High- dimensions, December 2021. URL http://////axi org/ ab1// 03791 arXiv:2106.03791 [stat].
[31] Hossein Taheri, Ramtin Pedarsani, and Christos Thrampoulidis. Sharp Asymptotics and Optimal Perfor- mance for Inference in Binary Models, February 2020. URL http: http://aawivv..r//////./20.00. arXiv:2002.07284 [math].
12

---

1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer:
[Yes]
Justification: We demonstrate theoretically and empirically the dependence of optimal loss weighting on the level of underparametterization of the model.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
A limitations section is included.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The au- thors should reflect on how these assumptions might be violated practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the ap- proach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to ad- dress problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't t acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: Assumptions are given in theorem statements and proofs are provided in the appendix.
13

---

Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theo- rems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple- mented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: All necessary hyperparameters are listed in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducíbility can accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., g., in the case of a large language model), releasing of model checkpoint, other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or a way re- produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case au- thors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
14

---

Answer: [Yes]
Justification:
Code is provided SM and will be released publicly after the review period.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.....
onpolicy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: These details are included in the appendix and relavent sections of the main body.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Simulations are repeated 30 times for confidence, while empirical results are captured over 10 runs. Error bars are calculated using the seaborn library.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
15

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should prefer- ably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Provided in appendix experimental details.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g.. preliminary or failed experiments that didn't t make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
No human subjects, and potential harms are mentioned.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there is a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Mentioned in introduction and limitations.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
16

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility MI).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
No high risk models or data are released.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by re- quiring that users adhere usage guidelines or restrictions access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Datasets are cited and license is respected.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name the license (c.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use the pack- age should be provided. For popular datasets, wwwppsswiittaaaaaa whwode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the li- cense of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
17

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documenta- tion provided alongside the assets?
Answer: [NA]
Justification:
No assets released.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the pa- per include the full text instructions given to participants and sereenshots, applicable, well as details about compensation (if any)?
Answer: [NA]
Justification: No human subjects are utilized.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, cura- tion, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
No human subjects are utilized.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equiva- lent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
18

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: LLM usage is not non-standard.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy (https://neurips) cct//onfeeeencce///2///////// for what should or should not be described.
19

---

A Proofs
A.1 Proof of Risk in
I
Proof.
Note we can decompose X as
= Z, -~N ~~N(0.1)
(13)
or in matrix notation as
X = yp
+2, 2, i [n]
(14)
Then
R4 Pr(V(X
1)
(15)
0 2
(16)
=>
(17)
-r((lQ] nz 10)
(18)
Pr(Z's 7
b
(19)
a
Q( 7 b
(20)
where Z I) and Z' NO 1).
A.2
Technical Tool: CGMT
key technical tool in this work is the convex Gaussian minimax theorem, and predecessor Gordon's 's comparison lemma. We restate the key result here, but refer the reader to Thrampoulidis et al. [23 for a more detailed view on the problem.
Consider pair of primary and auxiliary optimization problems:
h max min S
G')
min max Gw + (/, u),
(21)
wESw yES,
|| u 11 (i w)
(22)
wES yES.
We will show that for certain random inputs, these two problems are equivalent.
Theorem 4 (CGMT (22). Let Φ, ó be defined as in 21 and (2) is a matrix with standard
Gaussian entries, and G. H N00,
(r < 2Pr (M <)
(23)
additionally if convex-conaave, then Ve E TR
Pr (x > c) S 2Pr (o H1 2))
(24)
That is, we can bound the performance of & by the comparatively simpler ø.
We will leverage this result study the comparatively simple auxiliary optimization problem through a series of scalarizations.
A.3 Properties of the Moreau Envelope
Lemma 1 (Properties of Moreau). Let é: IR + R be proper, closed, convex convex function. The Moreau envelope Me(x; X) is jointly convex in its arguments and the following hold:
M2 A) prox;(x; X)
(25)
A
M= 1 (Me 2))
(26)
For complete list of useful properties see for example Lemma D.I in (23).
20

---

A.4
Proof of Theorem
Note that we can rewrite
by the constrained optimization problem
min 2.b
w,l(u,)
(27)
1=1
s.t.
(= +
vi [n]
which then allows us write the problem min-max:
min
vulu ya. 0 yıd) wsl(uu).
(28)
2,0,0
From this point, we can move to matrix notation
min max -u (u h. = 12
Lu (w),
(29)
where (u) & E wr)(u,) and Dy the diagonal matrix of the vector y. We can further decompose this by expanding X
min max B.b,a E 71
yp
Z) 2 2Tyb) yb) + (w).
(30)
Finally, noting that D Dyu = 1, we have the primary optimization:
min max 8.0,0
V 20 1(1 v. uT +
(u).
(PO)
n
Next we write our auxiliary optimization following CGMT Theorem
min max 1 0. E
[101|gT e + ||l|/ 0 + tTuu v T 11x - = + (w).
(AO)
We can separately optimize over the direction and magnitude of y:
1
min max 22..200.10
|100||
T v
+
3h + - 11(pp 0) - - yb + Lw)
(31)
1 j0
max min
([[@lg
10p () (b) 3h 0 + (u)
(32)
0. 220.1
1
min max 2,0.0 920
[
@ g 1( T 0 yو4l 3h 0 + Cu (w).
(33)
Next we use the AM-GM trickl rewrite the two norm the squared two norm, letting &
Vn
min max min Et E b 820 T>0
+
||u |00|g 1( (o) ybbl + nn
h To (w(
(34)
2
2mn
8T
Now optimize @ in the orthogonal subspace to lr setting ||@] and
min max min ŞT a,b 520 720
1u + ag
1(114) 17) yb1l
2
27n
(h H |
/ a2 1 '1 all + (2)
(35)
nn
1
min,
+ 27
22
2
20 (A
+ 1P-- 0
10
[n و² ||-0113 - |P-@| = -/aー
21

---

where Pl projection into the orthogonal subspace of μ. This can then be written in terms of the Moreau envelope:
min max min FT ab,sa E20 +20 2
5
(1
+
va² a² - 72 p211p=+b1))
vn
MMMMMaag + 11111l + b; Tw+)
w MI -ags + 1111l17 b; TW- }}.
(36)
==1
where we have grouped by y.
Letting n, - 00 with _12 6, we have
ET <vō(a² + T+ TTW, |1 + b,
TW+
min a>0,5,75,
max $20 2
E
+ + Ilully b;
---))) }} &
(37)
where we have switched the order of the min and max. See Taheri al. 31 for a justification and further technical details.
This leaves us with simplified systems of equations to solve based on first-order optimality condi- tions:
w
+
n
+ |1 + b; B[Mi
TW4 }]
2
&
&
w
B aG + |||||1 - b; TW
=]=0
(38a)
6
&
T
00aa2 - 72)
Tu
B[M
2 aa + ||u|17 + b; TW+)
}]
2
-
g²
Tw²
E.M 2( -o + 11x117 - b; TW
)] =
(38b)
8²
&
87
+ T44+ MM 1 (-aG + ||w|| + 6; TW+
8
}]
/ 1022-222
- | a + 1|x117 - b; TW-
= 0
(38c)
34 F4W+BIM -00G + ||a||7 + b; TW4 }]
g
- w BiM (-aG + ||u||> b; TW
)}
0
(38d)
da
T.W.M (-aG + || + b; TW+ }]
E V 0( 72)
&
- E a- + || b; 11=0 8
TW
(38e)
Let X' A T/C. Combining 38a) and 38b we obtain T
Vōla² 22), equivalently, we have the following simplifications:
{ vō - /// Substituting this into (38c) and 38e
8(a² - -2) + 2x
M M2p((aGG + ||24|/ 6; Xww)]
+2X² M 2 (-aG + || [w=_) =
(39a)
87
+ u
||
( ( + || y b; Xww1]
X
+ w || ( (aG || [w_> =
(39b)
da
T T.
E.W.EEGMMM. (-aG + ||lll b; Xw4)
X
22

---

- w E,, (- + ||lll 6; bw =
(39c)
T+ Bea, + ||al| b; [ww]]
--- -T---IIMMM MiM ||al| b; N' (ww))
(39d)
Note that the above depends only on the ratio of the weights, so let À - Nes+ and u+
ó(a² --)) + 2x T B..iMMMM22-aaaa.. + || b; A)]
A2
+2 T T-EMM 20-M + by A/p
(40a)
87p + T M B ( || b; A)]
+ I MLM || by A/p)] = 00
(40b)
8ap
7 x | ( + b; A)
-IGM, aG | b, A/p)] = 0
(40c)
M |1ul] bb; b;
-T BLM ( aG |1d1 A/D)
(40d)
We can rewrite using Stein's lemma
-a²) + 2X T (- 1| + b; A]
+2x² /22 BLM aa | by A/p) = =0
(41a)
67p + + ( 11/l/1 8; A)]
4y A/p) =0 + |I ( -
(41b)
8ap
T Ba, || b; A)]
+ *T_aB[MM,(-aG++ 11/// by A/p)) = =0
(41c)
T M (- |11411
b; A)]
π E.M ( aG 1114 by A/D)
(41d)
This completes the proof.
A.5 Proof of Corollary
The margin-based form of square loss can be written as
Esquace (2) 1 -2 1)2
(42)
2
where z is the margin. The prox operator (and therefore the Moreau envelope) has nice form for this loss:
I A 1+
prox (x;;
(43)
which thus reduces the Moreau envelope to
1 1 +
1)2 1 X(1 x) )2 21
Migum (x; A)
(44a)
2 1
1 X
1 1(x - 1)2
X(1 x(1=x)²
+
(44b)
2 X)²
2(1 + X)2
1 (x - 1)2
(44c)
2 1+X
For smooth function f and G standard normal: [G) (G) and E.G((a)) aEff((a)))
23

---

We have the following derivatives of the Moreau envelope:
1
M11(1; ))=; 1+1 A
(45)
1
(z; = 1+1
(46)
M2²(x;A) = - -1 1) 2(1 2)2
(47)
Using the properties of the Moreau envelope of square loss, we get the following from 41
8(a² - 72) + 1 [E[[[ + |||||7 + 1)31 (1+ A)2
-BI(-aC + || - b - 1)
+X2/0²
(48a)
(1 + A/p)²
87p + M+pl|kM
1E((-aG + ||lll + b 1)1
+ A
1E((-oG + 1111117 - b - 1)]
+ ||
(48b)
1 1P1/p
dap
1
1
TT ap
+ T
0
(48c)
A
1+A
1 A/p
E1( (1((-G + |1 + b 1)
+
EI( Bl(--G + |1111/7 b 1)]
(48d)
1 A/P
Taking the expectations, we get
26y ||l|| + >11111112 2>||1e|| 62 2b + a² +1
8(a² -22) - 1------0 1²
(1 + A)2
>2//0² 22ryllall + + -11111112 - 27|1/411 + b2 + 2h + 0² +1
(49a)
(1 + 2/p)²
87p
11 + b
|||| - b -
+ a +lll
+ π ||
(49b)
X
1+>
1 t/p
8p
1
+ T+
+ 1 2/p
(49c)
X
1
| + b -
| - b b-1
7+P P
(49d)
++ A
1 A/p
Further, we can examine the sum of the weighted expected Moreau envelopes and see by direct calculation that the sum is jointly strictly convex in a, 7, b, X. Of note, each expected Moreau envelope on its own is not strictly convex (in fact the determinant of the hessian is 0). This proves Corollary
A.6 Comparison of and أ
Note that we have closed forms for both the unweighted case (Corollary 2 and the weighted case such that b* = O (Theorem 2). Thus, it is natural to compare the two to see when it is advantageous to weight in this manner.
In the unweighted case, the WCE dominated by the positive (minority) risk in [ since 1* < On the other hand, the two class-conditional risks are equal for the model learned with by construction. Thus, we need not consider the max in the definition of WCE (2). The comparison of interest, then, is simply two Q functions. We will derive conditions under which á achieves lower WCE than
24

---

1. We denote the solution to the weighted problem as (% â) and the unweighted problem as (y", a², b*). Thus, order for the p-reweighted WCE be lower than the unweighted WCE,
7s Q
y Y (++t+-))
S
(50)
à
ar
requiring
78
7 +
(51)
a*
Substituting the closed-form solutions yields the following necessary and sufficient condition:
2
22 1
2 2 2T π4)
(4(1 m
This final expression demonstrates that the separation the classes is too large, then the un- weighted model will outperform the weighted model. This is perhaps intuitive given that large separations essentially act as leverage, possibly causing the weights to overcorrect. Note that this
large separation regime, 5a is approximately linear and so the WCE of the weighted model decays exponentially in This tells that when ý outperformed by p 1, the errors very small.
0.200
6= 0.2. = 502
0.00200
6- 0.2, =1+ 0m2
Class
0.375
0.00175
Method Simulation
0.150
Sim.lution
0.001.5
0.300
0.00075
0.050
0.00050
0.00025
0.000
0.0
2.5
5.0
7.5
10.0
12.5
15.0
17.5
26.0
0.00000
0.0
2.5
5,0
7.5
10.0
12.5
15.0
17.5
20.0
(a) 2, mild separation
(b) 4, significant separation
Figure 8: With mild separation, the weighted model (red) does significantly better than the un- weighted model (black), but this is reversed with large separation between classes. Note the differ- ence in scales of the error.
See Figure understand the behavior of the class-conditional risks with increasing separation. In Figure 8b note that is predictive of the crossover point, but that the weight where the WCE is actually minimized smaller than ý in the large separation case. This result provides clear insight into the regions where weighting is useful and emphasizes the intuitive point that larger separations demand less weighting. Indeed we see that increasing separation decreases the errors for each class while also decreasing the optimal weight; decrease from ç to 1. This regime of little impact, however, as the error for both will be extremely small pointed out previously.
conjecture that for - oo, the optimal weight will
25

---

B Additional Plots
B.1
PCA
We see in Figure 9 that very few features capture most of the variance of the retraining data. This motivates us to use the smaller "effective" dimension as mentioned in the main body. Note that for CIFARIO, the limited data limits the maximum number of meaningful features.
PCA Ververe Spectrum
PCA Veriece Spectrum
10
04-
130
150
203
250
(a) CelebA
(b) CIFARIO, planes trucks
PLA Vatvene
PCA Vanuraa Spadrum
90
64
02
05
00
CIFARIO, cats vs dogs
(d) CIFARIO, deer vs horses
Figure 9: PCA Spectra. We see that most of the variance is captured by very few features.
B.2
Ablation Studies
We provide additional plots in Figure 10 showing that our findings hold even when the latent di- mension of the ResNet34 model is different. Here we show results for a ResNet34 model with 128 dimensional latent space. The effective dimension (3) is still used to calculate p.
Backbone: resnet34, Latent Dim: 128, n: 1000
Backbone: resnet34, Latent Dim: 128, n: 20
Class
0.8
-1
0.8
Error
0.6
Error
0.4
0.2
0.2
10
(a) n 1000
(b)
20
Figure 10: Per-class errors on CelebA dataset. Even with lower latent dimension, (red) is still predictive the cross-over point modulo shift seen even the large setting. This likely due to non- Non-Gaussianity of the latent data.
26

---

Even for different class pairs in CIFARIO, we see similar behavior as shown in Figure I1 For this dataset, we select cat as the minority class +1 and dog as the majority class -. The imbalance selected as 17% class +1 and 83% class -1. For this dataset, the effective dimension is 3. p is still predictive the erossover point for both n. 90 and n 20 resulting in significant gains over the traditional ratio of priors.
Backbone: resnet34, Latent Dim: 256, n: 90
Backbone: resnet34, Latent Dim: 256, n: 20
1.0
Class
0.8
+1
0.8
0.6
Emor
Emor
0.4
0.2
Class
0.2
+1
0.0
0.0
10
20
30
40
50
60
10
20
30
40
50
60
P
(a) 90
(b) 20
Figure 11: Per-class errors on CIFARIO dataset, cats vs dogs. f (red) predictive of the cross-over point when using the effective dimension of the data (4).
The story is similar for deer vs horse with the same imbalance which has effective dimension 9. Note that for 20, is undefined, suggesting that the correct weighting strategy is to push + Do. We see that up to p = 60, the per-class errors do not meet.
Secibvn: varaC4 L 25C
Becdbvs Hamec4 Luaan 256 2
Becdons: rama24 Latan 250
3
(a) 90
(b) 30
(c) 20
Figure 12: Per-class errors on CIFARIO dataset, deer vs horses. / (red) is predictive of the cross- over point when it is defined using the effective dimension of the data (9). Is undefined for 20, suggesting that the per-class risks will not meet. This effect is seen in practice.
C Experimental Details
All empirical experiments were performed using NVIDIA A100 GPUs while simulations were com- pleted on CPU. Each experiment took less than 30 minutes of wall time to run after training base models. Base models took less than 4 hours to finetune from the pretrained weights. A full list of hyperparameters is provided in Table
27

---

Parameter
Value
Backbone
ResNet34
Pretrained Weights
Imagenetlk-V2
Latent Dimension
(128, 256, 512)
Optimizer
AdamW
Learning Rate
1e-3
Full fine-tuning epochs
10
MLP Dropout Rate
0.5
Fine-tuning epochs
Fine-tuning LR
le-2
Table 1: Hyperparameters for CelebA and CIFAR10
28


Reviews:
[{"review_id": "NNce49T0th", "reviewer": "Reviewer_T2Wg", "summary": "This paper studies optimal loss weighting in last layer retraining for the underparameterized regime ($\\omega = d/n \\in (0,1)$), a practically relevant setting between the well-understood population and overparameterized extremes. The authors employ the Convex Gaussian Minimax Theorem (CGMT) to reduce the high-dimensional weighted ERM problem to a tractable system of four scalar equations, deriving closed-form solutions for square loss and an optimal weighting formula that incorporates overparameterization effects beyond the classical ratio-of-priors. They validate their theoretical predictions on CelebA and CIFAR10 datasets.", "questions": "1. The analysis focuses on binary classification, but many real-world applications involve multi-class imbalanced problems. How would the theoretical framework extend to the multi-class case, and would the optimal weighting strategy change significantly?\n2. The work specifically targets the last layer retraining, but what if it were applied to broader fine-tuning scenarios? Would the optimal weighting formula still hold?", "limitations": "Please see the weakness and questions", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The paper tackles the understudied underparameterized regime ($\\omega \\in (0,1)$) in cost-sensitive learning, which is precisely where real-world last layer retraining typically operates, bridging the theoretical gap between well-characterised extreme cases.\n2. The authors construct a well-structured theoretical framework that progresses systematically from general principles to concrete applications, utilising sophisticated mathematical tools (CGMT, Moreau envelopes) to derive theoretical insights.\n\nWeaknesses:\n1. A key limitation is that the authors assume solution uniqueness in Theorem 1 without proof, creating a conditional statement where the essential premise remains unverified. Additionally, CGMT provides probabilistic bounds, yet Theorem 1 claims deterministic convergence without bridging this fundamental gap through concentration inequalities.\n2. A gap exists between theoretical analysis and practical implementations; specifically, the theory assumes $n, d \\to \\infty$ but practical applications use finite samples (e.g., $n=20$) with no error bounds or convergence rates provided.", "was_revised": true, "final_justification": "Although several concerns have been resolved, the multi-class scenario still raises questions for me. The author's two explanations have not fully convinced me.", "rebuttal": "We thank the reviewer for their thorough reading of our work. We are very glad that you found the paper relevant and engaging. We hope to address each of the weaknesses and questions mentioned.\n\n - **W1**: Thank you for pointing this out. The key to showing uniqueness (and therefore convergence to the asymptotic discussed in this work) is to show the joint strict convexity-concavity of the objective in (37) in the supplement. For square-loss, we can take a direct approach by evaluating the *expected* Moreau Envelope (eME) functions in (37) leveraging that the square-loss is self-dual, hence its Moreau envelope is an easy-to-work-with quadratic. After easy calculation of the expectations over the (scalar) Gaussian random variables, and after combining the two individual eME functions (corresponding to majorities/minorities respectively) we can show that the combination (which is of the form ($\\alpha^2+\\gamma^2+b^2+1)/\\lambda$) is jointly strictly convex (in $\\alpha,\\gamma,b,\\lambda$) by direct evaluation of the Hessian. We note that strict convexity-concavity holds only when both eME terms are considered together, as each individual eME is not jointly strictly convex-concave. Beyond square-loss, we can follow the steps of App. A.6.1 in [1]. The only key change, following the “lesson” learnt by the explicit evaluation of the square-loss above, is to combine the two eME terms. We will include this analysis in our final version and update corollary 1 to reflect this. We will also clarify in Thm. 1 that the convergence to the deterministic parameters holds in-probability (as previous works e.g. [1,24 in manuscript]). \n - **W2**: First, we highlight the fact that asymptotics kick in at relatively small dimensions and samples is a *positive* feature of our analysis. Non-asymptotic analyses are more challenging. While the CGMT is non-asymptotic in nature, the technical challenge comes in the analysis of the Auxiliary optimization. Recent literature [2,3,4] has made significant progress in this direction, but given the necessary technical overhead such an extension goes beyond our scope. We recognize the value of such non-asymptotic extension, which for example can yield confidence bounds and will certainly discuss this as current limitation and a possible direction for future work. For our main goal of re-weight optimization it is encouraging that direct optimization of the asymptotic formulas gives insights that match with our empirics.\n - **Q1**: Recent work [5,6] has considered the multi-class case, and our scalarization (Theorem 1) would be amenable to these updated analyses. However, the analysis of the auxiliary optimization and the weighting scheme would need to change. For instance, it is not obvious that the multi-class extension of square loss would be amenable to the same closed-form analysis. Additionally, the optimal weighting scheme in our work is chosen such that $b=0$ which is enough to ensure that both classes are fairly classified. In the multiclass setting, however, this condition would be insufficient without significant additional structure on the class means and variances. We hope to extend our work to this setting in the future, and see this manuscript as an important and valuable step in that direction. \n - **Q2**: The construction of the optimal weighting scheme $\\tilde\\rho$ relies on the linearity of the last layer retraining, but one could conjecture that the same trends would likely hold: namely the more parameters that are being finetuned (with respect to the number of samples and their dimension), the stronger of a weight is needed to achieve the same balance. A key challenge in settings rather than the last layer retraining would be quantifying the effective dimension as our current approach relies on a fixed latent distribution. Spreading weights across different layers would complicate this significantly. \n\n## References:\n\n[1] H. Taheri, R. Pedarsani, and C. Thrampoulidis, “Sharp Asymptotics and Optimal Performance for Inference in Binary Models,” Feb. 26, 2020, arXiv: arXiv:2002.07284. doi: 10.48550/arXiv.2002.07284.\n\n[2] L. Zhou, F. Koehler, P. Sur, D. J. Sutherland, and N. Srebro, “A Non-Asymptotic Moreau Envelope Theory for High-Dimensional Generalized Linear Models,” Oct. 21, 2022, arXiv: arXiv:2210.12082. \n\n[3] G. Wang, K. Donhauser, and F. Yang, “Tight bounds for minimum $\\ell_1$-norm interpolation of noisy data,” in Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, PMLR, May 2022, pp. 10572–10602. \n\n[4] Q. Han and Y. Shen, “Universality of regularized regression estimators in high dimensions,” Jun. 27, 2022, arXiv: arXiv:2206.07936. \n\n[5] B. Loureiro, G. Sicuro, C. Gerbelot, A. Pacco, F. Krzakala, and L. Zdeborová, “Learning Gaussian Mixtures with Generalised Linear Models: Precise Asymptotics in High-dimensions”.\n\n[6] D. Bosch and A. Panahi, “A Novel Convex Gaussian Min Max Theorem for Repeated Features”. Proceedings of The 28th International Conference on Artificial Intelligence and Statistics"}, {"review_id": "YAonOsbD9W", "reviewer": "Reviewer_odBN", "summary": "- Study the effectiveness of loss weighting in LLR to improve  minority class accuracy, which is highly correlated to the overparameterization ratio $\\delta$ \n-  Proves in theory and practice that loss weighting is still effective under the δ ∈ (0, 1) regime of LLR, but the optimal weight needs to consider the  overparameterization ratio $\\delta$ \n- Provide a closed‐form solution for the square‐loss weights.\n- Apply the method in the practical application of fine-tuning models.", "questions": "See weakness", "limitations": "Yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths: see the summary\n### Weakness:\n- The paper addresses only last‐layer retraining. While that setting is common, in many applications, a small number of top layers or additional modules are fine‐tuned jointly to improve performance in both sufficient and limited training data. Extending the analysis beyond one linear layer remains an open challenge.\n- The closed‐form weighting is derived for the square loss. Although the scalar‐equation framework can apply more generally, the most commonly used classification loss, cross‐entropy, should be discussed.\n- Strong assumption on the distribution of latent features being Gaussian with known covariances.", "was_revised": true, "final_justification": "The authors addressed my concern", "rebuttal": "We thank the reviewer for their careful consideration of our work. We will consider each of the weakness in turn:\n - **W1**: The construction of the optimal weighting scheme $\\tilde\\rho$ relies on the linearity of the last layer retraining, but one could conjecture that the same trends would likely hold: namely the more parameters that are being finetuned (with respect to the number of samples and their dimension), the stronger of a weight is needed to achieve the same balance. A key challenge in settings rather than the last layer retraining would be quantifying the effective dimension as our current approach relies on a fixed latent distribution. Spreading weights across different layers would complicate this significantly.\n - **W2**: A key challenge in characterizing losses other than the square loss is in computing the explicit form of the expected Moreau envelope; in other words, such closed form expressions do not exist for most losses making the optimization harder in theory. While the reviewer correctly points out that the scalarization holds for any loss, the reduction beyond that point is dependent on the form of the Moreau envelope. Recent work [1] has found that finetuning models using square loss can achieve equal or better classification performance to using standard cross entropy, which we discuss in the manuscript as motivation for studying the optimal weighting scheme for square loss. \n - **W3**: This assumption may appear strong, but recent work [2,3] has argued that such assumptions in the last layer can be sufficient to guarantee performance for distributions sharing only the first and second order statistics. Additionally, while the nature of the CGMT framework requires that the latent features have some variant of a Gaussian distribution, there is the possibility of extending this work to mixtures of Gaussians which may more closely resemble the true latent distribution of a deep model. Finally, our empirical results align with our theoretical findings, suggesting that the assumptions we make might not be too strong to be useful in practice.\n\n## References:\n\n[1] A. Achille, A. Golatkar, A. Ravichandran, M. Polito, and S. Soatto, “LQF: Linear Quadratic Fine-Tuning,” Dec. 21, 2020, arXiv: arXiv:2012.11140. doi: 10.48550/arXiv.2012.11140.\n\n[1] R. Ghane, D. Akhtiamov, and B. Hassibi, “Universality in Transfer Learning for Linear Models,” Feb. 23, 2025, arXiv: arXiv:2410.02164. doi: 10.48550/arXiv.2410.02164.\n\n[2] Q. Han and Y. Shen, “Universality of regularized regression estimators in high dimensions,” Jun. 27, 2022, arXiv: arXiv:2206.07936. doi: 10.48550/arXiv.2206.07936."}, {"review_id": "3ZoJYDsR3i", "reviewer": "Reviewer_k7Vs", "summary": "This paper investigate the loss weighting method in the understudied regime $\\delta \\in (0, 1)$, where $\\delta$ is the ratio between the latent data dimension and the retraining samples. The authors theoretically and empirically show that the loss weighting is still effective in the regime $\\delta \\in (0, 1)$ and they also derive optimal asymptotic weighting as a function of $\\delta$.", "questions": "- Would the results still hold under weaker assumptions than $d, n \\rightarrow \\infty$?\n\n- How is the conclusion affected by the input complexity? The conclusions are derived under a special case where both p(x|Y = y+) and p(x|Y = y-) follow Gaussian distributions with identical variances. However, in real-world settings, different classes may exhibit varying input complexities, which can significantly impact the optimal choice of $\\rho$. For instance, if the input x  corresponding to y+ is fixed to a single point, then selecting a small $\\rho$ would be reasonable.\n\n\n- Is there an overlap between $\\delta \\to 0$ and $\\delta \\in (0, 1)$? If so, what are the connections and differences between the conclusions drawn in this work and those in prior studies that focus on the $\\delta \\to 0$ regime?", "limitations": "yes", "rating": 4, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "I have limited knowledge in this specific area and the overall manuscript appears satisfactory to me. \n\nQuality:\n\n- [+] The theoretical and empirical demonstrations are both clear and technically sound.\n\nClarity:\n\n- [+] The presentation of the idea is clear.\n- [-] It is unclear why the case of $\\delta = 1$ is special, especially considering that many dimensions in the last layer of large models often go unutilized. Further justification or intuition would be helpful.\n\nSignificance& Originality:\n\n- [+] I think this work is novel and can be a contribution to the Machine learning community.\n- [-] The current formulation is limited to the specific case where p(x|Y=y_i)is a Gaussian distribution with an identity covariance matrix. Generalization beyond this setting would enhance its impact.", "was_revised": true, "final_justification": "My concerns are addressed and I will keep my original score", "rebuttal": "We thank the reviewer for their careful review of our work and are grateful for the opportunity to clarify these points. We will address each of the presented weakness and questions: \n - **W1**: While many of the dimensions in the last layer go unused or have little impact, the threshold of $\\delta=1$ is still critical in theory. At that point, the single optimizer of square loss is not well defined and theory must resort to discussing the minimum norm interpolator of the training data. This threshold, and in particular the regime of $\\delta>1$,  is also used to argue against the efficacy of loss weighting in large models [1]. In contrast, we study the underparameterized regime which we demonstrate is amenable to loss weighting. We will clarify this point in an update to our manuscript if accepted. \n - **W2**: The CGMT framework allows us to discuss more complex data distributions such as mixtures of Gaussians per class or more general covariance matrices. This work focuses on the identity covariance matrix for simplicity of presentation, but any shared covariance matrix is captured by our theory through a whitening transformation. We hope to extend this framework to weighting for mixtures of Gaussians per class, but this work is a first step towards that goal. Indeed we see that our analysis provides valuable insights in real datasets which are certainly non-Gaussian and non-isotropic. We will ensure that these clarifications are made in the revision.\n - **Q1**: In our analysis of the auxiliary optimization, the only use of the asymptotics is through the Law of Large Numbers and kicks in relatively quickly. Indeed, we report results for n=20 and show that the theoretical weighting scheme is still effective despite the very small number of samples and dimensions.\n - **Q2**: Indeed, one should suspect that the (relative) input complexity should also play a role in the selection of $\\rho$. The central aim of our manuscript is to capture the effect of parameterization in the selection of loss weights, and we focus on the identity covariance as a clean basis for this investigation. We hope that this work serves as inspiration for future work exploring other factors that should be considered when selecting loss weight, including relative input complexity. \n - **Q3**: One nice result of our work is that it captures the population case ($\\delta\\rightarrow0$) as a special case. We point out in [Section 3.1] that our weighting scheme $\\tilde\\rho$ recovers the ratio of priors which is optimal in the population setting. Additionally our results on downsampling in [Section 3.2] align with [2] when $\\delta\\rightarrow0$. Our results offer the flexibility to account for the parameterization ratio and are complementary to population results.\n\nWe hope that these clarifications enhance the reviewer's understanding of our work and will ensure that these clarifications are included in the revision.\n\n## References:\n\n[1] J. Byrd and Z. C. Lipton, “What is the Effect of Importance Weighting in Deep Learning?,” presented at the International Conference on Machine Learning, Dec. 2018. \n\n[2] M. Welfert, N. Stromberg, and L. Sankar, “Theoretical Guarantees of Data Augmented Last Layer Retraining Methods,” May 09, 2024, arXiv: arXiv:2405.05934. doi: 10.48550/arXiv.2405.05934."}, {"review_id": "mstAESHUDT", "reviewer": "Reviewer_xfu4", "summary": "The paper investigates the class imbalance problem using Empirical Risk Minimization. The authors assume a binary classification setting and derive its closed-form solution. They then apply their method to real-world data by using subsets of the CelebA and CIFAR-10 datasets, retraining only the last layer (LLR).", "questions": "I do not have any further questions beyond those mentioned above.", "limitations": "Yes", "rating": 4, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "[Strength]\n\nThe paper addresses classification under imbalanced data, a common issue in many natural datasets.\n\nIt is generally well written, and the accompanying code repository is clean and well-organized.\n\n\n[Weakness]\n\n The authors assume a binary classification task and evaluate their method only on specific attributes in CelebA and a single class pair (airplanes vs. trucks) in CIFAR-10. This narrow scope limits the practical generalizability of the proposed method, even though the problem of imbalanced data is highly relevant.\nThe authors compare their proposed $\\tilde{\\rho}$ with the conventional prior ratio, but they do not explicitly define what the prior ratio is or how it is computed.\n\nThe authors need to state the terminology more carefully. For example, the $\\textsf{prox}$ operator in L100 should be the proximal operator ($\\textsf{prox}).  Similarly, CS-SVM in L36 and w.p. In L88, the full name should also be mentioned.\n\n[Minor]\n\nIn L21, [1, 2] By -> [1, 2]. By\n\nIn Equation (8b), consider replacing $\\delta_\\gamma$ with $\\delta\\gamma$ if no subscript is intended.", "was_revised": true, "final_justification": "Several of my concerns are resolved. However, I still have a concern regarding a multi-class scenario. I do not think this is directly applicable to that setting from the current work.", "rebuttal": "We thank the reviewer for their comments and suggestions. We would like to address each point presented:\n - **W1**: We included additional results in the supplementary materials B which show similar results for additional class pairs. If accepted, we will update the manuscript with results for additional modalities such as text to expand the applicability of our work. Additionally, while our current results capture only binary classification for class imbalance, we aim to use similar techniques to tackle the more general problem of group imbalance in binary classification. We believe that this work is an important first step in understanding how the loss weighting process is affected by overparameterization. \n - **W2**: By the conventional prior ratio, we meant the inverse ratio of the class priors, $\\pi_-/\\pi_+$, which is very frequently used in practice. We will ensure that this point is clarified in the next revision of this work.\n - **W3**: We thank the reviewer for pointing out these oversights. We will ensure that any typos are fixed and the above terms are fully expanded earlier in the manuscript."}]

Author rebuttal:

Meta-review: "Thumb on the Scale" studies optimal loss weighting for last layer retraining, showing that it is still effective in the underparameterized regime if weights take relative overparameterization into account. Reviewers generally appreciated the work, acknowledging that it constructs a well-structured theoretical framework, and finding it to be a solid contribution to the machine learning community. While the study is limited to binary classification, and Reviewer T2Wg finds the extension to the multi-class scenario to be unconvincing, there is still sufficient material here to justify acceptance in my view. I urge the authors to include any elaborations provided during the discussion phase in the camera-ready submission.
Decision: Accept (poster)