When resampling/reweighting improves feature learning in imbalanced classification? A toy-model study
Authors: Tomoyuki Obuchi
Toshiyuki Tanaka
Venue: Accepted by TMLR ()

Abstract: A toy model of binary classification is studied with the aim of clarifying the class-wise resampling/reweighting effect on the feature learning performance under the presence of class imbalance. In the analysis, a high-dimensional limit of the input space is taken while keeping the ratio of the dataset size against the input dimension finite and the non-rigorous replica method from statistical mechanics is employed. The result shows that there exists a case in which the no resampling/reweighting situation gives the best feature learning performance irrespectively of the choice of losses or classifiers, supporting recent findings in~\citet{kang2019decoupling,cao2019learning}. It is also revealed that the key of the result is the symmetry of the loss and the problem setting. Inspired by this, we propose a further simplified model exhibiting the same property in the multiclass setting. These clarify when the class-wise resampling/reweighting becomes effective in imbalanced classification.

Published in Transactions on Machine Learning Research (4/2025)
When resampling/rewiighting improves feature learning in imbalanced classification? A toy-model study
Tomoyuki Obuchi
obuchi@iikyoto--..c..
Department of Systems Science
Kyoto University
Toshiyuki Tanaka
11 H...kyoooww.ec ac..jp
Department of Systems Science Kyoto University
Reviewed OpenReview: https: openreveee......ee.. net/ per//frmm? id= spgbyeCyLR
Abstract
A toy model of binary classification is studied with the aim of clarifying the class-wise resampling/reweigeeee
effect on the feature learning performance under the presence of class imbalance. In the analysis, a high-dimensional limit of the input space is taken while keeping the ratio of the dataset size against the input dimension finite and the non-rigorous replica method from statistical mechanics is employed. The result shows that there cxists case in which the no resampling/ resampling/rexveightinng situation gives the best feature learning performance irrespectively of the choice of losses or classifiers, supporting recent. findings in Kang al. (2020); Cao et al. (2019). It is also revealed that the key of the result is the symmetry of the loss and the problem setting. Inspired by this, we propose further simplifted model exhibiting the same property in the multiclass setting. These clarify when the class-wiss wisc resampling/reweightingg reweighting becomes effective in imbalancod classification.
1 Introduction
Real-world datasets for classification occasionally exhibit strong class imbalance with a long-tailed class distribution (Van Horn et al. 2018 NNaturalist 2018 Liu et al. 2019). Classifiers applied to such datasets tend to perform poorly for minority classes, which poses a major challenge areas such as visual recognition. Although several methods mitigate class imbalance have been proposed far (Chawla al. 2002] He
2009 Wallace et al. 2011), recent advances of deep lcarning have shed new light on this issuc, resulting numerous studies from the perspective of applying those approaches to classifiers based on deep
& Garcia
neural networks (DNNs) [Liu et al. 2019: Huang et al. 2016 Wang et al. 2017 Cui et al. 2018 Khan et al.
2019
Cui et al.
2019
Cao et al. 2019 Kang et al. 2020 Jamal et al. 2020 Tan et al. 2020 Menon et al.
2021
Kini et al.
2021
Among those approaches proposed so far, we focus on two simple strategies, reweighting and resampling,
which are commonly employed mitigate class imbalance. The resampling strategy tries to balance the samples in the dataset by oversampling the minority classes and/or undersampling the majority classes, while the reweighting strategy puts additional weight to each term of the loss in order to counterweight the class imbalance. The effectiveness of these strategics has been empirically verified in wide range of studies (Cui et al.) 2019 Cao el al.) 2019 Kang el. al. 2020 Jamal et al.) 2020; Chawla et al. 2002 He & Garcia 2009). In spite of these pieces of work, transparent description or understanding about when they are useful or not would still be incomplete. In particular, how class imbalance may affect the quality of feature learning would be an important problem in the context of representation learning in DNNs, but a thorough understanding of this issue still missing.

---

Published in Transactions on Machine Learning Research (4/2025)
Recently, Kang et al. (2020) reported an interesting observation that feature learning becomes better if no resampling is applied. More specifically, on the basis of their extensive experiment on visual recognition tasks using DNNs, they reported that the best classification performance was achieved when the whole network was first trained without any resampling and then only the last output layer (final classifier) was retrained with bass-balanced resampling. This observation can be interpreted as follows: one can learn the best feature representation in the initial training phase if one does not use resampling at all, and the good classification performance achieved by the retrained network is ascribed to exploitation of the good feature representation acquired in the initial training. A similar behavior was also reported in Cao et al. (2019). One can therefore expect that Kang et al's observation would provide useful generic insight into efficiency of resampling with regard to feature learning.
In this paper, we provide a theoretical analysis on a toy model to examine the effect of resampling and reweighting, especially aiming to clarify under what conditions the observation by Kang et al. holds. In our toy model, we treat a binary classification problem, in which the stochastic. More specifically, inputs are independent and identically-diisribbueed (i.i.d.) from a probability distribution on RN,, with parameters controlling the class imbalance and the variances. The two class centers are assumed to be located at t /VN E ERN. These constitute a standard setting for theoretical treatments of binary classification (Barkai & Sompolinsky 1994).
process is assumed to be
In the above setting, the performance of feature learning can be quantified as the accuracy of estimating wo. since it represents the most discriminative direction of the two classes under isotropic class-conditional input distributions. Our analysis, which considers the asymptotic N - so, reveals that the accuracy of estimating we is maximized when one does not employ resampling/reweighee the class imbalance, under the conditions that the variances of samples of the two classes are equal, that the decision boundary is located equidistantly from the two class centers, and that specific ansatz of the analytical framework we use, the so-called replica symmetric (RS) ansatz, is correct. This finding provides an analytical support for Kang et al.'s obsorvation. More interestingly, this finding remains valid with a rather wide range of classífiers and losses, which can be shown on the basis of the symmetry in formulae derived in our analysis. Although the equal-variance condition for sample distributions might seem somewhat artificial, it may be achieved in the feature representation at the last hidden layer of classification DNN as result of DNN training, and some recent studies partially support this (Papyan et al. 2020
at all, irrespectively of the degree of
Fang et al. 2021
Our results are derived via
mmehaiccalle techniques (Barkai & Sompolinsky
1994
Barkai et al.
1993 Biehl & Mietzner 1993 Watkin & Nadal 1994 Lootens & van den Broeck
1995
Tanaka 2013
which are applicable in the limit N + 0o. Especially, the replica method (Nishimori, 2001
Dotsenko 2005
Mezard & Montanari (209) plays a key role in computing the quantities of interest. Although the replica method is mathematically non-rigorous, the results derived in this paper are conjectured to be correct, which
is supported by an excellent agreement with results of numerical experiments, as well as by an accumulation of many model analyses over many decades in which the replica method is eventually shown to give the exact
results (Montanari & Tsc 2006 Talagrand
2003 2011a b).
The remainder of this paper is organized as follows. In the next section, the problem setup and the for- mulation are explained. In sec. 3 the analysis details using the replica method are explained. The derived formulac using the replica mothod are utilized to systematically examine behaviors of the quantities of inter- gives the best
est. The result shows that there exists a case in which the absence of wwwaimlliinng/weiilii/eo feature learning performance irrespectively of the choice of losses classifiers, yielding theoretical support for Kang et al.'s observation. On the basis of this theoretical result, we also provide further simplified model for multiclass classification, for which the same consequence about the resampling hwtphg//wwwwihtigggg holds. In sec. E numerical experiments are conducted to verify the replica results. The last section concludes the paper.
2

---

Published in Transactions on Machine Learning Research (4/2025)
2 Problem setup, formulation, and related work
2.1 Data-generation process
Let us consider a classification problem with two classes labeled by y = =1 whose distribution is
Pr(2) Σ Tyggs'' y'=il
(1)
where da,b denotes the Kronecker delta, which equals when a a= = and 0 otherwise, and where rii E [0. 1] with T+1 + r-1 = 1 control the degree of class imbalance: T+1 = r-1 1/2 corresponds to the balanced case. The input space is assumed to be IR and the input vector x E R is assumed to be generated from the following model given a label y E {-1,13:
W0
x
+ E(y),
(2)
where tw//V/ VN E R represent the class centers corresponding to the two classes y =1. We impose the normalization condition wwol3/NN In Eq. zero-mean Gaussian distribution
2 {()) is assumed to be random i.i.d. vector obeying the
PEY (E 0z) (2
N/2
203 141
(3)
where the label-y dependence appears only through the variance of which is assumed finite for both y 11 The variance of expresses the cluster size of the class in the input space. We call this model the Gaussian dlass-conditiona model (GCCM) (see, e.g., Chatterji & Long (2021 Let DM {( y be a dataset
of M i.i.d. datapoints following the above data-generation process:
M II N.. PElY
P(DM wo)
Fp
3
W0
(4)
VN
Up
Although we derive our results on the basis of the Gaussianity assumption 3 we expect that the same results hold for a much wider class of distributions with oi < oo thanks to the universality appearing through the central limit theorem in the limit N + cc. Also, it is possible that qualitatively similar results may hold for a broader class of distributions beyond the Gaussian universality, such as those studied in Adomaityte ct al. (024).
2.2 Classifier and loss
A generic classifier can be formulated as first mapping the input æ onto a ono-dimensional feature f(x)= e IR via a feature mapping /, and then, on the basis of the feature f(x),, producing soft decision P(y a)) which is an estimate of the conditional distribution of the class label for the input æ. In formulation, consider the simplest case where linear feature mapping f(x) w x /N is to be used, with w E RV its parameter, As the noft-decision classifier given the one one-dimensional feature = f(x), which we call the feature-based classifier, we assume P(y x)) = M(U(f(z)+ with a function M : - [0. The bias term the parameter of the feature- based classifier. We may also write P(y w, b) = MMy(w" ( +)) in order to make explicit the dependence of the classifier on the parameters w and By an appropriate choice of the function M, this formulation covers several standard classifier models, such as a perceptron and logistic function:
1,
h > 0,
1 sgn (h)
Mpa(h)
1/2,
h 0,
(5)
2
0,
h < 0,
M1) = ch)
(6)
The same model sometimes referred by other names, such the Gaussian mixture model.
3

---

Published in Transactions on Machine Learning Research (4/2025)
We say that feature-based classifier using M(h) is symmetric if the function M satisfies M(h)+M((- for all h E IR. Mp and Ml. shown above are examples defining symmetric feature-based classifiers.
The weights w of the feature mapping f are usually determined by minimizing an empirical loss, which is a (sometimes weighted) sum of loss values over all the datapoints. We let [(w;; x, y. M) denote the loss of the weights ur given the datapoint (ær,y) and the model M. According to the above assumptions of the linear feature mapping and those on the model, generic loss function can be written as function f(h, y) of two arguments h = w x////V/N + and y, in such way that the relation [(w; 2, MM) l(w "ooT=//VN + b, y) holds. A common loss function in the recent practice is cross entropy (CE) and the corresponding loss function takes the form
(CE(w" LiuTz/Ve "=////N++ + b. = log M(y(w x/VN 6)).
As (ce(h, y) depends on and y only through their product yh, it has the symmetry (ce(-h.-u) = LCE (h, (h,y) y): it is also the case with many other standard losses which are represented as functions of yh, such as zero-one, exponential, smoothed or non-smoothed hinge losses. Hercafter the symbol E is used to denote generic loss having this symmetry. To express a specifie model and loss, we use an appropriate subscript: for example, if use the logistic function and the CE, the resultant loss will be denoted as fCElo. Furthermore, if & is used with a single argument, we assume that it expresses the one for the positive label: E(h) E(h, +1). This is convenient shorthand notation when we work with the above symmetry.
The empirical loss considered in this paper has class-wite reweighing factors sy E [0, 1/, which are assumed to satisfy the condition 841 + 8-1 = 1. Given a loss and a dataset D ((p y1) W the empirical reweighted loss is thus written by
M H(w DM; s) = E' Sy ((w Ta x./VNN + +, B.) ==1
(8)
This class-wise reweighting of the loss is intended to mitigate possible undesirable effects of the class im- special case of the sample-wise weighting which
balance arising when ri1 * 1/2. It can be considered has been investigated, e.g., in He & Garcia (2009). We would also like to mention that resampling olfers yet another alternative for the purpose of mitigation of class imbalance. Resampling, however, amounts to deter- mining the sample-wise weights in the empirical loss by the respective sample counts, and honce the average effect of the resampling can be incorporated into the class- reweighting factors. We analyze properties of the above loss and its minimizer v arg min H(w DM,;,, b, s) under the constraint ||/||2 N, especially wo:[|[||==N
focusing on the overlap between v and Wh: wn defines the most discriminative direction of the two classes, and under the fixed-norm constraint || [w 12 = N how well the estimate v aligns with the direction of wo is fully captured by the overlap m = sb wo/N, implying that the quality of feature lcarning in our sctting is solely characterized by m. Although estimating the bias b may also be performed via minimization of H with respect to b, the analytical framework explained in the next section allows us to do it in more flexible manner and we leave b as tunable parameter.
Before proceeding, we have a noteworthy remark about the Bayesian inference. Suppose that we know the data-generation process but do not know the specific values of either we or f11- We thus introduce w as a random variable to estimate wo and T+ı as hyperparameters playing the role of class-wise reweighting factors. With an appropriate prior P(w)) the posterior distribution of w given the dataset DM becomes
P(w MM P(w) II
1==1 Tup PEIY (xp -
w
0
/N
ʃ ddw I Ty, PEY
Up
/N
1p))
P(w) IIMM PEIY [zp (1n-1117 w
/N
(9)
[dw II M PalY
(x,,.
U/ /N
02
Up
Hence the posterior distribution does not depend on T=1, meaning that the reweighting via adjusting r=i has effect on estimation of wo- If we treat. rii as random variables with prior P((ra1)), the result
4

---

Published in Transactions on Machine Learning Research (4/2025)
is scemingly different but the marginal posterior P(w DM) f P(w,r DM) is still independent of the choice of P((ra)) as long as the priors of r1l and w are independent. Hence, to study the effect of reweighting/reeampligg on the feature learning, the Bayesian inference framework based on the true data- generation process is inappropriate.
2.3 Statistical mechanical formulation
To investigate the estimator v arg min H(w D,; it is convenient to introduce the following willw//saN
distribution:
Ps(w) ;DM;,,8) = Z 5(N - [) 978(m) DM (MM.bs))).
(10)
where 8(.) denotes the Dirac measure, where B² 0 is the inverse temperature parameter, and where
Z = Z(D = / dw o(N -| - DM (b.be))
(11)
is the normalization coefficient. In the limit + oo, the distribution P3 concentrates on the set of min- imizers of H(w :b; s), and hence any properties of the estimator to can be computed from the av- erage the distribution in the limit. Further, Φ = -(NN)-1 Z plays the role of the cumulant generating function of w and converges to the per-variable average loss in the limit B + co, that is, lim
= min [iin=:////?[N H(w b,,,,))N. This means that ø contains all the necessary infor- mation for our purpose and hereafter we concentrate on computing it. According to the physics terminology, in the following we call P3 the Boltzmann distribution, 8-1 the temperature, the partition function, ø the free energy, and u the energy. The average over the Boltzmann distribution is denoted by the angular brackets as
C-9H(WD DM
((...))) f dw Ps(w) (b b, s....) TT
DM,
(...),
(12)
where the symbol Tr f dw 6(N - ) is introduced for notational simplicity of our development later. A problem arises in the computation of the free energy de it depends the random variable D and hence its direct evaluation is difficult. However, is expected to exhibit what is called the self-averaging property, implying that it converges to its expectation value over D in the limit N + co. Hence we may instead compute [6) par, where the square brackets express the average over the data-generation process:
M
=
II
Σ
dz
P(DM wo)(...),
(13)
where P(DM wo) is given in Eq. 4 Unfortunately, the evaluation of [0] -(NN [log Zl is still difficult. The replica method is great aid in such a situation, via making use of the following identity:
[log D) lim log [2n) pM
(14)
x 7
In addition to this identity, we assume that n is a positive integer. This assumption enables us to explicitly compute the average [(---))p on the right-hand side of Eq. [4]. After computing this average, we take the limit + by relying what is called the RS ansatz. The details are in sec. g
expression of the average that analytically continuable from N to R, under
2.4 Related work
The statistical mechanical approach neural networks and machine learning problems has long history: some pioneering pieces of work are for associative memory (Hopfield) 1982), generative model (Ackley et al.
5

---

Published in Transactions on Machine Learning Research (4/2025)
1985), simple perceptrons Garrdner 1988: Gardner & Derrida 1988), supervised learning (Seung et al. 1992
and unsupervised learning (Barkai & Sompolinsky 1994). These facilitated various related studies, leading a research area of statistical mechanics for information processing [Nishimori] 2001 Mezard & Montanari, 2009). This approach has gained renewed interosts due to the recent upsurge in the field of machine learning, and some interesting findings have been accumtlating. In this subsection, we review some contributions along this direction, highlighting their relevance to and distinction from our study.
Baik et al. [2005] investigated behaviors of large eigenvalues of what are called the random non-central Wishart matrices in the infinite-dimensionall limit and identified critical thresholds, which have implications for inference and estimation in high-dimensional settings. Although focusing on random matrix theory, their work shares similarities with ours in terms of utilizing infinite-dimensionall analysis techniques. Lesieur et al. (2017) applied tools from statistical physics to study low-rank matrix estimation, emphasizing phase transitions and the performance of approximate message passing algorithms. Although their use of the replica method in asscssing the estimation performance is conceptually related to our approach, their focus was on matrix factorization and Bayes-optinal in inference, which differs from our exploration of resampling and reweighting for imbalanced classification tasks.
Mignacco et al. (2200) provided an analysis of the classification performance of the empirical risk minimiza- tion (FRM) with generic convex loss functions and the f2 regularization ion when the data are generated from the two-class GCCMs as ours. Their setup is quite similar to ours, but they focused on the effects of regu- larization on classification performance, in contrast to our current aim to clarify the effect of resampling and reweighting on feature learning. A noteworthy point of their work is that it offers a mathematically rigorous proof for the formula derived using the replica method. In order to enable the rigorous proof, they restricted their loss function to convex ones, while the replica method can handle even non-convex loss functions and in fact we discuss general loss functions that may not necessarily be convex. Loureiro et al. (2021) studied behaviors of generalized linear models (GLMs) when applied to data generated from the multiclass GCCMs in the ERM framework. Their analysis provided procise asymptotic results for training and generalization errors in high dimensions, and they examined how regularization affects performance. They also offered a proof of their formula, provided that both the loss and the regularizer are convex. One interesting point of this paper is that it pointed out that when the training dataset is generated by Generative Adversarial Network (GAN), the corresponding learning curve shows fairly good agreement with the theoretical curve derived assuming a GCCM as the data-generating distribution, implying a potential relevance to real data. However, this work did not deal with the problem of feature learning for imbalanced classification focused in this paper. Loffredo et al.] (0224) tackled the problem of finding the optimal undersampling and oversam- pling strategies in imbalanced classification. To this end, they computed several quantities characterizing the classification performance in a systematic way using the replica method. Their problem setup is directly relevant to ours, as they analyzed the effects of under under/oversamp /vveraamplin on classification performance. However, their focus was primarily optimizing sampling strategies for accuracy, whereas our study explores how resampling and reweighting affect the quality of feature learning.
Takahashi 2022) investigated use of pseudo-labels in self-training for semi-supervised learning using the two- class GCCMs. Similarly to our work, this study employed the replica method to derive sharp asymptotic characterizations, focusing on iterative updates of model parameters and proposing heuristics for pseudo- label refinement that yield performance close to that of fully supervisod lcarning. In another study (Takahashi] 2024), the same author provided replica analysis of under-bagging (UB) for imbalanced classification, again using the two-class GCCMs, comparing it with undersampling and simple weighting methods. This study revealed similarity and difference among these three methods and established superiority of UB in terms of performance and simplicity of implementation. While these pieces of work examined the impact of label imbalance in classification performance and used the replica method as our current work, our study uniquely focuses on interaction between beaamplingrreweighiee strategies and feature learning performance under class imbalance.
Adomaityte et al. (024) analyzed the impact of non-Gaussian, heavy-tailed data. distributions on classifi- cation performance using the replica method. As result, they demonstrated deviations from the Gaussian case, indicating the importance of considering the non non-Saasiyy and heavy-tailedness for capturing real data in theory. This is an important contribution, but their emphasis on heavy-tailed distributions contrasts
6

---

Published in Transactions on Machine Learning Research (4/2025)
with focus the class imbalance. Francazi et al (2023 investigated behaviors of gradient- learn- ing algorithms under the class imbalance, to find that the imbalance affects the convergence of the dynamics. To mitigate this, they also introduced techniques of normalizing the gradient in class-wise manner. While this study focused on class imbalance, emphasized optimization dynamics and is rather different from our aim. Finally, the work by Mannelli et al. (2024) explored the geometry of data and clarified how it introduces bias machine learning models, via applying the replica method to synthetic data-generation model called the teacher-mixture model. While they also examined the effect of the imbalance, their focus fairness and bias, and hence it does not align with the objective of the present paper. We investigate when the resampling and reweighting strategies enhance feature learning through detailed analysis using the replica method.
3 Theoretical Analysis
3.1 Overview
In this section we derive a formula for the sample-averaged free energy [o] = -(NN)~ ZJD in the limit N + oo by using the replica method under the RS ansatz. The formula is characterized by a small number of quantitics which are called order parameters, and the order parameters satisfy a set of equations called equalions of state (EOS), both according to the physics terminology. The dependence of the order parameters on the parameters, especially the reweighting factor $11, is of our special interest. in the paper and is systematically studied the basis of the EOS, A further simplified model inspired from the replica results will also be introduced later in this section to discuss the case with more than two classes
For notational simplicity, we use the shorthand notation (+(+) to denote 8+1(8-1) hereafter. The same shorthand rule applies to Bg and Ty as well.
3.2 Derivation of free energy and EOS under RS ansatz
The computation starts from evaluating [Z"] If n E N, we may consider n "replicas" of the original system which have distinct parameters {wa)-=1 and share the same dataset DM and represent [Z"] using them as
EM pmi E... e fow a an/V/++b))
2 DM = Tr(we
(15)
DM
where Tr (mm.)) )
[T Trw., The average over the dataset D yiclds
B ExE Ei.. Sop f(10 n. V +baa))
Tr
Tr 1. 1d
(16)
DM
where we let
L=
Σ Ty dx Pejy
y
Wo VN
og
3 Ei syllw] (ww
(17)
y=#1
The integral over x in Eq. 17
is cunbersome. However, the integrand depends on only through the Conditional on y, {uaYa obeys a multivariate Gaussian, thanks to the (3] on E(y) in the present set up thanks to the central limit theorem in the large-N limit in a more generic case. Letting E [. y] denote the average over {(y) given y, the conditional mean of ua given is
quantities { Ui, w x//T//V
NV.
Gaussianity assumption
E [ua yl 1 LE [ v)
ymaas
(18)
N
where we let m. = wi! wo/N. N. The conditional covariance of ua and up given y
E. (u,uub ub.
[)]
E [ua
EE [ub
v] Qabo
(19)

---

Published in Transactions on Machine Learning Research (4/2025)
where we let Qab w wt/N.
To proceed further, we assume the RS as follows:
Ma = m, Qab = Qoab + q(1 8ab).
(20)
This assumption allows us to express ua given y
Ua -y (VQ v(z) + ym,
(21)
where ta and are independent standard Gaussian random variables. Hence the integral over æ is recast into those over 2, (laaaa Ma-a---- Using the shorthand notation
[
dz
- -..)) D=(...),
(22)
2T
we have
L Σ Ty / Dz
Df Besloo vez) tomtbss) (omttbbb)))) )
==#1
Σ ry T y=11
"
(23)
where we let
h(t, 2, Q.q.m. = (VQ- - vQz) - m b,
(24)
23 used the invariance of the result with respect to (w..t.) ) - -2, t + -t and the symmetry of the loss with the single-argumes shorthand notation ((yh) f(h, y) introduced in sec.
and in the last line of Eq.
Equation (33 reveals that L depends on {wa)² only through Q (((@(((())))))))))) (ama)a). 1), (( (( Jam1), and m m((w, ) Hence the integral Tr( 2u-1 in Eq. (16 can be rewritten as that w.r.t. Q, q, m.m. as
2.2
f dQ dq dm V(Q, q, m) M (Q, g,m),
1 = TT 12 M =
(25)
where we introdueed the notation L(Q,q, m) to denote the dependence of L on Q, q, m, and where V(Q, q, m) represents the volume of the subshell specified by Q, q, m in the space of [wa)==)
V(Q. (..mm) m. Tr(w )i
II (NQ /wwall) 8 (N
wi u.)
II a<b
8 [ I wb)
(26)
We defer the details of its computation to sec.
A and here only show the result:
lim 1 n--0 nN
1 Q
-00 mm + log(2r) 1 log( + + m² + Q + S
1
log V (1, m) = Extr
(27)
(Q.4.))
2
where Extr denotes the extremization w.r.t. x. This extremization appears as the consequence of the Moldle-point/LLpllee
method which is valid in the limit N + co. The volume becomes finite only when Q 1, due to the normalization condition ||w|| N. In the same way we can com- pute lim,-40NN n. log M (Q, (QQq.m) m). Substituting these expressions to Eq. (14), the free energy Φ -BBN)- log Z averaged w.r.t. DM and in the limit N + xo takes the following compact form:
DM = lim 1 log [Z"] [2"]
-B6- = lim n nN
[log V V(1, (1,g,m) m) + log M (1,g, m)
n-00,N
nN
1 m + Ω
Extr
120220 999 mm + log(2n)
2 o+
log( Q + ②) +
m.
2
2 Q +
+a Σ Ty f Dzlog 1-11
D (f
Do (
(28)
where a= M/N. The quantities ( G, m. q, m. the order parameters of the present system.
8

---

Published in Transactions on Machine Learning Research (4/2025)
Zero-temperature limit 8 - Next compute the zero-temperature limit B + 30. As discussed in sec. 2.3 the posterior of w concentrates on the set of minimizers of the loss in the limit + oc. We further expect that the minimizer is unique. In view of this, we adopt the following ansatz for the asymptotic behaviors of the order paramcters around the saddle points as 3 becomes large:
q - 1 1
X = O(1,
m 0(1,
@ = - + 3Q.
82x,
nn Bm.
(29)
performing the This ansatz will later turn out to be consistent. Inserting these relations into Eq. variable transform e = t/ 4/vB and taking the limit B - 00 yicld
28
{ { 2e+ 2xx + mm
1 m
Extr 8-000 (0..m...)
lim Φ =
Q
/
Σ F=+1
Ty
(30)
where we let
G(u,h,8)
1
sl (h),
(31)
2
e - syl lo (Vxu 2) +m ybb }
1
by arg max
(32)
2
hy By (Vxu, + +2) + m + yb.
(33)
Here, By denotes the maximum point of G w.r.t. the variable v, which appears in evaluating the inner integral w.r.t. & in the last term of Eq. (28 in the limit 3 + oo via the Boelle-poont/LLappeee loss function nonconvex, the optimization problem in Eq. (32) may have multiple local optima, and hence numerical solutions require careful consideration, such as exploring different initial conditions. Note also that the dependence of by and hy on the integration variable is implicit in the above formulac.
method. When the
Equations of state (EOS) The extremization condition in Eq. 30 yields the following equations:
Q² m 2 + X,
(34a)
m Q
(34b)
=
Q'
(34c)
Ty
|
m
Drugs
(34d)
y== σy
X = a Σ Tyo / ==
D2
(34c)
This set of equations is the EOS for the present problem. An intuitive interpretation of the EOS is given in sec. B The term By appearing in Eqs. (34dl and (34e) is defined via Eq. (32) as a function of m, X b, y. Sy. Determining by easy we are allowed to assume differentiability of the loss on I:: introducing the di shorthand notation g 2n we find that by satisfies
By Syduv Xg (hy)
(35)
It should be noted that by appearing on the rinht- side depends on By through Eq. equation is a (non- -ion-)inear) equation on vy. If there are multiple solutions to Eq. (35) the one yielding the
33
so that the above
largest value of G(u,, hy, sy) should be selected. Under this differentiability assumption, the extremization
condition w.r.t. m can be computed from the partial derivative of G w.r.t. h. aG(u,h,s) 0h sg():: this 8h
9

---

Published in Transactions on Machine Learning Research (4/2025)
is because the partial derivative of G w.r.t. v vanishes since is fixed at the extremum value of G given h and 8 shown in Eq. [32]. The term sg(h) can be rewritten using Eq. 335 yielding the right-hand side of Eq. (34dh. A similar rewriting using Eq. (35) is done for the extremization condition w.r.t. X yielding the right-hand side of Eq. [4ch. Although the differentiability assumption does not hold for some losses having singularities such as the zero-one loss, those losses are usually representable as a limit of certain
differentiable function: for example, the zero-one loss can be expressed as
(h < 0)
1 (1 (anh(rh)) lim f01,; (h).
(1)
1/2
(h 0)
lim 7-000
(36)
2
0
(h 0)
7-000
Hence, for such losses with singularities, the above discussion should be interpreted as that for such smoothed versions of the losses, and the smnoothness-cnntooliingg parameter (7 in Eq. (36)) is sent to appropriate limit after the computation. The resultant formula (34) still holds after the limit even when the derivative itself is not meaningful in the limit.
The EOS constitutes the basis of the following study. special focus is on the overlap m = (w) wo wo/N, because it characterizes the performance of the feature learning as discussed sec. 2.2
3.3
Behaviors of quantities of interest
3.3.1 Overview
By specifying the functional form of the loss é, one can numerically obtaín values of the order parameters and the energy, via computing vi with the loss and numerically solving the EOS. We consider two choices for the loss & as representative examples: the zero-one loss with the perceptron (ui and the CF loss with the logistic function (CBio(h, (h, v) = -hy + log(2 cosh(h)). Hereafter these two losses are referred to as Olpe and CElo, respectively. The aim here is to investigate how the order parameters depend on the parameters in order to assess the feature learning performance. In the following, we investigate two cases: one is the equal-variance case of = o², where the two classes y = +1 share the same variance, and the other is the conoqual-varianee case of + o². where the variances of the classes y = 11 are different.
sgn (yh))/2
3.3.2 Equal-variance case (o7 02)
We start from canonical situation where the two class variances are equal: of = o² =: 02. As examples, we compare the balanced case r+ 0.5 and an imbalanced case r+ 0.2, with a = 0.6. For illustration of these cases, the probability density functions (PDF) of æ projected onto wa/ are plotted in Fig, 1 Besides, we fix a = 20 in the following plots unless otherwise stated; the results for other values of a, not
0.6, r+ 0.5
0.6, r+ 0.2
0,25
0.3
0.2
PDF
0.15
PDF
0.2
0.1
0.1
-10
-5
0
5
10
-10
-5
0
5
10
w. x/ VN
w x /
N
Figure PDFs of æ projected onto wo with a = 0.6 for r+ = 0.5 (left) and 0.2 (right).
too small, were qualitatively similar far have checked.
10

---

Published in Transactions on Machine Learning Research (4/2025)
Firstly, show and plotted against b for s- = 0.1, 0.9: the results for r+ 0.5 and 0.2 are shown in Fig. 2 and Fig. respectively. It should be noted that the class centers are located at + V in our problem setting, so that a reasonable choice of the parameter b in view of the task of classification would intuitively be b [-1,1] indoed, otherwise, the two class centers + / N are located on the same side 0. the following, we nevertheless investigale the behaviors of the models over wider ranges of Fig. (a) and Fig. 3 (a) are the results with 01pe, whereas Fig. (b) and
of the decision plane æ/vN
a=20,0,=06. =, 1,66,,,05..,50....
a= 00.6, 0..6, r055
a=20, -0,6, =0.6. =0.5.__00.
0.9
0.85
0.85
0.85
2
0
b
(a) Zero-one loss with perceptron loipe-
a=20, 0 =0.6, a =0.6, =0.3, s =0.1
a=20, o =0.6, =0.6,1 =0.5. =05
a=20, a =0.6, a =0.6, =0.5, =0.9
0.99
0.955
0.985
0.985
0.98
50
20
20
0
0-
-2
0
2
-4
-2
2
-4
-2
0
2
(b) Cross-entropy loss with logistic function (CElo
Figure 2: Plots m and u against the balanced case r+ 0.5 for s = 0.1 (left), 0.5 (middle), and 0.9
(right). (a) Zero-one loss with perceptron lotpe- (b) Cross-entropy loss with logistic function fCElO
Fig. 3 (b) are those with CElo. As observed Fig. 2 the b-dependence of m in the balancod case is very different between Olpe and CElo: with Olpe, it is almost symmetric w.r.t. b even with the strong reweighting factors $+ = 0.1 and 0.9, while it exhibits clear asymmetry with CÉlo. The values of b at which achieves its maximum are also different: m achieves its maximum around b = +1 with 01pe, while with CElo the maximum is achieved at b less than -1 for $4 = 0.1, at b = 0 for 84 = 0.5, and at b larger than 1 for = 0.9. Meanwhile, as observed in Fig. 3 for the imbalanced case = 0.2, the maximum of m seems to obtained at a large positive b in all the cases: this is considered to be natural since the positive bias enhances the probability of the minority class at r+ 0.2. As an overall tendency, the value of m tends be larger with CElo than that with Olpe, suggesting the superiority of the CE loss in the feature learning.
Secondly, to examine the maximum performance of feature learning, compute the maximum of against b, mmmax max, m(b), and plot it against 84 in Fig. 4 Fig. (a) and (b) are with Olpe and CElo, respectively; the left and right columns are the plots with r_ 0.5 and 0.2, respectively. The maximum location b(mmux) = arg max mb)) and the corresponding energy value u(maaxx = u(b(mmax)) are also
plotted. As a general trend, Mmax shows very weak dependence 84 in all the cases investigated, can be seen in the very small ranges of the plots. An interesting observation is that with Olpe the maximum of Mmax is obtained at the no-reweighting situation 84 = 1/2 even in the imbalanced case. (Fig. 4 (a), right), while with CElo the m's maximum is located at some value different from 84 1/2 even in the balanced case r+ = 0.5 (Fig. (b), left). This property of 0lpe persisted as far as we have numerically investigated. This may be related to Kang et al's observation, although Olpe is not typically used in practical situations.
In practical situations one cannot directly maximize the overlap m since one does not know Wy- One will instead minimize the loss obtain a reasonable estimator. From this viewpoint, thirdly, the minimum of
11

---

Published in Transactions on Machine Learning Research (4/2025)
a=20), o, =0.6, =11,, r, r,=0.2, s,001
o=20, a, =0.6, =0.6, 1,=1122, ,
o=20. o, 0.=006,00=10 0,6666 =,220, =,-2...=0099
0.9
E 0.85
0.85
0.85
0.8
0.8
5
2
22
0
(a) Zero-one loss with perceptron lorpe- a=20. 0, =000066600000.... a =0.6. =0.2, =00.5
00= o 6, =0,66,, 5, s,02s,
==20 0_ =0.6. =0066. r.0.6.r==0...... s_=.2.._==099
0.98
0.98
0.96
0.96
0.96
50
10
50
5-
-2
-2
2
(b) Cross-entropy loss with logistic function EoElO-
Figure 3: Plots of m and u against b in the imbalanced case 7 = 0.2 for 84 = 0.1 (left), 0.5 (middle), and 0.9 (right). (a) Zero-one loss with perceptron loipe- (b) Cross-entropy loss with logistic function ECElo-
w.r.t. b, Umin = min; u(b), is plotted against 8+ in Fig. with the minimum location b(Mmin)) = arg min u(b) and the corresponding overlap value m(Umin) = m(b (((()))))) This figure shows an intriguing behavior of ml(uiaa in the balanced case r+ = 0.5, its curve symmetric around 8+ 0.5 and the maximum is obtained 84 0.5 as expected, with both Olpe and CElo. In the imbalanced case r+ 0.2, however, the tendency is different between Olpe and CÉlo: with Olpe, there is maximum at certain value of 84 greater than 0.5, which is natural because the values of 84 greater than 0.5 enhance the probability of the minority (positive here) class. With CFlo, on the other hand, the opposite occurs and the overlap maximum is obtained at 8+ = O, meaning the complete disregard of the minority class, and we numerically confirmed that the corresponding minimum location b(Umin) goes to oo. This rather counterintuitive, and provides a lesson that not straightforward to predict how the interplay among the loss, classifier, and reweighting factor would influence the feature learning performance.
In recent practices, the bias b is occasionally neglected (Cao el. al. 2019 Menon al. 2021: Kini el al. 2021). This is because DNNs learn feature vectors also from the data, and hence the adjustment of the feature space origin, that is the effect of the bias, can be incorporated by the lcarning even without the bias. Accordingly, we lastly examine the case b = 0, which is actually natural choice since the origin is located at the middle point of the two class centers in our setting. The overlap and energy in this case are plotted against 84 Fig. G very interesting observation that the maximum of m is achieved at the no-reweighting situation $ 0.5 all the cases. Actually, this property holds irrespectively of the loss, model, the degree of class imbalance: we will provide strong analytical evidence of this fact. later in 3.4 This property means that the feature learning has its best performance when resampling/ haanppliin//wwwghttiii /rwweihttiingggggg is applied if the bias is appropriately chosen to make the decision boundary to be located equidistantly from the two class centers. This provides an analytical support for Kang et al's observation and constitutes one of our main results in this paper.
To summarize, we enumerate our findings in sec. 3.3.2
1. The overlap value m tends to be larger with CElo than with 01 0p...
12

---

Published in Transactions on Machine Learning Research (4/2025)
a=20, of 00=0.6, =0.6, =0.5
a= o, -0-6.6, =0.6, =0.2
max
0.90105
0.92012
m
0.901
0.9201
u m mmx
max
1.046
b( m
1.044
0
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
8
(a.
Zero-one loss with perceptron folpe-
a=20, 0, =0.6, o =06, =0.5
a= σ =0.6, =0.6, r =022
max
0.986
0,9895
0.985
0.989
200
u ( m
100
0
max
2
6
b( m
4
2
0
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
(b) Cross-entropy loss with logistic function lCElo
Figure 4: Plots Mynax ul mmmmaax) b(m b(mmax) against 8+ for r+ 0.5 (left) and r+ = 0.2 (right). (a) Zero-one loss with perceptron lotper (b) Cross-entropy loss with logistic function fCElo-
13

---

Published in Transactions on Machine Learning Research (4/2025)
2=20, --= 0.=0.6, 0 =0.6, =.5
a= o 00=0,6, o =0666 =02
m (u min
0.85
0.825
0.84
0.82
min
0.05
0.02
0.1
b( u min
-0.12
0
-0.16
-0.2
-0.1
-0.24
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
(a)
Zero-one loss with perceptron Polpe-
(x=20, 0, =0.6, =0.6, r,0.5
a=20, 0-- =0.6, =0.6. =0.2
min
0.98
0.984
0.983
0.97
2
min
0
b(u min
0
0
-2
-2
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
(b) Cross-entropy loss with logistic function loElo-
Figure Plots of m (min) Umin (min) against 8+ for r+ 0.5 (left) and r+ 0.2 (right). (a) Zero-one loss with perceptron lotpe (b) Cross-entropy loss with logistic function forlo-
14

---

Published in Transactions on Machine Learning Research (4/2025)
a=20, =0.6, 0 =0.6, -
=0.5
a=20, of =0.60,6, -0.6, 002.
0.85
0.85
m (b=0)
0.84
0.8
0.83
0.1
n ((bb))
0.05
0.05
0
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
$
(8)
Zero-one loss with perceptron lorpe-
x=20, 0- 00,, 006,, r,0.5
a=20,, 0- =0.6, o =0.6,
r
=0.2
(b=0)
0.98
0.98
0.96
0.97
0.94
3
2.02
2
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
(b) Cross-entropy loss with logistic function lCElo.
Figure Plots of m and u at against 8+ for r+ 0.5 (left.) and r+ 0.2 (right). (a) Zero-one loss with perceptron loipe and (b) Cross-entropy loss with logistic function (CEb- A crucial observation is that the maximum of m is always obtained at the reweighting situation $ 1/2, irrespectively of the models, losses, and the degree of class imbalance.
15

---

Published in Transactions on Machine Learning Research (4/2025)
2. The values of b with which maximum values of achieved tend to be strongly dependent the choice of the model and the loss.
3. The maximum overlap Mmax max; m(b) also shows a strong dependence on the choice of the model and the loss. With Olpe, Maax takes its largest value at the no-reweighting situation 8+ 0.5, whereas with CElo it is obtained at some extreme values of 84-
4. The overlap m(mmin) at the point of minimizing the loss u shows a moderate dependence on 84- The dependence on 84 natural with Olpe but is counterintuitive with CElo. With CElo, the maximum at 8+ 0.5 in the balanced case but is the extreme value of $4 in the imbalanced case.
The overlap m takes ils maximum at the no-reweighting situation 84 0.5 irrespectively of the loss, model, or degree of class imbalance when the bias is appropriately chosen make the decision boundary to be located equidistantly from the two class centers.
The assumption on the bias in item 5 of the above list is what is considered desirable in the standard view of classification. Hence, Kang et al.'s observation a property that holds widely when features and bias are set to be in such a desirable situation, which conversely implies that their learning works well. Meanwhile, our other findings such as item in the above suggest some other bias values different from the desirable one; the resultantly selected value tends take an extraordinary value outside the reasonable range [-1, of the bias b in the present setting. Presumably, this has prevented researchers from examining such extreme bias values in practical situations, and it may be an interesting future work to study such extreme biases in real-world datasets.
3.3.3 Nonequal-variance case
We turn to the nonequal-variance case. As an example, we examine o. = 1, a_ = 0.5 with r, = 0.5 and 0.2 as depicted in Fig. 7 We compare the result for this case with the one in the equal-variance case, especially
0+ = 0,
a_ 0.5, r+ 0.5
0+ = 1, a_ 0.5, r- 0.2
0.25
0.3
0.2
PDF
0.15
PDF
0.2
0.1
0.1
0.05
-10
-5
0
5
10
-10
-5
0
5
10
wi x VN
w T
x /
Figure 7: PDFs for 0. = 1,0 = 0.5 at r+ = 0.5 (left) and r+ = 0.2 (right).
focusing on the reweighting factor dependence of m after erasing the bias dependence as in Figs.
Figure g is the counterpart of Fig. in which mmax and the related quantities are plotted against 84. As expected from the asymmetry between the classes, the maximum location of mmax is not 8 1/2 anymore with either 01pe CElo. Another interesting observation is that the maximum location is 8+ 1/2 with Olpe and 8+ 1/2 with CElo, showing that the effect of reweighting on the feature learning is not simple.
Next, we study the lossmiingg result in Fig. g which is the counterpart of Fig. 5 This time, again due the asymmetry, the maximum location of m is not 84 = 1/2 in all the cases. is at 84. > >1/2 with Olpe and at 8+ with CElo, which in contrast the wverlapmmaximizing result in Fig. Looking at both of the equal- variance and nonequal-variane results with CElo shown in Figs. and respectively,
16

---

Published in Transactions on Machine Learning Research (4/2025)
a=20, of 0,=1, 0.=0.5,r==055
a=20, =1=1, =05, -022
m max
0.92
0.935
0.91
0.93
b m max u max
2
0
1.2
1.2
1.1
1.1
0
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
(a) Zero-one loss with perceptron lolpe-
a=20, 0, 0,=1, 0 0,5
a=20, 1.=00l1... =1, 0 =0.5, r =0.2
m mnx
0.9879
0.9923
0.9877
0.9922
max
100
200
50
100
b( m max
6
6
4
0
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
8
(b) Cross-entropy loss with logistic function lOElo.
Figure Plots of mamx: ul mmxx) b(mmaxx against 8+ for r+ 0.5 (left) and r+ 0.2 (right) the nonequal-variance case. (a) Zero-one loss with perceptron loipe- (b) Cross-entropy loss with logistic function
lcElo-
17

---

Published in Transactions on Machine Learning Research (4/2025)
a=20, o =1, =0.5, -0.5.r_-0.5 =055
Q=20, o,=l, =05,, rrr,,-0 =0.2
m min
0.79
0.82
0.78
0.8
0.77
0.1
min
0.05
0.1
b( u min
0.2
0.4
0-
-0.2
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
$
(a) Zero-one loss with perceptron folpe-
a=20, 0 =1,
0-=1, σ =0.5, r- 1,50.5
x=20, 0 =1, =0.5, =0.2
m (u min
0.98
0.95
0.96
0.94
0.9
2
1
u
b( min
-1
-2
-3
0
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
(b) Cross-entropy loss with logistic function loRlo-
Figure 9: Plots of m (umuiu)), umin b( ((umi)) against 84 for r+ 0.5 (left) and r+ 0.2 (right) in the nonequal- variance case. (a) Zero-one loss with perceptron coppe- (b) Cross-entropy loss with logistic function ECE.-
18

---

Published in Transactions on Machine Learning Research (4/2025)
see that the asymmetry due to the imbalance both in the number of examples and the variance magnitude commonly leads to extreme values of for the best feature learning.
Finally, we examine the no-bias case b = 0 in Fig. 10 Similarly to the above two cases, the maximum of m
a=20, 0 =1, 000,0,1100000
=0.5, -0.5,r,=055
a= 0- -0=1, 0 =0.
m(b=0)
0.8
0.8
0.75
0.7
0.75
0.5
0.1
0.05
0
0.2
0.4
0.6
0.8
0
0.2
0.4
0.6
0.8
$
(a) Zero-one loss with perceptron foipe
Q=20, 0 =1, σ =0.5, r, =0.5
a=20, =1, =0.5, =0.2
m(b=0)
0.95
0.9
0.9
0.8
3
3
u(b=0)
2.5
2-
2
1
0.2
0.4
0.6
0.8
0
0.2
0.4
0.6
0.8
(b) Cross-entropy loss with logistic function fCRl
Figure 10: Plots of m(b = 0), u(b = 0) against 84 for r+ = 0.5 (left) and r+ = 0.2 (right) in the nonequal- variance case. (a) Zero-one loss with perceptron loipe- (b) Cross-entropy loss with logistic function fCE..
is not at 8+ 1/2. An interesting observation that the maximum located commonly in 8+ 1/2. This may not be surprising since the class with -1 has smaller variance and thus is considered to carry more information about the feature direction.
Overall, in the nonequal-variance case, we found no empirical evidence supporting Kang et al.'s observation. This implies that the equal-variance condition of the feature across the two classes may be an important ingredient for their observation.
3.4 Analytical evidence that the maximum value of m is achieved at 84 1/2
In this subsection, we provide an analytical evidence for the observation that m takes its maximum al the no-reweighting situation $+ = 1/2 irrespectively of the loss, model, or degree class imbalance when the input distributions of the two classes are oqual-variance and the bias is appropriately chosen to make the decision boundary be located equidistantly from the two class centers. The last two conditions mean 0+ (=-(0) and b in the present setting, and assume them throughout this subsection.
19

---

Published in Transactions on Machine Learning Research (4/2025)
The idea is very simple. What we want to show is the following equality:
dm
O.
(37)
ds+ 84
s+=1/2
It means that no reweighting/rrssapping point (84 1/2) is an extremum of m. In deriving this, we need to take the derivatives w.r.t. 8+ of the EOS (44. Taking the derivative and rearranging the formulae using the simple relation between Q and (m, X) in Eq. (34a),, we have the following set of linear equations among the first order derivatives:
T
(38a)
ds+
ds +
X
m
= c + T
d
m
(38b)
ds+
ds+
X
The matrices T, T and the vector e have simple analytical expressions depending on the order parameters, the explicit forms of which are given in Eqs. 66bb) and 69 in sec. C After lengthy but straightforward algebraic computations, we can show that Eq. specifying details of the loss, the model, or the degree of class imbalance. This strongly supports the empirical
(38) implies Eq. 37 under the above two conditions, without
observation that the maximum m is achieved when no recamplingrrewoihhting is applied. The detailed
computations are deferred lo sec. C
It can be seen from the algebras in sec. C that the symmetry ((h, y) e(-h, explained in sec. is important in deriving the above result, but not its details, implying the wide applicability of the obtained result.
3.5
Relevance to DNNs
Here, we discuss how our findings compare to those of Cao et al. or Kang et al. (Cao et al. 2019 Kang al. 2020).
First of all, we should keep in mind that in the experiments of Cao et al., as well as those of Kang el al., whatever backbone networks (bidirectional LSTM, variant of ResNet, etc.) were used as feature extractors, the final layer of the models are linear classifiers. Therefore, it is natural to consider that our input vector w corresponds to the feature representation h of the final layer in their models. Thus, if the distribution of h has equal variance across classes, this would match with the assumptions underlying our findings. Although the wlass-wise distributions of h have not been elucidated in their papers, it is not difficult to numerically check whether this point actually holds not.
Moreover, a recent paper reported that an interesting phenomenon widely occurs in successful DNNs for classification (Papyan et al., 2020). This phenomenon is called neural collapse (NC), and one of its important ingredients is willhin-class variation collapse, meaning that the final-layer feature vectors of the same class converge to an identical vector as a result of learning. This implies that the equal variance assumption holds in the limit where the variance approaches zero. The mechanism why NC occurs is also understood from simple model analysis (Fang et al. 2021), which suggests that this phenomenon universally occurs in sulficiently expressive DNNs with reasonable regularizations. Considering this, it is plausible that the equal-variance condition underlying our results may indeed hold in actual DNNs.
3.6 A further simplified model with more than two classes
Can the results so far be extended to the case with multiple classes more than two? While extending our problem setup to multiple classes and analyzing it using the replica method is possible, such an analysis would be rather complicated and we avoid doing it in this paper. Instead, in this subsection, we consider a further simplified model allowing us to analyze the multiclass problem easily.
From the discussion of the previous subsection, we understand that the symmetry in the loss and the problem setup is important to find the overlap maximum in the no-reweighting situation. On the basis of
20

---

Published in Transactions on Machine Learning Research (4/2025)
this understanding, reflecting the symmetry argument, propose the following "one-dimensional" feature generation process for multiclass classification:
WD Pp
+ Ept #= 1,2, 1.2..... M,
(39)
where the true feature is again normalized | = N. The label by R is assumed to take of K values, where K corresponds to the number of classes, and Sn is assumed to be a random vari- able following o² where IN is the identity matrix of dimension N. We further assume that 4, 1,,22...,4a 1.. E1, E2........ E,...., EM are independent. Here, only one true feature vector wo governs the feature space, and thus the feature learning performance can quantified in terms of the accuracy of estimating wo as in the case of binary classification discussed so far. For the sake of simplicity of the analysis, we con- sider, instead of a classification loss, a simple unsupervised loss for feature learning. Concretely, we consider hy w Fy and obtain the estimator by maximizing the variance of {hy } under the presence of set of sample-wise reweighting factors { satisfying 84 0 and MM 8p = 1. The loss is formulated as
M
H(w) = Σ
0k - 1122 = -w Aw,
(40)
M where denotes the weighted mean with the rewcighting factors {sp I (i..., = Ep=z Sufu), and where
M
A
Sp (1p
-X)
.
(41)
The minimizer of Eq. 40 under the normalization condition ||l|| N becomes our estimator of wo- Namely, the estimator given by the eigenvector of the largest eigenvalue of A. The matrix A can be rewritten as
Wu VN
Wu VN
Wu
Wu N
+E)
+E)
+
+
E
(42)
wp (Wo)
+ E R
(43)
N
where
T F (f)
(44)
EE - E36
(45)
R = (LE Tε)
We
(IE TE)
(46)
The last term R the right-hand side of Eq. Neglecting this term, have our estimator as
43 is mean zero and is negligible if M is large enough.
④ = arg max
T
We
Wo
+ Σ
(47)
We furthermore assume that the reweighting factors { are given depending on M and satisfy limar+400 M 5t 0. The empirical covariance E then converges to o²1 as M large-M limit, the maximizer w. which is the cigenvector associated with the leading eigenvalue of A, ap- proaches wp irrespectively of the choice of { yielding m + 1. Meanwhile, for large but finite M, stochastic fluctuations of S make w to deviate from Wo, resulting in decrease of m from 1. A detailed analysis reveals that minimizing the fluctuations leads to the best estimate of wo, which is achieved at the
IM
Hence, in the
21

---

Published in Transactions on Machine Learning Research (4/2025)
resampling/tewwiihttn situation 8p 1/M, Vy average. This can be easily shown by computing the mean squared deviation of the diagonal Σn from o², as
M Σ
M 1
Eg (Eu- 22) =0 {
(48)
Minimization of the right-hand side under the constraint Σx= S4 1 yields the uniform weights sn 1/M, Vu, implying that hoonppiinggrwwighinng leads to the best feature learning performance. This provides simple demonstration supporting Kang al's observation in the multiclass case.
4 Numerical experiments
In this section, we conduct numerical experiments to verify the correctness of the theoretical analysis pre- sented in secs. 3.2 3.3 Our theoretical framework built upon a method that assumess N + co. Thus, if our numerical results, computed with sufficiently large N, align well with the theoretical findings, this would support the validity of our analysis. For this purpose, we only examined the case with CElo in this section since the CE loss is convex and thus the numerical optimization is relatively easy; such good property is absent in the zero-one loss. The standard interior-point method was used for the optimization. In the following results, we conducted simulations with N = 400 and took a sample average over 100 different realizations of the dataset; the error bar is the standard error in the average. The parameter a was fixed to a = 20, which is identical to the value used in sec. 3.3
Equal-variance case We first experimented the oqual-variance case 0+ = =_ = 0.6. We start from plotting m and a against b for r+ 0.5, 0.2 and 84 = 0.1, 0.5 Fig. 11 The agreement between the numerical results (markers) and the theoretical ones (lines) excellent, justifying our theoretical treatment. Although our theoretical analysis assumees the high-dimensional limit N - co, this numerical result indicates that several hundreds of N can be regarded large enough.
For further quantification, we evaluated the loss-minimizing bias value and the corresponding Umin and m(Umin) from the numerical experiments. Their plots, along with the eheoretically-evaluaeed curves, are given in Fig. 12 The agreement between the numerical and theoretical results is again very good. Even the nontrivial location of the maximum point of m(Umin) is reproduced by the numerical experiments.
The last result for the equal-variance case is the plot at b O: m(b 0) and u(b = 0) are plotted against 8 with the theoretical curves in Fig. 13 The agreement is again good, and the maximum of m(b 0) is approximately obtained at the resampling/reweighii case 1/2, numerically supporting main result in this paper.
Nonequal-variance case We next experimented the nonequal-varianee case 0+ 1, 0.5, and show the results briefly. The plots of m and u against b for r+ = 0.5 and 0.2 are shown in Fig. 14 Yet again, the agreement between the numerical and the theoretical results is fairly good. Our theoretical result is thus validated even for the noncqual-variance case.
5 Conclusion
In this paper have studied toy model of binary classification for investigating the effect of the resam- pling/rew hhang//wweihhtng the feature learning performance in the imbalanced classification; special aim is providing a theorctical basis for Kang et al's observation that the best performance of feature learning is achieved without any resampling/ het////wweeinggggg The model's feature space is R and the class centers are assumed to be described by =wo EWo/VN(( RM). The data generation is on the standard i.id. assumption with a label distribution having a control parameter of class imbalance and input distributions being al- lowed have different variances on the two classes. this setting, the feature learning performance can be quantified as the accuracy of estimating wn- The analysis of this and related quantities is conducted in
22

---

Published in Transactions on Machine Learning Research (4/2025)
a=20, 0 =0.6, 0.= 0.=0.6, r $,=0.5, $--5,$=-00 =0.1 (CElo)
a=20, o -00666,00
0 r,-00,,,,-000
s, =,001 (CHlo)
0.985
0.98
0.98
0.97
N=400
N=400
0.975
0.96
-4
-2
0
2
6
-2
0
2
N-400
50
N=400
0
-4
2
6
-2
0
2
b
(a) 84 0.1 case.
2=20,
=0.6,
006,, r =0.5, 8 =0.5 (CElo)
a=20, 0 -06666 o =0.6, r s_00.5 (CElo)
0.985
N=400
0.98
N=400
0.97
0.96
0.98
-4
-2
0
2
4
6
-4
-2
0
2
4
6
N=400
N=400
50
.4
2
0
2
4
6
-4
-2
0
2
4
6
b
(b) 8+ 0.5 case.
Figure 11: Plots of m and u against & for r4 0.5 (left) and (right) at the equal-variance case a+ a_ 0.6 for (a) 84 0.1 and (b) 8+ = 0.5. The markers denote the numerical result and the lines represent the theoretical one. The agreement between them is fairly good.
a=20, o =0.6, a =0.6, =0.6,r,00.5 =0.5 (CElo)
a=20, 0. -0=666, o 0,2 (CElo)
0.2
(u min
0.98
0.984
m
0.983
N=400
0.975
N=400
min
2
1.5
1.5
40400
f N=400
0.5
b( u min
0-
-1
No400
N=400
-2
-3
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
Figure 12: Plots of mn(umin) and Mmin against 84 for r+ = 0.5 (left) and 0.2 (right) at the equal-variance
case 04 = a_ = 0.6. The agreement between the theoretical and numerical results is again very good.
23

---

Published in Transactions on Machine Learning Research (4/2025)
a=20, 0 =0.6, o. =0.6, + =055 (CElo)
a=20, o 0
=022 (CElo)
0.985
0.98
m(b=0)
0.98
f N=400
0.97
0.975
0.96
N=400
2.03
3
N=400
2.02
2.01
1 N=400
2
2
1.99
0.2
0.4
0.6
0.8
0.2
0.4
0.6
8
Figure 13: Plots of m(b = 0) and u(b = 0) against 8+ for 7+ = 0.5 (left) and 0.2 (right) at the equal-variance
case 0+ =0_ 0.6.
2= =1, 0 005, r,S55, 0, =011 (CElo)
o=20, 0. 00=. =0.1 (CElo)
0.98
E 0.97
N=400
E 0.95
0.96
No
0.9
-2
0
0
i N=400
= 50
= 50
N0400
0
0
-2
0
0
b
b
(a) 84 = 0.1 case.
2020, 0 =1, =0.,, =05, =0.5 (CElo)
a=20, =1, σ =0.5, r =02, =0.5 (CElo)
0.98
0.98
0.96
4=0
E 0.94
400
0.96
0.92
0.9
-2
0
0
2
4=000
N0400
50
= 50
0
0
b
b
(a) 84 0.5 case.
Figure 14: Plots of and against b for r+ 0.5 (left) and 0.2 (right) for the nonequal-variance case
0+ 1, 0.5. Two cases (a) 84 0.1 and (b) 84 0.5 are shown.
24

---

Published in Transactions on Machine Learning Research (4/2025)
the nigh-dimensional limit N - oo keeping the dataset size ratio M/N finite, using the replica method from statistical mechanics.
Our theoretical analysis has revealed that the best performance of feature learning is actually achieved with no caaammlling/eweignttiin for a fairly wide range of losses and classifiers when input distributions are of equal-variance and the bias set so that the decision boundary is located equidistantly from the two class centers: this is a desirable situation from the standard viewpoint of classification. The key of the derivation is the symmetry of the loss and the problem setup, explaining the wide applicability of the result. This provides theoretical basis for Kang et al's observation through the connection between their last layer feature representation and our input vector æ and also implies that their learning result achieved the desirable situation. The emergence of such desirable situations as a result of learning may be understood from the viewpoint of NC (Papyan et al.) 2020 Fang et al. 2021]. More quantitative analysis has been conducted in two cases: the combinations of the cross-entropy loss and the logistic function and of the zero-one loss and the perceptron. Although we have found that optimization of the energy or the overlap over the bias b yields the better overlap than the above desirable sit uation, the selected bias values tend to take extreme values and also be highly dependent The practicality of such sensitive results should be tested by experiments in more rcalistic situations, which constitutes interesting future work.
the choice of loss, classifier, and the parameters.
Numerical simulations on the N = 400-dimensional systems have been also performed to check the validity of our theoretical results. The result showed good consistency with the theoretical one, which verifies our theoretical results and also reinforces the practicality of our theory derived in the limit N + 0o.
As future direction, it will be interesting to extend the analysis in this paper to multiple classes. Although we have proposed a further simplified model to treat multiclass classification on the basis of the insight from the analysis, the model is too simple in that it is essentially a one-dimensional problem with an unsupervised loss: this is incompatible with the standard treatment of multiclass classification. Such an extension would involve some difficulties related to the arrangement of the cluster centers. Some reasonable assumptions simplifying the analysis would be necessary, and the accumulated practical knowledge machine learning communities would help find appropriate assumptions, just as Kang et al's observation which inspired the present study. Another interesting extension is to the semi-supervised setting with both labeled and unlabeled data. The effect of unlabeled data for performance is debatable from a. theoretical viewpoint (Yang & Xu 2020) while the benefit is evident in some applications (Jing & Tian 2020). A toy-model study would be helpful to resolve this puzzle and to find conditions under which unlabeled data improve the performance.
A Volume computation
To handle the Dirac measures, we employ the following identities and trick:
/ aQr (NQ ||wall | Cc f NoG-) 1Q
1 C
(49)
1 Ci ʃ dgō (Nq- w uv) = Cs/ f / dqddee -N ws
=Cs 1 1 C1 / dmo (Nm- wo wa) Cs3 / ddmdiine - Nohm-ttionn wa
(50)
(51)
where the Fourier expression of the delta function is used to obtain the expression on the rightmost side of each equation, and where C1, C2, C3 are normalization constants that are irrelevant the following computations and will be discarded hereafter. Using these, we obtain
V / dQdgdmdQddddi e (ttnQq fn(n-1) ())g--mmm))))))
x Tr [w. 12 ||wa|l 7 Ei wo wath Eneo w. wh
= f dQdqdmddddddn e N (2nkQ-- -o000---are- ja(n 1)19- nomm I.
(52)
25

---

3.2 Published in Transactions on Machine Learning Research (4/2025)
The factor I is computed as follows:
I =
/
[I
8(N
1| 13) - 1 E... w/ wat E acc w. wb
rico
idAa 2nT
# E. Ao
-000
/
dwa
e
$
Aallma - =0 [w Eis my mato E w0 my
rioc
idAa
N Ao
Dis log Dz IT (s dwe frout ((0+0)) +m/p://)ummuuuuueeeee vizu)
II
e
(53)
2T
ioc
Here, the integration w.r.t. Aa can be replaced by its oxtremization condition thanks the saddle- point PointLapplce method. This yields the symmetric solution A A Hence,
I= [ + E.n, log Dz (a (fee"" - 2=-===0-00000 +Q- + +imwort Visjw
(mowps 1 (
|nna+ Exn log Dz ( Vx+tg+o"
a 2++Q44
(54)
The last term in the exponent can be rewritten in the limit - 0
(mannn tua) n
lim n n nN
Σ log
/ D=
(V
2m
A +++
A"
+
N
(Mune: + /42)²
= log(2T)
log(A* +Q+ +
D2 A +
m +
= log(2T) log(A* + + ①)
(55)
2
2 A + Q 4
where the relation Σ, w N used in the last equality. Overall, the limit n yields
1
lim
log g
a a- +00 nN
1 n +
Extr
QQ 2 -09 mm + A" log (2T)
2 log(A" + +Q+
(56)
2 A+ + 2 +
The extremization conditions w.r.t. Q and A are degenerating. Rewriting Q - Q A erases this degeneracy and makes the A d'ddependence very simple as (-(1/2)Q+1/2)A* in the equation. The extremization condition w.r.t. A thus yields Q = 1, leading to Eq. 27
Interpretation of EOS
In this appendix, we provide an intuitive interpretation of the EOS (34 derived in see. the first line of 53 for the factor I. One can rewrite the exponent of the integrand as
We start with
m W. 2
w W. + Σ T wT
in Wo + vg
Wa
m
|wol'
Q g
ww. ||
2
2ÿ
2
(57)
Using the Hubbbard-Stratonoviic transform
e||a||?/
N/2
/
=dz
(58)
2T
26

---

Published in Transactions on Machine Learning Research (4/2025)
with c= and [(m/V])wo + VQΣ
wal/Va. rewrite the integrand as + Ees || 1| +h Σm_1 mn my +4 Encs "n
e
wwlll
2t Dam l/wall"
1
N/2
/
|=||
2 wall
1| dz 2oolll++iih +iiwo x+ Ea= w. 2i Eom1
Q
1
N/2
/
3/ 2 1
II du (12---z00 gmm 11 |tou edd @ dz.
(59)
This formula,, after taking the limit n - O, implies that the problem of eesttmaating wo is equivalent, in the limit N + xo, to estimating it from its scaled and noisy version muw + n with Gaussian noise (( gIN) by assuming the likelihood z turns out to be N ( .+q IN). and the posterior average with respect to this model turns out to be corresponding to the average ((...) over the Boltzmann distribution defined in Eq. 12
1=ll=-@ and the prior e -lll, The posterior distribution of given
Let ú á be independent samples from the posterior distribution p(w z). One then has
④ wol
E Elw0
z]
m
(60)
N
Q + á
N
Q +
|1
[
Q 22 + m²
+ Q+1
(61)
(Ô + Q)2
(Ô + Q)2
E 1
w0']
=|
mi² +
(62)
N
(Q + Q)
N
(Ô+
which should be equal to m = ((w) T WolD /N, Q ([ /N, and q y= [1]][w))]]] (/M)))l)MM//// /N, respectively.
These provide an interpretation of the EOS: in the limit N + c0, estimation of w may be regarded as being performed with the Gaussian model defined as above, whose parameters m, Q, q are to be determined via scalar estimation problem defined in terms of the loss l. The parameters m,, Q, should be taken so that
the estimate of w has length vN, which implies that Q= holds and hence Q 24 + m = (Q + g) from Eqq. 61]. Under this condition one may forget the constraints on |wa]l2 in Eqq 53 as they will be satisfied automatically in the limit + xc
We have assumed the scaling of the order parameters in the limit 8 + 00 as Fq. 29 In this limit, the signal-to-noise ratio of z riw + n with (N, g(N) is m² /q m²/X which is finite, whereas the Signal-to-ooise ratio of in the likelihood model x e 1= is q = f + oo, implying that the likelihood model is asymptotically noiscless. It can be understood as corresponding to the deterministic nature of the minimum-loss estimator v = arg min H(w DM;b, One furthermore has wi||w||?=N
Q + 24 + n² = (Q + q) + X m =Q²
(63)
m Q'
m
+ m
(64)
Q + 9
1
1
q= Q +
9
= Q
(65)
30
These reproduce Eqs. (34a) (3cc of the EOS.
C
Computational details for sec. 3.4
Here we show the detailed algebras necessary for sec. 3.4 For notational convenience, we write the derivative
of quantity A w.r.t. 4 as A = dA, and the order parameter vectors f2 (m, x) and f (m, x) det
27

---

Published in Transactions on Machine Learning Research (4/2025)
Taking the derivatives w.r.t. 8. of the EOS (34. we can derive a of linear equations of $2, S.. For example, the derivative of Eq. (34a] yields
Q m +
2Q X-
(66)
Similarly, taking the derivative of Eqs. transforming 52 to 52
(4bb and 34c and rewriting Q by using Eq. (66), we have a relation
Ω = Tá,
(67a)
π 1
X m
m
(67b)
Q3
which corresponds to Eq. (38a). In the same way, after lengthy algebras, the derivatives of Eqs. 34d and 34e lead to another relation transforming Ω to Ω:
i = c TTΩ
(68)
which corresponds to Eq. 38b where
9u g(hy); dg(h)
(69a)
d, dh
I1=by
(69b)
Dg 1 OXSyPy:
(69c)
Su Ey=di yT u Dz
Dv Hu Thy
c= a
(69d)
Ey VT S Dz
T11
T12
T
(69e)
T21
T22
D= gg
Tn =
Σ
Ty8y
(69r)
Dy
1=#1
Σ sy /
Dz Iu
-²
(69g)
Dy
#=
Iugl
T21 = 202a
Dz
(= 2T12).
(69h)
Dy
w=#l
/
93
T22 2020
Tyo2s s
Dz
(69i)
D
Note that the expressions Eq. 69 assume that the loss twice differentiable on R. For losses having
singularities such as the zero-one loss, those expressions should be interpreted as appropriate limits those for their smoothed versions, as discussed in sec. 3.2 The smoothness-controllin parameter (y in Eq. (36
be arbitrary because the following discussion holds irrespectively of its value.
28

---

Published in Transactions on Machine Learning Research (4/2025)
Using the identity Dy/D, = (-1+ b/))
/(o2x8y), we rewrite Eq. (69) as
T11 a
-1+ Σ Ty /
D=
Dy Dy)
1
(70)
d²x
y=1l
Σ Tygy / Dzgy+
T1 X
+
E
Tygy
Dz Iu Dy
(71)
y=+l
y=+1
T21 2T12:
(72)
T22
20²a
(-
Σ Tu Ero
Dzgg + Σ T
Dz 9 Dy
(73)
x
M=#1
Putting Eqs.
67a and
68
together, we obtain the equation to 22:
Ω TTc + ATΩ =: b + A52.
(74)
If we assume the extremization condition rh = 92 = 0, the following relation can be derived:
b1
b2 x).
(75)
A12
A22
Our discussion completes by showing that this condition is satisfied if 8+ = = 1/2 =. In the situation s with = O, the symmetry 9+ =g== D+ = D_ =: D holds. This simplifies many terms:
m- + as Dzg.
(76a)
- d'as / {D29² D=g"
(76b)
9 + a(2r+ - -1) / D2 9
(76c)
D'
22 + 20 assass22 (2r+ - 1) / D2 D'
g²
(76d)
Tn a
-1+ / Dz -1/
(76e)
xx
DDs''s + / Dz 2x
T12 X
(76f)
Th1 = 27'12
(76g)
20
[-]DE(0/P+ fD= Dz (g)) )
T22 +
(76h)
X
Using these, we obtain
6 = Tua + T1202 + o²a2s2 22 1) { Dzgz /
D2 9 - - D2g
Dz g²
(77)
Q3
D
DJ
69 T2ic + T2202 + as(2r+ 1)
{ Dzg
Dz D2 o +02 { Dz Dz
g²1
(78)
Q3
DJ
2zz
{ | /
/ |D2
D2g /
92
A TiiTi2 + Ti2T22 +
Dz D
Dz D
(79)
Q2
as?
{(
[ Dzg /
Dz 9 +02 / D2
A2 Ti2 T22 +
(80)
Q
In the transformations above, we have used the EOS relation
x = =Q-1 and
/ Dzg²
Q² + ase +a / Dzg²+ Dagz
+ = Dzg² +o'a²
(81)
29

---

Published in Transactions on Machine Learning Research (4/2025)
Then, we finally arrive at
0i
2r+
b2
(82)
A12
Qs
1
A22
showing that Eq. 75 holds at 8+ = 3- =. This supports the empirical observation that the maximum of m is achicved when no haamplinngrowcighhinn is applied.
Acknowledgments
This work was partially supported by JST, CREST Grant number JPMJCF1862, Japan (TO, TT), JSPS
KAKENHI under Grant 22K12179 (TO), Grant-in-Aid for Transformative Research Areas (A), "Foundation of machine learning physics" (22H05117) (TO), and Grant-in-Aid for Transformative Research Areas (A), "Shin- bioloy regulated by protein lifetime" (24H01895) (TT).
References
David H. Ackley, GeolTrey E. Hinton, and Terrence J. Sejnowski. A learning algorithm for boltzmann machines. Cognitive Science, 9(1):147-169 1985. doi: http:://oaaooro//00./2/////55150
URL httpp:://onltneelrraaa catpey..wwlyy wiley. htwpeaarwwwleey..o//o//////a// http:///..aaaa.....22////6616669
Urte Adomaityle, Gabriele Sicuro, and Pierpaolo Vivo. Classification of heavy-tailed features in high dimen- sions: a superstatistical approach. Advances in Neural Information Processing Systems, 36, 2024.
Jinho Baik, Gérard Ben Arous, and Sandrine Péché. Phase transition of the largest eigenvalue for nonnull complex sample covariance matrices. The Annals of Probabiliiy, 33(5111643 1697, 2005. doi: 10.1214/ 0091119055000000233. URL https::/doo
N Barkai and Haim Sompolinsky. Statistical mechanics of the maximum-likllhhod density estimation. Physical Review E, 50(3):1766 1769, 1994.
N Barkai, HS Seung, and Haim Sompolinsky. Scaling laws in learning of classification tasks, Physical Review Letters, 30120):3167-3170, 1993.
M Biehl and Mietzner. Statistical mechanics of unsupervised learning. EPL (Europhysies Letters), 24(5): 421, 1993.
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with label-distributtoon-waaa margin loss. Advances in Neural Information Processing Systems, 32, 2019.
Niladri S. Chatterji and Philip M. Long. Finite-sample analysis of interpolating linear classifiers in the overparameterized regime. Journal of Machine Learning Research, 22:1-30, 2021.
Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer. SMOTE: synthetic minority over-sampligg technique. Journal of Artificial Intelligence Rresearch, 16:321-357, 2002.
Yin Cui, Yang Song, Chen Sun, Andrew Howard, and Serge Belongie. Large scale fine-grained categorization and domain-specific transfer learning. In Proceedings of the TEEE Conference Compuler Vision and Pattern Recognition (CVPR), pp. 4109-4118, 2018.
Yin Cui, Menglin Jia, Tsung-Yi Lin, Yang Song, and Serge Belongie. Class-balanced loss based on effec- tive number of samples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9268-9277 9277, 2019.
Viktor Dotsenko. Introduction to the Replica Theory of Disordered Statistical Systems, volume 4 of Collection Alea-Saclay; Monographs and Texts in Statistical Physics. Cambridge University Press, 2005.
Cong Fang, Hangfeng He, Qi Long, and Weijie J. Su. Exploring deep neural networks via layer- aayer-peeled model: Minority collapse in imbalanced training. Proceedings of the National Academy of Sciences, 118 2103:1110 2021. doi: (0.1073/pnas.2 2103091111 URL BtRLhhtts:://nn //wt:s:://www. wwwppaan.....p.p..ppppppppppp... wrtg/////ab//1..1000 pnas 2103091118
30

---

Published in Transactions on Machine Learning Research (4/2025)
Emanuele Francazi, Marco Baity-Jesi, and Aurelien Lucchi. A theoretical analysis of the learning dynamics under class imbalance. In Proceedings of the 40th International Conference on Machine Learning, ICMT''2. Jrg,.org, 2023.
Elizabeth Gardner.
The space of interactions in neural network models. Journal of Physics A: Mathematical and General, (1)):257-,77, 1988.
Elizabeth Gardner and Bernard Derrida. Optimal storage properties of neural network models. Journal of Physics A: Mathemalical and General, 21(1) (1111:271228) 1988.
Haibo He and Edwardo A Garcia. Learning from imbalanced data. IEFE Transactions on Knowledge and Data Engineering, 21(9):1263 1284, 2009.
Hopfield. Neural networks and physical systems with emergent collective computational abilities. Pro- ceedings of the National Academy of Sciences, 79(5)22554 2558, 1982. doi: 10.1073/pnas.79.8 wtt.pa................. 2554. URL https://wwiz pnas. orgooog//oo.. 10733//aaa 79 79.8.22... 2554
Chen Huang, Yining Li, Chen Change Loy, and Xiaoou Tang. Learning deep representation for imbal- anced classification. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5375-5384, 2016.
iNaturalist. iNaturalist 2018 competition dataset: https://githnb.com wwt..oo//vviiiiiiiiaaaoeeee catnattooom/tt/////aaaa www.tr//w/////e//e/eeeeeeeeeee/e /2018, 2018.
Muhammad Abdullah Jamal, Matthew Brown, Ming-Hsuan Yang, Liqiang Wang, and Boqing Gong. Re- thinking blass-balanced methods for long-tailed visual recognition from a domain adaptation perspective. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7610-7619, 2020.
Longlong Jing and Yingli Tian. Self-supervised visual feature learning with deep neural networks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, (3(11):4037-40558 2020.
Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalan- tidis. Decoupling representation and classifier for long-tailed recognition. In International Conference on Learning Representations, 2020. URL https::/openreviieet http:///w......ooom/oaaaaaao////
Salman Khan, Munawar Hayat, Syed Waqas Zamir, Jianbing Shen, and Ling Shao. Striking the right balance with uncertainty. In Proceedings of the TEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 103 112, 2019. doi: 10.1109/ N0.1109/CVPR.220 /CTCC/VV...........00
Ganesh Ramachandra Kini, Orestis Paraskevas, Samet Oymak, and Christos Thrampoulidis. Label- imbalanced and group-sensitiv classification under overparameterization. Advances in Neural Information Processing Systems, 34, 2021.
Thibault Lesieur, Florent Krzakala, and Lenka Lenka Zdeborová. Constrained low-rank matrix estimation: phase transitions, approximate message passing and applications. Journal of Statistical Mechanics: Theory and Experiment, 0077(33, jul 2017. doi: 10.1088 10.1088/1742-5540 1172--4111288// aa724. URL https://ddx doi. org/ 10.
aa7284
Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella Yu. Large-scale long- tailed recognition in open world. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognition (CVPR), pp. 2537 2546, 2019.
Emanuele Loffredo, Mauro Pastore, Simona Cocco, and Remi Monasson. Restoring balance: principled un- der der/oversampling of data for optimal classification. In Ruslan Salakhutdinov,, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (cds.), Proceedings of the 41st Inter- national Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 32643 326700 PMLR, 21-27 Jul 2024. URT https::t//pptt httpps:/proceeeingg. mlr. http:/aeeiiaaa......eeee//////// (rress/v225/ooffred024 html
31

---

Published in Transactions on Machine Learning Research (4/2025)
E Lootens and Christian van den Broeck. Analysing cluster formation by replica method. EPL (Europhysics Letters), 30(7):881, 1995.
Bruno Loureiro, Gabriele Sieuro, Cédric Gerbelot, Alessandro Pacco, Florent Krzakala, and Tenka Zde- borová. Learning gaussian mixtures with generalized linear models: Precise asymptotics in high- dimensions. Advances Newral Information Processing Systems, 34:10144 10157, 2021.
Stefano Sarao Mannelli, Federica Gerace, Negar Rostamzadeh, and Luea Saglietti. Bias-inducing geometries: exactly solvable data model with fairness implications. In 1CML 2024 Workshop on Geometry-ground Representation Learning and Generative Modeling, 2024.
Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and San- jiv Kumar. Long-tail learning via logit adjustment. 2021. URL http::://penrrvvien.ett///eeeeee
37nvvg
Mare Mezard and Andrea Montanari. Information, Physics, and Computation. Oxford University Press, 2009.
Francesca Mignacco, Florent Krzakala, Yue Lu, Pierfrancesco Urbani, and Lenka Zdeborova. The role of regularization in classification of high-dimensional noisy gaussian mixture. In International conference on machine learning, pp. 1874-6885 PMLR, 2020.
Andrea Montanari and David Tse. Analysis of belief propagation for non-linear problems: The example of CDMA (or: How to prove Tanaka's formula). In 2006 IEEE Information Theory Workshop-ITW'00 Punta del Este, pp. 160-164. IEEE, 2006.
Hidetoshi Nishimori. Statislical Physiic of Spin Glasses and Informalion Processing: An Introduction. volume 111. Clarendon Press, 2001.
Vardan Papyan, X. Y. Han, and David L. Donoho. Prevalence neural collapse during the terminal phase deep learning training. Proceedings of the National Academy of Sciences, 117(40):24652 24663, 2020. doi: 10.1073 1113//pas.221 ppass2015509117. URL htttps////w...passrr///o/////// hrtpaas.rrr////o///////// 107./////................... 2015559117
H. S. Seung, H. Sompolinsky, and N. Tishby. Statistical mechanics of learning from examples. Phys. Rev. A, 14:0556-6091 Apr 1992. doi: 10.1103/PhysRev/M A...//RReeeA..6665... URL https:://ink.... aps wtt.:/ik.aaps..oroo/////./. 0rg/3o1/10.11033/ PhysRevA. 60066
Takashi Takahashi. The role of pseudo-labels in self-training linear classifiers on high-dimensional gaussian mixture data. arXio preprinl, arXiv:2205.07730 07.07999,,,,, 2022. URL https://arxiv.. http://awww..rrraaa///...//.///.
Version updated in 2024. Accessed on December 22, 2024.
Takashi Takahashi. A replica analysis of undernbagging. Transactions Machine Learning Research, Published: July 2024, 2024. URL
Reviewed on Open-
Review.
Michel Talagrand. Spin Glasses: A Challenge for Muthematicians: Cavily and Mean Field Models, volume 46 A Series of Modern Surveys in Mathematics. Springer Science & Business Media, 2003.
Michel Talagrand. Mean Field Models for Spin Glasses, Volume 1: Basic Examples, volume 54 of A Series of Modern Surveys in Mathematics. Springer Science & Business Media, 2011a.
Michel Talagrand. Mean Field Models for Syin Glasses, Volume H: Advanced Replica-Symmelry and Low Temperature, volume 55 of A Series of Modern Surveys in Mathematics. Springer Science & Business Media, 2011b.
Jingru Tan, Changbao Wang, Buyu Li, Quanquan Li, Wanli Ouyang, Changqing Yin, and Junjic Yan. Equalization loss for long- tailed object recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11662 11662-11671, 2020.
32

---

Published in Transactions on Machine Learning Research (4/2025)
Toshiyuki Tanaka. Statistical-mochanies analysis of Gaussian labeled-mmlableed classification problems. In
Journal of Physics: Conference Series, volume 473, pp. 012001. TOP Publishing, 2013.
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The iNaturalist species classification and detection dataset. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition CVPR), pp. 8769-8778, 2018.
Byron C. Wallace, Kevin Small, Carla E. Brodley, and Thomas A. Trikalinos. Class imbalance, redux. In 2011 IEEE 11th International Conference on Data Mining, pp. 754-763, 2011. doi: 10.1 101109IIIDM.2211....
Yu-Xiong Wang, Deva Ramanan, and Martial Hebert. Learning to model the tail. Advances in Newral
Information Processing Systems, 30, 2017.
TLH Watkin and J-P Nadal. Optimal unsupervised learning. Journal of Physics A: Mathematical and General, 27(6))18999, 1994.
Yuzhe Yang and Zhi Xu. Rethinking the value of labels for improving class-imbalancedd learning. Advances in Neural Information Processing Systems, 33:19290 19301, 2020.
33


Reviews:
[{"review_id": "PnAFQbmDjI", "reviewer": "Reviewer_9A8H", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The strengths of the previous submission carry over to this one. For brevity, I will only list the improvements.\n\n\n**Improvements:**\n\n1. Similar to my observation on their previous manuscript, I would say that the paper is extremely thorough. The marked improvement in this version of the paper is that the authors have made efforts to enhance readability without sacrificing thoroughness.  The most noticeable of these are both in the Theoretical Analysis section - (a) the shifting of the proof for the main result of the paper to the appendix, and (b) the inclusion of section 3.5 to explain how the authors' findings apply to the deep learning papers they reference at the beginning of the manuscript. The inclusion of a short explanation on why the overlap parameter can be used to quantify the quality of feature learning is an important addition to make the paper more accessible to those from a non-statistical mechanics background. \n2. In response to a number of concerns raised by reviewers of their previous manuscript, the authors have included a detailed related work section (Section 2.4). Although I am not intimately familiar with related work in this area, I see that it contains a good blend of older and more recent work. The works by Baik et al. (2005), Lesieur et al. (2017), Loureiro et al. (2021), Francazi et al. (2023), and Loffredo et al. (2024) were all requested by a  reviewer in the previous version of the authors' manuscript which have all been included here.\n\n\n**Weaknesses:**\n\nThe weaknesses I listed for the previous version of the manuscript have all been sufficiently addressed by the authors, so I believe I do not have anything major to list here.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Since the paper is essentially a re-submission of a previous manuscript (https://openreview.net/forum?id=0mULkj0Lyp), the original summary of contributions hasn't changed. With this in mind, I will re-use the text from my previous review and include it here:\n\nThe motivation of this paper is to develop a toy binary classification model that attempts to explain an interesting observation discussed in papers such as *Cao et al. (2019)*; *Kang et al. (2019)*, etc. The observation in question is that the best features are learned *without* any kind of re-sampling/re-weighting in unbalanced binary classification problems, which is both surprising and counter-intuitive. To explain this odd phenomenon, the authors propose a toy binary classification model in which they theoretically prove that the no re-sampling/no re-weighting condition yields the best features subject to a set of assumptions.\nThe assumptions are as follows:\n1. The positive and negative class-centers are located at $\\frac{+w_0}{\\sqrt(N)}$ and $\\frac{-w_0}{\\sqrt(N)}$ respectively ($N$=number of dimensions)\n2. The number of dimensions $N \\rightarrow ∞$ and the ratio $\\frac{M}{N}$ is finite ($M$=number of samples)\n3. The loss must be symmetric i.e. $l(+h,+y)=l(-h,-y)$ where $h$ is the hypothesis function of the form $\\frac{w^T \\cdot x}{\\sqrt{N}} + b$\n4. The positive and negative classes have the same variance $\\sigma_{+}^2 = \\sigma_{-}^2$\n5. The bias is set such that the decision boundary is equidistant from both class centers\nThe authors then proceed to lay out their framework in great detail and provide a formal proof of their result using the replica method. In addition, the authors also provide a detailed discussion of various configurations of their model including the one where the result of interest is observed. Finally, the authors run some numerical experiments and show the agreement between their experiments and their theoretical results for $N=400$.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "None from me", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "zOfOyvyzXN", "reviewer": "Reviewer_kqHA", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper has been carefully rewritten in different parts implementing the suggestions of the reviewers. This results in a more readable paper, accompanied by interesting observations. \n\n\nHowever, the paper still contains some weaknesses that have not been completely addressed. The writing and the statement regarding the mathematical assumptions need to be further improved.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "I refer to my detailed review of the first version of the manuscript (previous reviewer code: MA55). I discuss below the revised version.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Most importantly, the authors are not mathematically precise in what are the assumptions needed for the theoretical results to hold. This was highlighted in the previous round of reviews, but it is not thoroughly discussed in the revised version. In the introduction on page 2, the authors say \"wide range of classifiers and losses....\". Where is this discussed in the theoretical analysis? The analysis relies on a Replica Symmetry ansatz, and the precise implications on the learning setting (classifier setup, loss, etc.) need to be explicitly stated. This should also be reflected in the discussion of related works where the authors classify the setting of some references as \"restricted\" to the convex setting.\n\nMoreover, I find the writing still too dispersive. The authors provide an overview of the statistical mechanics formulation that guides the reader towards the computation of the log-partition function $\\log Z$ (page 4). The pages 7 to 9 still contain a long computation without enough guidance for the non-expert reader (e.g., at the most basic level, adding paragraphs separating the usual blocks in a replica computation). As highlighted in the previous round of reviews, the computation in itself is not the most interesting part of the present manuscript (as also the authors discuss in the revised related works section), and the core of the manuscript should be the discussion of the behaviors of the quantity of interest (from section 3.3 onwards). \n\nThe plot management still remains suboptimal. Pages 13-14-15-17-18 are occupied only by a single plot, reinforcing the dispersive nature of the writing as pointed out above. Many legends on the y-axis are not readable neatly due to the presence of the subscript, e.g.$u(m_{\\rm max})$. The captions of the plots are not communicating to the reader; e.g., Figure 6 explains what is the \"crucial observation\" but this is not done in all the figures. I would suggest moving some of the plots to the appendices and keeping only a few selected plots that are thoroughly discussed.", "broader_impact_concerns": "I have no broader impact concerns as the contribution is mostly theoretical.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "lxf6bC0pkX", "reviewer": "Reviewer_hpg6", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "As per previous reviews, the paper's key contribution is a theoretical justification for Kang et al.'s recent observation that resampling may not be necessary for classification tasks with imbalanced classes. The analysis is conducted using a simple \"mean-field\" model, examined through the replica trick. Additionally, all theoretical findings are validated through numerical experiments. The addition of the literature review and Section 3.4 are an improvement on the previous version, although the former is not very clear per se as it introduces quantities that can be clarified by taking a look at the appendix only. \n\nOn the other hand, the paper is only marginally original, as it heavily relies on very well known and exploited replica techniques for classification tasks in the proportional asymptotic regime, with minimal variations with respect to known results, and mostly relies on the numerical exploration of the fixed-point equations for the order parameters. The presentation looks sometimes verbose (some plots are redundant, eg, Fig. 1 and Fig 7, or the obvious symmetry that appears in Fig 2, and the numerical experiment might be presented directly on top of the theoretical predictions in Sec 3.2-3.3, as their separate discussion does not add much to the paper). In the binary classification setting, classes are assumed to be spherically symmetric, a fact that might introduce some important symmetry effects, as it happens for the derivation of the result in Section 3.4, and as stated by the authors in Section 3.6: it looks therefore that the irrelevance of the reweighting is just the accidental consequence of the symmetry of the problem in the specific considered setting. Finally, the multiclass model might appear as a little bit artificial (classes's centroids live on the same straight line).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper examines binary classification in high-dimensional settings, with a particular focus on the impact of resampling. By analyzing a set of fixed-point equations for order parameters—derived using the replica method—the authors investigate how resampling affects classification performance under varying class balances and the role of the bias term in estimation. Furthermore, they demonstrate that, under certain conditions, avoiding resampling may be the optimal approach, providing a theoretical basis for recent findings by Cao et al. (2019) and Kang et al. (2019).", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The paper is a resubmission and has been revised by the authors to address some concerns emerged during the first round of reviews. Other questions rising from the chosen setup (eg, the dependence on the regularization in case of  soft norm constraint on the weights) are left unexplored. A list of possible (minor) modifications is the following:\n- At the end of Section 2.1, it might be worthy to stree that the results are expected to be *qualitatively* the same for a wider class of distributions, not quantitatively, as shown in the added reference by [Adomityte et al. (2023)](https://proceedings.neurips.cc/paper_files/paper/2023/hash/88be023075a5a3ff3dc3b5d26623fa22-Abstract-Conference.html).\n-  The authors should specify what difficult arises when dealing with non-convex loss, as they initialy mention ```while the replica method can handle even non-convex loss functions and in fact we discuss general loss functions that may not necessarily be convex``` whereas it seems to me that the solution of Eq. 32 might become a delicate thing to deal with if $\\ell$ is nonconvex.", "broader_impact_concerns": "I have no broader impact concern about the submission.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper is a resubmission. In the previous iterations, all reviewers agreed that the content is interesting, but the manuscript required substantial revision, mostly for what concerned clarity and context. After a major revision and a second round of reviews, the suggestions made were taken into account and all reviewers agreed that the manuscript has improved. Therefore, I am happy to recommend it for acceptance at TMLR.
Decision: Accept (poster)