Neural Collapse in Cumulative Link Models for Ordinal Regression: An Analysis with Unconstrained Feature Model
Authors: Chuang Ma
Tomoyuki Obuchi
Toshiyuki Tanaka
Venue: NeurIPS 2025 poster ()
Keywords: neural collapse
unconstrained feature model
ordinal regression
cumulative link model
deep learning theory
Primary area: deep_learning

Abstract: A phenomenon known as ``Neural Collapse (NC)'' in deep classification tasks, in which the penultimate-layer features and the final classifiers exhibit an extremely simple geometric structure, has recently attracted considerable attention, with the expectation that it can deepen our understanding of how deep neural networks behave. The Unconstrained Feature Model (UFM) has been proposed to explain NC theoretically, and there emerges a growing body of work that extends NC to tasks other than classification and leverages it for practical applications. In this study, we investigate whether a similar phenomenon arises in deep Ordinal Regression (OR) tasks, via combining the cumulative link model for OR and UFM. We show that a phenomenon we call Ordinal Neural Collapse (ONC) indeed emerges and is characterized by the following three properties: (ONC1) all optimal features in the same class collapse to their within-class mean when regularization is applied; (ONC2) these class means align with the classifier, meaning that they collapse onto a one-dimensional subspace; (ONC3) the optimal latent variables (corresponding to logits or preactivations in classification tasks) are aligned according to the class order, and in particular, in the zero-regularization limit, a highly local and simple geometric relationship emerges between the latent variables and the threshold values. We prove these properties analytically within the UFM framework with fixed threshold values and corroborate them empirically across a variety of datasets. We also discuss how these insights can be leveraged in OR, highlighting the use of fixed thresholds.

Neural Collapse in Cumulative Link Models for Ordinal Regression: An Analysis with Unconstrained Feature Model
Chuang Ma²
Tomoyuki Obuchi
Toshiyuki Tanaka'
'Kyoto University, "RIKEN AIP,
ctaacchang.5555 52h@st, obuchi@i, (ooi).kyotouu.cc.
Abstract
A phenomenon known "Neural Collapse (NC)" in deep classification tasks, in which the penultimatellayer features and the final classifiers exhibit an extremely simple geometric structure, has recently attracted considerable attention, with the expectation that it can deepen our understanding of how deep neural networks behave. The Unconstrained Feature Model (UFM) has been proposed to explain NC theoretically, and there emerges a growing body of work that extends NC to tasks other than classification and leverages it for practical applications. In this study, we investigate whether a similar phenomenon arises in deep Ordinal Regression (OR) tasks, via combining the cumulative link model for OR and UFM. We show that phenomenon we call Ordinal Neural Collapse (ONC) indeed emerges and is characterized by the following three properties: (ONCI) all optimal features in the same class collapse to their within-class mean when regularization is applied; (ONC2) these class means align with the classifier, meaning that they collapse onto a one- mie-dimensional subspace; (ONC3) the optimal latent variables (corresponding logits or preactivations classification tasks) are aligned according to the class order, and in particular, in the zero- rerroaegullrizationn limit, a highly local and simple geometric relationship emerges between the latent variables and the threshold values. We prove these properties analytically within the UFM framework with fixed threshold values and corroborate them empirically across variety of datasets. We also discuss how these insights can be leveraged in OR, highlighting the use of fixed thresholds.
Introduction
In classification tasks on balanced datasets, it has been observed that, after sufficient training, the outputs (or features) of the penultimate layer and the final classifier weights in sufficiently expressive Deep Neural Networks (DNNs) exhibit remarkably simple symmetric structure. Papyaa al. [2020] conducted thorough experiments across architectures and datasets to corroborate this phenomenon, and identified its four intertwined signatures, which are summarized as Neural Collapse (NC): (NC1) all feature vectors from the same class converge exactly onto their shared class mean, extinguishing within-class variance; (NC2) once these class means recentered at the global mean, they occupy the vertices of a maximally symmetric Simplex Equiangular Tight Frame (Simplex ETF); (NC3) each classifier weight vector becomes parallel to its corresponding class mean vector, reflecting self-dual alignment; (NC4) the network simply classifies by nearest class mean. NC, being considered to offer valuable clue toward understanding DNNs, has inspired number of theoretical studies (Mixon) et al. 2022, Fang et al. 2021 Zhu et al. 2021]] E and Wojtowytsch, 2020 Lu and Steinerberger 2022 which introduced Unconstrained Feature Model (UFM) being a central theoretical tool in
this context.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

UFM has allowed extending NC to broader problem settings and facilitated the analytical investigation
of its properties (Zhou al., 2022 Thrampoulidis al., [022] Dang et al. 2023 2024, et al.)
2024a]. For example,/Andre Andriopoulos et al. [024] applied this framework to multivariate regression, finding phenomenon called Neural Regression Collapse (NRC) in which features collapse to a target subspace and weight vectors align accordingly. The proliferation of these extensions suggests
that NC a universal phenomenon in DNNs.
Beyond classification and regression, there is task called Ordinal Regression (OR) which aims to solve prediction tasks where labels are discrete categories with a natural order. Unlike classification, not all misclassifications are equally wrong in OR tasks; unlike regression, label values in OR do not bear quantitative information and only their ordering relationship is essential. A common approach to OR problems is to use threshold models [Verwaeren et al. 2012] which assume that an unobserved continuous latent variable generates the observed ordinal response: the map to latent space is traditionally assumed to be linear with respect to (w....) input datapoints [Herbrich et al. 2000]. Even within this framework, more challenging scenarios can be addressed by applying an appropriate transformation (feature extractor) the input datapoints, and the effectiveness of DNNs as feature extractors has been reported (Doado-Morent et al. 2012 Vargas et al. 2020]. The latent variable in the threshold models occupies the same role as the logit in classification networks, prompting the question of whether phenomena similar to NC may also emerge in the feature space of OR.
To address this question, in this paper we explore phenomena analogous to NC within the context of OR. To that end, we adopt the Cumulative Link Model (CLM) [Agresti] 2010] classical type of threshold models- and analyze it in combination with UFM. As a result, we find that such a phenomenon indeed occurs when the l regularization is applied, and we name it Ordinal Neural Collapse (ONC). ONC is characterized by the following three properties:
(ONC1) Within-class Mean Collapse: all optimal features in the same class collapse to their within-class mean.
(ONC2) Collapse to One- tne-Dimensional Subspace: these class means align with the classifier, meaning that they collapse onto a one-dimensional subspace.
(ONC3) Collapse to Ordinal Structure: the optimal latent variables are aligned according the class order, and in particular, in the sero-regularization limit, a highly local and simple relationship emerges between the latent variables and the threshold values.
We note that this result is obtained under the assumption that all the threshold values of CLM are fixed. Although it is not necessarily a standard assumption in recent studies, we argue that it is necessary for the emergence of ONC and, morcover, can be practically meaningful assumption. A discussion about this point will be given later in Section 6
We also validated ONC through experiments using five imbalanced ordinal datasets and DNN architecture. The result provides clear empirical evidence of ONC under fixed threshold values. Furthermore, our experiments with learnable thresholds still exhibit ONC, implying its robustness.
2 Related work
We cite here only recent results that are particularly relevant to the present work.
UFM and related issues. UFM and the related models were proposed almost concurrently in number of pieces of work [Mixon et al. 2022 Fang et al., 2021 Zhu et al. 2002 E and Wojtowytsch 2020 Lu and Steinerberger] [002]]. The core idea of UFM lies in decoupling the features from the data by treating the output of a specific layer, typically the penultimate layer, as free learnable variables, while explicitly modeling the nonlinear functions and weight vectors in the subsequent layers. This simplifying assumption enables us to analytically derive nontrivial results. Even in such a simplified model, the analysis can remain nontrivial. For example, UFM generally admits multiple local minima in its loss landscape. Among them, it was shown by Zhu et al. (202T) that only the global minimum exhibits the NC structure in the case of balanced classification. Departing from the typical analysis of the single-layer UFM, the first investigation of UFM with multiple layers was carried out by Tirer and Bruna [2022], finding that NC still emerges as the unique global optimum. Furthermore, to understand the phenomenon called Deep NC, in which NC propagates not only to the

---

final layer but also to intermediate layers Hui et al. [022, He and Su 2023 Rangamani et al. 2023
Parker al., 2023 Masarezyk al. 2023 Sūkenik et al. [003] extended UFM to multiple nonlinear layers and proved that, in the case of binary classification, Deep NC emerges as the unique global optimum. Their follow-up study [Sákeník et al. 2024] demonstrated that, in multi-layer architectures, increasing the number of classes causes Deep NC to cease being the optimal solution.
NC extensions under varied settings on classification. In classification, the NC framework has been adapted to different settings from the balanced case. Zhhou et al. 2022) showed that cross-entropy,
focal loss, label smoothing, and even mean squared error lead to the same NC geometry using UFM.
Thrampoulidis et al. [2022] and Hong and Ling (2024] found that with cross-entrope loss and & regularization, class imbalance does not prevent NCT but the global geometry generalizes from
Simplex ETF to more general structure. In extremely imbalanced cases, Fang et al. [202]] found a
"minority collapse" phenomenon where minority-class features collapse onto a single point. Dang et al. [2023] proved in deep linear UFM that every global minimizer forms orthogonal directions whose magnitudes scale proportionally to class sample sizes, and et al. [2024] proved essentially
the same statement for rectified linear unit (ReLU) UFM.
NC extensions beyond classification. Li al. 2024a extended NC to multi-label classification by showing that multi-label embeddings lie in the linear span of label-means. Andriopoulos et al. [2024] generalized NC to multivariate regression to find NRC. Wu and Papyan (2024) introduced the concept of "linguistic collapse" in large-scale language models, showing that token embeddings tend toward an approximately uniform distribution on the hypersphere as model scale increases. Furthermore, Nguyen et al. [2024) demonstrated that NC-style embedding collapse also occurs in diffusion models. NC-like phenomena have also been observed in self-supervised learning [Ben-Shau et al., [203) and in transfer learning [Galanti et al.] 2022 [i et al. [2024b]: the latter studies discussed the re relationship between the degree of NC and transfer performance, and also proposed some strategy for leveraging NC insights to improve generalization performance.
3 Formulation
OR. An OR task is formulated as follows. Let X' be an input space and y = {1, 1..,......) Q} be an ordered label set with ordering
< Q. Given a training set D (Z.,. y/ )= with (æ;, y:) E X x ) our goal is to learn an order-respecting mapping r X' - Y. For each label g, we
let D,, ( v.)= D Vi o} and n, its size, so that Σ ng N holds.
CLM. To express the ordinal structure, threshold models introduce a latent variable 2 E IR and also strictly ordered set of "thresholds" b = (bo, .., bq) which partitions the z-axis. One typically assumes (bo, bq)) (-o0, oo) to partition R properly, and thus each interval is uniquely associated with one category via the decision rule y = q + z E (bg-1.bg).
In CLMs, the probability of a specific category is expressed through a cumulative probability P(y S q 2) conditioned on the latent variable 2, which is modeled by using strictly monotone inverse link function IR + (0, 1) as
(bbb-))) z).
(I)
There are several typical choices for g. including the logistic function g(x) = (1+e-=)-1 the normal cumulative distribution function (CDF) g(x)) = (1x) = f e 1=² dz/ /2T, and the Gumbel CDF g(x) = - e -e"" which correspond to the logit, probit, and clog-log models, respectively.
An input datapoint x is transformed to a value 2 in the latent space through certain map. When using a feature extractor such as DNNs |Vargas et al. 2020), the map expressed as
w ho()),
where w E R" and hg : X + R are the classifier weight vector and feature extractor, respectively. Here, 0 represents the parameters of the feature extractor.
Under the model @T. (2), since the probability that y belongs to class a given expressed as P(y q 22) = P 2) P(y I q- 1 2). the empirical negative log-likelihood given the
3

---

dataset (Dylg=) becomes
(NLL(W,0,) 0, =
1
Σ
Σ
L(zi, b bg),
4= fw.0()))
(3)
N
4=1 (D
where let (zz, a, b) == llolggb =) g(a 2)]. consider the & regularization on the parameters:
As in recent practices using DNNs,
R(w, 0) A wll2
Ae 1012
(4)
The overall optimization problem is therefore
min ( (NU 0, b) R( 0))-
w
UFM for CLM-based OR. In the single-layer UFM, the feature vector ha(a;) itself is treated as a free learnable variable. As result, for each datapoint æ;, a free variable h; is associated. For notational simplicity, we relabel this variable as fy.i where indexes the class and 1,..., "y indexes the datapoints within Dg. The regularization on the parameter is assumed to converted to that on H = (hgg)),i- UFM thus allows to convert ( into
min (CNLL,UFM(()) H, b) + RUFM H)), wz.l
(6)
where
GNLL,UFM(W,H,)) H,
L(w hggs b bg)
(7)
q=1
Au
A
RUFM(W, H)
hy
(8)
2
2.V
4 Theoretical results based on UFM analysis
Let us state our main theoretical results. Thanks to structure of our CLM and UFM, (6) can be decomposed into multi-stage optimization as follows:
min fo(w, hn.s) } }.
min 10
2 w
+
(9)
N
Aga
q=1
where
fy(w, h) L(wa T h, bh. + An hhl 2
(10)
and where w 2 and a are the norm of w and the unit vector representing the direction of w. and ||]]| = hold. Since our objective function to be minimized in
respectively, so that (6) is invariant under any orthogonal transformation w - Ow, + Oh, AO O(p), we can fix the direction of w without loss of generality. Furthermore, we assume that the derivative g' of the inverse link function is logarithmically concave (log-concall concave:::: some standard choices of g in OR such the logistic function, the standard normal CDF, and the Gumbel CDF satisfy this assumption.
Under these assumptions, the conditions of ONC can derived. Before presenting the concrete statements, we first show the following theorem.
Theorem 4.1. Let p(x) log-concave function on R2 and P(x) = f p(u) du. Then, for any a < b, the function p(x:) = P(b - x)) P(a x.) is log-concave.
Proof. One write p(z:) as
p(x) = P(b - x) - P(a x)) / b-z
p(u) du /' p(y + x) dy,
(11)
where we let y u æ. Since p④y + x) log-concave in TR. one can apply Theorem A.1 with A |a, b] conclude that p(x: is log-concave in

---

This means that the log-concavity of g' leads to the convexity of L(2, b) = loglo(b-2))g(( w.r.t. for any (a.b) satisfying > One can further show that, if g' strictly log-concave, then M(2,,, ((2,,,) b) is strictly convex in 2: see Appendix A.2
We now ready state the ONC theorem.
Theorem 4.2 (ONC). Assume that the inverse link function g(x) defined on R. is differentiable, and that its derivative g' is log-concaves Consider (6) with thresholds b (bo, bi.... (Q) satisfying bq. and let ,w",,", denote the global minimizer. Under the assumption
bo < bi < Aw; Xo > O, the following three properties hold:
(ONC1) For any class q V, the optimal features (hg.)s ts in class q become identical:
higs h Vi Wi=1,- 1,...,0pp nq.
In other words, the optimal features collapse their within-class mean hig.
(ONC2) For any class q, the class mean h becomes parallel to w", meaning that all class means collapse onto the oneddimensional subspace spanned by w².
(ONC3) The optimal latent variables zn = ( satisfy zं S 22 S ..... S <z. Moreover, if g' is strictly log-concave and if ww + O, then these inequalities hold strictly.
Proof. By technical reason, separately treat the two cases w² 0 and w" + +0, and here provide only the derivation of ONCI and 2, deferring the proof of ONC3 to Appendix B
Thanks to the structure of (9), for any fixed w every hy i can be optimized separately from the other variables, and the objective function is identical for all i E (1, (1,...,ny)
.,y)- Its explicit form is
An 11
SaicaTh,b,, arg min (w haga) arg min L(waa h, b1, by) Bug.
(12)
2
First suppose + O. Since L(z, b1 by) is proven to be convex in through Theorem L(wa h. 1 byy (.g(wa) h) is also convex in h. Since the term (Aa/2)|[A|β is strictly convex, the total objective function to be minimized is strictly convex w.r.t. h. On the other hand, let Vy be the gradient of (ww h) at h 0. Thanks to the convexity of L,(wa h). one has
(wga ThaThh) T.g (0) 2 o h, Vh,
(13)
which implies that the objective function is bounded from below:
(wa h) + An | (0) + An h +
0g An
1 2Ah
1l 12 (g)
|
-00
2
2An
(14)
Hence, the strict convexity and the boundedness of L.g h)) + A || guarantee the uniqueness of the minimizer, proving ONC1. The proof of ONC2 is more straightforward. Let hy denote the projection of h on a and h - hhj- Then we have
2
(wa T h) + An |1A112 = (wa T hn) +
An 1 112
An
+
h 113.
(15)
2
2
Hence, the minimization of this w.r.t. h1 yields h- = O, showing ONC2.
Next assume 0. In this case, the dependence of the objective function h only appears in the regularization term and the optimization thus yields h = 0 for all g, i. Hence, the ONC properties appear trivially.
In contrast to ONC1 and 2, which only require the convexity of L(z, a, b), ONC3 has more quantitative information about the problem. Actually, the values of w* 2 are determined from set of equations deduced from the stationarity condition of (9) Borrowing the terminology from statistical physics, we call this set equations Equations Of State (EOS). Analyzing EOS leads to a derivation of ONC3, but it is involved and is deferred to Appendix B The solution of EOS exhibits some singularity at certain parameter values, and also some simple behaviors certain limits. The next theorem summarizes these findings.
5

---

Theorem 4.3 (EOS, phase transition, and some limiting behaviors), Consider the same situation as in Theorem 4.2 If the optimal norm value satisfies w" > 0, w* and the optimal latent variables
obey the following set of equations which we call EOS:
g' (b, 29) g (b 1 -21) 2nj
3q
+ Ah
=0, q= Q
(16a)
g(b, 2gg g(b - 29g)
(w+)²
An
Aw :r
ag (2)))) =0
(16b)
(w*)
where Og = /NgNN. Additionally assumiig the continuity and monotonicity of u w.E.t An and A this EOS implies a phase transition with the phase boundary in the (Ans Xw) Jpllne characterized by
Q Any = aq
(g((b) -g'(b-- -g((b--))
(17)
g(bb) g(bg-1)
Namely, for An 2 C. the trivial solution w" 0, x² = -0 becomes the optimal solution to
9
while for >n C the nontrivial solution w² > 0, x" + O. which obey EOS, becomes the optimal one.
Moreover, this EOS admits a simple behavior emerging in the limit where the product An ap- proaches zero. In that limit, z" is determined by
g (bg xg)== = d'(by-1 - 20), q= 1, -=....,, Q, and one has w² = ((Ann/2m.))/))))
(18)
Proof. Applying ONC2, we have 2y = wa hg. Thus we may rewrite the squared norm |1h 12 as =g/"²² and the optimization w.r.t. w and (hg)), in can be reduced to those w.r.t. w, z. Taking the stationarity condition w.r.t. and wt lead to EOS.
Next, we examine the phase transition and the phase boundary. One subtlety in analyzing the nature of the phase transition is that the trivial solution (w", =*) (0, 0) does not satisfy EOS within the whole parameter region where it is optimal. However, thanks to the assumed continuity and monotonicity of w", exactly on the phase boundary the trivial solution must satisfy EOS. Therefore, we search for a condition under which EOS admits the trivial solution. By substituting (16a) into (Tbb) to eliminate the explicit dependence on w', we obtain the following equation:
g'(b, 27) g (bp 4
2
Ape
(19)
g(h, =x) g(b 2
Inserting the trivial solution 2 into this leads to the phase boundary 17 Thanks to the assumed monotonicity, once the solution becomes the trivial one, it continues to be so above the boundary.
Finally, the limiting behavior is investigated. By inserting (16b) into (6aa) to eliminate the explicit dependence on w", we have
g' (b, =/)) - g (b 2gg) g(bg 2 g(bg-1 2/)
Awds
zq
q 1= Q.
(20)
V
Eg==1 Og (zg))
This yields 18) when An is sent to zero, as long as x" + 0. Then, solving (6b) w.r.t. w", we have the scaling wt = ((An/\))))
The scaling w' O ((An/>w)) (awAn/Aw) 1/44 means that w" in the limit A and/or Aw + 0 may diverge, vanish, or remain finite depending on how one takes the limit.
Equation (8) in the vanishing regularization limit is fairly striking since it provides a simple local relation between b and 2" Especially, if the inverse link function satisfies a symmetry g(z))
g()2), which is the case for the logit and probit models, one has g (x) = g (-x), and (18 thus
implies
b, + by
(21)
2q

---

10
10a
10
10
0.2
0.4
10
10
10
10
Figure 1: Solution behavior of EOS in the logit model for Q = 3 with = (-10, -8, 3, 10) at A = 1. (Left) w" and x" plotted against Au linear scale. A clear phase transition appears at Aw.c = C/AA, (vertical broken line), and the values of z" in the limit A - match well with the theoretical prediction (2 (bg b2-1 (Right) w" is plotted the small-Aw region. A power-law divergence with exponent - 1/44, corresponding to the scaling w = (A) with fixed Xy is clearly observed.
log-log scale in
This simple relation will be verified later in experiments using real-world datasets.
For illustration, we numerically solved (I6) for the logit model and plotted the solution in Fig. [ The analytical prediction about the critical point and the limiting behaviors were certainly reproduced.
Experiments
5.1 Experimental setting
Inverse link functions. Two symmetric inverse link functions, the logistic function and the normal CDE, which correspond to the logit and probit models, respectively, were treated in the experiment.
Datasets and neural networks. We used five tabular OR datasets with the largest number of data points-Eii ERR, LE, SW, CA, and among those publicly available from Gutiérrez al. (2016). Thirty pre-defined training-validation splits with identical label distributions have been officially released, and we used them as-is in this study. Additionally, UTKFace age estimation dataset (Zhang et al., 2017), which contains face images labeled with ages. We grouped ages into classes with five-year intervals. For the tabular datasets, we employed multilayer perceptron with residual connections, while for UTKFace, we used ResNet101 and ResNet50 [He al.) [0166), and DenseNet201 (Huang et al. [0177] as backbones. The weight decay coefficient was set to small values, with the exact choice varying by condition. The motivation for this setting that, in the regul-regularizzation limit, a very simple result emerges as shown (18). which facilitates experimental verification. Further details are provided in Appendix
conducted experiments
the
Treatment of thresholds.
We considered two cases: fixed and learnable thresholds.
For the fixed case, to ensure that the ignored tail probabilities sufficiently small, the edge thresholds by and by were symmetrically fixed (bp = bq) to sufficiently large values. The remaining thresholds were evenly spaced over the interval [bn, bo). Under this setting, solved (5), where 0 denotes the DNN parameters.
In the learnable case, we set by = -o and by +o and learned by with L..,,,Q---- To guarantee the strict ordering between the threshold values, we parameterized them with E RQ-1 as
Q-1 log + + q= 1.. 1
(x) = Σ log(1 + +
Σ
(22)
Q Q-1
Correspondingly, solved min..9 (ENLL(W, 0 b(s)) R(w, 0)) instead of
7

---

Evaluation metrics. We used two basic training metrics for evaluation: LNLL and the mean absolute error (MAE) for label prediction, MAE t Ei=1d y/ where ik is the predicted label. Additionally, we evaluated classification accuracy, quadratic weighted kappa (QWK), within-1 accuracy (proportion of predictions within one class of the true label), and minimum sensitivity (worst-case per-class recall). Besides, let ho(+) denote the penultimate-layer output of our DNN, and let h2 = (1/l)) (( ho(2;) and h (1/N) Ei=1 ha(4;) represent the class-wise and global feature means, respectively. Using these, we introduced the following four quantitative indicators for ONC:
ONC1 = (1/Q) E N, E LamneED, 11
h
(23)
(1/N) > im1 11o (2;)) h2ll
Eg=1 11(h- h) (u (hg- h)) 21l2
w
u
ONC
ONC = -1
(24)
EØ= |hg- hol2
||
EQ-1 -==1
1bg
(2g + 1q+11/2
ONC3
(25)
Σy=1 (bg+1 - bq)
where u is the unit first principal component (h, - - ONC, the indicator for ONCI and becomes zero when ONCI exactly happens; ONC2-1 quantifies whether each class mean collapses onto the dominant one-dimensional subspace represented by while ONC2- measures whether w also collapses onto the same subspace; since our experiments focus on the small-regullrrizttion regime under the use of symmetric g. ONC3 is expected to emerge the form (2T). and accordingly ONC3 serves as an appropriate indicator for it.
5.2 Results
In the main text, the results for the ER and UTKFaace datasets are shown; the logit model is used in both the cases, and ResNet101 among three backbones mentioned above is used for the UTKFaaee case. Experiments conducted under different settings also yielded consistent results and are reported in Appendix D The results obtained using the clog-log model, as well as the quantitative comparison between the theoretical and experimental results-where the latter even exhibits phase transitions-are also reported in the same section.
Figures and plot the evolution of all evaluation metries for ER and UTKFace, respectively.
Fixed Train
Foodd
Learnable Train
Learmable Val
10'
0.3
0.8
0.0
MAC conser)
207
2
30
0.4
0.4
20
2000
2003
<000
Epoch
Epsch
Epech
Epoch
(8temum Senatrurty
32'
0.8
0.8
0.8
357
30-1
30
10/14
107
2006
4000
4202
.000
4000
Epoch
Epech
Eooch
Epoch
Figure 2: Epoch-wise average metries curves for the ER dataset with the logit model.
For the ER dataset, both training and validation MAE approached zero, while accuracy, within-1 1 accuracy, QWK, and minimum sensitivity all approached one, indicating that samples were correctly classified. ONC2.1 and ONC2. rapidly approached zero, showing that the feature vectors collapsed onto the one-dimensional subspace spanned by w. As training proceeded, ONC, decreased steadily, confirming that features collapsed toward their class means. For ONCs we observed clear
8

---

Find
Roed
Loamaale
Learnable
10
92
MAX: conoent
W0thi ACC
6.6
107
0.2
302
322
320
200
302
Esoch
Epoch
Cooch
Epoch
Epoch
Neemmu Senstsity
0.3
107
04
0.2
BA
0.4
10"
0.2
0.
0.2
100
Epoch
200
300
Rpoch
105
Epoch
200
300
100
Epech
200
320
300
spach
200
300
Figure 3: Epoch-wise average metrics curves for the UTKFace dataset with ResNet101 backbone.
difference between the two threshold strategies. With fixed thresholds, ONC3 took small value from an early stage and continued to decrease as training progressed. In contrast, with learnable thresholds, ONC3 seemed converge to a non-zero value. These observations indicate that the simple form of ONC3, given by (T), practically holds under fixed thresholds but does not hold under learnable ones.
For the more complex UTKFace dataset, the ONC metrics exhibit the same trends as observed on the ER dataset. Across metrics including LNLL accuracy, MAE, and minimum sensitivity, fixed thresholds demonstrate better performance. Notably, both training and validation minimum sensitivity for learnable thresholds remain at zero throughout training, indicating that at least one class completely ignored by the model. This phenomenon consistently observed across the other two backbones presented in Appendix DD4
To illustrate the ONC emergence, visualization the feature and latent space evolution throughout training are shown in Figs. and The red dashed lines denote the thresholds. Feature points are wwo-dimensional PCA-projeco feature vectors and are color-coded by class, with validation features in lighter shades. Class means highlighted with star markers, and the black arrow indicates the classifier weight. This visualization demonstrates the emergence of ONCI-3, but the ONC3 behavior is clearly different between the threshold strategies, observed in the latent space.
(Epoch
-42
-00
28
-33
-12
(Spocc
Feature
Feature Space (Epoch
Feature
Mok
-20
Labent ve Thresholds (Epoch B)
Latent vs Thresholds (Eporh 18)
Labent ve Thrasholds (Caoch 50)
Labent vs Thresholds (Epoch 5000)
Learnable
LoWe Varace a Pg50
Lavie Valakee 0320
Latine Vwrad 2m A,141
Feature Space (8porch
Peature Space (lpach 10)
Feature Space (Rpoch
Feature Space (Rpochh 5000)
2
2
2
PC1
Figure 4: Latent and feature space visualization for the ER dataset with the logit model.

---

C6 2494
530 Ro
LAS Nocto
Lebent Thresholda (tpech 0)
Labent Threaholds IEsoch 10)
Latent Thresholds (taech 200)
Labent Threaholds (Epoch 300)
Fixed
Litert Vartatle hund
Variable
Varisk hyx
Feature Space IEpoch 01
Feature Space (Spoch 10)
Festure Space (Epech 100)
Feature Space (Epoch 200)
g
PC1
PC1
PCE
(Rpoch
Thresholds (Spach
Learnable
10
-3750-3500- 1150 2000 150 -do -350 Vanatr
13
20
-10
Thins
Space
ves WESISEIEE
Figure 5: Latent and feature space visualization for the UTKFace dataset with ResNet101.
Discussion
Perspectives and future directions. ONC can also offer practical advantages. Although fixed thresholds were introduced primarily to establish ONC, shown in Fig. 2 they tended to yield faster and more stable convergence compared with learnable thresholds. Moreover, as shown in Fig. 3 for the more complex UTKFace dataset, it can be seen that using fixed thresholds leads to dramatic improvement classification accuracy. Judging from the behavior of the minimum sensitivity, this improvement is largely attributed to better classification of the minority classes. This is actually natural since fixed thresholds provide a fairer allocation of the latent space- and hence of predicted probabilities across all classes and they can offer greater robustness and generalization under label imbalance or label shift. We believe that this insight will be valuable also for practitioners.
Furthermore, the geometric structure induced by ONC can be directly utilized in the design of regularization terms or loss functions. For instance, adding lightweight penalties that attract each class mean toward the classifier axis or to the corresponding threshold midpoint may accelerate training, especially in scenarios with scarce labels or significant class imbalance. We leave such extensions as promising directions for future exploration.
Limitations. The theoretical development in Section assumes that the thresholds b are fixed. Although our experimental results suggest that ONC1-2 also emerges even when b is learnable, this has not yet been theoretically established. Moreover, we believe that there exist only two phases-on one with the trivial solution and the other with non-trivial solution-but we have not succeeded in rigorously proving this. Instead, Theorem 4.3 we cireumvented this gap by assuming the continuity and monotonicity of w" w.r.t. A and Ar.
During the revision, we realized that the current definition of ONC3 (55 is only valid when the thresholds are fixed and evenly spaced in the latent space, which is the case for our fixed threshold experiments but not for the others. Even though, our overall conclusion for the learnable threshold case remains unaffected because it is clear from other analyses including the latent space visualization that ONC3 behavior does not occur the same manner as with fixed thresholds. Developing generally applicable ONC3 metrics is possible but the corresponding experiment is left for future work.
This study extended the NC theory to CLM-based OR through the UFM framework, to find ONC involving three hallmark properties. In the regularggulriization limit, we also showed that highly local and simple relation between the thresholds and the latent variables emerges. Experiments on real-world datasets confirmed the theoretical predictions and additionally demonstrated that fixed thresholds not only exhibit ONC but also lead to training speedup and better classification accuracy. These suggest that ONC can be an effective concept for generic OR tasks.
Conclusion.
10

---

Acknowledgments and Disclosure of Funding
This work was partially supported by JST, CREST Grant number JPMJCF1862, Japan (TO, TT), ISPS KAKENHI under Grant 22K12179 (TO), Grant-in-Aid for Transformative Research Areas (A), "Foundation machine learning physics" (22H05 117) (TO), and Grant-in-Aid for Transformative Research Areas (A), "Shin-biology regulated by protein lifetime" (24H01895) (TT).
References
Alan Agresti. Analysis of Ordinal Categorical Data. Wiley Series in Probability and Statistics, John Wiley & Sons, Inc., 2nd edition, 2010. doi: 10. 10.1002/9780470594000.
George Andriopoulos, Zixuan Dong, Li Guo, Zifan Zhao, and Keith Ross. The prevalence of neural collapse neural multivariate regression. In Globerson, L. Mackey, Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 126417-126451, 2024.
Mark Bagnoli and Ted Bergstrom. Log-concave probability and its applications. Economic Theory, 26(2):445449, August 2005.
Arie Ben-David. Automatic generation of symbolic multi-attribute ordinal knowledge-based DSSs: Methodology and applications. Decision Sciences, 23(6):1357-1372 1992. Data accessible the http://pppoppnnllooorgo
repositnn wwtp////////////////// 1/2028 (SWD).
(ERA), httpss//ap..oemmmooo/////0222 (LEV),
and https://api.openmlll wwwpp////...o//////ooe/eee
Ido Ben-Shaul, Ravid Shwartz-Ziv Ziv. Tomer Galanti, Shai Dekel, and Yann LeCun. Reverse engineering self-supervised learning. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 58324-58345, 2023.
Marko Bohanec and Vladislav Rajkovič. Knowledge acquisition and explanation for multi-attribute decision making. In Proceedings of the 8th International Workshop 'Expert Systems and Their Applications'. volume 1, pages 59-78, 1988. Data accessible at the UCI repository https://archiveic......edduaaua (atlu//atttse//lliaeeevlallaalt 9/cartevaluation.
Ame Brøndsted. An Introduction to Convex Polytopes, volume 90 of Graduate Texts in Mathematics. Springer, 1983.
Paulo Cortez, António Cerdeira, Fernando Almeida, Telmo Matos, and José Reis. Mod- cling wine preferences by data mining from physicochemical properties. Decision Sup- port Systems, 47(4)5547-553 November 2009. Data accessible at the UCI repository http://cchcceeceecccccccdd////// 18//inettuuallttyy
Hien Dang, Tho Tran Huu, Stanley Osher, Hung The Tran, Nhat Ho, and Tan Minh Nguyen. Neural collapse in deep linear networks: from balanced to imbalanced data. In ICML Proceedings of the 40th International Conference on Machine Learning, volume 202, pages 6873-6947. PMLR, 2023.
Hien Dang. Tho Tran, Tan Nguyen, and Nhat Ho. Neural collapse for cross-entropy class dlas-imballnced learning with unconstrained ReLU features model. In (CML'24: Proceedings of the 41th Interna- fional Conference on Machine Learning, volume 235, pages 10017-10040. PMLR, 2024.
Egbert Dierker. Competition for customers. In William A. Barnett, Bernard Coreet, Claude d' Aspremont, Jean Gabszewicz, and Andreu Mas-Colell, editors, Equilibrium Theory and Applica- fions: Proceedings of the Sixth International Symposium in Economic Theory and Econometrics, pages 383-402. Cambridge University Press, 1991.
Manuel
Pedro Antonio Gutiérrez, and Cesar Martínez. Ordinal elassification using hybrid artificial neural networks with projection and kernel basis functions. In Emilio Corchado, Václav Snášel, Ajith Abraham, Michael Wožniak, Manuel Graña, and Sung-Bae Cho, editors, Hybrid Artificial Intelligent Systems, 7th International Conference, volume 7209 of Lecture Notes Computer Science, pages 319-330. Springer, 2012. ISBN 978-3-642-28930-99 doi: 10. 000007978--642-2888 31.

---

Weinan E and Stephan Wojtowytsch. On the emergence of simplex symmetry in the final and penultimate layers of neural network classifiers. In Proceedings of the 2nd Mathematical and Scientific Machine Learning Conference, volume 145, pages 270-290. PMLR, 2020.
Cong Fang, Hangfeng He, Qi Long, and Weijie J. Su. Exploring deep neural networks via layer- peeled model: Minority collapse in imbalanced training. Proceedings of the National Academy of Sciences, 11 118(43):e2103011 118, October 2021. doi: 10. 20.1073/pnas22103011 0010009911.... URL https
// www.. www..wpp.................. org/ /wwd/os//////////// abs/aaasaa.00/ 107. pnas. 2103091118
Tomer Galanti, András György, and Marcus Hutter. On the role of neural collapse in transfer learning. In International Conference on Learning Representations 2022, 2022.
P. A. Gutiérrez, M. Pérez-Ortiz, J. Sánchez-Monedere, F. Frenández-Navande and C. Hervás-Martinez. Ordinal regression methods: Survey and experimental study. IEEE Transactions on Knowledge and Data Engineering, 281)21127- January 2016. doi: 10. 10.1109/TKDDE22255245597)
Hangfeng He and Weijie J Su. A law of data separation in deep learning. Proceedings of the National Academy of Sciences, 120(36):c2221 (2216))222277122222 7744 120, 2023,
K. He. X. Zhang, S. Ren, and J. Sun. Delving deep into rectifiers: Surpassing human-level per- formance on ImageNet classification. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 1026-1034, December 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016. doi: 10.1 10.1109/CVP.22016.99
Ralf Herbrich, Thore Graepel, and Klaus Obermayer. Large margin rank boundaries for ordinal regression. In Alexander J. Smola, Peter Bartlett, Bernhard Schölkopf, and Dale Schuurmans, editors, Advances in Large Margin Classifiers, chapter 7. The MIT Press, 2000. doi: 10.7551/ mitpres//111.0033.000
Wanli Hong and Shuyang Ling. Neural collapse for unconstrained feature model under cross-entropy loss with imbalanced data. Journal of Machine Learning Research, 25(192):1-485 2024. URL
http://jmlr org/ http//apers////// 23- 1215. 12ml.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q. Weinberger. Densely connected convolutional networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2261-22612222 1222269,,2222022 2017. doi: 10.1109/CVPR.2017..24
Like Hui, Mikhail Belkin, and Preetum Nakkiran. Limitations of neural collapse for understanding generalization in deep learning. arXiv preprint arXiv:2202.083884 [cs.LG], 2022.
László Leindler. On certain converse of Hölder's inequality. II. Acta Scientiarum Mathematicarum, 33:217-223, 1972.
Pengyu Li, Xiao Li, Yutong Wang, and Qing Qu Neural collapse in multi-label learning with pick-all-label loss. In /CML 24: Proceedings of the 41th International Conference on Machine Learning, volume 235, pages
PMLR, 2024a.
Xiao Li, Sheng Liu, Jinxin Zhou, Xinyu Lu, Carlos Fermandez-Grande Zhihui Zhu, and Qing Qu Understanding and improving transfer learning of deep models via neural collapse. Transactions on Machine Learning Research, May 2024b.
Jianfeng Lu and Stefan Steinerberger. Neural collapse under cross-entropy loss. Applied and Computational Harmonic Analysis, 59:224-241, July 2022. doi: 10. 10.1016/j.acha, attp://...aa............ 2021.
Wojciech Masarczyk, Mateusz Ostaszewski. Ehsan Imani. Razvan Pascanu, Piotr Mitoš, and Tomasz Trzcinski. The tunnel effect: Building data representations deep neural networks. A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 76772-76805, 2023.
Dustin G. Mixon, Hans Parshall, and Jianzong Pi. Neural collapse with unconstrained features. Sampling Theory, Signal Processing, and Data Analysis, 20(2):11, 2022.
12

---

Bac Nguyen, Chieh-Hsin Lai, Yuhta Takida, Naoki Murata, Toshimitsu Uesaka, Stefano Ermon, and Yuki Mitsufuji. Mitigating embedding collapse in diffusion models for categorical data. arXiv preprint ar arXiv:2410. N11:22410.14758v [cs.I.G], 2024.
Vardan Papyan, X. Y. Han, and David L. Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117(40): URL BtL.http:://ww: pnas org/
24652- 24663, September 2020. doi: 10.
/a////bbs100... pnas. 2015509117
Liam Parker, Emre Onal, Anton Stengel, and Jake Intrater. Neural collapse in the intermediate hidden layers of classification neural networks. arXiv preprint arXiv:2308.0276600 [cs.LG], 2023.
András Prékopa. On logarithmic concave measures and functions. Acta Scientiarum Mathematicarum, 34:335-343, 1973.
Akshay Rangamani, Marius Lindegaard, Tomer Galanti, and Tomaso A. Poggio. Feature learning in deep classifiers through intermediate neural collapse. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 28729-28745.. PMLR, 2023.
Barry Simon. Convexity: An Analytic Viewpoint, volume 187 of Cambridge Tracts in Mathematics. Cambridge University Press, Cambridge, 2011.
Peter Súkeník, Marco Mondelli, and Christoph H Lampert. Deep neural collapse is provably optimal for the deep unconstrained features model. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances Neural Information Processing Systems, volume 36, pages 52991 1-30224, 2023.
Peter Súkeník, Christoph Lampert, and Marco Mondelli. Neural collapse versus low-rank bias: Is deep neural collapse really optimal? In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 338250-132288 2024.
Christos Thrampoulidis, Ganesh Ramachandra Kini, Vala Vakilian, and Tina Behnia. Imbalance trouble: Revisiting neural-collapse geometry. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 27225 27238, 2022.
Tom Tirer and Joan Bruna. Extended unconstrained features model for exploring deep neural collapse. In Kamalica Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, International Conference Machine Learning, volume 162, pages 21478 21501. PMLR, 2022.
Victor Manuel Vargas, Pedro Antonio Gutiérrez, and César Marttn--arttinz.. Cumulative link models for deep ordinal classification. Neurocomputing, 401:48 58, August 2020.
Jan Verwaeren, Willem Waegeman, and Bernard De Baets. Learning partial ordinal class memberships with besedl-based proportional odds models. Computational Statistics Data Analysis, 56:928- 942, April 2012. doi: wttp://.ssaa.22...2222.
Robert Wu and Vardan Papyan. Linguistic collapse: Neural collapse in (large) language models. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 137432-137473. 2024.
Zhifei Zhang, Yang Song, and Hairong Qi. Age Progression/Regrress
by Conditional Adver- sarial Autoencoder In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4352-4360, Los Alamitos, CA, USA, July 2017. IEEE Computer Society. doi: 10.1109/CVPR. 463. URL. htttt::////:iii http:////dooi.eeecccccmeeeeeee htw.:aaiieeeeeeoppetreeeeeee.e.e http:///c....aaaoo.ooooooooaoo/a
2017.468 463
13

---

Jinxin Zhou, Chong You, Xiao Li, Kangning Liu, Sheng Liu, Qing Qu, and Zhihui Zhu. Are all losses created equal: A neural collapse perspective. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 31697-31710, 2022.
Zhihui Zhu, Tanyu Ding. Jinxin Zhou, Xiao Li, Chong You, Jeremias Sulam, and Qing Qu A geometric amalysis of neural collapse with unconstrained features. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information 2021.
Processing Systems, volume 34, pages
14

---

A Log-concavity
A.1 Prékopa's theorem
We first recall theorem, due Prekopa [973], which is derived from the Prékopa-Leindler inequality (Theorem A.2) on log-concave functions. basically states that marginalization preserves the log-concavity. We use prove Theorem 4.1 in the main text.
Theorem A.1 (Theorem 6 of Prékopa 1973 ). Let f(x, y) be function of (n + m)) variables where a and y are n-- and m-dimensional, respectively. Suppose that f is log-concan on R=+m and let A be convex subset Rm.. Then the function of defined by
J. ((x,3)) dy
(26)
is log-concave R
A.2 Strict log-concavity
Here we show that the strict log-concavity of gz(x) on R ensures the strict convexity of L(z, a, b) in
2. Although proving it would be easy one can assume differentiability g', as demonstrated in Appendix B holds even without the differenntiaiiity assumption, as shown in the following.
We start by recalling the Prékopa-Leindllrr inequality (Prékopa 1973 Leindler 1972].
Theorem A.2 (Theorem 3 of Prékopa 1973]. Suppose that X I....,k, k, are positive constants satisfying Σ A 1. Let f ft be nonnegative and Borel measurable functions IR, and
let
r(t) = sup f( ... -. fx(x))) t EER.
(27)
Σ Aæt
Then the function rt) is Lebesgue measurable and we have
[ r(t) dt 2
II (J 1=1
doz
(28)
We need some more preparation.
Lemma 1. Let f D > [0, oo) be a strictly log-concave function on convex set D. If there exists 3o E for which f(x) = 0, then ag is an extreme point of D.
Proof. We prove that if æo is not extreme point of D (that is, zo can be represented as convex combination of x, y E D such that Da Ax + (1 A)y. 3X E (0, 1)), then f (aro) cannot be zero. From the strict log-concavity of f, one has
f(xo) (Aa + A)y) f(x) 20,
(29)
showing that f(xo) positive.
It should be noted that Lemma implies that any strictly log-concave function on R" is strictly positive.
Let A be subset of R" For R", let Aa fx + E A] and Aia A\ (Aa)).
Lemma Let A convex subset of Th and R. be a non-zero vector. Then one has:
(a) If Aya 0, then one has (a ta A.t [0, 20)3 A.
(b) Consider the set Auta indexed by t e R. IF A = for some to > 0, then A holds for any 2 0.
(c) Aqta is nondecreasing [0, oo) (that is, for any 0 S < to one has Asaa C Alega) and nonincreasing (-o0, 0).
15

---

Proof. We first prove (a). Aa = 0 implies A C Aa, that is, for any æ E A one has x E A. By induction, for any æ E A one has a ma E A. for any m E {0, 1, .. Due to the convexity of A. one obtains æ ta E for any & 2 0, proving (a).
We next prove (b). From (a), the condition A4aa = implies that for any a E one has x E Ata for any & 2 0, which in turn yields A/ = 0 for any 2 0.
We now prove (c). Fix 0 < 4 < 12- Take any point x: E Aqua which implies that æ E and
e-t1a fia & A. We show that æ t2a & A. For this purpose, assume, to the contrary, that æ -2a E A holds. One has
1 1- 2/1 (a (2-
(2- t2a)
41 12
- tia
x.
(30)
which shows that æ a convex combination of æ and 12a, both lying in A. Due the convexity of A, it would follow that æ - a E A, which is a contradiction. We have thus proved that for any Avr. one has æ laa E A, and hence x E A It in turn implies Avna C A proving that Avaa is nondecreasing on [0, po). The statement that Asta is nonincreasing on (-oo, 0
can be proved in the same manner.
For subset A of I2, let ((A) denote its volume. Lemma 3. Let A be convex subset of R", and E R" be a non-zero vector.
(a) If (((A) = 0 for some to > 0, then one has (((Atta) = 0 for all / //2 0.
(b) If (AAA)ta)) 0 for some to > 0, then one has (u(Auua) > 0 for all t 0.
Proof. We first prove (a). The condition p4 = O implies that for almost every x E A one should have E Atsa: eqivalently, --taa E A. One then has x 2fy A for almost every such x E A satisfying x - tga E A. By induction, for almost every a E A one has x mtga E A for any m 21. Due the convexity of A, for almost every A one has (ær la 2 0} A. It then implies that, for any L 2 0 and for almost every x E A, one has x E Ata. showing that ((A holds for all /. 2 0.
Proving (b) is straightforward by noting that if one has ((A > 0 and (()))))) = 0 for la, 4 > 0, la A 4,, the latter would imply, via (a), (aa = 0 for all / 2 0, which is in conflict with the former.
We need the following assumption. Assumption 1. The subset A of R is convex. Furthermore, for any non-zero vector a E R", either ((Aaa)) (())a) is positive.
The last positivity assumption in Assumption does not always hold (consider the case where A is half-space and a is parallel to the boundary of A). Lemma below gives a necessary and sufficient condition for that.
does
Lemma 4. Let A be a convex subset of R" with μ(A) > 0. Assume that has n supporting hyperplanes Hi...., Hn with their normal vectors ..,..., nr forming a basis of R=. Then for any non-zero vector a E R", either p(Ara) a or p(A;-a) is positive. Conversely, if the normal vectors of the supporting hyperplanes of A do not span IR"", then there exists * 0 with which ((A,a) = =[(A-a)) = 0 holds.
Proof. We first prove the former statement. As {Tt..... is basis of R", not all the inner products {(n,, a) j= are simultancously zero for any non-zero vector a. Assume without loss of generality that (n1, a) of 0. Let p H1 be boundary point of A. Then one has Hi {xr E IR" (n1.2 x - p) 0}. Take -n, in place of nı if necessary, one can assume that {I R" (n1, p) 0} A holds.
Take a point z in the interior of A, and a ball (z) of small enough radius > centered at z, so that (z) C A holds. Consider the hyperplane H = {Ix E R" (n), x z) 0). It is parallel to H, and passes through z. cuts the set into two parts, each of which contains the half of the ball
B, (æ) and is thus of positive volume. Let A' {x A (ni, x z) < 0). that is, A' is the set of
16

---

points in A which resides on the same side of H as the point p. One consequently has p( > 0 since A' contains the half-ball.
Let
p' = + ia. t= n11 2 - P) (n1, a)
(31)
The point p' is on H because
(n1.7 p' - 2) (n1,p- 2) (n1, P) (n1, =0
(32)
(n), a)
holds. Furthermore, H is supporting hyperplane of Atai noting that p' +p+ e Ata tottal a A}, one has
(n1, x + --=) =2) (n1, x p) + (n p + ta =)
(n1, x p) + (n1.7 p' 2) =)>0 0,
(33)
where the last inequality is due A for (n1, p) 0) and (32). It implies that Ata C {
(n1, x - 2) 2 0} holds. It should be noted that the point lies on the opposite side of H to Ata- which can be confirmed by noting that E A C (æ (m1, x p) 0) implies (n1,p- 2) 0.
One consequently has A.a A' 0. which furthermore implies A' (Ata).
One therefore has At An (Ata) C A', and thus u(A{taa 2 p(A') O. One can then apply Lemma to show that either aa))a) or pa) is positive.
We next prove the converse. Let W the linear span of the normals of the supporting hyperplanes of A, and let TW R" W be the orthogonal projection onto W. The supporting hyperspace of A with the normal n E W can be represented as on 2 (n, x) = (n, (w))). Since the closure A of A is convex and thus is the intersection of its supporting hyperspaces [Brondsted] [983] Theorem 4.5), one has
A
n for (n, S cm)
neW
n fx (w))) cnl nEW
N (lV E W (n, S cm) + W
neW
(y E W (n, S s0)) W-, nnew
(34)
showing that A is cylinder since W + {0). It immediately implies that for any E W one has (A9), = 0, and consequently, /a( (Aaa) =
The positivity assumption in Assumption 1 is automatically satisfied if A is convex and p(A) E (0, 0o), as shown in the next lemma. Lemma 5. Let A be convex subset of R and a E IR be a non-zero vector. Assume p(A) E (0,
Then one has a(Ara) > 0.
Proof. The condition μ(A)) E o0) implies that A is bounded and that its closure A is compact. For any nonzero n E R", the function æ + (nt, x) is continuous, and it attains its maximum on on A". Then (n, x)) = Cn the supporting hyperplane of A with normal n. One can then observe that, by taking nı = = n in the proof of Lemma 4. one has ((A,.)) > 0.
We now state the main theorem.
Theorem A.3. Assume that f() is strictly log-concave on R For a subset A TR satisfying Assumption 7 let p(az) be defined by
IR"
p(x) / flu - x) du.
(35)
Then p(x) is also strictly log-concave.
17

---

Proof. Take arbitrary 2u, 21 E I"" with æ1 show the strict inequality p(ax) > ((x=)) to hold for any À E (0, 1), which proves the theorem. Let
ô o 0 and let æ = (1 \)zo + Az1. We will
Ex(u)= (x)) A).
(36)
One then has
p(xx)= J, f(u f(u-xx) (xx)du
Fx(u) du. JR"
(37)
(u) is log-concave in because it is a product of the two log-coneave functions f(u æx) and and because multiplication log-concave
1uu E A) (note that A is convex due to Assumption I functions preserves log-concavity.
Lemma 6. For any À E (0. 1) and any uu U1 E R", let uy = (1 N)eo + Xun, One then has Fx( 2 Fo(uu)) (o(uu))->F(uu)) with strict inequality when Fxxx)) 0 and uo + 8.
(38)
Proof of Lemma 6
Fix À E (0, 1). Assume that uo un such that uY A and u1 Uu o ô hold.
One then has (ux) = f(ux axx) > 0, where the positivity of f is due to Lemma For such uo, U1 one has
Fx(ux) ʃ(ux 2x) 5(11 \)(uo xn) + >(wi -x1))
> 1(un - 20)) 1- -(( - x²))^
2 Fa(uo)) 1- 1- 1 (ui)²,
(39)
where the first inequality due to the strict log- log-concavity of f and the condition ug - zo F u1 æ1 which is derived from u, uo # 5 = I To. and where the second inequality results from multiplication with the indicator functions.
uo 26n and yy x (1- A) x1) + A(ug- - zo) = ue Ho so that f(ux- - 2xx is constant for A E [0, 1]. and hence the inequality (38) holds. For uo, 21 such that ux & A holds, either un u₁ should lie outside A. so that one has the equality Fx
For uo, such that ux E A and un uo 8 hold, one has u
(wo))) a (ur) =0.
We return to the proof of Theorem A.3 Let
r(u)=
sup
(40)
u0.
Xjua Aui=u
Then for u & A one has (u) r(u) = 0. On the other hand, for u E A one cannot have
ug Un E A which satisfy both 41 Ug = 5 and (1 - A)uo + Au = u: indeed, since up E A, one lets u1 uu + ō. one has Ass8- This, together with Lemma in turn implies the strict inequality Fx(u > r(u) for u E A A One can similarly show the strict inequality Fx) > r(u) to hold for u E A (11-)6 as well.
(1 A)uo Auu utp 2o E Axss which should not lie in
As have shown the strict inequality Fx) > r(u) to hold for in the set Asso - which has a positive volume for 8 & O due to Assumption ] one has
p(ax)= = [ Fx(u) du
f r(u) du
2 ( Fo(u)) du
(i(w))du )
p(xo)) 1-> p(
(41)
where the first inequality is due to the strict inequality shown above, and where the second inequality derived by applying the Prékopa--einee Leindler inequality (Theorem A.2 This proves the strict log- concavity of p.
18

---

Strict log-concaving of the function
g(b 2) -9(a 2) [ g'(u g'(u-20 2) du
(42)
in under the assumption of strict log concavity of g' is immediate from Theorem A.3 It proves the strict convexity of L(z, a, b) logig 2) -( 2)] as well.
should be noted that the function f(u x v E A), whose integral with respect to u E lk"" yields p(x), is itself log-concave but not strictly log-concave in (æ, u), so that one cannot apply the argument in [Prékopa| 1973] on the strict log-concavity to our case.
B Derivation of ONC3
We show in this section the ordering z1 S 22 S 2Q of the optimal latent variables. For this purpose, we discuss how the minimizer of the optimization problem
A )
min
A> )
+
<b
(43)
with the function Mxx, a, b) being of the form
[(x, b) log log x) P(a x)).
(44)
behaves one changes a, h. where P iss an indefinite integral of a function p. which is log-concave on R. because the optimal latent variable 2i wa. h given w > O is determined the optimal solution of the following minimization:
An 2w
min
[xx, by-1; by) +
x22
(45)
with [xx, a. b) defined as in 44 using
= P g(u) du.
Theorem 4.1 ensures that L(x, a, b) is convex in for any a, b, which in turn ensures that the minimizer of (43) unique. We show in the following that the minimizer & = á(a, b) monotonically
non-decreasing in and b. This monotonicity will prove the desired ordering of {2
The minimizer å satisfies the stationarity condition
( a. b) + A# = 0,
(46)
where the subscript of T. denotes the partial derivative of L with respect to æ. It should be noted that particularizing (46) in the optimization of the latent variable zy yields EOS (16a).
We first assume differentiability of p. Taking the derivative of both sides of (46) w.r.t. a, one has
L (f, a. b)f. + Lza (ā, a, b) + Ada = 0,
(47)
yielding
Liza (f, a, 6)
f.
(48)
Iian (5, b)
Similarly, one has
(i, a,
3, L (ē, a, b)
(49)
19

---

Since L(z, b) is convex in z, one has (2, az a, b) 2 0. One also has
p(b x) p(a x)
Lz( P(b - σ) - P(a - z)
p 2)
pp(b - 2) -) - p(a x))p(a- x)
P(b 2) P(a- x))
P(( - 2) P(a x))
p(a z)
p'(a z) p) p(a x)
(50)
P(b 2) P(a- 2)
p(a - z) P(b (1b-1) r) P(a 2)
p'(b z)
[p(6 x) p(a r)]p(b - zx)
(2b b.
P(b z) P(a x)
[P(b 2) P(a x]]
p(b 2)
pb(bb -b)b)) p(b x) p(aa x) p((b x) P(b =) P(a x)
(51)
P(b - z) - P(a x)
p' (b- x) (a 2)
[p(b z) p(a =)}²
Lua (z, b)
P(b - x) P(a - x)
[P(b x) P(a - x)}²
- a) + Las(a, a, b)].
(52)
Since p(u) is assumed log-concave, (log p(u))' p'(u) d'(u)/p(u) /p(() is monotonically non-increasing. One therefore has, via the technique used in [Dierker 1991 Proof of Lemma 1] and Bagnoli and Bergstrom 2005].
pa xx) [P(b x) P((a [)]] p(a 2
pl(a x)
( p(u) du
p(a r) aa-x
b- = p'(u)
2
p(u) p(u) du
p'(u) du a-
p(b x) pla- - x),
(53)
which, together with (50). implies that Lea b) S 0 holds. Combined with Lex (z, b) 2 0
and (48). in turn proves C. 2 0. Similarly, one has
p'bb x) p(b x) [P x) P(a - x)]
pb(b x) f p(u) du
p(b x)
= p'(u)
p(u) p(u) du
p'(u) du
- pla - x),
(54)
which, together with (TT) implies that (xs a) < 0 holds. Combined with (49) it in turn proves i 2 0.
We next discuss the case where p is not necessarily differentiable. Since p(u) is log-concave, q(u) log p(u) is convex, that it is continuous and differentiable except countable set. Let ((u) be any function such that Φ(u) takes a value in the subderivative of q(v) for any u. As
p(u) = -tt)), one has, at any point u at which p(u) is differentiable,
p' p'(u)= = -q'(u)e -q(u). = (u))((u)
(55)
One therefore has, for any b > a,
p(b) p(a) - f (())p(u) du.
(56)
which can be proved in the same way [Simon] 2011 Theorem 1.28). On the other hand, we know that d(u) is monotonically non-decreasing in u. One thus has
((PP))(((a))))))) = -((a) S. p(u) du S f d(u)p(u) S {bb J."
'' p(u) du ((b))())-P(a)),
(57)
20

---

which, via replacing (a. b) with (a- x,-x0 b. z). proves inequalities corresponding to (33 and 54
One has therefore proven the following proposition. Proposition B.I. For a < b, let L(x, a, b) as defined (44 and let ø(a, b) be the minimizer of the optimization problem 43 Then, for < b and a' 8' with S a and I 8', one has B(a, b) & á(a' bb).
We would like to note that if is differentiable and strictly log-concave, then we have strict inequalities in the above proposition, which in turn implies, via (52). that Liz (z, a, b) 2 This constitutes an alternative proof of the strict convexity of L(x, b) in z under the differentiability assumption. Even without the differentiability assumption, one can note that, under the strict log- cognccivvvtty of p. d(u) is increasing, so that the strict inequalities hold in (57), proving the striet inequalities in the above proposition as well.
The above argument proves the first half of ONC3, that is, for any fixed w one has 2 S 2 for q E (1..... Q 1}. Furthermore, if g' is strictly log-concave one has the strict ordering when + 0: 2q < 2 for 9 E {1, {1...., 1}, thereby proving the latter half of ONC3.
C Details of the Experimental Setup
Dataset statistics. We used the five publicly available real-world OR tabular datasets of Gutiérrez et al. 2016 ERR, LE, and SW (Employee rejection/acceptancee, Lecturers evaluation, and Social workers decisions (public domain) David 1992]). CA (Car evaluation (CC BY 4.0) [Bohanec and Rajkovič 1988)). and WR (Wine quality Red (CC BY 4.0) [Cortez et al. 2009] exactly as released. The website offers 30 pre-defined training-validatie distributions are identical across the two partitions. We ran our experiments on all the 30 splits for each dataset and report the averages.
hold-out splits whose label
Table summarizes the key statistics of the five datasets used in our study. For consistency, all ordinal labels were remapped to consecutive integers starting from one. We constructed input vectors by concatenating two types of preprocessed attributes: one-hot encoded categorical attributes and normalized numerical attributes.
Table 1: Summary of tabular datasets used in the experiments. Attr. denotes attributes and Input dim. denotes the dimension of input vectors.
Dataset (code)
Subset
#Samples (per split)
#Attr:
Input dim.
#Classes
Distribution (comnts per label)
Train Val
750
11:69, 2:106, 3:136, 4:129, 5:118, 6:89, 7:66, 8:23, 9:141 [1:23, 2:36, 3:45, 4:43, 5:40, 6:29, 7:22, 8:8, 9:4]
ERA (ER)
13
250
Train
10:70, 1:210, 2:302, 3:148, 4:20)
LEV (LE)
Val
250
[0:23, 1:70, 2:101, 3:49, 4:7]
Train
750
12:24, 3:264, 4:294, 5:163]
SWD (SW)
14
250
[2:8, 3:88, 4:100, 5:54]
Train
1296
[ace:288, good:52, unace:907, vgood:49]
Cur (CA)
21
Iace:96, good:17. unace:303, vgood:166
Train
1199
13:8, 4:39, 5:510, 6:479, 7:150, 8:13)
Wine (WR)
Val
400
11
17
4:14, 5:171, 6:159, 7:49, 8:5]
Additionally, we conducted experiments on the UTKFace dataset (non-commercial research use) (Zhang et al.] 2017]. large-scale face image dataset for age estimation. The dataset con- tains 23,708 facial images with age labels. We grouped ages into 20 ordinal classes using 5-year intervals. The dataset exhibits natural class imbalance, with the most frequent class (25-29 years) containing 5,034 samples and the rarest class (95-116 years) containing only 67 samples. Following standard practice, performed stratified 80/20 train-vall cain-validation split to maintain class distribution across subsets. All images were resized to 224 x 224 pixels and normalized using ImageNet statistics for pre-trained backbone compatibility. Table presents the detailed class distribution.
httpp:://wwtt uco. http:/wwwwwww/.p/////////aa///e/ ht//grppo//yymm//ottvvvee
21

---

Table 2: Class distribution of UTKFaae dataset with 5-year age grouping.
Class
Age
Train
Val
Class
Age
Train
Val
0
[0-4]
1,733
434
10
[30-54]
1,076
269
1
[5-9]
716
179
11
[55-59]
763
191
2
[10-14]
471
118
12
[60-64]
586
146
3
[15 19]
753
189
13
[65 69)
469
117
4
[20-24]
1,848
462
[70-74]
298
75
5
[25-29]
4,027
1,007
15
[75-79]
261
65
[30-34]
[80-84]
7 [35-39]
1,797
450
17
[85-89]
214
53
8
[40-44]
945
236
[90-94]
82
20
[45-49)
851
213
19
[95-116]
54
13
4,742
Overparameterized network. Here we describe the neural network architecture used in our experi- ments. For the tabular datasets, we employed a multilayer perceptron with residual connections as follows:
An input æ E Rd is first mapped to a 128-dimensional representation by linear layer, then passed through parametric rectified linear unit (PReLU) (He al., 2013].
It then passes through four residual blocks, each defined as
x x PRRLU (W₂ (W + 62), W1, W2 E 128x122 b, b2 E R128
It subsequently passes through three consecutive linear layers with linear activation, yielding the 64-dimensional feature (x).
Finally, linear layer without bias-whose weight vector is the classifier weight analyzed in this study- maps the he (æ) to the one-dimensional latent variable 2 = w ho(x).
The PReLU activations together with the linear tail give the network enough flexibility map inputs to any location the feature space, aligning with the UFM assumption. Figure illustrates the architecture of the network.
Unver Lategta g2
Unear Lager in
Laver opar
Laae Lave
Liver Layarg
Friul
Lane Loen
Uneer Lager
Liver
Unwer
Figure Architecture of the overparameterized network for tabular datasets.
For the UTKFaae dataset, we employed pre-trained ResNet50, ResNet101 (He et al. 2016 and DenseNet201 (Huang al., 2017) as backbones. To better satisfy the UFM assumption, augmented each backbone with two additional linear layers that maintain the same dimension as the backbone output. The mapped features he (x) are then passed through final linear layer without bias to produce the dieedimenli latent variable = w he where is the classifier weight analyzed in this
study.
Other experimental settings. For the tabular datasets, all the models were trained for 5000 epochs with the Adam optimizer, starting from the initial learning rates listed in Table 3 We made learning rate to decay by factor of 0.1 at epochs 200, 800, and 3000. We used a batch size of 2048 and applied weight decay of 5 x 10-3 to all network parameters (except that the lcarnable thresholds, when present, received zero weight decay). For each dataset tested the four possible combinations the three link functions (logit, probit, clog-log) and the two threshold strategies (fixed, learnable). Running every configuration the 30 predefined hold-out splits yielded total of 600 training-var validation wraiiing-vliidation runs.
To ensure that the training error reached (near- -aarr-terroeee regime where feature collapse is observed we tuned only the initial learning rate and, fixed thresholds, the threshold range. Because logistic function has heavier tails than the normal CDF, the logit runs used a wider fixed range [-20, 220,200 20), whereas the probit runs used 2). The complete hyper-parameter grid summarized
in Table 3
22

---

For the UTKFace dataset, models were trained for 300 epochs using the Adam optimizer with batch size of 512 per GPU across 5 GPUs (total effective batch size 2560). We tested two threshold strategies (fixed, learnable), using only the logit link function. Starting from the initial learning rates listed Table the learning rate was decayed by a factor 0.1 epochs 100 and 200. Each configuration was repeated with three different random seeds. The other Syper--arameee settings for each backbone are summarized in T
Experiments compute resources. For the tabular datasets, every run was executed on single NVIDIA RTX A6000 (48 GB) GPU, taking roughly 5-10 minutes and requiring only a few hundred megabytes of GPU memory. For the UTKFace dataset, training performed across 5 NVIDIA RTX A6000 GPUs using distributed data parallel, with each configuration taking approximately hours to complete 300 epochs.
Table 3:
Hyper- parameter settings for tabular datasets with three link functions (logit, probit, clog-log) and two threshold strategies (fixed, learnable).
Dataset
Link function
Threshokds
Threshold range (when fixed)
Initial learning
logit
fixed
[- 20, 20]
x 10
logit probit
learnable
1
ER
fixed
[-2, 2]
10-3
probit
learnable
1x 10-3
clog-log
fixed
2, 2]
5 10-3
clog-log
Iearnable
5 10-3
logit
fixed
20, 20]
1 10
logit
learnable
X 10
LE
probit
fixed
[-2, 2]
5 10
probit
learnable
5 10-3
clog-log
fixed
[-2, 2]
10 10-3
clog-log
Isarnable
10
logit
fixed
[- 20, 20]
1 10
logit
learnable
1 2
Sw
probit
fixed
[-2, 2]
10
prohit
learnable
5 10
clog-log
fixed
[-2, 2]
5 10
clog-log
learnable
G 10
logit
fixed
[- 20, 20
1 10
logit
learnable
1 10
CA
probit
lixed
[-2, 2]
X 10
probit
learnable
5 10
clog-log
fixed
[- 2]
clog-log
Isarnable
10
logit
fixed
[ 20, 20]
1 10
logit
learnable
1x 2
WR
prohil
lixed
[-2, 2]
probit
learnable
1x 10
clog-log
fixed
[-2, 2]
1 10
clog-log
learnable
1 10
Table 4: Hyper-paramee parameter settings for UTKFace experiments with logit link function and two threshold strategies (fixed, learnable).
Backbone
Thresholds
Threshold range (when lixed)
Initial learning rate
Weight decay
fixed
40, 40)
1 10-3
1 10-4
ResNe150
lenrahll
I x 10
1 10
[-40,
10-3
1x-30-
ResNet101
learnable
1 x 10
1 10-3
fixed
[-40, 40]
5 x 10-4
1 x 10-4
DenseNer201
learnable
x
x 10
23

---

D Additional experimental results
D.1 Results with the logit model
This section presents the experimental outcomes obtained using the logistic function (i.c., g(x) (1+e-")-) which corresponds the logit model. Figures 10 show the evolution of evaluation- metric curves for the datasets LE, SW, CA, and WR, respectively. Figures [ visualization of the latent and feature spaces for the datasets LE, SW, CA, and WR, respectively. These exhibit
consistent behavior with the one of the ER dataset in the main text (Figs, and
Fixed Tain
Foxed
Learnable Train
Learnable Val
1.0
0.3
0.0
2.
10-1
0.4
00"h
207
2030
4020
Epoch
Epoch
Epoch
0.7
0.0
0.2
32'
10
0-4
0.4
10'14
2000
4002
3000
4000
Spath
booch
Epacc
Figure 7: Epoch-wise average metrics curves for the LE dataset with the logit model, comparing fixed- and ceamable-threshol/ models.
Fixed Tain
Fixed
Learneble Train
Learmable Val
1.0
0.3
0.8
10
MAC (oncent
30
2
0.6
0.4
30t1
2000
4030
Epoch
Epoch
Epoch
Wermum Sersittrry
0.7
0.3
0.2
30'
0.4
10+1
10"1
2000
1002
3000
4000
Epoth
Eooch
Epech
Figure 8: Epoch wise average metries curves for the SW dataset with the logit model, comparing fixed- and learnable-treesholdd models.
24

---

1071
MMC (occett
Witho ACC
0.6-
10
0.3
0.6
0.2
02
0.2
10
2333
4233
2003
4000
Esoch
10"
0.2
0.2
0.2
2000
2000
CCCS
Epoch
Epoch
Epach
Sparc
Figure 9: Epoch- wise average metrics curves for the CA dataset with the logit model, comparing fixed- and tearsole models.
Learnable Train
Learnable
3.0
1.0
0.0
10
Withe
0.4
2077
43
0.0
0.0
Esoch
2000
Epoch
4300
Epoch
2000
Epoch
4000
2006
Epoch
4000
3.2
1.0
1.2
0.6-
0.66
ONC.
30-1
10-1
2003
.000
Spoch
1203
Spoch
Figure 10: Epoch- average metrics curves for the WR dataset with the logit model, comparing fixed- and caamable-threshole models.
UME
Threchalds (poch D)
Lateet Thrnshokis (Rpach 18)
Lateat Thonsholde (Rpoch 50)
ve Thresholds (Spach 5000)
Liter Va-11e m-m Nole
Leved Varihe 1 Puce
Liter Virith d (Nle)
Later Vereble 2- Auli
Space (Epoch
Festure Space
Space (tpach
Peature Space (Epoch
-04
25
00
Threshalds
Throsholds
Ora
Oua
Learnable
12
-23
Feature (Epoch
Feature
Feature (Feoch
Feature (Spech S000)
02
83
52
04
05
10
3
-5
10
-5
11
Figure 11: Visualization of the latent and feature spaces for the LE dataset using the logit model, comparing fixed- and learnable-threshold models.
25

---

-250
-135
000
035
850
375
-200
-000
10
-5
Latent Thresholds (Ppoch 0)
Latent us Threshalds (Spech 10)
Latent vs Threshates (tpach so)
Ladent vo Thresholds (Rpochh 5600)
Learnable
Litet Add1
Latert Vertable hyr
Latent Veriable Adai
Latert Vartable
Festure Space (Epech 0)
Feature Space (Epoch 10)
Festure Space (Epech 50)
Fasture Space (Epoch 5000)
2
1
Figure 12: Visualization of the latent and feature spaces for the SW dataset using the logit model, comparing fixed- and tearsalle-treesolld models.
Thresholds (Epech
Threcholds lEpoch
Latent Taracholds (Gpoch
Latent Thresholde (Spach 5060)
Cas
Fixed
-10
Feature Space (Epoch
Feature Space (Epoch
Festure Space (tpech
Feature (Epach 5000)
Latent ve Thresholds (fiach 0)
Labent ve Threcholds (Epoch 10)
Latest us Threcholds (Epech 50)
Latent ve Thresholds (Rgoch 5000)
Leaet Vortileee
Variade
Varible..c But
Feature Space (Epoch 0)
Feature Space (Epoch 10)
Feature Space (Epoch 50)
Feature Space (Epoch 5000)
2
*
Figure 13: Visualization of the latent and feature spaces for the CA dataset using the logit model, comparing fixed- and leamhale-threshold models.
26

---

CHe
-10
-21
Feature Space (Epoch
Feature Space (Epach
Feature (Epach
Feature Space (Epoch 5000)
wxpe +
Latent ve Thretholde (Epoch
Latent ve Threcholds (Epach 10)
Lacent ve Threchelds (Epech 50)
Latent ve Threcholde (Epach 5000).
Leamable
Liart Faride A:141
Letert
Lifert Verlabk Thun
Vortade 2M (1/X)
Festure Space (Spoch 0)
Festure Space (Epech 10)
Festure Space (Eaoch 55)
Festure Space (Epoch 5000)
25-
23
MC1
Figure 14: Visualization of the latent and feature spaces for the WR dataset using the logit model, comparing fixed- and learnable-threshold models.
27

---

D.2 Results with the probit model
This section presents the experimental outcomes obtained using the normal CDF (i.e., g(z) (2))) which corresponds to the probit model. [15 show the evolution of evaluation metric curves for the datasets ER. LE, SW, CA, and WR, respectively. Figures 20 24 show visualization of the latent and feature spaces for the datasets ER, LF,, SW, CA, and WR, respectively. These again show consistent behavior with that the main text.
Fieeo Train
Rited Val
Learable Train
Learnable Val
MAE (onden
Withoo3
0.4
12
0.2
2000
9900
2000
2000
4000
2000
4005
Spoch
Spoch
Epoch
1.0
1.0
0.0
64-
10
0.4
0.0
0.0
2000
4033
2000
4297
2005
0000
2000
Epoch
Spoth
Epoch
Figure 15: Epoch-wise average metrics curves for the ER dataset with the probit model, comparing fixed- and leamable-threshold models.
Rxed
Leamable Train
Learnale
1.0
0.3
NAVE Condert
0.6
02
10
2003
Spoch
4000
2000
4000
Epoch
Epoch
Hom Spr5
0.0
0.0
0.0
13
ObCr
0.2
62-
10.
X001
4390
3000
4000
Epach
tpach
tpoch
Epach
Figure 16: Epoch-wise average metrics curves for the LE dataset with the probit model, comparing fixed- and cearnable-thrshol models.
28

---

Foeed Train
Fooed
Leamable Train
Learnale Val
0.0
0.3
MAE conter
Wh-r-1
0.4
12
64
50
Spoch
Spech
Epoc
Helmun S000 DAit
0.5
0.0
10
0--
0.4
8.A-
18.h
00-
4023
2000
1002
4000
2000
Eooch
tpoch
Epcch
Figure 17: Epoch- wise average metrics curves for the SW dataset with the probit model, comparing fixed- and leamsble-thresholdd models.
Fixee Thain
Fixed
Learable Train
Learnable
10
3.0
357
MAC (octert
0.-
Unoo3
0.6
02-
0.6
0.
Epoch
2000
Epech
4000
2000
Epacc
2008
Epcch
4020
2000
Epoch
4000
1.0
1.0
1.0
CMC
3.60
ONCs
2006
4030
2085
Epech
Figure 18: Epoch-wise average metrics curves for the CA dataset with the probit model, comparing fixed- and tearsoll-thresholdd models.
12 30"
22"
30
0.3
19
NAE tontert
Withe-
0.2
11
2008
Spoch
2000
2006
4000
Homum Scnoitiit
0.0
3.0
0.0
ONC.
02
0.2
02-
13"
2091
8393
2000
Epoch
Spach
Spoch
Epach
Figure 19: Epoch- wise average metrics curves the WR dataset with the probit model, comparing fixed- and teamsallethresholld models.
29

---

Sr. wraple
Latent Thrusholds (Epech 6)
Letent Thresholds (Epoch 10)
Latent Thresholds (Cpoch
Latent Thresholds (Eooch 5000)
Clea
Daos
Choce
Learrable
Viribble
Penture Space (tpoch
Peature Space (tpoch 10)
Feature Space (taech 50)
Peature Space (tpoch 5000)
2
2
seieed to drrk
121
Figure 20: Visualization of the latent and feature spaces for the ER dataset using the probit model. comparing fixed- and learnabee-threshol models.
Om03
Threshelds (tpech
Threshelds (froch
Thresholds (tpech
Tireshuld (tpock
byo0
Libert co
Lntant huit
Aisi
14
15
Latent Thresholds (Epach 6)
Lutent Thresholds (tpech 10)
Latent Thresholds (tpech
Lutest Threahotds (tpech 5000)
Learnable 031
Laterd
Varadie
Labee Adt
Labert
Peature Space (Spech
Feature Space (laech 16)
Festure Space (Spech 50)
Feature Space (Epoch 5000)
#
803
Figure 21: Visualization of the latent and feature spaces for the LE dataset using the probit model, comparing fixed- and learnable-threshold models.
30

---

Gaiz
Thresholds (tooch
Thresholds (tpech
Lebent Thresholds (tanch
Thresholds (tpech
Liter Varabi hy00
Libert bybt
Variible hylal
Adi
22x
-1.
Labent Thresholds (Epoch
Labent Threshelds (Epech 10)
Labent Thresholds (tpech
Labent Threahalde (Epech 3000)
Learnable 0h0
Levet Variate by00
Litert Variable bydd
Latent Vartobbe ha/r
Labert h//0
Festura Space (Spoch 0)
Festure Space (Epoch 16)
Feature Space (Epoch 50)
Festure Space (Epoch 5000)
PC1
PC
PC1
PC1
Figure 22: Visualization of the latent and feature spaces for the SW dataset using the probit model, comparing fixed- and leamable-threshold models.
CB557
DM5c + Casceo
(Spoch
Fixed
Latest va Thresholda Itpech
Latent va Thresholds (tpoch 10)
Latent Threaholds (Epoch 501
Letent va Thresholds (Epoch 5000)
Leamable
Latne verile N/2
Lateet Veridan Bu35
Lower ver stw. Auis
(ateet Vwitele Bux
Feature Space (Rloch 6)
Peature Space Hlpoch 10)
Feature Space (tecch 96)
Feature Space (lpech 5000)
PCL
K23
PC3
Figure 23: Visualization of the latent and feature spaces for the CA dataset using the probit model, comparing fixed- and learnable-tresholdd models.
31

---

Caii
Caes
Cla
Gae
Cass
Dao Pas
Caudter Tradhees
Latent us Threcholde (Epoch 0)
Latent ve Thresholds (Epoch 18)
Latest vs Threcholds (Epech 50)
Latene Taracholde (Spach 5000)
Peature (Epech
Peature (Epoch 10)
Peature Space (Epoch
Peature Space (Epech 5000)
05
1.5
3.0
05
Latent Thresholds (Epach
Labent Thresholds (Spach 10)
Latent vs Threshalds (Gaech
Latent Threshalds (Epach 5000)
Feature Space (Epach
Feature Space (Epach
Featura Space (Gpoch
Feature (Epach 5000)
* 2 --
2.8
05
00
35
Figure 24: Visualization of the latent and feature spaces for the WR dataset using the probit model, comparing fixed- and learnable-threshold models.
32

---

D.3 Results with the clog-log model
This section presents the experimental outcomes obtained using the Gumbel CDF (i.e., g(x)
1 1-e )) which corresponds to the clog-log model.
Unlike the symmetric link functions (logit and probit), the clog-log link is asymmetric. The solution to
(88) thus no longer places optimal latent variables at the simple midpoint between adjacent thresholds, but shifts them by an intrinsic offset.
As the examined fixed threshold cases so far, we assume the uniformly spaced fixed thresholds b 1/ + qár where A (b@) /QQ Q. In this case, we have the optimal solution as
q + b
(58)
2
for a constant offset o.. To verify this, we substitute the above expression into 18 which requires g'(b@e 2g) = g' (6 2q)) Computing the arguments gives:
lix
Alik
A fix
bix q
Zig
61
4
0,
(59)
2
and
bax + 1
A
fi
q-1
2n
blx
9-1
(60)
2
2
Substituting these into (88 we obtain:
A g'
2 = -0)
A
2 20)
(61)
Since this equation depends only on A5* and not on q. the same offset a applics uniformly to all class intervals. In practice, a can be computed as a = यं (bf b/xs// once zं is determined from (18)
Accordingly, for the uniformly spaced fixed thresholds, we introduce modified ONC3 metric that accounts for the asymmetry of the link function:
Bfx 8/dcal (
ON ONC3.clog-log
9
(62)
Q
b6x)
-=
where the ideal threshold position gideal is detined as
q
ideal
2g + 1 0.
(63)
2
Note:
To derive this expression, recall that for uniformly spaced fixed thresholds, the optimal latent
variables satisfy 2i
(biix + 6üx 1)/2 + and 2 (bii + bbix) + a. Adding these two
equations yields:
büx 6
flx
+ blx
blx + 26 + blx
+
2a
2a.
(64)
2
For uniformly spaced thresholds with 6lix Substituting this gives:
blix
fix
flx
lix we have 60in + 6
26fxx
9
41,0x
2g
29+
2
20
26 + 22.
(65)
Rearranging, we obtain:
6ix
29
1
5.
(66)
2
This establishes the form of the ideal threshold position.
In the above analysis, the assumption that fixed thresholds are evenly spaced, but this assumption does not hold for learnable thresholds. Even for symmetric link functions such as the logit model, the
33

---

same limitation applies; however, in the main text, we have presented the results without resolving this issue. The rationale for this choice is discussed Section
Fortunately, for symmetric cases, the error caused by this simplification is not severe, and therefore the conclusions in the main text remain valid. In contrast, for asymmetric link functions, the error introduced by this issue is expected to be more significant. Thus, for the general case-including learnable thresholds-it is necessary to appropriately extend ONC3 and define a new quantity that can properly quantify the phenomenon. However, doing so would make systematic comparison with the existing experiments in the main text difficult, as would create a mismatch from the experimental setup in the main text. Therefore, for the clog-log model, we only quantify ONC3 for the thed-lh case and omit the ONC3 quantification for the jearnable-threshnt case.
Figures 25 29 show the evolution of evaluation-metric curves for the datasets ER, LE, SW, CA, and WR, respectively. Due to the limitation for learnable thresholds, only present ONC log curves for the fixed threshold model. Figures 30 34 show visualization of the latent and feature spaces for the datasets ER, LE, SW. CA, and WR, respectively. These results align with the ones in the case of the symmetric link functions, suggesting the wide applicability of ONC.
Train
Fiod
Learnalee Train
Learnable
1.0
2.0
PVAE (onter)
0.5
0.6
2-10
0:2
10
0.2
2000
2903
Spech
4000
2003
2000
03.0
1.0
Nosmam Seneeeeeee
0.4
0.0
CMC
0.2
0.21
3071
0.2
2000
Spoch
4002
2003
Epoch
4000
Figure 25: Epoch-wise average metrics curves for the ER dataset with the clog-log model, comparing fixed- and leamsble-threshold models.
Flxdd Train
Fixed Val
Learnable Train
Learnable Val
2'100
0.5
MA2 (occert
0.4
0.4
1.7
2003
Spoch
4000
2000
Epoch
4303
Epoch
0.7
2003
Epoch
4000
2000
Epoch
4000
1.2
3.0
Hesmum Sernaturty
0.2
0.6
10'
0.18
CMC
201
0.3
00
0.2
2003 Spach
6000
2000
4003
3003
Spocc
4000
Epoch
Figure 26: Epoch-wise average metrics curves for the LE dataset with the clog-log model, comparing fixed- and haamable-threshee models.
34

---

Feed Tain
Food val
Learnable Train
Leamaale Val
2x107
1-0
0.3
0.8
MWC (onceet
WHBmm ACC
100
0.6
30
3071
10:11
0.2
Epoch
Epech
Epoch
0.8
0.8
0.8
04
1071
7000
5000
Epozh
Epoch
Epoch
Figure 27: Epoch- Epsch-wise average metries curves for the SW dataset with the clog-log model, comparing fixed- and learnale-thresholdd models.
Flood Train
Flxed Val
Learnalic Train
Leamable Val
10"1
MAC (sccent
30"1
30
0.4
0.4
0.5-
20F1
2006
4000
2000
Epoch
4303
tpoch
2003
Epech
4000
2000
Epoch
4300
1.2
3.0
Memmm Serattiry
0.8
DMC
0.44
0.2
0.
0.2
2003 Epoch
4000
2000
1003
3003
Epecch
4000
Epoch
Figure 28: Epoch-wire wise average metrics curves for the CA dataset with the clog-log model, comparing fixed- and sormable-threshlle models.
1.0
3.0
1.0
10"1
0.8
10"7
3 18"x22
0.6
2014
0.8
0.6
3.4
30
02
tooch
4202
2000
2001
Epech
dice
2003
2000
02.00
3.0
1.2
0.1
6.5
Weimum
ONC
02-
0.2
0.2
0.7
2000
2003
4000
Epoch
Bzach
Figure 29: Epoch-wise wise average metrics curves for the WR dataset with the clog-log model, comparing fixed- and learnable-tee threshold lea models.
35

---

10
-25
Latent Thrusholds (Epech 6)
Letent Thresholds (Cpach
Letent Thresholds (fasch
Labent Thresholds (Epoch 5000)
Peature Space (Eooch
Peature Space (Epoch 10)
Peature Space (tpoch
Feature Space (Epochh 3000)
0
Stk
+
Figure 30: Visualization of the latent and feature spaces for the ER dataset using the clog-log model, comparing fixed- and learnable-threshold models.
Om
Om03
Gradter
Trudetis
Threchelds (Spach
Thrasholds (Coach 10)
Thrasholds (Gpach
Thresholds (Epach
Looel Variate aom Nv00
Lotel uu 1u00
Lotel Varitte Jum Nu00
Loter a. 11x0
(fpoch
Space
Latent Thresholds (tpech
Latent Thresholds (Booch 10)
Latent Thresholds (tpoch 50)
Latent Thresholds (Epach 3000)
Learnable 050
Liber Variate N/l0
Libert Variaie h/X0
Liberd Variade zow hvll
Libert Variable hyX
Feature Space (Epoch
Festure Space (Epach 10)
Festure Space (Epoch 58)
Feeture Space (tpech 5000)
g
+
901
K31
Figure 31: Visualization of the latent and feature spaces for the LE dataset using the clog-log model, comparing fixed- and learhabl-thresholld models.
36

---

Feature (Epoch
Feature (Spach
Feature (Epach
Feature (Spach
Latent vs Thresholds (Epoch D)
Lotent vs Thresholds (Gooch 10)
Latent vs Thresholds (Easch 50)
Latent Thresholds (Epoch 5000)
Learnable
Lotert Vardtre 2ww hul0
Laten Veridde TbX
Latent Verebte com (adel
Leiot Verieble baa)
Peature Space Itpoch 01
Peature Space (Cooch 18)
Peoture Space (Epoch 50)
Festure Spece (Epoch 5000)
*
PC1
Figure 32: Visualization of the latent and feature spaces for the SW dataset using the clog-log model, comparing fixed- and learnable-hhesholld models.
Casl
Okn
CznFaer
Clans
-1
-658
648
0.5
150
075
-10
-05
Lo
Lebeet Thresholds (Spoch 6)
Latent va Thresholds (tpech 10)
Latent va Thresholds (Epech 50)
Latest va Threahoids (tpech 5000)
Chass
Later veriit 1120
Latern Verietle bub0
Later Varatee- Till
Lateet vwristeee Tu2e1
Peature Space (Raoch
Feature Space (Epoch 10)
Peature Space (Spech 50)
Peatuure Space (Epoch 5000)
2
PC3
PC3
Figure 33: Visualization of the latent and feature spaces for the CA dataset using the clog-log model, comparing fixed- and leamaale-threshold models.
37

---

(Epoch 58)
Threthelds (oach 5000)
-3
Fixed
1.0
2.5
Lebentt vo: Thresholds (tpoch 0)
Latent vo Thresholda Itooch 101
Latent vo Thresholde (Epech 50)
Latent vo Tiresholds (Epech 500g)
Dase
Laar Wardd: [..al
Liti Vardic T1/
Littl Va Tcinl
Laton Vusidi ThCA)
Pesture Space (tpoch 0)
Peature Space (Eooch 10)
Footure Space (tpech 50)
Posture Space (tpech 5000)
0
80
Figure 34: Visualization of the latent and feature spaces for the WR dataset using the clog-log model, comparing fixed- and learnable-thresholdd models.
38

---

D.4 Results on FTKeaae dataset with the ResNet50 and DenseNet201 backbones
This section presents the experimental outcomes for the UTKFace dataset using the logit model. Figures and [6 show the evolution of cvaluation-metriy curves for the ResNet50 and DenseNet201 backbones, respectively. Figures 37 and 38 [38sshw visualization the latent and feature spaces for the ResNet50 and NenseNer201 backbones, respectively (one random seed per backbone). 'These results are essentially consistent with those obtained using ResNet101 in the main text, suggesting that the effectiveness of ONC and fixed thresholds broadly holds even for complex backhone networks used practice.
Fixed
Fixed
Learnable Trein
Learnable
3.2
1.0
soae: (onber))
A0C
0.2
0.6
2
Witthm
0.2
0.2
320
Epoch
Edoch
Epoch
10
10'
0.2
Netmum
19:
301
02-
200
310
10:
200
300
Fooch
Epoch
Epech
Figure 35: Epsch-wise average metrics curves for the UTKFace dataset with the ResNet50 backbone.
Fixed train
--- Fixed Vol
Learnable train
Learnable Val
MARE curdery
ACC
3
0.38
30n4
0.3
0.0
P52
100
300
300
100
200
300
100
200
300
spoch
tpoch
Spach
1.2
1/7
1.0
1.2
No WommmmSerette Seretteyty
6.5-
66
0.E.
ONC)
10"
022-
20'1
62-
0.-
0.2
220
-00
203
Eaocc
Epech
Epoch
Figure 36: Epoch-wise average metrics curves for the UTKFace dataset with the DenseNet201 backbone.
39

---

CL.E4
O 23
05:8544
10:0005
C13 80
ELS 10.66
E2X 5336
Can Mon
Threshaids (Spech
Threthalds (Spoch 10)
Labent Thracholde (Epach 1000
Latent Thrachalds (Spach 260)
Lewer
Verialle
Uniiall
Feature Space (lppcch
Feature Space (Epech 10)
Festure Space llipech 160)
Feature Space (Spech 200)
de desddeooeeeiiis doda
-50
Thresholds (Epoch
(taech
Thresholds (tpech 1009
Thresholds (Epoch 500)
Learnable
-30
-20
-10
-43
+2
-33
30
30
200 330
200
123
200
300
Figure 37: Visualization of the latent and feature spaces for the UTKFaae dataset with the ResNet50 backbone.
50
Go 1335
(6-25-3
2U0 4-4p
052-5539
CA 55:0
CU6 15.79
CSR 35-39
E3X 85-16 - Curr
vc Thrashalde (Spach
Latest Threcholds (Egach
Latent un Threcholds (Spach 180)
Lataot Thrnchalds (Eporh 2001
Lawel Vovinde Buu
Space
100
10
Latent Threaholds (Epoch
Latent Thresholds (Gaech 101
Labent Threshelds (Epach 100)
Latent Threaholds (Epach 2001
Learnable
Lafert Variable Ausi
Variiii
Verialle
Levet Variit
Festure Space (Epech 6)
Festure Space (Epoch 18)
Festure Space (Eaech 100)
Feature Space (Epoch 200)
-3
-35
FC3
PC1
PC1
P01
Figure 38: Visualization of the latent and feature spaces for the UTKFace dataset with the DenseNet201 backbone.
40

---

D.5 Comparison of theoretical and experimental results
We here compare theoretical predictions from EOS T6) with experimental results on the LE dataset (Q = 5 classes) as representative example, assuming the logit model. For the experiment, we use the same network architecture and the dataset splits as the main text, but the training objective slightly changed to the following one:
A 2N
Aw
Liotal LNLL
Σ
(
1| Ao1101
(67)
2
where CNLL denotes the negative log- ligeliiklhoodd. and A x 10-3 and X₁ 0.01 are the regularization coefficients for the backhone network parameter and the feature vector he (output of the backbone network), respectively: the latter regularization term was absent the experiments in the other sections but is here introduced to make direct quantitative comparison with our theory.
We trained the whole network under the fixed thresholds set as b (-20,- 12, 4, 4, 12, 20) for different 35 values of X E [0, 10*].
The results of the experiment (datapoints with error bars) given in Fig. 39
The left panel
LE data, b= (22202222224444112220
LE data, b=( 0=2-200-122 4..1222
15
2.5
5
1
1.5
-5
1
-10
0.5
-15. 669
10
10
10
10
100
Figure 39: Theory (solid curves from EOS) versus experiment (datapoints, 30-holdout average with error bars) for the LE dataset. Left: |w=|| versus >y. Right: 2g versus X,
shows w while the right one exhibits z". In the plots, the theoretical predictions (solid curves) are simultaneously shown: they are computed from EOS (16) with the above parameter values (b, As). the dataset size ratios a (0.094, 0.281, 0.472, 0.231, 0.031) of the LE dataset, and the respective Ay value. The right panel shows good agreement for x" in the small-X, Aw region where the relation 27 - (by + /2 holds, but the left panel exhibits a huge gap in w' |wo" || the DNN exhibits a saturating behavior at small X while the theory predicts the divergence, implying that our theory quantitatively not accurate. On the other hand, both the theoretical and experimental results show similar qualitative dependence on Ag: they vary monotonically with X and in both cases, phase transitions occur certain specific values of A though the locations of the transition points are quantitatively different. We confirmed that similar behavior is also observed for the other four tabular datasets.
In summary. although the theory developed in this study does not quantitatively predict the behavior of actual DNNs with full accuracy, it can qualitatively explain the observed phenomena.
41

---

NeurIPS Paper Checklist
1. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We have provided a paragraph in the Discussion section to discuss the limita- tions of the theoretical results, including the assumption of the fixed threshold, as well as that on the number of phases in the (Aa, ) Additionally, we acknowledge that the current definition of ONC3 only valid when the thresholds are fixed and evenly spaced in the latent space, which is the case for our fixed threshold experiments but is not for the others. We note that developing generally applicable ONC3 metrics is possible but the corresponding experiment is left for future work.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
2. Theory assumptions and proof's
Question: For each thcoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: We have provided all the assumptions for the theorems in the main text
(Theorems 4.1
2 and 4.3). as well as those in the appendices (Theorems A. A.2
A.3
and Proposition
B. We have also provided complete proofs to them except Theorems A.
and A.2 (which not original contribution), and do believe that they are all correct.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short. proof sketch provide intuition.
42

---

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
3. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Section and Appendix C provide all the information necessary for repro- ducibility. We provide the URL for downloading the datasets in Appendix [ along with detailed information for tabular datasets (see T and UTKFace dataset (see Table 2) Figure 6 shows the specific structure of the neural network for tabular datasets, and the backbone architectures for UTKFaace are detailed in the text. Other settings, including the number of epochs, learning rate (see Tables 3 and 4. learning rate scheduling strategy. regularization coefficients, batch size, etc. also clearly listed in Appendix C Loss functions are defined in Section and evaluation metrics are defined in Section S The link functions, thresholding strategies, and exed-threshol ranges are also detailed in Tables 3 and Finally, the supplementary material contains the complete source code, enabling exact reproduction of all the experiments.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
4. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
43

---

Answer: [Yes]
Justification: Our study used publicly available datasets. For the tabular datasets, the URL for downloading is shown in Appendix C The UTKFaae dataset is also publicly available. We have included the complete experimental code for the tabular datasets with logit and probit models in the supplementary material, along with the necessary instructions and environment setup for reproduction.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm cc/
iuplic/guides http://uuuddee/eeeeubbieeeeee
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to rum to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips.cc/publl http:ccccppbbclccccc heciiiccgiies///e httpa/////gu.ee///eeeuubbbsb////
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
5. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Section and Appendix C provide detailed experimental settings. Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
6. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: For the tabular datasets, publicly available datasets following 30-holdout were used; every experiment was conducted on all the training-validation hold-out splits and the results were averaged, while the metric curves included the corresponding error bands. For UTKFace, each configuration was repeated with three different random seeds, and the results were averaged with error bands shown in the metric curves.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
44

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
7. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Appendix lists the computational resources for each experiment, including the GPU type, the GPU memory and runtime of each run, and other relevant details.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't itdn't make it into the paper).
8. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http::///oerrpp..ccpbbbii//iiii hwtpppuwwww.eccecciiieeeeeeeeeee
Answer: [Yes]
Justification: We did not do any experiments involving human subjects. We have confirmed the copyright information of all the datasets used in our experiments, and mentioned it in Appendix C This paper is of theoretical nature, that there should be societal impact.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations their jurisdiction).
9. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer:
[NA]
Justification: This paper is of theoretical nature, that there should be societal impact.
Guidelines:
45

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e-g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
10. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper is of theoretical nature, so that there should be no such risks of misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
11. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We have properly credited the creators of all the datasets used in our ex- periments in Appendix C All the license information has been explicitly mentioned in Appendix Cl as well.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
46

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thoode com comddatasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
12. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The code used in the experiments is uploaded in the supplementary material with scripts, the license, and other related files.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
13. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We did not do any experiments involving human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
14. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: We did not do any experiments involving human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
47

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
15. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core methodology and experimental pipeline of this study do not employ any LLMs. LLMs were used exclusively for manuscript editing, Al search, and routine debugging assistance during code development; hence, they do not constitute an important, original, or non-standard component of the research.
Guidelines:
The answer NA means that the core method development in this research does not involve L.MMs as any important, original, non non-standard components.
Please refer to our LLM policy (https: /ttpa/////euip....ee http///uipppsccooofoeeeeeeenneee co/ooaffeencccs/222555 for what should or should not be described.
48


Reviews:
[{"review_id": "uYOlF61ujg", "reviewer": "Reviewer_ZsuL", "summary": "This paper studies the phenomenon of neural collapse in ordinal regression models. The authors propose a set of conditions for ordinal neural collapse, and through the unconstrained features model show that the global minima satisfy these conditions. The authors also empirically validate their findings on ordinal regression datasets.", "questions": "1. What is the role of weight decay in the analysis of the UFM model? Is weight decay necessary?\n\n2. Comparing the emergence of NC in OR vs standard classification, is there a reason to prefer one setup over another? Can the UFM analysis provide an answer for this?\n\n3. The analysis seems to exclude the case where the thresholds are learned. Can the UFM analysis prescribe a proper choice of thresholds?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: This paper has both analytical and empirical characterizations of neural collapse in a new setting. The mathematical results seem correct (I have not thoroughly checked the derivation of ONC3), and the empirical validation in small label settings seems valid.\n\nWeaknesses:\n\n1. The mathematical results do not seem to provide prescriptions for training methods, and only analyze the landscape of the unconstrained features model. This is the case for all UFM analyses however, not a particular problem with this paper.\n\n2. The empirical validation is performed on relatively simple datasets with a small number of classes. Ordinal regression provides an advantage over regular classification in situations with extremely large label spaces. How does NC emerge in that case?", "was_revised": true, "final_justification": "This paper is a solid contribution to the literature on Neural collapse - indicating that NC is a more fundamental phenomenon in training neural networks across a range of tasks and loss functions. While I have read the rebuttal and appreciate the addition of experiments on UTKFace, my main concerns - regarding limited practical insights, and not having demonstrations with a large number of classes - still remain. I will keep my score which indicates that on balance I believe the paper should be accepted.", "rebuttal": "We are grateful for your constructive criticism, which provides an opportunity to clarify our contribution.\n\n**[W1] Limited prescriptions for training methods** \n\n**[A1]** Indeed, our theoretical analysis does not directly address training methods. However, UFM-based analyses can inspire practical training methods through their insightful analytical results. In fact, several prior studies based on UFM have proposed some practical prescriptions, such as the use of a fixed ETF classifier to counter class imbalance or the introduction of new loss functions to accelerate convergence; associated experiments have demonstrated that such prescriptions are actually effective. Our paper also leads to some operational prescriptions: one such example is the use of fixed thresholds to preserve ONC3 and stabilize training, which, moreover, is likely to alleviate the common issue of degraded classification performance for minority classes in the presence of class imbalance, as argued in the Discussion section of the manuscript. When submitting this manuscript, this last benefit in the imbalance case was just an expectation, but inspired by the comments of reviewers including you, we have conducted additional experiments using a more challenging task (UTKFace age-estimation dataset which has a strong class imbalance)  and found that the fixed thresholds indeed improve the minority class accuracy. This is, in our view, a practical outcome naturally deduced from our theoretical arguments.  \n\nAdmittedly, UFM introduces a strong simplifying assumption at the beginning. But thanks to that simplification, the derived results are often transparent. They thus can reveal what the model is doing and where to intervene, which potentially yields concrete design principles for training DNNs. This is precisely one of the key advantages of the UFM framework and is something difficult to achieve with other approaches.\n\n**[W2] Limited label scale of experimental validation** \n\n**[A2]** We admit that our experiments only treated the cases with a small number of classes. To address this point, as also explained in [A1], we have conducted additional experiments using the UTKFace age-estimation dataset binned into 20 classes; three additional model architectures have also been treated in the experiments to assess the robustness of the results. For more details of the experimental setup, please see our response [A1] for Reviewer 7zzE. The result clearly shows ONC in such a case with a large number of classes, supporting the practicality of our finding.\n\n**[Q1] The necessity of weight decay in UFM** \n\n**[A3]** Definitely necessary. In ONC2, the feature vector components orthogonal to the classifier weight collapse to the zero vector thanks to the regularization on the feature $\\boldsymbol{h}$​. The same is true for ONC1: without the regularization, it is not possible to show the strong convexity of the total loss function, and hence the uniqueness of the solution leading to ONC1 is not shown. As for the regularization on the weight vector $\\boldsymbol{w}$, it is necessary since otherwise $\\boldsymbol{w}$ diverges, as observed in the right panel of Figure 1. Therefore, to ensure that the problem is mathematically well-defined, those regularizations are necessary.\n\n**[Q2] OR vs standard classification comparison from NC perspective** \n\n**[A4]** We did not consider this perspective so far. Thank you for your insightful comment. Indeed, since OR tasks can be approached as classification problems, a comparison of the two approaches could be feasible. We have examined this point from a theoretical viewpoint, but we could not draw any conclusion regarding which approach is better in this short time. A detailed investigation and comparative experiments would be necessary, and thus we would like to leave this as a direction for future work.\n\n**[Q3] UFM analysis insights for choice of thresholds**\n\n**[A5]** We believe that this is feasible. For example, in the class-balanced case, it is natural to assign equal probabilities to each class. Once the inverse link function $g$ and the number of classes $Q$ are fixed, the appropriate placement of thresholds $\\boldsymbol{b}$ is uniquely determined. However, in such cases, we have $b_0 = -\\infty$ and $b_Q = +\\infty$, which implies that in the context of ONC, the optimal latent values $z_1^{\\ast}$ and $z_Q^{\\ast}$  diverge in the weak regularization limit. This divergence could make deep-learning-based experiments less aligned with the theoretical predictions, and therefore, we fix those edge threshold values at large enough values in the experiments, to make the ignored tail probabilities small enough while keeping the alignment with the theory. We believe that this is one of the best practices in the class-balanced setting in that the result is theoretically well-controlled, but still, the classification performance is optimal. \n\nIn the case of class imbalance, one can also assign probability values to each class based on some criterion (e.g., setting (the probability) $\\times$ (the imbalance ratio) becomes constant), and then determine the corresponding appropriate values for $\\boldsymbol{b}$. The corresponding ideal latent values $\\boldsymbol{z}^*$ can be computed using the UFM framework. While we already discussed this point in the current manuscript’s Discussion section, we will make an effort to clarify it further in the revised version."}, {"review_id": "gSmxvFBYlT", "reviewer": "Reviewer_zTjz", "summary": "This work extends Neural Collapse (NC) to ordinal regression (OR) tasks, which differ subtly from both classification and regression. Building on the Unconstrained Feature Model (UFM) and the Cumulative Link Model (CLM), the authors provide a theoretical characterization of a collapse phenomenon under $\\ell_2$ regularization applied to both classifier weights and “free” hidden features. The theoretical results can be decomposed into two sets: the first set ONC1 and ONC2 mirror classical NC behavior, showing within-class feature collapse and alignment of class means with the classifier direction, the other set is more task-specific, where the authors demonstrate that with vanishing regularization strength, the collapsed latent values respect the class order, and under zero regularization, they match threshold midpoints. Empirical results validate these findings and suggest that the collapse structure persists even when thresholds are learned rather than fixed.", "questions": "Please see the weakness section.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n* The paper is overall well-written, with clear technical exposition and smooth narrative flow. The related work section on NC is comprehensive and positions the paper well within the broader literature.\n* As far as I can tell, the extension of the NC framework to OR tasks appears to be novel and well-motivated.\n* The theoretical analysis and empirical validation are tightly aligned. In particular, the behavior under vanishing regularization is quite interesting and elegant.\n\nWeaknesses:\n* The paper falls somewhat short on practical insights.. The authors discuss that the fixed threshold values \"yield faster and more stable convergence compared with learnable thresholds\" and \"These make ONC a potentially effective concept for improving practical performance in generic OR tasks.\" but not pursue this direction more clearly. For example, Figure 2 shows similar MAE for fixed and learnable thresholds, but notably better $\\ell_{NLL}$. Could this translate to improved generalization or calibration in more challenging settings?\n* In Figure 2 and its associated discussion, the authors mention that both training and validation MAE approach zero. However, the curves appear to be nearly identical throughout training, which raises the question of whether the chosen tasks are too easy. Would it be possible to test on more comprehensive tasks to broaden the applicability of the results?", "was_revised": true, "final_justification": "After reading the reviews and responses from both the authors and other reviewers, I noticed that a common concern was the simplicity of the experiments presented in the main paper. The authors have addressed this point in their response, including additional results, which is great.\n\nI will maintain my initial recommendation of borderline accept, as I find the task to be novel and the paper overall interesting, but the practical implications are not yet entirely clear.", "rebuttal": "Thank you for your valuable and constructive feedback.\n\n**[W1] Limited Practical Insights**\n\n**[A1]** The purpose of this paper is to investigate the existence of NC-like phenomenon in OR tasks and to provide its characterization. Hence, providing practical utility is indeed not the primary purpose of this paper. We hope for your understanding in this regard. However, our results also provide practical insights, such as suggesting appropriate ranges for regularization parameters, indicating the potential for designing new loss functions based on UFM analysis, and highlighting the usefulness of fixed thresholds. While we have already addressed these points in the Discussion section, we will revise the text to make these contributions more explicit.\n\n**[W2] Concerns About Task Simplicity**\n\n**[A2]** Similar comments are also raised by Reviewers 7zzE and wf6P, and we agree with this point. To address this weakness, we have conducted additional experiments using the UTKFace age-estimation dataset with 20 ordinal classes (5-year intervals, 0–4 yrs up to 95–116 yrs), which exhibits a strong class imbalance. For treating this complicated task, we have used more powerful backbone architectures (ResNet-50, ResNet-101, and DenseNet-201). All architectures utilize ImageNet pretrained weights and are fine-tuned with the Adam optimizer (lr=0.001, weight_decay=0.001). We employ the logit link function with both fixed thresholds (uniformly spaced from -40 to 40) and learnable thresholds. Tables 1 and 2 present the summarized results. Across all architectures, the four ONC metrics demonstrate significant reductions, confirming successful ONC. This consistent emergence of collapse indicates that the ONC phenomenon represents a fundamental characteristic of over-parameterized neural networks. As a noteworthy outcome, Table 1 reveals that learnable thresholds cause the model to neglect minority classes. Conversely, fixed thresholds mitigate this problem (as evidenced by the \"min sensitivity\" metric). This is expected and already argued in the Discussion section of the original manuscript, and the expectation has now been experimentally confirmed, providing one reasonable response to your comment regarding the weakness of our paper ([W1]). We will report this finding in the revised manuscript. \n\n**Table 1: OR Metrics After Training**\n| Backbone | Threshold | Data Split | loss ↓ | acc ↑ | mae_order ↓ | mae_age ↓ | qwk ↑ | min_sensitivity ↑ | within_1_acc ↑ |\n|----------|-----------|------------|--------|-------|-------------|-----------|-------|-------------------|-----------------|\n| ResNet101 | fixed | train | **0.3451** | **0.9908** | 0.0261 | 0.1304 | 0.9940 | **0.9333** | 0.9969 |\n| ResNet101 | fixed | val | **0.8358** | **0.8683** | **0.2035** | **1.0174** | 0.9834 | **0.7500** | **0.9599** |\n| ResNet101 | learnable | train | 1.0741 | 0.9523 | **0.0250** | **0.1249** | **0.9988** | 0.0000 | **0.9980** |\n| ResNet101 | learnable | val | 1.3540 | 0.5996 | 0.2066 | 1.0332 | **0.9869** | 0.0000 | 0.9573 |\n| ResNet50 | fixed | train | **0.3312** | **0.9909** | 0.0264 | 0.1318 | 0.9931 | **0.9545** | 0.9972 |\n| ResNet50 | fixed | val | **0.8450** | **0.8722** | 0.2061 | 1.0306 | 0.9819 | **0.7500** | 0.9526 |\n| ResNet50 | learnable | train | 0.5326 | 0.9879 | **0.0177** | **0.0887** | **0.9987** | 0.0000 | **0.9983** |\n| ResNet50 | learnable | val | 0.9335 | 0.7837 | **0.1924** | **0.9620** | **0.9883** | 0.0000 | **0.9573** |\n| DenseNet201 | fixed | train | **0.4172** | 0.9454 | 0.0177 | 0.0885 | 0.9964 | **0.9500** | 0.9982 |\n| DenseNet201 | fixed | val | **0.7893** | **0.8776** | **0.1835** | **0.9174** | **0.9877** | **0.7866** | **0.9598** |\n| DenseNet201 | learnable | train | 0.7825 | **0.9884** | **0.0141** | **0.0706** | **0.9993** | 0.0000 | **0.9984** |\n| DenseNet201 | learnable | val | 1.0224 | 0.8042 | 0.1908 | 0.9538 | 0.9866 | 0.0000 | 0.9538 |\n\n**Table 2: ONC Metrics Evolution (Initialization → Post-training)**\n| Backbone | Threshold | Data Split | onc1 ↓ | onc2_1 ↓ | onc2_2 ↓ | onc3 ↓ |\n|----------|-----------|------------|--------|----------|----------|--------|\n| ResNet101 | fixed | train | 8.91×10⁻¹ → 6.27×10⁻² | 4.82×10⁻¹ → **1.21×10⁻⁸** | 9.65×10⁻¹ → **2.62×10⁻⁴** | 4.71×10⁰ → **9.24×10⁻²** |\n| ResNet101 | fixed | val | 8.98×10⁻¹ → 1.54×10⁻¹ | 5.11×10⁻¹ → **1.23×10⁻⁸** | 9.80×10⁻¹ → **2.62×10⁻⁴** | 4.72×10⁰ → **2.40×10⁻¹** |\n| ResNet101 | learnable | train | 8.92×10⁻¹ → **5.80×10⁻²** | 4.84×10⁻¹ → 7.33×10⁻⁵ | 9.65×10⁻¹ → 5.01×10⁻³ | 6.28×10⁰ → 2.53×10⁻¹ |\n| ResNet101 | learnable | val | 8.99×10⁻¹ → **1.49×10⁻¹** | 5.12×10⁻¹ → 7.21×10⁻⁵ | 9.81×10⁻¹ → 5.03×10⁻³ | 6.29×10⁰ → 3.16×10⁻¹ |\n| ResNet50 | fixed | train | 8.90×10⁻¹ → 7.65×10⁻² | 4.82×10⁻¹ → **3.12×10⁻⁷** | 9.64×10⁻¹ → **5.52×10⁻⁴** | 4.71×10⁰ → **8.27×10⁻²** |\n| ResNet50 | fixed | val | 8.97×10⁻¹ → **1.56×10⁻¹** | 5.10×10⁻¹ → **2.84×10⁻⁷** | 9.80×10⁻¹ → **5.51×10⁻⁴** | 4.72×10⁰ → **2.23×10⁻¹** |\n| ResNet50 | learnable | train | 8.92×10⁻¹ → **6.72×10⁻²** | 4.83×10⁻¹ → 5.91×10⁻⁴ | 9.65×10⁻¹ → 4.56×10⁻² | 6.28×10⁰ → 3.27×10⁻¹ |\n| ResNet50 | learnable | val | 8.98×10⁻¹ → 1.57×10⁻¹ | 5.12×10⁻¹ → 5.65×10⁻⁴ | 9.81×10⁻¹ → 4.55×10⁻² | 6.29×10⁰ → 4.27×10⁻¹ |\n| DenseNet201 | fixed | train | 8.93×10⁻¹ → 5.64×10⁻² | 4.84×10⁻¹ → **1.08×10⁻⁵** | 9.66×10⁻¹ → **2.66×10⁻³** | 4.72×10⁰ → **8.04×10⁻²** |\n| DenseNet201 | fixed | val | 9.00×10⁻¹ → **1.27×10⁻¹** | 5.13×10⁻¹ → **9.81×10⁻⁶** | 9.82×10⁻¹ → **2.67×10⁻³** | 4.73×10⁰ → **1.50×10⁻¹** |\n| DenseNet201 | learnable | train | 8.94×10⁻¹ → **4.94×10⁻²** | 4.85×10⁻¹ → 7.14×10⁻⁴ | 9.66×10⁻¹ → 1.39×10⁻¹ | 6.30×10⁰ → 4.48×10⁻¹ |\n| DenseNet201 | learnable | val | 9.01×10⁻¹ → 1.45×10⁻¹ | 5.14×10⁻¹ → 6.94×10⁻⁴ | 9.82×10⁻¹ → 1.39×10⁻¹ | 6.30×10⁰ → 5.36×10⁻¹ |"}, {"review_id": "5UByVNJucT", "reviewer": "Reviewer_7zzE", "summary": "This paper explores the phenomenon of Neural Collapse in the context of ordinal regression, while previous studies primarily focused on classification and regression tasks. The authors theoretically demonstrate that regularized empirical risk minimization (ERM) in unconstrained feature models leads to Neural Collapse even when the learning objective is ordinal regression. Specifically, they show that the class-wise mean features collapse onto a one-dimensional real line, and the ordering of these projected means corresponds to the ordinal structure of the classes. The authors validate their findings through experiments on several benchmark datasets, providing evidence that overparameterized models trained for ordinal regression display the prescribed Neural Collapse properties.", "questions": "1. How do the theoretical findings translate to practical benefits for ordinal regression tasks? Could you please demonstrate improved performance on benchmark datasets when applying insights from your neural collapse analysis?\n\n2. Could the authors please justify how their experimental design addresses the concerns raised in the review?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "This paper makes a valuable theoretical contribution by extending neural collapse analysis to ordinal regression, though its experimental validation has some limitations that reduce its potential impact. While I lean toward accepting the paper, some concerns regarding the experimental results should be addressed in the rebuttal.\n\nThe paper is well organized and clearly written. The main theoretical results are well-presented with rigorous mathematical notations. The objective of the empirical evaluations is clearly highlighted as bridging the gap between theoretical findings and practical applications. The clarity of the paper is high except for the demonstrations of the experimental results.\n\nThe theoretical results are supported by rigorous mathematical proofs. While I did not check the proofs in detail, I cannot find any major issues. The quality of the theoretical contents appears to be at a publishable standard.\n\nThe authors incorporate existing techniques for analyzing neural collapse (i.e., unconstrained feature models) to analyze neural collapse in ordinal regression. While the basic idea of the theoretical analyses combines existing techniques, this combination itself represents a novel contribution. Therefore, the present paper exhibits originality, though not at the highest level.\n\nThe theoretical analyses reveal three properties regarding neural collapse in ordinal regression. OCN1 and OCN2 are analogous to existing results in classification tasks. OCN3 and Theorem 4.3 are non-trivial findings specific to ordinal regression. Particularly significant is the suggestion regarding choices of regularization parameters that avoid trivial meaningless solutions.\n\nI have concerns about the experimental validation being conducted on a single non-common architecture designed by the authors. The generalizability of these results to widely-used architectures such as CNN, ResNet, and Transformer remains unclear. Since these common architectures are basis of many current successes in deep learning, the inability to demonstrate that the theoretical findings apply to these architectures diminishes the paper's potential impact in explaining real-world deep learning successes.\n\nThe quality of the experimental design raises some questions. The authors use a non-standard accuracy metric (mean absolute error of the value of true and predicted labels) despite their own cited reference (Gutierrez et al., 2016) recommending classification accuracy and mean absolute error of the *order* of predicted and true labels as accurate metrics for ordinal regression tasks. This raises a concern that the empirical confirmations of OCNs were done for inaccurate models, reducing the practical relevance of the theoretical results.\n\nWith respect to clarity of experimental results, the presentation in Figure 2 could be improved. The authors inconsistently use different scales for the vertical axis across the evaluated metrics—linear scale for MAE, ONC2-1, and ONC2-2, while using log scale for LNLL, ONC1, and ONC3. Using a consistent log scale for all metrics would facilitate better interpretation of the results.", "was_revised": true, "final_justification": "I lean toward accepting this paper, as it makes a valuable theoretical contribution by extending neural collapse analysis to ordinal regression. However, I would like to lower the score from the initial evaluations because there remains a concern that the empirical confirmations of OCNs were conducted on inaccurate models, reducing the practical relevance of the theoretical results.", "rebuttal": "Thank you for the thoughtful suggestions and detailed critique, which helped us improve the paper.\n\n**[W1] Generalization to standard neural network architectures**\n\n**[A1]** We sincerely thank you for this valuable feedback. To address this concern, we have conducted additional experiments. We have examined three different backbone networks (ResNet-50, ResNet-101, and DenseNet-201) and the UTKFace age-estimation dataset binned into 20 ordinal classes (5-year intervals, 0–4 yrs up to 95–116 yrs). All models are pretrained using ImageNet and the Adam optimizer (lr=0.001, weight_decay=0.001) is used for fine-tuning. The logit link function is used with fixed thresholds (uniformly spaced from -40 to 40) and learnable thresholds. The result is summarized in Table 2. All four ONC metrics show substantial decreases across architectures, indicating successful collapse during training for both training and validation datasets. The consistent emergence of collapse patterns across different architectures suggests that the ONC phenomenon is a general characteristic of over-parameterized neural networks.\n\nIf the paper is accepted, all the additional experimental results will be added to the paper's appendix. Besides the above three additional backbone networks, we also plan to supplement experimental results of transformer-based backbones. That experiment is currently underway. \n\n**[W2] Clarification on evaluation metrics**\n\n**[A2]** We thank you for pointing out the incompleteness of our evaluation metrics. We have additionally computed five metrics: classification accuracy, MAE of the *order* of predicted and true labels, QWK, minimum sensitivity, and accuracy within 1-off (mentioned by Reviewer wf6P). We will add plots of all these new evaluation metrics to help readers better judge the training and validation situations if the paper is accepted.\n\n**[W3] Improving clarity of Figure 2**\n\n**[A3]** Thank you for this suggestion. In the revision, we will consistently use a log scale in all the plots in Fig. 2.\n\n**[Q1] Practical implications of our theoretical findings**\n\n**[A4]** This is an interesting question, and it may be possible to address it through the appropriate design of thresholds or the development of new loss functions based on the UFM framework. While we have already mentioned the issue of degraded accuracy for minority classes in the Discussion section of the original manuscript, additional experiments presented in Table 1 show that with learnable thresholds, the model indeed tends to ignore the minority classes. In contrast, this issue is alleviated under fixed thresholds (see the “min sensitivity” metric). We plan to discuss this point in greater depth in the revised version.\n\n**[Q2] Justification of experimental design**\n\n**[A5]** In the previous experimental design, it was indeed unclear how relevant our findings were to more realistic scenarios. In response to your suggestion, we conducted additional experiments, which we believe have significantly strengthened our claims. We appreciate your valuable feedback.\n\n**Table 1: OR Metrics After Training**\n| Backbone | Threshold | Data Split | loss ↓ | acc ↑ | mae_order ↓ | mae_age ↓ | qwk ↑ | min_sensitivity ↑ | within_1_acc ↑ |\n|----------|-----------|------------|--------|-------|-------------|-----------|-------|-------------------|-----------------|\n| ResNet101 | fixed | train | **0.3451** | **0.9908** | 0.0261 | 0.1304 | 0.9940 | **0.9333** | 0.9969 |\n| ResNet101 | fixed | val | **0.8358** | **0.8683** | **0.2035** | **1.0174** | 0.9834 | **0.7500** | **0.9599** |\n| ResNet101 | learnable | train | 1.0741 | 0.9523 | **0.0250** | **0.1249** | **0.9988** | 0.0000 | **0.9980** |\n| ResNet101 | learnable | val | 1.3540 | 0.5996 | 0.2066 | 1.0332 | **0.9869** | 0.0000 | 0.9573 |\n| ResNet50 | fixed | train | **0.3312** | **0.9909** | 0.0264 | 0.1318 | 0.9931 | **0.9545** | 0.9972 |\n| ResNet50 | fixed | val | **0.8450** | **0.8722** | 0.2061 | 1.0306 | 0.9819 | **0.7500** | 0.9526 |\n| ResNet50 | learnable | train | 0.5326 | 0.9879 | **0.0177** | **0.0887** | **0.9987** | 0.0000 | **0.9983** |\n| ResNet50 | learnable | val | 0.9335 | 0.7837 | **0.1924** | **0.9620** | **0.9883** | 0.0000 | **0.9573** |\n| DenseNet201 | fixed | train | **0.4172** | 0.9454 | 0.0177 | 0.0885 | 0.9964 | **0.9500** | 0.9982 |\n| DenseNet201 | fixed | val | **0.7893** | **0.8776** | **0.1835** | **0.9174** | **0.9877** | **0.7866** | **0.9598** |\n| DenseNet201 | learnable | train | 0.7825 | **0.9884** | **0.0141** | **0.0706** | **0.9993** | 0.0000 | **0.9984** |\n| DenseNet201 | learnable | val | 1.0224 | 0.8042 | 0.1908 | 0.9538 | 0.9866 | 0.0000 | 0.9538 |\n\n**Table 2: ONC Metrics Evolution (Initialization → Post-training)**\n| Backbone | Threshold | Data Split | onc1 ↓ | onc2_1 ↓ | onc2_2 ↓ | onc3 ↓ |\n|----------|-----------|------------|--------|----------|----------|--------|\n| ResNet101 | fixed | train | 8.91×10⁻¹ → 6.27×10⁻² | 4.82×10⁻¹ → **1.21×10⁻⁸** | 9.65×10⁻¹ → **2.62×10⁻⁴** | 4.71×10⁰ → **9.24×10⁻²** |\n| ResNet101 | fixed | val | 8.98×10⁻¹ → 1.54×10⁻¹ | 5.11×10⁻¹ → **1.23×10⁻⁸** | 9.80×10⁻¹ → **2.62×10⁻⁴** | 4.72×10⁰ → **2.40×10⁻¹** |\n| ResNet101 | learnable | train | 8.92×10⁻¹ → **5.80×10⁻²** | 4.84×10⁻¹ → 7.33×10⁻⁵ | 9.65×10⁻¹ → 5.01×10⁻³ | 6.28×10⁰ → 2.53×10⁻¹ |\n| ResNet101 | learnable | val | 8.99×10⁻¹ → **1.49×10⁻¹** | 5.12×10⁻¹ → 7.21×10⁻⁵ | 9.81×10⁻¹ → 5.03×10⁻³ | 6.29×10⁰ → 3.16×10⁻¹ |\n| ResNet50 | fixed | train | 8.90×10⁻¹ → 7.65×10⁻² | 4.82×10⁻¹ → **3.12×10⁻⁷** | 9.64×10⁻¹ → **5.52×10⁻⁴** | 4.71×10⁰ → **8.27×10⁻²** |\n| ResNet50 | fixed | val | 8.97×10⁻¹ → **1.56×10⁻¹** | 5.10×10⁻¹ → **2.84×10⁻⁷** | 9.80×10⁻¹ → **5.51×10⁻⁴** | 4.72×10⁰ → **2.23×10⁻¹** |\n| ResNet50 | learnable | train | 8.92×10⁻¹ → **6.72×10⁻²** | 4.83×10⁻¹ → 5.91×10⁻⁴ | 9.65×10⁻¹ → 4.56×10⁻² | 6.28×10⁰ → 3.27×10⁻¹ |\n| ResNet50 | learnable | val | 8.98×10⁻¹ → 1.57×10⁻¹ | 5.12×10⁻¹ → 5.65×10⁻⁴ | 9.81×10⁻¹ → 4.55×10⁻² | 6.29×10⁰ → 4.27×10⁻¹ |\n| DenseNet201 | fixed | train | 8.93×10⁻¹ → 5.64×10⁻² | 4.84×10⁻¹ → **1.08×10⁻⁵** | 9.66×10⁻¹ → **2.66×10⁻³** | 4.72×10⁰ → **8.04×10⁻²** |\n| DenseNet201 | fixed | val | 9.00×10⁻¹ → **1.27×10⁻¹** | 5.13×10⁻¹ → **9.81×10⁻⁶** | 9.82×10⁻¹ → **2.67×10⁻³** | 4.73×10⁰ → **1.50×10⁻¹** |\n| DenseNet201 | learnable | train | 8.94×10⁻¹ → **4.94×10⁻²** | 4.85×10⁻¹ → 7.14×10⁻⁴ | 9.66×10⁻¹ → 1.39×10⁻¹ | 6.30×10⁰ → 4.48×10⁻¹ |\n| DenseNet201 | learnable | val | 9.01×10⁻¹ → 1.45×10⁻¹ | 5.14×10⁻¹ → 6.94×10⁻⁴ | 9.82×10⁻¹ → 1.39×10⁻¹ | 6.30×10⁰ → 5.36×10⁻¹ |"}, {"review_id": "YxL9dW6fil", "reviewer": "Reviewer_wf6P", "summary": "The paper investigated whether the Neural Collapse (NC) phenomenon arises in deep Ordinal regression (OR) tasks by combining the cumulative link model for OR and the unconstrained feature model (UFM). They extended the NC theory to CLM-based OR and found some properties the authors demonstrated analytically and empirically across benchmark ordinal datasets.", "questions": "•\tCan you clarify the intended purpose of Figure 3 (evolution of latent and PCA-projected features)? How does it support or relate to the ONC properties?\n•\tHave you considered evaluating ONC properties on additional (more challenging) ordinal datasets?\n•\tWhy did you not consider other standard ordinal evaluation metrics like Quadratic Weighted Kappa (QWK), Minimum Sensitivity, and Accuracy within 1-off? Including them could provide complementary evidence for ONC convergence.\n•\tHave you considered evaluating your approach with alternative ordinal loss functions, such as those based on QWK? These could affect ONC convergence properties.\n•\tHave you explored other link functions, such as the complementary log-log link, which may be more suitable for skewed ordinal class distributions?", "limitations": "The authors acknowledge relevant limitations in a specific discussion subsection. The additional limitations outlined in the Weaknesses and Questions sections should also be carefully considered and addressed.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "(+) The authors investigate whether a phenomenon analogous to Neural Collapse (NC) arises in ordinal regression (OR) tasks. They introduce the  Ordinal Neural Collapse (ONC) concept and support its existence through analytical derivations and empirical analysis.\n(+) The work contributes to the theoretical understanding of ordinal regression by extending the theory of Neural Collapse to the ordinal domain, offering new insights into the behavior of deep models trained on ordinal tasks.\n\n(–) As acknowledged in the limitations, the theoretical analysis assumes fixed thresholds $\\mathbf{b}$. The ONC phenomenon under learnable thresholds is not theoretically addressed. Although the empirical analysis includes fixed and learnable thresholds, further investigation is needed to characterize the behavior of ONC_{3} in the learnable case. In particular, experimental results suggest that the ONC_{3} metric converges to a non-zero value when thresholds are learned, indicating that Eq. (21) may not hold in that setting.\n(–) The purpose and interpretation of Figure 3, which depicts the evolution of latent representations and their PCA projections, is unclear. It is not sufficiently connected to the analysis of ONC properties shown in Figure 2 (which shows the evolution of ONC-related metrics). More explicit descriptions of Figure 3 are needed to clarify the contribution of this result better. \n(–) While hyperparameter settings and additional experiments are reported in the Appendix, the main paper would benefit from presenting at least one additional experiment on a different dataset to reinforce the empirical evidence for ONC and improve the generalizability of the findings.\n(–) As discussed by the authors, the ONC properties were confirmed on standard benchmark ordinal datasets. However, to better assess the robustness of these findings, it would be important to test the ONC phenomenon under more challenging datasets and conditions such as ordinal imbalance or distributional shift, which are more representative of real-world scenarios.\n(–) The evaluation of convergence is currently limited to a few metrics. It is unclear why standard ordinal metrics such as Quadratic Weighted Kappa (QWK), Minimum Sensitivity, or Accuracy within 1-off were not included in the analysis. Their inclusion could offer complementary insights.\n(–) The authors do not explore alternative ordinal loss functions, such as those based on QWK, which could influence convergence behavior and ONC dynamics.\n(–) Finally, the choice of link function is limited. The authors do not consider alternatives like the complementary log-log link, which might better model skewed ordinal class distributions.", "was_revised": true, "final_justification": "The authors answered my comments. As a result I upgraded my score to accept. I also encourage the authors to include these explanations and additional experiments in the final version of the paper.", "rebuttal": "We appreciate your careful reading of our manuscript and your insightful comments.\n\n**[W1] Theoretical limitation regarding learnable thresholds**\n\n**[A1]** We recognize this as a limitation of our results. In the case of learnable thresholds, the outcome likely depends on how the thresholds are optimized. We conjecture that if the learning of thresholds proceeds sufficiently slowly compared to the learning of $\\boldsymbol{w}$ and $\\boldsymbol{h}$, ONC3 may still hold. However, this setup deviates from standard DNN training practices, and we have not conducted experiments under such conditions. As this consideration clarifies, the emergence of ONC3 would depend on the details of the learning dynamics, and hence it would be nontrivial to resolve this issue. Accordingly, we would like to leave this point as a direction for future work.\n\n**[W2, Q1] Clarification on the purpose and interpretation of Figure 3**\n\n**[A2]** We apologize that the explanation of Figure 3 was insufficient and confusing. The purpose of Figure 3 is to intuitively visualize the occurrence of ONC1, ONC2, and ONC3. Figure 3 is divided into upper and lower parts, corresponding to the fixed and learnable thresholds, respectively. Each threshold setting is further divided into upper and lower panels, with the upper ones representing the latent space and the lower ones expressing the feature space. \n\nFeature Space: This refers to the space where $\\boldsymbol{h}$ resides (via PCA dimensionality reduction). Different colors represent different classes, and pentagrams represent the class means. Black arrows represent the classifier weights $\\boldsymbol{w}$ in the formula. At epoch 0, features are scattered in the feature space; as training progresses, features collapse to their class means, reflecting ONC1. The class means then collapse to a one-dimensional subspace aligned with $\\boldsymbol{w}$, reflecting ONC2.\n\nLatent Space: This refers to the one-dimensional space where $\\boldsymbol{z}$ and $\\boldsymbol{b}$ in the formula reside. Different colored points represent $z$ of different classes. Thresholds $\\boldsymbol{b}$ are represented by red dashed lines. At epoch 5000, the $z$ class means collapse in class order and are positioned between the appropriate thresholds, which directly reflects the occurrence of ONC3. For the fixed thresholds case, the thresholds are (approximately) located at the midpoints of adjacent $z$ class means. For the learnable case, since the boundary thresholds are set to negative and positive infinity, the $z$ class means at both ends do not show an apparent collapse. This explains why the ONC_3 metric curves converge to different values for the two threshold strategies.\n\nWe will improve the main text to clarify the above points in the revision. \n\n**[W3] Insufficient Dataset Diversity in Main Text**\n\n**[A3]** Thank you for this excellent suggestion. In response to [W4, Q2], we have added experiments on UTKFace, a classic facial age estimation dataset in ordinal regression tasks, with three additional backbones (details in [A4]). If the paper is accepted, we plan to include partial results from the UTKFace experiments in the main text, which would be more convincing than simply moving results of one dataset from the current appendix to the main text.\n\n**[W4, Q2] Evaluation of robustness of the theoretical findings under more challenging and realistic conditions**\n\n**[A4]** Thank you for your constructive suggestion. Reviewers zTjz and 7zzE also emphasized the importance of validating the ONC phenomenon on more challenging tasks. Therefore, we have examined three different backbone networks (ResNet-50, ResNet-101, and DenseNet-201) on the UTKFace age-estimation dataset binned into 20 ordinal classes (5-year intervals, 0–4 yrs up to 95–116 yrs): the maximum class-imbalance ratio is approximately 75:1. Facial age estimation using the UTKFace dataset is one of the most renowned tasks in the ordinal regression field, and the ONC phenomenon in this task will provide more robust validation for our theoretical findings. \n\nThe experimental results are summarized in Tables 1 and 2 in our response to Reviewer 7zzE (due to character limitations in this response, it was not possible to show it here). All four ONC metrics show substantial decreases across architectures, indicating successful collapse during training for both training and validation datasets in all cases. \n\nIf the paper is accepted, all these additional experimental results will be added to the paper.\n\n**[W5, Q3] Use of non-standard ordinal evaluation metrics [Q4]**\n\n**[A5]** We thank you for your suggestion. Reviewer 7zzE also provided a similar suggestion. According to these comments, we have additionally computed five additional metrics, including QWK, minimum sensitivity, and accuracy within 1-off. \n\nUsing the five tasks presented in the original manuscript, we have computed these metrics and found that the mean values of QWK, minimum sensitivity, and accuracy within 1-off almost converge to 1 with variance close to 0 at the final epoch. The same observation is obtained for the supplementary experiments mentioned in [A4]. Table 1 in our response to Reviewer 7zzE reports this. Notably, the learnable threshold method consistently achieves 0 minimum sensitivity on UTKFace, while the fixed threshold method overcomes this issue, implying that the fixed threshold strategy actually mitigate the minority class problem.\n\nIn the revision, we will add plots of all these new metrics to help readers better judge the training and validation situations.\n\n**[W6, Q4] Exploration of alternative ordinal loss functions beyond standard CLM framework**\n\n**[A6]** Indeed, changing the loss function would naturally affect the dynamics of ONC. However, our analysis is not concerning the dynamics, but rather focuses on the stationary properties of solutions obtained by minimizing the loss function. Moreover, our framework is fundamentally grounded in a statistical perspective based on maximum likelihood estimation. While it is interesting to explore the implications of using QWK, particularly in terms of its impact on learning dynamics, we consider such investigations to be beyond the scope of the present study and therefore leave them out of consideration. We would appreciate your understanding.\n\n**[W7, Q5] Limited exploration of alternative link functions suitable for skewed class distributions**\n\n**[A7]** As the complementary log-log (cloglog) link is associated with a log‑concave probability density function, it satisfies our theoretical assumptions and thus falls within the scope of our theory. We will stress this point in the revision.\n\nTo validate its empirical effect, we have conducted additional experiments using the cloglog link function, with the same architecture and datasets as in the original manuscript. The result is summarized in Table 1. A noteworthy point of the cloglog function is that it is not symmetric, in contrast to the logistic and normal CDF functions, and hence the equation (21) does not hold. Instead, the optimal $z_q^*$ at the weak regularization limit should be computed by numerically solving the equation (18), and thus the appropriate metric for ONC3 (equation (25)) should be modified. The result using the modified ONC3 metric (written as onc3_cloglog) is reported in Table 1.\n\nFrom the result, we can see that all the three characteristics of ONC actually occur in the cloglog function as well. We will add this result in the appendix in the revision.\n\n**Table 1: ONC Metrics Evolution for LE Dataset (Initialization → Post-training)**\n| Threshold | Data Split | onc1 ↓ | onc2_1 ↓ | onc2_2 ↓ | onc3_cloglog ↓ |\n|-----------|------------|--------|----------|----------|----------------|\n| fixed | train | 8.20×10⁻¹ → **1.15×10⁻²** | 3.85×10⁻¹ → **2.18×10⁻¹⁴** | 9.43×10⁻¹ → **2.80×10⁻⁶** | 1.34×10⁰ → **2.27×10⁻²** |\n| fixed | test | 8.48×10⁻¹ → **1.36×10⁻²** | 3.95×10⁻¹ → **7.92×10⁻¹⁵** | 9.37×10⁻¹ → **2.86×10⁻⁶** | 1.34×10⁰ → **2.35×10⁻²** |\n| learnable | train | 8.30×10⁻¹ → 1.57×10⁻² | 3.10×10⁻¹ → 2.77×10⁻¹⁴ | 8.66×10⁻¹ → 2.41×10⁻⁴ | 1.99×10⁰ → 5.57×10⁻¹ |\n| learnable | test | 8.44×10⁻¹ → 2.42×10⁻² | 2.74×10⁻¹ → 1.22×10⁻¹⁴ | 8.73×10⁻¹ → 2.41×10⁻⁴ | 2.00×10⁰ → 5.53×10⁻¹ |"}]

Author rebuttal:

Meta-review: This paper extends the theory of Neural Collapse (NC), a phenomenon observed in standard classification, to the domain of Ordinal Regression (OR). The authors introduce and formally characterize "Ordinal Neural Collapse" (ONC) by combining the Cumulative Link Model (CLM) with the Unconstrained Feature Model (UFM). Their key contribution is a theoretical proof demonstrating that optimal solutions exhibit three properties: within-class feature collapse (ONC1), alignment of class means onto a 1D classifier subspace (ONC2), and an ordered alignment of latent variables that relates simply to the model's thresholds (ONC3). Reviewers were in agreement that the work is novel, technically sound, well-motivated, and clearly written, representing a valuable extension of NC theory to a new and important task.

The primary concerns raised by reviewers centered on the empirical validation, which was initially perceived as being conducted on relatively simple datasets with a non-standard architecture, thus limiting the generalizability of the findings and practical insights. However, the authors provided a thorough and convincing rebuttal. They conducted substantial new experiments on the more challenging and imbalanced UTKFace dataset using standard architectures like ResNet and DenseNet. These new results suggested that ONC robustly emerges in more complex and realistic settings and also uncovered a valuable practical insight: the use of fixed thresholds, as suggested by their theory, can improve performance for minority classes. The authors also addressed other reviewer concerns regarding evaluation metrics and technical details, which led one reviewer to raise their score to "Accept".

While some reviewers maintain borderline scores, citing remaining questions about the full practical implications, I believe the paper's contributions are significant enough to warrant acceptance. The primary contribution is a novel and rigorous theoretical extension of NC, and the authors have successfully demonstrated this phenomenon both analytically and empirically. The additional experiments conducted during the rebuttal have substantially strengthened the paper's claims and addressed the most critical initial weaknesses. Therefore, I recommend this paper be accepted. I strongly encourage the authors to integrate the new experimental results and the reviewers' feedback into the final manuscript to maximize its impact.
Decision: Accept (poster)