Title: Neural Collapse in Cumulative Link Models for Ordinal Regression: An Analysis with Unconstrained Feature Model
Venue: NeurIPS 2025
Accepted: yes
Keywords: neural collapse

unconstrained feature model

ordinal regression

cumulative link model

deep learning theory
Abstract: A phenomenon known as ``Neural Collapse (NC)'' in deep classification tasks, in which the penultimate-layer features and the final classifiers exhibit an extremely simple geometric structure, has recently attracted considerable attention, with the expectation that it can deepen our understanding of how deep neural networks behave. The Unconstrained Feature Model (UFM) has been proposed to explain NC theoretically, and there emerges a growing body of work that extends NC to tasks other than classification and leverages it for practical applications. In this study, we investigate whether a similar phenomenon arises in deep Ordinal Regression (OR) tasks, via combining the cumulative link model for OR and UFM. We show that a phenomenon we call Ordinal Neural Collapse (ONC) indeed emerges and is characterized by the following three properties: (ONC1) all optimal features in the same class collapse to their within-class mean when regularization is applied; (ONC2) these class means align with the classifier, meaning that they collapse onto a one-dimensional subspace; (ONC3) the optimal latent variables (corresponding to logits or preactivations in classification tasks) are aligned according to the class order, and in particular, in the zero-regularization limit, a highly local and simple geometric relationship emerges between the latent variables and the threshold values. We prove these properties analytically within the UFM framework with fixed threshold values and corroborate them empirically across a variety of datasets. We also discuss how these insights can be leveraged in OR, highlighting the use of fixed thresholds.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigated whether the Neural Collapse (NC) phenomenon arises in deep Ordinal regression (OR) tasks by combining the cumulative link model for OR and the unconstrained feature model (UFM). They extended the NC theory to CLM-based OR and found some properties the authors demonstrated analytically and empirically across benchmark ordinal datasets.\n\nSTRENGTHS AND WEAKNESSES:\n(+) The authors investigate whether a phenomenon analogous to Neural Collapse (NC) arises in ordinal regression (OR) tasks. They introduce the  Ordinal Neural Collapse (ONC) concept and support its existence through analytical derivations and empirical analysis.\n(+) The work contributes to the theoretical understanding of ordinal regression by extending the theory of Neural Collapse to the ordinal domain, offering new insights into the behavior of deep models trained on ordinal tasks.\n\n(–) As acknowledged in the limitations, the theoretical analysis assumes fixed thresholds $\\mathbf{b}$. The ONC phenomenon under learnable thresholds is not theoretically addressed. Although the empirical analysis includes fixed and learnable thresholds, further investigation is needed to characterize the behavior of ONC_{3} in the learnable case. In particular, experimental results suggest that the ONC_{3} metric converges to a non-zero value when thresholds are learned, indicating that Eq. (21) may not hold in that setting.\n(–) The purpose and interpretation of Figure 3, which depicts the evolution of latent representations and their PCA projections, is unclear. It is not sufficiently connected to the analysis of ONC properties shown in Figure 2 (which shows the evolution of ONC-related metrics). More explicit descriptions of Figure 3 are needed to clarify the contribution of this result better. \n(–) While hyperparameter settings and additional experiments are reported in the Appendix, the main paper would benefit from presenting at least one additional experiment on a different dataset to reinforce the empirical evidence for ONC and improve the generalizability of the findings.\n(–) As discussed by the authors, the ONC properties were confirmed on standard benchmark ordinal datasets. However, to better assess the robustness of these findings, it would be important to test the ONC phenomenon under more challenging datasets and conditions such as ordinal imbalance or distributional shift, which are more representative of real-world scenarios.\n(–) The evaluation of convergence is currently limited to a few metrics. It is unclear why standard ordinal metrics such as Quadratic Weighted Kappa (QWK), Minimum Sensitivity, or Accuracy within 1-off were not included in the analysis. Their inclusion could offer complementary insights.\n(–) The authors do not explore alternative ordinal loss functions, such as those based on QWK, which could influence convergence behavior and ONC dynamics.\n(–) Finally, the choice of link function is limited. The authors do not consider alternatives like the complementary log-log link, which might better model skewed ordinal class distributions.\n\nQUESTIONS:\n•\tCan you clarify the intended purpose of Figure 3 (evolution of latent and PCA-projected features)? How does it support or relate to the ONC properties?\n•\tHave you considered evaluating ONC properties on additional (more challenging) ordinal datasets?\n•\tWhy did you not consider other standard ordinal evaluation metrics like Quadratic Weighted Kappa (QWK), Minimum Sensitivity, and Accuracy within 1-off? Including them could provide complementary evidence for ONC convergence.\n•\tHave you considered evaluating your approach with alternative ordinal loss functions, such as those based on QWK? These could affect ONC convergence properties.\n•\tHave you explored other link functions, such as the complementary log-log link, which may be more suitable for skewed ordinal class distributions?\n\nLIMITATIONS:\nThe authors acknowledge relevant limitations in a specific discussion subsection. The additional limitations outlined in the Weaknesses and Questions sections should also be carefully considered and addressed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores the phenomenon of Neural Collapse in the context of ordinal regression, while previous studies primarily focused on classification and regression tasks. The authors theoretically demonstrate that regularized empirical risk minimization (ERM) in unconstrained feature models leads to Neural Collapse even when the learning objective is ordinal regression. Specifically, they show that the class-wise mean features collapse onto a one-dimensional real line, and the ordering of these projected means corresponds to the ordinal structure of the classes. The authors validate their findings through experiments on several benchmark datasets, providing evidence that overparameterized models trained for ordinal regression display the prescribed Neural Collapse properties.\n\nSTRENGTHS AND WEAKNESSES:\nThis paper makes a valuable theoretical contribution by extending neural collapse analysis to ordinal regression, though its experimental validation has some limitations that reduce its potential impact. While I lean toward accepting the paper, some concerns regarding the experimental results should be addressed in the rebuttal.\n\nThe paper is well organized and clearly written. The main theoretical results are well-presented with rigorous mathematical notations. The objective of the empirical evaluations is clearly highlighted as bridging the gap between theoretical findings and practical applications. The clarity of the paper is high except for the demonstrations of the experimental results.\n\nThe theoretical results are supported by rigorous mathematical proofs. While I did not check the proofs in detail, I cannot find any major issues. The quality of the theoretical contents appears to be at a publishable standard.\n\nThe authors incorporate existing techniques for analyzing neural collapse (i.e., unconstrained feature models) to analyze neural collapse in ordinal regression. While the basic idea of the theoretical analyses combines existing techniques, this combination itself represents a novel contribution. Therefore, the present paper exhibits originality, though not at the highest level.\n\nThe theoretical analyses reveal three properties regarding neural collapse in ordinal regression. OCN1 and OCN2 are analogous to existing results in classification tasks. OCN3 and Theorem 4.3 are non-trivial findings specific to ordinal regression. Particularly significant is the suggestion regarding choices of regularization parameters that avoid trivial meaningless solutions.\n\nI have concerns about the experimental validation being conducted on a single non-common architecture designed by the authors. The generalizability of these results to widely-used architectures such as CNN, ResNet, and Transformer remains unclear. Since these common architectures are basis of many current successes in deep learning, the inability to demonstrate that the theoretical findings apply to these architectures diminishes the paper's potential impact in explaining real-world deep learning successes.\n\nThe quality of the experimental design raises some questions. The authors use a non-standard accuracy metric (mean absolute error of the value of true and predicted labels) despite their own cited reference (Gutierrez et al., 2016) recommending classification accuracy and mean absolute error of the *order* of predicted and true labels as accurate metrics for ordinal regression tasks. This raises a concern that the empirical confirmations of OCNs were done for inaccurate models, reducing the practical relevance of the theoretical results.\n\nWith respect to clarity of experimental results, the presentation in Figure 2 could be improved. The authors inconsistently use different scales for the vertical axis across the evaluated metrics—linear scale for MAE, ONC2-1, and ONC2-2, while using log scale for LNLL, ONC1, and ONC3. Using a consistent log scale for all metrics would facilitate better interpretation of the results.\n\nQUESTIONS:\n1. How do the theoretical findings translate to practical benefits for ordinal regression tasks? Could you please demonstrate improved performance on benchmark datasets when applying insights from your neural collapse analysis?\n\n2. Could the authors please justify how their experimental design addresses the concerns raised in the review?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work extends Neural Collapse (NC) to ordinal regression (OR) tasks, which differ subtly from both classification and regression. Building on the Unconstrained Feature Model (UFM) and the Cumulative Link Model (CLM), the authors provide a theoretical characterization of a collapse phenomenon under $\\ell_2$ regularization applied to both classifier weights and “free” hidden features. The theoretical results can be decomposed into two sets: the first set ONC1 and ONC2 mirror classical NC behavior, showing within-class feature collapse and alignment of class means with the classifier direction, the other set is more task-specific, where the authors demonstrate that with vanishing regularization strength, the collapsed latent values respect the class order, and under zero regularization, they match threshold midpoints. Empirical results validate these findings and suggest that the collapse structure persists even when thresholds are learned rather than fixed.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n* The paper is overall well-written, with clear technical exposition and smooth narrative flow. The related work section on NC is comprehensive and positions the paper well within the broader literature.\n* As far as I can tell, the extension of the NC framework to OR tasks appears to be novel and well-motivated.\n* The theoretical analysis and empirical validation are tightly aligned. In particular, the behavior under vanishing regularization is quite interesting and elegant.\n\nWeaknesses:\n* The paper falls somewhat short on practical insights.. The authors discuss that the fixed threshold values \"yield faster and more stable convergence compared with learnable thresholds\" and \"These make ONC a potentially effective concept for improving practical performance in generic OR tasks.\" but not pursue this direction more clearly. For example, Figure 2 shows similar MAE for fixed and learnable thresholds, but notably better $\\ell_{NLL}$. Could this translate to improved generalization or calibration in more challenging settings?\n* In Figure 2 and its associated discussion, the authors mention that both training and validation MAE approach zero. However, the curves appear to be nearly identical throughout training, which raises the question of whether the chosen tasks are too easy. Would it be possible to test on more comprehensive tasks to broaden the applicability of the results?\n\nQUESTIONS:\nPlease see the weakness section.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-01"}

comments: SUMMARY:
This paper studies the phenomenon of neural collapse in ordinal regression models. The authors propose a set of conditions for ordinal neural collapse, and through the unconstrained features model show that the global minima satisfy these conditions. The authors also empirically validate their findings on ordinal regression datasets.

STRENGTHS AND WEAKNESSES:
Strengths: This paper has both analytical and empirical characterizations of neural collapse in a new setting. The mathematical results seem correct (I have not thoroughly checked the derivation of ONC3), and the empirical validation in small label settings seems valid.

Weaknesses:

1. The mathematical results do not seem to provide prescriptions for training methods, and only analyze the landscape of the unconstrained features model. This is the case for all UFM analyses however, not a particular problem with this paper.

2. The empirical validation is performed on relatively simple datasets with a small number of classes. Ordinal regression provides an advantage over regular classification in situations with extremely large label spaces. How does NC emerge in that case?

QUESTIONS:
1. What is the role of weight decay in the analysis of the UFM model? Is weight decay necessary?

2. Comparing the emergence of NC in OR vs standard classification, is there a reason to prefer one setup over another? Can the UFM analysis provide an answer for this?

3. The analysis seems to exclude the case where the thresholds are learned. Can the UFM analysis prescribe a proper choice of thresholds?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-09