Title: Improved Balanced Classification with Theoretically Grounded Loss Functions
Venue: NeurIPS 2025
Accepted: yes
Keywords: balanced loss

imbalanced data

consistency

surrogate loss

learning theory
Abstract: The *balanced loss* is a widely adopted objective for multi-class classification under class imbalance. By assigning equal importance to all classes, regardless of their frequency, it promotes fairness and ensures that minority classes are not overlooked. However, directly minimizing the balanced classification loss is typically intractable, which makes the design of effective surrogate losses a central question. This paper introduces and studies two advanced surrogate loss families: Generalized Logit-Adjusted (GLA) loss functions and Generalized Class-Aware weighted (GCA) losses. GLA losses generalize Logit-Adjusted losses, which shift logits based on class priors, to the broader general cross-entropy loss family. GCA loss functions extend the standard class-weighted losses, which scale losses inversely by class frequency, by incorporating class-dependent confidence margins and extending them to the general cross-entropy family. We present a comprehensive theoretical analysis of consistency for both loss families. We show that GLA losses are Bayes-consistent, but only $H$-consistent for complete (i.e., unbounded) hypothesis sets. Moreover, their $H$-consistency bounds depend inversely on the minimum class probability, scaling at least as $1/\mathsf p _{\min}$. In contrast, GCA losses are $H$-consistent for any hypothesis set that is bounded or complete, with $H$-consistency bounds that scale more favorably as $1/\sqrt{\mathsf p _{\min}}$, offering significantly stronger theoretical guarantees in imbalanced settings. We report the results of experiments demonstrating that, empirically, both the GCA losses with calibrated class-dependent confidence margins and GLA losses can greatly outperform straightforward class-weighted losses as well as the LA losses. GLA generally performs slightly better in common benchmarks, whereas GCA exhibits a slight edge in highly imbalanced settings. Thus, we advocate for both GLA and GCA losses as principled, theoretically sound, and state-of-the-art surrogates for balanced classification under class imbalance.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper motivates a new algorithm for conducting class-balanced cross entropy (CE) minimization. The paper states that class-balanced CE minimization has been addressed via heuristic methods such as class-weighted CE and logit-adjusted loss, but that these methods lack desired theoretical properties such as Bayes consistency. The paper then develops two new loss functions - Generalized Logit-Adjusted Loss (GLA) and Generalized Class-Aware Loss (GCA), as algorithms that upper bound the true cost-sensitive loss while having strong theoretical properties of Bayes consistency and H-consistency. The experimental results show that the new loss functions perform the best in terms of balanced-error when used in multi-class classification, for different kinds of class-skew in the training data.\n\nSTRENGTHS AND WEAKNESSES:\n**Quality**: This paper is overall excellent. The GLA and GCA algorithms are very well-grounded, as they are motivated by the true balanced loss and naturally derived as being tractable surrogates through manipulation of the margin loss. Additionally, the theorems go beyond traditional convergence under convexity style proofs and show Bayes consistency/H-consistency, adding another layer of theoretical backing to the algorithms. The experiments are well-conducted and benchmarks against several well-known, relevant algorithms in this domain (e.g. focal loss, logit-adjusted, class-weighted CE, etc.). The experimental setting is slightly limited (three CV datasets) but is nonetheless appropriate and well-justified, as there are indeed several classes and the authors clearly explain how class imbalanced is induced. The proofs are difficult to read as they do not have much annotation, but from what I can gauge, they appear correct and have a logical development (e.g. $\\rho$-margin loss -> upper bound -> GLA loss). The differences between the GLA and GCA loss are also clearly elucidated. \n\n**Clarity**: Despite the excellent theoretical results, this paper was difficult to parse in multiple parts and needs some improvement (although I fully acknowledge the space limitations). At a high level, the lack of clarity is in some details, such as why weighted CE does not work well when data is easily separable, what the confidence margin parameters are supposed to represent, what the evaluation \"balanced error\" actually is, etc. Most importantly the proofs are hard to read. More details in the suggestions.\n\n**Significance**: This is a very significant advancement in the development of class-balanced error minimization. Class-balanced error indeed has several applications, including in fairness, fraud-detection, rare-event detection, etc. In practice, class-weighted CE is used (or equivalently, data resampling), but this paper proposes a new path forward that presents a more theoretically disciplined, yet tractable approach. The proofs are also quite novel, and may be useful for future research in this area.\n\n**Originality**: This paper is very original. It appropriately reviews existing literature around this topic and identifies the key works. The methods use to develop the GCA/GLA are relatively novel, as they take a principled approach to upper bounding the class-weighted 0/1 loss and develop a surrogate accordingly. I am not as familiar with proofs in Bayes consistency/H-consistency to judge if they are novel, but considering these results are rarely developed in research regarding loss-functions, they feel novel to me.\n\nQUESTIONS:\nThe paper is as a whole excellent and significant, the following are my suggestions to make it more accessible to readers not as familiar with this style of work and further increase adoption:\n1. Add further details for why class-weighted CE does not work well. This is (in my experience) by far the most common approach to handling this very general problem. Identifying the errors more clearly can go a long way to further boost the significance of the results. For example, the graph on the (lack of) H-consistency of LA loss is great.\n2. When introducing the GCA loss, it says that $\\rho$ is \"is a vector of positive confidence margin parameters for each class.\" Practically speaking, what is this supposed to be? More hyperparameters typically yields more complications when trying to use a loss in practice, so helping clarify what these are meant to represent may provide more transparency around using this loss.\n3. The proofs could really use more annotation. A \"plan\" before the proof starts would go a long way, as would more details about how one step flows into the next. There appear to be some interesting techniques used, such as in the H-consistency results of GCA where the authors transform GCA to GCE via the change of measure and then use known consistency results. Hence, it would be really helpful to have some description. \n4. Most of the notation seems to be clearly defined, but I could not find where $\\mathcal{E}$ was defined. I'm guessing it's the empirical loss, but overall I recommend one more parse through the paper to check that notation is clearly defined.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes two novel surrogate loss functions for multi-class classification under class imbalance: Generalized Logit-Adjusted (GLA) loss and Generalized Class-Aware weighted (GCA) loss, building on the existing general cross-entropy (GCE) framework (Mao et al. ICML 2023). GLA losses generalize Logit-Adjusted losses, which shift logits based on class priors, to the broader general cross-entropy loss family. It includes theoretical analysis of consistency of the two proposed surrogate loss functions. Extensive experiments are conducted to evaluate the performance of GLA and GCA, in comparison with existing surrogate loss functions, demonstrating different operating characteristics of GLA and GCA.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: Building on the existing general cross-entropy (GCE) framework (Mao et al. ICML 2023), this work proposes and studies two surrogate loss functions, the generalized logit-adjusted (GLA) loss and the generalized class-aware (GCA) loss. In GCA, $\\rho$ is motivated by Cortes et al. (2025). Strong theoretical results are provided for consistency of GLA and GCA, showing that GLA is Bayes-consistent, but only $\\mathcal{H}$-consistent for unbounded and complete hypothesis sets whereas GCA is $\\mathcatl{H}-consistent for any hypothesis set that is bounded or complete and offers significantly stronger theoretical guarantees in imbalanced settings. Extensive experiments are conducted to demonstrate the superior performance of GLA and GCA in comparison with existing surrogate loss functions and also show different operating characteristics of GLA and GCA.\n\n\nWeakness: It is unclear why directly optimizing the balanced classification loss is intractable.\n\nIt is unclear whether the key assumptions underlying the theoretical results in Section 5 are plausible in real-world data.\n\nIt is unclear How the proposed GCE and GCA compared to IMMAX (Cortes et al. 2025) and other existing surrogate losses in terms of theoretical properties.\n\nSection 6: The numerical experiments did not include more recent baseline methods such as  IMMAX (Cortes et al. 2025) and LSC (Wei et al., 2024).\n\nQUESTIONS:\n1) How to assess whether the key assumptions underlying the theoretical results in Section 5 hold in real-world data? Under what real-world settings are these assumptions plausible?\n\n2) How do the proposed GCE and GCA compared to IMMAX (Cortes et al. 2025) in terms of their theoretical properties? \n\n3) It would strengthen the experiemental results to more recent baseline methods such as \nIMMAX (Cortes et al. 2025) and LSC (Wei et al., 2024).\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present generalized versions of the logit-adjust (LA) loss and the class-aware (CA) loss. They extend these through the generalized cross-entropy framework and show that they admit strong theoretical guarantees in terms of bayes consistency and $\\mathcal{H}$ consistency.  The authors also demonstrate the efficacy of their methods on CIFAR10 and 100 with imbalance.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n1. The paper is clear and well-written\n2. The theoretical guarantees for both GLA and GCA are quite strong, and the proofs themselves have novelty. The favorable scaling of GCA with $p_\\text{min}$ is especially compelling.\n3. The empirical performance of these methods is compelling, especially compared with more specialized methods like focal loss. \n\n# Weaknesses\n1. The novelty of the generalization is somewhat limited, simply be a simple extension of LA and CA under the GCE framework. \n2. The additional positive margins $\\rho$ included in GCA are new hyperparameters to tune, and their impact is not completely clear. Similarly with $q$ in both GLA and GCA. Including ablations to demonstrate the sensitivity to these choices would be helpful.\n3. The choice of the inverse of the prior for GCA is not clearly justified in the setting of $q\\ne1$ as it could be for $q=1$.\n\nQUESTIONS:\n1. How well-behaved are these loss functions in terms of optimization? Similar methods like $\\alpha$-loss (Sypherd et al.) can make the optimization procedure quite slow in practice. \n2. Ablations demonstrating the impact of the choice of $q$ could help readers better internalize its meaning and importance for tuning.\n3. Ablations for the inclusion of $\\rho$ in GCA would also be helpful.\n4. A key shortcoming of standard weighted cross entropy is in the highly overparameterized setting. How do GCA and GLA fare when data is limited?\n5. The weighting factor for GCA need not be the class prior. Tuning over this would be very interesting and give insights into whether the prior is an optimal choice for $q\\ne1$.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors give two novel surrogate loss function families for addressing class imbalance in multi-class classification: Generalized Logit-Adjusted (GLA) losses and Generalized Class-Aware weighted (GCA) losses. \n\nThe authors provide theoretical analysis showing that GLA losses are Bayes-consistent but only $H$-consistent for complete hypothesis sets with bounds scaling as $1/p_{min}$, while GCA losses are $H$-consistent for both bounded and complete hypothesis sets with more favorable bounds scaling as $1/\\sqrt{p_{min}}$. \n\nExperimental results on image datasets show that both loss families outperform existing baselines.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nThe paper provides rigorous theoretical analysis (in the supplement) with well-established consistency results. \n\nThe presentation is good, especially Sections 2 and 3, which provide a good foundation for the main results of the paper.\n\nThe experiments include proper statistical reporting.  They compare against relevant baselines across different datasets.\n\nWeaknesses:\n\nThe theoretical novelty is limited. It would be great if the authors clarify the proof novelty compared to Mohri et al., 2018 and Mao et al., 2023b,a in the main part of the paper.\n\nThe experiments are restricted to image classification tasks (CIFAR-10/100, Tiny ImageNet) with relatively simple architectures (ResNet-32). The generalizability to other domains and modern architectures is unclear.\n\nThe paper mentions that $\\rho_k$ values in GCA losses can be tuned via cross-validation but provides limited guidance on computational complexity or sensitivity analysis. The suggestion to use $[m_k^{1/3}]_k$ as starting points without any intuitions theoretical justification.\n\n\nWhile the paper mentions training details, it doesn't thoroughly analyze the computational cost of the proposed methods compared to baselines.\n\nQUESTIONS:\nLine 14: \"H-consistent for unbounded and complete hypothesis sets\" - The conjunction \"and\" is confusing here. Should clarify if both conditions are required or if it's \"unbounded or complete.\"\n\nLine 92: \"The margin $\\rho_h(x, y)$ for a predictor $h \\in H$\" - The margin notation $\\rho$ conflicts with the imbalance ratio $\\rho$ used later.  Could you clarify? \n\nLine 130: The LA loss definition uses $\\tau$ but the text doesn't clearly explain what happens when $\\tau \\neq 1$ until much later.\n\nLine 177: $\\rho = (\\rho_1, \\ldots, \\rho_n)$ - This reuses the margin notation from Page 3.\n\nLine 191: \"Note that while the $\\rho_k$ values can be freely tuned\" - What do you mean? \n\nLine 258: The bound $\\Gamma(t) = \\sqrt{\\frac{2t}{p_{min}}}$ -  Is this a tight bound?\n\nLine 283: The theorem statement is quite dense and could benefit from more intuitive explanation.\n\nLine 343: \"imbalance ratio, $\\rho = \\frac{\\max_{k=1}^c m_k}{\\min_{k=1}^c m_k}$\" - The notation $c$ for number of classes conflicts with the usage of $n$ elsewhere.\n\nLine 351: \"200 epochs\" - No justification for this choice or sensitivity analysis.\n\nEq (5): The confidence margin parameters $\\rho_y$ should be clearly distinguished from the margin function $\\rho_h(x,y)$ defined earlier.\n\nHow sensitive are the GCA losses to the choice of confidence margin parameters $\\rho_y$?\n\nWhat is the computational complexity comparison between GLA and GCA losses?\n\nAre there convergence guarantees for the optimization of these loss functions?\n\nIt would be great if the authors clarify the proof novelty compared to Mohri et al., 2018 and Mao et al., 2023b,a.\n\nLIMITATIONS:\nN.A.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}