Reducing Class-Wise Performance Disparity via Margin Regularization
Authors: Beier Zhu
Kesen Zhao
Jiequan Cui
Qianru Sun
Yuan Zhou
Xun Yang
Hanwang Zhang
Venue: ICLR 2026 Poster (2026)
Keywords: image classification
class-wise performance gap
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We propose MR^2, a theorical grounded margin regularization framework that reducing class-wise performance gap without sacrificing overall accuracy, validated across seven datasets including ImageNet, and foundation models like MAE, MoCov2, and CLIP.

Abstract: Deep neural networks often exhibit substantial disparities in class-wise accuracy, even when trained on class-balanced data—posing concerns for reliable deployment. While prior efforts have explored empirical remedies, a theoretical understanding of such performance disparities in classification remains limited. In this work, we present Margin Regularization for performance disparity Reduction ( $MR^2$ ), a theoretically principled  regularization for classification by dynamically adjusting margins in both the logit and representation spaces. Our analysis establishes a novel margin-based, class-sensitive generalization bound that reveals how per-class feature variability contributes to error, motivating the use of larger margins for ''hard'' classes. Guided by this insight,$MR^2$ optimizes per-class logit margins proportional to feature spread and penalizes excessive representation margins to enhance intra-class compactness.
Experiments on seven datasets—including ImageNet—and diverse pre-trained backbones (MAE, MoCov2, CLIP) demonstrate demonstrate that our $MR^2$ not only improves overall accuracy but also significantly boosts ''hard'' class performance without trading off ''easy'' classes, thus reducing the performance disparities. Codes are available in https://github.com/BeierZhu/MR2.

Published as a conference paper at ICLR 2026
REDUCING CLASS-WISE PERFORMANCE DISPARITY VIA MARGIN REGULARIZATION
Beier Zhu', Kesen Zhao?, Jiequan Cui Qianru Sun", Yuan Zhou?, Xun Yang', Hanwang Zhang² University of Science and Technology of China Nanyang Technological University, Hefei University of Technology *Singapore Management University beier. eew.uuuttc.u.......... cn
ABSTRACT
Deep neural networks often exhibit substantial disparities in class-wise accuracy, even when trained on class-balanced data posta- concerns for reliable deployment. While prior efforts have explored empirical remedies, theoretical understanding of such performance disparities in classification remains limited. In this work, we present Margin Regularization for performance disparity Reduction (MR²), a theoretically principled regularization for classification by dynamically adjusting margins in both the logit and representation spaces. Our analysis establishes a
sessisen generalization bound that reveals how per-class feature variability contributes to error, motivating the use of larger margins for "hard" classes. Guided by this insight, MR² optimizes per-class logit margins proportional to feature spread and penalizes excessive representation margins enhance intra-class compactness. Experiments on seven ImageNet-and and diverse pre- preitrained backbones (MAE, MoCov2, CLIP) demonstrate that MR² not only improves overall accuracy but also significantly boosts "hard" class performance without trading off "easy" classes, thus reducing the performance disparity. Codes are available in https :
com Bwt.BBBeeerehheeee
INTRODUCTION
Deep neural networks have achieved strong generalization across a wide range of domains and tasks. However, as noted by Cui al. (202)), substantial disparity class-wise accuracy persist-even even when training on class-balanced data. As illustrated in Figure models such as ResNet-50 (He
cet al. 2016a), and ViT-B/16 (Dosovitskiy et al. 2021) trained on ImageNet (Deng et al. 2009) exhibit extreme class-wise accuracy imbalance: for instance, with ResNet-50, the top-perlorming class reaches 100% 1 accuracy, while the worst performs at only 16%. The under-performance
of certain categories threatens the safe deployment of deep models, making it imperative to design frameworks that combine high accuracy with balanced class-wise performance.
Several works have investigated the causes and remedies for class-wise performance disparity. For instance, [alestrierro et al. (2022) shows that common techniques such as data augmentation and weight decay are unfair across classes and can lead to severe performance degradation on certain classes. Cui et al. (2024) attributes the disparity to the problematic representation rather than classifier bias-unlike in long-tailed classification (Menon al. 2021 Ren al. 2020 Kang et al. 2020) and advocates appropriate data augmentation YYnn et al.
et al. 2019) and representation learning (e.g., He et al.) (2020) 2022) to mitigate class-wiss wise unfairness. However, these approaches remain largely empirical and lack theoretical grounding. In this work, we provide principled explanation and framework for reducing class-wise performance disparity.
Our theoretical analysis is both motivated by and consistent with the empirical finding by Cui et al. (2024), which reveals that "hard" (under-pertee performinof classes exhibit greater feature variability than "easy" (well-performing) ones. This is reflected in Figure where the Le norm of the per-class feature variance, increases from "easy" to "hard" classes. In this work, we derive a novel margin- based, dass-sensitive learning guarantee (Proposition II that connects per-class feature variability and margin design. The result reveals that classes with larger feature variability contribute more to the generalization enror. Guided by theoretical results, we propose Margin Regularization for

---

Published as a conference paper at ICLR 2026
100
ResNet-30 50
12
80
Variamoo norm
60
200
400
600
800
1000
0
200
400
500
800
1000
Sorted class index
Sorted dlass index
(a) Per-class accuracy on ImageNet
(b) Per-class variance norm on ImageNet
Figure 1: (a) Deep models exhibit severe class-wise accuracy disparity on class-balanced dataset such as ImageNet. (b) Feature diversity imbalance: feature distribution of "hard" classes is more diverse than that of "easy" classes. Classes are sorted in descending order per-class accuracy.
performance disparity Reduction, dubbed MR², framework that dynamically adjusts margins in both logit and representation spaces during training (Section 3). The logit margin is designed proportional to the feature spread reduce the generalization bound (Corollary D.. while the representation margin regularization further tightens this bound (Section(4.2) Intuitively. the logit margin loss naturally promotes larger margins for "hard" classes, which are known to improve generalization (Bartlett et al. 1998 Koltchinskii & Panchenko 2002). and the representation margin loss encourages intra-class feature compactness.
To validate the effectiveness of our MR?, we conduct extensive experiments on seven datasets including ImageNet (Deng et al. 2009) using both convolutional networks (He et al. 201 b) and Vision Transformer (Dosovitskiy et al. 2021) architectures. In addition, we evaluate MR on diverse pre-trained backbones with both end to-end fine-tuning and linear probing iondiims--nccldiin
MAE (He et al. 2022), MoCov2 (He et al. 2020). and CLIP (Radford et al. - assess its broad applicability on recent foundation models. We make the following observations: (1) reduced performance disparity: MR² achieves clear improvements
"hard" classes across all models and datasets, effectively narrowing the performance gap with "easy" classes; and (2) improved wverall-erformaace: our MR² attains moderate gains "easy" classes as well, indicating that MR² reduces disparity without sacrificing "easy" class performance. This differs markedly from reweighting methods in debiasing (Nam al., 20200 Liu et al., 202T]. margin-
methods (Cao et al. 2019; Ren et al. (2020), distributionally robust optimization (Sagawa et al. 22200] Jung et al.
2023 etc., which entail performance trade-offs across "hard" and "easy" groups (Section 5.
RELATED WORK
Margin-based long-tail learning. In long-tailed classification, where the training data exhibit a skewed label distribution, introducing asymmetric margins into the loss function has proven to be an effective strategy. Typical methods include LDAM (Cao et al., 2019), EQL Tann et al.) (020). Balanced Softmax (Ren al., 2020), Logit Adjustment (Menon et al. 2021). GLA (Zhu al.) 2023). PPA (Zhu et | 2025) and DRO-LT (Samuel & Chechik 2021). However, these losses are specifically designed for class-imbalanced settings and reduce to the standard cross-entropy loss when class priors are uniform. In contrast, our margin-based loss provides meaningful regularization for improving balanced pla-class performance under balanced class priors. Hyperspherical learning methods (Zhou et al. [022 Son et al. 2025) regulate class sample margins, often adapting them to data difficulty and class frequency. However, these approaches not address class- wise performance disparities, which is the central focus of our work. A detailed discussion is deferred to Section 3.1
Mitigating performance disparity in classification. Recently, several studies (Cui et al. 2024 Balestriero et al. 2022 Li & Liu| 2023 Ma et al. 2022 Jin et al. 2025) have explored ways to address the imbalan lnce accuracies across classes on balanced data. Such work primarily focuses on faimess under adversarial attacks (Li & Liu 2023 Ma al. 2022 Jin et al. (025) conducts
purely empirical analyses of image recognition task (Cui et al. 2024: Balestriero et al., 2022). In
contrast, we provide general class-sensitive analysis that links class-level unfairness imbalanced
2

---

Published as a conference paper at ICLR 2026
intra-class diversity. Building this, derive principled margin regularization at both the logit and feature levels to promote balanced class-wise generalization.
3 METHOD
Setup. Considering a classification problem with inputs E X' and labels E ) = [K] drawn from
an unknown distribution P with balanced class priors, f.e., P(y) = 1/K. Let F be hypothesis set of
functions mapping from X' x ) to the prediction score (logit) R. For hypothesis f E F, the predicted
label f(x) of x is assigned to the one with the highest score, i.e., f(x) = argmax y f(x, 8). Let (-)
be the indicator function, we define the risk of ʃ as the 0-1 loss over P:: R(() = Ex Ex [1 + f(x))
and the best-in-class risk as RD (FF) inf fez R(f)) Our goal is to learn function f that attains the best-in-class risk R However, as the distribution IP is unknown, the risk of hypothesis is not directly accessible. The learner instead measures the empirical risk given training dataset
D {(x,; y:) pp;; Ro(J)- - tV Ex.geD + f(x))].
Notations. Assume that the function f consists of feature encoder Φ - Rdd and a classification head parameterized by K weight vectors (wx) ) For input x, the feature representation is ((x) and the logit for class y is represented as: f(x. y) = w d(x). Let Dy denote the subset that contains all instances x belonging
the class label y and Ny N/K the sample size of class y. The empirical per-class mean is computed as fu N, EXED, [o(x)] and the mean squared deviation is computed as 1 = 1 N, ΣxED, [1l((x) 11). During training, we maintain exponential moving average (EMA) of the squared norm of mean fll 1B) and mean squared deviation {|Sx (5x11) |
We now present MR? framework that adjusts margins in both logit and representation space.
Logit margin. Let ly denote the one-hot vector for class y, and z = [f 1). f(x K)]
corresponding logit vector. The 7-margin cross-entropy loss fy F x X x y R+ for input with label is:
C K.KKle
S 11)
(n.elf.x,, xf -1g In|softmax(z/77)) where "/
(I)
Ex
1|x
where E is hyper-arrameter and [7,,..., 7 specifies per-class margins, normalized such that the mean 7 = R Ex 7 č. Note that when * 1 for all & E [K]. the y-margin cross-entropy loss reduces to the canonical multi-class cross-entropy loss.
At a high level, hard classes (i.e., those with low accuracy) typically exhibit greater feature variability, as observed in both our analysis (Figure [b) [nd et al. (2024). Intuitively, our loss adaptively assigns larger margins to such classes, leveraging the well-established fact that larger margins lead to better generalization (Bartlett et al.) 1998). Therefore the proposed loss help improve the performance on hard classes. The per-class margin 7' Eq. (1) is formally justified in Corollary [ where show that it also minimizes the generalization risk bound. (The proof is deferred for clarity of exposition.)
Representation margin. Let 1/4 Σ || denote the average of the mean squared deviation. The representation margin loss la : F X
y
R is:
la ((f,x,))= 1
Σ x+E2,A(x)
exp(Il((x) o(x ((xx) 23)
Intuitively, ls encourages intra-class compactness with the parameter 2s serving as the margin to control the slack. The loss Ea can be regarded as a differentiable relaxation of & (proof in Section B.l):
Ea(,,x,0 max
max SxtEE,VNE
(1lx((x) - +(xx) 23))
which encourages the pair-wise distance within the same class no more than 23. Observe that reducing the pairwise distance is equivalent minimizing the mean squared deviation, [[]lØ(x*) o(x) = 2| (proof in Section B.2). This also explains the origin of the factor that appears in front of s. In Section 4.2 our theoretical results detail that penalizing large mean squared deviation
3

---

Published as a conference paper at ICLR 2026
helps tighten the generalization error bound. The overall objective of our MR² is:
min
Σ
(f, y) x,
FEJ
X.EED
where À > 0 hyper-parametee that controls the strength of representation regularization.
3.1 DISCUSSION WITH EXISTING WORK
This section compares our two losses (ly,ce and (z) with prior methods that modify margins at both the logit and representation levels, respectively.
Logit level. To facilitate the comparison, we first rewrite our fyy.e loss in the form:
2. K ا1 1B5
(f x. = In 1 +
Σ expl(zy - 2y)/mu} where "/ = Σの A
(5)
||a
Next, contrast our by.ce with variants of cross-entropy that add logit margins in the unified form:
La (f.,))) = In 1 +
exp A by + z - nn
(6)
v'
where Ayy adds per-class margin into the cross-entropy loss. Typical choices for A include:
(1) LDAM Cao et al.
2019]: Agy = PPy) à (2) EQL. Tan et al. (2020): Any = Ply)) (3)
Balanced Softmax Ren al. (2020): Awy' In P(w') and (4) Logit Adjustment Menon al. 2021
Ply)
P(y") Ayy' = T- In Fly) with - > O. Existing logit-margin losses are tailored to class-imbalance: their margins A vy' are functions of skewed class priors. In contrast, our loss &,ce is derived from non-asymptotic, class-sensitive bound (Section 4.2) and aims to improve balanced accuracy under balanced priors. Note that when the priors equal-Awy -Ayy' becomes constant-existiin a collapses to cross-entropy with equal margins, whereas our ly,ce still imposes class-sensitive margins.
Representation level. The most commonly techniques used to regularize the feature representation the contrastive objectives which take the form of
Rcon ( (J,,,))
E In 1+ Ex((
exp(o(x) xx(=)x(xx))))))))))) exp(x(x)T(xxx)) (x)
Σ
-x(x)
((xx))) ((x))x(xxt)))
In the unsupervised setting He et al. (2020): random augmentations of x. while negative samples x
Chen et al. (020a). positive samples x E P((x) are
E (x)) are randomly drawn from the dataset. In the supervised setting Khosla et al. (2020); Radford et al. (2021), positives share the same label as x, and negatives come from different classes. However, these methods do not incorporate margin constraints into the objective. Some methods incorporate fixed margins into contrastive objectives, either tuned on a validation set SShn [2066): et al. (2022); Barbano et al.) (2023) derived from class priors for imbalanced learning Samuel & Chechik (2021). In contrast, loss f adaptively sets the margin based on the average mean squared deviation based on our analysis of our generalization bound (Proposition I].
Discussion with Neural Collapse. Neural Collapse (Papyan et al. 2020 Behnia et al. 2023 Kini
al. 2024 Galanti al. 2021 Ma al. 2025 rang al. (021) characterizes idealized geometric patterns that emerge in the terminal phase of training, including the vamishing of within-clk class feature variance, i.e., ||و + 0, Vy E y' However, on large-scale and hard datasets such ImageNet, this variability collapse does not materialize. Consistent with recent empirical studies, we observe that within-class feature variations remain substantial and class-dependent rather than all collapsing toward zero. These deviations motivate modeling of nen-negligible feature spreads. Detailed analysis and empirical evidence are provided in Sec.A
Discussion with temperature scaling. Our logit-level loss in Eq. 5 can also be viewed as introduc- ing a class- desendeennd temperature, similar to temperature scaling used in knowledge distillation and model calibration (Xu et al.) 2020; Guo et al. (2017). where (typically global) temperature is applied to smooth or calibrate prediction confidences. By contrast, our y is spread-awar and is explicitly designed to balance margins between easy and hard classes, rather than improving calibration quality.
4

---

Published as a conference paper at ICLR 2026
20
0-1 loss
Ramp
Weggt [|wi
05-
1000
Sorted class index
Figure 2: Plot of 0-1 loss, ramp loss and py-
Figure Weight-norm |/w on ImageNet.
THEORETICAL ANALYSIS
In this section, we explain why MR² reduces classification disparity by showing that margin regular- ization the logits and representations balances the per-class error and tightens the upper bound of generalization risk. We start with some preliminaries on y-margin loss and Rademacher complexity.
4.1 PRELIMINARIES
Recall that the margin of a function f E F on a data point (x, is defined as:
75(x, v) f(x, v) max 5'*4
fx,3')
(8)
which measures the difference between the score assigned to the true and that of the runner-up.
y-margin loss Tian & Man-Cho So (022) is an extension of ramp loss which is defined as follows:
Definition 1. (7-margin loss) Let T (u) min(1, max(0, 10,11-u//h u//))- For any y hu 7kl >
the multi-class "-margin loss by F X
[0, 1] is:
4f. y) = 2 (x,)
(9)
Figure 2 provides comparison between Φ,. the 0-1 loss, and ramp loss to better illustrate their differences. Note that the 0-1 loss is upper bounded by ramp loss and 4,. The empirical risk of y-margin loss is defined as RJ)
and the expected risk is RT())- Ex x. y)). Next, give the definition of empirical 7-margin Rademacher complexity.
Ex
Definition 2. (Empirical 7-margin Rademacher complexity) Let Tt denote the indices of the instances that belong to class k. Given non-negative margins = [ти]ие[K]. the empirical 7-margin Rademacher complexity of F For a sampled dataset D is defined as:
S(x;, v)
}
RF(FF Ne
sup TEF
ΣΣΣ NEY
Ciy
(10)
7
where e (Cig)i. with ciy being i.i.d. Rademacher variables sampled uniformly from 1-1, +1}.
4.2 MARGIN REGULARIZATION PROMOTES GENERALIZATION AND REDUCES DISPARITY
In the sequel, we show that the margin regularization introduced in Eq and Eq. improves both generalization and balanced accuracy. Our key result relies on the following lemma, which provides bound on the expected risk in terms of the empirical 7-margin risk and its Rademacher complexity.
Lemma 1. (7-margin bound). Let F be set of real valued functions. Given 7 > 0, then for any 5 > O, with probability at least 5, for all E F:F
In 2
R(J) S RT()) + 4V2K
(11)
2N
5

---

Published as a conference paper at ICLR 2026
The proof is provided in Section B.3 5.8 in Mohri et al. (2018). and is similar to that Cortes et al. (2025). Furthermore, following the techniques in Theorem 5.9 Mohri et al. (2018). Lemma can be generalized to hold uniformly over all y, with only a modest overhead of In-In terms. Next, we aim to (1) bound the empirical y-margin Rademacher complexity (FF in Lemma and (2) bound the empirical risk of y-margin loss R.B. by its cross-entropy surrogate RTD in Lemma
This learning guarantee is multi-class extension of Theorem
Lemma 2. (Bound the empirical Rademacher complexity.) For non-negative O, consider {xx y) - wy y Φ(x) Wy E R2 1R2,11, S A}. Let {Il [ and (| k- denote the per-class squared norms the feature means and the mean squared deviations, respectively, as defined in Section 3 Then, the following bound holds for all F:
K N
K
1
1
+
|
(F)
(12)
7k
2
k=
The proof is provided in Section B.4. This bound is class-sensitive, as depends on the per-class squared norms of the means, per-class mean squared deviations and the class- margins. i.e.,
| ||& and 2/- Note that we use uniform bound A on the L22 norms of the classification head weight |//|2 rather than class-wise bounds, motivated by the key finding of al. (2024):
disparity does not stem from classifier bias, and the Le norms of the weights ww are already balanced, illustrated in Figure which shows |/w|]2 values from ResNet-50 trained ImageNet.
In training, we use the cross-entropy surrogate εy,ce in place of the y-margin loss Ey. Let R7.50 (f)
N Ex.ye Ex.ggell.ccc oo x, The following Lemma relates R.J.B and R.
Lemma 3. For O. Then, Vf E F, we have RT) S I R7.
The proof is provided in Section B.5
Building on Lemmas I
2 and 3
we derive the following
class-sensitive learning guarantee margin-basee cross-entropy.
Proposition 1. For 7 O, consider F {(x, y) + w o(x)
Wy E R || S A}. Let
(1lAa1)]] and [| denote the per-class squared norms of the feature means and the mean squared deviations, respectively. Then, the following bound holds for all f E F:
k=1
1 (2) R() < m
4/2AK VN
L + 1
+ O0/VVN)
(13)
This bound implies that if maintains low empirical risk R7.00 (J) often achievable with parameterized deep nets in practice - with relatively large margin values %, then it enjoys a lower generation risk, as reflected by the second (complexity) term. However, overly large Y is undesirable. Because Ey grows monotonically with y, overly increasing 7h inflates the empirical margin risk and makes the loss harder to optimize. Let the average margin budget be fixed as ē = π Σx 7/k- Under this constraint, our choice of y is determined below.
Corollary 1. (Our choice of 7) For fixed ē, the complexity term of the bound is minimized when
CK Bay
1
Yy
for all y E [K].
(14)
Σ K
12)
This follows from solving Lagrangian optimization problem, with the proof provided Section B.6
Remark. Analogous Eq.
13
bound the per-class error Rg for class w.p. at least 1 - 6:
R&()) S 1 In2
R7. + R. (F) + e(6,N),
(15)
where Rg x / 1 + || and e a low-order term (see proof in Section B.8
Corollary motivates our logit-level margin regularizer: choosing 7k & (1) + 11)x
minimizes the generalization bound and balances Eq. (15), since the feature variability term

---

Published as a conference paper at ICLR 2026
Table 1: Comparison against existing methods in reducing class-wise accuracy disparity on CIFAR- 100 and ImageNet. ResNet-32 and CLIP ResNet-50 backbones are adopted respectively.
(a) CIFAR-100
(b) ImageNet
Method
Overall
Easy
Medium
Hard
Method
Overall
Easy
Medium
Hard
ERM
70.9
84.5
71.0
56.7
ERM
75.2
91.1
78.3
56.4
LIF
69.1
83.6 (0.9)
70.1 (0.9)
53.7 (330))
LIF
74.4
90.2 (.09)
77.8 (0.55
55.2 (-12)
JTT
70.6
84.3 (-02)
70.8 (-02)
56.2 (-0.5)
JIT
74.8
90.7 (-0.4)
77.9 (-04)
55.7 (-0.4)
EQL
70.7
84.4 (-0.1)
70.9 (..1)
56.4 (-0.3)
EQL
75.3
91.3 (.0.2)
78.4 (10.3)
56.2 (-0.2)
LDAM
71.1
84.7 (+0.2)
71.2 (+0.2)
57.0 (+0.3)
LDAM
75.4
91.5 (+0.4)
78.6 (+0.3)
56.1 (-0.3)
LGM
70.8
84.0
71.4 (+0.4)
56.6
LGM
75.3
90.9 (-0.))
78.3 (+0.0)
56.6 (+0.2)
CMIC-DL
71.1
85.1
70.9
56.8
CMIC-DL
75.2
91.0 (-0.1)
78.2 (-0.1)
56.5 (+0.1)
SAM
71.0
84.6 (+0.1)
71.1 (+0.1)
56,9 (40.2)
SAM
75.6
91.4 (+0.3)
78.8 (40.5)
56.6 (+0.2)
DFL
71.3
84,8 (+0.3)
71.3 (+0.3)
57.1 (+0.4)
DFL
75.8
91.7 (+0.6)
78.6 (+0.3) 57.1 (+0.7)
SupCon
71.5
85.0 (40.5)
72.1 (+1.1)
57.5 (40.8)
SupCon
75.7
(40.3)
78.9 (40.6) 56.7 (40.3)
DRL
71.9
85.5 (+1.0)
72.5 (+1.5)
57.7 (+1.0)
DRL
75.5
91.4 (v0.3) 78.5 (v0.2) 56.6 (+0.2)
CSR
71.2
85.1 (+0.6)
71.3 (+0.3)
57.1 (+0.4)
CSR
75.1
90.8 (-0.)) 77.9 (-0.1) 56.6 (+0.2)
DRO
71.6
85.1 (40.6)
72.2 (+1.2)
57.2 (40.5)
DRO
75.9
91.0 (.:1)
79.3 (+1.0)
57.2 (40.8
FairDRO
72.0
85.7 (+1.2)
72.5 (+1.5)
57.7 (+1.0)
FairDRO
76.1
91.5 (+0.4)
79.6 79.6 (+1.3) 57.3 (+0.9)
MR² (ours)
73.9
85.9 (+1.4) 73.8 (+2.8)
61.9 (+5.2)
MR² (ours)
76.9
91.5 (v0.4) 79.7 (+1.4) 59.6 (+3.)
(vl p + | 1
appears in the numerator while Y controls the denominator. Their propor- tionality yields a more balanced contribution across classes.
Proposition justifies our representation-level regularizer: the loss la reduces the mean-squared deviations | 12. thereby tightening the complexity (second) term in Eq. (13). By suppressing large
variances in harder classes, this regularization also balances the per-class error bounds, preventing them from being dominated by high-variance classes.
4.3 GENERAL NORM BOUNDS ON MARGIN-BASED COMPLEXITY
Astute readers may wonder whether Proposition remains informative for cosine classifier models, e,g., CLIP models [Radford et al., 2021 where features are Lz-normalized, i.e., ||ø(x)ll2 = 1. In this case, the factor | Syy Th Eier [lx(x;) |12 = in the complexity term becomes constant, making it class-agnostic. Fortunately, Proposition I nnatrrally extends to other norm-based formulations, in Proposition
This allows margin regularization under any Lp norm with 2 1. 2
Proposition 2 (General bound on the y-margin Rademacher complexity). Fix conjugate expo- nents p. q E [1, oo) such that 1/p + 1/q = 1. Let Fg = ((x, y) w x(x) | Wa E Rdd |Wylle llg S 1
Vk. = TTl Eier, (1x(x)) Then, A1}- For each class & E [K], we write the average L norm as 2 for all f E Fgg the empirical 7-margin Rademacher complexity satisfies
K
(F.(F)) S C()A,
Σ Th
(16)
N
where C(p) a constant factor depends on p (its exact expression and proof appear Section B.7
Analogously to Corollary I the corresponding choice of y under Lp norm with E [1, oo]
cKr Typp
Ty.p
Σn Tkpp
2/3
(17)
For cosine classifiers, one can restore class-dependent margin by selecting non-L2 norms, e.g. 3.
5 EXPERIMENTS
Datasets and backbones. We select seven datasets: CIFAR 100 (Krihevssyy & Hinton
2009).
ImageNet (Deng et al. 2009), StanfordCars (Krause al., 2013), OxfordPets (Parkhi et al. 2012)
Flowers (Nilsback& & Zisserman, 2008). Food (Bossard et al. 2014). and FGVCAireraft (Maji et al
2013), using both CNNs (He et al. 2016a b) and Vision Transformers (Dosovitskiy al. 2021
7

---

Published as a conference paper at ICLR 2026
Table Evaluation across diverse model architecture. We perform linear probing on MoCov2 ResNet-50 using only Ey.se. CLIP models are cosine classifier models. RN: ResNet. W-RN: WideResNet. TIS: Train from Scratch.
CIFAR-100
(h) ImageNet
Model
Overall
Easy
Medium
Hard
Model
Overall
Easy
Medium
Hard
ResNet-20 MMR
68.7
83.3
70.3
53.0
RN-50 TES + MB?
71.7
88.5
74.1
52.6
70.9
84.4 (+1.)
72.2 (+1.9)
56.7 (+3.7)
74.2
89.9 (+1.4)
76.9 (+28)
55.9 (43.3)
PreAct RN-20
69.1
83.8
70.5
53.4
MoCov2 RN-50
71.1
89
73.5
50.7
MR²
71.3
84.7 (40.9)
73.2 (+2.7)
56.6 (+3.2)
+ MR (Exyce)
72.4
89.6 (41.6)
74.8 (+1.3)
52.7 (42.0)
PreAct RN-32 MR?
71.2
85.0
71.7
56.7
CLIP VIT-B/32
75.6
91.4
78.4
57.0
73.3
86.1 (+1.)
73.9 (+2.)
60.1 (+3,4)
+ MB?
77.1
91.7 (+0.3)
79.9 (+1.5)
59.8 (+2.8)
W-RN-22-10
78.4
89.6
80.0
65.4
MAE VIT-B/16
80.4
94.5
83.9
62.7
+ MRR
81.2
90.7 (+1.1)
82.5 (+2.8)
70.2 (+4.8)
+ MR?
82.0
94.8 (+0.3)
85.3 (+1.4)
66.1 (+3.4)
For CIFAR- 100, we use ResNet- (20, 32) (He et al. (016a) PreAct ResNet- (20, 32) (He et al. 2016b), and WideResNet-22-I1 (Zagorykoo & Komodakis (016) our backbones. For training from scratch on ImageNet, we use ResNer-50 (He et al. 200 as the backbone. For fine-tuning on ImageNet, we adopt four pre-trained models: MAE (He et al.) 2022 ViT-B/16, MoCov2 (Chen ct al. 2020b) ResNet-50, and CLIP (Radford et al. 2021 with ResNet-50 and ViT-B/32. For the rest datasets, we use CLIP ResNet-50.
Baselines. We compare our MR² against 14 baselines: (1) ERM (Vapnik 1991). empirical risk
minimization with cross- enssopy loss. Error-based reweighting methods: (2) LiF (Nam al. 2020
and (3) JTT (Liu et al. (2021). Margin adjustment methods: (4) EQL (Tan et al., (020) and (5)
LDAM (Cao et al.) 2019). Contrastive representation learning methods: (6) SupCon (Khosla et al.
2020) and (7) DRI. (Samuel & Chechik (021). (8) CSR (Jin et al. 2025). a regularization technique
for enhancing robust fairness. Distributionally robust optimization methods: (9) DRO (Sagawa et al.
2020) and (10) FairDRO (Jung et al. 2023). (11) Sharpness- Aware Minimization (SAM) FForet et al.
202T. Methods that promote class separability: (12) DFL (Wen et al.) 2016) (13) LGM (Wan et al.
2018
(14) CMIC-DL (Yang et al. 2025 Further details of the baselines given in Section C.I
Evaluation metrics. We report the average top-1 accuracy as the overall performance metric. To assess performance gap across classes, we follow Cui et al. (2024) to partition each dataset into three equal-sized subsets-"easy". "medium", and "hard" based on per-class accuracy ranking. For instance, classes in the "easy" group correspond the top one-third in per-class accuracy. We report the average accuracy for each of the three subsets. In Section accuracies of MR² without the partition.
C,3
we also report the per-class
Implementation details. For CIFAR-100, our training configurations are aligned with Cao et al. (2019). We follow the training setup He al. (2022) for fine-tuning MAE and He et al. (2020) for MoCov2, and adopt the configuration from Wortsman et al. (2022) for CLIP ViT-B/32 and ResNet-50. The decay rate for exponential moving average (EMA) is fixed 0.9 across all exper- iments. The hyper-parameter é in (q. is selected from (1,2,3) and À in Eq.(4) is tuned over (0.1, 0.5, 0.7, ] on validation sets. the case of cosine classifier models (e.g., CLIP), we adopt p 3 in Eq. (7). By default, we use standard augmentations such as random cropping and flipping for fair comparison. We further evaluate our MR² under advanced augmentation strategies (e.g., SunndAuument(uuuuk et al. (020)) and AutoAug (Cubuk et al. 2019)): ); the results appear in T A| All experiments are repeated three times, standard deviations are reported in Tab. [5
See Section 2 for more details.
5.1 MAIN RESULTS
Comparison against existing methods. We select ResNet-32 and CLIP ResNet-50 as backbones for CIFAR-100 and ImageNet, respectively, and compare the results with various baselines in Table We observe that: (1) Reweighting methods (LfF, JTT) underperform ERM, often degrading performance on "hard" classes. (2) Margin adjustment methods (EQL, LDAM) yield minimal gains and fail to reduce llael disparity. (3) Representation learning and robust optimization methods (SupCon, DRL, CSR. DRO, FairDRO) improve overall accuracy but fail to reduce disparity as the gain on ges"hard" classes is even lower than that on "medium/easy" classes (e.g., FairDRO and SupCon)
8

---

Published as a conference paper at ICLR 2026
Nocurncyy Improvement
Easy
Middo
Hard
Averaae
Figure 4: Relative improvements on fine-grained datasets: StanfordCars, OxfordPets, Flowers, Food and FGVCAircraft with CLIP ResNet-50. Numerical values are provided in Table TO
Table 3: Main component analysis on CIFAR-100 using ResNet-32.
Model
Overall
Easy
Medium
Hard
(a)
Baseline (fa)
70.9
84.5
71.0
56.7
(b)
fy.ce with uniform 7
70.6 (:0.3)
84.6 (+0.1)
(:0.2)
56.5 (.0.2)
(c)
fy.ce
72.8 (+1.9)
85.1 (+0.6)
72.9 (+1.9)
60.4 (+3.7)
(d)
lea + Als with 8-0
71.6 (+0.7)
85,0 (+0.5)
72,2 (+1.2)
57.6 (+0.9)
(e) la + Nls
72.3 (+1.4)
85.4 (+0.9)
73.1 (42.1)
58.8 (+2.1)
(n) MR²: l7,ce Xle 73.9 (+3.0)
85.9 (+1.4)
73.8 (+2.8)
61.9 (+5.2)
and sometimes exhibit trade-offs, e.g., on ImageNet, DRO improves "hard" classes by 0.8% while sacrificing "easy" classes by -0.1%. (4) Our MR² achieves consistent improvements across both datasets, substantially boosting "hard" class accuracy while mildly improving on "easy" one, thus reducing the performance disparity without trade-offs.
Evaluation across diverse model architectures. Table Sabble@pre presents all metrics on CIFAR-100 100 and ImageNet. For MoCov2, we follow its standard evaluation protocol to perform linear classification on frozen features by omitting the representation margin loss Lg. Recall that CLIP models employ cosine classifiers, and
p 3 in Eq. (17). Our MR² achieve consistent improvements across all metrics, indicating improved generalization. Notably, our method effectively reduces class- wise performance disparity, with substantially larger gains on "hard" classes than those "easy" classes. Our evaluation across linear classifiers, cosine classifier models, training from scratch, diverse pretraining objectives and model architectures validates the general effectiveness of MR?.
Evaluation across fine-grained datasets. Figure 4 illustrates the relative performance gains of MR² on five fine-grained benchmarks using CLIP ResNet-50, reinforcing its benefit for both balanced performance and generalization. Complete results for all metrics are in Table 10 in Section C.3
5.2 ABLATION STUDIES
Main component analysis. In Table 3 we ablate the effect of our margin regularization on logits and feature embeddings CIFAR-T00 using ResNet-32. Row (a) is the baseline trained with cross-entropy loss; Rows (b) and (c) apply our logit margin loss ce. where Row (b) uses simple uniform margin tuned on validation
(7/- Vk E [K). and Row (c) uses our proposed class-wise margins; Rows (d) and (e) apply our representation margin loss fg on top of standard cross-entropy loss fee, where Row (d) sets $ (removing feature-level margins), and Row (e) uses our full &
formulation. Row (f) is our MR?, combining both b,ce and fg. Table in Sec. C.3 provides further results on ImageNet and CIFAR- across diverse backbones.
From Rows (a,b,c), we observe that: (1) uniform margins does not improve balanced results or generalization-RRos (b) performs similarly to Row (a); and (2) our class-wise margin design in Eq. ①1 leads to clear improvements on "hard" classes and moderate gains "casy" classes. From Rows (a,d,e), we find that our representation margin loss yields notable improvements across all three subsets. In contrast, setting the representation margin "hard" classes is even lower than that on "medium" classes. Finally, Row (f) shows that combining logit and representation margin losses further enhances both faimess and overall performance.
fails reduce disparity, the gain
Effeet of ē. Figure 5 we study the impact in Eq. ( by varying its value from 0 to 5. Using ResNet-32 CIFAR- 100, plot the average accuracy for both overall and "hard" classes. A

---

Published as a conference paper at ICLR 2026
75
Overall
74
Accuracy
Hard
81
W.D.
60
A
Figure 5: Effect of a
Figure 6: Effect of X.
12
MR²
Table 4: Effect of MR? output-level and classifier-level margins. (t): higher is better. C.-
10
level: classifier-level margin. (Formal definition of the two margins in text)
16,1
MAyy:
Output-level (t)
C.-level (t)
Maay:
Avg.
Easy
Med.
Hard
100
200
300
Epoch
ERM
0.158
0.236
0.144 0.093
58.4
MR 2
0.373
0,435
0.396
0.289
81.8
Figure 7: Visualization of the evolving ||Sy||-
small é leads to insufficient regularization, while large ē imposes overly strong constraints, making optimization difficult. Both curves exhibit similar trends, with optima located around
Effeet of X. In Figure 6 we examine the effect of À in Eq. (4) by varying its value from 0 to 0.9 and reporting the accuracy of overall and "hard" classes on CIFAR-106 using ResNet-32. MR² achieves optimal overall and "hard" classes performance around À 0.5, and the performance remains stable in this region, indicating low sensitivity to A. See Figure
for the study of č and À on ImageNet.
Using different norms p. By default, applied to cosine classifiers. Table g and Table [9 (deferred to Appendix [C.3 due to space limits) report the results with various Tp norms CIFAR- 100 and ImageNet. As expected, using p 2 consistently improves both fairesss and generalization, supporting our theoretical framework.
L2 norm for all experiments except for L3 norm when
Visualization of the evolution of | Figure 7 illustrates the evolution of the average ||8 1l2 for easy, medium, and hard classes when training ResNet-50 from scratch on ImageNet. The contrast clear: MR² consistently compresses the gap between easy and hard classes throughout training, whereas the baseline exhibits steadily widening disparity.
Effect of MR² output-level and classifier-level margins. Comparing logit margins models is difficult because their logit scales may differ. We therefore evaluate the output-level margin using prediction probabilities, defined as the gap between the ground-trutt probability and the runner- up one: mx(x y) = Po(y x) - max,' zy
Po x). For classifier-level margin, we adopt the minimal pairwise angle distance among the classifier weights {w from Zhou et al. 2022): me((wa) (()))))
min.g, Z(w,, w,) = arccos [max4 w, w (///(// (wwlll/lwssll))) As in Table 4 MR² significantly enlarges output-level margins across all subsets, reduces the margin gap between
easy and hard class, and raises the classifier-level margin from 58.4 to 81.8.
6 CONCLUSION
We present MR?, theoretically grounded framework that reduces class-wise performance gap by regularizing both logit and representation margins. Built upon a novel class-sensitive learning guaran- tee, our method links generalization error to per-class feature variability and margin design. Guided by this insight, MR² assigns adaptive margins to balance generalization bounds across classes and promotes intra-class compactness. Extensive experiments across seven datasets, diverse architectures, and pre-trained foundation models demonstrate that MR² consistently improves performance on "hard" classes without degrading accuracy "easy" ones, enhancing both fairness and overall performance.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENT
This project is partially supported by the Ministry of Education, Singapore, under its Tier-l Academic Research Fund (No. 24-SIS-SMU-D440). This research is also supported by the RIE2025 Industry Alignment Fund - Industry Collaboration Projects (AAF-Id)(Awarrd 12301E0026), administered by STAR,, as well as supported by Alibaba Group and NTU Singapore through Alibaba-NTU Global e-Sustainability CorpLab (ANGEL).
REFERENCES
Randall Balestriero, Leon Bottou, and Yann LeCun. The effects of regularization and data augmenta- tion are class dependent. In NeurlPS, 2022.
Carlo Alberto Barbano, Benoit Dufumier, Enzo Tartaglione, Marco Grangetto, and Pietro Gori. Unbiased supervised contrastive learning. In ICIR, 2023.
Peter Bartlett, Yoav Freund, Wee Sun Lee, and Robert E Schapire. Boosting the margin: A new explanation for the effectiveness voting methods. The annals of statistics, 1998.
Tina Behnia, Ganesh Ramachandra Kini, Vala Vakilian, and Christos Thrampoulidis. On the implicit geometry of cross-entropy parameterizations for label-imbalanced data. In AISTATS, 2023.
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food- Food-101-man 101--mi--iinnggggg discriminative compo- nents with random forests. In ECCV, 2014.
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with abbel-iisributionn-waar margin loss. In NeurIPS, 2019.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML, 2020a.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning, arXiv preprint arXiv:2003.0420 2020b.
Corinna Cortes, Vitaly Kuznetsov, Mehryar Mohri, and Scott Yang. Structured prediction theory based on factor graph complexity. NewrIPS, 2016.
Corinna Cortes, Anqi Mao, Mehryar Mohri, and Yutao Zhong. Balancing the scales: A theoretical and algorithmic framework for learning from imbalanced data, 2025.
Ekin D. Cubuk, Barret Zoph, Dandelion Mané, Vijay Vasudevan, and Quoc V. Le. Autoaugment: Learning augmentation strategies from data. In CVPR, 2019.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with reduced search space. In CVPR workshops, 2020.
Jiequan Cui, Beier Zhu, Xin Wen, Xiaojuan Qi, Bei Yu, and Hanwang Zhang. Classes are not equal:
An empirical study on image recogmition fairness. CVPR, 2024.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. CVPR, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
Cong Fang, Hangfeng He, Qi Long, and Weijie J Su. Exploring deep neural networks via layer-peeled model: Minority collapse imbalanced training. PNAS, 2021.
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-awar minimization for efficiently improving generalization. In ICLR, 2021.

---

Published as a conference paper at ICLR 2026
Tomer Galanti, András György, and Marcus Hutter. On the role of neural collapse in transfer learning. In ICLR, 2021.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In ICML, 2017.
Uffe Haagerup. The best constants in the khintchine inequality. Studia Mathematica, 7063):231-28,, 1981.
Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In ECCV, 2016b.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders scalable vision learners. In CVPR, 2022.
Gaojie Jin, Sihao Wu, Jiaxu Liu, Tianjin Huang, and Ronghui Mu. Enhancing robust fairness via confusional spectral regularization. In ICLR, 2025.
Wittawat Jitkrittum, Aditya Krishna Menon, Ankit Singh Rawat, and Sanjiv Kumar. Elm: Embedding and logit margins for long-tail learning, arXiv preprint arXiv:2204. 230000288, 2022.
Sangwon Jung, Taecon Park, Sanghyuk Chun, and Taesup Moon. Re-weighting based group fairness regularization via classwise robust optimization. In ICLR, 2023.
Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling representation and classifier for long-tailed recognition. ICLR, 2020.
Prannay Khosla, Piotr Teterwak, Chen Wang. Aaron Samaa, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. In NeurlPS, 2020.
Ganesh Ramachandra Kini, Vala Vakilian, Tina Behnia, Jaidev Gill, and Christos Thrampoulidis. Symmetric neural-collapse representations with supervised contrastive loss: The impact of rel.U and batching. ICIR, 2024.
Vladimir Koltchinskii and Dmitry Panchenko. Empirical margin distributions and bounding the generalization error of combined classifiers. The annals of statistics, 2002.
Jonathan Krause, Michael Stark, Deng, and Fei-Fei. 3d object representations for fine-grained categorization. In CVPRW, 2013.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images, 2009.
Boqi Li and Weiwei Liu. Wat: improve the worst-class robustness in adversarial training. In AAAI, 2023.
Evan 7. Liu, Behzad Haghgoo, Annie S Chen, Aditi Raghunathan, Pang Wei Koh, Shiori Sagawa, Percy Liang, and Chelsea Finn. Just train twice: Improving group robustness without training group information. ICML, 2021.
Chuang Ma, Tomoyuki Obuchi, and Toshiyuki Tanaka. Neural collapse in cumulative link models for ordinal regression: An analysis with unconstrained feature model. arXiv preprint arXiv:2506.0528 05801, 2025.
Xinsong Ma, Zekai Wang, and Weiwei Liu. On the tradeoff between robustness and fairness. In NeurlPS, 2022.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint 13X6v:1306.51:: 2013.
12

---

Published as a conference paper at ICLR 2026
Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar. Long-tail learning via logit adjustment. In ICLR, 2021.
Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of Machine Learning. The MIT Press, 2018. ISBN 026201825X.
Junhyun Nam, Hyuntak Cha, Sungsoo Ahn, Jaeho Lee, and Jinwoo Shin. Learning from failure: De-biasing classifier from biased classifier. In NeurIPS, 2020.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In Indian Conference on Computer Vision, Graphics & Image Processing, 2008.
Vardan Papyan, XY Han, and David L Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. PNAS, 2020.
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In CVPR, 2012.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021.
Jiawei Ren, Cunjun Yu, Shunan Sheng, Xiao Ma, Haiyu Zhao, Shuai Yi, and Hongsheng Li. Balanced meta-softmax for
visual recognition. In NeurlPS,
Alessandro Rinaldo. Advanced statistical theory.
https: www. stat. cmu. edu/
Fari
pd 2018.
Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust neural networks. ICLR, 2020.
Dvir Samuel and Gal Chechik. Distributional robustness loss for long-tail learning. In ICCV, 2021.
Kihyuk Sohn. Improved deep metric learning with multi-class n-pair loss objective. In NeurIPS. 2016.
Minseok Son, Inyong Koo, Jinyoung Park, and Changick Kim. Difficulty-wwar balancing margin loss for long-tailed recognition. In AAAI, 2025.
Jingru Tan, Changbao Wang, Buyu Li, Quanquan Li, Wanli Ouyang, Changqing Yin, and Junjie Yan. Equalization loss for long-tailed object recognition. In CVPR, 2020.
Lai Tian and Anthony Man-Cho So. Computing d-stationary points of p-margin loss svm. AISTATS, 2022.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. JMLR, 2008
Vladimir Vapnik. Principles of risk minimization for learning theory. In NeurlPS, 1991.
Weitao Wan, Yuanyi Zhong, Tianpeng Li, and Jiansheng Chen. Rethinking feature distribution for loss functions image classification. In CVPR, 2018.
Yandong Wen, Kaipeng Zhang, Zhifeng Li, and Yu Qiao. A discriminative feature learning approach for deep face recognition. In ECCV, 2016.
Mitchell Wortsman, Gabriel Ihaarco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannanch Hajishirzi, Ali Farhadi, Hongseok Namkoong, and Ludwig Schmidt. Robust fine-tuning zero-shot models. In CVPR, 2022.
Kunran Xu, Lai Rui. Yishi Li, and Lin Gu. Feature normalized knowledge distillation for image classification. In ECCV, 2020.
En-Hui Yang, Shayan Mohajer Hamidi, Linfeng Ye, Renhao Tan, and Beverly Yang. Conditional mutual information constrained deep learning for classification. TNNLS, 2025.
13

---

Published as a conference paper at ICLR 2026
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo.
Cutmix: Regularization strategy to train strong classifiers with localizable features. In ICCV, 2019.
Sergey Zagoruyko and Nikos Komodakis, Wide residual networks. In BMVC, 2016.
Xiong Zhou, Xianming Liu, Deming Zhai, Junjun Jiang, Xin Gao, and Xiangyang Ji. Learning towards the largest margins. In ICLR, 2022.
Beier Zhu, Kaihua Tang, Qianru Sun, and Hanwang Zhang. Generalized logit adjustment: Calibrating fine-tuned models by removing label bias in foundation models. In NeurIPS, 2023.
Beier Zhu, Jiequan Cui, Hanwang Zhang, and Chi Zhang. Project-prob--gggeeen Efficient fine- tuning for group robustness. In CVPR, 2025.
14

---

Published as a conference paper at ICLR 2026
CONTENTS
Introduction
Related Work
3
Method
3.1
Discussion with Existing Work
Theoretical Analysis
4.1
Preliminaries
4.2
Margin Regularization Promotes Generalization and Reduces Disparity
4.3
General Norm Bounds on Margin-Based Complexity
Experiments
5.1
Main Results
8
5
Ablation Studies
6
Conclusion
10
Additional Discussions
16
Missing Proofs
18
B.1
Connection between Eg and
18
B.2
Connection between Pairwise Distance and Mean Squared Deviation
18
3
Proof of Lemmal
18
4
Proof of Lemmal2
20
B.5
Proof of Lemmal
21
B.6
Proof of Corollary
22
B.7
Proof of Proposition]
23
8.8
Per-Class Error Bound
24
Additional Experimental Details and Results
25
Additional Details of Baseline Methods
25
Additional Implementation Details
25
Additional Experimental Results
26
Societal Impact and Limitation
LLM Usage Statement
28
15

---

Published as a conference paper at ICLR 2026
A ADDITIONAL DISCUSSIONS
Discussion 1: Would it be better to make y trainable after initialization according to Eq.
We do not find this beneficial. Since loss cy.ce is monotonically increasing in y, treating 7 trainable parameter would introduce trivial descent direction: the optimizer can always reduce the loss by shrinking 7, As result, 7 would systematically converge toward small values, collapsing the class-dependent margins and failing to enforce a balanced margin across easy and hard classes.
Table 5: MR² with advanced data augmentations on ImageNet using CLIP-ResNet
Model
Overall
Easy
Medium
Hard
Standard data augmentation
75.2
91.1
78.3
56.4
+ MR²
76.9 (+1.7)
91.5 (+0.4)
79.7 (+1.4)
59.6 (+3.2)
RangAug + MR
75.8
91.5
78.8
57.2
77.3 (+1.5)
92.0 (+0.5)
79.8 (+1.0)
60.1 (+2.9)
AutoAug
76.2
91.9
79.2
57.5
+ MR?
78.1 (+1.9)
92.3 (+0.4)
81.0 (+18)
61.1 (+3.5)
Discussion 2: Effectiveness of our MR² with advanced data augmentations.
In our main experiments and ablation studies, we adopt standard data augmentations, i.e., random cropping and flipping. Here, further evaluate the effectiveness of MR² under stronger data augmen- tation strategies. repoot] results using RandAug (Cubuk et al. (220)) and AutoAug (Cubuk) et al. 2019) on ImageNet with Cest-Res Again, MR consistently boosts the accuracy of "hard" classes while providing mild improvements for easy ones, thereby reducing performance disparity without introducing trade-offs.
Discussion 3: Relation to Neural Collapse.
Neural Collapse (NC, (Papyan et al. 2020 provides useful lens for understanding the geometry of deep classifiers in the terminal phase of training (TPT). Two empirical properties are particularly relevant our setting:
(NC1) Classifier collapse to an ETF structure. The classifier weights converge to equal-norm vectors whose pairwise angles are all equal, forming an equiangular tight frame (ETF).
(NC2) Variability collapse of last-layer features. Within-cla feature variance collapses and features concentrate tightly around their respective class means.
In our experiments, we observe that these idealized NC patterns do not fully emerge on large-scale datasets, such ImageNet. Specifically, equal norms in NC1 are approximately realized (which is consistent with our theoretical assumption of a uniform L2-norm bound A on the classifier weights, rather than class-dependent one), but the ETF structure of equal pairwise angles and the full variability collapse in NC2 are violated under our settings.
Our observations are consistent with the empirical study of Cui al. (0224). who adapt NC di- agnostic tools to measure feature and classifier variability. Their results show that (i) although classifier weights become approximately equal-norm, their pairwise angles do not converge to an ETF configuration "ccpuurrttioooaaa""""""a classes exhibit larger separation angles (Sec. 4.1 and Fig. 5 in Cui et al. (2024); and (ii) the within-class feature variations remain negligibe,,,, with an average ratio between variance norm and mean feature norm of Avg, [||s, / |!] ~ 0.3285, and these variations are imbalanced between easy and hard classes (Sec. 4.3 and Fig. 2 Cui al. (2024)).
The gap between the ideal NC behavior and the empirical geometry observed on datasets such as ImageNet can attributed to at least two factors. First, task hardness. The difficulty a classification problem depends on the number of classes, the sample size, and the intrinsic complexity of the visual concepts. Inspecting Fig. 6 (first row) in Papyan et al.] (2020), one observes that as task difficulty increases, the within-class feature variation ceases collapse toward zero: the magnitude of the within-class to between-class variance is on the order of 10-4 for the easiest task (MNIST), but exceeds 1 for the most challenging subsampled ImageNet setting. To further
16

---

Published as a conference paper at ICLR 2026
Table 6: Per-class | 222 and I| on MNIST. We adopt the same training configuration as in ImageNet using R8- On MNIST, the variability collapse of features (NC2) clearly emerges, with Avg, |||| / = 7.5 x 10-4, In contrast, this phenomenon does not occur on ImageNet, where Avg, ||||| y 2 A = 0.3285.
class id
3
4
8
10
0.0041 6.6982
0.0032
0,0048
0,0051
0,0056
0.0063
0,0040
0.0056
0.0056
0.0063
6.7474
6.5227
6.8819
6.7275
6.9866
6,4091
6.4091
6.7820
6.6932
illustrate that NC2 tends to appear only on simpler datasets, we repeat the same analysis on MNIST, using the same training configuration in ImageNet but with ResNet-18 18 backbone. The results appear in Table 6; On MNIST, the variability collapse of features (NC2) does emerge: the ratio Avgy ||| i is approximately 7.5 x 10-4. However, recall that ImageNet this ratio is much larger, Avg, ||| ا [|]] 0.3285, indicating that NC2 does not hold in this harder, large-scale setting. Moreover, Papyan et al. (2020) use only 600 training samples per class roughly half of the full ImageNet training size yet still observe substantial residual within-class variability. This suggests that non-vanishing ||yy|| is expected for large-scale, high-complexity datasets.
Second, practical regularization. Modern training pipelines routinely employ regularization strate- gies that were not considered in Papyan et al.] (2020) and that hinder complete variability collapse. Early stopping prevents networks from entering the extremely long TPT required for NC to fully emerge, since training is typically stopped according to validation performance rather than continued for many additional epochs. In addition, strong data augmentation substantially increases the effective difficulty of the task by injecting extra intra-class variation, while Papyan et al. (2020) train without augmentation.
Together, these factors naturally inhibit complete within-class variability collapse in contemporary large-scale settings, and justify modeling and regularizing the residual, class-dependent spreads ||š, as we do in this work.
Discussion 4: Applying our MR² slass-imbalanced settings.
In the main experiments, we omit evaluation under class-imbalane settings, because our goal is to disentangle two fundamentally different sources of class- wise performance disparity: (1) imbalanced class numbers (2) imbalanced class feature spreads (margins). Evaluating on imbalanced datasets would entangle these effects and obscure the specific phenomenon we aim to study.
For class-imbalanee scenarios, extensive prior work already provides well-established solutions. These methods typically attribute class-wise disparity to shifted decision boundaries and thus in- troduce classifier-level corrections (often termed decoupled learning in long-tail recognition), such as freezing the backbone and retraining or reweighting classifier weights (Kang et al. 2020), or Mitably,MMeoon
adjusting classifier biases/margins (Menon et al., 2021). ). Notably, Menon et al. (2021) show that consisttn solution for balanced error is achieved by subtracting the log class prior from the predicted logits (the logit-adjustment method, LA).
These classifier-level techniques are orthogonal to our approach and can be combined with MR? in a modular way. To demonstrate this, we incorporate LA into MR² and evaluate the hybrid method on the standard long-tail benchmark CIFAR- 1O-----00-I--100 (Cao et al. (2019) using ResNet-32. The results in Table 7 show that incorporating MR² with the imbalanced-Iearme learniin method LA further improves overall accuracy and reduces performance disparity.
Table 7: Applying MR² to CIFAR-100 100-IIB--0..
Model
Overall
Easy
Medium
Hard
ERM
37.7
69.6
38,9
5.4
LA
41.9
66.9
43.9
15.6
LA MR.
44.2
68.1
46.3
18.9
17

---

Published as a conference paper at ICLR 2026
B MISSING PROOFS
B. CONNECTION BETWEEN Ea AND Eg
In this section, we prove that
(ff In 1+
E exp(Il((x) o(xx) 23)
(18)
x+ED,\(}
a differentiable relaxation of (s(,x,,)) = max
0. max Et Mx]
(1lx(x) xxxx* 23)
(19)
Let 2j = (1x(x) - (((x())1) ) 28, where j E [m]] and [Dx\(x)] Denote 2m+1 = 0. We can
rewrite la and lg
m++1
(,,)),)) = In
Σ exp(z1)
L.lsxx1))
max Zj jE[m+1]
(20)
According to the property of LogSumExp, we have:
m+1
(s( = max 2j In exp( max je[m+1]
2ÿ) - In
exp(2j) = =f((fx,,).
(21)
je[m+1]
Therefore, lg is a smooth approximation to the f..
B.2 CONNECTION BETWEEN PAIRWISE DISTANCE AND MEAN SQUARED DEVIATION
For random vectors (x. y) independently drawn from some distribution, let μ E[x] denote the mean and define the expected squared deviation as ||s|| [ 1) Then
[|[| y [2 = [| [x 1s) (y -
(22)
= [ (x - (4) + B - 1))1 2F[(( 11) T 14)
(23)
2X[][(x plll = 2|
(24)
First equality in Eq. (24 follows from the independence of (x (1) and (y 11).
B.3 PROOF OF LEMMA 1
Restated Lemma (Lemma 7-margin bound). Fet be a set of real valued functions. Given Y O, then for any > with probability at least 1 5, for all F:
In 2
R() 5 (0))
(25)
2N
Proof. Let G be a set real valued functions that map labeled instances (x, y) into the 7-margin loss:
G = (xx,x) Cy IS
(26)
Since G maps into [0. 1]. applying Theorem 3.3 from Mohri et al. (2018) yields that, with probability at least 1 1-6 8, the following holds for all g E G:
N
In
E [[((
Σ g(x,, vi) + (g) + 3/
(27)
N
2N
1=1
18

---

Published as a conference paper at ICLR 2026
By substituting the definition of g, obtain
N Σ 1=1
In 2
E (1,(fx,,,)) S
vi)
+ +31
(28)
N
2NN
Recall the definitions of the empirical and expected 7-margin risks: R2(1)= & Di= (J, X, vi) and (J) = Ex,yle (J, x. w)]- Using the fact that the 0-1 loss is upper-bounded by 7-margin loss, we have R()) < RJ (f). Thus, applying Eq. (88 we obtain
R(ʃ) - RT(1) + (g) +31 2N
(29)
Next, we aim bound (g) using (FF RD( can be written as:
1
U, x v.)
RD)
E
sup
(30)
N
FEF
N
sup
0, 1 TH.
(7/ (xx,
(31)
N
FEF
i=1
N
75 Xx+ Mi)
E
sup
(32)
N
where a = (o:); with o; being i.i.d Rademacher variables sampled uniformly from (-1,1}. Eq. 32 holds from the Lipschitz continuity of Ty. We define the function W as:
T's 7g(xx, V.)
f(x, Ue)
max,' tv. f(x;,1 1/)
wi(f)
(33)
You
Then, for any f. F, have
1
[D,5) v.(f))
(f((x;)))
max f(x,, v/)) - (f'(x,,))) -mxx (xxx,,)))
(34)
Yyu
*/#hh
1
max (f yi) (x(x,,))) + (f'(x,,
f(x;,
(35)
Yy.
1/ AV:
2
<
max [x 11) - -x',
(36)
yEy
2
[[(x;,)) v) - (xx,,))
(37)
Yuu
DEV
2
[f(x.. - f'(x,, 1)
(38)
Yo:
vEY
Let ay f' v)), by f(xi,y)), and 3² arg max,y ag', Eq.35) holds from the sub- additivity of the maximum:
max f'(x,y')) max (xx,v') Ogy #y.
max f(x,, v') ay- by < max max by:). (39) y g'#2.
y *'
v Zy..
We have Eq. 38 because of the Li-L2 norm inequality for K-dimensional x vector: ||x||1 S 7x.
/ K | From (8). it is evident that V is 2VK -Lichitzz with respect the || ||2 norm. Thus,
by the vector contraction lemma (Lemma of Cortes et al. (2016)), we have:
}
1 No
E sup |EF
Σ a (.(())
2/ N
E
sup
ΣΣΣ I vey
Cj
((xx,))
222K
fEF
(40)
Combining Eq.440 and Eq. 29
we arrive at (5)
19

---

Published as a conference paper at ICLR 2026
B.4 PROOF OF LEMMA 2
Restated Lemma (Lemma
2 Bound the empirical Rademacher complexity). For
negative y > O, consider F [x, u) Y w, ((x) Wy E 1R,l22 S AJ. Lei {ll and
{ denote the per-class squared norms of the feature means and the mean squared deviations, Then, the following bound holds for all f E F:
respectively, as defined in Section
K N
1 + ||
(D(F)SA A
Σ
(41)
Proof.
1
f(x.: V)
(FF
E
sup fer
Eigl
(42)
N
7kk
k=liEla BEY
wy ((x,)
E
sup
Elyy
(43)
ww 11
7k
k= DEY
1
GiH ((x;)
E
sup IwwllaSA
(44)
N
M
7k
DEY
1= SEI
1
(xx;)
E
sup
|| l12
Cry
(45)
N
(w, 11aSA
7Kk 112)
Jim Tx
A N
E
Σ
cyy
(xx,)
(46)
7k
vEy
1 lE
A
K
(bxx)
Σ
E
Eiy
(47)
N
Tik
VEJ
k=1ieh
A
((x;)
Σ
E
Σ
Giy
(48)
N
7kk
1E Tx
A
E
(49)
N
7676
DEy
KA
K
Eier (xx(x)111
(50)
N
V
K
K
p 12 + ||S
AV
(51)
N
7
Eq.
45 holds by Sauchy-Schwarzine inequality. We have Eq. by Jensen's inequality, i.e., B|| [|2] <
E
X
Eq.
49
49
follows using the fact that E(ese]] 0 for i of / and E(eres) 1. Given the
20

---

Published as a conference paper at ICLR 2026
class- baas-balaaced setting, i.e., [1x|
k
Eq holds by the following equality:
Σ
11x(x;
Σ ((0(x;; (m)) jax) ((Ø(x;) (xx)) + jaxe)
(52)
1
Σ [x))(x;)) Pull + Σ
i1 + Σ
[(((x;;) (4a)) mx)
(53)
it
sEL
iElt
=0
= 1Txll [&
[1(((x;)) fa + [Tl | l
(54)
VELe
N K
(11S 1 + (xx111
(55)
B.5
PROOF OF LEMMA 3
Lemma (Log-sum-ew) exp dominates the ramp). For every y > 0 and all R,
Φ (u) = min(1, m0ax(0,1 )) S loga (1 + expl exp(--])
"
?
Proof. Let g(u) denote the RHS: g(u) logz (1 + -xxp-=))) is strictly decreasing and convex
-xp(-u/7) because g' (u) 7 lu2 1+exp(-u/11 and g g"(u) م Feeppp-uu/1) O. We verify the
exp(-9/7)
inequality in three disjoint regions:
u 2 y. (u) = 0 by definition, while g(u) > 0. Hence the inequality holds.
u < 0. Now 2. (u) 1. Since g(u) is decreasing, g(u) 2 g log, 2 = 1.
u uu<. y. In this interval, let ア/ (0, 1). We aim to prove that
(tt)) = log, (1 - exp( +}} (1 - t) = g(u) (u) 2 0. Vt E (0,
(56)
Let ((t) Ite be the sigmoid function, the derivative of htt) is:
1
1
(t)
h((t) 2n
(57)
1n
ln2
Since (t) is monotonically increasing, h'(t) > (1 - .n2 Therefore, h(t) is strictly increasing for (0,1). Since ln2
a
= 1
1
= 0.279 > 0.
2
h(0)
log,
0. we have
h(t) > 0, Vt E (0,
Combining the three regions proves the claim.
Restated Lemma (Lemma For non-negative y O. Let Ex 4) and Then, the following bound holds for all f E F.F
R2 = 1x (f,,,
RB(1) < 1n
())
(58)
Proof. Let z [f(x, 1), 1), fxx K))T denote the logit vector. Using our Lemma loga + exp( }). and the fact that logz(a) = In(a)/ l2()
4 that (,, <
( x, v) =2,, (x, 1)) S logg 1+exp(- 2r 9)
75(xx, y)
1n2 In + exp(
7'v
(59)
21

---

Published as a conference paper at ICLR 2026
Because max, 0₁ S In E, exp(a,), we have
In expl 7,(x,3), 1)
(x. Y
max
2:1:
1 In
Σ
exp(
Zy
- 2k
(60)
kzy
Yv
7v
A
Adding inside the logarithm on both sides yields
3,(xx1))
x(f,x,0)=
In
exp(
(61)
In2
Zzy
In
1
exp(
(62)
In 2
Y/
by cc (J: x,
(63)
In 2
1 Since by wiint-wise upper-bounded by In2 y,e it follows immediately that
1
RT(1) - 1n2
Rba (s).
(64)
B.6 PROOF OF COROLLARY
Restated Corollary (Corollary Optimal choice of y). For fixed c, the complexity term of the bound is minimized when
εK (K( 11 +
1|))
for all y E [K].
(65)
Σ k==
+ 18)
Proof. To minimize the complexity term in Eq.
13 under the total margin constraint c, we solve the
following constrained optimization problem:
K
112
||
7 argmin
S.L.
Σ 7k = c.
(66)
>=/m|]
AE[K]
Denote =1 11x + ||su > O. It is equivalent to solve the following Lagrangian problem with
multiplier v E R.
K
mim IaxX
O'k
vl
Σ - c)
(67)
7k
kE[A]
Setting the partial derivatives w.r.t. % to zero gives
20 y +v=0
79 = 20ky )
(68)
Thus every optimal yy is proportional to ay
1/3
Since k ΣE 7/k = C, we have
εK = (2 12 a
2z (2 ax)
(69)
čK
It follows directly that:
eKa
7v =
(70)
Σ k
22

---

Published as a conference paper at ICLR 2026
B.7 PROOF OF PROPOSITION
We begin with lemmas leading to the proposition of our general bound.
Lemma 5. If X1,,.., X,, are independent uub-Gassiian K.V. with variance proxies of...., d then 2 - - 1i=1 X; is sub-Gaussian with variance proxy Σi=1 o7.
Lemma 6. (Maximal Inequality Rinaldo (018)) Suppose Xi,..,, Xn are sub- Sab-Gaussian EVS with
variance proxy g², Then
E E|max X,) S log(n) [E|n]
(71)
Restated Proposition (Proposition 2 General bound the y-margin Rademacher complexity).
Fix conjugate exponents 9 E [1, oo such that 1/p + 1/9 = 1. Let Fg = ((x, v) + wy ((x)
Wy R ||wlll I Aqg. For each class ls E [K], we write the average L norm
(72)
TR
iET
Then, for all f E Fg. the empirical 7-margin Rademacher complexity satisfies
v /2log(d)
K
22
if = 00,
k=
7
(l((F)) S
23
Ag
K
=k=1
i 0o,
(73)
V
7/2
K AgV
Σ k=1
22p
$f1$$52
N
7k
where I is the gamma function.
Proof. Using Hölder's inequality and the bound on ||wllp we have
K
f(c,, 8)
RF(FF)
=
sup SEF₂
(74)
N
Tke
k= EE DEy
K
Ciy ((x;)
sup Iwolls
W
(75)
N
DEy
Rim IET
K
o(xx)
sup
/wullo
ΣΣ
Eisy
(76)
| A
7k
BEy
k=1iEl
KA
E (xxx)
(77)
N
iE
We use the shorthand
(xx))
E RR
(78)
7
and uy as the j-th coordinate of u:
E o(xx))
Uj
(79)
TE
Using Lemma and the fact that Rademacher variable is ssub-Gaussian, the proxy variance of of My
K
Diel, o(xx)3
K Eien 1lx(x;)113
N
K
<
(80)
7
2
7k
K
T
k=
23

---

Published as a conference paper at ICLR 2026
The inequality holds for any 2 1, as for any vector E R and Proof of upper bound, case p = 0c.
[d], we have Ivyl $ ||vl|p-
KA
(FF) -
E
||]|u||bo]
(81)
N
KA
E max
suj
(82)
N
Thus, by Maximal Inequality Lemma (Lemma 6
(2(F)) V2log(d)
K
K
(83)
N
Proof of upper bound, case 1 S p < 00.
By Khintchine's inequality Haagerup (198T), the following holds:
N
E
||||||p]]
u
S Cpp Σ
of)+
Cp
(84)
jeld
jEId
Therefore, have:
K
R3(F)) S C.A,A
Σ
(85)
N
where Cp = 1 for 2, and
Cp = 24 1
p+1
(86)
π
for p > 2.
B.8 PER-CLASS ERROR BOUND
We bound the per-class error R2 for class is w.p. at least 8: +
Ra(f) < m D.
4 7 RR (F) + E),
(87)
where Rig
M + Six _h2 and is low-order term. It suffices express R4 (F) as the empirical Rademacher complexity restricted to class A:
R (F)
Ee
sup Σ JEF
cxf(xi, (x)
(88)
Ix
iET
where Rademacher variables.
Proof. By definition,
1
Ee
sup
Σ S. ((xi)
(89)
Nx
[iwllASA
Ee
sup
w
Σ (g((s))
(90)
N
||w]|ISA
NET
24

---

Published as a conference paper at ICLR 2026
Applying Sauuch-Scchwaz,
(Fx)
E - Ns
Σ
(xi
(91)
ii
Now note that each coordinate of = Eiel (x((x;) is Rademacher sum with variance proxy Eier, )xxx)). Thus,
Ee Sellul8 =
Σ
Σ iElN
(x(())))
(92)
1=1
Σ
Σ
(xxx)}
(93)
1=1
Σ
(x(
(94)
NE
Taking square roots,
(Fx S
1x(x1111
(95)
N
Finally, decomposing into squared mean and variance gives
|0(x))1 | + 1|xx
Therefore,
RR (F) < A AV AVIl + 1|s.112.
C ADDITIONAL EXPERIMENTAL DETAILS AND RESULTS
C.1 ADDITIONAL. DETAILS OF BASELINE METHODS
We adopt the official implementations of the baselines to ensure fair comparison, with the exception of CSR Jin et al. (2025), ) DRO Sagawa et al. (2020), and FairDRO Jung et al. (2023), where differences in settings require modifications, as described below. CSR designed to enhance robust fairness under adversarial attacks. To align with our clean-data training setting, remove the adversarial
perturbations during training. DRO and FairDRO target worst-group performance. In our setting without group labels, we substitute group labels with class labels.
C.2 ADDITIONAL IMPLEMENTATION DETAILS
Implementation details for CIFAR-100. Following et al. (2019), we use simple data augmenta- tion: each image is padded by 4 pixels on all sides, then a 32 32 crop is randomly sampled from the padded image its horizontal flip. The model is trained using stochastic gradient descent with momentum 0.9, weight decay x 10 a batch size of 128, and for 300 epochs. The initial Icarning rate is set to 0.1 and follows cosine decay schedule. Experiments performed using single NVIDIA A100-40GB GPU.
Implementation details for training from scratch ImageNet. We train the ResNet-50 in 300 epochs with a cosine and batch size of 128 on NVIDIA A100-40GB GPUs. The initial learning rate is set to 0.1 and the weight decay is set to 10-4 We use the SGD optimizer with momentum of 0.9.
Implementation details for CLIP ResNet-50 and CLIP ViT-B/32. Following the fine-tuning configurations of WiSE-FT Wortsman al. (2022),
the default PyTorcch Adam W optimizer for
25

---

Published as a conference paper at ICLR 2026
Table 8: Effect of different Lp norms on CIFAR-I00 with ResNet-32. improvements across all metrics for different p values.
We observe consistent
p
5
9
Overall
73.4 (v2.6)
73.7 (+2.9)
73.9 (+3.1)
73.7 (+2.9)
73.2 (+2.4)
Easy
85.8 (+1.3)
85.9 (+1.4)
85.9 (+1.4)
85.9 (+1.4)
85.8 (+1.3)
Medium
73.5 (+2.5)
73.7 (42.7)
73.8 (+2.8
73.6 (+2.6)
73.4 (+2.4)
Hard
60.7 (+4.0)
61.4 (+4.)
61.9 (+5.2)
61.5 (+4.8)
60.2 (+3.5)
Table 9: Effect of different norms ImageNet with CLIP ResNet-50. For the cosine classifier models, we note consistent improvement across all metrics for different p values.
P
5
Overall
76.5 (+0.9)
77.1 (+1.5)
76.9 (+1.3
77.1 (+1.5)
76.8 (+1.2)
Easy
91.6 (40.2)
91.7 (40.3)
91.5 (+0.1)
91.7 (+0.3)
91.5 (40.1)
Medium
79.3 (40.9)
79.9 (+1.5)
79.6 (+1.2)
79.9 (+1.5)
79,4 (+1.0)
Hard
58.8 (+1.8)
59.8 (+2.8)
60.0 (+3.0)
59.7 (+2.7)
59.6 (+2.6)
10 epochs with weight decay of 0.1. We set an initial learning rate of 3 x 10-5 and follows cosine decay schedule with 500 warm-up steps. We use a batch size of 512 and minimal data augmentation as in Radford et al. (202T); Wortsman et al. 2022). Since CLIP models are cosine classifier models, we set p = 3 for Lp norm. Experiments are performed using four NVIDIA A100-40GB GPUs.
Implementation details for MAE ViT-B/16. Following the fine-tuning setup of He et al. (2022), we use the default AdamW optimizer with base learning rate of 10-3 for 100 epochs. The batch size set to 1024 and cosine decay learning schedule with warmup epochs. For fair comparison, avoid advanced augmentation strategies (e.g., RandAugment, CutMix and AutoAug) and use only standard augmentations such as random crop and flip. Experiments are performed using eight NVIDIA A100-40GB GPUs.
Implementation details for MoCov2 ResNet-50. Using a pretrained MoCov2 He et al. (2020) backbone, train supervised linear classifier frozen features. The classifier is optimized with SGD using learning rate of 30, momentum of 0.9, and a batch size of 256. Training runs for 100 epochs, with the learning rate decayed by factor of 0.1 at epochs 60 and 80. We apply standard data augmentations, including random cropping and horizontal flipping. Experiments are performed using single NVIDIA A100-40GB GPU.
C.3 ADDITIONAL EXPERIMENTAL RESULTS
Effect of different norms on CIFAR-100 and ImageNet. Table and Table report the results with various Lp norms on CIFAR- 100 and ImageNet. As expected, using p 2 consistently improves both balanced accuracies and generalization, supporting our theoretical framework.
Details numerical results for StanfordCars, OxfordPets, Flowers, Food and FGVCAircraft. In To we present all the metrics for five fine-grained datasets. We observe significant improvement on "hard" classes and moderate gains on "casy" classes, indicating the effectiveness of our MR² in improving balanced accuracies and generalization.
Additional main component analysis. In Table [1 we present additional analysis the effects of the logit margin loss &,.e and the representation margin loss lg. The results show that both regularization components effective in enhancing balanced accuracies and overall performance.
Numerical results for hyper-parametery ē and The detailed values are in Table 12 and Table I3
Per-class accuracies of our MR². As datasets such CIFAR-100 and ImageNet involve large number of classes, making tabulated reporting less clear, we use CIFAR- 10 for clarity. Additionally, train models 10% of the CIFAR-10 10 training set to ensure meaningful class-wise variation, as training on the full CIFAR- dataset leads to near-saturated performance. The results in Table [4] clearly indicate that our MR² method improves performance most significantly for the harder-to-
26

---

Published as a conference paper at ICLR 2026
Table 10: Fairness improvements on StanfordCars, OxfordPets, Flowers, Food and FGVCAircraft.
Metric
Cars
Pets
Flowers
Food
Aircraft
Overall
75.8
89.2
96.7
81.4
34.5
CLIP
Easy
92.0
97.4
100
91.0
65.7
RN-50
Medium
77.9
90.6
99.3
82.9
29.4
Hard
57.4
79.7
90.9
70.2
8.4
Overall
76.4 (40.6)
90.4 (+1.2)
97.3 (+0.6)
81.9 (+0.5)
35.5 (+1.0)
Easy
92.2 (40.2)
97.8 (+0.4)
100 (+0.0)
91.1 (+0.1)
65.8 (+0.1)
MR?
Medium
78.4 (+0.5)
91.7 (+LI)
99.9 (+0.6)
83.3 (+0.4)
30,0 (+0.6)
Hard
58.7 (+1.3)
81.6 (+1.9)
92.2 (+1.3)
71.3 (+1.1)
10.8 (+2.4)
Table 11: Additional main component analysis. "RN", "P", "C" and "M" stand for ResNet, PreAct, CLIP and MAE, respectively.
(a) CIFAR-100
(b) ImageNet
Model
Overall
Easy
Medium
Hard
Model
Overall
Easy
Medium
Hard
PRN-32 PRN-20 RN-20
68.7
83.3
70.3
53.0
MVTT-B CVTT-B CRN- 50
75.2
91.1
78.3
56.4
Ey.
70.3
84.1 (408)
71.7 (+1.4)
55.8 (+28)
75.9
91.2 (40D))
78.6 (40.3)
58.1 (+1.7)
Ey.ce + X6g
70.9
84.4 (+1.1)
72.2 (41.9)
56.7 (43.7)
Eye + A6z
76.9
91.5 (40.4)
79.7 (+1.4)
59.6 (43.2)
tee
69.1
83.8
70.5
53.4
Ea
75.6
91.4
78.4
57.0
Ey.o
70.2
84.2 (+0.4)
71.8 (+1.3)
55.1 (+1.7)
Eyyon
76.3
91.5 (40.1)
78.7 (40.3)
58.9 (+1.9)
Ey. + Ae.
71.3
84.7 (10.9)
73.2 (+2.7)
56.6 (+3.2)
Ey,or + Ae,
77.1
91.7 (40.3)
79.9 (+1.5)
59.8 (+2.8)
tee
71.2
85.0
71.7
56.7
80.4
94.5
83.9
62.7
Eyyme
72.7
85.8 (40.)
73.4 (41.7)
59.2 (42.5)
Eyyon
81.3
94.6 (40.1)
84.5 (40.6)
65.0 (42.3)
E,,00
Aea
73.3
86.1
73.9
60.1
Eyyou
Ae,
82.0
94.8 (+0.3)
85.3
66.1
classify classes. For example, the two worst-performing classes, bird and cat, see improvements of 8.2% and 9.1% in accuracy, respectively. This aligns with the goal of reducing class-wise performance disparity.
Effect of é on ImageNet. In Figure we study the impact of using CLIP ResNet-50 on ImageNet. Both curves exhibit similar trends, with optima located around =
Effect of on ImageNet. In Figure we examine the effect of À using CLIP ResNet-50 ImageNet. MR² achieves optimal overall and hard class performance around À 0.4, and the performance remains stable in this region.
Standard deviations. In Table 15 we report the standard deviations across diverse model architec- tures on CIFAR-10 10 and ImageNet.
Visualization of the features. Figure 10 visualizes the effect our loss on the feature geometry of hard 110 classes. We randomly selected hard classes and extract their last-layer test features. These features are then projected to 2D using t-SNE van der Maaten & Hinton (2008). Under standard cross-entropy (Figure 1Oa), the clusters of different classes many hard classes exhibit large within-class spread. In contrast, with our MR² loss (Figure 10p), )), the features of each hard class become much more compact and different classes are better separated.
highly entangled and
D SOCIETAL IMPACT AND LIMITATION
Disparities in class wise accuracy, even under Hlss-balanced training, pose fairness risks in real- world deployments. Our work contributes toward building more trustworthy and fair machine learning systems. Although MR² focuses on class-level disparity, does not account for unfairness arising from data-related issues such as label noise and class co-occurrence.
27

---

Published as a conference paper at ICLR 2026
Table 12: Numerical values for Figure S
Table 13: Numerical values for Figure
0.5
1
3
5
À
0.01
0.05
0.1
0.3
0.5
0.7
0.9
Hard
60.4
60.8
61.6
61.9
61.2
61.0
Hard
60.7
61.0
61.7
61.8
61.9
61.6
61.5
Overall
72.8
73.2
73.7
73.9
73.4
73.1
Overall
73.1
73.5
73.9
73.9
73.9
73.8
73.7
Table 14: Per-class accuracies on CIFAR-10. A denotes the relative improvements.
automobile
truck
ship
frog
horse
airplane
deer
dog
bird
ERM
91.4
87.1
85.5
80.4
79.0
76.5
76.
70.5
70.1
62.5
MR² (ours)
92.1
89.0
87.4
83.0
82.5
78.9
79.2
77.3
78.3
71.6
+0.7
+1.9
+1.9
+2.6
+3.5
+2.4
+3.1
+6.8
+8.2
+9.1
E
LLM USAGE STATEMENT
We used ChatGPT only for minor language editing to improve clarity and conciseness. No part of the research idea, methodology, or analysis was generated by LLMs.
28

---

Published as a conference paper at ICLR 2026
Overal
77
Overall
2 78.0
Accuracy (%)
78
Accuraen Boy
60.5
Hard
80
88.0
Hard
68.5
Figure 8: Effect of c.
Figure 9: Effect of X.
Table 15: Standard deviations across diverse model architectures CIFAR-IO and ImageNet.
(a) CIFAR-100
(b) ImageNet
Model
Hard
Medium
Easy
Overall
Model
Hard
Medium
Easy
Overall
ResNet-20
0.0033
0.0029
0.0019
0.0035
ResNet-50 TFS MR²
0.0023
0.0039
0.0015
0.0028
+ MR²
0.0012
0.0027
0.0016
0.0022
0.0018
0.0029
0.0032
0.0015
PreAct RN-20
0.0016
0.0030
0.0033
0.0019
MoCov2 RN-50
0.0015
0.0013
0.0024
0.0008
+ MR?
0.0027
0.0039
0.0015
0.0024
MR² (c),,))
0.0019
0.0032
0,0011
0.0027
ResNet-32
0.0029
0.0036
0.0019
0.0032
CLIP RN-50
0.0036
0.0018
0.0039
0.0015
+ MR²
0.0019
0.0029
0.0037
0.0017
MR²
0.0029
0.0035
0.0012
0.0023
PreAct RN-32
0.0026
0.0016
0.0034
0.0039
CLIP ViT-B/32
0.0019
0.0017
0.0020
0.0028
+ MR?
0.0030
0.0012
0.0037
0.0022
MR²
0.0021
0.0033
0,0029
0.0037
W-RN-22-10
0.0025
0.0038
0.0019
0.0034
MAE VIT-B/16
0.0018
0.0022
0.0034
0.0017
+ MR?
0.0013
0.0027
0.0040
0.0022
MR²
0.0026
0.0038
0.0023
0.0035
(a) Baseline
(b) Our MR Loss
mouse
train
bee
chair
telephone
can
sunflower
poppy
crocodie
вea
whale
lamp
Figure 10: t-SNE visualizations of feature representations for randomly selected hard CIFAR-100
classes using (a) standard cross-entropy and (b) our MR?
29


Reviews:
[{"review_id": "6RnRMjdQDM", "reviewer": "Reviewer_MUKn", "summary": "In this work, the authors propose a margin-based regularization framework to mitigate performance disparities between “hard” and “easy” classes that persist even when the training data are class-balanced. They begin by empirically showing that class-wise accuracy differences correlate with disparities in feature variance and margin distribution across classes. Building on this observation, the authors introduce a modification to the cross-entropy loss composed of two complementary terms: a logit-level margin regularizer that adaptively scales per-class margins based on feature variability, and a representation-level regularizer that enforces intra-class compactness by minimizing the spread of embeddings around each class mean.\n\nThe paper then provides theoretical analysis, first deriving an upper bound on the standard cross-entropy loss via their margin-regularized formulation, and subsequently identifying the optimal hyperparameter configuration that minimizes this bound. The analysis is further extended to normalized embedding spaces—commonly used in contrastive and transformer architectures—showing similar generalization guarantees and optimal margin design.\n\nFinally, the authors conduct extensive experiments, training from scratch on smaller datasets such as CIFAR-100 and fine-tuning or linearly probing large pre-trained models on ImageNet and other fine-grained datasets. Across all cases, the proposed method improves the accuracy of “hard” classes without degrading, and sometimes even enhancing, performance on “medium” and “easy” classes. The authors also include ablations showing the robustness of their results to the choice of hyperparameters.", "questions": "**Q1:**  \nIn line 289, the authors mention that overtly increasing the value of the margin parameter is not desirable and may lead to a higher empirical margin. Could the authors expand on this point and provide a more precise theoretical explanation? I find this statement somewhat unclear. A related follow-up question is how, according to the proposed loss terms, simply reducing the inter-class spread parameter $s_y$ to zero would not already minimize the overall loss objective.  \n\n**Q2:**  \nWhy do the authors not evaluate their method under class imbalance settings? From my reading of the results, the imbalance scenario should conceptually produce a similar difference in margin distribution as the one discussed here. This relationship has been explicitly examined in the Neural Collapse literature (Fang et al., 2021). Furthermore, given that the ImageNet dataset already exhibits some class imbalance, wouldn’t the improvements observed on it indirectly reflect performance under imbalance? Am I correct to assume that classical ImageNet is indeed imbalanced?  \n\n**Q3:**  \nWere the experiments involving other architectures and loss formulations conducted only in the fine-tuning stage, rather than full training? If so, do the authors believe that training from scratch on harder benchmarks such as ImageNet with their loss formulation would yield noticeable improvements?  \n\n**Q4:**  \nRegarding the representation margin term, is this component designed to enforce a balance on the value of $\\bar{s}$—that is, to keep it small but non-zero? Could the authors elaborate on this point or clarify whether any part of their theoretical analysis addresses the optimal regime or equilibrium value of this term?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is clearly written, and the ideas it explores are relevant to our broader understanding of optimization. The theoretical outline is well presented and easy to follow, and the experimental setup is generally sound and consistent with prior work. The proposed approach makes a meaningful contribution by improving performance on hard classes without hurting the easier ones, leading to a more balanced overall accuracy across classes.", "weaknesses": "**W1)**: I believe this work, given its focus on margin geometry and embedding compactness, overlooks a closely related and highly relevant area known as Neural Collapse (Papyan et al., 2020). This phenomenon shows that in over-parameterized networks—such as those considered in this paper—class embeddings tend to collapse to a single prototype per class with maximal inter-class separation as training progresses. Subsequent works have analyzed Neural Collapse under class imbalance (Behnia et al., 2023), supervised contrastive losses (Kini et al., 2023), and in relation to generalization and transfer learning (Galanti et al., 2021). Even more recently, Neural Collapse principles have been extended to other domains such as federated learning, where margin adjustment inspired by NC has been proposed to address class imbalance (Li et al., 2025).\n\nFrom a theoretical standpoint, if we follow the Neural Collapse arguments, the regime where $ ||s_y|| \\to 0$  would eventually reduce the proposed loss to standard cross-entropy, since the representation compactness term becomes inactive and the per-class margins converge. This connection appears important but is not acknowledged or discussed in the paper. While the authors do a solid job referencing prior work on logit adjustment and class-wise accuracy imbalance, they omit this major line of research that directly studies how margin-controlled and embedding-controlled objectives influence model geometry and generalization. I would be interested to see the authors comment on this relation during the discussion period, given the conceptual overlap between their margin regularization intuition and the Neural Collapse framework. Much of the NC work struggles with how to relate the geometry of features during training to generalization and much of the theoretical upper bound for optimal geometry struggle in practice on improving test performance.\n\n**W2)**: While the authors primarily focus on the margins of hard versus easy classes, their evaluation relies solely on accuracy as the performance metric. Although the reported improvements are clear and likely related to the margin and embedding-spread–based logit adjustments, additional experimental analysis would help solidify these claims. In particular, it would be useful to visualize how the margins and the $s_y$ parameter evolve during training under their objective. Without such analysis, it remains unclear how much of the observed improvements stem from the proposed margin regularization itself versus other implicit effects of regularization.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HPK2kWL8Ws", "reviewer": "Reviewer_aWLX", "summary": "This paper proposes a Margin loss for the balanced datasets, which tries to balance the variance in hard classes by using a larger margin. The margin loss is theoretically explained by proving a bound on the Rademacher complexity. Experiments have been provided on numerous datasets such as CIFAR-100, Pets, ImageNet etc.", "questions": "Could the authors explain the difference in theoretical ideas from Cortes et al. (2025)?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Paper introduces a theoretically motivated solution to the problem.\n\n2. The paper is well structured with relevant experiments.", "weaknesses": "1. Experiments are done on an older setup: I find that the experiments are done on older SOTA setups. The newer setups, like Sharpness Aware Minimization (SAM) [R1], WideResNets, have not been considered for comparison. Hence, the performance reported for datasets like CIFAR-10 and ImageNet is much lower than the current SoTA. Further, the margin-based algorithms like LDAM, compared with MR2, perform much better when compared to SAM [R2].\n\n2. Missing Comparison: There are some contrastive learning methods that use Supervised Labels, like SupCon. As MR2 also uses lables it would be better to compare with them. It would be great to see some comparison with SotA (State-of-the-Art) frameworks in mind. \n\n3. Novelty: I find the novelty of the paper to be a little limited, as I found an existing paper that talks about balancing the feature across classes [R3]. Further, a lot of the theoretical ideas are similar to those presented in Cortes et al. 2025 [R4].\n\n[R1] Foret, Pierre, et al. \"Sharpness-aware minimization for efficiently improving generalization.\" arXiv preprint arXiv:2010.01412 (2020).\n\n[R2] Rangwani, Harsh, Sumukh K. Aithal, and Mayank Mishra. \"Escaping saddle points for effective generalization on class-imbalanced data.\" Advances in Neural Information Processing Systems 35 (2022): 22791-22805.\n\n[R3] Zhong, Ke, et al. \"Class-Center-Based Self-Knowledge Distillation: A Simple Method to Reduce Intra-Class Variance.\" Applied Sciences 14.16 (2024): 7022.\n\n[R4] Cortes, Corinna, et al. \"Balancing the scales: A theoretical and algorithmic framework for learning from imbalanced data.\" arXiv preprint arXiv:2502.10381 (2025).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NSWlLKYRd5", "reviewer": "Reviewer_5Lxp", "summary": "This study aims to address the problem of class disparity that arises in classification under balanced settings. To this end, the authors propose regularizers that control the margins in both the logit and representation spaces, assigning larger margins to more challenging classes to achieve balance in the error bound. Furthermore, the paper provides theoretical evidence supporting the effectiveness of the proposed approach through well-formulated theorems and corresponding proofs. In addition, extensive empirical evaluations across various datasets, including fine-grained datasets, as well as architectures such as CLIP, demonstrate the effectiveness of the proposed method", "questions": "**Q1** (w.r.t **W1**). Ablation studies on CIFAR-100 using ResNet-32 were conducted to examine hyperparameter tuning; however, since they are limited to a single dataset, providing additional insights could help alleviate concerns about tuning sensitivity. For instance, is there any guideline on how to adjust hyperparameter values—such as increasing or decreasing them—depending on the number of samples or labels?\n\n**Q2.** Although the proposed method shows improved performance compared to other approaches across various experiments, the class disparity still appears to remain severe. Could the authors provide a more detailed explanation of why the issue has not been completely resolved, along with potential limitations or directions for future work to address it?\n\n**Q3.** How many times were the experiments conducted with different random seeds? To address concerns about potential cherry-picking and demonstrate the robustness of the results across different random environments, please provide this information.\n\n\n**Things to improve the paper that did not impact the score**:\n\n* (lines 022-023) typo: \"demonstrate demonstrate\"\n\n* (line 800) In Eq. 39, $a'\\_{y}$ and $b'\\_{y}$ newly appear without being mentioned or defined (it seems a typo error for $a\\_{y'}$ and $b\\_{y'}$)", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "* By providing background on the class disparity problem—an increasingly critical issue in modern classification settings—and analyzing its underlying causes, this study effectively establishes the motivation for addressing this problem\n\n* The study also supports the validity of the proposed approach with solid theoretical analysis, comprising precisely stated theorems and corresponding proofs\n\n* Furthermore, extensive experiments conducted on a wide range of datasets, including fine-grained benchmarks, and on architectures such as CLIP, confirm the robustness of the proposed approach. Ablation studies (e.g., main component analysis, effect of $\\bar{c}$ and $\\lambda$) also demonstrate its effectiveness", "weaknesses": "**W1.** As illustrated in Eq. 13 (lines 286–291), there exists a trade-off between the first and second terms depending on the value of $\\gamma$. Although this trade-off is bounded through Corollary 1, further tuning of the coefficient $\\bar{c}$ is still required. This remains a tuning issue, in combination with another hyperparameter $\\lambda$, which increases the overall burden of hyperparameter tuning.\n\n**W2.** The proposed approach indirectly verifies its effectiveness in addressing the class disparity problem through performance improvement. However, it would be more convincing to more directly validate whether the margins for hard classes have indeed increased, using metrics such as the class margin introduced in [R1].\n\n**W3.** The related work section provides a thorough review of not only other works on class disparity but also long-tail learning, which is a similar area of work in that it focuses on controlling class margins. However, despite the presence of conceptually related approaches in hyperspherical learning—such as those that separately control the margins between classes and samples or adjust margins according to sample difficulty, as shown in [R1,R2]—the paper does not include a survey or discussion of hyperspherical learning.\n\n\n[R1] Zhou et al., Learning Towards the Largest Margins, ICLR 2022\n\n[R2] Son et al., Difficulty-aware Balancing Margin Loss for Long-tailed Recognition, AAAI 2025", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "N0IT3jMGPB", "reviewer": "Reviewer_dj7P", "summary": "The paper presents MR$^2$, a novel objective function for training DNNs with reduced inter-class performance disparity and improved generalization. The MR$^2$ objective consists of a margin scaled cross-enropy loss and representation regularization term for encouraging intra-class feature compactness. It's shown that the proposed methods is connected to logit margin loss and contrastive learning and can tighted generalization bound. Extensive experimental results on image classification are provided to validate the effectiveness of the proposed method.", "questions": "1. Would it be better to make $\\gamma$ trainable after initialization according to Eq. 1?\n2. In Fig. 5, how can $\\bar{c}$ be 0? This will lead to a \"dividing by 0\" error in Eq. 1.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper addresses a largely overlooked issue: inter-class performance disparity under class-balanced setting.\n2. The paper provideds extensive theoretical analysis to justify the proposed method by linking it to $\\gamma$-margin loss and generalization bound.\n3. Comprehensive experimental results are presented.", "weaknesses": "1. The loss in Eq. 1 can be regarded as cross-entropy loss with class-dependent temperature scaling. Temperature scaling has been studied a lot in knowledge distillation and model calibration. It's better to discuss and compare with some of those methods such as [1].\n2. In experiments, it's better to compare with more methods that promote class separability, such as [2], [3] and [4].\n3. The statement \"hard classes ... to such classes\" in Line 137~140 seems to assume that $||\\hat{\\mu}_y||$ is roughly constant across all classes so the margin mainly depends on $||\\hat{s}_y||$. It's better to provide evidence to support that $||\\hat{\\mu}_y||$ roughly stays constant across classes.\n4. All experiments on ImageNet are based on fine-tuning. It's better to provide some results for training from scratch on ImageNet, even just for small models such as ResNet-18 and ViT-tiny.\n5. It's mentioned that advanced augmentation strategies are avoided in experiments. However, it's better to show results with those augmentations to show the effectiveness of the proposed method when combined with modern augmentaion settings.\n\n[1] K. Xu et al. Feature Normalized Knowledge Distillation for Image Classification. ECCV 2020.\n[2] Y. Wen et al. A discriminative feature learning approach for deep face recognition. ECCV 2016.\n[3] W. Wan et al. Rethinking Feature Distribution for Loss Functions in Image Classification. CVPR 2018.\n[4] EH. Yang et al. Conditional mutual information constrained deep learning for classification. TNNLS 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)