Fine-Grained Class-Conditional Distribution Balancing for Debiased Learning
Authors: Miaoyun Zhao
Qiang Zhang
Venue: ICLR 2026 Poster (2026)
Keywords: group robust classification
spurious correlations
short-cut mitigation
distribution balancing
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Achieving group-robust generalization in the presence of spurious correlations remains a significant challenge, particularly when bias annotations are unavailable.
Recent studies on Class-Conditional Distribution Balancing (CCDB) reveal that spurious correlations often stem from mismatches between the class-conditional and marginal distributions of bias attributes. They achieve promising results by addressing this issue through simple distribution matching in a bias-agnostic manner. 
However, CCDB approximates each distribution using a single Gaussian, which is overly simplistic and rarely holds in real-world applications. 
To address this limitation, we propose a novel Multi-stage data-Selective reTraining strategy (MST), which describes each distribution in greater detail using the hard confusion matrix.
Building on these finer descriptions, we propose a fine-grained variant of CCDB, termed FG-CCDB, which enhances distribution matching through more precise confusion-cell-wise reweighting. FG-CCDB learns sample weights from a global perspective, effectively mitigating spurious correlations without incurring substantial storage or computational overhead.
Extensive experiments demonstrate that MST serves as a reliable proxy for ground-truth bias annotations and can be seamlessly integrated with bias-supervised methods.
Moreover, when combined with FG-CCDB, our method performs on par with bias-supervised approaches on binary classification tasks and significantly outperforms them in highly biased multi-class and multi-shortcut scenarios.

Published as a conference paper at ICLR 2026
FINE-GRAINED CLASS-CONDITIONAL DISTRIBUTION BALANCING FOR DEBIASED LEARNING
Miaoyun Zhao
Key Laboratory of Social Computing and Cognitive Intelligence Dalian University of Technology
Liaoning, China
Qiang Zhang f
Key Laboratory of Social Computing and Cognitive Intelligence Dalian University of Technology Liaoning, China
ABSTRACT
Achieving group-robust generalization in the presence of spurious correlations re- mains significant challenge, particularly when bias annotations unavailable. Recent studies on Class-Conditional Distribution Balancing (CCDB) reveal that spurious correlations often stem from mismatches between the class-conditional and marginal distributions of bias attributes. They achieve promising results by ad- dressing this issue through simple distribution matching in bias-agnostic manner. However, CCDB approximates each distribution using a single Gaussian, which is overly simplistic and rarely holds in real real-world applications. To address this lim- itation, we propose a novel Multi-stage Lata-Selective re reTraining strategy (MST), which describes each distribution greater detail using the hard confusion matrix. Building on these finer descriptions, we propose a fine-grained variant of CCDB, termed FG-CCDB, which enhances distribution matching through more precise confusion-cell--ee -ise reweighting. FG-CCDB learns sample weights from global perspective, effectively mitigating spurious correlations without incurring sub- stantial storage computational overhead. Extensive experiments demonstrate that MST serves as a reliable proxy for ground-truth bias annotations and can be seamlessly integrated with bias-supervised methods. Moreover, when combined with FG-CCDB, our method performs on par with bias-supervised approaches on binary classification tasks and significantly outperforms them in highly biased multi-class and multi-shoricut scenarios.
INTRODUCTION
Neural networks trained with standard Empirical risk minimization (ERM) Vapnik 1998) often suf- fer from spurious correlations: shortcuts that are predictive of the target class in the training data but irrelevant to the true underlying classification function LaBonte LaBonte et al. (2023b). Samples exhibit-
ing such spurious correlations typically dominate the training distribution and form the majority groups, while samples with different or conflicting correlations constitute the minority groups Rad- ford et al. (2021). This imbalance across groups is also referred to biased data, which results in poor ERM performance on the minority ones, sometimes even no better than random guessing prevalent in many high-stakes applications, including toxic comments identification Borkann et al. (2019), medical diagnosis Castro et al.] (2020), and au- tonomous driving Pourkeshavarz al. (2024). where both robustness and fairness are critical but overlooked by conventional methods. Take the traffic sign classification task as a vivid example Liu et al. (2023), in which the training data exhibits a strong bias: 99% of stop signs appear in red, whereas stop signs of other colors are rare and constitute minority group Beery et al. (0018)
Shah et al. 22020). Spurious correlations
"mi "miacyun3 9zhac @ gmail.com. Code: https::/ / h//gggttbbb//g///// com www.a//Maayunzaac/E..ooom zhac
1 "Corresponding author, zhangg@dlut.cdu.co

---

Published as a conference paper at ICLR 2026
Consequently, the classifier relies on the red color as shorteut for recognizing stop signs, ignoring the textual "stop" features. This leads to biased predictions and poor generalization when the color cue is absent or misleading. (Arjvssy et al. (2019); Geirhos et al. (2020); [Beer al. (2018). These challenges underscore the urgent need to develop classification methods that remain reliable across diverse data subgroups, especially in the presence of spurious correlations.
One of the most effective strategies for improving robustness against spurious correlations is to re- train models using group-balanced subsets derived from bias annotations Kirichenko et al. (2023). However, given the massive scale of modern datasets, manually labeling bias attributes is often pro- hibitively expensive, which motivates the development of annotation-free alternatives. Recent stud- ies have shown that models trained with naïve ERM tend to favor biased solutions. which generalize poorly to minority groups - offering a "free lunch" for bias modeling Pezeshki et al.) (2024); Puli] er al. (2023). Accordingly, various methods have been developed to identify misclassified samples as belonging to minority groups. These approaches either explicitly highlight such samples or im- plicitly simulate group-balancing during the debiasing process to enhance group robustness LaBonte ct al. (2023a) Pezeshki et al. (2024); Li et al. (2023)) Liu et al. (202T). However, they often rely on empirically chosen hyperparameters to control the upweighting of minority groups, which can easily lead to overemphasis on these groups and, in turn, degrade performance on the majority ones. As a result, held-out annotations often required for effective hyperparameter tuning. Recent research on Class-conditional distribution balancing (CCDB) Zhao et al. (2025) reveals that spurious cor- relations arise from the mismatches between class conditionl and marginal distributions (usually caused by bias cues), and addresses it by reweighting samples to minimize the mutual information between bias cues and class labels without hyperparameter searching. However, CCDB performs coarse distribution matching by treating each distribution as single Gaussian, which rarely holds in real-world applications. In practice, instances within the same class often exhibit multi-modal dis- tributions due to hidden bias cues. Thus, this coarse matching fails to capture intra-class variations, leaving residual spurious correlations unaddressed.
To resolve these limitations, we propose a fine-grained distribution matching technique based on CCDB, termed Fine-Grained Class-Conditional Distribution Balancing (FG-CCDB), which achieves stronger mitigation of spurious correlations without relying bias annotations. Our ap- proach is developed from two key perspectives: (i) Fine-grained distribution description. Inspired by the "free lunch" phenomenon in ERM where models tend overfit to spurious correlations we introduce a Multi-stage data- Selective reTrainiig strategy (MST) for bias characterization, which capable of tackling multi-shorteuts by relate the hard confusion matrix to bias-aligning and conflict- ing partitions, and employing a multi-stage, data-selective retraining strategy to enhance the relia- bility of these partition assignments, which iteratively refines predictions from the overfitted model. This process yields a confusion matrix that approximates the ground-truth group partition when spurious correlations arise from a single shortcut. (ii) Fine-grained distribution matching. Build- ing on the confusion matrix identified by MST, we extend CCDB into a fine-graines formulation, termed Fine-Grained Class-Conditional Distribution Balancing (FG-CCDB). It provides a discrete multi-modal approximation of both class-conditional and marginal distributions, enabling precise mode-wise alignment and thus more thorough mitigation of spurious correlations than the original CCDB. The main contributions of this work are follows: (i) We propose an annotation-fre bias exploration method with multi stage refinement, based on model overfitting, which generalizes beyond singular shorteut and serves a reliable alternative to human annotations. (ii) We intro- duce FG-CDDB, a lightweight and scalable debiasing method that enables fine-grained mode-wise reweighting and is well-suited for multi-class classification and tensive experiments show that our method matches or surpasses bias-supervised baselines, achieving strong performance without requiring bias annotations.
mitigation. (iii) Ex-
RELATED WORK
The related work is structured around the two core aspects of our contribution.
2.1 BIAS EXPLORATION
Primary approaches define bias as texture Bahng et al. (2020), background Venkataramani et al. 2024). or image style Li et al. (2025) features presumed irrelevant class labels. These methods
2

---

Published as a conference paper at ICLR 2026
often rely on tailored architectures training schemes to detect specific bias cues Hong & Yang
(202I) but generalize poorly to unknown biases. To overcome this, recent data-driven strategies
interpret bias group imbalance latent substructures. Some methods, like JTT Liu et al. (202I),
LfF[Nam et al. all (020). and RIDGE|Pezeshkiee et al. (2024), identify bias via consistently misclassifie
(hard) samples under ERM. Others rely on model disagreement, e.g. DebiAN [i al. (2022h) iteratively trains a bias "discoverer" alongside main classifier, XRM XRM Pezeshki et al. (2024) uses
a pair of biased auxiliary models generate pseudo group labels across the training set, DDB Ciranni et al. (2025) utilizes diffusion model to generate bias-aligned data, which amplifies the bias reliance. Other methods, such as GEORGE [Sohoni et al.] (2020), apply unsupervised feature clustering to decompose each class into latent subgroups. Few of these methods conduct thorough evaluation on the quality of bias prediction. Another trend leverages vision-lange lsiiun- models (e.g..
CLIP Radford et al. (202T) to infer explainable bias attributes Jain et al. (2023), Kim et al. (2024); Wiles et al. (2022). though they are constrained by predefined vocabularies and may miss unexpected biases.
2.2 BIAS MITIGATION
Bias annotation dependent. With the assistance of bias annotations, a variety of methods have been developed to mitigate spurious correlations. GroupDRO Sagawa et al. (2020) groups data based on class and bias annotations and optimizes for the worst-group performance. DFR Kirichenko et al. (2023) improves robustness by retraining only the last layer using a small, balanced validation set. MAPLE Zhou et al. (2022) uses measure based validation set with explicit bias annotations to reweight training samples. LISA Yao et al. (2022) utilizes data augmentation technique to encourage bias-invariant features. Though effective, relying on costly bias annotations limits their scalability in real applications.
Bins-cong samples dependent. To mitigate spurious correlations without manual annotation, recent studies often leverage disagreements among auxiliary models to identify bias-conflicingg sam- ples and focus learning on them Naa et al. (2020); Liu et al. (2023); Chu al. (202T); ); Liu et al. (202I). To better identify bias-conflicting samples, LaBonte et al. (2023)) proposes to split the training data and applying early stopping for effective bias-conflicting detection. ulAA [sirig- otis et al. (2023b) leverages pretrained self- self-super models to extract bias-relevant information. HaR et al. (2024) uses the Generalized cross-entropy loss Nam et al. (2020) to amplify model coneeters-oftert requiring split of an- notated subsets-an their simple binary partitioning into bias-aligned and bias-conflicting samples insufficient to fully capture the structure of bias, ultimately limiting generalization.
bias. However, these methods rely on empirically tuned
Bins-agn.stic. Beyond bias- aware techniques, several bias ttas-agnostic approaches have emerged, mo- tivated by diverse perspective:(uuii et al. (2023); Jain et al. (2024). MASKTUNE[Asgart et al. (2022), ExMap Chakraborty et al. (2024), and DaC Noohdani et al. (2024) reduce reliance on spurious fea- tures by identifying thas-related regions via heatmaps, which restricts their applicability to the image domain. Stable learning approache<Zhane
et al. 202I): Yu et al. (2023) treat spurious correlations as effects of unknown confounders and attempt mitigate them by decorrelating features, though this is difficult to achieve in practice. GERNE Asaad et al. (2025) leverages the gradient differences between two batches identify a debiasing direction, along which the model is optimized. CCDB Zhao et al. (2025) seeks to mitigate spurious correlations by minimizing the mutual information between spurious features and class labels via distribution matching. Although effective, its coarse matching strategy limits generalization performance.
3
OUR METHOD
Our work builds on the existing method CCDB, which attributes spurious correlations to distribution mismatches and addresses them through sample reweighting without requiring bias annotations. However, CCDB performs distribution matching in a relatively coarse manner by modeling each distribution as a single Gaussian. To enable more accurate alignment-and thereby more effective spurious correlations elimination-we propose fine-grained extension. Specifically, we introduce a stage data-selective retraining strategy (MST) that characterizes bias structure via the hard confusion matrix, allowing for discrete multi-modal deseription of each distribution. Based on these modai distributions, we develop Fine- Grained Class Claas-Coodttional Distribution Balancing (FG-CCDB), which performs alignment at the mode level.
3

---

Published as a conference paper at ICLR 2026
Random Split
Biased prediction
Split
Prediction
worst accuracy
-
cMNIST 5pt
cCIFAR.10-5p
WaterBinds
Celebok
Bast
Rind Aucinn
repear)
repearl repeat? repeats
(2) hiased learning
Bius onhancement learning (maltiplc times)
He/WGA along Ete repestation of (b)
The MST framework progresses from an initial partition with limited minority group coverage (a) to more complete identification in stage (b). (c) WGA under different bias-capturing qualities.
Figure
We consider the task of predicting a label y E V. (1....,C) C} based on an input 2: E X A' C Rd Following prior work, we define shortcut as an explainable attribute (e.g., color, background) that is spuriously correlated with class labels and highly predictive, and focus on a more general setting in which each data point (ar, y) may be associated with one more shortcuts. Motivated by Tsirigotis et al. (2023a), we use an auxiliary biased model to predict the bias label & E S, which share the same label space y, i.e., [S] 12. Note that goal is for to capture general and harmful bias information that humans may not preconceive Li et al. (2022)), rather than only phys- ically interpretable attributes. The value s represents spurious signals that an ERM model prefers over core features and that consequently cause evaluation failures. & = denotes all spurious cues that cause samples from other classes be misclassified as class These cues may correspond to interpretable shortcuts, combinations of multiple shortcuts, or entangled, uninterpretable patterns. By combining s and y, we partition the dataset into modes M S V, which exactly corre- sponde to the hard confusion matrix. When the bias corresponds to a single shortcut, this reduces to conventional group partitions. To distinguish our data partitioning approach from traditional group- based methods, we refer to the partitions derived from the confusion matrix as modes. Accordingly, diagonal entries represent majority (bias-aligning) modes, and off-diagonal entries correspond to minority (bias-conflictingg modes. With the confusion matrix, one infer a discrete multi-modal approximation of both the class-conditional and marginal distributions over bias information. The goal is twofold: (i) to train biased model that can effectively explore the underlying bias cues; (iz) to train a debiased model that invariant to bias information and achieves uniform performance across all modes.
3.1 BIAS EXPLORATION THROUGH OVERFITTING
In this section, we introduce the proposed multi-stage data-selective retraining (MST) technique and demonstrate its compatibility with existing bias-supe methods. It is well established that, in the presence of spurious correlations. ERM tends to overfit to majority groups in training data, leading to an over-reliance on bias cues and poor generalization to minority groups. Recent studies LaBonte et al. (023b); [Tirigotis et al. (2023b) have made preliminary attempts to exploit this over- fitting behavior to mitigate spurious correlations, revealing that the predictions of overfitted models are strongly aligned with bias cues. Furthermore, Lee et al.] (2023); Ciranni et al. (2025) find that removing bins-conflicting samples improves bias prediction and point out that, principle, if all bias-conflicting samples were removed, one could train a bias-capturing model that provides ideal learning signals for debiasing. Inspired by these insights, we propose multi-stage framework for refined bias prediction, which further leverages model overfitting and serves as an approximate sub- stitute for human annotations. The overall framework consists of two basic stages (Figure (b)): initial bias learning and bias enbancement learning. The first stage extracts primary bias patterns, while the second amplifies them in the model's predictions, yielding a reliable bias predictor.
Initial bias learning. Given an accessible train dataset D == {(x M1 J.V of N samples and C classes. Following prior works Zhao et al. (2025); [ LaBonte et al. (202bb): : Pezeshki et al. (2024), we explore bias information by randomly splitting 'D into two subsets D, and D2, where D1 contains a fraction 7 of the original data (Figure [a). left). We then perform naive ERM on D, to train a biased model fe,, which typically performs well on majority groups but poorly minority groups. Unlike prior Larkse et al. (2023b); Pezeshki et al. (0024), which use 95% of the data for biased training, our goal is to maximize the model's 's alignment with bias cues to better reveal underlying spurious correlations. As demonstrated in our experiments (Figure rright),. a smaller y proves more effective for bias exploration, with 7 10% emerging sweet spot. Since the data is
4

---

Published as a conference paper at ICLR 2026
M
Margiaal
weights
M
12.3
Way
P
neee"
0000
Figure 2: (a) The framework of our FG-CCDB (b) The joint distribution J of bias and class labels estimated by our method. (c) Toy example to show how FG-CCDB differs from group balancing.
randomly split, some samples from minority modes inevitably participate in training, which weakens the model's tendency align its predictions with bias cues (Figure Ta) right). To counteract this effect, we introduce a subsequent amplification stage.
Bias enhancement learning. Amplifying bias in model predictions is non-trivial. Our key idea is to guide the bias prediction model to focus exclusively on majority modes, i.e., to construct a training subset D1 that contains little to no samples from minority modes. This idea of removing bias-conflicting samples has been shown to effectively amplify bias in prior work Lee et al. (2023); Ciranni et al. (2025). Such setup forces the model to overfit to the majority modes and align more strongly with the corresponding bias cues, thereby behaving like bias predictor and exhibiting near-zero generalization ability on minority modes. To achieve this, we introduce a data selection procedure based on the predictions of Jo, forming an extremely biased subset Dj, which more biased model is trained. Specifically, for each sample (æx, y1) in D. we infer the softmax output as h, fe, (x Within each class, select the top 3 fraction of samples (3 E [0, 1J) with the highest prediction confidence (measured by h:;), and aggregate them to form Dj- In our experiments, we find that 3 50% offers a stable and reliable choice. Since fo, is biased toward majority modes, the high-confidence samples are more likely to come from those modes. Consequently, D' filters most minority mode instances and is thus more biased than D,. (Figure [b) left). We then train a new biased model Seg using naïve ERM on Dj. The resulting model serves as the final bias predictor produce bias labels for D. Combined with the target class labels, the resulting hard confusion matrix yields estimated mode partitions over the space (S] x (J%. which can serve as proxy for group annotations bupe-supe methods (Figure bb) right).
Notably, the "Bias enhancement learning" stage can be repeated to further improve bias prediction accuracy. Only the biased model from the final repetition used to generate bias labels. As shown in the experinents(Figure[]] ()), single iteration already achieves performance comparable to existing methods, while further iterations lead to gradually converging performance with diminishing gains.
3.2 FINE-GRAINED CLASS- COASS-CONDITIONAL DISTRIBUTION BALANCING
In this section, we present the Fine-grained Class-Conditional Distribution Balancing (FG-CCDB) approach. With the hard confusion matrix obtained via MST, FG-CCDB improves both the quality of distribution matching and the efficiency of sample reweighting.
The original CCDB proposes to mitigate spurious correlations by directly minimizing the mutual in- formation between bias cues and target classes, which is achieved by aligning each class-conditional distribution with the marginal distribution, while simultaneously balancing class proportions generalization to traditional class balancing technique. Specifically, the objective is to minimize:
Lu [ Hyy) = Ep_(v) DKL v
(p) log Pw (1)
where 2 denotes the latent feature (with gradients detached) extracted by the biased model prior to the fully connected layer, which predominantly captures bias cues. w denotes the sample weights to be optimized, and Dkt.[-l-) refers to the Kullback-Leibler divergence Kullback Leibler (155]) )
Since the true distributions associated with z and y are unknown, CCDB approximates them ing single Gaussian, which is insufficient for complex data with inherently multi-modal structures. Moreover, CCDB's sample-level reweighting requires storing and processing feature representations for the entire dataset, incurring additional computational cost.
Our work adopts the same objective Equation I
To overcome the aforementioned limitations, we derive a discrete multi-modal approximation of both class-conditional and marginal distributions from the hard confusion matrix, which enables localized, mode-wise distribution matching and leads
5

---

Published as a conference paper at ICLR 2026
to more accurate and scalable reweighting. As shown in Figure 2 (a), we represent the confusion matrix as M E RCC' where Mr, denotes the number of samples belonging to mode (s. = (i,5), Thus, the joint distribution over (z, y) is approximated with a discretized version over modes (s, v). which is characterized by matrix E RCXC with J& M. represents the probability of N
sample belonging to mode (s, (i, j)), N is the total number of training samples. By design,
we define a class-conditional distribution matrix P E RCxC such that the j-th column P.,. encodes
p(zly = j), and a marginal distribution vector E IRC that captures p(z). Both P and g are
computed directly from J as follows:
def
def R q Σ J.s
p<zly j) ~ P
p(z)
E.
Figure 2 (b) shows the joint distribution matrix estimated by our MST across four datasets. Clear spurious correlations are observed, evidenced by the strong diagonal elements (aligned along the yellow line), which indicate a high dependency between labels and bias cues. To eliminate these spurious correlations and minimize equatior[ [
we introduce mode-level weighting parameter W
CCC to adjust each class-conditional distribution so that it aligns with the marginal distribution. A straightforward solution for W is,
q: P
W WJ
for i,j = ,,,..,
Note, equation 3 achieves exact distribution matching, i.e., W @ P... = q. meaning that all class- conditional distributions are reweighted to align with the same marginal distribution q. where @ denotes the Hadamard product. For given mode (s,y) = (i,j), assuming uniform contribution from its samples, the corresponding sample weight is,
Wij Wi.j M..
for 1,.......,
(4)
Note that beyond distribution matching, Equation 4 jinherently solved the class imbalance issue: the mode with more samples gets smaller weights. As result, FG-CCDB simultaneously minimizes both terms in Equation []] These weights are subsequently used during debiasing to reweight training samples according to their mode identities.
is worth noting that our distribution matching fundamentally differs from conventional group bal- ancing (see example in Figure(ce)): (i) Unlike group balancing, which aims to reduce differences across all entries in the mode matrix, our method focuses solely on aligning the classccondiiional distributions-i.e.. reducing the variation among columns in P-while preserving intra-column im- balance. This allows for more flexible training by merely minimizing mutual information rather than enforcing strict equality. (ii) By minimizing the divergence between conditional and marginal distributions, our method and CCDB implicitly achieve "covariate balance" from the view of causal inference, specifically, by finding reweighting that makes the confounder (bias) independent of the treatment (core feature), ultimately forcing the statistical model to rely solely on core features for inference Neal (2020). (iii) Simple scale balancing between majority and minority modes is insufficient for generalization, as majority modes typically exhibit greater diversity. Our method applies more aggressive reweighting strategy. For example, the ratio between the largest and smallest mode weights in FG-CCDB reaches 1000, compared to just 100 in conventional group bal- ancing. Compared to CCDB, our sample reweighting approach offers several key advantages: (i) It performs distribution matching across multiple localized regions defined by the confusion matrix, enabling more precise alignment and more thorough removal of spurious correlations; (ii) The sam- ple weights computed efficiently in closed form, without requiring any iterative optimization; (iti) Instead of assigning weights individually to each sample, FG-CCDB assigns a shared weight to samples within the same mode, resulting in negligible computational and memory overhead.
After completing MST and FG-CCDB, we train a debiased model fo by incorporating sample weights into the data sampling process using PyTorch's forch.utiis..de WeighnedRandomSampler following Zhao et al. (2025). Unless otherwise specified, we refer to the entire procedure as FG- CCDB for brevity. A full algorithm of the proposed method is provided in Appendix A

---

Published as a conference paper at ICLR 2026
Table I: Classification performance on real-world datasets. We report the average test accuracy(%) and std.dev. over 5 random seeds. Best bias-agnostic results in bold.
Bias label
Waterbirds
CelehA
CivilComments
Mothods
Train
Val
L.d.
WGA
1...d
WGA
1.d.
WGA
GroupDRO
Yes
Yes
93.50
91.40
92.90
88.90
84,2
73.7
DFR
Yes
Yes
94,20=0a
92.90=a2
91.30
88,30
87 87.2=003
70.1
No
Yes
97.50
75.20
86.00
77.20
68.2
50.3
JTT
No
Yes
93.60
86.00
88.00
81.10
83.3
64.3
LCC
No
Yes
90.50a11
88. 10
70.30
SELF
No
Yes
93.00
83,90
79.10=2.1
DaC.
No
95.3+a0
92.3+04
91.4
81.9+a7
ERM
No
No
97.30
72.60
95.60
47.20
81.6
66.7
MASKTUNE
No
No
93.00=07
86.40=19
91.30
78.00
ul.A
No
No
91.50200
86. 10:15
93.90 33.90
86.50
XRM
No
No
90.60
86.
91.0
88.5
83.5
70.1
DebiAN
No
No
90.80
78.19
84.0
52.9
DDB
No
No
90,34
GERNE
No
No
89.88=0047
74.24
13.10=a2
CCDB
No
No
92.59
90,48:025
90.08 89.71
85.27 89.22.9.1
83.60=0.11 86.99_0.14
75.00
FG-CCDB
No
No
92.50
90.5610024
78.52
Table 2: Results on UrbanCars.
Table 3: Ablation study on four datasets.
Methods
Bias lubel
LD.
Gap due to shortcuts(()
Methods
Waterbirds
CelebA
MNNIST CIFARIO
Train Val
BG CoObj BG+CoObj
Lid.
WGA
Lid.
WGA
Lid.
L.d.
GroupDRO
Yes
Yes
91.6
10.99
-3.6
-16.4
GroupDRO
93.50
91.40
92.90
88.90
84,20
57.32
JIT
No
Yes
95.9
-8.1
-13.3
-40.1
GrouDRO
DaC
No
No
98.17
3378
-9.78
-58.58
-MST
90.82 88.47
88
85.21
84.07 -uz 55.73
ERM
No
No
97.6
-15.3
-11.2
-69.2
DFR
94.20 92.90
91.30
88.30
ExMap
No
No
-5.9
-9.9
-30.7
DFR-MST
92.53 coe 91.49 cmm
88, 10
85.87 am
DebiAN
No
No
98.0
14,99
-10.5
-69.0
FG-CCDB
92.50.co
90.56 ao
89.7
89.22 cm
98.21
78.06
DDB
No No
86.39
-1.85
-0.52
-0.12
FG-OCDB
91.54 91.76....
93.14 14. 89.09
98.26
78.53
FG-CCDB
No
No
92.98
4.17
-7.37
.4.9
ssu
4 EXPERIMENTAL RESULTS
In this section, we demonstrate the effectiveness of our method from five perspectives: (i) We con- duct experiments real-world binary classification benchmarks with either single multiple short- cuts, such as Waterbirds /hhou et al. (2022), CelebA Zhou et al. (2022). CivilComments Koh et al.) (202I). and UrbanCars Li et al. (2023b) validate the overall effectiveness of our method; (ii) We further evaluate our method on challenging multi-class datasets, including cMNIST Li et al. (022a) and ICIFARIO Hendrycks & Dietterich (2018) to assess its robustness under highly biased conditions: (iii) To evaluate the reliability of the bias cues explored by MST, we compare them with ground-trutt truth bias annotations and analyze the effects of repeating the "bias enhancement learning" procedure; (iv) we conduct ablation study to demonstrate that each technical component (MST and FG-CCDB) makes distinct and independent contribution to the final performance. (v) Finally, analyze the effects of hyperparameters (7 and (3) on the performance of MST.
For all datasets, we adopt the same vain--valiation-tess split following Liu et al. (202I); Tsirigotis let al. (2023b) for fair comparison. Results are averaged over 5 random seeds, and for each seed, the best-performing model (the one with the highest worst-class accuracy on the validation set) is selected Tsirigotis et al. (2023b). Unless otherwise stated, we repeat the "bias enhancement learning" process three times for FG-CCDB. See the appendix for the detailed experimental setup.
Compared methods. To demonstrate the superiority of method in addressing spurious correla- tions and its potential to serve as an approximate substitute for bias-super methods, we compare it with both bias bias-supervised and bias-agnostic techniques. GroupDRO (Sgaawa et al. (2020) and DFR Kirichenko et al. (2023) are fully bias-supervised during both training and validation, and serve as strong baselines. Nam et al. (2020). JTT Liu al. (202T), LC et al. (2023), DaC Noohdani et al. (2024). and SELF LaBonte et al. (2023b) rely on pseudo-bii bias supervision during training, but L
still require bias annotations during validation achieve optimal performance. In contrast, ERM, uLA Tsirigotis et al. (2023b), MASKTUNE, Asgari et al. (2022). XRM Pezeshii et al. (2024). De- biAN, ExMap, DDB Ciranni et al. (2025), GERNE Asaad et al. (2025), and CCDB Zhao et al.
2025 similar to our method, are entirely bias-agnostic throughout both training and validation.
4.1 BINARY CLASSIFICATION WITH A SINGLE OR MULTIPLE SHORTCUTS
The results on real-world binary classification with single shortcut are shown in Table Although i...d performance reflects overall accuracy, can mask disparities across groups. In contrast, worst-

---

Published as a conference paper at ICLR 2026
Table 4: Results on cMNIST and cCIFARIO with various bias-conficting ratios in the training set. The test accuracy(%) is averaged over 5 random seeds. The best results are indicated in bold.
Bias label
cMNIST
eCIFARIO
Methods
Train Val
0.5%
1%
2%
5%
0.5%
1%
2%
5%
GOO
Yes
Yes
63.12
68.78
76.30
84,20
33.44
38,30
45.81
57.32
LIF
No
Yes
52.50:243
61.891499
71.03124
80.571384
28.57:1130
33.07
39.91
50.27:1.56
LC
No
Yes
71.25+3317
82,25
86,21+102
91.16*097
34.56=a09
37.34=009
47.81
14.55
DaC
No
Yes
53.24
75.02
87.60
94.70
21.01
28.01
36.56
51.06
ERM
No
No
35.1913.49
52,091288
65,86.3359
82.17.10.4
23.08.1125
25.82.003
30.0610.11
39,42.00.4
uLA
No
No
75.13=0.8
41
84.79+1.10
92.79
34.39=1.14
62.49=0774
63.88+1107
74.49+0.58
GERNE
No
No
77.25=a017 83,98+0.6
=0.41
90.98+0.13
39.90=aa8
45.60=0023
50. 19.198
56.53++0.2
CCDB
No
No
83.20.2.17
87.95
91,02.0.28
96.37.0.25
55,07
63.28
67.78.0078
74.6410.4
FG-CCDB
No
No
89.02:0045
94.9320.17 96.18 98.21 20.m 55.28
64.66 71.69.0031 78.06220:0
group accuracy (WGA) directly measures robustness by focusing on the most challenging subpop- ulations. With bias annotations available during both training and validation, GroupDRO and DFR demonstrate strong generalization performance on the worst group, serving as challenging upper bound. In contrast, methods that only use bias annotations during validation show a bit inferior per- formance. The situation becomes more challenging when access to bias annotations is not permitted. In this case, existing bias-agnostic methods consistently fall short of the supervised upper bound on at least one of the datasets. Remarkably, SELF, CCDB and our method surpass the supervised upper bound on CivilComments by a large margin. This is because they apply stronger upweighting to the minority groups/modes. Among all compared methods, including those with full supervision, our method consistently achieves the best or competitive WGA across all three datasets, highlighting its effectiveness in eliminating the need for human annotations.
T presents the results on UrbanCars with multiple shortcuts: background (BG) and co- occurring object (CoObi). The in-distribution accuracy(I.D. Acc) and gap-related metries adopted from Li et al. (See appendix for details). The BG/CoOb//GG+COOh Gap is the drop in accuracy between mean and cases when only the BG/CoOhj/BG+Cooh is uncommon. A smaller drop indicates better generalization. On average, BG+CoObj is the most challenging one and most compared methods suffer a significant drop on GroupDRO can mitigate multiple shortcuts; however, they require access to labels of both shortcuts. Although DDB shows the smallest over- all drops across all bias-confficting scenarios, its base I.D. Ace is the lowest among all compared methods. Overall, our method consistently achieves the best balance between high I.D. Acc and small drops compared to other bias-agnostic methods (particularly on the challenging BG+CoObj generalization). It performs comparably to, or better than, methods that rely on bias annotations. These results confirm that our approach provides a general framework for handling multi-shortcut scenarios. Please refer to Appendix [ more details.
4.2 MULTI-CLASS CLASSIFICATION UNDER EXTREME SPURIOUS CORRELATIONS
In this section, of our method under challenging multi-class settings with extreme spurious correlations. For each, vary the ratio of bias-conflicting samples in the training set to control the strength of spurious correlations and evaluate performance on a completely unbiased test set. Following Tsirigotis et al. (2023b), the bias-conflicting ratios are set to (0.5%, 1%, 2%, 5%} for both datasets, where 0.5% indicates an extremely biased scenario. The generalization accuracies are reported in Table 4 We observe that: (i) On both datasets, our method consistently achieves the best performance. In partic- ular, it outperforms the second-best method by large margin on cMNIST; (it) On eCIFAR10 10, the improvements become more pronounced as the bias-conficting ratio increases (i.e., at 2% and 5%).
the synthetic datasets cMNIST and cCIFAR1O to evaluate the effectiveness
A comparison of the results in Table and Table reveals a phenomenon similar to that reported in Zhao et al. (2025): bias-supervised methods tend to perform well on basic binary classification tasks, whereas bias-agnostic methods are relatively more effective in complex multi-class classification scenarios. In contrast, CCDB demonstrates strong performance across both scenarios. With fine- grained distribution matching, FG-CCDB further boosts performance over CCDB by significant margin, highlighting the effectiveness of more thorough spurious correlations elimination.
To demonstrate the effectiveness of FG-CCDB in mitigating spurious correlations, analyze how sample reweighting influences the correlation between feature dimensions and class/bias in- formation, shown in Figure leftt) We compute the correlation of each feature dimension with
8

---

Published as a conference paper at ICLR 2026
class/bias information, and visualize their distributions using box plots Zhao al. (2025) Before sample reweighting, strong spurious correlations in the training data lead the biased model fo. to rely heavily on bias-related features, with most dimensions exhibiting high correlation with bias and low correlation with class. After applying FG-CCDB weights on features from sog. the correlation with bias drops significantly, while the correlation with class increases. Moreover, after debiasing training on the reweighted data, this shift toward class-relevant features is further amplified, con- firming that FG-CCDB effectively reduces the model's reliance on spurious features.
4.3 THE QUALITY OF BLAS EXPLORATION
In this section, evaluate the effectiveness of MST by measuring its mode-prediction F1-score, precision, and recall against the ground-truth annotations. Results regarding the smallest-mode are shown in bb) Since our method progressively filters out bias-conflicingg samples, it retains far fewer such samples than XRM and JTT. achieving the highest F1 Fl-score across the four datasets. This confirms the principle that removing bias-conflicting samples improves bias prediction. JTT misidentifies large number of majority samples as belonging to the smallest-mode(() samples as majority modes(low recall). With
precision).
In contrast, XRM tends to misidentify minority-mone multi-stage refinement, our method achieving the best overall performance. As discussed in Section 3 repeating the "bias enhancement learning" process can further improve both bias prediction ac- curacy and consequently mode prediction accuracy. To validate this claim, we conduct experiments with different numbers of repetitions. The mode prediction performance across varying repetition counts shown in figure(4aa) The dashed lines represent the standard accuracy across all modes, while the solid lines show the recall for each individual mode. We observe that repetition has a particularly strong effect on minority groups (highlighted in bold), evidenced by the significant improvement in their recall with more repetitions. Please refer to Appendix Figure for convergence results with additional repetitions.
Figure (e) shows the final WGA for classification as repetition increases. Notably, performance improves substantially after the first repetition and then plateaus, especially on cMNIST and CelebA suggesting that single repetition is often sufficient to achieve satisfactory performance.
4.4 ABLATION STUDY
Our method comprises two core technical modules: MST and FG-CCDB, which together demon- strate superior performance. In this section, we integrate these modules with existing methods and observe the resulting performance improvements to verify the effectiveness and versatility of our approach, as detailed below.
(i) To assess the effectiveness of MST, we replace ground-truth annotations in bias-supervised meth- ods, i.e., GroupDRO and DFR, with bias predictions generated by MST. This results in their unsu- pervised counterparts, denoted as GroupDRO-MST" and DFR-MST, respectively. The results are reported the top of Table 3 Remarkably, the generalization performance of these unsupervised variants is comparable to their supervised versions using human annotations. Although the bias pre- dictions are not perfect, they are sufficienlly accurate to identify most minority modes, confirming the effectiveness of our MST as approximate substitute for human bias annotations.
(ii) To evaluate the effectiveness FG-CCDB independently of MST, we replace the predicted bias with human annotations, resulting in supervised version, FG-CCDB-sup The results reported at the bottom of Table 3 When bias annotations are available, FG-CCDB-sup further boosts performance, achieving results comparable existing supervised methods on Waterbirds,
Waterlirh
eMNIST(!5)
C(FARRINSG)
CrlbAA
1.0
Befar xfw mevigggie
Jiles
After evvigiig Afa uttuued loaning
Afy snarightin Alli wct aaaiing
Xfae unbiadd larnin laaning
82x
Usbnn Cann-andi
30-
10
Bla Condliiee
Class Conatiie
Clacs Flertae
Class Concaaion
Figure 3: Left: effect FG-CCDB sample reweighting reshaping the data distribution and miti- gating spurious correlations. Right: mode prediction accuracy and Smallest-mool mode recall along 7.

---

Published as a conference paper at ICLR 2026
(FAAR1N5%)
WaterBirds
Culbb
Precision
Recall
Bia cecanay plo
ohenge
Ofack friiak)
(bkend_arakk)
(bedht-
Offinei Samwry)
Rin-corlliing
Crac
Hanehoch Conoin
speat squal ropa2 repai?
ropas) rspaai
rppez
squad
squail
repart repoat
(N)
Figure 4: (a) The mode prediction accuracy along the repeating of the "bias enhancement learning"
procedure; (b) Smallest-mode Fl-score, precision, and recall compare with existing methods.
Table 5: The F1 Fcore of the mmaleest-mode prediction under different top high-confidence ratio B.
cCIFAR1065%)
Waterbirds
CelebA
UrbanCars
Bias-align ratio
95.00%
94.97%
51.72%
90.25%
30%
0.65
0.53
0.32
0.47
50%
0.72
0.64
0,47
0.62
70%
0.79
0.67
0.40
0.64
and outperforming them the others, especially in multi-class settings. This justified our statement on a more aggressive reweighting and indicates that FG-CCDB is a more effective strategy than natve group balancing for handling spurious correlations. Moreover, the performance gap between FG-CCDB-sup and the original FG-CDBB is marginal, further confirming the effectiveness of our method in reducing reliance human bias annotations.
4.5 HYPERPARAMETER ANALYSIS
In this section, evaluate the effect of the hyperparameters 7 for "Initial Bias Learning" and for selecting top ceghhcoofideece samples on MST's final performance. The results are presented in Figure Bright) and Table
The hyperparameter 7 controls the proportion of samples selected for training the initial bias model. Intuitively, a smaller 7 leads to stronger overfitting to bias cues and thus greater reliance them. As expected, the results in Figure righht show that when y S 0.2, both prediction accuracy and smallest-mode recall remain high. However, as 7 increases 0.5, the performance drops signifi- cantly. We find that = 0.1 serves as a sweet spot, while also saving computation compared to y 0.2.
Fl-score with 3 E {30%, 50%, 70%} are reported in Table 5 The hyperparameter 3 controls the proportion of top high-confidence samples selected filter bias-conflicting samples and amplify the model's bias. Intuitively, this value relates to the smallest bias- bias-aligned ratio across classes, as shown in the first row of Table 5 Except for CelebA, whose ratio slightly above 50%, all other datasets have ratios exceeding 90%. Accordingly, B = 50% serves as reasonable middle- ground choice. For CelebA, which has relatively low bias-aligned ratio, = 50% achieves the best performance; whereas for datasets with ratios exceeding 90%, both f 50% and 33 70% yield high Fl-scores, with / = 70% performing the best. Intuitively, when bias annotations are unavailable, selecting the top 50% high-confidence samples is likely to capture the bias-aligned subset while excluding bias-conflicting samples.
5 CONCLUSIONS
In this paper, we address the challenge of robust group generalization under spurious correlations without requiring bias annotations. Following the distribution matching paradigm, we propose method that integrates a reliable bias prediction module with fine-grained class-conditional distri- bution matching. Our approach demonstrates strong performance on real-world datasets with single or multiple shortcuts, as well as highly biased multi-class datasets, often matching or outperforming methods that rely on human-pron prooiddd group annotations. By leveraging the model's overfitting be- havior, our method offers novel alternative to traditional group balancing strategies and effectively reduces reliance on manual supervision. However, its effectiveness may be limited in scenarios where the overfitting signal fails to capture bias cues-for example, in CelebA, which has only one minority group, or in CivilComments, where majority groups dominate one class while minority groups appear in another. These settings present different spurious correlation patterns that weaken the overfitting signal used for bias prediction. Addressing this limitation remains an important di- rection for future research.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work was supported by the National Key R&D Program of China under Grant No. 2024YFA1012700. We thank Chenrong Li and Ruolan Liu for their assistance with figure prepara- tion.
REFERENCES
Kartik Ahuja, Ethan Caballero, Dinghuai Zhang, Sean-Christophe Gagnon-Audet, Yoshua Bengio, Ioannis Mitliagkas, and Irina Rish. Invariance principle meets information bottleneck for out-of- distribution generalization. Advances in Neural Information Processing Systems, 34:3438-3450, 2021.
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv: 2019.
Ihab Asaad, Maha Shadaydeh, and Joachim Denzler. Gradient extrapolation for debiased represen-
tation learning. ICCV, 2025. Jtt.httss://aree /tattps://arxiv.... wwwpwv............oooere.o 2503. 13236
Saeid Asgari, Aliasghar Khani, Fereshte Khani, Ali Gholami, Linh Tran, Ali Mahdavi Amiri, and Ghassan Hamarneh. Masktune: Mitigating spurious correlations by forcing to explore. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 23284-23296. Curran Associates, Inc., 2022. URL ltttps://proceedinnn wwt.:sss.eeriggee cc cc/ File/93be2e6 htee993be25555 3 Coneeern-oofefeecceeeommooo pddf
paper cc/psper
files/paper/202/
Hyojin Bahng, Sanghyuk Chun, Sangdoo Yun, Jaegul Choo, and Seong Joon Oh. Learning de-biased representations with biased representations. In International Conference on Machine Learning. pp. 528-539. PMLR, 2020.
Sara Beery, Grant Van Horn, and Pietro Perona. Recognition in terra incognita. In Proceedings of the European conference on computer vision (ECCV), pp. 456-473, 2018.
Daniel Borkan, Lucas Dixon, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. Nuanced metrics for measuring unintended bias with real data for text classification. In Companion pro- ceedings of the 2019 world wide web conference, pp. 491-500, 2019.
Daniel C Castro, Ian Walker, and Ben Glocker. Causality matters in medical imaging. Nature Communications, 11(1):3673, 2020.
Rwiddhi Chakraborty, Adrian Sletten, and Michael C Kampffmeyer. Exmap: Leveraging explain- ability heatmaps for unsupervised group robustness to spurious correlations. In CVPR, pp. 12017- 12026, 2024.
Sanghyeok Chu, Dongwan Kim, and Bohyung Han. Learning debiased and disentangled repre- sentations for semantic segmentation. Advances in Neural Information Processing Systems, 34: 8355-8366, 2021.
Massimiliano Ciranni, Vito Paolo Pastore, Roberto Di Via, Enzo Tartaglione, Francesca Odone, and Vittorio Murino. Diffusing debias: Synthetic bias amplification for model debiasing. NeurIPS, 2025.
Jacob Devlin, Ming- Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume (long and short papers). pp. 4171 1171-4186, 2019.
Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11):665-673, 2020.
Hyeonggeun Han, Sehwan Kim, Hyungjun Joo, Sangwoo Hong, and Jungwoo Lee. Mitigating spurious correlations via disagreement probability. Advances in Neural Information Processing Systems, 37:74363-74382. 2024.
11

---

Published as a conference paper at ICLR 2026
Moritz Hardt, Eric Price, and Nati Srebro. Equality of opportunity in supervised learning. Advances in neural information processing systems, 29, 2016.
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In CVPR, pp- 770-778, 2016.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common cor- ruptions and perturbations. In International Conference on Learning Representations, 2018.
Youngkyu Hong and Eunho Yang. Unbiased classification through bias-contrastive and bias-
balanced learning. Advances in Neural Information Processing Systems, 11-26499-264611 2021.
Saachi Jain, Hannah Lawrence, Ankur Moitra, and Aleksander Madry. Distilling model failures as directions latent space. In ICLR, 2023.
Saachi Jain, Kimia Hamidich, Kristian Georgiev, Andrew Ilyas, Marzyeh Ghassemi, and Aleksander Madry. Improving subgroup robustness via data selection. Advances in Neural Information Pro- cessing Systems, 37:94490 -94511,991 2024.
Younghyun Kim, Sangwoo Mo, Minkyu Kim, Kyungmin Lee, Jaeho Lee, and Jinwoo Shin. Discov- ering and mitigating visual biases through keyword explanation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1192-11092, 2024.
Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson. Last layer re-training sufficient for robustness to spurious correlations. In The Eleventh International Conference on Learning Representations, 2023. URL https::////cc.. htt.://e//peneeeeeeeeeeeeetettee
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Bal- subramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al. Wilds: A benchmark in-the-wild distribution shifts. In International conference on machine learning, pp. 5637-5664. PMLR, 2021.
Solomon Kullback and Richard A Leibler. On information and sufficiency. The annals of mathe- matical statisties, 22(1):79-88) 86, 1951.
Tyler LaBonte, Vidya Muthukumar, and Abhishek Kumar. Towards last-layer re- training for group robustness with fewer annotations. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural In- formation Processing Systems, volume 36. pp. 11552-11579. Curran Associates. Inc., 2023a. URL
ngs. neurips www.:ewwwreeeeeeeeeeeeeee wwtrriisscccpeaee fil File/266 htteeehhhheeeeeeeeeeeeeee3333333 78 oat.aa//eeeeeeaea-eeeeee---errre
Tyler LaBonte, Vidya Muthukumar, and Abhishek Kumar. Towards last-layer retraining for group robustness with fewer annotations. Advances in Neural Information Processing Systems, 36: 11552-11579, 2023b.
Jungsoo Lee, Jeonghoon Park, Daeyoung Kim, Juyoung Lee, Edward Choi, and Jaegul Choo. Re- visiting the importance of amplifying bias for debiasing. In 37th AAAI Conference on Artificial Intelligence, AAA/ 2023, pp. 14974-14981 AAAI Press, 2023.
Bo Li, Yifei Shen, Yezhen Wang, Wenzhen Zhu, Dongsheng Li, Kurt Keutzer, and Han Zhao. In- variant information bottleneck for domain generalization. In Proceedings the AAAI Conference on Artificial Intelligence, volume 36, pp. 7399-7407, 2022a.
Gaotang Li. Jiarui Liu, and Wei Hu. Bias amplification enhances minority group perfor- mance. CoRR, abs/2309.06717, 2023a. URL
2309. 76717
Ruimeng Li, Yuanhao Pu, Zhaoyi Li, Chenwang Wu, Hong Xie, and Defu Lian. Invariant rep- resentation learning via decoupling style and spurious features. Machine Learning, 114(2):37, 2025.
12

---

Published as a conference paper at ICLR 2026
Zhiheng Li, Anthony Hoogs, and Chenliang Xu. Discover and mitigate unknown biases with debi- asing alternate networks. In European Conference on Computer Vision, pp. 270-288. Springer, 2022b.
Zhiheng Li, Ivan Evtimov, Albert Gordo, Caner Hazirbas, Tal Hassner, Cristian Canton Ferrer, Chenliang Xu, and Mark Ibrahim. A whac-a-mole dilemma: Shortcuts come in multiples where mitigating one amplifies others. In CVPR, pp. 20071-20082, 2023b.
Evan Z Liu, Behzad Haghgoo, Annie S Chen, Aditi Raghunathan, Pang Wei Koh, Shiori Sagawa, Percy Liang, and Chelsea Finn. Just train twice: Improving group robustness without training group information. In International Conference on Machine Learning, pp. 6781-6792. PMLR, 2021.
Sheng Liu, Xu Zhang, Nitesh Sekhar, Yue Wu, Prateek Singhal, and Carlos Frrnandez-Granda. Avoiding spurious correlations via logit correction. In The Eleventh International Conference on Learning Representations, 2023.
Junhyun Nam, Hyuntak Cha, Sungsoo Ahn, Jaeho Lee, and Jinwoo Shin. Learning from failure: De-biasing classifier from biased classifier. Advances Neural Information Processing Systems, 33:20673-20684 2020.
Brady Neal. Introduction to causal inference. Course lecture notes (draf(), 132, 2020.
Fahimeh Hosseini Noohdani, Parsa Hosseini, Aryan Yazdan Parast, Hamidreza Yaghoubi Araghi, and Mahdieh Soleymani Baghshah. Decompose-nn--compps:: A compositional approach to mitigating spurious correlation. In CVPR, pp. 27662-27671, June 2024.
Mohammad Pezeshki, Diane Bouchacourt, Mark Ibrahim, Nicolas Ballas, Pascal Vincent, and David Lopez-Paz. Discovering environments with XRM. In Forty-first International Conference on Machine Learning, 2024. URL
id wttpe//oooommnddptttteee 3FSS9
Mozhgan Pourkeshavarz. Junrui Zhang, and Amir Rasouli. Cadet: a causal disentanglement ap- proach for robust trajectory prediction in autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14874-14884 June 2024.
Aahlad Manas Puli, Lily Zhang, Yoav Wald, and Rajesh Ranganath. Don't 't blame dataset shift! shortcut learning due to gradients and cross entropy. Advances in Neural Information Processing Systems, 36:71874-771111
2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp- 8748-8763. PMLR, 2021.
Shiori Sagawa, Pang Wei Koh, Tatsunori Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generaliza- tion. In ICLR, 2020.
Harshay Shah, Kaustav Tamuly, Aditi Raghunathan, Prateek Jain, and Praneeth Netrapalli. The pitfalls of simplicity bias in neural networks. Advances in Neural Information Processing Systems, 33:9573-9585, 2020.
Nimit Sohoni, Jared Dunnmon, Geoffrey Angus, Albert Gu, and Christopher Ré. No subclass left behind: Fine-grained robustness in coarse-grained classification problems. Advances in Neural Information Processing Systems, 33:19339-19352. 2020.
Christos Tsirigotis, Joao Monteiro, Pau Rodriguez, David Vazquez, and Aaron C Courville. Group robust classification without any group information. In A. Oh, T. Naumann, A. Globerson, K. Sacnko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 56553-56577 Curran Associates, Inc., 2023a.
Christos Tsirigotis, Joao Monteiro, Pau Rodriguez, David Vazquez, and Aaron C Courville. Group robust classification without any group information. Advances in Neural Information Processing Systems, 36:56553-56677 565755, 2023b.
13

---

Published as a conference paper at ICLR 2026
V. Vapnik. Statistical learning theory. Wiley New York, 1998.
Rahul Venkataramani, Parag Dutta, Vikram Melapudi, and Ambedkar Dukkipati. Causal feature alignment: Learning to ignore spurious background features. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 4666-4674, January 2024.
Olivia Wiles, Isabela Albuquerque, and Sven Gowal. Discovering bugs in vision models using off-the-shelf image generation and captioning. In NeurlPS ML Safety Workshop, 2022.
Huaxiu Yao, Yu Wang, Sai Li, Linjun Zhang, Weixin Liang, James Zou, and Chelsea Finn. Im- proving out-o--distriuution robustness via selective augmentation. In International Conference Machine Learning, pp. 25407-25437. PMLR, 2022.
Han Yu, Peng Cui, Yue He, Zheyan Shen, Yong Lin, Renzhe Xu, and Xingxuan Zhang. Stable learning via sparse variable independence. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 10998-11006, 2023.
Xingxuan Zhang, Peng Cui, Renzhe Xu, Linjun Zhou, Yue He, and Zheyan Shen. Deep stable learning for out-of-distribution generalization. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 3372-5382, 2021.
Zhilu Zhang and Mert Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. Advances in neural information processing systems, 31, 2018.
Miaoyun Zhao, Qiang Zhang, and Chenrong Li. Class-conditional distribution balancing for group robust classification. arXiv preprint 17314 2025.
Xiao Zhou, Yong Lin, Renjie Pi Weizhong Zhang, Renzhe Xu, Peng Cui, and Tong Zhang. Model agnostic sample reweighting for out-of-distribution learning. International Conference on Ma- chine Learning, pp. 27203-27221. PMLR, 2022.
14


Reviews:
[{"review_id": "EpXvBHrJX1", "reviewer": "Reviewer_2Y79", "summary": "This paper addresses debiased learning under spurious correlations without relying on bias annotations. It targets the problem that existing methods mitigate spurious correlations by matching class-conditional to marginal distributions but rely on overly coarse single-Gaussian approximations that fail in multi-modal, multi-class settings. The authors propose MST to exploit ERM overfitting, training on a small split, and amplifying bias via per-class top-confidence selection to derive discrete modes from the final confusion matrix. Then they propose FG-CCDB to transform this matrix into mode-wise weights that align class-conditional to marginal distributions, reducing spurious reliance with lightweight, scalable computation. Experiments that MST can effectively substitute for human bias annotations in supervised methods, and FG-CCDB consistently outperforms or matches state-of-the-art bias-agnostic baselines while maintaining low computational and memory overhead.", "questions": "Why fix the top-50% per-class in bias enhancement? Did you explore adaptive thresholds (e.g., based on class-wise confidence distributions, entropy) or rank-based schedules across iterations? Any results on temperature scaling to improve confidence reliability?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "-The paper offers a thorough and compelling analysis of the limitations in prior work, clearly diagnosing CCDB’s single-Gaussian assumption and proposing an elegant, scalable remedy via mode-wise matching derived from confusion-matrix–based distributions. \n-The approach demonstrates strong practical value, showing robustness to multi-shortcut scenarios (e.g., UrbanCars) with competitive or superior performance relative to both bias-agnostic and supervised baselines.\n-The experimental evaluation is comprehensive. Ablations cleanly disentangle the contributions of MST and FG-CCDB, and the correlation-shift analyses substantiate the mechanism that the method reduces reliance on bias-related features.", "weaknesses": "-The proposed mechanism lacks theoretical grounding. There is no formal analysis of how iterative bias amplification improves minority-mode recall. While the empirical evidence is compelling, a theoretical treatment would substantially strengthen the contribution.\n-Several methodological details require further clarification. The selection of top-confidence samples hinges on the biased model’s calibration. Miscalibration could distort mode discovery, yet no temperature scaling or calibration baseline is reported.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RxLJncUeYf", "reviewer": "Reviewer_exPH", "summary": "This paper builds upon the existing Class-Conditional Distribution Balancing (CCDB) framework and proposes two key components: \n- MST (Multi-stage data-Selective reTraining): a strategy to characterize the bias structure through the hard confusion matrix, serving as a proxy for bias annotations.\n- FG-CCDB (Fine-Grained Class-Conditional Distribution Balancing) - a fine-grained extension of CCDB that performs distribution alignment at the mode level, enabling a more detailed representation of multi-modal data distributions. \n\nThe method aims to achieve annotation-free debiasing and robust generalization by modeling complex intra-class variations caused by spurious correlations. Experimental results show that MST can substitute for bias annotations in supervised baselines (e.g., GroupDRO, DFR), and FG-CCDB outperforms prior approaches in multi-class, bias-heavy scenarios.", "questions": "As the paper claims, can the authors experimentally demonstrate how well the MST matches human labels?\nCan the authors of the paper present performance comparison results with the latest methods using 'other label free + generative model methods'?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Well-structured and clearly written:\nThe paper is logically organized, and the presentation of ideas—from motivation to methodological formulation—is easy to follow.\n\n- Conceptual improvement over CCDB:\nBy replacing the single-Gaussian assumption with a multi-modal, mode-based distribution matching framework, the paper effectively extends CCDB to more realistic data distributions.\n\n- Novel use of confusion matrix:\nEmploying the confusion matrix to infer bias-aligned and bias-conflicting “modes” is elegant and intuitively appealing.\nIt offers a discrete, bias-agnostic way to describe intra-class spurious correlations.\n\n- Annotation-free contribution:\nThe approach is promising in scenarios where human bias annotations are unavailable or infeasible, showing comparable results to bias-supervised baselines.", "weaknesses": "- Indirect validation of MST as bias substitute:\nThe core assumption that MST can replace human-provided bias annotations is only indirectly validated through final task performance. The paper does not report any direct quantitative measure (e.g., F1, ARI, or NMI) of how well MST’s predicted bias partitions align with human-labeled bias groups. Without this, it is unclear whether MST truly captures bias structure or simply produces partitions that happen to improve performance.\n\n- Limited comparison with recent label-free debiasing methods:\nAlthough the paper positions itself within the “annotation-free” literature, it lacks comparisons with the latest label-free or label-free-from-features (LFF) methods, such as those that employ generative modeling or causal data augmentation for debiasing. Including these would strengthen the empirical validity and demonstrate broader applicability.\n\n- Experimental generality:\nMost experiments are conducted on benchmark datasets (e.g., Waterbirds, CMNIST) with relatively simple, well-defined bias sources.\nThe performance of MST and FG-CCDB under multi-bias or entangled bias scenarios remains uncertain.\n\n- Ablation analysis depth:\nWhile ablations for MST and FG-CCDB are presented, the interaction between the two modules is not deeply analyzed. It is unclear how errors from MST propagate to FG-CCDB, or whether FG-CCDB can compensate for imperfect bias predictions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kvV3v6sy2L", "reviewer": "Reviewer_FipK", "summary": "This paper proposes a novel method named Fine-Grained Class-Conditional Distribution Balancing (FG-CCDB) to mitigate spurious correlations in deep learning models under the condition of no bias annotations. The authors point out that existing methods (e.g., CCDB) model class-conditional and marginal distributions as single Gaussian distributions, which are too coarse-grained to capture the complex multi-modal structures in real-world data. To address this, they design a **Multi-stage data-Selective reTraining (MST)** strategy that leverages model overfitting behavior to identify \"modes\" (i.e., bias patterns) through multi-stage data filtering. Based on this, FG-CCDB performs distribution alignment at the mode level, achieving more precise sample reweighting. Experimental results show that the method can rival bias-supervised approaches in binary classification tasks and even outperform them in multi-class tasks, with low computational overhead.", "questions": "（1）FG-CCDB assumes samples within a mode are homogeneous and assigns them equal weights. However, a mode (e.g., a specific background for a class) may still have substructures or diversity. Does ignoring intra-mode heterogeneity affect reweighting effectiveness? Have you considered further subdivision within modes or introducing continuous weights?  \n\n（2）In multi-stage MST, repeating \"bias enhancement learning\" up to three times improves performance. Is there a saturation point? Would further iterations cause model collapse or overfitting to noise? Could you provide performance curves across iteration counts to demonstrate convergence?  \n\n（3）Although FG-CCDB is computationally efficient, MST requires training multiple auxiliary models, involving multiple training stages. Compared to single-training methods like ERM or uLA, how much additional training time does it incur? Is this computational cost acceptable in practical applications?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "（1）Proposed Multi-stage data-Selective reTraining (MST), which utilizes model overfitting to construct highly biased training sets through multi-stage data filtering. This generates reliable pseudo-bias labels and forms a hard confusion matrix to characterize fine-grained \"mode\" structures, offering a new perspective for unsupervised bias exploration.  \n（2）Proposed FG-CCDB, which performs class-conditional and marginal distribution alignment at the mode level based on MST. This achieves finer and more effective distribution matching than the original CCDB, effectively mitigating spurious correlations.  \n（3）The method requires no ground-truth bias annotations, is computationally efficient, and allows sample weights to be computed via closed-form solutions. It achieves superior or comparable performance to existing unsupervised and partially supervised methods on multiple binary and multi-class benchmarks, particularly in strong-bias multi-class scenarios.", "weaknesses": "（1）The definition of \"mode\" is dependent on specific bias model architectures and training processes, lacking semantic clarity and theoretical guarantees.  \nThe authors define a mode as (s, y), where s is a \"pseudo-bias label\" predicted by an auxiliary bias model. However, the physical meaning of s is ambiguous—it may correspond to a single shortcut, a combination of shortcuts, or even entangled uninterpretable patterns. This makes \"mode\" a black-box concept, with unclear mapping to latent bias factors in the data generation process. Moreover, the quality of s heavily relies on the initial ERM model's overfitting behavior. If the model fails to capture major biases, the entire MST process may collapse.  \n\n（2）Hyperparameter choices in MST (e.g., γ=10%, top 50% high-confidence samples) lack systematic analysis and generalization guarantees.  \nWhile the authors claim γ=10% is a \"sweet spot,\" this conclusion seems empirical, with no sufficient ablation studies to validate its robustness across datasets or bias strengths. Similarly, selecting the top 50% high-confidence samples appears arbitrary, with no discussion of alternative proportions (e.g., 30% or 70%). These critical hyperparameters lack theoretical justification or adaptive mechanisms, reducing the method's universality and reproducibility.  \n\n（3）The method's validity critically depends on the strong assumption that \"overfitted models accurately reflect bias structures,\" which may not hold in complex real-world scenarios. \nThe core idea is to exploit ERM models' overfitting to reveal biases. However, in real-world data, biases may be subtle or diverse, and models might overfit to noise or irrelevant features instead of true bias cues. Additionally, when multiple competing biases exist, the model may capture only one, leaving MST unable to reveal the full bias structure. While experiments show strong performance, the robustness of this assumption has not been validated in more challenging scenarios with hidden or complex biases.  \n\n（4）Insufficient comparison with existing methods (e.g., XRM, DebiAN) to highlight FG-CCDB's core innovations. \nThe paper includes XRM and DebiAN as baselines but provides brief descriptions of their mechanisms. XRM uses auxiliary models to generate pseudo-group labels, while DebiAN iteratively trains a \"discoverer\" model to identify biases. MST is essentially another pseudo-labeling strategy. Reviewers expect a more detailed comparison: e.g., does MST's pseudo-label quality significantly outperform XRM or DebiAN? On which bias types does MST excel? Otherwise, FG-CCDB may appear as another variant of existing pseudo-labeling + reweighting paradigms rather than a fundamental breakthrough.  \n\n（5）The experimental evaluation lacks direct quantitative analysis of mode partitioning quality. \nWhile Figure 2(b) shows a heatmap of the joint distribution J and claims it reflects true biases, there are no quantitative metrics (e.g., mutual information, F1-score with ground-truth biases) to evaluate the accuracy of MST-generated mode partitions. If the mode partitioning itself contains significant errors, subsequent FG-CCDB distribution matching may amplify these errors, leading to performance degradation. The authors claim MST \"serves as a strong proxy for ground-truth bias annotations,\" but this assertion requires stronger empirical or theoretical evidence.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)