DeNetDM: Debiasing by Network Depth Modulation
Authors: Silpa Vadakkeeveetil Sreelatha
Adarsh Kappiyath
Abhra Chaudhuri
Anjan Dutta
Venue: NeurIPS 2024 poster (2024)
Keywords: Trustworthy Machine Learning
Debiasing
Robustness
Primary area: fairness
TL;DR: A method for obtaining a debiased classifier by modulating the network depth.

Abstract: Neural networks trained on biased datasets tend to inadvertently learn spurious correlations, hindering generalization. We formally prove that (1) samples that exhibit spurious correlations lie on a lower rank manifold relative to the ones that do not; and (2) the depth of a network acts as an implicit regularizer on the rank of the attribute subspace that is encoded in its representations. Leveraging these insights, we present DeNetDM, a novel debiasing method that uses network depth modulation as a way of developing robustness to spurious correlations. Using a training paradigm derived from Product of Experts, we create both biased and debiased branches with deep and shallow architectures and then distill knowledge to produce the target debiased model. Our method requires no bias annotations or explicit data augmentation while performing on par with approaches that require either or both. We demonstrate that DeNetDM outperforms existing debiasing techniques on both synthetic and real-world datasets by 5\%. The project page is available at https://vssilpa.github.io/denetdm/.

DeNetDM: Debiasing by Network Depth Modulation
Silpa Vadakkeeveetil Sreelatha" 1 Adarsh Kappiyath" Abhra Chaudhuri Anjan Duttal
1 University of Surrey 2 University of Exeter Fujitsu Research of Europe
{a. madaktanvaattlareelaata a-kappiyath, anjan.dutta jijaaaduttaaaaurrey.. ac.uk, abhra. caarra.caadhhrreee winedtrrrrfffuuj taa..co con
Abstract
Neural networks trained biased datasets tend inadvertently learn spurious correlations, hindering generalization. We formally prove that (1) samples that exhibit spurious correlations lie on lower rank manifold relative the ones that do not; and (2) the depth of network acts as an implicit regularizer on the rank the attribute subspace that encoded in its representations. Leveraging these insights, we present DeNetDM, a novel debiasing method that uses network depth modulation as a way of developing robustness to spurious correlations. Using a training paradigm derived from Product of Experts, we create both biased and debiased branches with deep and shallow architectures and then distill knowledge to produce the target debiased model. Our method requires no bias annotations explicit data augmentation while performing on par with approaches that require either or both. We demonstrate that DeNetDM outperforms existing debiasing techniques on both synthetic and real-world datasets by 5%. The project page is available https://vssilpaa httpss//ssiipa.guuubboo/eddddd github.io//deetdmm
Introduction
Deep neural networks (DNNs) have made remarkable progress across various domains by delivering superior performance on large-scale datasets. However, while the benefits of training DNNs on large-scale datasets are undeniable, these algorithms also tend to inadvertently acquire unwanted biases Shah et al. (2020), hampering their generalization. For instance, a classifier predominantly trained to recognize camels in desert landscapes could encounter difficulties when attempting to identify a camel situated on road Kim et al. (202T). While certain degree of bias can enhance model performance, exemplified by the assumption that cars usually travel roads Choi et al.) (2020), it remains critical identify and address unwanted biases. Previous methods to address this problem rely on bias annotations suggested in Majumdar et al. (2021); Kim et al. (2019);
Sagawa et al. 20200)] |aan et al. (2020). and may involve predefined bias types, such as texture bias mitigation approach in Geirhos et al. 2019). However, acquiring bias labels with human resources is expensive and time-consuming. Recent studies, including Nam et al. (2020) and Lee et al.) (2021) )
have shifted towards debiasing methods without bias labels, with approaches like Nam et al. (2020)
emphasizing bias-aligned samples and reweighting bias-confticting samples, while others like Lee
al. 2021
Kim al. (202T) introduce augmentation strategies diversify bias-conflicting points.
We propose DeNetDM (Debiasing by Network Depth Modulation), a novel approach to automatically identify and mitigate spurious correlations in image classifiers without relying on explicit data augmentation or reweighting. We start by showing that a sample set that exhibits bias through spurious correlation of attributes lies manifold with effective dimensionality (rank) lower than its bias-free counterpart. We then leverage this finding to formally derive relationship between the depth of network and the true rank of the attribute (not sample) subspace that it encodes. We find for a set of attributes that are equally likely to minimize the empirical risk, a deeper network prefers
"Equal contribution.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

to retain those with a lower rank, with a higher probability. This implies that the depth of network acts as an implicit regularizer the rank space of the attributes. We find that deeper networks tend to generalize based on bias attributes and shallower networks tend to generalize based on core attributes. This finding is in line with a number of works that show that deeper networks tend to learn low rank solutions in general (Roy and Vetterli| 2007 Huu et al. 2023 Wang and Jacot (024). Note, however that prior works do not establish the relationship between network depth and the rank of the attribute subspace, link we establish in our work for the first time, to the best of our knowledge.
Our theoretical claims are confirmed by our preliminary empirical study on linear feature decodability, which quantifies the extent to which specific data attributes can be accurately and reliably extracted from a given dataset or signal. Our study focuses on the feature decodability of bias and core attributes in the neural networks of varying depths, following the approach outlined in Hermann and Lampinen (2020). Our observations in untrained neural networks reveal that the feature decodability tends to diminish as the networks become deeper. We also investigate how attribute decodability varies with Empirical Risk Minimization (ERM) based training on networks of varying depths.
Our hypothesis posits that in a task requiring deep and shallow branches to acquire distinct information, the deep branch consistently prioritizes bias attributes, while the shallow branch favors core attributes. We utilize a technique inspired by the Product of Experts (Hinton 2002), where one expert is deeper than the other. Empirical analysis shows that the deep branch becomes perfectly biased and the shallow branch becomes relatively debiased by focusing solely on the core attributes by the end of the training. Since the shallow branch may lack the capacity to capture the nuances of the core attributes adequately due to less depth, we propose a strategy where we train a deep debiased model utilizing the information acquired from both deep (perfectly biased) and shallow (weak debiased) network in the previous phase. Our training paradigm efficiently facilitates the learning of core attributes from bias-conflicting data points to the debiased model of any desired architecture.
In summary, we make the following contributions: (I) We theoretically prove that the deep models prefer to learn spurious correlations compared shallower ones, supported by empirical analysis of the decodability of bias and core attributes across neural networks of varying depths. (2) Building
upon the insights from our decodability experiments, we present a novel debiasing approach that involves training both deep and shallow networks to obtain a desired debiased model. (3) We perform extensive experiments and ablation studies on a diverse set of datasets, including synthetic datasets like Colored MNIST and Corrupted 100 as well as real-world datasets, Biased FFHQ, BAR and CelebA, demonstrating an approximate 5% improvement over existing methods.
2
Related Works
Several works, such as Hermann and Lampinen] (2020); Mehrabi et al. (202T), have highlighted neural networks' vulnerability to spurious correlations during empirical risk minimization training. Recently, various debiasing techniques have emerged, which can be categorized as follows.
Supervision on bias: A variety of approaches (e.g.. Majumdar et al. 2002 ; Kim et al. (201));
Sagawa al. p020); Wang et al. (2020) assume readily accessible bias labels for bias mitigation. Some approaches assume prior knowledge on specific bias types without using explicit annotations,
(2019;; Ge et al. (202I): Geirhos al.] (2019). Recent works such as Karimi Mahabadi et al. 22020); Clark et al. (2019) apply the Product of Experts method to mitigate bias in natural language processing, assuming biased expert's availability. However, obtaining bias labels can be resource- intersivee. In contrast, DeNetDM, our proposed method, does not require pre-access to bias labels or types. Instead, it leverages diverse network architecture depths within the Product of Experts framework to implicitly capture relevant bias and core attributes.
like texture bias in Wang et al.
Utilization of pseudo bias-labels: Recent approaches avoid explicit bias annotations by obtaining pseudo-labels through heuristics to identify biased samples. One heuristic suggests that biases easy to learn are captured early in training, as seen in Nam et al. (2020); Lee et al. (202T) [Liu et al. (2023); Kim et al. (2021); Tiwari and Shenoy (2023); Lee et al. (2023). Nam et al. (2020) employ generalized cross-entropy loss to identify and reweight bias-conflitingg points. On the other hand, [ee et al.
(202T) augment features of bias-conflicingg points for debiasing, while Hhi et al. (2023) employ logit
correction and group mixup techniques to diversify bias-conflicting samples. Other methods like
Sohoni et al. 22220) and Seo et al. (2022) acquire pseudo-bins bias labels through clustering in biased

---

Stage
Stage
Dear Branch
ds
Dect Brands 4s
Clasiia
Cres--ctinpyy
Lees fg
Bsand Candifiation
Shand Weights
Hirr
Claspieer
Crose-cntuppy Loss
Tirge Brand 4%
Clactiir
Cez
Duaiiaaiin
lins Blased
Loger
Dchisual
Bixsed Dutoect
Clasitication
Shillos Rranc
Dehisaod
Weat
Suilis Touck
du
Classffer
Classifiation
Figure I: Illustration of the DeNetDM framework: In Stage 1, an ensemble of shallow and deep branches produces outputs linearly combined and trained as product of experts. The cross-entropy loss with depth modulation aids in separating biases and identifying target attributes. In Stage 2. we further introduce a target branch with the desired architecture, which also requires debiasing. This phase exclusively focuses on refining the target branch's 's feature extractor (ox) and classifier head (fi) while leveraging knowledge from the initial stages.
network feature spaces. Our approach does not explicitly require pseudo-bias labels; it implicitly uses them during training to learn both biased and debiased models.
Dependence on network architectures: Diffenderfer et al. (2021) employ lottery-ticket-Sie style p algorithms for compressed robust architectures. Similarly. approaches like Park et al. (2023); Zhang et al. (2021) introduce pruning to extract robust subnetworks. Our method aligns with this category but does not target specific robust subnetwork discovery. Instead, we utilize training dynamics of varied-depth architectures to enhance debiasing. Meanwhile, (Shressaa et al. (2022) applies Occam's ' razor principle to optimize network depth and visual regions, enhancing overall robustness. Both DeNetDM and OccamNets (Shrestha et al. (022) aim to simplify learning for better generalization and reduced spurious correlations. DeNetDM uses depth modulation with separate deep and shallow branches address bias where the shallow model captures biases and the deep model learns complex, unbiased patterns. In OccamNets, simplicity is a core design principle, with the architecture adaptively minimizing complexity per-sample basis. Both methods tackle spurious correlations without extra annotations or data augmentation but through distinct architectural strategies.
3 Debiasing by Network Depth Modulation
First, we thcoretically justify that the deeper models are more inclined to learn spurious correlations compared shallow networks, as discussed in Section 3.1 We then provide empirical evidence to support our theoretical claims by utilizing feature decodability, detailed in Section 3.2 Based on these, we introduce DeNetDM, a debiasing approach centered on network depth modulation. Our training process comprises two stages: initially, a deep and shallow network pair is trained using a training paradigm that originates from Products of Experts (Hinton) 2002), yielding both biased and debiased models, which is detailed in Section 3.3 Subsequently, recognizing the limitations of the shallow debiased model in capturing core feature complexities due to its depth, we proceed to train a target debiased model, ensuring it possesses the same or higher depth compared to the deep biased model. This phase leverages information acquired from the biased and debiased models in the previous step. elaborated in Section 3.4 An illustration of DeNetDM is provided in Figure [
We operate on a dataset X, where a fraction of the data points, denoted with X,,, are bias- bias-aligned and the remaining points, denoted with X,, are bias conflicting. Let Φ X - R2 be an encoder that produces an embedding E RD for an input x E X. We denote the effective rank (Roy and Vetterli (007) of matrix A as p(.A), which gives us a continuous notion of the size of the span (rank) of A, a quantity that maximized under equally distributed singular values, and minimized when single singular value dominates over the rest (Huh et al. 2023). Let B and C' be the set of bias and core attributes respectively, both with strictly positive ranks, defining bases that orthogonal to cach other, i.e., B 1 C. A summary of notations is provided in Section 7.1
Notations:
3

---

3.1 Simplicity Bias and Spurious Correlations
Debiasing with network depth modulation requires understanding how the depth of a neural network affects its learning of bias- slas-eligned or bias- ctas-oonflittin subsets of X with lower generalization error. These results finally let us build up to our finding that deeper networks are more susceptible to learning spurious features over their shallower counterparts. All proofs are deferred to Section 77.
Definition 1 (Stability). A partitioning X = X1 U Xg... U U,m of a sample set X is stable wrr. an attribute w when:
p(x;) = P(X Vi E [1,m]]
where X and X;" are the respective subspaces of X and X, corresponding to the attribute w, and P(-) is the associated probability distribution.
For example, if w follows uniform distribution in X, stable partitioning would ensure that each of the partitions X, also have w distributed uniformly. Stability ensures that a partitioning does not introduce sampling bias into any of the partitions wrt. a particular attribute.
Theorem (Partition Rank). When the partitioning X = X. U Xe is stable wrt. C, the rank of the alisnal partition upper- by the rank of the bias- ctas-conftictin partition, i.e.,
rank( (Xa) rann((X))
Intuition: The theorem assumes a stable partitioning of the sample set X. It implies that, in both the bias-aligned and conflicting subsets, the distribution of the core attributes equal to that of the original sample set, i.e., P(Xx) = P(X.) = P(X*). Under this condition, the only component in either of the subsets that determines the subset's rank should be the bias attributes, assuming (without loss of generality) that the attribute space is made up of only the core and the bias attributes. The proof proceeds by establishing that the rank of the bias attributes is lower in the bias-aligned points (resulting from the lack of intra-class variation due to spurious correlation with the class label) than the bias-conflicting points.
Theorem 2 (Depth-Rank Duality). Let A = [Ao, A1.... .a......,,, ] be the attribute subspace of X with increasing ranks, i.e., rank (Ao) < rank(Ai) < ... < &ran((A)) such that every A E A is maximally and equally informative of the label Y. i.e., I(Ao, Y)= I(A1, =... I(An, Y). Then, across the depth of the encoder o, SGD yields a parameterization that optimizes the following objective:
An]
min C(f(6(X)), Y) min Σ old)(() Ωd c AA 0.
ERM
where ((--)) is the empirical risk, f(-) is a classifier head, +b1((-) is the output of the encoder & (optimized end-to-end) at depth d, I1-11z is the (--orrm, @ the element-wise product, is the lz-normalized version of X, Ω2d = [d); 1x2(d) ...: (d)) 1 is a random binary function that outputs with a probability π, and n. (d) the propagation probability of A, at depth d bounded us:
(((d)= = O (p(old)
where p(<bld) is the effective rank of the Φ[d] representation spuce, and r = (A,).
Intuition: For a set of attributes, all of which equally minimize the training loss, Theorem 2 describes the strategy adopted by SGD to parameterize a neural encoder, for capturing the above set attributes. At a given depth d of the encoder & (represented as <(d)), each attribute A, E A gets encoded in the representation space of d[d] according to its corresponding probability mass ग. (d). According to Equation (2), the probability of survival of all attributes decrease with increasing depth. However, the probability of survival of an attribute with a higher rank drops faster with increasing depth than that of one with a lower rank, prioritizing the usage of lower rank attributes at greater depths. In other words, the depth of a network acts as an implicit regularizer in the attribute rank space.
As an example, say, a neural network á of depth d (denoted ø(d) has 3K available dimensions, and depth > d, ((D) has K available dimensions (the rank reduction with increasing depth stemming from the simplicity bias (Huh et al. 2023 Wang and Jacot 2024 Say the attribute space has to learn from is composed of two attributes: (1) Ag, with rank of K, and (2) A,, with a rank of K i, where 1 < < K, where both Ao and A, are equal minimizers of the empirical risk.

---

Linear Decodability of Attributes Unirained MI.P
Early Training Dynamics of MLPs Varying Depth
Lxbe
Lincar Docodabiiitty (%)
Decodability
-
Dait laye MILP)
Color (3 laye MLP:
Dagtt layer NELP)
Coler d5 layer MLP!
Network Depth
Training Iterations
(a) Linear Decodability vs. Network Depth
(b) Linear Decodability vs. Training Iterations
Figure 2: Exploring the effect of depth modulation: (a) illustrates how the linear decodability of features decreases as neural network depth increases, while (b) dives into the training dynamics of MLPs with varying depths under ERM.
So, according to Theorem 2 at <fd]] the encoder has no constraint over the number of attributes accommodate, since 33 > 2K i. However, at depth D, & [D] Φ|D] with K dimensions. Since both Ag and A₁ result in the same solution for ERM (Empirical Risk Minimization), SGD would parameterize ΦDD] capture Ag with a higher probability.
only choose an attribute
3.2 Effect of Depth Modulation
Theorem D @establishes a relationship between the depth of a network and the nature of the features learns in terms of its rank. To empirically validate this, probe MLPs of depths feature decodability technique proposed by Hermann and Lampinen (2020), to uncover the types of features that get encoded in them. We the Colored MNIST dataset Nam al. (2020) (CMNIST), where digit identity (core attribute) is spuriously correlated with color (bias attribute). We experiment with the decodability of the digit identity and color attributes in the CMNIST dataset. Additional
and 5, using the
information on the computation of feature decodability can be found in Section 7.5 We regard
digit identity have a higher rank than that of color, due to its higher representational complexity / information content in terms of the number of bits required for storage, a notion also confirmed in the experiements Hermann and Lampinen (2020). We start by looking the decodabilities at random initialization of the networks, and interestingly observe in Figure 2a that the decodabilities of both attributes decrease with increasing depth, but that of digit identity drops faster than color. Since at random initialization, there is no notion of empirical risk, the min C(::)) term in Theorem 2 cancelled out. Thus, the observation aligns with our prediction of the second term in Il-112 of Theorem that the higher the rank of a feature, the less likely is to get encoded in the later layers. the theoretically predicted behavior specifically for random networks being discussed in Corollary 2.1 We then proceed to investigating how feature decodability evolves during the early stages of Empirical Risk Minimization (ERM) training across the networks of varying depths, i.e., under the presence of min C(-, :), the results of which summarized in Figure 2b We perform similar linear decodability analysis on C-CIFAR1O dataset and the observations are presented in Section [7.6.]]
As observed in Figure 2b the initial phases of training for both networks emphasize color attribute (since bias is easy to learn), leading to notable improvements in color decodability for both models. Also, as training progresses, the 3-layer model exhibits higher digit decodability compared to the 5-layer model. Hence, the difference in decodability between color and digit attributes becomes more pronounced in the 5-layer compared to the 3-layer MLP. This again confirms the prediction of our Theorem
2 that when two attributes equally minimize the empirical risk, a deeper network is more likely to select the one with lower rank, while shallower network will try to accommodate as much of both as possible. Based on these observations, the deep models may prefer bias attributes, while shallow models focus
attributes when tasked with capturing distinct information.
This prompts us to explore whether similar behavior can be induced in models of equal depth. In this scenario, both models, undergoing ERM training, may exhibit a similar trend, with the disparity in decodability between biased and core attributes becoming nearly identical in both models due to same depth. Consequently, when compelling each model to learn distinct information, they may
5

---

capture biased or core attributes, or even divide attribute information between them, leading to loss of control over the bias identification process. We also present empirical evidence in Table to support these claims. Therefore, using models of different depths introduces an inductive bias suitable for the bias identification process.
3.3 Stagel: Segregation of Bias & Core Attributes
Theorem predicts that bias-aligned points lie lower-rank manifold than bias conflicting points. Theorem predicts that as we go deeper into neural network, the likelihood that higher rank feature, that equally minimizes the empirical risk as that of other lower rank features, retained, decays exponentially with depth. Based on this, we present training procedure to obtain the biased and debiased classifier for an M class classification problem. Let Φi and Φa denote the parameters of the feature extractors associated with the deep and shallow branches, where depth(())) > depth((a). We use f to represent the classifier head shared by @b and Φd. Here, f, @b and Φd are trainable parameters. Considering image-label pair (I, v). the objective function is expressed as:
LCE(0. y) = E Ue log(Pe)
where ş = softmax (f(asoxo((( + aldd(x))). process, we get:
If we set = = throughout the training
p = softmax (f (db(x)) pa(x)))
To evaluate the performance an individual expert, we assign value of 1 to the corresponding a while setting the other equal to 0.
Our training methodology derived from the Products of Experts technique (Hinton 2002) where multiple experts are combined to make final prediction, and each expert contributes to the prediction with a weight. However, in our approach, the role of the experts is assumed by Φ6 and Φa. whose features are combined through weighted contributions. The conjunction of features is then passed to the shared classifier to generate predictions. We provide a detailed proof elucidating the derivation of Equation through the Product of Experts in Section 7.3 of the Appendix. Due to the architectural constraints we imposed by modulating their capacities, the deep expert tends to prioritize the learning of bias attribute, while the shallow expert is inclined towards the core attribute. The model leverages the strengths of both experts to effectively learn from their combined knowledge. We investigate the training dynamics in Section
3.4 Stage2: Training the Target Debiased Model
The initial phase effectively separates the bias and core attributes into deep and shallow branches, respectively. However, relying solely on the debiased shallow branch may not be practical, as might not capture the complex features representing the core attributes, given the less depth of the shallow model. This limitation does not apply to the deep biased model. To tackle this challenge, we introduce a target branch with the desired architecture for debiasing.
Let de be the parameters of the feature extractor associated with the target branch and ft be the classifier head whose weights are initialized using the weights of f. During this phase, our training exclusively focused on 6; and f.. We freeze Φb and @u since we leverage these models only extract the necessary knowledge for debiasing the target branch. To capture information orthogonal to dh, we employ the same training approach described Section 3.3 where da, and di serve as the experts. The objective function can be written as:
(.(g.)) -
Σ ye log(pe)
(5) The training and evaluation of the experts follow the procedure described in Section 3.3 with the key difference being that in this phase, only a single expert, dr, which is the target branch and classifier fú. undergoes updates.
where
أ - (f + (:b:(x)))

---

We further leverage the knowledge pertaining to the core attributes, which is encapsulated in da. by transferring this knowledge to the target branch o, through knowledge distillation. Here, or acts as the student, whereas da corresponds to the teacher. We set 8, 0 and 3r = 1 in Equation (5) to obtain the predictions of the student de. Therefore, the distillation loss is given by
M
Laia(De.D)) =
Pto log(Pe.)
(6)
p. softmax f((0r(2))
pr softmax [f(0((x))
where
(7)
(8)
where À is a hyperparameter chosen from the interval [0, 1]. The pseudocode for the entire training
process of DeNetDM is provided in Section 7.4
Experiments
In this section, we discuss the experimental results and analysis to demonstrate the effectiveness of DeNetDM training in debiasing. We evaluate the performance of the proposed approach by comparing with the previous methods in debiasing, utilizing well-known datasets with diverse bias ratios, consistent with the prior works in debiasing. Additionally, conduct empirical study to analyze the training dynamics of DeNetDM. We also perform ablation studies to assess the effectiveness of individual components within the proposed approach.
4.1 Experimental Setup
We evaluate the performance of DeNetDM across diverse domains using two synthetic datasets (Colored MNIST Ahuja et al. (2020), Corrupted CIFAR R 10 Hendrycks and Dietterich (2019) al. (202T). BAR Nam et al.] (2020)) and CelebA Liu et al. (0015). In Colored MNIST (CMNIST), the digit identity is spuriously correlated with color, while in Corrupted CIFARIO (C-CIFARIO) 10), the texture noise corrupts the target attribute. Biased FFHQ (BFFHQ) comprises human face images from the FFHQ dataset Karras et al. (2019) such that the age attribute is spuriously correlated with gender. BAR consists of human action images where six human action classes are correlated with six place attributes. We conduct experiments by varying the ratio of bias-cong points the training set to demonstrate the efficacy of our approach across diverse scenarios. Following the experimental settings used by the previous works Liu et al.) (2023): Lee et al. (202T): Qi et al. (2022), we vary the ratio of bias-conflicting samples, specifically setting at [0.5%, 1%, 2%, 5%) for CMNIST and C-CIFAR 10, (0.5%) in BFFHQ and (1%, 5 5% in BAR datasets. We employ a subsampled version of CelebA as described in Hong and Yang (202) maintaining the same data splits for consistency.
Datasets:
and three weal-world datasets (Biased FFHQ
Baselines: We compare the performance our proposed approach to the following bias mitigation techniques; ERM Vapnik (1999). GDRO Sagawa et al. (2020), LIF Nam et al. (2020), JTT Liu et al. (2021) DFA Lee al. (2021) and LC et al. (202)). Among these, GDRO utilizes supervision on bias whereas LiF and JTT assumes no prior knowledge on the bais labels. DFA and LC utilizes augmentation techniques increase diversity of minority groups. More details on the baselines are provided in Section 7.8.2 of the Appendix.
We evaluate CMNIST and C-CIFAR10 on unbiased test sets, with target features randomly correlated to spurious features, following the evaluation protocol commonly used in prior debiasing works Nam et al. (2020); Liu al. (222I: : Lee et al. (202I). Nevertheless, for BFFHQ, we do not use the unbiased test set since half of them are bias-aligne aligned points. To ensure fair evaluation on debiasing, we adhere to previous methods Liu al. 2023 Lee al. (202T) by exclusively utilizing a test set comprising bias-conflicting points from the unbiased test set. Notably, the BAR test set consists solely of bias-conflicting samples, posing a significant evaluation challenge. Our primary metric is accuracy, with aligned accuracy and conflicting accuracy calculated separately for some ablations on CMNIST and C-CIFAR 10 (see Section 4.4). Aligned accuracy computed solely on bias-aligned data points while conflicting accuracy is determined exclusively based on the bias-conflicting points. For CelebA, we report worst-group accuracy specifically focusing on the bias-conflicting group (Blonde Hair 0, Male 0), which contains substantial number of samples.
Evaluation protocol:
7

---

Table 1: Testing accuracy on CMNIST and C-CIFARIO, considering diverse percentages of bias- conflicting samples. Baseline results for C-CIFARIO are taken from Liu et al. (2023), as we employ the same experimental settings. For CMNIST, we utilize the official repositories to obtain the models.
Model requirements for spurious attribute annotations (type) are indicated by x (not required) and / (required).
Methods
Grop
CMNIST
C-CIFARIO
Info
1.0
1.0
Group DRQ
39.67
71.33
76.30
84.40
33.44
38.30
45.81
57.32
ERM JTT
35.34 (0.13)
50.34 (0.16)
62.29(1.47)
17.63 (0.13)
23.08 (1.25)
25.82 (0.33)
30.06 (0.71)
39.42 (0.64)
(3.89)
74.23 (3.21)
(0.60)
(0.31)
33.40
42.20 (0.31)
LIF
63.39 (1.97)
74.01 (2.21)
80.48 (0.45)
85.39 (0.94)
28.57 (1.30)
33.07 (0.77)
39.91 (0.30)
50.27 (1.56)
DFA
X
59.12
71.04 (1.02)
82.86 (2.27)
88.29 (1.50)
29.95 (0.71)
36.49 (1.79)
11.78 (2.29)
51.13 (1.28)
63.48 (5:22)
78.41
83.63 (1.43)
88, (8.59
34,56 (0.69)
47.81 (1.26
54.55 (1.26)
DeNeDDM
74.72 (0.99)
(0.76)
89.29 (0.51)
(0.22)
(1.16)
44.20 (0.77)
47.35 (0.70)
56.30 (0.42)
Table 2: Testing accuracy BAR, BFFHQ, and CelebA. The test set for BAR and BFFHQ contains only binfcconf samples. Baseline method results are derived from Lim et al. (202)) for BAR Liu et al. 2023) for BFFHQ, and Park et al. (2023) for CelebA on the same dataset split since we utilize identical experimental settings.
Methods
Group Info
BAR
BFFHQ
CelehA
1.0
5.0
1.0
ERM
x
57.65 (2.36)
68.60 (2.25)
56.7 (2.7)
47,02
JTT
x
58.17 (3.30)
68.53 (3,29)
65.3 (2.5)
76.80
LIF
X
57.71 (3.12)
67.48 (0.46)
(1.6)
DFA
X
52.31 (1.00)
(1.47)
63.9 (0.3)
65,26
LC
X
70.94 (1.46)
74.32 (2.42)
70.0 (1.4)
DeNetDM (ours)
X 73.84 (2.56)
79.61 (3.18)
75.7 (2.8)
81.04
We conduct five independent trials with different random seeds and report both the mean and standard deviation to ensure statistical robustness.
Implementation details: We perform extensive hyperparameter tuning using a small unbiased validation set with bias annotations to obtain the deep and shallow branches for all the datasets. We consistently utilize the same debiasing model architectures used by the previous methods for our target branch to ensure fair comparison. Additionally, linear layer is employed for the classifier for all the datasets. The additional architecture details for different datasets are as follows: (1) CMNIST: we use an MLP with three hidden layers for the deep branch and an MLP with single hidden layer corresponding to the shallow branch. During the second phase of DeNetDM, we use an MLP with three hidden layers for the target branch. (2) C-CIFARIO, BAR: we use the ResNet-20 architecture for the deep branch and a 3-layered CNN model for the shallow branch. The target branch used the second stage of DeNetDM is ResNet-18. 18. (3) BFFHQ, CelebA: we use the ResNet- 18 architecture as the biased branch and a 4-layered CNN as the shallow branch. We also use the ResNet-18 architecture for the target branch, following the approaches of Liu et al. (2023): Lee et al. (222)) Further details on the datasets and implementation are presented in Section 7.8
4.2 Evaluation Results
We present a comprehensive comparison of DeNetDM with all the baselines described in Section 4.1 across varying bias conflicting ratios on CMNIST, C-CIFAR 10, BFFHQ, BAR and CelebA in Table and Table respectively As evident from Table I and Table DeNetDM consistently outperforms all basclines across different bias ratios for CMNIST, BFFHQ. BAR and CclebA datasets. Notably, on the C-CIFARIO dataset, DeNetDM exhibits superior performance when bias ratios are at 0.5%, 1,, and 5%, and closely aligns with Liu et al. (2023) in the case of 2%. These findings provide evidence for the practical applicability of DeNetDM. is worth mentioning that the proposed approach demonstrates significant performance enhancement across all datasets compared to Group DRO, which relies on predefined knowledge of bias. DeNetDM achieves this improvement without any form of supervision on the bias, highlighting the effectiveness of depth modulation in the debiasing.
An intriguing observation from Table iis is that DeNetDM demonstrates better performance compared to the basclines when the bias-conflicting ratio is lower, particularly evident in the C-CIFAR10 dataset.
8

---

We believe that the effectiveness of inductive bias enforced by DeNetDM in distinguishing between core and bias attributes superior that of LC, thereby allowing to adeptly capture core attributes even when dealing with data points that exhibit fewer bias conflicting points. This emphasizes the applicability of DeNetDM in scenarios where the training data exhibits a significant amount of spurious correlations. Another noteworthy observation in Table 2 is that DeNetDM outperforms LC and DFA by a considerable margin across all datasets, particularly on the complex real-world datasets, BAR and BFFHQ. Both LC and DFA rely on augmentations to enhance the diversity of bias-conflicting points, whereas our approach utilizes depth modulation to efficiently capture the core attribute characteristies in the existing training data. Despite this, DeNetDM still achieves superior performance compared to LC and DFA without relying on augmentations.
4.3 Analysis of Training Dynamics
In Section 3.2 we discussed the variability in linear decodability at various depths and its sig- nificance as motivation for debiasing. To further validate this intuition and identify the ele- ments contributing to its effectiveness, we delve into the training dynamics of DeNetDM during initial stages. We consider the training of Colored MNIST with 1% skewness due to its sim- plicity. Figure 3 shows how linear decodability of attributes varies across different branches of DeNetDM during training. As depicted in Figure 3 prior training. the deep branch demon- strates lower linear decodability for both the digit identity (core attribute) and color (bias at- tribute) compared to the shallow branch. As training progresses, the bias attribute, easier to learn, rapidly increases in linear decodability in both branches, labeled 'A' in Figure 3 Here, the disparity linear decodability be- tween digit identity and color attributes becomes more pronounced in the deep branch than in the shallow one. This distinction serves as prior, influencing the deep branch to effectively capture the bias. Since employ Product of Experts technique, the deep branch becomes proficient in classification using the spurious at- tribute, thereby compelling the shallow branch to rely other attributes such digit for the classification. It is worth noting that the lin- ear decodability of core attributes is more pro- nounced in the shallow branch, allowing them capture the core attributes. Thus, the training paradigm of DeNetDM leads to a shallow branch that is robust to spurious correlations, and deep branch that majorly relies on the biased attribute. This analysis confirms our intuition and provides empirical evidence of effective debiasing.
Limwer Decodaabley 1941
Shallow Ketwor' (TAye)
Shallow Keewor' (Tabor
Doeg Netooek (Digt)
Training Beerston
Figure 3: Early training dynamics of DeNetDM.
4.4 Ablation Studies
We perform several ablation studies evaluate different facets of DeNetDM. We scrutinize the effect of various loss components on the performance of DeNeIDM. Additionally, we explore the influence of network depth, fundamental element of DeNetDM, and the sensitivity of DeNetDM number of parameters which are discussed in Section 7.6 All the experiments are conducted on CMNIST and C-CIFAR10 10 datasets where the ratio of conflicing points is set to 1%. Additional experiments and ablations are also provided Section 7.6
Table Ablation study of different losses used in DeNetDM on C-CIFAR1O.
Lee (Stage-1)
East (Siage-2)
E (Stage-2)
Accuracy (%)
Conflicting Accuracy (%)
Aligned Accuracy (%)
1)
37.47
42.89
35.74
81.60
42.25
38.34
68.52
43.12
39.46
69.53
Effect of loss components: We conduct ablation studies C-CIFAR10 by selectively removing components to analyze their impact on the testing set accuracy as well as accuracy on bias-aligned and bias-confficting points. The results are summarized in Table 3 When considering Cee alone,

---

corresponding to the first stage of MeNetDM involving depth modulation, the model achieves 37.42% accuracy, showing a strong ability learn target attributes. However, introducing the second stage of DeNetDM training with L; alone leads to capturing significant bias information alongside core attributes, evidenced by high accuracy on aligned points (81.60%) When introducing Lais alone, the model distills knowledge from the shallow branch obtained in the first stage, resulting in performance similar to stage training. However, performing the second stage DeNetDM training using both C, and Cdist prevents capturing bias, focusing more learning core features and resulting in improved conflicting and overall accuracy. similar trend can be observed for CMNIST dataset and the results summarized in Section 7.7.2
5
Conclusion
We introduce DeNetDM, novel debiasing method leveraging variations in linear decodability across network depths. Through extensive theoretical and experimental analysis, we uncover insights into the interplay between network architecture, attribute decodability, and training methodologies. DeNetDM employs paired deep and shallow branches inspired by the Product of Experts methodology, transferring debiasing capabilities to the desired architecture. By modulating network depths, it captures core attributes without explicit reweighting or data augmentation. Extensive experiments across various datasets, including synthetic ones like Colored MNIST and Corrupted CIFAR- 10, as well as real-world datasets like Biased FFHQ and BAR, validate its robustness and superiority. Importantly, DeNetDM achieves performance comparable to supervised approaches, even without bias annotations.
Acknowledgments
Silpa Vadakkeveetil Sreelatha is partly supported by the Pioneer Centre for AI, DNRF grant number Pl.
References
Kartik Ahuja, Karthikeyan Shanmugam, Kush Varshney, and Amit Dhurandhar. Invariant risk minimization games. In JCML, 2020.
Sungha Choi, Joanne T. Kim, and Jacgul Choo. Cars fly up in the sky: Improving urban-scene segmentation via height-driven attention networks. CVPR, 2020.
Christopher Clark, Mark Yatskar, and Luke Zettlemoyer. Don't 't take easy way out: Ensemble based methods for avoiding known dataset biases. In EMNLP-IJCNLP. 2019.
James Diffenderfer, Brian R Bartoldson, Shreya Chaganti, Jize Zhang, and Bhavya Kailkhura. A winning hand: Compressing deep networks can improve
robustness. In NeurlPS, 2021.
songwei Ge, Shlok Mishra, Chun-Liang Li, Haohan Wang, and David Jacobs. Robust contrastive learning using negative samples with diminished semantics. In NeurlPS, 2021.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland
Brendel. Imagenet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. In ICLR, 2019.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In JCLR, 2019.
Katherine Hermann and Andrew Lampinen. What shapes feature representations? exploring datascts, architec- tures, and training. In NeurlPS, 2020.
Geoffrey Hinton. Training products of experts by minimizing contrastive divergence. Neural Comput. 2002.
Youngkyu Hong and Eunho Yang. Unbiased classification through bias-contrastive and bias-balanced learning. In NewrlPS, 2021.
Minyoung Huh, Hossein Mobahi, Richard Zhang, Brian Cheung, Pulkit Agrawal, and Phillip Isola. The low-rank simplicity bias deep networks. TMLR, 2023.
10

---

Rabeeh Karimi Mahabadi, Yonatan Belinkov, and James Henderson. End-to-end bias mitigation by modelling biases in corpora. In ACL, 2020.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In CVPR, 2019.
Byungju Kim, Hyunwoo Kim, Kyungsu Kim, Sungjin Kim, and Junmo Kim. Learning not to leam: Training deep neural networks with biased data. In CVPR, 2019.
Eungyoup Kim, Jihyeon Lee, and Jaegul Choo. Biaswap: Removing dataset bias with bias flasttallored swapping augmentation. In ICCV, 2021.
Pang Wci Koh, Shiori Sagawa, Hemrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, Tony Lee, Etienne David, Ian Stavness, Wei Guo, Berton Earnshaw, Imran Haque, Sara M Beery, Jure Leskovec, Anshul Kundaje. Emma Pierson, Sergey Levine, Chelsea Finn, and Percy Liang. Wilds: benchmark in-the-wild distribution shifts. In ICML, 2021.
Jungsoo Lee, Eungyeup Kim, Juyoung Lee, Jihyeon Lee, and Jaegul Choo. Learning debiased representation via disentangled feature augmentation. In NewrlPS, 2021.
Jungsoo Lcc, Jeonghoon Park, Dacyoung Kim, Juyoung Lee, Edward Choi, and Jacgul Choo. Revisiting the importance of amplifying bias for debiasing. AAAI, 2023.
Jongin Lim, Youngdong Kim, Byungjai Kim, Chanho Ahn, Jinwoo Shin, Funho Yang, and Seungju Han. Biasadv: Bias-adversarial augmentation for model dehiasing. In CVPR, 2023.
Evan Z Liu, Behzad Haghgoo, Annie S Chen, Aditi Raghunathan, Pang Wei Koh, Shiori Sagawa, Percy Liang, and Chelsea Finn. Just train twice: Improving group robustness without training group information. In ICML, 2021.
Sheng Liu, Xu Zhang, Nitesh Sekhar. Yue Wu, Prateek Singhal, and Carlos Femandez- Avoiding spurious correlations via logit correction. In ICLR, 2023.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep Learning Face Attributes in the Wild. In ICCV, 2015.
Puspita Majumdar, Richa Singh, and Mayank Vatsa. Attention aware debiasing for unbiased model prediction. In ICCVW, 2021.
Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. survey on bias and fairness machine learning. ACM Comput. Surv. 2021.
Junhyun Nam, Hyuntak Cha, Sungsoo Ahn, Jaeho Lee, and Jinwoo Shin. Learning from failure: De-biasing classifier from biased classifier. In NeurlPS, 2020.
Geon Yeong Park, Sangmin Lee, Sang Wan Lee, and Jong Chul Ye. Training debiased subnetworks with contrastive weight pruning. In CVPR, 2023.
Jiaxin Qi, Kaihua Tang. Qianru Sun, Xian-Sheng Hua, and Hanwang Zhang. Class is invariant to context and vice versa: On learning invariance for duttr--distributtonn generalization. In ECCY, 2022.
Olivier Roy and Martin Vetterli. The effective rank: A measure of effective dimensionality. In ESPC, 2007.
Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distribationally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. In ICLR, 2020.
Samuel S. Schoenholz, Justin Gilmer, Surya Ganguli, and Jascha Sohl-Dickstein. Deep information propagation. In ICLR, 2017.
Seonguk Seo, Joon- Joon-Young Young Lee, and Bohyung Han. Unsupervised learning of debiased representations with pseudo-atrriutes. In CVPR, 2022.
Harshay Shah, Kaustav Tamuly, Aditi Raghunathan, Prateek Jain, and Praneeth Netrapalli. The pitfalls of simplicity bias in neural networks. In NewrlPS, 2020.
Robik Shrestha, Kushal Katle, and Christopher Kanan. Occamnets: Mitigating dataset bias by favoring simpler hypotheses. In ECCV, 2022.

---

Nimit Sohoni, Jared Dunnmon, Geoffrey Angus, Albert Gu, and Christopher Ré. No subelass left behind: Fine-grained robustness in coarse-grained classification problems. In NeurlPS, 2020.
Rishabh Tiwari and Pradeep Shenoy. Overcoming simplicity bias in deep networks using feature sieve. In ICML,202.
Vladimir Vapnik. An overview of statistical learning theory. IEEE TNN, 1999.
Haohan Wang, Zexue He, 7achary I.. Lipton, and Eric P. Xing. Learning robust representations by projecting superficial statisties out. In ICIRR, 2019.
Zihan Wang and Arthur Jacot. Implicit bias of SGD in $1 2S-regularized linear DNNs: One-way jumps from high to low rank. In ICLR, 2024.
Zeyu Wang, Klint Qinami, Ioannis Karakozis, Kyle Genova, Prem Nair, Kenji Hata, and Olga Russakovsky. Towards fairness in visual recognition: Effective strategies for bias mitigation. In CVPR, 2020.
Wikipedia. Cosine similarity, sttps://en..ikipediaaooog. Chclidean, distance atsiiddeaanddananeeeeeeeeeee
mormallized
2024.
D.H. Wolpert and W.G. Macready. No free lunch theorems for optimization. IEEE TEC, 1997.
Dinghuai Zhang, Kartik Ahuja, Yilun Xu, Yisen Wang, and Aaron Courville. Can subnetwork structure be the key to out-of-dissrihution generalization? In ICML, 2021.
Hongyi Zhang, Moustapha Cissé, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In ICLR, 2018.
12

---

7 Appendix
In the primary text of our submission, we introduce DeNetDM, a novel debiasing framework that leverages the variation of linear decodability across network depths effectively disentangle bias from core attributes. This technique sets a new benchmark for bias mitigation, achieving unparalleled performance without reliance data augmentations. To ensure manuscript's integrity, provide an extensive appendix designed to complement the main text. This includes a series of additional experiments, expanded ablation studies, comprehensive implementation protocols, and deeper analyses of our findings. The Appendix is presented to bridge the content gap necessitated by the page constraints of the main manuscript, providing detailed exposition of our methodology and its broader impact on the domain.
7.1 Notations
B: Bias attributes
C: Core attributes
X, Y: Sample set (X: Inputs, Y: Labels)
x Bias-eligned points
X6: Bias-conflicting points
Φc Encoder
|d] Encoder at depth d
p(o(d)): Effective rank of an encoder at depth d
T2 (d: Propagation probability of an attribute (indexed i) at depth d
Q4: Propagation probability distribution of an attribute set at layer d of a neural network
rank(-): Rank of matrix
dim(-): Dimensionality of tensor / space
e: Knock-off probability when transitioning from depth d +
7.2 Proofs
Theorem (Partition Rank): When the partitioning X, U.2, is stable wrr. C, the rank of the bias-aligned partition is upper-bounded by the rank of the bias-conflicting partition, i.e.,
rank(Xa) rank(X.)
Proof. The theorem assumes stable partitioning of the sample set X, i.e., in both the bias-aligned and conflicting subsets, the distribution of the core attributes are equal to that of the original sample set, i.e.,
P(X.) P(X P(X
Under this condition, the only component in either of the subsets that determines the subset's 's rank should be the bias attributes, under the simplifying assumption (without loss of generality) that the attribute space is made up of only the core and the bias attributes.
For the bias aligned partition X,, all the data points within class have very low variance within the set of values for the bias attribute B, since is spuriously correlated with the class label. So, B within class collapses
much lower dimensional manifold b B, such that rank(b) Extending this across classes, without loss of generality, assuming that the number of classes higher than the variance in B among the bias aligned samples, i.e., rank(b), B over the set of all classes in X. would map to manifold of dimensionality rank(b). Therefore, since the whole of B in X, can be represented by manifold of dimensionality rank(b) orthogonal to the basis of C, the rank of X. is given by:
dim( (B).
rank(Xa) rank(C) + rank(b)
13

---

For the bias conflicting partition, since there is no correlation between the class labels and B, within each class, the bias attributes would require a dim (B) dimensional subspace independent of C, be represented, since B 1. C. This implies that the rank of the bias conflicting points would be:
rank(X.) rank(C) + dim(B).
dim(B) it is ultimately implied that Since we know that S B, which leads rank(b) x rank(Xa) S rank(X.)) (X.).
This completes the proof of the theorem.
Lemma 1. Let A = [Ao, A1,..., An) be the attribute subspace of X with increasing ranks, i.e., < rank( A,), such that every A E A maximally and equally informative of the label Y, Le., T(Ao, = T(A1, =.... -...:[(An.. Y). Then, at any given depth d of neural network, the probability of propagation (ı(d) of an attribute A, directly proportional to the effective rank plold) of the network that depth, i.e.
rank( Ao) < rank A1)
(r (o(old))
Proof. Let the total rank of A be R. Consider some reference attribute A E A with rank According to the results on the low rank simplicity bias (Huh al. 2023 Wang and Jacot 2024) and deep information propagation (Schoenholz et al., (017), after propagation through each layer, slt of the bases would be knocked off, resulting in a pruned version of A. The total number of ways in which
A can be pruned given by
{R Also, the number of ways that A features in that pruning is
given by (eR) Thus, the probability of A being knocked-off in layer-1 1 of á is given by:
eR
R ER
r!
(eR-r).R((1. R!
Therefore, probability of survival layer d:
r! R R
Tx(d)=
1
(ER
Therefore, the probability of survival n. (d) of any attribute A, at depth d increases exponentially with increasing rank r of A;, and decreases exponentially with the knock-off rate E. is the part of the knocked-off basis not in Ac. is the part of the complete basis of A not affected by first knock-off at layer 1. Thus, at depth d, &d indicates the size of the subspace of A that survives depth d, therefore being proportional to the effective rank of Φ[d]. Based on this, the effective rank at depth d can be written as:
T. (d) (1- e) O () (p(old)))
This completes the proof of the lemma.
Lemma Let A = [Ao, AI,.., An] be the attribute subspace of X with increasing ranks, i.e., rauk(Aoo) < rank(Ai) < < rank(AA), such that every A E A is maximally and equally informative of the label Y, i.e., I [(Ag, =)= =.... of neural network. the propagation probability of attribute decreases with rank, i.e..
-...= I[AA,, Y). Then, at any given layer
(id T2(d) 2 (d),,,
at rate that is polynomial the attribute rank, with degree equal to the depth, i.e.,
n; (d)
(d-
n (d)
14

---

Proof. Continuing from Equation 9 we have the propagation probability of A; at depth d as:
d
r! RR (1 R
Tz(d)=
(ER
Note that when increases, i.c., for higher rank attribute, it leads to drop in and in subsequent exponential decrease in (d) as follows:
T++ (1 (ER (r k)).R!( R.
(r + RD)
d
Te+b(d)
(d(d) S
Tith (d) S (()d)
(i) 2 (2(d) 2 2...... (d),,,
which proves the first part of the lemma. Now, taking the ratio the propagation probabilities of attributes with rank (i + k) and i depth d, get:
(r 1) (m+ O ((+=+)) =0((== k: d) ) 0(K4T-))
t (d) T (d)
O
For propagation on to the next layer, ht = 1. We thus have:
T 1 (d) = (r
(d)
This completes the proof of the lemma.
Theorem 2 (Depth-Rank Duality): Let = [Ao, Ai..., An] be the attribute subspace of X with increasing ranks, i.., rank (Ao) < rank(A1) < ... < (rak(A,)) such that every A E A is maximally and equally informative of the label Y, i.e., [(A,, Y) [(A,Y) =... =(A,, Y). Then, across the depth the encoder Φ, SGD yields a parameterization that optimizes the following objective:
min [(f(o(())) Y) min Σ [l Ωd о AAl 0.
ERM
where -(.,-) is the empirical risk, f(-) is a classifier head, (-1](-) is the output of the encoder (optimized end-to-end) at depth d, 11-1l/ is the [--norm, @ is the element-wise product, X is the lz-normalized version of X, X2 = (d); Il m.) 1I n. (a)). 1 is random binary function that outputs with probability æ, and n. (d) () is the propagation probability of A; at depth d bounded as:
n (d)
(p(old));
where p(old)) is the effective rank of the @ b representation space, and r; = rank(A.).
Proof. Since all A AA are equally informative about the label Y, they all equally minimize C(,, +). Thus, the representations learned by & are solely determined by the second term in the summation of Equation ( This means that the SGD must employ a selection mechanism to choose from the A that optimally utilizes the available parameters A.
If éld] has sufficiently many parameters to accommodate all of A, SGD should have no reason to discard any of them. However, a number of works that analyze the representational properties of DNNs have found that as we go deeper into network, the effective number of dimensions available for encoding information, formally known as the effective rank and denoted as p(o(d) (effective rank of e at depth d), decreases (Huh al. 2023 Wang and Jacot 2024). This characteristic is also known
15

---

as the simplicity bias of DNNs. Given the simplicity bias, SGD must learn a parameterization for ø that optimally selects from A when the effective rank at particular layer is lower than rank(AA). In order to stay at the minimum of ((: & must rely on the complete basis of at least one attribute, as only partially learning an attribute would cause deviation from the minimum. So every attribute that is retained for prediction, has to be retained fully. Given this condition, the optimum choice for SGD under constrained effective ranks is thus, to choose A E in increasing order of effective ranks. In other words, the Ag has the highest likelihood of getting chosen, followed by A1, then A2, and so on.
Lemma I and Lemma 2 provide bounds for the quantification of the associated probabilities at given depth, for an attribute of a given rank. Combining them, we get the propagation probability of A; at depth d as:
T&. (d) = (((d))))
where r; the rank of A, We denote the distribution of π for an attribute across a network as S2+- Without loss of generality, assuming the retention of all attributes at depth d - 1, we get the forward pass output at depth d as:
([d[()) - <WW, øld 1](X)),
where Wa is the weight matrix at layer d and gamma is non non-linearitty Under the most general setting where the elimination of attributes comes only with a decrease in the effective rank and not in the reduction in the dimensionality of the weight matrix, applying Lemmas [ [ we obtain the survival probability of the basis corresponding to all A E A in W as:
Wd = [1 (d)) W (d) Wi:... 1 Ix (c) Will
Wd. X' (d) W Ao: m (n) WT A1: ... 1 Wa)
= 22 ② A.A. W
where 1, is random binary function that outputs 1 with probability Ω [1 (a); 1 (-)) 1 (.)( and X' = Φ|d - I)) To keep & at a minimum, W must cor- rectly activate for the informative features in x', for which it must maximize Ωd A. Wd. Now, Qd @ A. Wd is maximized when n' Ω @ A. Thus, the optimal strategy for SGD is param- eterize W' such that it captures the attributes in A according to the distribution Ω Over the full depth, the optimization objective would then be:
max Σ
22dt C A (l)(((()) min Σ E1lodd(x)) - 52d o A412
where X is the lz-normalized version of X, and the equivalence comes from the equivalence of maximizing the dot product and minimizing the ly-distance of the normalized samples (Wikipedial 2024].
This completes the proof of the theorem.
be the attribute subspace of X with increasing ranks, i.e., rank(Ag) (Ao) rank(A, < rank(A,) such that every A A is maximally and equally informative of the label Y, i.e., ([A,YY)= I[(1,,)) I..,, Y). Then, across the depth of randomly initialized encoder 6, the outpul of & at depth follows the propagation distribution Qi of the attribute space A aas:
Corollary 2.1. Let A [Ao,
(X) x Ω2d @ A,
(10)
where (.) is the output of the encoder 6 at depth d, O is the element-wise product, X is the Ia-normalized version of X, Ω = [1mm (d); [x(d) ...; 1 (d)), 1, is a random binary function that outputs with a probability n, and (T(d) is the probability of propagation of A, of rank r; at depth bounded as:
T (d) = O (p(old)r=
(11)
Discussion: Let IL be the space of all empirical risks {L1, L2... over X. According to the No Free Lunch theorem (Wolpert and Macready 1997). an attribute minimizes some L. E LL, there exists another L, E IL which it maximizes. So, if we consider the probability of survival of attributes in a randomly initialized network, we need to marginalize the ERM part of Equation 4T) across the entirety of L. Assuming an unbiased random initialization scheme, the distribution associated with
16

---

L would be uniform (because concrete form of empirical risk is defined, we can consider all functions be equally likely, under the unbiased initialization assumption) follows:
/ CU(()(X))) Pr(C) dC, DEL
where Pr(C) the probability associated with the function L L, which can be assumed to be uniform, as argued before. Then, due to the No Free Lunch Theorem (Wolpert and Macready 1997). the expected informativeness of all attributes in X is the same, satisfying the [A4., Y) I (An, Y) criterion in the theorem, where the nature of Y is determined by the specific choice of L. The remainder of the reasoning for (X) x Q C A is the same as the proof for mins >n [X) 22d @ All in Theorem
[[A1 Y) =
7.3 Equivalence with Product of Experts Framework
In Section 3.2 of the main text, we asserted that our training methodology is derived from the Product of Experts. In this section, we elucidate this mathematically:
f f:R. R.F.F. lincar Rº f(x) softmax(f(x))
Φb RCCHW -RF where F' is the feature dimension
Φd RCCHW R such that depth(ds) > depth(os)
C
L(2,8i 0d) - Σ fc(ov(z)) Fc(daxx)
Ue log(Ps (Loss function definition)
dos
(Product of Experts)
E. f((o)()))) fc(d((())
softmax.((og((((((())) + log(F(pa(z)))
(Softmax log-sum-exp trick)
+ +
(Translation invariance softmax)
+ (Linearity of classifier f')
We utilize oo to compute the probabilitics in DeNetDM which is the same as Equation 2 presented in the main paper.
7.4
Pseudocode
The pseudocode for the entire training process of DeNetDM is provided in Algorithm 1
7.5
Feature Decodability
We utilize feature decodability to gauge the extent to which specific dataset features can be reliably decoded across models of varying depths. Hermann and Lampinen (2020) demonstrated that the visual features can be decoded from the higher layers of untrained models. Additionally, they observed that the feature decodability from an untrained model has significant impact in determining which features are emphasized and suppressed during the model training. Following their approach, we specifically focus on assessing the decodability bias and core attributes from the penultimate layer of untrained models. In order to evaluate the decodability of attribute in dataser, we train decoder to map the activations from the penultimate layer of frozen, untrained model attribute labels. The decoder comprises single lincar layer followed by softmax activation function. The
17

---

Algorithm DeNetDM: Training
Input: Data: ((x,
Output: Øz, ft.
Initialize: @c. ft, fo 4o. da such that (épth(6)) > depth(oa)
1: repeat
2: Fetch minibatch data {(z, 3=)
3: for - 1 to K (in parallel) do
4:
Compute p using (4) to obtain (f. y);
5: end for
6: Update Φb. Φb. I by minimizing Lcx (3 via SGD
7:
until Convergence
stagel
8: repeat
9:
Fetch minibatch data {(z, y).
10: for 1 to K (in parallel) do
11:
Compute D. Ps. Pe. via 5
respectively
12: end for
13: Update di fo by minimizing C in via SGD
14:
until Convergence
stage2
Early ERM Training Dynamies for C-CIFAR-10
Linear Devodability (%)
Shallow Network (Cijeet)
Shallow Network (Corruption)
Deep Netweck (Ofject)
Deep (Comuption)
8
Training iterations
Figure 4: Early training dynamics of DeNetDM on C-CIFAR10 dataset.
decoder is trained using an unbiased validation set associated with the dataset, where each instance is labeled according to the attribute under consideration. Subsequently, the linear decodability of the attribute, measured in accuracy, is reported on the unbiased test set. We investigate the decodability of digit and color attributes in the CMNIST dataset from MLP models with varying depths, including 3, 4, and 5 layers, and the results are depicted in Figure 2a] To investigate how feature decodability evolves during the early stages of Empirical Risk Minimization (ERM) training across networks with varying depths, we train 3-layer and 5-layer MLPs on the CMNIST dataset. Following the training, we evaluate the model's linear decodability for digit and color attributes.
7.6
Additional Experiments
7.6.1 Feature Decodability C-CIFAR10
Analogous to Figure 2 in the main paper. Figure 4 iillsstrates the variation in feature decodability for corruption (bias) and object (core) in the C-CIFARIO dataset as ERM training advances. We chose ResNet 20 as the deep network and 3-layer CNN as the shallow network since these are the architectures used for DeNetDM. The training dynamics show similar trend those observed in ColoredMNIST concerning bias and core attributes. As training progresses, corruption (bias) becomes highly decodable by both deep and shallow networks, with the deep branch slightly outperforming the shallow branch. However, the object attribute (core) is more decodable by the shallow network as training progresses, during the initial training dynamies. These observations align with the early training dynamics observed in CMNIST.
18

---

7.7 Generalization to other tasks
We evaluate the performance of DeNetDM on the CivilComments dataset Koh et al. (202I), which involves natural language debiasing. The task requires classifying online comments toxic toxic. with labels spuriously correlated with mentions of certain demographic identities. As shown in T our approach performs comparably to state-of-the art methods. Due to the constrained rebuttal timeline, we just applied our model out of the box, without any reasonable hyperparameter tuning. The observations illustrate the applicability of DeNetDM to domains beyond vision.
Table 4: Worst group accuracy (%) comparison between different methods on CivilComments dataset.
Method
Worst Group Acc (%)
ERM
(8.661.7)
JTT
69.3 (-)
LIF
58.3 (0.5)
LC
70.3
DeNetDM (ours)
68.33 (-)
7.7.1 Effect depth modulation
To validate our hypothesis on the significance of network depth in DeNetDM, we conduct an ablation by setting the same depth for both branches and compare with the default DeNetDM, where one branch is deeper than the other. We focus on the first stage of DeNetDM training for 5 different random seeds, reporting the averaged test accuracy on bias-aligned and bias-conflicting points for individual branches in Table Branch I and Branch 2 in Table S correspond to the deep and shallow branches in DeNeIDM, respectively. We ignore the second stage of training since our focus was primarily on the segregation of bias and core attributes. An interesting observation is the significant standard deviation in accuracies when the branches have the same depth, observed in both datasets. This phenomenon occurs because in such a configuration, DeNetDM loses its ability to clearly distinguish between branches. This due to the similarity feature decodability of bias and core attributes across both branches, as discussed in Section 3.2 As result, DeNetDM may distribute information across multiple branches or still separate core and bias attributes, but the specific branch capturing core attributes varies with different initialization. In contrast, when depths unequal, the deeper branch tends to focus on aligned points, disregarding conflicting ones, as seen in the test accuracies provided in Table 5 Additionally, the shallow branch emphasizes capturing core attributes, consistently enhancing conficting accuracy. This shows the pivotal role depth modulation in the DeNetDM framework for effectively segregating bias and core attributes.
Table 5: Performance of DeNetDM using different network depths for the two branches of DeNetDM.
Dataset
Depth
Branch
Conflicting Accuracy (%)
Aligned
(Branch Branch 2)
Accuracy (%)
Branch
44.94 (22.25)
74.85 (12.71)
(5
Branch
17.25 (7.89)
88.57 (9.50)
CMNIST
Branch
1.921 (0.29)
99.92 (0.25)
(5. 3)
Branch
83.17 (0.96)
88.25 (2.254)
Branch
19.54 (11.16)
85,83 (8.19) 19)
(ResNet-20,, ResNet-20)
Branch
24.42 (16.93)
86.95 (11.04)
C-CIFARIO
Branch
3.0 (1.29)
99.34 (0.47)
(ResNet-20,, 3-layer CNN)
Branch 38.52 (0.99)
76.72 (2.19)
7.7.2 Effect of loss components on CMNIST
The primary text, constrained by spatial limitations, only includes an ablation study detailing the effect of individual loss components of DeNetDM on the C-CIFAR10 10 dataset. However, this section extends the scope of our analysis to encompass the CMNIST dataset and the results are summarized
19

---

in Table 6 The proposed approach exhibits a similar trend as observed in the case of C-CIFARIO
(presented Section
Table 6: Ablation study of different losses used in DeNetDM on CMNIST dataset.
LCE (Stage-1) 1)
Ldist (Stage-2)
L (Stage-2)
Accuracy (%)
Conflicting Accuracy (%)
Aligned Accuracy (%)
81.61
83.28
89.66
/
82.96
81.53
95.85
84.05
83,41
89.86
/
/
84.97
84.44
89.17
7.7.3 Depth vs. Number of parameters
DeNetDM employs depth modulation its principal strategy for mitigating bias. We investigate the influence the number parameters of both branches on DeNetDM performance. We opt for the optimal configuration of the proposed approach on C-CIFAR 10 and conducted an ablation study, employing NesNet-20 (depth(0s) = 20) as the deep network and a 3-layer CNN (depth(ou)) = 3) as the shallow network. We explore three scenarios where 1 loal. l0bl loal. and [øs] loal- The first stage of DeNetDM training is then performed to analyze learning in the deep and shallow models in each of the cases, and the results shallow model exhibits increased resilience to spurious correlations, while the deep model captures bias in all three cases. This suggests that DeNetDM effectively segregates bias and core attributes regardless of the number of parameters in both branches. Interestingly, a notable finding is that the shallow model exhibits better robustness against correlations when the shallow branch possesses greater number of parameters compared to the deep model, as evident from Table
presented in Table 7 As indicated in Table 7 the
The findings for CMNIST mirror those observed for C-CIFARIO presented in Table the shallow branch demonstrates robustness to spurious correlations, whereas the deep branch consistently assimilates bias irrespective of the number of parameters in both branches. These consistent patterns across datasets reinforce the efficacy of DeNetDM in distinguishing between bias and core attributes.
Table 7: Ablation study the number of parameters of deep and shallow branches in DeNetDM using C-CIFAR10 10 dataset.
Case
Branch
Conflict (%)
Align (%)
> Ød
Φ6 3.08
96.8
@d
29.78
62.61
Φb
R3
dd
Ø 3,48
95.91
28.64
64.32
Φb dd
Φb
2.04
99.01
39.05
67.68
Table 8: Ablation study on the number of parameters of deep and shallow branches in DeNetDM using CMNIST dataset.
Case
Branch
Conflicting Accuracy (%)
Aligned Accuracy (%)
Φb
<
0d
db
11.90
99.93
Ød
83.89
88.78
Φb
R
0d
db Φd
11.87
99,90
83.07
89.09
$b
Ød
db
10.79
98.26
@d
83.32
88.61
20

---

7.7.4 Effect of Network Depth on DeNetDM
In the main text, have illustrated how the variation in network depth affects the performance of Me We provide an in-depth analysis in this section. As observed in the first three rows of Table 9 the difference in network depth of deep and shallow progressively increases, the performance of the debiased model increases monotonically. Further, when we decrease the difference depth of shallow and deep branches (rows 3 and 4) the performance decreases 80.42% compared to 87.37% Similar performance degradation can be seen when we increase the depth of the shallow network from
(rows 5 and 6). Hence, DeNetDM able to distinguish bias and core attributes better when there is significant difference between the depths of shallow and deep branches. This aligns with the observations presented Section 3.2 of the main text (Effect of depth modulation).
Table 9:
Performance comparison of DeNetDM for various depths of shallow and deep branches.
Depth (Shallow, Deep)
Conflicting Accuracy (%)
Aligned Accuracy (%)
(3, 4)
72.2
98.33
(3, 5)
80.46
92.87
(3, 7)
87.37
93.62
(6, 7)
80.42
96,45
(4, 8)
91.19
94.62
(6, 8)
69.55
93,83
7.7.5 Performance on varying bias-conflicting ratios
We perform experiments the CMNIST dataset with bias-conflicting ratios of 10% and 20% to evaluate our method's 's efficacy across a broader range of ratios. The findings, presented in Table IO show that DeNetDM performs as expected, effectively capturing core attributes the shallow branch for varied bias ratios.
7.7.6 Early training dynamies in ResNet architectures
We also examine the early training dynamics of ResNet-8, ResNet-32, and ResNet-50,, akin to Figure 2b in C-CIFARIO dataset to assess the scalability of DeNetDM to larger ResNet models. After 200 iterations, texture (bias) decodability in all architectures neared 99%, while core attribute decodability for ResNet-8, ResNet-32, and ResNet-50 was 18. 18774% 24.32%, and 12.91 % respectively. This aligns with our hypothesis that ResNet-50 would prefer texture attribute over core when paired with R or ResNet-32. To confirm, we tested two setups: (1) ResNet-8 and ResNet-50, and (2) ResNet-32 and ResNet-50. The results, shown Table IT indicate high bias-aligned accuracy for 5esNet-50 and high bias-conflicting accuracy for ResNet-8 and ResNet-32 respectively. Since ResNet-50 has lower core attribute decodability than ResNet-8 and ResNet-32, favors bias attributes, while the shallow branches capture core attributes. This experimental results suggest DeNetDM's applicability to diverse, complex and larger models architectures.
7.8 Additional details
In this section, we provide an in-depth discussion of various datasets used along with finer implemen- tation details that enhance the reproducibility of our method.
Table 10: Results on CMNIST with wider bias conflicting ratios.
Bias ratio
Branch
Conflicting Accuracy (%)
Aligned Accuracy (%)
Deep Shallow
1.84(0.5) 93.12(0.8)
99.14(0.2)
10%
96.47(1.3)
Deep
3.23(2.8)
97.93(2.1)
20%
Shallow
94.49(2.4)
97.51(3.4)
21

---

Table 11: Comparison of the performance of Mr using different network depths for the two branches of Dr
Depth (Branch 1, Branch 2)
Branch
Conflicting Accuracy (%)
Aligned Accuracy (%)
Branch
3.48 (0.98)
97.15 (2.10)
(ResNet-50, ResNet-32)
Branch 2
30.88 (1.22)
81.72 (0.73)
Branch
9.38 (1.52)
98.60 (0.86)
(ResNet-50, ResNet-8)
Branch 2
20.32 (1.90)
59.94 (2.61)
7.8.1 Datasets
We provide a detailed description of various datasets used along with a representative sample of all of them.
Colored MNIST(CMNIST), CMNIST is an adaptation of the MNIST, that introduces color variation to the images. For each digit class, the majority (1 - a) of the images are correlated with the corresponding color G, with i matching the digit label The remaining images randomly assigned one of the other colors Cj. where j + y. The challenge of this dataset lies in identifying the digits despite the strong color bias. To incorporate color variability, a noise vector drawn from a normal distribution is added to c.. The dataset and its characteristics are illustrated in Figure Among multiple choices of severity, we choose the most severe corruption to simulate the worst-case scenario as done in other works.
Corrupted CIFAR10 (C-CIFARIO): The Corrupted CIFAR dataset represents evolved form of the classic CIFAR set, with an emphasis on two particular features: the object depicted and the type of corruption applied. In approach akin to that used for CMNIST, this dataset adopts an array of corruption styles, labeled from Co. symbolizing blurring, to Cg. indicative of snow. Within each object category, a proportion 1 of the images is intentionally altered with the corruption type ç; corresponding to the object's 's label The remainder of the images is processed with a randomly selected corruption type Cj, chosen ensure j A y. In dataset, we employ the highest degree of corruption out of the five levels outlined in the original CMNIST dataset. Illustrative samples from this dataset demonstrated Figure
Biased FFHQ (BFFHQ P The BFFHQ dataset is a selectively reduced subset derived from the larger FFHQ database of facial images, with a focus the attributes of gender and age. Gender is designated as the primary attribute of analysis, with age being the secondary attribute that could introduce bias. The gender classification is binary, encompassing male and female categories. The dataset predominantly features male images of subjects aged between 40 and 59, whereas female images generally of subjects aged between 10 and 29. Samples that defy these age associations-sute as younger male or older female subjects are are also present, countering the main age distribution.
The Biased Action Recognition (BAR) dataset comprises real- world images classified into six action categories, each biased towards specific locations. The chosen pairs encompass six common action-location combinations: Climbing on a Rock Wall, Diving underwater, Fishing on a Water Surface, Racing on a Paved Track, Throwing on a Playing Field, and Vaulting into the Sky. The testing set consists solely of samples that present conflicts in bias. Consequently, achieving higher accuracy results on this set indicates superior debiasing performance.
Biased Action Recognition (BAR):
7.8.2 Baselines
In this section, we provide detailed overview of the baselines:
Empirical Risk Minimization (ERM) Vapnik 1999); Standard ERM using cross-entropy loss.
Group DRO (GDRO) Sagawa et al. (2000) supervised approach that utilizes group labels to identify the worst group and learn an un unbiased classifier.
22

---

Learning from Failure (LfF) Nam et al. (202)); Identifies bias-conf points through the Generalized Cross Entropy (GCE) loss and upweighting for debiasing.
Just Train Twice (JTT) Liu al. (021): Treats misclassified points by ERM-based classifiers as bias-conflicting and upweights them for debiasing.
Disentangled Feature Augmentation (DFA) Lee et al. (021I): Introducing feature augmentation improve the diversity of bias-conflicting points and enhance unbiased accuracy.
Logit Correction (LC) Liu et al. )2023) Proposes logit correction for bias mitigation along with
MixUp Zhang et al. (2018) inspired data augmentation for increasing diversity.
0
0
0
0
C
O O
1
2
2.
2
2
A 3 4 5
2 3
} 3
5
8
3
3
3
3
3
4
4
9
/
4
4
4
5
S
6
5
5
5
l
6
7
7
7
1
7
7
7
7 8
7 8 4
9
6
8
8
8
9
9
9
Figure Samples from training data of CMNIST, Corrupted-CIFARI and Biased FFHQ
Table 12: Optimal hyperparameters for the CMNIST, C-CIFAR10, BAR and BFFHQ datasets determined through extensive experimentation. The tuples represent optimal hyperparameters for Stage and Stage 2, respectively.
Parameter
CMNIST
C-CIFARIO, BAR
BFFHQ
Learning Rate (LR) Batch Size
(1.0 x 10 x 10-3) (64, 64)
(1.0 x 10-3,10v 1.00 10-4) (1.0 x 10 1.0 x 10-4) (256, 256) 0.9 (1.0 x 10 0) (100, 200)
(64, 64)
Momentum
0.9
0.9
Weight Decay
(1.0 x 10-3.0 0)
(0, 0) (10, 100)
Epochs
(100, 100)
7.8.3 Implementation details
In this section, we detail the optimal hyperparameters identified for various datasets, which were instrumental in achieving the results reported in the main manuscript. The optimal hypeparameters obtained for various datasets are listed Table 12 Additional parameters not mentioned in Table I2 follow the default values of PyTorch.
Data Augmentations: The training phase of DeNetDM incorporated specific data augmentation techniques tailored to each dataset. For instance, the CMNIST dataset did not utilize any form of augmentation. In contrast. the C-CIFARIO and BFFHQ datasets applied Random Horizontal Flip and random cropping, with the latter involving crops from images padded by 4 pixels. These augmentations are critical as they introduce variability into the dataset, aiding the generalization ability of the neural network.
Experimental compute: We utilize RTX 3090 GPUs for all our experiments.
Architectural Details: Depth modulation critical component of our debiasing strategy. We enumerate the architecture specifics of the shallow branches tailored for each dataset below.
CMNIST:
(shallow branch): Sequential
(c1): Linear (in_ fea2ures=2352, out_features-100, bias-True)
23

---

(x1): ReLU()
(s1): MiPHiideellyyere(( laders_ ModuleList(
(0): Linear (in_teatures-100,, out_features--00, bias-True)
(act): : ReLU()
C-CIFARIO and BAR:
(shallow branch): Sequential(
(c1): (3 32, kernel_size--6, 5), stride-(1, 1))
(b1): Batchlorm2d eps-1e-05, momentun=0.1, 1, affine-True, track_rumin meaik_uunnnggsttteTTnut
(r1): ReLU()
(s1): (axemol ceil_mode=Palse)
2), stride=2, padding=O, dilation-1,
(c2): : (32v 64, kernel_size=-6, 5), stride=(1, 1))
(b2) BatchNorm2d eps=1e-05, momentum=0.1. 1, affine=True,
prack_rumiing_state-Tvvu
(x2): : ReLU()
(s2) : MaxPool22(termmllssize (2, 2), stride-2, padding-O, dilation-1,
ceil_mode=False
(c3) (6nv2d(64, 64, bernel_size=fe 5), stride=(1, 1))
(b3) BatchMorm2d eps=1e-05, momentum=0.1, affine=True,
track_ Frack_running_stats-te
(r3): : ReLU()
(f1): Platten(start_ddim:, end and_dim=-1)
(classifier): (act): ReLU()
out_ featureer,,,-0,, bias=True)
BFFHQ:
(shallow branch): Sequential(
(c1): Conv2d(3, 64, kernel_size=(( 7), stride=(1, 1))
(b1): BatchNorm22d64, eps-1e-05, momentum-0.1, affine-True, track_rumiigg_statt.
(r1): ReLU(inplace-True)
(a1): MaxPoolldd(eerree_sizeee
2), stride=2, padding=0, dilation=1,
ceil_mode=Palse)
(c2) : (64v 128, ternel_size=(( 3), stride= (1, 1))
(b2): BatchNorm2d(128, eps-1e-05, momentum-0.1, affine-True,
(racs__umming_ssats=rrree)
(x2): : ReLU((iplaace=True)
(s2): MaxPool2a(temmelssseee (2, 2), stride=2, padding=0, dilation=1, ceil seil_mode=False)
ternel_size=(3 (3, 3), stride=(1, 1)) (c3): Conv22(128, 512, kernel size=
(s3) MaxpPol12((eerree_ssize
2), stride=2, padding=0, dilation=l,
ceil_mode=Palse)
(b3) : (5tchMorm22d511, eps=1e-05, momentun=0.1, affine=True, track_running.e Fpack_rumning_staate-Tree
(x3): ReLU((inpaace-True)
(c4): : Conv2d(512, 512, kernel_size-33, 3), stride- (1, 1))
(b4) : BatchNorm2dd511,, eps=1e-05, momentun=0.1, affine=True,
track_rumning track_.umiiggsstate-Tiu
(x4): : ReilU(inplace=True)
(a1): AdaptiveavgPoolla((utttuu _ize=_t,, 1))
(f1): Flatten(start__dim end dim=-1)
24

---

7.9 Limitations & Broader Impact
The primary challenge with this approach is the scalability issue when applied to multi-bias setting. As the number of bias attributes increases, the subtle variations in linear decodability across the various branches could become so refined that accurately identifying biases may fail to achieve high fidelity. Moreover, depending on the network architecture might compel the model to depend excessively on intricate hyperparameter adjustments.
The societal impacts of identifying and mitigating biases in neural networks are extensive, resulting in fairer, more equitable, and trustworthy Al systems. Some of them are as follows:
1. Bias Mitigation in AI: contributes more equitable Al systems by reducing the influence of spurious correlations.
contributes to societal fairness by reducing biased decision-making AI systems and potentially decreases the risk of discrimination in Al applications.
Societal Benefits:
3. Ethical AI Development: encourages transparency and accountability in Al research and deploy- ment.
25

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Refer to Section 1
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Refer to Section 7.9
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
26

---

Justification:
All proofs are provided in Section 7.2
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: All the experimental details required for reproducibility are covered in the Sections and 7.8
and
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
27

---

Answer: [Yes]
Justification:
Source code is provided in https//gahhbbbbookkaaamll//////n
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https guwdes/ddeeeeee Cooeeeoodeesssseeeeee //nips.com htte://alllllgiiiee////eeeuaaass ssioiolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
Refer to Sections 4.1 and 7.8
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: All the experiments conducted for 5 different random seeds and we report the mean and standard deviation.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
28

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please refer to Section 7.8
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
The proposed approach conforms with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please refer Section 7.9
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
29

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The paper uses existing public datasets and models and are properly credited wher ever required throughout the paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the batasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
30

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
31


Reviews:
[{"review_id": "wNdyaD4dFU", "reviewer": "Reviewer_2yWj", "summary": "This work makes several contributions in relation to network depth and dataset bias. It shows that bias-aligned samples lie on a lower rank manifold compared to bias-conflicting samples. This is linked with network depth by showing how deeper networks tend to prefer spurious correlations, which is demonstrated by the decodability of bias and core features from networks of different depths. Based on these insights, the authors propose DeNetDM, a debiasing approach to train deep (biased) and shallow (debiased) branches, with target debiased model corrected via knowledge distillation. Empirical results demonstrate effectiveness on both synthetic and real-world image classification datasets.", "questions": "The questions correspond to the points listed in the weaknesses section:\n\n[Q1] Why not build a shallower biased branch? Is there any advantage a deeper biased branch offers compared to a shallower one?\n\n[Q2] Does the approach need to be multi-staged? Could the debiasing occur in parallel to the training of the biased branch?", "limitations": "The limitations section should mention that is not tested on non-vision/non-classification tasks. Would the approach need any modifications for other tasks?", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "[S1] The claims in the paper are backed both theoretically and empirically. Theorem 1 predicts lower-rank manifolds for bias-aligned samples and Theorem 2 shows how deeper networks prefer lower-rank features, thereby providing a solid theoretical backing for the argument that deeper networks prefer bias-aligned samples. The decodability vs depth plots support these claims.\n\n[S2] The proposed debiasing method does not rely on bias annotations or data augmentations, which is advantageous. The efficacy is demonstrated on evaluation benchmarks with varying ratios of biased/unbiased samples, which clearly shows benefits over existing debiasing methods.", "weaknesses": "[W1] It is unclear why the biased branch needs to be deep. Clearly, both core and spurious features are available at shallower depths. Would it not be more beneficial to build a shallower bias detector instead? Other works including: [1, 2] have already explored leveraging shallower layers for bias correction, which is more efficient.\n\n[W2] DeNetDM is trained in two stages, but it is unclear why this is necessary. Is it not possible to train both biased branch and perform debiasing in parallel as done in [1]?\n\n[W3] The study does not take into account the impacts of loss functions/regularization. For instance, would the deeper networks exhibit similar proclivity to spurious features with spectral decoupling [3]?\n\n[W4] All the experiments are performed on image classification tasks, which raises questions on generalization to other tasks.\n\n[1] Shrestha, Robik, Kushal Kafle, and Christopher Kanan. \"Occamnets: Mitigating dataset bias by favoring simpler hypotheses.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.\n\n[2] Clark, Christopher, Mark Yatskar, and Luke Zettlemoyer. \"Learning to model and ignore dataset bias with mixed capacity ensembles.\" arXiv preprint arXiv:2011.03856 (2020).\n\n[3] Pezeshki, Mohammad, et al. \"Gradient starvation: A learning proclivity in neural networks.\" Advances in Neural Information Processing Systems 34 (2021): 1256-1272.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the comments and questions.\n\n**W1/Q1.** Why not build a shallower biased branch?\n\n**A.** TL;DR: Both core and spurious features are indeed available at shallower depths. However, since we do not use explicit bias annotations, we have no a priori way of telling them apart. So, we aim to segregate them automatically. We observe (in Theorem 2) that although both core and spurious features are available at shallower depths, at greater depths, only spurious features are available, while core features are not. So, deep networks provide us with a unique way to identify spurious features (by looking at the features that survive to greater depths), something that is not possible with shallow networks.\n\nLonger answer: In Theorem 2 (via Lemma 2), we show that the propagation probability $\\pi_i$ is inversely proportional to the rank of an attribute $r_i$. Specifically, for any two attributes $a_1$ and $a_2$ with ranks $r_1$ and $r_2$ respectively, if $r_2 > r_1$, then at any given depth $d$, $a_1$ is more likely to propagate through depth $d$ than $a_2$. Simply put, attributes with a lower rank would propagate to a greater depth than those with a higher rank.\n\nSince spurious attributes are typically of a much simpler nature (say, color, texture, etc.) relative to core attributes (such as shape), they are more likely to lie on a lower dimensional manifold, effectively having a lower rank. Combining this with Theorem 2, we thus have that for deep networks, spurious attributes survive to greater depths than core attributes. For this reason, deep networks give us a unique way to segregate bias and core attributes that shallow networks cannot offer since both bias and core attributes can survive in the early layers (shallow network) due to their higher propagation probabilities (stemming from higher effective ranks (Huh et al., 2023)), but only spurious / bias attributes can survive up to the later (deeper) layers.\n\n**W2/Q2.** Does the approach need to be multi-staged? \n\n**A.** It is possible to train the deep model, shallow model, and target model in a single stage. In each iteration, the deep and shallow models can be updated using a product of experts (PoE) approach, followed by updating the target model through distillation within the same iteration. We believe that this approach may be less stable than the two-stage method. This instability could arise because the segregation of the deep and shallow models as experts in bias and core attributes, respectively, would not have been established in the early epochs. Consequently, the target model may receive mixed signals during the initial iterations. While parallel training is feasible, it introduces the additional challenge of stabilizing the training dynamics of the target model. Also,   training and backpropagating through all three networks simultaneously demands significant GPU memory resources which can be reduced using multiple stages of training. Due to time constraints, we could not experiment with DeNetDM in this setting, so the single-stage approach could perform better or worse than anticipated.\n\n**W3.** Would deeper networks show similar tendencies toward spurious features with spectral decoupling?\n\n**A.** Thanks for the suggestion, which led to valuable insights. To test the hypothesis that “depth captures biases” under debiasing regularizers like spectral decoupling, we performed ERM on the CMNIST dataset with both deep and shallow networks. We found that while accuracies on bias-aligned points remained near 100%, spectral decoupling(**SD**) improved accuracies on bias-conflicting points more significantly for the deep network than the shallow network (see \"Reference to Fig. A in the PDF attached to global response\" for results and the best bias-conflicting accuracies in the table below).\n\n|        Method        |  Without SD  |  With SD  | Improvement (With SD - Without SD) |\n|:--------------------:|:------------:|:---------:|:----------------------------------:|\n| Shallow (3-layer) MLP |    43.56     |  47.44    |              +3.88                 |\n| Deep (5-layer) MLP    |    36.90     |  48.74    |             +11.84                |\n\nThis indicates that the deep network, being more prone to capturing bias, benefited more from SD in terms of debiasing and accuracy on bias-conflicting points. The shallow network, already focusing on core attributes, saw only marginal improvement. This supports our claim that deeper networks are more bias-prone than shallower ones.\n\nAdditionally, the deep network with SD slightly outperformed the shallow network. We hypothesize that SD alleviates rank bottlenecks in deep networks, improving the propagation of core attributes while leveraging the network's larger capacity. This suggests that a deeper network with SD can be both bias-free and more capable, a phenomenon warranting further theoretical exploration.\n\n**W4.** Generalization to other tasks.\n\n**A.** We evaluated the performance of DeNetDM on the CivilComments dataset [A], which involves natural language debiasing. The task requires classifying online comments as toxic or non-toxic, with labels spuriously correlated with mentions of certain demographic identities. As shown in Table 1, our approach performs comparably to state-of-the-art methods. Due to the constrained rebuttal timeline, we just applied our model out of the box, without any reasonable hyperparameter tuning. This was just to illustrate its applicability to domains beyond vision. We believe that with further hyperparameter tuning, we would be able to even surpass the marginal differences that we have with SOTA on CivilComments.\n\n| Method   | Worst Group Acc (%) |\n|:--------:|:-------------------:|\n| ERM      | 58.6 (1.7)          |\n| JTT      | 69.3 (-)            |\n| LfF      | 58.3 (0.5)          |\n| LC       | 70.3 (1.2)          |\n| DeNetDM  | 68.33 (-)           |\n\n[A] WILDS: A benchmark of in-the-wild distribution shifts, ICML 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VHDB6Y2lqB", "reviewer": "Reviewer_9dFt", "summary": "The submission proposes that deeper networks are more likely to use biased features than shallower ones. Using this idea, they develop a training algorithm to encourage reliance upon non-spurious attributes. This is done by training a deep and shallow network as a product-of-experts, then distilling the shallow network into a target network.\n\nAnalytic experiments on synthetic datasets indicate that deeper networks do tend to fixate on simpler features as training progresses. A theoretical development of the key intuitions are provided, along with experimental results suggesting that the proposed algorithm can improve over baselines.", "questions": "The submission mentions use of a “small” validation set, could they clarify the exact size, relative to training and testing (I couldn’t find it easily upon a quick look)?\n\nIt would be interesting to look at the decodability dynamics on the more complex datasets such as BFFHQ and BAR. Are there technical difficulties in showcasing these trends that I missed?\n\nWould it be possible to compare both aligned and conflicting accuracies for all methods?", "limitations": "The submission acknowledges limitations and broader impact in the Appendix.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Originality: The core idea in the submission about depth modulation is original to my knowledge. Some existing work (e.g. Gradient Starvation: A Learning Proclivity in Neural Networks) has discussed how being able to learn some target-correlating features can impede the learning of others, so it seems plausible that a network that learns a simpler feature quicker is less likely to also learn other features. \n\nQuality: There is a good set of comparisons to related methods, on some of the standard datasets in the literature. The ideas presented and the intuitions appear sound. Limitations such as anticipated difficulties in scaling beyond single-biases are acknowledged.\n\nClarity: The submission is generally intuitively clear and easy to follow.\n\nSignificance: The ML community continues to consider ways to approach bias in datasets, since specific domains that are data-scarce and unique enough that foundation models may not be available continue to motivate explorations along these lines.", "weaknesses": "One key weakness in approaches of this flavour is the reliance upon use of a “debiased’ validation set to pick crucial hyper-parameters. This presumes knowledge of the bias, and if one knows it exactly, one can likely adopt other bias-informed approaches in practice. There is some work (e.g. Systematic Generalization with Group-Invariant Predictions) that have used a differently-biased validation set to pick hyperparameters relative to the test set, which can be a slightly better alternative than use of an “oracle” validation set. \n\nIn practice, it is unclear if a model is always going to be operating in an OOD setting. In fact, it might be the case that OOD settings arise relatively infrequently in deployment. From this perspective, it might make more sense to consider both aligned and conflicting accuracies of all baselines. A competing model that takes less of a hit in-distribution might be preferable.\n\nThe learning dynamics of decodability throughout training are interesting, but based only on the simpler datasets with a more drastic difference in the “complexity” of the core and biasing attributes. It is unclear how these trends play out for more realistic data.\n\n\nMinor:\n\nIs an equation with the full loss for training the target model missing? I’m assuming the objective used for training the target network is really L_t + \\lambda*L_dist.\n\nA related baseline might be relevant for inclusion: Simple and Fast Group Robustness by Automatic Feature Reweighting, Qiu et al., ICML 2023.\n\nFor the mathematical development, it might be better to include some of the key results from Huh et al., Wang and Jacot, etc. for sake of completeness, and also make it easier to gauge the additional contributions brought in this submission. \n\nUsing \\citep to might be better to enclose citations in parentheses/brackets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's insightful suggestions and comments. Below, we provide detailed responses to each of the questions and concerns raised.\n\n**W1/Q1.** One key weakness in approaches of this flavour is the reliance upon use of a “debiased’ validation set to pick crucial hyper-parameters.\n\n**A.** The authors of [A] provide empirical evidence that group annotations in the validation set are crucial for hyperparameter tuning in bias mitigation. They show that tuning hyperparameters for worst-group validation accuracy significantly improves worst-group test performance compared to tuning for average validation accuracy. Following this approach, as well as the methodology in [B], we tune hyperparameters using a small validation set with group annotations whenever available. For the CMNIST and C-CIFAR10 datasets, we create an unbiased validation set with 250 samples to optimize hyperparameters. In the case of BFFHQ, where an unbiased validation set is not available apart from the test set, we use 40 out of 1000 samples from the test set for validation. For the BAR dataset, we do not perform hyperparameter tuning due to the lack of group annotations and instead apply the same hyperparameters used for the C-CIFAR10 dataset.\n\nWe thank the reviewer for citing a reference that employs a differently biased validation set for hyperparameter tuning. However, we believe this is a completely different scenario that warrants independent investigation and could be explored as a future work.\n\n**W2/Q3.** It might make more sense to consider both aligned and conflicting accuracies of all baselines. Would it be possible to compare both aligned and conflicting accuracies for all methods?\n\n**A.** We computed the aligned and conflicting accuracies for all baselines, as well as for our approach, on the BFFHQ dataset. Due to rebuttal time constraints, we could not compare the accuracies with the baselines for the other datasets. As shown in the table, approaches like LfF and DFA reduce accuracy on aligned points while attempting to improve accuracy on conflicting points for BFFHQ dataset. This is likely because these methods focus on enhancing performance in conflicting distribution by reweighing or augmenting minority groups. In contrast, our approach achieves strong performance on conflicting points without sacrificing in-distribution accuracy.\n\n|  Method  | Bias Aligned Acc (%) | Bias Conflicting Acc (%) |\n|:--------:|:---------------------:|:------------------------:|\n|   ERM    |       93.9 (0.2)      |        56.7 (2.7)        |\n|   JTT    |      96.56 (2.9)      |        65.3 (2.5)        |\n|   LfF    |      87.49 (4.5)      |        62.2 (1.6)        |\n|   DFA    |      79.59 (2.7)      |        63.9 (0.3)        |\n|    LC    |      97.8 (1.8)       |        70.0 (1.4)        |\n| DeNetDM  |      **98.55 (0.7)**      |        **75.7 (2.8)**        |\n\n\n**W3/Q2.** The learning dynamics of decodability throughout training are interesting, but based only on the simpler datasets with a more drastic difference in the “complexity” of the core and biasing attributes. It is unclear how these trends play out for more realistic data.\n\n**A**. Please refer to the global response.\n\n**W4.** Minor corrections.\n\n**A.** We thank the reviewer for these minor corrections. We will address them in the final version of the paper.\n\n[A] Liu et al. (2021). Just Train Twice: Improving Group Robustness without Training Group Information, ICML 2021.\n\n[B] Wang et al. (2022). On Feature Learning in the Presence of Spurious Correlations, NeurIPS 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UDpXOdjeGx", "reviewer": "Reviewer_CcWa", "summary": "This paper proposes the unsupervised debiasing strategy via modulating network layers depth. It proves that the network with deep layers exploits bias attributes more than that with shallower layers, and shows that training on such less-biased network with shallow layers exhibit debiased learning. The proposed method outperforms other baselines in several datasets including CMNIST, C-CIFAR10, BAR, and BFFHQ.", "questions": "See weaknesses.", "limitations": "See weaknesses.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- It is novel contribution that identifies how the rank of attributes, specifically regarding dataset bias, is related to network depth.\n- DeNetDM shows superior debiasing performances against existing baselines across different benchmark datasets and correlation severities.", "weaknesses": "1. Empirical analysis on networks' depth to learning of different ranks (bias-aligned / bias-conflicting) is limited to CMNIST. Additional experiments on other benchmark datasets, e.g., C-CIFAR10, BAR, and BFFHQ, are required for validity.\n2. As shown in Tables 8 and 10, different pairs of networks result in significantly different results (about 20% differences in conflicting accuracy in Table 8). How sensitive are the bias-aligned (bias-conflicting) attributes with regard to depth of deep (shallow) networks, and the following performances of DeNetDM in stage 2?\n3. Although authors focus on evaluation on biased dataset, it is unclear whether DeNetDM maintains the accuracy when applied to unbiased dataset.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable feedback and suggestions.\n\n**W1.** Empirical analysis on networks' depth to learning of different ranks (bias-aligned / bias-conflicting) is limited to CMNIST. Additional experiments on other benchmark datasets, e.g., C-CIFAR10, BAR, and BFFHQ, are required for validity.\n\n**A.** Please refer to the global response.\n\n**W2.** As shown in Tables 8 and 10, different pairs of networks result in significantly different results (about 20% differences in conflicting accuracy in Table 8). How sensitive are the bias-aligned (bias-conflicting) attributes with regard to depth of deep (shallow) networks, and the following performances of DeNetDM in stage 2?\n\n**A.** As the reviewer correctly pointed out, Table 8 in the main paper provides insights into how variations in network depth affect the performance of DeNetDM. As shown in Table 8, DeNetDM can better distinguish between bias and core attributes when there is a significant difference between the depths of shallow and deep branches. However, this also depends on the complexity of the features. For instance, in the case of C-CIFAR10, ResNet 32 may capture core features more effectively than ResNet 8 due to the architectural differences. Therefore, pairing ResNet 50 with ResNet 32 in our approach enhances the learning of core attributes in ResNet 32, as reflected in the conflicting accuracy shown in Table 10. \n\nRegarding sensitivity, if the shallow architecture can partially learn the core attributes and there is a significant difference in the depth between the deep and shallow networks, DeNetDM efficiently aids in learning the bias and core attributes within the deep and shallow models, respectively. Debiasing the target architecture in Stage 2 is more effective when the deep and shallow models accurately capture the bias and core attributes respectively in the first stage, allowing us to leverage the distilled information from these networks. Thus, it can be concluded that DeNetDM is sensitive to the depth variations between the deep and shallow branches.\n\n**W3.** It is unclear whether DeNetDM maintains the accuracy when applied to unbiased datasets.\n\n**A.** Our approach operates under the assumption that the data is biased. In such scenarios, deep networks tend to prioritize biased attributes, while shallow models focus on core attributes. When the data is unbiased, the model treats all attributes equally, allowing them to be considered core attributes for the prediction task. In a Product of Experts (PoE) setting, each expert learns a portion of these core attributes, which are then used for joint prediction using both the deep and the shallow branches. \n\nAs a result, it cannot be guaranteed that either the deep or shallow branches will perform well individually on the prediction task, since both models contribute to the joint prediction through PoE. We validate this intuition by training stage 1 of DeNetDM on an unbiased CMNIST dataset using a 5-layer MLP and a 3-layer MLP. The deep model and shallow model achieve accuracies of 75.04% and 67.95%, respectively. However, the PoE model, which learns to jointly predict from both deep and shallow experts, achieves an accuracy of 98.64%.\n\nOur approach remains effective if we consider the PoE model from stage 1 as a whole (as the core model) in stage 2 of DeNetDM, rather than focusing on individual shallow or deep models as in the current setting. However, this requires a strategy for model selection (PoE or the shallow model) based on the degree of bias in the data, which could be explored in future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "trZiS7HDlt", "reviewer": "Reviewer_Bkbf", "summary": "This paper presents a useful theoretical framework that shows that samples that exhibit spurious correlations lie on a lower rank manifold and that the depth of a network acts as an implicit regularizer for the rank of the attribute subspace. Building upon this, the paper proposes a method *DeNetDM* that creates a biased strong encoder (deep network) and a debiased weak encoder (shallow network) and then leverages this to train a final strong encoder that is debiased.", "questions": "See above in weaknesses.", "limitations": "Empirical evaluation of method: discussed in greater depth in weaknesses.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The theoretical characterization provided in this paper is extremely intuitive and useful. Confirming the intuitions that examples with spurious attributes lie on a lower-dimension manifold formally is the most important contribution of this paper in my opinion. Supplementing this with the explanation that the depth of the network acts as implicit regularizer makes the method of this paper theoretically sound. \n\n2. The confirmation of theoretical findings using synthetic experiments helps validate the theoretical claims. \n\n3. Putting the theoretical findings together into a method that does outperform prior work illustrates the effectivness of this approach.", "weaknesses": "1. Empirical Evaluation: The datasets chosen by the authors to evaluate their method are not standard in this literature. Common datasets such as Waterbirds [1] and CelebA [2] are missing. Moreover, including newer more challenging datasets such as UrbanCars [3], SpuCoAnimals [4] and SpuCoMNIST [4] could further improve this paper. \n\n2. Insufficient discussion of similarity to related work: Two related works seem very similar in method to the proposed method. 1) Overcoming simplicity bias in deep networks using a feature sieve and 2) Learning from failure: De-biasing Classifier from biased classifier. The paper would greatly benefit from discussing the similarities and differences between the proposed approach and these 2 closely related approaches. Moreover, Identifying Spurious Biases Early in Training through the Lens of Simplicity Bias is another relevant theoretical / empirical study of spurious correlations which would be useful to compare & contrast with to better understand the contributions of the authors' work. \n\n[1] https://arxiv.org/abs/1911.08731\n[2] https://arxiv.org/abs/1411.7766v3\n[3] https://arxiv.org/abs/2212.04825\n[4] https://arxiv.org/abs/2306.11957", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable feedback.\n\n**W1.** Empirical Evaluation: Including newer more challenging datasets could further improve this paper. \n\n**A.** In response to the reviewers' suggestions, we have evaluated the effectiveness of our approach on the CelebA dataset, where blonde hair is the target attribute and gender is the spurious attribute. Due to time constraints, instead of using the full CelebA dataset, we employed a subsampled version as done and described in [A], maintaining the same data splits for consistency. We employed the same architectures that were applied to the BFFHQ dataset, ensuring that the target architecture remains consistent with [A]. To ensure a fair comparison, we reference results from [A] on the same dataset split for methods such as ERM, JTT, Disentangled Feature Augmentation, and DCWP. Currently, we lack results for LC and LfF on this specific dataset version, as they used the original CelebA dataset in their study. We plan to incorporate these comparisons before finalizing the camera-ready version of our paper.\n\n| Method   | Worst Group Accuracy |\n|:--------:|:--------------------:|\n| ERM      |       47.02          |\n| DisEnt   |       65.26          |\n| JTT      |       76.80          |\n| DCWP     |       79.30          |\n| DeNetDM  |       **81.04**          |\n\nAs the results indicate, DeNetDM achieves relatively high worst-group accuracy compared to other approaches.\n\nWe also conducted experiments on two variants of SpucoMNIST with low and medium spurious difficulty magnitude respectively and compared our results to those reported in [B]. The results were averaged over three random seeds. It can be observed that DeNetDM performs on par with its supervised counterparts, such as GroupDRO and Group Balancing on SpucoMNIST dataset.\n\n| Method           | Worst Group Accuracy (Medium difficulty) | Worst Group Accuracy (Low difficulty) |\n|:----------------:|:----------------------------------------:|:-------------------------------------:|\n| ERM              |              66.0 (5.0)                  |              97.0 (0.5)               |\n| GDRO             |              90.4 (1.9)                  |              96.6 (0.4)               |\n| JTT              |             65.0 (18.4)                  |              94.8 (1.0)               |\n| Group Balancing  |              92.9 (0.1)                  |              95.8 (0.6)               |\n| DeNetDM          |             **93.91 (1.1)**                  |              96.85 (0.1)              |\n\n**W2.** Insufficient discussion of similarity to related work: Two related works seem very similar in method to the proposed method. 1) Overcoming simplicity bias in deep networks using a feature sieve and 2) Learning from failure: De-biasing Classifier from biased classifier.\n\n**A.** We appreciate the reference to the following papers: [C] \"Overcoming Simplicity Bias in Deep Networks Using a Feature Sieve,\" [D] \"Learning from Failure: De-biasing Classifiers from Biased Classifiers,\" and [E] \"Identifying Spurious Biases Early in Training through the Lens of Simplicity Bias.\" A commonality between our work and these approaches is the emphasis on examining the early training dynamics of neural networks to identify and address biases. Specifically, the authors of [C] propose that simple features are learned quickly, appear in early layers of the neural network, and tend to propagate throughout the subsequent layers. Similarly, [D] posits that biases that are easy to learn are captured during the initial phases of training. In [E], the authors demonstrate that, in the early training iterations, the influence of a spurious feature on the network output increases linearly with the level of spurious correlation. While our approach also focuses on the early training dynamics of neural networks, it uniquely characterizes the differences in these dynamics across models with **varying depths** using a Product of Experts architecture, an aspect not explored by the aforementioned approaches. Instead of concentrating on a single neural network, we investigate how the early training dynamics differ between deep and shallow models, demonstrating how core and bias attributes are learned throughout the training process for each model. As suggested by the reviewer, we will incorporate this discussion into the final version of our paper.\n\n[A] Training Debiased Subnetworks with Contrastive Weight Pruning, CVPR 2023.\n\n[B] Towards Mitigating Spurious Correlations in the Wild: A Benchmark & a more Realistic Dataset, Arxiv, Sep 2023.\n\n[C] Overcoming Simplicity Bias in Deep Networks Using a Feature Sieve, ICML 2023.\n\n[D] Learning from Failure: De-biasing Classifiers from Biased Classifiers, NeurIPS 2020.\n\n[E] Identifying Spurious Biases Early in Training through the Lens of Simplicity Bias, AISTATS 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank the reviewers for their valuable feedback and thoughtful comments. We appreciate the opportunity to address the concerns raised and provide clarifications. We offer detailed responses to each reviewer, aiming to address the issues and enhance the clarity of our work. We utilize the global repsonse to address a common concern raised by **Reviewer 9dFt**  and **Reviewer CcWa**.

Reviewer 9dFt raised a concern that the decodability experiments are limited to simple dataset, CMNIST. Reviewer CcWa also raised a concern that the empirical analysis on networks' depth to learning of different ranks (bias-aligned / bias-conflicting) is limited to CMNIST. We provide a common response to both the concerns below:

Analogous to Figure 2b in the main paper, Fig. B in the attached PDF illustrates the variation in feature decodability for corruption (bias) and object (core) in the C-CIFAR10 dataset as ERM training advances. We chose ResNet 20 as the deep network and a 3-layer CNN as the shallow network since these are the architectures used for DeNetDM. The training dynamics show a similar trend to those observed in ColoredMNIST concerning bias and core attributes. As training progresses, corruption  (bias) becomes highly decodable by both deep and shallow networks, with the deep branch slightly outperforming the shallow branch. However, the object attribute (core) is more decodable by the shallow network as training progresses, during the initial training dynamics. These observations align with the early training dynamics observed in CMNIST.
	
As explained in Section 3.2 of the main text, an unbiased dataset is crucial for training the feature decodable layer to assess attribute decodability. For synthetic datasets like CMNIST and C-CIFAR10, we can generate unbiased datasets. However, this is not the case with natural datasets. The only option is to use existing unbiased test data, but the number of test data points in BFFHQ and BAR is as low as 1,000 and 656, respectively, which we believe is insufficient to train the decodable layer effectively. Additionally, BAR lacks group annotations even for the test set. As a result, we have concerns about the accuracy of feature decodability assessments on these datasets. However, we believe that the early training dynamics observed in C-CIFAR10 provide valuable insights into how these dynamics unfold for complex features.

Meta-review: The paper proposes a novel debiasing method (DeNetDM) that uses network depth modulation to address spurious correlations in biased datasets. The core idea is that deeper networks tend to encode spurious correlations more strongly than shallower networks, which, as reviewer 2yWj pointed out, is an idea that other papers have pursued without requiring bias annotations. The authors present theoretical and empirical evidence to support their approach, demonstrating its effectiveness across various datasets.

**Strengths:**
   - **Theoretical Foundation**: The reviewers appreciated the method's theoretical underpinnings, particularly the proof that bias-aligned samples lie on a lower-rank manifold and how deeper networks are more likely to capture these biases.
   - **Empirical Validation**: The method’s performance on synthetic and real-world datasets is strong. 
   - **Novelty**: The approach is novel, although the novelty may be oversold in terms of past work focusing on similar ideas. 

**Weaknesses:**
   - **Limited Dataset Evaluation**: Multiple reviewers noted that the empirical analysis was primarily limited to simpler datasets like CMNIST, raising concerns about the method's generalization to more complex, real-world datasets. Many methods fail to generalize from the toy datasets studied here to harder ones (e.g., COCO-on-Places and Biased MNIST).
   - **Sensitivity to Architecture**: There were concerns about the sensitivity of DeNetDM to the choice of network architectures
   - **Lack of Discussion on Related Work**: The paper was critiqued for not adequately discussing related work, especially methods that also address simplicity bias in neural networks. While these papers were cited, e.g., the OccamNet paper, the significant overlap in the core motivation was not discussed. Given that the motivations behind this approach and OccamNets are similar, the paper would be strengthened by a head-to-head comparison, as the baselines are not especially comprehensive.

**Response and Discussion:**
- The authors provided additional experiments. 
- The response clarified the rationale for the two-stage training process.
- The authors acknowledged the need for a more thorough discussion of related work and committed to addressing this in the final version of the paper. Given that the authors cited those papers, it is critical that they correctly attribute their ideas. 

**Recommendation: Accept.** This paper presents a technically solid contribution to debiasing in machine learning. While there are some concerns regarding the empirical evaluation's scope and the method's sensitivity to architecture, the authors have addressed most of these issues in their rebuttal. However, the authors must address concerns regarding the appropriate attribution of ideas and include an expanded discussion of related work.
Decision: Accept (poster)