Introduction
Deep neural networks (DNNs) have made remarkable progress over the past years in many different fields by delivering superior performance on large-scale datasets. However, while the benefits of training DNNs on large-scale datasets are undeniable, these algorithms also tend to inadvertently acquire unwanted biases [ 28 ] . Such biases can lead to systematic and unfair predictions based on certain attributes or groups, often reflecting societal prejudices. For instance, a classifier predominantly trained to recognize camels in desert landscapes could encounter difficulties when attempting to identify a camel situated on a road [ 15 ] . However, it is worth noting that not all biases are necessarily unwanted; some level of bias can be beneficial for model performance, such as the grounded assumption that cars typically operate on roads [ 4 ] . Nevertheless, identifying the unwanted biases and mitigating them is of utmost importance.

Image classifiers, in particular, are prone to capture spurious correlations within the available data, hampering their generalization and robustness. Previous efforts to address this issue have typically depended on having prior knowledge of the bias. Certain methods assume the availability of bias annotations as suggested in [ 21 , 14 , 26 , 34 ] , and may involve predefined bias types, such as texture bias mitigation approach in [ 8 ] , which limits models’ ability to address biases without prior knowledge.
However, acquiring bias labels with human resources is expensive, time-consuming, and laborious. Recent studies, including [ 23 ] and [ 16 ] , have shifted towards debiasing methods without bias labels, with approaches like [ 23 ] emphasizing bias-aligned samples and reweighting bias-conflicting samples, while others like [ 16 , 15 ] introduce augmentation strategies to diversify bias-conflicting data points.

We aim to propose a novel approach that automatically identifies and mitigates spurious correlations of image classifiers without relying on explicit data augmentation or reweighting. To achieve this objective, we propose the DeNetDM ( De biasing by Net work D epth M odulation) model, which is grounded on insights gained from our preliminary study on
feature decodability, which quantifies the extent to which specific data attributes can be accurately and reliably extracted from a given dataset or signal. In our study, we focus on the feature decodability of bias and core attributes in the neural networks of varying depths, following the approach outlined in [ 10 ] . This involves training specific linear decoders dedicated to mapping penultimate features of neural networks to bias and core attribute labels respectively, utilizing unbiased datasets. Our observations in untrained neural networks reveal that the attribute decodability tends to diminish as the neural networks become deeper. Additionally, we investigate the variation in attribute decodability when employing Empirical Risk Minimization (ERM) based training on networks of different depths. This aspect of our study is particularly enlightening, as it sheds light on how training methodologies and network architectures jointly influence the ability of a model to discern between different types of data attributes. A detailed discussion of the experimental setting and the conclusions can be found in Sec. 3.1 .

We hypothesize that when we design a task that requires a pair of deep and shallow branches to acquire separate information, the deep branch consistently prefers bias attributes while the shallow branch favors core attributes. In practice, we employ a technique derived from the Product of Experts [ 11 ] , where one expert is deeper compared to the other as illustrated in Fig. 1 . Empirical evidence and analysis demonstrate that the deep branch becomes perfectly biased and the shallow branch becomes relatively debiased by focusing solely on the core attributes by the end of the training. Since the shallow branch may lack the capacity to capture the nuances of the core attributes adequately due to less depth, we propose a strategy where we train a deep debiased model utilizing the information acquired from both deep (perfectly biased) and shallow (weak debiased) network in the previous phase. Our training paradigm efficiently facilitates the learning of core attributes from bias-conflicting data points to the debiased model of any desired architecture without the need for explicit reweighting or augmentation as done in the previous approaches.

In summary, we make the following contributions:

We conduct a comprehensive analysis of the decodability of bias and core attributes across neural networks of different depths;

Leveraging insights gained from the aforementioned experiments and observations, we introduce a novel debiasing method that involves training a pair of deep and shallow networks to obtain a desired debiased model.

We perform extensive experiments and ablation studies on a diverse set of datasets, including synthetic datasets like Colored MNIST and Corrupted CIFAR-10, as well as real-world datasets, Biased FFHQ and BAR.

Our results demonstrate the superiority of our approach over existing state-of-the-art methods, achieving an improvement in performance of approximately 5%.