Effect of Depth Modulation
This section presents an experimental investigation to gain insights into the impact of depth modulation of neural networks in the context of debiasing. We use the Colored MNIST dataset [ 23 ] (CMNIST), where digit identity (core attribute) is spuriously correlated with color (bias attribute).

We employ the concept of feature decodability to assess the extent to which the specific features of a given dataset can be reliably decoded from the models with varying depths. Hermann et al . [ 10 ] demonstrated that the visual features can be decoded from the higher layers of untrained models. Additionally, they observed that the feature decodability from an untrained model has a significant impact in determining which features are emphasized and suppressed during the model training. Following their approach, we specifically focus on assessing the decodability of bias and core attributes from the penultimate layer of untrained models. In order to evaluate the decodability of an attribute in a dataset, we train a decoder to map the activations from the penultimate layer of a frozen, untrained model to attribute labels. The decoder comprises a single linear layer followed by a softmax activation function. The decoder is trained using an unbiased validation set associated with the dataset, where each instance is labeled according to the attribute under consideration. Subsequently, the linear decodability of the attribute, measured in accuracy, is reported on the unbiased test set. We investigate the decodability of digit and color attributes in the CMNIST dataset from MLP models with varying depths, including 3, 4, and 5 layers, and the results are depicted in Fig. 2(a) . An intriguing observation is that the decodability of both attributes decreases as the depth of the neural network increases, which is also observed in [ 10 ] . To investigate how feature decodability evolves during the early stages of Empirical Risk Minimization (ERM) training across networks with varying depths, we train 3-layer and 5-layer MLPs on the CMNIST dataset. Following the training, we evaluate the model’s linear decodability for digit and color attributes. The results are summarized in Fig. 2(b) .

As observed in Fig. 2(b) , the initial phases of training for both networks emphasize color attribute (since bias is easy to learn), resulting in a notable enhancement in the decodability of color in both models. Also, as the training progresses, the decodability of the digit is higher in a 3-layer model when compared to a 5-layer model. Hence, the disparity in decodability between color and digit attributes becomes significantly more pronounced in a 5-layer MLP in comparison to a 3-layer MLP. Drawing upon these observations, our intuition is as follows: the significant gap in the linear decodability of bias and core attributes in the deep models might incline them towards favoring bias attributes while providing shallow models the opportunity to focus on core attributes when forced to capture distinct information.

This prompts us to explore whether similar behavior can be induced in models of equal depth. In this scenario, both models, undergoing ERM training, may exhibit a similar trend, with the disparity in decodability between biased and core attributes becoming nearly identical in both models due to same depth. Consequently, when we compel each model to learn distinct information, either model may capture biased or core attributes, or they might even divide the attribute information between them. This leads to a loss of control over the bias identification process. We also present empirical evidence in Sec. 4.4 to support these claims. Therefore, using models of different depths introduces an inductive bias suitable for the bias identification process, where deep and shallow models tend to capture biased and core attributes, respectively. Based on this rationale, we introduce the DeNetDM framework.