Ablation Studies
We perform several ablation studies to evaluate different facets of DeNetDM. Initially, we explore the influence of network depth, a fundamental element of DeNetDM. Additionally, we scrutinize the effect of various loss components on the performance of DeNetDM. Lastly, we conduct an analysis to ascertain the sensitivity of our approach to the number of parameters. All the experiments in this section are conducted on CMNIST and C-CIFAR10 datasets where the ratio of conflicting points is set to 1%.

Effect of depth modulation: To validate our hypothesis regarding the significance of network depth in DeNetDM, we conduct an ablation by setting the same depth for both the branches which is then compared with the default DeNetDM where one branch is deeper compared to the other.
We perform the first stage of DeNetDM training for 5 different random seeds and the averaged test accuracy on bias-aligned points and bias-conflicting for individual branches are reported in Tab. 3 . Branch 1 and Branch 2 in Tab. 3 correspond to the deep and shallow branches in DeNetDM respectively. We ignore the second stage of training in this case since we majorly focus on the segregation of bias and core attributes. One interesting observation is that DeNetDM exhibits a huge standard deviation in the accuracies when the branches are of the same depth which is evident in both datasets. This phenomenon arises because DeNetDM, in such a configuration, loses its ability to clearly distinguish between the branches since the feature decodability of bias and core attributes would be almost similar in both the branches as discussed in Sec. 3.1 . As a result, DeNetDM either distributes information across multiple branches or, alternatively, it may still separate core and bias attributes, but the specific branch capturing core attributes varies with different initialization. In contrast, when the depths are unequal for both datasets, the deeper branch tends to disregard conflicting points and concentrate on the aligned points, as evident from the test accuracies provided in Tab. 3 . Additionally, the shallow branch emphasizes capturing core attributes and thus consistently enhancing the conflicting accuracy. This shows the pivotal role of depth modulation in the DeNetDM framework for effectively segregating bias and core attributes.

We also examine the early training dynamics of ResNet-8, ResNet-32, and ResNet-50, akin to Fig. 2(b) in C-CIFAR10 dataset to assess the scalability of DeNetDM to larger ResNet models. After 200 iterations, texture (bias)

decodability in all architectures neared 99%, while core attribute decodability for ResNet-8, ResNet-32, and ResNet-50 was 18.74%, 24.32%, and 12.91%, respectively. This aligns with our hypothesis that ResNet-50 would prefer texture attribute over core when paired with ResNet-8 or ResNet-32. To confirm, we tested two setups: (1) ResNet-8 and ResNet-50, and (2) ResNet-32 and ResNet-50. The results, shown in Tab. 3 , indicate high bias-aligned accuracy for ResNet-50 and high bias-conflicting accuracy for ResNet-8 and ResNet-32 respectively. Since ResNet-50 has lower core attribute decodability than ResNet-8 and ResNet-32, it favors bias attributes, while the shallow branches capture core attributes. This experimental results suggest DeNetDM’s applicability to diverse, complex and larger models / architectures.

Effect of loss components: To assess the impact of various loss components, we conduct ablation studies by selectively removing components and analyzing their effects on classification accuracy, specifically on the test set, as well as accuracy on bias-aligned and bias-conflicting points. The results of the study are presented in Tab. 4 . As observed from Tab. 4 , when considering \mathcal{L}_{\text{CE}} alone which

corresponds to the first stage of DeNetDM involving depth modulation, the model exhibits a relatively strong ability to learn the target attributes, achieving 37.42% and 83.28% accuracy on C-CIFAR10 and CMNIST, respectively. However, upon introducing the second stage of DeNetDM training with \mathcal{L}_{t} alone, the target model tends to capture a significant amount of bias information alongside core attributes compared to the robust model obtained in the first stage. This is evident from the high accuracy on aligned points, reaching 81.60% and 95.85% on C-CIFAR10 and CMNIST, respectively. When introducing \mathcal{L}_{\text{dist}} alone, the model distills knowledge from the shallow branch obtained in the first stage, resulting in the target model exhibiting a similar performance as that of the stage 1 training. However, performing the second stage of DeNetDM training using both \mathcal{L}_{t} and \mathcal{L}_{\text{dist}} prevents the model from capturing bias, thus focussing more on learning the core features. This leads to improved conflicting and overall accuracy.

Sensitivity to number of parameters: We also investigate the influence of the number of parameters of both branches on DeNetDM performance. We opt for the optimal configuration of the proposed approach on C-CIFAR10 and conducted an ablation study, employing ResNet-20 ( \operatorname{depth}(\phi_{b})=20 ) as the deep network and a 3-layer CNN ( \operatorname{depth}(\phi_{d})=3 ) as the shallow network. We explore three scenarios where \left|\phi_{b}\right|<\left|\phi_{d}\right| , \left|\phi_{b}\right|\approx\left|\phi_{d}\right| , and \left|\phi_{b}\right|>\left|\phi_{d}\right| . The first stage of DeNetDM training is then performed to analyze learning in the deep and shallow models in each of the cases, and the results are presented in Tab. 5 .

As indicated in Tab. 5 , the shallow model exhibits increased resilience to spurious correlations, while the deep model captures bias in all three cases. This suggests that DeNetDM effectively segregates bias and core attributes regardless of the number of parameters in both branches. Interestingly, a notable finding is that the shallow model exhibits better robustness against correlations when the shallow branch possesses a greater number of parameters compared to the deep model, as evident from Tab. 5 . We also perform a similar ablation on the CMNIST dataset and the results are provided in Section C of the supplementary.