Related Works
Several works, such as [ 10 , 22 , 8 ] , have highlighted neural networks’ vulnerability to spurious correlations during empirical risk minimization training, and recent years have seen the emergence of various debiasing techniques, which can be categorized as follows.

Supervision on bias: A wide variety of approaches, such as [ 21 , 14 , 26 , 34 ] , operate under the assumption that bias labels are readily accessible for mitigating the required bias. Some specific approaches in this domain assume prior knowledge of the bias type instead of explicitly using the bias annotations. For instance, approaches, such as [ 33 , 7 , 8 ] delve into methods focused on mitigating texture bias. Recent works by [ 12 , 5 ] apply the Product of Experts method to mitigate bias in natural language processing where they presume the availability of a biased expert. Nevertheless, this assumption has its limitations as obtaining bias labels can be a non-trivial and resource-intensive task, making it a less practical approach.
In contrast, our proposed DeNetDM does not necessitate pre-access to bias labels or bias types; rather, it leverages diverse network architecture depths within the Product of Experts framework to implicitly capture information related to bias and core attributes.

Utilization of pseudo bias-labels: Instead of relying on explicit bias annotations, recent approaches obtain pseudo-labels through heuristics to identify biased samples. One notable heuristic suggests that biases that are easy to learn are captured in the early stages of training, and this approach has been adopted by several studies, including [ 23 , 16 , 20 ] , and numerous others [ 15 , 31 , 17 ] . Nam et al . [ 23 ] employ generalized cross-entropy loss to identify and subsequently reweight bias-conflicting points. On the other hand, Lee et al . [ 16 ] augment the features of bias-conflicting points to achieve a debiased model, whereas Liu et al . [ 20 ] introduce logit correction and group mixup techniques to enhance the diversity of bias-conflicting samples. Another category of approaches which includes [ 30 ] and [ 27 ] acquire pseudo-bias labels through clustering in the feature space of biased networks. Our approach does not explicitly need to acquire pseudo-bias labels in advance, as they are implicitly employed by the training of DeNetDM to facilitate the learning of both biased and debiased models.

Dependence on network architectures: Diffenderfer et al . [ 6 ] employ lottery-ticket-style pruning algorithms for obtaining compressed robust architectures. Similarly, other approaches, such as [ 24 , 35 ] , introduce pruning algorithms to extract robust subnetworks. Meanwhile, Shrestha et al . [ 29 ] applies Occam’s razor principle to optimize network depth and visual regions, enhancing overall robustness. Our method falls within this category but differs in that we do not specifically target the discovery of robust subnetworks. Instead, we leverage the training dynamics of architectures with varying depths to enhance debiasing.

Ensemble Knowledge Distillation: Our work shares conceptual similarities with the domain of ensemble knowledge distillation, as explored by Asif et al . [ 3 ] and Allen-Zhu et al . [ 2 ] . They focus on distilling knowledge from an ensemble of teacher models with different parameter sizes to enhance the performance of a student model. Similarly, our approach involves utilizing models with varying depths to achieve an unbiased model. Different from the existing works, we solely distill knowledge from the shallow model, while transferring knowledge orthogonal to the biased deep model via the Product-of-Experts training paradigm.