Mitigating Biases in Blackbox Feature Extractors for Image Classification Tasks

Authors: abhipsa basu saswat
subhajyoti  mallick
r  venkatesh babu
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 07:54:18

Abstract: In image classification, it is common to utilize a pretrained model to extract meaningful features of the input images, and then to train a classifier on top of it to make predictions for any downstream task. Trained on enormous amounts of data, these models have been shown to contain harmful biases which can hurt their performance when adapted for a downstream classification task. Further, very often they may be blackbox, either due to scale, or because of unavailability of model weights or architecture. Thus, during a downstream task, we cannot debias such models by updating the weights of the feature encoder, as only the classifier can be finetuned. In this regard, we investigate the suitability of some existing debiasing techniques and thereby motivate the need for more focused research towards this problem setting. Furthermore, we propose a simple method consisting of a clustering-based adaptive margin loss with a blackbox feature encoder, with no knowledge of the bias attribute. Our experiments demonstrate the effectiveness of our method across multiple benchmarks.

Main paper: 
 introduction: 
 Deep learning models are to known to inherit harmful stereotypical biases with respect to the different genders, races, cultures, from the datasets they are trained on [1, 2, 3] . For example, a model trained on a gender-biased dataset with images of people having blond and non-blond hair may wrongly learn a correlation between the label 'blond hair' and the gender of the person in the image. Thus the model fails to classify images belonging to minority groups (in this case blond males and non-blond females). These biases can affect the performance of AI systems handling job recruitment, e-commerce, health care, face detection and recognition [4, 5, 6] . Models can also learn spurious correlations between irrelevant training features and the target labels, instead of focusing on the relevant ones [7, 8] . Several works focus on mitigating such biases from trained models in a variety of tasks [9, 10, 11] .In recent times, large-scale models, pretrained on enormous amounts of data, are being used by machine learning practitioners as feature encoders for numerous downstream applications, as their features are shown to be semantically rich [12, 13] . However, do these powerful pretrained features themselves exude the harmful stereotypical biases that are known to affect traditional deep learning systems? A previous work [14] sheds light on these questions (when the pretrained model can be fully finetuned on the downstream dataset) -the downstream models finetuned on top of pretrained models can inherit their biases and such biases can be mitigated simply by manipulating the finetuning data.In this paper, we investigate a more constrained yet practical problem setting. With billion-parameter models gaining impetus in today's world, it may not be feasible to finetune/retrain such models due to scarcity of resources or unavailability of model weights for privacy concerns [13, 12, 15, 16, 17, 18] . A frozen pretrained encoder is used as a feature extractor for a downstream task. An adapter is attached on top of the encoder that learns the bias in the downstream dataset. Finally, the bias is mitigated with the help of an adaptive margin loss, leading to unbiased predictions. No knowledge of the bias attribute is assumed apriori.Hence, for downstream classification tasks, simply the classifier layer on top of the pretrained features is trained on the corresponding dataset, keeping the rest of the network frozen. We ask, to what extent can such systems be debiased, given that the pretrained model weights are unavailable?For the scope of this paper, we assume that the downstream dataset consists of multiple groups (e.g. blond males, non-blond males, blond females, non-blond females) due to the presence of a certain bias attribute (e.g. gender). We find that the effect of the bias in the downstream model depends on how well the pretrained feature encodes the target attribute as compared to the bias attribute. The system remains unbiased if the pretrained features are highly aware of the target attribute (even when the bias correlation in the downstream dataset is high). However, if the pretrained features instead predominantly encode the bias attribute, the downstream system becomes biased. A simple solution to the problem is to group-balance, or reduce the bias-correlation of the finetuning dataset, similar to the suggestion of Wang et al. [14] . However, as manipulating the bias correlation of a dataset is not straightforward, we advocate for a specific strategy to debias the system in the given scenario.To simplify the bias-mitigation task, we insert a trainable adapter module between the pretrained feature extractor and the classifier. With such a setting, after evaluating existing methods, we find that a large number of these methods do not yield the expected performance gains compared to the ERM-trained model, indicating the challenges in the problem setting as well as the necessity of designing specific debiasing strategies. To aid the mitigation process, we forcefully amplify the bias in the downstream model into the adapter (following previous works [19, 20] ), and then investigate a number of simple debiasing techniques utilizing this biased adapter. Finally we propose a method involving clustering-based adaptive margin loss which first clusters the biased feature space, and accordingly sets the margin value for a given sample such that it is inversely proportional to the frequency of the sample's ground truth class in the cluster it belongs to. The problem setting is depicted in Fig. 1 . We summarize our key contributions in this paper below:• We highlight a practical yet under-explored problem setting on how harmful biases creep into a model when it uses pretrained but blackbox feature extractors to obtain features. • We explore the scenarios in which biases can propagate from the pretrained features to the downstream tasks and demonstrate the necessity and feasibility of debiasing strategies in such cases. • To debias, we first amplify the bias learnt by the model from the downstream data and propose a simple mitigation strategy by utilizing the identified biases and applying an adaptive margin loss.Extensive experiments show that the proposed method is effective across multiple benchmarks. 
related works: 
 Known biases: Mitigation of biases with the knowledge of the bias attributes, as well as their labels, is a widely explored problem [21, 22, 23, 24, 25, 26, 27, 28] . Roh et al. [29] propose a sampling algorithm by formulating a combinatorial optimization problem for unbiased sample selection. While Sagawa et al. [23] minimize worst-case training loss over the groups formed using the available bias labels and the target class labels, Zhang et al. [30] disentangle the bias and target representations and debias the network using a mutual information estimator. Some methods adopt a semi-supervised approach, where bias attributes are annotated for only a few data samples [20, 31] . Another body of work assumes knowledge of the bias attribute but not its specific labels [32, 33, 34, 35] .Unknown biases: Recent works consider the more practical scenario of unavailability of the knowledge of the bias attribute as well as its labels [36, 37, 38, 39, 40, 19, 41, 42, 43] . This area encompasses a wide variety of works, a subset of which is described here. JTT [19] debiases models using misclassifications from an ERM-trained model. Some approaches like BPA and GEORGE employ clustering to identify the biases in the dataset [44, 45] . Certain works employ the Generalized Cross-Entropy (GCE) loss [46, 47] to amplify the biases learnt by the network and then debias it. While most of these works, like LfF [20, 37, 48] , employ two branches in the network, with one branch over-learning the bias and the other debiasing it, Ahn et al. [40] utilize the GCE loss to learn the bias, and then find the per-sample gradient of the trained model to obtain a balanced dataset. DebiAN [38] identifies multiple biases by having a discoverer model which optimizes an equal opportunity violation loss. Correct-n-Contrast [49] identifies training samples having the same class labels but dissimilar bias features via Empirical Risk Minimization (ERM) training and then applies contrastive loss-similar to Contrastive Adapter [50] -to bring the target features of these same-class samples closer. Qraitem et al [51] propose a sampling method to reduce dataset biases. Kim et al. [52] use a committee of auxiliary classifiers to identify the biases in the network, assigning large weights to the identified bias-conflicting samples during the training of the main classifier. Jeon et al. [53] show how bias in a CNN network is more pronounced at the top layers, and hence, leveraging features from the lower layers can help the model to exploit less biased representations. Kirichenko et al. [54] observe that retraining only the last layer is enough for having an unbiased model. However, they require bias annotations in the 2 nd stage of training. This is avoided by LaBonte et al. [55] by constructing a reweighting dataset using model disagreements for the second stage of training.Biases in pretrained models: Utilizing pretrained feature extractors for downsteam tasks is a common trend in recent times. Such encoders may carry biases that crept in from the datasets used for training them -many recent works focus on quantifying the fairness of such pretrained models [56] . Srinivasan et al. [57] study biases in multimodal vision-language systems. Goyal et al. [58] shows how pretrained models trained with self-supervision exhibit lesser biases than those trained with supervision. Recent works show that biases of pretrained models can be reduced by manipulating the fine-tuning dataset in both Computer Vision and NLP [14, 59] . Salman et al. [60] show that bias transfer happens from pre-trained models to the downstream tasks, even when the target data is unbiased. Our problem setting on the other hand consumes features from a blackbox pretrained encoder and aims to mitigate the biases arising from that encoder. This is challenging, especially when the bias annotations are not available.3 Problem Statement and Methodology 
preliminaries and problem setting: 
 This work focuses on the task of image classification. Let X = {x 1 , x 2 , . . . x N } be a set of training images of size N , and {y 1 , y 2 , . . . , y N } be the corresponding labels, where each y i ∈ C . Each data point (x i , y i ) is associated with a hidden spurious attribute a i ∈ A, and consequently a group g i ∈ G, where G = C × A. We assume that each group g ∈ G is present in the training data. In an unbiased dataset, number of training samples in each g remains approximately equal. However, models tend to learn spurious correlations when there is an imbalance in these numbers. We refer to samples favoured most by the ERM trained models as bias-aligned, and the rest as bias-conflicting [20, 38] .The goal is to train a model to optimize a mapping function f : X → R |C| . In ERM training, we optimize the Cross-Entropy (CE) loss as defined below for a sample (x, y):EQUATIONwhere [p 1 , p 2 , . . . , p |C| ] is a one-hot vector representing y. The corresponding predicted probability vector for the same sample is given by [ p1 , p1 , . . . , p |C| ].Extracting features from popular pretrained encoders for downstream applications is a common norm, as being trained on large amounts of data equips these models to perform well on a variety of tasks. We freeze these models to avoid backpropagating into their architectures that are generally large-scale [12] and often only accessible through API calls [15, 18, 16, 17] . In this subsection, we choose the WaterBirds dataset [23] to analyse the effect of the pretrained model on the downstream performance (see Section 4 for details). We observe two different scenarios here:Scenario 1: Pretrained features are target-aware. Mehta et al. [61] find that if one chooses a proper pretrained encoder for their specific downstream task, ERM training on a linear classifier attached to the pretrained embedding is enough to obtain unbiased predictions. For instance, using a ViT-H 14 encoder pretrained on the SWAG dataset [62] followed by end-to-end ImageNet [63] finetuning can achieve state-of-the-art results on the Waterbirds dataset. We find that even if all the bias-conflicting samples are removed from the training data, the worst group accuracies remain sufficiently high for the test set (see Table 1 ). Thus, the pretrained model is highly aware of the downstream target attribute, and no debiasing is required. However, a single pretrained model is hardly a panacea for bias mitigation. For instance, the worst group accuracy for the CelebA [64] dataset's Blond Hair Classification is merely 6.71%! Scenario 2: Pretrained features are bias-aware. Contrary to Scenario 1, when we use an ImageNetpretrained ResNet-18 as the pretrained encoder, we notice that the worst group accuracies are considerably low for the original Waterbirds dataset itself (see Table 1 ). An even further drop is seen when the bias-conflicting samples are removed. This shows that this feature encoder exhibits the biases present in the downstream dataset. One simple mitigation strategy is to reduce the biascorrelation in the training set. Group-balancing the Waterbirds dataset leads to considerably uniform accuracies across all groups, irrespective of the underlying pretrained model, as shown in Table 1 . This may not always be simple -firstly, the bias attribute may not be known apriori to a practitioner, and secondly, group-balancing or reducing the bias in the downstream dataset may be expensive. Further analyses on these lines are presented in Appendix subsection A.1. Thus, for such systems, an explicit mitigation strategy is required so that the model predictions are unbiased.To mitigate biases in this problem setting, we consider an image classification model that consists of 3 primary components (see Fig. 1 ): a) A pretrained feature extractor m to extract the image features f locked = m(x) (m is blackbox, i.e. frozen) , b) an adapter consisting of a multi-layer perceptron model h comprising of a single non-linear hidden layer, projecting f locked to a new latent space defined as f = h(f locked ), c) a classifier C, attached to f to obtain the final predictions ŷ = C( f ).Performance of existing methods. A large body of work exists in the domain of bias mitigation. However, most of these works assume a fully trainable feature encoder. We pick a few representative ones like DebiaN [38] , BPA [45] , GEORGE [44] , LfF [20] , JTT [19] and Contrastive Adapter (Co-Ada) [50] (see Section 2 for details) and explore their efficacy in this setup. We choose three popular benchmarks: Waterbirds, CelebA, and ColorMNIST-0.995 (descriptions available in Sec 4) and use a frozen ResNet-18 feature encoder pretrained on the ImageNet dataset for this experiment. We have a number of key observations (see Table 2 ):• LfF performs well on Waterbirds and ColorMNIST-0.995, however, it drops by 14% on CelebA. Similar trends are seen for other methods as well. This shows that existing methods exhibit inconsistencies in this constrained problem setting. • Co-Ada, which was designed to improve the zero-shot performance of foundation models (without backpropagating into the underlying model), works consistently well across all datasets. However, Contrastive Adapter is computationally expensive (Appendix Table 17 ). The above observations serve as motivating principles behind the proposed method for this problem setting that is computationally efficient and effective across benchmark datasets.In this subsection, we present our approach. Inspired by previous works [20, 19, 38] , we first amplify the biases in the system. Following this, we discuss a few alternatives for mitigation that eventually lead upto our final method.Bias-amplified Training. We propose learning features f first using an adapter (i.e. an MLP layer) on top of the pretrained features f locked through ERM training, and then identifying the biases from these learnt features. The goal is to reduce the worst-group accuracies of the training set while maintaining the overall training performance -thereby amplifying the bias learnt by the model. As indicated in previous works [65, 23] , we find that increasing the weight decay λ reduces worst-group performance of the training set considerably. We next show the effect of changing λ on 3 popular datasets: Waterbirds, CelebA and ColorMNIST-0.995. In Fig. 2 (a), we observe that with increasing weight decay, the worst group accuracies reduce for CelebA's Blond Hair classification [64] (i.e. for Blond Males), and also dips for Non-blond Females, while the other two groups' performances remain high. This points to the amplification of gender bias, i.e., the model tends to predict Blond Hair for female images and Non-blond Hair for male images. A similar phenomenon is noticed in the case of Waterbirds [23] , as shown in Fig. 2(b ). Although the same pattern is exhibited in the case of ColorMNIST-0.995 in Fig. 2(c ), an interesting observation also emerges. For λ = 0.1, training accuracy drops off quite steeply. So, with the goal of reducing the performance on bias conflicting samples, the model may end up not learning anything meaningful at all (as also seen in Fig. 2(a) ). This hints at a tradeoff, where we fix λ to a high value while ensuring the training accuracy does not fall drastically. We select the model based on the training accuracy as higher training accuracy ensures optimal learning of the bias-aligned data points even when there is less learning of the bias-conflicting samples. To avoid all predictions from collapsing to a single class, we Before delving into the rest of our method, we discuss a caveat here. An attentive reader might enquire as to whether the above technique of bias amplification will always work. We reiterate that we consider the bias in the downstream dataset to align with that in the feature encoder. An example to the contrary has been discussed in Scenario 1 (subsection 3.2). This presents an interesting future direction where one might attempt to debias a feature encoder which does not capture the downstream dataset bias.Technique 1: Loss-weighted (LW) Cross-Entropy Loss. We use a weighted CE loss to trainD: L D LW = -L B T |C| j=1 p j log pj, where L B T is the CE loss computed from B for the sample with respect to its ground truth label. The intuition is that B being biased, it would upweight the CE loss for the bias-conflicting sample in the mitigation stage. On implementing this for the Waterbirds, CelebA and ColorMNIST-0.995, we find that even though the mitigation performance increases as compared to the ERM-trained model, the improvement is not satisfactory (see Table 3 ).Technique 2: Cluster-weighted (CW) Cross-Entropy Loss. The adapter features f in the biased model B are expected to encode the bias in the downstream dataset. Hence, we explore a cluster-based weighting scheme for bias mitigation by clustering the bias-amplified feature space f in B. We define K as the number of clusters obtained and mz c as the proportion of class c within the cluster z, i.e.,EQUATIONwhere n z c is the number of times samples of class c have occurred in cluster z. For any sample in the training data, we find the cluster it belongs to (say z), and then weight the CE loss for training D in the following way:L D LW = - |C| j=1 (1 -mz j )p j log pj .Intuitively, the loss is upweighted if the sample belongs to a minority class in its cluster, otherwise it is downweighted. From Table 3 , we find that it is a significant improvement over LW. Infact, from Table 2 , we find that it performs on par with Co-Ada [50] for Waterbirds and ColorMNIST-0.995.In Technique 2, we see that clustering-based weighting methods can be effective, where we upweight samples belonging to the minority classes in a cluster, whereas we downweight the others proportionally to increase the importance of the sparse samples. We further ask, inside each cluster, can we make the features of the individual classes more discriminative? We conjecture that in such a case, it will be easier for the debiasing model D to distinguish among the samples from the different groups in the dataset (assuming that the clusters are an approximate representation of the true groups in the dataset). To this end, margin losses are an efficient class of loss functions that can reduce intra-class distance and increase inter-class distance in the feature space. Enforcing such margin constraints on hyperspherical feature spaces has been shown to be beneficial in case of deep face recognition [6, 5, 66] , long-tailed learning [67] , few-shot learning [68] and the language-bias problem [69, 70] in Visual Question Answering (VQA). Especially, inspired by the works of long-tailed learning and VQA, we make the margins adaptive to ensure discriminative features among the frequently and infrequently occurring classes within a cluster. To achieve this, we utilize the weights used in the cluster-weighted (CW) CE loss (defined above) as the margin values. We begin by defining the following:Normalized CE loss: First, we reformulate the CE loss as a cosine loss [5, 6] , by L2-normalizing the classifier weight vectors W k ∈ C (recall that C is the classifier) for each class c k ∈ C (k = 1, 2, . . . |C|), and the trainable feature f . We define Ŵk = W k ∥W k ∥ and fnorm = s f ∥ f ∥ , where s is a scaling parameter. The bias term is set to 0 for simplicity. Let θ k be the angle between f and W k . Therefore, the logit for each class c k becomes:ŷk = Ŵk ⊤ fnorm = ∥ Ŵk ∥∥ fnorm ∥ cos θ k =s cos θ k . The features fnorm are thus distributed on a hypersphere with a radius s. This makes the normalized CE loss for a single sample as:EQUATIONAdaptive Margin Loss. Inspired by the ArcFace loss [6] used in face recognition, we define the adaptive margin loss here. The loss adds a margin penalty to the angle between the features f , and the classifier weights W k for the k th class. Since the margin is placed on the angle, it maps exactly to the "geodesic" distance on the hypersphere [6] , leading to highly discriminative features. For face recognition, it suffices to have a constant value for the margin penalty (0.5 for ArcFace). Since we want discriminative features for the frequently and infrequently occurring classes in a cluster, we ensure that the training samples assigned to a cluster z and belonging to the majority class y in that cluster are allowed a smaller margin than those belonging to the minority class. We assign the loss weights from CW as the margin values (i.e. eq. 2). Specifically, for a data sample belonging to class c with cluster id z, we denote its margin as m z c = (1 -mz c ) (see Fig. 3 for an overview of the system). Finally, we define the angular adaptive margin loss for each sample belonging to the tuple t = (c, z) by combining eq.s 3 and 2:EQUATIONGaussian Randomization of the margins: While we estimate the margins for the margin loss by clustering the features from the biased model B, the obtained clusters can capture noisy signals, leading to erronous results. Alluding to our previous example of a dataset of people with blond and non-blond hair with gender bias (Section 1), since most of the females have blond hair, all such females should be given lesser margin values, whereas blond-haired males should be given higher values. But clustering leads to noisy grouping of samples in the dataset, whereby a certain group may include less blond females and more blond males. Therefore, for those female blond samples, comparatively large margin penalty will be assigned, while the blond male samples will get a low margin penalty. Also, the margins being high for sparse classes and low for the frequent ones in a cluster, they may over-learn the former, ignoring the latter. In face recognition, Boutros et al. [71] suggest that in typical margin losses, setting constant margins can limit the generalizability and discriminative power of a model and advise the introduction of stochasticity in the margin values to boost the same. This stochasticity can help us smooth out the effect of the noises in clustering on one hand and make the model more generalized to all groups. To enforce this, we use a randomized version of mz c , called rz c , where rz c ∼ N ( mz c , σ). Recall that mz c denotes the proportion of samples belonging to class c and cluster z. N is the Gaussian distribution, and σ is the standard deviation (a hyperparameter). This impedes the model from overcorrecting with respect to the rare classes in a group, thus increasing its generalizability while also compensating for the errors in margin values due to the noisy cluster labels. Finally, we obtain the randomized margin r z c = 1 -rz c for each c ∈ C. We then replace m z c with r z c in eq. 4. Our overall method is summarized in Fig. 3 . 
experiments and results: 
 Dataset Details. We evaluate our method on multiple benchmarks. Waterbirds [23] is a dataset of birds, labeled as waterbird if the bird is a seabird, and landbird otherwise. A spurious correlation exists between these labels and the background-land or water. The dataset has 4795 training samples. The ColorMNIST dataset (CMNIST) [72] is generated from MNIST [73] , where each digit is associated with one color in its background most of the time (see Fig. 3 ). It has 50000 training and 10000 test images. We evaluate our margin loss method on two variants of CMNIST: in CMNIST-0.9, each digit is associated with one color 90% of the time and other colors only 10% of the time (moderate bias). In CMNIST-0.995, each digit is associated with a single color 99.5% of the time (severe bias). The real-world CelebA dataset [64] consists of 202, 599 face images of celebrities along with 40 attributes. We choose Blond Hair as the target attribute, as it is known to suffer from severe gender bias [28, 38, 45] . For evaluating models on CelebA and Waterbirds, we find the accuracy of each group g = (c, a) in the test set [20, 45] , and report the worst of all the groups in G and their average. For CMNIST, we report the overall bias-aligned (i.e. images of digits with their maximally associated colors) and bias-conflicting (i.e. images of digits with other colors) accuracies, along with the average of all the groups [38] . We also show the results for two more real-life datasets (BAR [20] and UTKFace [74] ) in Appendix Table 12 . Implementation Details. We evaluate our method on multiple pretrained encoders: ImageNetpretrained ResNet-18 [75] and ViT-Base [76] encoders, and CLIP ResNet-50 image encoder [12] (pretrained on other datasets). All our implementations use a 1-hidden layer of M neurons followed by a non-linearity in the adapter, and the classifier is a linear layer. For clustering, we use the KMeans algorithm. We discuss the hyperparameters and their analyses in Appendix subsection A.3. We assume the availability of a small group-balanced (but unannotated) validation set and calculate the overall accuracy over this dataset for model selection during the bias-mitigation phase.We compare our baseline against Contrastive Adapter [50] , given that it performs the best among all other methods across different benchmarks (see Table 2 ), and the ERM model. The results are presented in Table 4 , where we compare our method against the ERM method and Co-Ada. For comparison with other competing methods, see Appendix Table 8 . We also present the results for the ViT-H 14 encoder in Appendix Table 13 . For the CLIP encoder, we observe the two methods to perform similarly. The CMNIST-0.9 dataset is affected by a moderate degree of the background color bias, whereas for CMNIST-0.995 the bias strength is severe. In case of both datasets, our method outperforms Contrastive Adapter for all backbones by a large margin, especially for CMNIST-0.9 (see results for CMNIST-0.995 in Table 5 ). Compared to Contrastive Adapter, our method has another advantage: it is time-efficient, as discussed in the Appendix Table 17 . Our method is effective even when finetuning to the encoder weights is possible (see Appendix subsection A.7). In this subsection, we discuss the important components of our approach and the margin loss strategy. We first show what happens when the margin penalty is constant as the ArcFace loss [6] itself. Then, we show the role played by the Gaussian randomization of the group-based margins (read subsection 3.5 and Appendix subsection ??). Finally, we show what happens when we estimate the different bias-groups in the training data by clustering the blackbox features instead of the bias-amplified adapter layer. All evaluations are performed on Waterbirds, CelebA (Blond Hair classification) and CMNIST-0.9 with ResNet-18 as the backbone. 
ablations: 
 The results are shown in Table 6 .Constant Margin. We present the results obtained by keeping the margin value constant at 0.5 as suggested by ArcFace [6] (no Gaussian randomization is applied here). This study enables us to judge the utility of the adaptive nature of the margins. While for Waterbirds and Blond Hair, the scores are better than those of ERM, there's a sharp drop in both worst and average group accuracies compared to our method. The trends are similar in CMNIST-0.9 as well. Hence, we conclude that while margin losses improve model performance, a constant margin is not enough.Without Randomization. We remove the Gaussian randomization of the margins, but here the margins are adaptive (as per eq. 2). For Waterbirds, we see that both the worst and average group accuracies reduce compared to the final scores. Worst group accuracy drops for CelebA as well, though the average group score remains similar. In CMNIST, the bias-conflicting accuracies slightly improve at the cost of a small drop in bias-aligned performance. Thus, overall we find that randomizing the margins for each sample is helpful.Clustering from f locked . Our model identifies biases by clustering the adapter features f . However, the pretrained features f locked can be useful as well, if the pretrained model and the downstream dataset share similar biases. Hence, we cluster the pretrained features f locked instead of f to obtain the margin penalties. The rest of the training pipeline remains the same, as described in subsection 3.3. For all 3 datasets, we see a reduction in the worst and average group accuracies, however, the scores are still close to that of the proposed approach. Since clustering the pretrained features does not require an extra stage of ERM training, therefore in presence of time-constraints, f locked can be used as a proxy for f , and subsequently, relatively decent model predictions can be obtained. 
discussion and limitations: 
 Our method is specifically targeted towards cases where the bias in the downstream dataset is already encoded in the pretrained model. We believe that detecting if this assumption holds apriori is highly challenging in the absence of the bias labels. We put forward a few suggestions to identify the scenarios that fit this assumption: a) Obtain bias annotations for the small validation set. If the worst group accuracy of the validation set does not reduce substantially with increasing weight decay, it indicates that the features have stronger signals of the target class than that of the bias, making it harder to capture the bias. b) If the bias annotations of the validation set cannot be obtained due to privacy concerns, the overall validation accuracy can indicate strength of the bias. For example, the difference between the validation and training accuracy is 25% for an ERM trained method for the Waterbirds dataset on the ResNet-18 backbone. The higher this difference, the more the indication that the model is overfitting to more and more samples. Such overfitting can indicate that the model is learning the bias in the dataset, thus not generalizing on the bias-conflicting samples. While we have mentioned the situation when the pretrained model is aware of the target attribute in Section 3, another potential use case might be when the pre-training data distribution may be different from that of the downstream dataset. We prescribe finetuning the pretrained models to amplify and mitigate the biases in such a case. A more practical approach would be to choose another model, as also suggested by Zhang et al. [50] .One limitation of our method is that it currently relies on the existence of biased groups in the training set (i.e., the bias labels are expected to be categorical) as it uses clustering to identify the biases. Furthermore, since our approach (along with most competing methods) relies on amplifying the bias first and then mitigating the same, it involves a risk wherein if the mitigation module fails, the bias in the system may be exacerbated. With these limitations in mind, we hope this work initiates a much required discussion in this direction leading to more sophisticated and targeted solutions in the near future.Scalability of the proposed approach to large datasets and models. Since our method involves clustering the features, if the dataset is large-scale, one can randomly sample a small percentage of it to do the clustering. For example, on clustering only 10% randomly sampled images from the CelebA dataset, we find that the worst group accuracy is 81.11%, whereas the average group accuracy is 85.6% on the ResNet-18 backbone. The scores become 80.55% and 85.6% respectively when the clustering is performed only on 1% of the images (with full clustering, the scores are 81.61% and 86.04% respectively). On the other hand, for very large models, the requirement is to be able to load the model into a GPU memory. Our method adds negligible overhead owing to the addition of only an adapter and a classifier layer.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: All implementation details are present in Section 4 and hyperparameter details are in Appendix Section A.3.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
conclusion: 
 In this work, we explored the effect of using pretrained but frozen feature extractors on downstream applications with biased datasets and found the need for specific bias mitigation strategies in cases where the biases in the downstream dataset align with the pretrained encoders. Such mitigation is challenging as the encoder is blackbox. While we found many of the existing works to be inadequate, we proposed a simple method where we first amplified the biases present in the downstream dataset and then employed a clustering-based margin loss to mitigate the same. Our experiments showed this method to be an efficient and effective technique across multiple benchmarks. Lastly, it is our hope that this work will initiate a much-required discussion among the scientific community on encouraging similar works in this highly practical problem setting. We share further studies, detailed hyperparameter analyses and experiments of the proposed method in the appendix.Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: All experiments have been done over 3 seeds, and Tables 4 and 5 have the corresponding standard deviations.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The does not pose any such risk.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: All datasets used have been cited. Only open-source methods have been replicated and all of them have been cited.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: No new assets/datasets have been proposed in the paper.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: No crowdsourcing is involved in this work.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA]Justification: No crowdsourcing is involved in this work.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: A.1 Biases in Pretrained Features -Further AnalysisIn the main paper, we looked at two scenarios on how the absence or presence of biases in the pretrained feature encoder affects the downstream model's performance on finetuning data. However, for the analysed cases, we did not look into the specific relation between the pretraining data and the downstream data. In the appendix, we perform a controlled and detailed analysis to answer 4 questions in this regard:Q1: What happens if the pretraining data is heavily biased and so is the downstream dataset?To answer this question, we pretrain a ResNet-18 model with the CelebA dataset to make it heavily gender-biased by performing gender prediction itself as the pretraining task [14] . During the downstream step, we choose to predict the Blond Hair attribute of the CelebA dataset, which is known to be heavily correlated with gender. We class-balance the finetuning dataset to have equal number of blonds and non-blonds, so as to avoid the downstream predictions from collapsing into a single class. Finally, we also sample from the class-balanced downstream data to ensure that the total number of data points in the downstream data is approximately 1/4 th of the pretraining data.Q2: What happens if the pretraining dataset is heavily biased but the downstream dataset is less biased?Table 7 : Dependence of downstream model performance on pretraining data. When the pretraining model has a high amount of bias, the downstream data performance is heavily affected if the latter itself is biased. On the other hand, the performance is more stable if i) the pretraining data has high bias and the finetuning data has low bias, ii) the pretraining data has low bias and the finetuning data has high bias. The downstream predictions get biased even if the finetuning data has biases which are not common with those in the pretraining data, but the performance drops.Pretraining For pretraining, we use gender prediction again as mentioned in the above case, but for the downstream dataset, we group-balance the CelebA dataset in terms of Blond Hair and gender, keeping the total number of downstream samples same as described above.Q3: What happens if the pretraining data is less biased but the downstream dataset is heavily biased?For the downstream dataset, we choose the same one as mentioned in case of Q1. For the pretraining task, instead of predicting gender, we predict the 'Smiling' attribute, as we find it to be much less biased in terms of gender.Q4: What happens if the pretraining data is highly biased and so is the downstream dataset, but the two biases are unrelated?To simulate this scenario, we pretrain the ResNet-18 encoder by predicting the gender attribute again in the CelebA dataset as mentioned in the above cases, and for the downstream task, we choose a class-balanced version of Waterbirds, which has a completely different bias.Findings. Note that Q1, Q2 and Q3 are related. From our experiments, we observe the following. When both the pretrained encoder and the downstream dataset are heavily biased with a common attribute (Q1), the worst group performance on the downstream dataset is the least of all the 3 cases. On the other hand, for Q2 (i.e. when the pretrained model is heavily biased but the downstream dataset is not), the worst group performance improves considerably. This corroborates with our findings in the main paper (Sec 3) that group-balancing the downstream task can improve model performance. The best performance is seen for Q3, where the pretraining model is less biased, but the downstream dataset is still highly biased -here we find that the worst group performance is infact higher than the previous case (Q2). This shows that the lesser the bias in the pretraining data, the more unbiased downstream application is going to be.When the bias in the pretrained model and the downstream task are unrelated, we find that the model is indeed affected by the bias in the downstream data, with a drastic drop in overall performance. This indicates that if the pretraining and downstream data distributions do not match, one should choose a different feature encoder to achieve better performance on the downstream task. All the above findings are summarized in Table 7 . Note that the overall performance of the downstream task is considerably lower in comparison to using an Imagenet pretrained encoder for all above cases. This is likely because the pretraining task involves a binary prediction of an attribute that is different from the downstream target attribute -leading to overfitting of the pretrained features to the former.In the main paper, we discuss the performance of our proposed baseline and compare it with Co-Ada [50] across various benchmarks and pretrained encoders. In this section, we describe the performance of the other methods in Table 8 . We find that for the ResNet-18 encoder, among all methods, LfF [20] performs well on Waterbirds and both the versions of CMNIST, but the worst group accuracies for the same method is poor for CelebA. On the other hand, while BPA has high worst group accuracy for CelebA, its performance deteriorates for Waterbirds and both the CMNIST versions. This inconsistency is present for other methods as well, for every encoder. As mentioned in the main paper, for model selection, we use the training accuracy in the biasamplification stage, whereas for the mitigation stage, we use the overall validation accuracy. Hyperparameter tuning happens via the validation accuracy of the model checkpoint saved during the mitigation stage. The bias-amplification stage has the following hyperparameters: LR (learning rate), BS (batch size), λ (weight decay) and number of epochs. In the clustering stage, the number of clusters K is a hyperparameter. Finally, in the mitigation stage, we have two hyperparameters specific to the margin loss: the scaling parameter s (i.e. the radius of the hypersphere on which the features are projected) and the standard deviation for the Gaussian randomization, σ. We next define the range of each hyperparamter on which we evaluate our model and the other methods. For learning rate LR, we explore in the range 0.0001, 0.0005, • • • , 0.05 in step sizes of 5 and 2 respectively. Similarly, for batch size BS we evaluate on the range {64, 128, 256, 512}. We explore higher values of weight decay λ ∈ {1, 0.1, 0.05, 0.01} for the bias-amplication stage and for the mitigation stage, we search λ in the range 10 -6 , • • • , 10 -2 in step sizes of 10 and also consider λ = 0. Number of epochs for training is kept in the range {50, 100}. For the clustering stage, we choose K ∈ {2, 4, 6, 8} for Waterbirds and CelebA, and explore K = 10, 20, • • • , 60 with a step size of 10 for the CMNIST variants. In the mitigation stage, we select s ∈ {4, 8, 12, 16} and σ ∈ {0, 0.05, 0.1, 0.15, 0.2}. We show the selected hyperparameters in the bias-amplification stage in Tables 9 and 10 , and for the mitigation procedure, the chosen hyperparameters are present in Table 11 . For number of neurons M in the MLP layer, we fix the value to be 128. For CelebA experiments, we use SGD as the optimizer, whereas for Waterbirds, and both the variants of CMNIST, we use Adam.Finally, we show the changes in scores by varying the number of clusters K, the standard deviation σ for the Gaussian randomization of margins and the hypersphere radius s for CelebA and Waterbirds (with ResNet-18 as the feature encoder) in Fig. 4 . We find that performance remains consistent for K ≥ 4 for both datasets. Similarly, similar scores are seen with σ > 0.1. The best values for s are found to be 8 and 12. We also vary the weight decay λ in the bias amplification stage and show its effect on the final model performance. While for CelebA, the effect of changing λ is minimal, Waterbirds can be seen to be more sensitive towards higher values.Here we describe two real-world datasets: UTKFace [74] and BAR [20] and compare our model against the ERM model and the Contrastive Adapter [50] . In the main paper, we evaluate on Waterbirds, CelebA (for Blond Hair classification) and CMNIST-0.9, CMNIST-0.995. Here, we evaluate on UTKFace [74] , which is a dataset of faces, with gender, race and age annotations. Specifically, we predict race, with gender as the bias attribute. The dataset is stored in 3 parts with 10137, 10719 and 3252 images respectively; we use the first part as the training set, second part as test, and third part as the validation set. Like all other experiments, we manipulate the validation set to contain group-balanced data. There are 5 races -White, Black, Asian, Indian and Other, and two perceived genders -females and males. Out of the 10 resultant groups, we report the worst group's accuracy as well as the average group score. We observe that our method far outperforms the ERM scores (as well as the competing Contrastive Adapter) for the ResNet-18 image encoder, thus further demonstrating the effectiveness of our method. The results are summarized in Table 12 .BAR (Biased Action Recognition) [20] is an image dataset with actions as the class labels and places as the spurious attribute. Note that the test set only consists of bias-conflicting samples, hence, we report its test accuracies following previous work [38] . As with UTKFace, we find that our method outperforms both the ERM method and the Contrastive Adapter (see Table 12 ).In a previous work [61] , it is shown that spurious correlations can be mitigated simply by using a stronger pretrained encoder. Further, the authors show that the strength of mitigation depends on the underlying pretraining data. They observe that using a ViT-H 14 encoder pretrained on the SWAG dataset [62] followed by end-to-end ImageNet [63] finetuning can achieve state-of-the-art results on the Waterbirds dataset. Likewise, to test whether the same holds for other datasets too, we use this same encoder and train for CelebA and CMNIST-0.995 using the ERM method. We observe that while indeed the worst group score for Waterbirds is considerably high, for CelebA and CMNIST-0.995, the performance is still weak. This leads us to the important observation that bias mitigation strategies are indeed required for all pretrained encoders, as even though they might perform well for some biased datasets, they may not be free of all biases that can affect model predictions. We compare the ERM scores with that of our margin loss approach and Contrastive Adapter, and find that in Waterbirds, though the ERM scores outperform the mitigation methods marginally, for the other two datasets, the mitigation scores considerably improve. Upon comparing our method with Contrastive Adapter, we find that the former outperforms the latter in CMNIST-0.995 and Waterbirds. The results are shown in Table 13 .Table 13 : Performance on ViT-H 14 (pretrained on SWAG and finetuned on ImageNet). In this table we show the performance of ERM, Contrastive Adapter and our margin loss-based method for Waterbirds, CelebA and CMNIST-0.995. We observe that while the ERM scores are high for Waterbirds, the same does not hold for CelebA and CMNIST-0.995 -showing that a single feature encoder, however powerful, may not mitigate all biases. Thus we advocate strongly for bias mitigation strategies that can debias models in presence of untrainable feature encoders. In this subsection, we further discuss the importance of the adaptive margin components of our approach and the margin loss strategy. Specifically, we present the results when the randomization is applied to constant margin values. As with Table 6 in the main paper, all evaluations are performed on Waterbirds, CelebA (Blond Hair classification) and CMNIST-0.9 with ResNet-18 as the backbone.The results are shown in Table 14 .Observations. Here we show the effects of randomizing a constant margin value to further understand the role of the adaptive margins. We perform this experiment on 3 constant margins -0.0, 1.0 and 0.5.We observe that for all three datasets, both worst and average accuracies decrease drastically. This shows the effectiveness of the adaptive margins, i.e., randomizing the margins do not help unless their underlying values represent the frequency of a training sample's class label in its assigned cluster.We next show further ablations of our method. As described in the main paper, in the biasamplification stage, we set the weight decay λ to a high value (weight decay is the weight used for L2 regularization) to ensure amplification of the bias in the model. Instead of the L2 loss, we investigate what happens when we instead use L1 and L1 + L2 regularization during this stage. The performances for all these different regularizations appear similar. We also show what happens when we do not use any of these regularizations -the scores drop drastically for Waterbirds. Moreover, in many of the previous works like JTT [19] , misclassifications in the ERM stage are used to identify the biases. When we use the same in our method to calculate the margins (instead of clustering), we find that clustering works better than using misclassifications for bias-identification. Further, we find that using the Generalized Cross-Entropy loss [20] with L2 regularization can have similar effects as that of our original approach, though the latter outperforms the former for both the datasets. We also investigate what happens when we replace the ArcFace loss [6] with another margin loss called CosFace [5] . Here we only change the loss formulation, but calculate the margins in the same way as that our method as described in the main paper. We call this version of our method CM + CosFace, and find that it is indeed effective, however the original margin loss still outperforms the former on both Waterbirds and CelebA. The results are summarized in Table 15 .In DFR [54] , Kirichenko et al. show that simply retraining the last layer of a biased model can achieve considerably unbiased predictions. However, they utilize a group-balanced reweighting Table 15 : In this table we summarize the roles of the different regularization techniques as well as misclassifications as an alternate for clustering on the ResNet-18 pretrained encoder. We find that using L1 regularization can be a good (even better) proxy for weight decay. We also show what happens when we use the GCE loss [20] along with high weight decay for the bias amplification stage. Finally, we evaluate the changes in performance when we use an alternate margin loss to mitigate the bias in the network. dataset (specifically from the validation set) for this retraining. The difference of this setting with ours is that ours does not assume a feature encoder which is already trained on the biased dataset; rather we assume a blackbox feature encoder which does not interact with the downstream training data. We summarize the results in Table 16 . While we see that the performance of Waterbirds and CelebA are comparable to our method, DFR has superior performance in CMNIST-0.9. However, it is to be remembered here that DFR not only uses part of the validation data for training, but also this excess data is group-annotated. This gives additional advantage to DFR, leading to improved scores in some datasets. While the problem statement described by Zhang et al. [50] is similar to ours, there are subtle differences. E.g., Contrastive Adapter is meant to boost the group robustness of foundation models. It uses the zero-shot capabilities of the existing foundation models (as a result, utilizes their text modalities) to find positive and negative samples with respect to an anchor sample and apply a contrastive loss for bias mitigation purposes. On the other hand, our aim is more generic: to detect and mitigate biases in case the feature encoder is untrainable (only accessible through API calls) and does not rely on any other modality. That is why, when we compare our method against other methods, we freeze the feature encoder for all of them and investigate how these methods perform when no backpropagation is allowed into the frozen backbone. Moreover, since ResNet-18 and ViT-Base do not have the text encoder, we cluster the backbone features as a proxy for the zero-shot accuracies of Contrastive Adapter using the KMeans algorithm (suggested by the authors in their code submission). Since KMeans is an unsupervised algorithm, the cluster label might not always correspond to the correct class index. Simple heuristics are employed to properly label each obtained cluster. In the binary classification setting (WaterBirds and CelebA), the cluster that has a higher number of correct predictions is assigned the label 0 (since WaterBirds has a higher number of landbirds (class 0) and CelebA has a higher number of Non-blonde samples (class 0)). This heuristic fails when the number of classes exceeds 2 (e.g. ColorMNIST), and in such cases, we use consensus to handle the prediction. For each cluster, we count the number of predictions for each class (digit), and the cluster label is the class with the largest number. This associates the most probable label with each cluster in the KMeans model -i.e., we take the ground truth class (digit) of each sample, find the majority class per cluster and assign that digit as the predicted class.Table 18 : Effect of full finetuning. Upon finetuning the encoders fully, our method achieves decent performance compared to the existing methods across two pretrained encoders. The numbers of all competing methods are taken from previous works for ResNet-18 [45] and ResNet-50 [19] . Impact of number of positives and negatives.Contrastive Adapter requires the creation of a similarity matrix, which consists of positive and negative samples for every sample in the dataset. Typically, a reasonable number of positive and negative samples are necessary for the method (contrastive learning) to function properly. The caveat is that when dealing with multi-class classification problems, gathering a sufficient number of negative samples takes a toll on the system memory, and a server of 128GiB can support only about 128 positives and negatives before overflowing. In addition to this, even if a sufficiently large server is procured to be able to support a higher number of samples, the training time grows significantly, which is a further drawback on the total runtime as well.We show the differences in training times of the proposed method and Contrastive Adapter in Table 17 . It is to be noted that we perform all our experiments on a single Nvidia-RTX A5000 GPU. In a time-constrained setting, this renders Contrastive Adapter inefficient, whereas our method, even with considerable less training time, outperforms Contrastive Adapter in most cases.In this subsection, we discuss the effect of full finetuning on the proposed loss, i.e., when the loss is allowed to backpropagate into the pretrained encoder. We compare our method with LfF [20] , BPA [45] and JTT [19] . We report the results for LfF and BPA on the ResNet-18 encoder as per the results shown by Seo et al [45] , whereas those for JTT (along with LfF) are with respect to the ResNet-50 model [19] . From Table 18 , we note that for ResNet-18, our method outperforms BPA and LfF. For ResNet-50, our method surpasses that of JTT by 3% in the worst group accuracy. In comparison, for our proposed setting, our method outperforms JTT by a huge margin (30.45%)! This study shows how our method performs satisfactorily in the setting of full finetuning, in addition to decent and consistent performance on the problem setting proposed in the paper.Since our work aims to create unbiased models in a unique problem setting, we believe that it promotes a positive social impact. Pretrained models are not free of biases [14] , and our method is an attempt to generate unbiased predictions without having to finetune these pretrained encoders -we believe that this is a step towards ensuring that future deep learning models are free of the prejudices already present in the society towards marginalized groups, and are fair to all. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 


Decision: The paper introduces a computationally efficient training of an unbiased classifier using a large and frozen pre-trained model, where a new loss function is introduced: the adaptive margin loss. The paper received 4 reviews, where the average score is 6.25 (Min: 5, Max: 8), with the following strengths: 1) original extension from the debiasing literature, 2) clear contributions, 3) experimental protocol is comprehensive with many datasets, 4) extensive ablation studies, 5) paper is well written, 6) results are convincing, and 7) problem is of high impact. As for the weaknesses, the paper lists the following issues: 1) strong assumption on the alignment of the bias in the pretrained model and downstream task, 2) missing details about clustering, 3) missing uncertainty results from competing approaches, 4) missing details about design decisions, and 5) missing deeper theoretical analysis. Most issues have been successfully addressed in the rebuttal, so I’m recommending the acceptance of the paper.

Reviews: review: paper_summary: paper_summary: The papers address the debiasing problem using pretrained but frozen feature extractors on downstream applications. Then, they propose a clustering-based method that relies on bias-amplified training through cross-entropy loss. After training, they cluster the biased features and mitigate the biases using the resultant clusters. The experimental protocol considers CIFAR, Waterbirds, and CelebA as evaluation datasets. Also, several ablations exist about distinct loss functions and comparison with literature methods under two scenarios.
strength_weakness: strength_weakness: In my opinion, the main weakness lies in assuming that the bias aligns with the feature encoder. There are many cases where the practitioner does not know the model's or the dataset's biases. Also, the paper lacks instructions on how to identify scenarios that fit the previous assumption.
questions: questions: - About Table 8, Is there any explanation about ResNet-18 achieving best performances than a Vit-B on waterbirds? I Would expect that ViT achieving higher performances in this dataset.
limitations: limitations: Despite the primary focus being on debiasing, one step of the proposed method involves amplifying existing biases. It introduces a significant risk: if the following step, designed to compensate for this amplification, fails to function as intended (which is a possibility in specific scenarios), the method may ultimately exacerbate the very biases it aims to mitigate. This could potentially allow malicious actors to exploit the technique, leading to harmful outcomes.
paper_summary: paper_summary: This work explores a problem setting where one wants to train a classifier on top of a large and frozen pre-trained model, while avoiding bias and improving fairness. It proposes a computationally efficient methodology to accomplish this objective, centered around a novel loss function, the Adaptive Margin Loss.
strength_weakness: strength_weakness: My comments here are minor suggestions, as I have no major concerns.

1- Figure 2: this analysis is very interesting. It would be nice to start the weight decay as 0, not 5e-2 in the plots.

2- A question I have is, how many unbiased samples do we need for this methodology (and the alternative ones) to work well?

Table 8 shows some results that help us answer this question, as we can see how performances vary from CMNIST-0.9 to CMNIST-0.995. What would happen for CMNIST-0.999? Where is the limit where the debiasing methodologies fail?

In summary, it would be great to expand these experiments, varying the proportion of the majority group in all datasets (not only CMNIST, but also CelebA and Waterbirds), until the point where all methods fail. This experiment would show the reliance of each methodology on the number of samples in the minority group.

3- In the tables, we have standard deviations only for the proposed methodology. They should also be reported for the competing methods as well. Moreover, although authors say that “All experiments have been done over 3 seeds”, multiple tables show no standard deviation.
questions: questions: My main question is why we do not see standard deviations in most tables, if all experiments have been done over 3 seeds?
limitations: limitations: Authors have correctly stated the method's limitations.
paper_summary: paper_summary: In this paper, the authors propose a simple method with a clustering-based adaptive margin loss for debiasing blackbox pretrained models. Whereas prior works have explored settings where pretrained models are tunable, the authors instead explore a more constrained and realistic setting, where a black-box network is frozen and only a classifier is trained on the fine-tuning dataset. The proposed approach involves training an adapter to amplify biases in the frozen encoder; then, a mitigation procedure is executed using a novel cluster-based margin loss. The proposed approach appears to outperform state-of-the-art methods across three datasets.
strength_weakness: strength_weakness: 1. **Applicability**: One important weakness of this approach is that it is difficult to know when it can be used. The authors claim that their "method is specifically targeted towards cases where the bias in the downstream dataset is already encoded in the pretrained model". However, this is difficult (and in some cases, impossible) to know a priori. 
    - In their evaluations in Table 1, the authors assume that spurious attribute labels are available in order to determine whether a pretrained model encodes the same bias as the downstream dataset. However, these subgroup labels are not likely to be available in real-world settings (which the authors claim as well later in the paper), making it difficult to know when the authors' approach is useful.
    - The authors also make a note about how their adapter-based bias amplification procedure will consistently work because they assume that the bias in the downstream dataset aligns with that of the pretrained model. As stated above, this is a strong assumption that is difficult to verify a priori, especially since the authors assume that bias annotations are not available for their downstream dataset.
    - As a result, I am unsure on the real-world applicability of this approach.
2. Presentation: The organization of this manuscript is somewhat confusing, particularly in section 3.
3. Additional clarification on design decisions: Some critical design decisions are somewhat vague in the main text.
    - How was the value of $\lambda$ selected in practice for each dataset? The paper states that $\lambda$ was selected as a high value while ensuring that the training accuracy did not drop drastically. Are there specific thresholds at which the training accuracy drop was viewed as "drastic"?
    - The paper could benefit from additional details on the clustering approach. How was the number of clusters selected? In Table 11, it appears that the number of clusters K selected by the authors varies significantly across encoders and datasets. Is this a hyperparameter that needs to be set by the user? How does performance vary depending on the number of clusters selected? It seems that this is a critical design decision that can affect efficacy of the proposed approach.
questions: questions: I have listed my questions above in the “weaknesses” section.
limitations: limitations: Yes, the authors have adequately described the limitations of their work.
paper_summary: paper_summary: The paper tries to address the critical issue of biases in blackbox feature extractors used for image classification tasks. These biases can impact the performance of models when adapted for downstream tasks. The authors investigate existing debiasing techniques and propose a novel method using a clustering-based adaptive margin loss, which does not require prior knowledge of bias attributes. Their experiments demonstrate the effectiveness of this approach across multiple benchmarks, highlighting its practical applicability in scenarios where feature extractor weights are not accessible.
strength_weakness: strength_weakness: (1) The paper could benefit from a deeper theoretical analysis of why the proposed method works, which would enhance its scientific rigor.
questions: questions: (1) How does the clustering-based adaptive margin losses compare to other adaptive loss functions in terms of computational complexity?
(2) Could the proposed method be applied to other types of pre-trained models beyond image classification tasks? If so, what adjustments would be necessary?
(3) How sensitive is the performance of the proposed method to the choice of clustering algorithm and the number of clusters?
limitations: limitations: (1) A more detailed discussion on the scalability of their approach to extremely large datasets and models would be beneficial. 
(2) While the method shows effectiveness in mitigating biases, exploring its impact on model interpretability could provide valuable insights.

Comments: title: Thank you for your response
Official Comment by the Authors
Summary of the discussion phase
Thank you to the reviewers
Response to the Reviewer
Response to the reviewer
Response to authors
Please read the rebuttal to check if the authors addressed your concerns
Please read the rebuttal to check if the authors addressed your concerns
Please read the rebuttal to check if the authors addressed your concerns
Please read the rebuttal to check if the authors addressed your concerns
comment: We thank the reviewer for appreciating our rebuttal and updating the score.
We once again appreciate the thoughtful feedback provided by the reviewer. We hope our responses have adequately addressed the concerns of the reviewer.
We summarize the rebuttal phase here. Overall, all reviewers have appreciated and positively rated our work. They have recognized the importance of our problem setting and the comprehensive experiments performed. During the rebuttal phase, we have attempted to address most of the concerns raised by the reviewers:

* We have discussed in detail on our assumption on the alignment of the bias in the pretrained model and the downstream task (**pwPo**, **wKPv**), and put forward suggestions on identifying a priori if this alignment is present. Our ViT-B experiments show how the extent of alignment affects all compared methods. Therefore, mitigating bias in the downstream dataset, especially when it does not align with the pretrained encoder, is an important area for future research..
* We have shown our method's performance to remain invariant to the number of clusters required by the KMeans algorithm, and similarly, we found the scores to remain similar when we changed the underlying clustering algorithm from KMeans to GMM (**wKPv**, **oZ3K**).
* Based on the feedback of reviewer **qb1H**, we have also shown that increasing the bias in the datasets reduces performance for all methods. Our experiments demonstrate that the proposed approach outperforms competing methods even with increasing bias ratio.
* We have also shown that the problem setting and our solution is applicable to other classification tasks like text, where we find that on the CivilComments dataset, our method far outperforms the ERM performance (Reviewer **oZ3K**).

Overall, we feel that this research direction is crucial, and we hope our paper leads to more discussions in the community regarding the same.
I appreciate the authors responses, and all my concerns were addressed very well. I am increasing my score to 8.
As we are nearing the end of the discussion phase, we would like to express our gratitude to every reviewer for their feedback and reviews. We are grateful to Reviewers pwPo and wKPv for acknowledging and responding to our rebuttal. We are eagerly waiting for responses from Reviewers qb1H and oZ3K.
We thank the reviewer for the response. We will definitely revise our manuscript to include the 'bias exacerbation' issue upon acceptance.
We thank the reviewer for the response. As the reviewer pointed out in the original review, we agree that identifying if the bias in the pretrained encoder aligns with that in the downstream dataset is highly difficult. This challenge is posed due to the nature of the proposed problem setting, since no finetuning/backpropagation is allowed into the pretrained encoder. In this regard, during our experiments, we find that the features of the ViT-B backbone do not align very strongly with the bias in Waterbirds (we refer the reviewer to the general responses section for more details). As a consequence, we find that most competing methods are unable to outperform the ERM method. Our method is able to surpass the ERM scores by a significant margin, though the scores are not higher than that of the ResNet-18 backbone for the same dataset. This finding highlights the difficulty of this problem setting, and we firmly believe that answering whether it is possible to mitigate biases when they are not encoded in the pretrained model is a very vital research direction.
I thank the authors for their detailed responses, which have addressed most of my concerns. However, my concerns on the applicability of this approach still stand, since it is difficult to know a priori whether the method will be effective. I will maintain my rating.
Thank you for your response.

The authors have adequately addressed my primary concerns, and I have no further questions. I will maintain my previous rating, but conditioned to 'bias exacerbation' revision.
Dear Reviewer oZ3K, 

Can you have a look at the rebuttal and see if your concerns have been addressed? 

Best regards
Your AC.
Dear Reviewer wKPv, 

Can you have a look at the rebuttal and see if your concerns have been addressed? 

Best regards
Your AC.
Dear Reviewer qb1H, 

Can you have a look at the rebuttal and see if your concerns have been addressed? 

Best regards
Your AC.
Dear Reviewer pwPo, 

Can you have a look at the rebuttal and see if your concerns have been addressed? 

Best regards
Your AC.