A Unified Debiasing Approach for Vision-Language Models across Modalities and Tasks
Authors: Hoin Jung
Taeuk Jang
Xiaoqian Wang
Venue: NeurIPS 2024 spotlight (2024)
Keywords: Vision-Language Model
Fairness
Debias
Primary area: fairness

Abstract: Recent advancements in Vision-Language Models (VLMs) have enabled complex multimodal tasks by processing text and image data simultaneously, significantly enhancing the field of artificial intelligence. However, these models often exhibit biases that can skew outputs towards societal stereotypes, thus necessitating debiasing strategies. Existing debiasing methods focus narrowly on specific modalities or tasks, and require extensive retraining. To address these limitations, this paper introduces Selective Feature Imputation for Debiasing (SFID), a novel methodology that integrates feature pruning and low confidence imputation (LCI) to effectively reduce biases in VLMs. SFID is versatile, maintaining the semantic integrity of outputs and costly effective by eliminating the need for retraining. Our experimental results demonstrate SFID's effectiveness across various VLMs tasks including zero-shot classification, text-to-image retrieval, image captioning, and text-to-image generation, by significantly reducing gender biases without compromising performance. This approach not only enhances the fairness of VLMs applications but also preserves their efficiency and utility across diverse scenarios.

A Unified Debiasing Approach for Vision-Language Models across Modalities and Tasks
Hoin Jung, Taeuk Jang, Xiaoqian Wang"
Elmore Family School of Electrical and Computer Engineering
Purdue University
West Lafayette, IN 47907
(jung414, jang141, joynang)
Abstract
Recent advancements in Vision-Language Models (VLMs) have enabled complex multimodal tasks by processing text and image data simultaneously, significantly enhancing the field of artificial intelligence. However, these models often exhibit biases that can skew outputs towards societal stereotypes, thus necessitating debias- ing strategies. Existing debiasing methods focus narrowly on specific modalities or tasks, and require extensive retraining. To address these limitations, this paper in- troduces Selective Feature Imputation for Debiasing (SFID). a novel methodology that integrates feature pruning and low confidence imputation (LCI) to effectively reduce biases VLMs. SFID versatile, maintaining the semantic integrity of outputs and costly effective by eliminating the need for retraining. Our ex- perimental results demonstrate SFID's effectiveness across various VLMs tasks including zero-shot classification, text-to-image retrieval, image captioning, and text-to-image generation, by significantly reducing gender biases without com- promising performance. This approach not only enhances the faimess of VLMs applications but also preserves their efficiency and utility across diverse scenarios. The code is available on GitHub
Introduction
Vision-Language Models (VLMs) have revolutionized the way we handle multimodal tasks by enabling simultancous processing of text and image data. Models such as CLIP [1] and XVLM 43 serve as foundation models 41 for various downstream tasks demonstrating the remarkable versatility of these systems such as shot classification and text-to-image retrieval. Additionally, models like BLIP (25] and CoDi [37 enhance this spectrum by facilitating tasks such as image captioning and text-to-ine rext-to-image generation. These examples highlight the diverse capabilities of VLMs adapting to various specific multimodal interactions.
Despite their remarkable capabilities, VLMMs have a critical bias issues, often skewing the model outputs in ways that reflect societal stereotypes [21] such gender [46] racial (45) biases in assigning professions or describing scenarios. For example, studies have identified biases in multi-class zero-shot classification [17), where models might disproportionately associate certain
professions with specific genders. Similarly, biases in text-to-image retrieval [18] 39 can lead to the preferential retrieval of images that reinforce stereotypical narratives. The implications of these biases extend to image captioning [55] 20 and text-to-iman image generation [10], where the descriptive and generative capacities of VLMs may perpetuate and even amplify existing societal prejudices.
These issues highlight critical need for effective debiasing strategies that can ensure the fairness of
VL.Ms applications.
"Corresponding Author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

"A photo of man who works as nurse."
CLIP-CAP
woman wetsuit surfing wave.
man riding skis down
snow covered slope.
CoDI
CoDi
CLIP-CAP + SFID
CLIP-CAP + SFID
person on surfboard in the water.
skier going down a snowy hill,
(a) Debiasing VLM in image captioning task
(b) Debiasing VLM in image generation task
Figure 1: Bias in VLMs' various downstream tasks. VLMs tend prefer certain gender for a subject in image or text, while SFID mitigates the bias issue in VLMs.
Given the bias issues in the realm of VLMs, debiasing approaches have been proposed. However, the existing debiasing frameworks for VLMs tend to focus on specific modalities and tasks. For example. (20] and [23] are tailored to particular tasks such as image captioning or text-to-image generation, respectively. On the other hand, while some debiasing strategies are originally designed for mitigating bias in the image or text encoder's output for retrieval task, like DeAR 34 and CLIP- clip (88, they offer potential for broader application across various modalities and tasks. For instance, DeAR can extend the use of adversarial training neutralize biases the decoder's intermediate representations by aligning them near the decision boundary for sensitive attribute by deceiving the attribute classifier. Yet, this method's effectiveness hampered by intrinsic complexities and sensitivities of adversarial training techniques, which are highly dependent precise hyperparameter tuning. Similarly, CLIP-clip could potentially adapt to various tasks by pruming features the frozen representation that exhibit high mutual information with sensitive attributes. However, this approach does not consider dependencies between feature columns. Furthermore, the process zeroing out pruned features might inadvertently distort the semantic meaning of embeddings, leading to loss in the quality and relevance of the output. Moreover, Prompt-Debias [1], which relies on pre-defined text prompts debias the text encoder, is limited to text encoders and cannot be applied to text decoders or other components.
To overcome these limitations, we propose Selective Feature Imputation for Debiasing (SFID) incor- porating feature pruning and low confidence imputation (L.CT). Unlike existing methods, this approach can be seamlessly integrated into various parts of VLMs, whether applied encoders, decoders, or both, enhancing its utility diverse contexts, while effecitvely maintain the dimensionalty and semantical meaning of debiased representation. In details, by utilizing feature selection techniques such as RandomForest [ to identify gender-specific (or race) biases within the frozen representation, SFID prunes and subsequently replace bias-causing features with bias-free representation obtained from ambiguous samples identified by RandomForest thereby maintaining the semantic content while effectively reducing bias.
Furthermore, SFID eliminates the need for costly retraining of pre-trained VLMMs and does not require paired text-image datasets. It simply utilizes datasets with sensitive attributes in individual images or texts for debiasing. For instance, to debias gender bias in VLMs, datasets like FairFace [22] for image inputs and Bias-in-Bios [12) for text inputs are employed. This approach not only maintains the utility and efficiency of VLMs but also broadens their applicability across varied scenarios, thereby
enhancing fairness without compromising performance.
The experimental results demonstrate the efficacy of the proposed method in mitigating bias across various downstream tasks, such as zero-shot classification, text-to-image retrieval, image captioning, and text-to-image generation. Our method consistently outperforms other debiasing methods without compromising the performance of the downstream tasks. Consequently, SFID enhances the fairness and utility of VLMS across a wide range of multimodal tasks, setting new standard for debiasing strategies in this field.

---

2 Related Work
2.1 Bias Evaluation in VLMs
Recent research has focused significantly on identifying and evaluating bias in VLMs. Agarwal et al. [2] and Wolfe al. [40] observed that CLIP [3] embeddings exhibit substantial racial and gender biases. Additionally, Chuang et al. [M highlighted that text prompts can capture spurious features in visual representations, exacerbating the bias. Kim et al. [24 further investigated how the association between sensitive attributes and specific keywords contributes to bias issues in downstream tasks. These biases lead unfair outcomes in zero-shot binary classification and text-to-image retrieval,
Moreover, Slyman et al. [35 extend the bias in zero-shot classification in multi-class setting using the FACET dataset [17 and its evaluation metric. Seth et al. M suggested new metric for fairness in text-to-image retrieval, considering the gender distribution in the query set. Beyond zero-shot classification and text-to-image retrieval, other downstream tasks also reveal biases. For instance, Hirota et al. [20] and Zhao et al. [45] investigated bias in image captioning, where specific genders or races are disproportionately represented leading to generate biased caption. Similarly, Cho et al. [TO] raised concerns about bias in text-to-image generation and suggested evaluation methods. Finally, Sathe et al. (32) emphasized the need for unified evaluation approach for various downstream tasks to address these pervasive biases comprehensively.
as noted by 13 18
2.2 Debiasing VLMs
As bias issues have arisen, many debiasing methods have been proposed. Zhang and Ré 44 trained adapter frozen representations to debias spurious correlations in zero-shot binary classification. Additionally, Chuang et al. [1]. Adila et al. [Α] and Berg et al. [6] suggested methods to manipulate input prompts text tokens for debiasing spurious correlations in VLMs' encoders, covering downstream tasks such as zero-shot binary classification and text-to-image retrieval. For image captioning, Hirota et al. [20] proposed fine-tuning method for both the encoder and decoder to mitigate biases. For text-to-image generation, Kim et al. (23) recommended a de-stereotyping prompt design to address biases. Some methods, such as DeAR 34 and CLIP-clip [88) manipulate frozen representations without training the entire model and can be generalized across various tasks, as discussed in Section 4 Dehdashtian et al. [13] also aimed to debias frozen representations from both the image and text encoders, though this approach requires class labels and text-image pair datasets, which are challenging to define and obtain across various downstream tasks. To address these limitations, we propose the first unified debiasing method for VLMs across various downstream tasks, which demonstrates outstanding performance compared the extensions of other debiasing methods, such as DeAR and CLIP-clip.
3 Bias Analysis in VLMs
3.1 Zero-shot Classification
Multi-class zero-shot classification leverages the capability of VLMs that train image and text encoders jointly. Specifically, the predicted class is determined by providing text prompts about classes to the text encoder and selecting the class with the highest cosine similarity to the encoded image. For instance, the text prompt "a photo of a/an [CLASS NAMEJ" is used for the zero-shot classification. Bias issues arise from the difference in accuracy between genders for a given class. For example, in the class Carpenter, the accuracy for male carpenter images and female carpenter images might differ, as VLMs are likely to associate the concept of "male" more strongly with "carpenter." Therefore, bias is defined by the average demographic disparity, which is determined by the gender disparity in recall for each class following 17.
p(Ý = k|a P(ý k|a = 0)/
ADP
KC
KEA
where Y is the predicted class, and & E K. represents each class in the multi-class classification, and E (0, 1} is the sensitive attribute. The overall accuracy is used as an evaluation metric for the performance of VLMS. Lower ADP indicates fair classification across the sensitive attribute, while higher overall accuracy denotes higher classification ability. As a bascline for zero-shot classification,
3

---

CLIP [3 with ResNet-50 19 and ViT-B/32 15), and XVLM 443 are adopted. We utilize the dataset, which includes 49,551 images across 52 classes with gender sensitive attribute.
FACET 17
3.2 Text-to-lmage Retrieval
Text-to-image retrieval leverages the matching ability of image and text encoders, For given ground truth caption, images in the query set are retrieved by sorting them according to the cosine similarity between the image embeddings and the text embedding of the caption. Bias retrieval arises when the gender distribution in the retrieved set is skewed to a certain gender. For example, for gender-neutral caption such as "a person
suit hurrying across the street." VLMS retrieve male images more frequently than females by associating the concept of 'suit' and 'male'.
The evaluation metric for fairness in text-to-image retrieval is defined as Skew@ M, as suggested in [34). Let Pa = NN// N for a E 10, where N₂ is the number of images for each gender in the original dataset and N the total number of images. For M retrieved images, calculate P. Ma /M, where Ma is the number of images for each sensitive attribute in the retrieved set. The metric is then defined as Skew, log (Pa/P. Pa) for each sensitive attribute a indicating if particular gender is retrieved more frequently. The final evaluation metric, Skew, is defined by averaging the maximum Skew values over the set of text prompts T,
1 Σ LET
Skew
max Skew
[
The performance metric for text-to-image retrieval is defined as Recall@K. K, which measures the probability of the ground truth image being among the top K, where K E (1,5, 103. Thus, lower Skew and higher Recall are desired. We use the same baselines: CLIP ResNet-50, CLIP ViT-B/32, and XVLM. the experiments, we utilize the Flickr30K 422 dataset which includes ground truth captions and gender attributes. After modifying the ground truth captions to be gender-neutral, we select 1,000 images for testing (38) and retrieve the top 100 images for each caption for bias measure.
3.3 Image Captioning
Image captioning aims to generate a caption given an image. Fairness issues arise from differences between the gender mentioned in the caption and the ground truth gender of the subjects in the image [20]. as VLMs may prefer certain genders for particular subjects. For example, as shown in Figure| image captioning models tend to associate contexts and genders, such as (athlete, male) and (long hair, female).
For the evaluation metric, we first detect the gender in the generated caption by its pronoun. Specifi- cally, for an image the query set, we measure the gender mismatch rate for a k-th image,
4 = {10
if (original gender) * (detected gender) if (original gender) = (detected gender) or (neutral detected gender)
where the misclassification rate for each gender is defined as MR = TM ΣLEM 4s MRp
[F ExeF 4r and MRo b ΣkED 4. with M, F, and O indicating male, female, and overall, respectively. Although the overall misclassification rate is used in (20), it cannot perfectly reflect fairness. For example, in two different situations where (M RM. MRF, MRo) are (3.0%. 3.0%, 3.0%) and (0.0%, 6.0%, 3.0%), respectively, the overall misclassification rates are the same, but the rates for each gender are not fair. To address this, derive a Composite Misclassification Rate, defined as MR / MR + (MRF MR which can be minimized when both the overall misclassification rate and the disparity in misclassification rates between genders are low.
On the other hand, the caption's quality is measure by METEOR 44 and SPICE B. METEOR measures the balance between precision and recall of n-grams in generated captions, incorporating synonyms, while SPICE focuses on the semantic content of captions by comparing sets of proposi- tional semantic tuples extracted from candidate and reference captions. (See Appendix B for details.) Considering fair image captioning, the evaluation metric for caption quality should account for both

---

Important Features
Unimportant Features
Gender-related Features
Race-reiated Features
Original
Top 10
Top 20
Top 30
Top 500
Top 510
Figure 2: GradCAM visualization for feature indices sorted by their importance in predicting an attribute (e.g., gender). Highly important features (left) focus on attribute-related characteristics such face the image, while the least important features (right) are associated with the background. SFID not only identifies the crucial biased features but also imputes these biased features with ambiguous values derived from low-confidence samples.
the ground truth caption and neutral caption.
MaxMETEOR max (METEOR(Turee ) MaxSPICE = max (SPICE(T Tcaptiion SPICE(T neutral; T caption))
METEOR (T neutral Teaption))
In image captioning, a lower M Rc score and higher METEOR and SPICE scores are preferable.
In the experiment, due ambiguity in images containing
than person, such images
excluded from the evaluation. We adopt two image captioning models, CLIP-cap [27] and BLIP The MS-COCO dataset [9] used the query dataset, which consists of 10,780 images with ground truth captions for each image.
(55
3.4 Text-to-Image Generation
VLMs can be used for text-to-image generation, creating image given text prompt. However, bias issues in text-to-image generation arise in two ways: generating an image with the wrong gender despite of specific gender is given in the prompt, generating biased gendered image for neutral text prompt. We utilize the input prompt from [10] to generate images for captions: "a photo of G who works a/an P." where gender set {a man, a woman, a person and profession P = aacconttnt,, engineer, ... ). We use 83 professions as in 10]
For the evaluation, first detect the gender of the generated image by utilizing BLIP-2 26 by asking "Does the person look like a male or female?". For gender-specifis prompts, a fair text-to- image generation model should produce exact gender images irrespective of the profession. Thus, measure the mismatch rate: [D] ΣLED It. where D a set of prompts for either the male or female subgroup. We adapt the composite misclassification rate, . Rc, as the same as the image captioning to verify a balanced outcome between mismatch rates for each gender. For neutral prompts, fair model should produce an equal number of images of each gender across all professions. The bias metric thus measures skewed distribution the model:
max Np.m, Np.1)
Skew
(I)
P
PEP
where Np. and Np.f are the numbers of detected genders for each profession, and C = 10 is the number of generation for each prompt. text-to-image generation, lower values of both Rc and Skew indicate faimesss. We use SDXL (28 and CoDi [37] as baselines for text-to-image generation.

---

Ohard
RandomFurest
Gender Label
Encoder
Training Set
Train Embedding
Po Encoder
Encoder
Law Gorfitecs right.
Validation Set
Validation Emhedding
j2 Feature Importance
J1
Real world Test Query
Imputing Values P2 1at.P2.....
(a) Feature Selection Imputing Value Extraction
(b) Debiasing Downstream Tasks
Figure 3: Selective Feature Imputation for Debiasing (SFID) utilizes RandomForest extract feature importance (jk,) identifying bias-related features, and low-confidence samples in the validation set which indicate ambiguous representations. During the inference stage, the extracted feature indices and imputing values (f4)) from low-confidence samples are imputed into the embedding used the downstream task.
4 Proposed Method
4.1 Selective Feature Imputation for Debiasing
Let Xp be a debiasing dataset with the sensitive attribute label yn. Let Xę be a query dataset that the user is interested in and wants to evaluate in debiased downstream tasks. For a frozen component in VLMS g. whether it is encoder decoder, processes image text, obtain the frozen representations DD (XD) and ZQ g( g(Xq). respectively.
For the debiasing representation Zp. Selective Feature Imputation for Debiasing (SFID) uses RandomForest [8] to predict the sensitive attribute y/D. Given the interpretability of RandomForest, it is capable of providing feature importance for predictions, allowing to identify which features in the frozen representation are relevant to the sensitive attribute. Additionally, RandomForest is known for not requiring hyperparameter tuning [29] and for its computational efficiency ( making it easily implementable choice. As the objective of SFID is to lead VLMs' components to produce a fair outcome, free of bias regarding sensitive attribute, SFID prunes the important features that show higher relevance to the sensitive attribute. This procedure is beneficial as it considers the dependency between features, whereas methods like CLIP-clip (38) extract feature importance by measuring the mutual information between each feature and the sensitive attribute, assuming each feature is independent.
However, simply dropping features cannot maintain the dimensionality of the representation, which crucial for using the embedding for generation tasks. For example, in CLIP-CAV CPPC (27). the decoder (GPT-2 (30)) takes the image representation from the CLIP ViT-B/32 31 image encoder as input, and the input dimension of GPT-2 is fixed. Therefore, must maintain the dimension of the representation after pruning to utilize the pre-trained decoder but approaches like filling with zero- values Gaussian noise may mislead the semantic meaning of the representation, described in Figure 4 and ablation study in Section 5.3
To address this. SFID replaces important features with ambiguous features through Low Confidence Imputation (LCI). LCI defined as the average of the features in low-confidence samples from the validation set as determined by RandomForest. RandomForest is known for its robustness against overfittinng and can provide reliable confidence levels, identifying which samples are more ambiguous with low confidence. These owwccofiieence samples likely to be 'hard to identify the sensitive attribute,' implying they are free of biased features.
We visualize how the important features are correlated to social biases by showing GradCAM 33 visualizations, as presented in Figure 2 For example, the more important features highlight human faces, while the least important features correlated with the image background. SFID imputes

---

Algorithm Selective Feature Imputation for Debiasing (SFID)
Input: Frozen representation of debiasing training and validation dataset, (Zp, yD) and (Z)).
Representation of query set in the downstream task, ZQ.
Output: Debiased representation in downstream task, Z22
f + RandomForest( D. yD)
// Run RandomForest for attribute prediction.
[mp f Importance( f, j). // Obtaai feature importance for each dimension of embedding. // Top k feature indices based on their importance ranking. J} // Identify low confidence samples in the validation set. t 1 Eiec Fij.j E S // Calculate the average value j4j from the low confidence samples. 2Q. t Ms for all j E S // Impute the values of j-th feature in the query embedding with py-
S f [j rank(Imp(i)) < k}
C {i: Confidence(
[c
the representations at important indices with the values from low-confidence samples, making the face-related features ambiguous. Despite this imputation, the replaced values remain within the distribution of the original samples.
Consequently, SFID not only considers the dependency of the features but also effectively imputes the attribute-related related features with ambiguous features. The overall algorithm is described in Algorithm ! and Figure 3 The number of pruned feature k is set as 50 by choosing an elbow point of the feature importance described Appendix A.I Moreover, the impact of hyperparameter for thresholding low sonfieence samples is studied in Section 5.3 The extension strategy for applying SFID to the decoder explained in Appendix A.2
Top-2 features in CLIP ViT-B/32 Image Encoder
Most Important Feature X2
0.05
-0.05
-0.10
Gender 0
-0.20
Gender
Zero-centered Gaussian Distribution
-0.25
Centroid of lower confience samples Zero centeroid (0,0)
-0.30
-0.15
-0.10
-0.05
0.00
0.05
0.15
0.20
0.25
0.30
Most Important Feature X1
Figure 4: Comparison of zero-value imputation, zero-centered Gaussian noise, and low confidence samples. Different colors of points indicate different sensitive attributes. Gray points represent zero-centered Gaussian noise, which is -atto--distribution from the original embedding. SFID utilizes the centroid of low confidence samples (red x), which remain in-distribution of the original samples.
4.2 High-Confidence Imputation in Text-to-Image Generation
In SFID, we utilize low-confidence samples to obtain imputation values that mute bias-related information within the embedding. However, in text-to-image generation, users may sometimes wish to specify the gender the generated image through gender-specific prompts, such "a photo of man who works as nurse." In such cases, rather than suppressing bias-related features, impute
7

---

Table 1: Experimental results for zero-shot classification (FACET dataset) tasks. Bold indicates the best result for each baseline, while underline denotes the second result.
Zero-shot Multi-class Classification
Model
Accuracy
A DP
Baseline
51.87
11.08:10,90
DeAR
52.08=+0.63
10.04+0.80
CLIP
CLIP-clip
50.73:0.58
10.09
(ResNet50)
Prompt-Debin
52.58:#0.56
10.37=0.91
SFID (Ours)
50.93++0.57
9.63
Baseline
52.17:0.58
11.60:0.93
DeAR
50.09=+0.45
10.7
CLIP
CLIP-clip
51.56:+0.53
10.80=0.80
(VIT-B/32)
Prompt-Debias
51.96=t0.53
10.56
SFID (Ours)
52.14:10.53
10.15
Baseline
55.74:10.48
11.72++0.72
DeAR
56.30++0.52
11.26-+0.84
XVLM
CLIP-clip
5..5
9.98++0.88
Prompt-Debias
56.37-+0.48
10.35+00.78
SFID (Ours)
53.69:10.59
9.91
features from samples that are classified with high confidence as belonging to the specified gender, using RandomForest classifier, to retain the desired gender attributes. For text-to-image generation, we report results using both low-confidence (LC) and high-confidence (HC) imputation approaches.
4.3 Dealing with Multiple Sensitive Attribute
We extend our approaches address more complex bias scenarios in VLMs,, focusing on multiple sensitive attributes. Specifically, we conduct additional experiments that examine racial bias by considering more than two sensitive attributes, enabling capture a broader spectrum of bias patterns. Since RandomForest can accommodate multi-class classification. SFID is applicable in this context, illustrated in Figure 2 by employing multi-class classification with RandomForest.
For training the attribute classifier, we use the FairFace dataset, which includes seven racial categories: East Asian, Indian, Black, White, Middle Eastern, Latino Hispanic, and Southeast Asian. A detailed analysis and the corresponding results on multiple sensitive attributes are provided in Appendix
Experimental Result
5.1 Implementation Detail
We use the FairFace dataset [22] for image inputs and the Bias-in-Bior dataset [12 for text inputs as our debiasing datasets, denoted as X Each dataset is split into training and validation sets. A RandomForest classifier is applied to a 2D-shaped embedding representing all training samples, with low-confidence samples selected from the validation set. All hyperparameters and model settings for each baseline follow the default configurations provided their respective open-source repositories. Detailed experimental settings, along with evaluation metrics and query datasets, are described in Section 3 We also adopt DeAR [4]] CLIP-clip [8], and Prompt-Debin [ as baseline comparison methods. A detailed discussion of these methods is provided in Appendix
For a fair comparison, we conduct 10 experiments using different subsets and report the mean and standard deviation for the text-to-image retrieval task. For zero-shot classification and image captioning, we employ 1000 bootstrapping iterations to calculate the confidence intervals. For text-to- image generation, images are generated using 10 random seeds. The mismatch rates are reported as the mean and standard deviation over runs, while the Skew for the Neutral prompt reported as single value based on 10 runs. Further analysis of this evaluation metric can be found in Appendix

---

Table 2: Experimental results for text-to- cox-tto-image retrieval (Flickr30K dataset) tasks. Bold indicates the best result for each baseline, while underline denotes the second result.
Text-to-Image Retrieval
Model
R
R@5
R@ 10
Skew @ 100
Baseline
57.24-10.58
81.66:10.61
88.12:1..56
0.1883=000939
DeAR
57.02=#0.57
81.62:0.76
87.95+0.61
0.1817=0.122 1200
CLIP
CLIP-clip
56.83+0.43
80.99=0.54
87.39=0.52
0.1542=0.100
(ResNet50)
Prompt-Debias
57.47=10.57
81.81
88.23+0.51
0.2030=0.0971
SFID (Ours)
56.94+00551
80.89+0.62
87.41+0.60
0.1414=0.0955
Bascline
58.9110.51
83.0810.62
89.2120.48
0.1721
DeAR
59.46=0045
83.26+0.66
89.23+0.51
0.1387=00.012
CLIP
CLIP-clip
57.66=+0.73
81.80+0.46
87.98=0.45
0.0920=0.0932
(ViT-B/32)
Prompt-Debias
58.86=00559
82.71
89.08=0.42
0.1496=0.1097
SFID (Ours)
58.53
82.7310.556
88.90:00.56
0.074410.0616
Baseline
80.77:0.56
96.67:0.26
98.55+0.23
0.2355=0.1425
DeAR 78.82++0.57
96.03++0.39
98.17+0.22
0.2066--0.1667
XVLM
CLIP-clip
75.99+00.54
94.77+00.33
97.43+0.31
0.2205+0.1224
Prompt-Dehias
79.02+0048
96.03+00.66
98.24+0.21
0.2355+0.1658
SFID (Ours)
78.00:0.466
95.67
98.01#0.25
0.2032-0.1229
Table Experimental results for image captioning. Bold indicates the best result for each baseline, while underline denotes the second-best result.
Caption Quality
Misclassification Rate
Model
Max
Max
Hale-Femmael
Overall
Composite
METEOR
SPICE
Baseline
34.57-00.83
25.41
2.20++1.81
2.10=0.70
1.24
DeAR
33.9010.91
24.73:10.33
1.58
2.93=0.98
3.53
CLIP-CAP
CLIP-clip
32.28-1077
23.44:10.65
3.73:12.32
2.90
4.34:12.48
SFID (Ours)
32.08
23.74.10.69
2.16.12.03
2.07
3.12.1
Bascline
24.01=0.62
17.06=+0.60
1.72+1.37
1.15=0.65
2.26
DeAR
21.76=0.59
10.51
2.62+1..4
1.07=0.63
2.84+2.13
BLIP
CLIP-clip
23.74.10.54
16.96.10.54
2.29
2.59 1.15-0.65
SFID (Ours)
23.38-L0.49
16.74
1.37
0.92 0.53
1.88
Table 4: Experimental results for text-to-image generation. Bold indicates the best result for each baseline, while underline denotes the second-best result.
Mismatch Rate (Gender prompt)
Neutral prompt Skew
Model
Male-Female/
Overall
Composite
Baseline
3.87++2.23
2.32
4.42+22.57
83.25
DeAR
89.28
44.64+1.04
99.81 +2.33
99.88
CLIP-clip
3.78
2.11#1.03
4.31
82,05
SDXL
Prompt-Debias
39.72-16.83
42.53+3.85
58.4943.64
82.77
SFID (LC)
1.6940.72
0.96+0.42
1.97+0.67
81.57
SFID (HC)
1.54+1.14
0.84
1.74+1.57
81.57
Baseline
3.94:12.71
5.54
6.85+2.16
84.94
DeAR
5,63+22.8
5.42+1.10
8.05+33.00
86.14
CLIP-clip
4.73+22.2
5.00+11.9
1.01
84.58
CoDi
Coompt-Debias
20.11+5.15
41.99+2.57
46.7743.43
81.57
SFID (LC)
3,83+2.07
1.64
6.22+1.48
82.17
SFID (HC)
4.70+1153
2.59
82.77

---

Table 5: Ablation study for low confidence imputation (LCT) and hyperparameter in SFID.
Zero-shot Classification
Flickr30K
XVLM
Accuracy
A DP
R@D
R@5
@@ 10
Skew @ 100
Base
55.70
10.71
81.42
96.70
98.52
0.4463
SFID w/ Zero Filling
54.66
8.78
71.62
92.94
96.44
0.2231
SFID w/ Gaussian Noise
42.85
8.59
68.32
90.26
94.52
0.4463
SFID w/ LCI,
0.7
53.65
8.11
73.82
93.62
96.76
0.0408
SFID w/ LCI.
0.6
53.53
8.78
73.74
93.58
96.76
0,0619
SFID w/ LCI. T = 0.7
53.65
8.11
73.82
93.62
96.76
0.0408
SFID w/ LCI, T- 0.8
53.63
8.11
73.74
93.62
96.76
0.0408
SFID w/ LCI, T= 0.9
53.68
8.07
73.66
93.62
96.70
0.0619
5.2 Result Analysis
As shown 2 SFID effectively mitigates biases in CLIP RN50, CLIP ViT-B/32, and XVLM for multi-class zero-shot classification on the FACET dataset and text-to-image retrieval on the Flickr30K dataset. Specifically, the fairness metrics such as ADP in zero-shot classification and Skew 100 in text-to-image retrieval outperform existing debiasing methods, including DeAR, CLIP-clip, and Trompt-Debiee without compromising performance metrics such as accuracy and recall.
In Table 3 for the image captioning task, SFID consistently improves both the overall misclassifi- cation rate and the composite misclassification rate, outperforming other debiasing methods. This indicates that SFID not only reduces the likelihood of gender misclassification but also balances the misclassification rate across genders.
As shown in Table 4 for text-to-image generation, SFID demonstrates improvements in both overall and composite mismatch rates. Notably, SFID with the high-confidence strategy (HC) shows a significant reduction in mismatch rates, though SFID with the low-confidence strategy (LC) also achieves a considerable level of improvement. This suggests that the debiased generative model adheres more closely to the intended gender specified by the user, rather than associating certain genders with professions in a biased manner, outperforming other methods. In particular, DeAR with gender, resulting in higher overall mismatch rate and increase
SDXL tends produce only in Skew. Although there were improvements in Skew for neutral prompts, the score remains notably high, indicating that further refinements necessary for this task.
5.3 Ablation Study
To verify the impact of the low confidence imputation and the hyperparameter т, we conduct an ablation study using XVLM, one of the state-of-the-aa foundation models. As shown in Table 5 our low confidence imputation strategy demonstrates outstanding performance compared to zero and Gaussian imputation. Morcover, the hyperparameter 7 shows optimal performance at 0.7, thereby we set T = 0.7 across the experiments.
Conclusion
In conclusion, study addresses the critical issue of bias in Vision-Language Models (VLMs) by introducing the Selective Feature Imputation for Debiasing (SFID) method. This approach effectively reduces biases across various downstream tasks, including zero-shot classification, text-to- image retrieval, image captioning, and text-to-image generation, without compromising performance. Additionally, SFID does require extensive hyperparameter tuning costly retraining, making cost-efficient. By training a debiasing dataset separate from the test query set, SFID demonstrates its transferability and maintains zero-shot capability. Furthermore, SFID's s ability generalize across different VLM components, such as encoders and decoders, highlights its versatility in diverse multimodal contexts. Experimental results show that SFID outperforms existing debiasing methods, making it versatile and efficient solution. This advancement paves the way for fairer and more reliable applications of VLMMs diverse multimodal scenarios, promoting both fairness and practicality in real-world applications.
10

---

Acknowledgements
This work was partially supported by the EMBRIO Institute, contract #2 1210000, a National Science Foundation (NSF) Biology Integration Institute, Purdue's Elmore ECE Emerging Frontiers Center, and NSF IIS #1955890, IIS #2146091, IIS #2345235.
References
[1] D. Adila, C. Shin, L. Cai, and F. Sala. Zero-shot robustification of zero-shot models with auxiliary foundation models. 2023.
[2] Agarwal, G. Krueger, J. Clark, A. Radford, J. W. Kim, and Brundage. Evaluating clip: towards characterization of broader capabilities and downstream implications. arXiv preprint arXiv:2108. 02818, 2021.
[3] Anderson, Fernando, M. Johnson, and S. Gould, Spice: Semantic propositional image caption evaluation. In Computer Vision-ECCV 2016: 14th European Conference, Amstendam, The Netherlands, October 11-14, 2016, Proceedings, Purt 14, pages 382-398. Springer, 2016.
[4] S. Banerjee and A. Lavie. Meteor: automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the aci workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65-72, 2005.
[5] Y. Bengio, A. Courville, and P. Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 13(88)1778811 1828, 2013.
[6] H. Berg, M. Hall, Y. Bhalgat, W. Yang. H. R. Kirk, A. Shtedritski. and M. Bain. A prompt array keeps the bias away: Dehiasing vision-language models with adversarial learning. arXiv preprint arXiv:2203. 1933, 2022.
[7] G. Biau. Analysis of random forests model. The Journal of Machine Learning Research, 13 (1):1063-1095, 2012.
[8] L. Breiman. Random forests. Machine learming, 45:5-32, 2001.
[9] X. Chen, H. Fang, T.-Y. Lin, R. Vedantam, S. Gupta, P. Dollár, and C. L. Zitnick. Microsoft captions: Data collection and evaluation server. arXiv preprint a504 00325, 2015.
[10] J. Cho, A. Zala, and M. Bansal. Dall-eval: Probing the reasoning skills and social biases of text-to-image generation models. In Proceedings of the IEEE/CVT International Conference on Computer Vision, pages 3043-3054, 2023.
[11] C.-Y. Chuang. V. Jampani, Y. A. Torralba, and S. Jegelka. Debiasing vision-language models via biased prompts. arXiv preprint arXiv:2302.00070, 2023.
[12] M. De-Arteaga, A. Romanov, H. Wallach. J. Chayes, C. Borgs, A. Chouldechova, S. Geyik, K. Kenthapadi, and A. T. Kalai. Bias in bios: A case study of semantic representation bias in a high-stakes setting. In proceedings of the Conference on Fairness, Accountability, and Transparency, pages 120-128, 2019.
[13] S. Dehdashtian, L. Wang, and V. N. Boddeti. Fairerclip: Debiasing clip's zero-shot predictions using functions in rkhss. arXiv preprint S..i::2403.15993 2024.
[14] C. Denis, Elie, M. Hebiri, and Hu. Fairness guarantee multi-class classification. arXiv preprint arXiv:2109, 13642, 2021.
[15] A. Dosovitskiy, I.. Beyer, A. Kolesnikov, D. Weissenbom, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11 201.,999...,,,, 2020.
[16] J. R. Foulds, R. Islam, K. N. Keya, and S. Pan. An intersectional definition of fairness. In 2020 IEEE 36th International Conference on Data Engineering (ICDE), pages 1918-1921. IEEE, 2020.
11

---

[17] L. Gustafson, C. Rolland, N. Ravi, Q. Duval, A. Adcock, C.-Y. Fu. M. Hall, and C. Ross. Facet: Faimess computer vision evaluation benchmark. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 20370-20382, 2023.
[18] K. Hamidieh, H. Zhang, T. Hartvigsen, and M. Ghassemi. Identifying implicit social biases in lasion-aagguage models. 2023.
[19] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770- 778, 2016.
[20] Y. Hirota, Y. Nakashima, and N. Garcia. Model-agnostic gender debiased image captioning, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15191-15200, 2023.
[21] S. Janghorbani and G. De Melo. Multimodal bias: Introducing framework for stereotypical bias assessment beyond gender and race vision language models. arXiv preprint urXiv:2303.12734 2023.
[22] K. Kärkkäinen and Joo. Fairface: Face attribute dataset for balanced race, gender, and age. arXiv preprint /008:/99000491. 2019..
[23] E. Kim, S. Kim, C. Shin, and S. Yoon. De-stereotyping text-to ext-to-image models through prompt tuning. 2023.
[24] Y. Kim, Mo, M. Kim, Lee, Lee, and Shin. Bias-to-text: Debiasing unknown visual biases through language interpretation. arXiv preprini arXiv:2301.11104 2023.
[25] J. Li, D. Li, C. Xiong, and S. Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning, pages 12888-12900 PMLR, 2022.
[26] J. Li, D. Li, S. Savarese, and S. Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730-19742. PMLR, 2023.
[27] R. Mokady, A. Hertz, and A. H. Bermano. Clipeap: Clip prefix for image captioning. arXiv preprint arXiv:2111.00
2021.
[28] D. Podell, Z. English, K. Lacey, A. Blattmann, T. Dockhorn, J. Müller, J. Penna, and R. Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952 2023.
[29] Probst, M. N. Wright, and A.-L. Boulesteix. Hyperparameters and tuning strategies for random forest. Wiley Interdisciplinary Reviews: data mining and knowledge discovery, 9()): e1301, 2019.
[30] A. Radford, J. Wu, Child, D. Luan, D. Amodei, I. Sutskever, al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
[31] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021.
[32] A. Sathe, P. Jain, and S. Sitaram. A unified framework and dataset for assessing gender bias in vision-language models. arXiv preprint arXiv:2402.13663 2024.
[33] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision, pages 61 6186226, 2017.
[34] A. Seth, M. Hemani, and C. Agarwal. Dear: Debiasing vision- models with additive residuals. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 2820-6829, 2023.
12

---

[35] E. Slyman. S. Lee, S. Cohen, and K. Kafle. Fairdedup: Detecting and mitigating vision- Languaee fairness disparities semantic dataset deduplication. arXiv preprint 0rXi:2444..66133 2024.
[36] Y. Sui, M. Yin, Y. Xie, H. Phan, Aliari Zonouz, and B. Yuan. Chip: Channel independence- based pruning for compact neural networks. Advances in Neural Information Processing Systems, 11:24604-246111 2021.
[37] Z. Tang, Z. Yang. C. Zhu, M. Zeng, and M. Bansal. Any-to-any generation via composable diffusion. Advances in Neural Information Processing Systems, 36, 2024.
[38] Wang, Liu, and X. Wang. Are gender-neutral queries really gender-neutrall mitigating gender bias in image search. arXiv preprint arXiv:2109.0543; 2021.
[39] J. Wang, Y. Zhang, and J. Sang, Fairclip: Social bias elimination based on attribute prototype learning and representation neutralization. arXiv preprint arXiv:22 14562, 2022.
[40] R. Wolfe, R. Banaji, and A. Caliskan. Evidence for hypodescent in visual semantic ai. In Proceedings of the 2022 ACM Conference Fairness, Accountability, and Transparency, pages 1293-130 1304, 2022.
[41] M. Xu, W. Yin, D. Cai, R. D. Xu, Q. Wang. B. Wu, Y. Zhao, C. Yang. S. Wang, et al. A survey of resource-efficient IIm and multimodal foundation models, arXiv preprint 2024.
[42] P. Young, A. Lai, M. Hodosh, and J. Hockenmaier. From image descriptions to visual denota- tions: New similarity metries for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67-78, 2014.
[43] Y. Zeng, X. Zhang, and H. Li. Multi-grained vision language pre-training: Aligning texts with visual concepts. arXiv preprint arXiv:2111.08276, 2021.
[44] M. Zhang and C. Ré. Contrastive adapters for foundation model group robustness. Advances in Neural Information Processing Systems, 15:21682-2169 2022.
[45] D. Zhao, A. Wang, and O. Russakovsky. Understanding and evaluating racial biases in image captioning. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 4830-14840, 2021.
[46] K. Zhou, Y. LAL, and Jiang. Vistereoset: A study of stereotypical bias in pre-trained vision-language models. Association for Computational Linguistics, 2022.
13

---

A Details of the Proposed Method
A.1 Selection of Number of Pruned Feature
In SFID, the number of features to be pruned, denoted as k, is critical hyperparameter. To determine an appropriate value for k, we analyzed the feature importance by plotting Figure which shows the sorted feature importance ranks. The plot reveals that the first few features have significantly higher importance, stabilizing around the top 100 features for all components. Therefore, we k 50.
11:30 Image
I1:50 not
VITB32 Image
VITB12 Text
Feature Importance
XX1M Imsge
XVLM Toxt
635-
634-
632-
Feature Rank
Figure 5: Feature importances for gender prediction by RandomForest for each frozen representation.
A.2 Extension to Decoder
In some scenarios, embeddings may not be in 2D shape, which poses challenges for further processing. For instance, in decoder, outputs are tensors with shapes such as Z E NSxC for text decoder and Z NxCxHx> for image decoders, where N is the number of samples, S is the sequence length, is the number of channels, H the height of the feature, and W is the width of the feature. These non-2D shaped tensors are suitable for extracting feature importance via RandomForest which requires 2D data Z E IR xC To address this, we transform the data into a 2D shape by averaging over the sequence length or applying global average pooling.
H
W
1
ZaRS Or Z
Z...kk.i.
H
This averaging is applied solely for extracting feature importance indices Sand imputation values p.-
B Evaluation Metric for Image Captioning
METEOR measures the balance between precision and recall of n-grams in generated captions, incorporating synonyms. Let P and be the precision and recall of matches between the generated caption and ground truth, including exact, synonym, and paraphrase matches: METEOR F Fmean
(L-Pen) where Fmenn 10- is harmonic mean, and Pen 0.5 number term where chunk is set of contiguous words in the generated captions that are in the reference. SPICE [3 focuses on the semantic content of captions by comparing sets of propositional semantic tuples extracted from candidate and reference captions. SPICE is the F1 score of precision and recall between the tuples of generated captions and ground truth.
number of matches is penalty
R
C Comparison with Other Debiasing Approaches
As described Section I two existing methods, DeAR 34 and CLIP-clip 388, datasets with gender labels and pre-trained models to debias embeddings. While DeAR is specifically designed for image encoders and CLIP-clip for both image and text encoders, their methodologies could potentially be extended to text and image decoders, well as other encoders. In contrast,
utilize debiasing
14

---

Acouacy 0.63
Ooa Imporaant Feature
Voon mevo teee SARUY
Most Iepertant Fosture
Original Emhedding
(b) Transformed Embedding by DeAR (c) Imputed Embedding by SFID
Figure 6: (a) A linear classifier can distinguish the attribute (e.g., gender) from the extracted image embedding of each sample using only the top 2 most important features determined by the pre-trained CLIP model. (b) The DeAR method attempts to fool the attribute classifier by perturbing the features. However, it fails to do so, as the features from the two groups remain distinguishable, as indicated by the comparable accuracy achieved by the attribute classifier. (c) In contrast, the proposed method, SFID, replaces all values the important features with ambiguous values, effectively obscuring the attribute of the embedding. This method aims to remove bias-related properties from the embedding, which is demonstrated by the resulting very low classification accuracy in classifying the sensitive attribute when using the transformed embeddings. Note that the replaced features still remain within the distribution of the original embedding.
Prompt-Debias [1I] is designed primarily for text encoders, using pre- text prompts to achieve debiasing. These three approaches are selected for comparison in our analysis.
C.1 Debiasing with Additive Residuals (DeAR)
DeAR 34 involves two-step training process. First, it trains attribute classifier, he using the
frozen representation by minimizing the cross-entropy loss. Second, trains an linear transformation h. for Zp to produce additive feature to make debiased representation Za i.e., Zu ha (Z
Zp. The objective function for optimizing hta
arg min (111/ Zpl12 + X2 max(Sofftmaz(hh(22.)) NaCroos(Sntrooy(ttt22 he
indicating the need to maintain the representation's 's meaning while guiding the linear transformation ha to minimize the maximum probability of he and maximize the cross-entropy loss of he- The goal is for the additive feature to make the representation ambiguous, placing it near the attribute classifier's decision boundary. However, DeAR does not consider feature importance and applies adversarial training to all feature columns. Moreover, requires several hyperparameters for the adversarial training, such learning rate, weight decay, and the weights between loss functions (A1, λ2. and Ag). This complexity may lead to inefficiencies in adversarial training, which is sensitive to hyperparameter settings, resulting in less effective translation of the feature, as shown in Figure For DeAR training, all hyperparameters are set according to 34
C.2 CLIP-clip
CLIP-clip [38] aims to identify features related to the sensitive attribute by measuring the mutual information between each column feature and the sensitive attribute. However, this approach assumes that each column independently contains meaningful information about the sensitive attribute, which can be overly restrictive. The dependency between feature maps in the neural networks' representation output dell-doce 5 36). supporting SFID's 's capability in more effective feature selection.
Moreover, CLIP-clip is designed to clip the representation, reducing the dimension size of the embedding. The reduced embedding not applicable as input for decoder, which necessitates imputing the pruned features. Since CLIP-clip does not provide any suggestions for the imputed values, we adopted zero-value imputation across our experiments for CLIIP-lipp where the number of pruned feature is set as & 60, showing best sacuracy-faireest trade-off described in Appendix D
15

---

C.3 Prompt-Debias
Prompt-Debic [[]] seeks to mitigate bias in text embeddings by projecting out biased directions from the text input. However, Prompt-Debias is specifically designed for text-based tasks such as zero-shot classification and text-to-image generation. Unlike SFID, lacks the flexibility be applied across other components, limiting its effectiveness. This results in performance constraint, as bias amplification can occur the decoder or image encoder, components that Prompt-Debias cannot address. We include Prompt-Debias in our experiments for zero-shot classification, text-to-image retrieval, and ext-to-image generation, excluding image captioning.
D The Impact of k in CLIP-clip
CLIP-clip 38 requires a hyperparameter & to determine the number of pruned features. We selected k for CLIP-clip based on empirical results with XVLM, as shown in Figures 7 and The value k == 60 performs best in text-to-iman to-image retrieval for the Flickr30K dataset, compromising slightly on accuracy and recall. Meanwhile, SFID demonstrates the best trade-off 50, corresponding to Appendix A.I
Accuracy
SFID
SFID CLI--cip
TACET Accuracy
56
FACET DP
100
Figure The impact of SFID and CLIP-clip with XVLM on FACET dataset.
Recal1g1
HlcerB0K Skew@100 vs
SFID CLIP-clip
SFID
FlickrJ0K Recallig1
Flickatok Stew@g000
0.3
X0
Figure The impact of SFID and CLIP-clip with XVLM on Flickr dataset.
E Confidence Interval in Text-to-Image Generation
We conduct text-to-iman -omat generation 10 times with different seeds and use unified evaluation metric to measure the skewed distribution across 10 different generations for each neutral prompt:
maxxNNpm. Np.r)
Skew =
Σ PEP
P
10
where Npm and Np.r are the numbers of detected genders for each profession, a profession set. For example, if model generates the same gender for class 9 times out of 10, the Skew value for this profession becomes 90%. Although this metric does not include a confidence interval, accounts for the randomness in text-to-image generation.
16

---

On the other hand, a metric used in [ measures gender discrepancy as follows:
N p.m 0.5) 0.5)² +
No.. |P]
0.5 0.5)
Discrepancy
This metric can be used in a single generation, so it may produce a confidence interval when conducted on multiple seeds. However, this metric does not effectively reflect bias in text-to-image generation. For example, assume we have a set of four gender-dominated professions: nurse, dancer, doctor, engineer. If biased text-to-image model always produces female images for nurse and dancer, and male images for doctor and engineer, the evaluation metric becomes O, even though the bias prevalent. This metric fails demonstrate bias, even with confidence interval over 10 Therefore, our evaluation metric in Eq. ( which computes Skew over 10 runs, more effectively reflects the bias text-to-image generation, accounting for randomness.
F Multiple Sensitive Attribute
We extend our analysis to more complex bias scenarios Vision-Languee Models (VLMs), involving multiple sensitive attributes. Specifically, we conduct additional experiments that focus racial bias, considering more than two sensitive attributes to capture a broader spectrum of bias patterns.
Firstly, we adopted the FairFace dataset for training the attribute classifier, as it contains seven racial attributes: East Asian, Indian, Black, White, Middle Eastern, Latino Hispanic, and Southeast Asian. Given that RandomForest can handle multiple classes, SFID is also applicable in this context. During the evaluation stage for zero-shot classification, we used the FACET dataset, which contains 'skin tone' labels instead of race. We categorized the skin tone attributes into three categories: 'lighter,' 'middle,' and 'darker'' this setting, the biased zero-shot classifier tends to produce higher accuracy by associating certain skin tones with specific professions (e.g., bartender with lighter skin, trumpeter with darker skin).
To mitigate this bias, SFID demonstrates its effectiveness even though the attributes in the training set and test set do not exactly match, i.e. race in FairFace and skin tone FACET. We adopted an evaluation metric inspired by [16] and [14,, which measures the maximum discrepancy across the sensitive attributes for each class. This metric is defined as follows:
( c - -) a- - P ac
DP.
max JEST(()
where i E S, e is a class, and S denotes the set of multiple sensitive attributes. We consider the mean and maximum values of DP. to evaluate the bias across the classes. By applying this metric, we can effectively measure and demonstrate the reduction in bias achieved by SFID, despite the differences in sensitive attributes between the training (debiasing) and sets. The results in Table [6) show that even with limited data and racial bias present only in the image dataset, SFID can effectively mitigate the racial bias. We expect even more advanced results we have access to mnce-profsssion related text datasets well.
Table 6: Comparison of Mean Accuracy, Mean DP, and Max DP for different methods
Method
Mean Acc.
Mean DP
Max DP
CLIP-RN50 (Baseline)
51.92
13.94
33.89
CLIP-RN50 (SFID)
51.35
13.27
32.56
ULIP-VYT-B/32 (Baseline)
52.48
13.54
44.62
CLIP-ViT-B/22 (SFID)
51.97
13.31
32.71
XVLM (Baseline)
56.61
14.85
45.30
XVLM (SFID)
56.51
14.59
43.00
G
Computational Resource
17

---

Table 7: Compute Resources Used for Experiments
Component
Details
CPU
AMD EPYC 7313 16-Core Processor NVIDIA RTX A5000
GPU
(CLIP ViTB-32 Image Encoder)
54.60s
Training RandomForest Data used for debiasing
20,000 (training), 10,000 (imputation value) from FairFace
(CLIP ViTB-32 Text Encoder)
60.75s
Training RandomForest
Data used for debiasing
20,000 (training), 10,000 (imputation value) from Bias-in in--oss
FACET inference data
34,686
1,000
Flickr30K inference data
(Picked from original with balanced gender distribution.)
Inference on FACET dataset w/o SFID Inference on FACET dataset SFID
6.82s 196 ms / sample) 7.06s (0.204 ms / sample)
Inference Flickr30K dataset w/o SFID Inference on Flickr30K dataset w SFID
14.62s (0.1 (0.1462s/ / sample)
15.21s (0.1521s / sample)
Training RandomForest (CoDi Text Encoder) Training RandomForest (CoDi Image Decoder)
65.90s
104.14s
Data used for debiasing Inference on CoDi w/o SFID Inference on CoDi SFID
20,000 (training). 10,000 (imputation value) from Bias-in- Bioo
11.80s (25 prompts at once)
12.05s / (25 prompts once)
18

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims are explicitly outlined in the abstract and introduction, as
detailed in Section 1 particularly in the last paragraphs. These sections clearly state the paper's s contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Stated in Section 5.2 result analysis, regarding the performance the proposed method in text-to-image generation task.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
19

---

Answer: [NA]
Justification: This work doesn't t include theoretical result and proof. However, the empirical observation to justify the selection of hyperparameter and the effectiveness of the proposed method are demonstrated in Appendix A. D and Section
Guidelines:
The answer NA means that the paper does not include theoretical results, All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The implementation details are introduced in Section 5.1 and the code for the proposed method is provided supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
20

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: The code for the proposed method and detailed instructions provided in the supplementary material. It will be released via GitHub upon the paper's 's acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ httpppccccpubccc/uuuccccec
Codeeubmii ssiinn criionPooiccy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All the experimental details such as the training and test dataset and hyperpa- rameter, and their justification stated in Section Section 5.1 Appendix A. Appendix
A.2 and Appendix D
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the
the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: The reported experimental results provide confidence interval produced by either multiple experiments bootstrapping.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
21

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: Detailed information on the computational resources provided in Appendix G
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http:///wwrrpppccccpppppt/eteeaa
Question:
Answer: [Yes]
Justification: The research conducted in the paper conforms the NeurlPS Code of Ethics. The study ensures scientific integrity through transparent methods and reproducibility, addresses societal impacts positively, and promotes fairness by mitigating biases.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The research primarily focuses on positive societal impacts and does not have any negative socictal implications.
22

---

Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considcrations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning conrectly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The research utilizes publicly available datasets and pre-trained models, ensuring there is no risk of misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All code and datasets used in the research are properly credited, with the appropriate license (CC-BY 4.0) explicitly mentioned and respected.
Guidelines:
The answer NA means that the paper does
existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
23

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the balaaets.ppaerssttt paperssii thcode. com con/datases package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: No new assets are introduced in this research.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
This research does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This research does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
24

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "1rvsn5PWXT", "reviewer": "Reviewer_ystz", "summary": "The paper introduces a new method to reduce biases in VLMs, which works by using a random forest to identify the bias-related features in model representations and then imputes the values for those features with values from the low-confidence samples. The authors test it on tasks like zero-shot classification, image captioning, and text-to-image generation.", "questions": "- In line 188 the authors mention that “For a frozen component in VLMs g, whether it is an encoder or decoder, or processes image or text, we obtain the frozen representations”. Can the authors provide a breakdown of the computation cost when the encoder or the decoder is a language model that is large? \n- Would it be possible to generate embedding translation (figure 4) for SFID as well?", "limitations": "The authors have provided some limitations in section 5.2. The paper could benefit from an extended limitations section.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The method works across different tasks (classification, captioning, generation) and model components (encoders, decoders).\n- SFID doesn't require costly model retraining or extensive hyperparameter tuning.\n- The authors test SFID on multiple state-of-the-art VLMs (CLIP, XVLM) and compare it against existing debiasing methods.", "weaknesses": "- The paper focuses primarily on gender bias without exploring how the method performs on more complex biases, or even biases on other attributes -- e.g. race or age attributes are provided in the FairFace dataset, it would be good to show how SFID can mitigate biases w.r.t these attributes. \n- The paper doesn't address potential limitations of using Random Forest for feature importance. There could be cases where complex, non-linear relationships between features and bias are missed by this approach.\n- Insufficient comparison with a wider range of existing debiasing methods. The paper only compares SFID with DeAR and CLIP-clip, and is potentially missing out on how it compares to other methods e.g. https://arxiv.org/pdf/2302.00070.\n- Notation in Section 3 could be simplified (especially 3.3)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### How SFID Mitigates Bias in Various Attributes, Including Racial Bias\nThank you for your detailed observations on how SFID mitigates different types of bias, which is even more complex.\n\nSFID effectively addresses biases when attribute labels are provided such as gender and race in the FairFace dataset. The RandomForest attribute classifier is trained to identify important features within the frozen image embedding. For instance, as illustrated in **Figure 1 of our rebuttal PDF**, GradCAM highlights that the 'important features' pertain to human faces, allowing for the recognition of identity in both gender and race attribute cases. SFID aims to impute the values in these important features with those of low-confidence samples, rendering the final embedding ambiguous in terms of the targeted attribute.\n\nTo demonstrate the adaptability of SFID to different types of attributes, we have included experimental results addressing racial bias in our **global rebuttal**. Both **Figure 1 in the PDF** and the **Table in the global rebuttal** illustrate SFID's capability to mitigate more complex biases, showcasing its overall effectiveness.\n### Effectiveness in using RandomForeset\nWe appreciate the reviewer's insightful feedback regarding the potential limitations of using RandomForest for feature importance. However, we argue that using RandomForest for feature importance is both theoretically and empirically sufficient and effective.\n- Contrary to some concerns, RandomForest is capable of capturing complex, non-linear relationships due to its ensemble nature. By combining multiple decision trees, each considering different feature subsets and data splits, RandomForest can effectively model complex interactions and dependencies among features.\n- Specifically, given the ample representation of images or text extracted by pre-trained neural networks, RandomForest's role is to identify important features while considering the dependencies between them.\n- In short, RandomForest is an interpretable method for determining feature importance, robust against overfitting, and efficient in handling large datasets.\n- Moreover, the empirical results in the paper demonstrate the effectiveness and versatility of using RandomForest within our framework.\n\nWhile we have demonstrated the effectiveness of RandomForest in our current work, we acknowledge the importance of exploring additional techniques that might further enhance the modeling of complex relationships. Future research could incorporate advanced methods for identifying important features while improving the quality of representation.\n### Comparison beyond DeAR and CLIP-clip\nThank you for your feedback. We acknowledge that many debiasing methods have been proposed recently, but most focus on specific downstream tasks, with only a few demonstrating versatility across various tasks, such as DeAR and CLIP-clip. The paper suggested by the reviewer, Prompt-debias [1], is limited to scenarios where the VLM input is text, making it inapplicable to tasks such as image captioning.\n\nFor the other three tasks, we have reported the results in **our rebuttal PDF file**. The results in Table 1 show that while Prompt-debias mitigates bias in zero-shot classification and text-to-image retrieval, ***our method, SFID, is superior in mitigating bias in these tasks compared to Prompt-debias***. In the case of text-to-image generation, Prompt-debias shows slightly better performance with neutral prompts with CoDi, but significantly deteriorates the bias with gender prompts.\n\nIn conclusion, few methods are versatile across various downstream tasks. SFID demonstrates superior performance in mitigating various types of bias across all the downstream tasks we evaluated.\n\n[1] Chuang, Ching-Yao, et al. \"Debiasing vision-language models via biased prompts.\" arXiv preprint arXiv:2302.00070 (2023).\n### Notation in Section 3\nThank you for pointing out the readability issues. We will revise the notations to be simpler and will add an appendix to provide more detailed explanations and visualizations for each evaluation metric to increase readability.\n### Computational Cost\nWe are happy to address the computational cost for each component and data type. **The overall computational cost is significantly low** since SFID only utilizes frozen embeddings from a pre-trained network, without involving any training or fine-tuning of the neural network. Regardless of the network size, we break down the computational cost into two parts: a) identifying feature importance and b) inference. Both tables demonstrate that applying SFID does not increase the computational cost in practice.\n#### Identifying Feature Importance (One-time for each component)\n|Component|Details| \n|-|-|\n|CLIP Encoders|54.60s to 60.75s|\n|CoDi Text Encoder|65.90s|\n|CoDi Image Decoder|104.14s|\n#### Inference\n|Component|Details| \n|-|-|\n|Zero-shot classification|0.196ms / sample| \n|Zero-shot classification + SFID| 0.204ms / sample|\n|T2I retrieval | 0.146s / sample|\n|T2I retrieval + SFID| 0.152s / sample|\n|T2I generation (CoDi)| 11.80s / 25 prompts|\n|T2I generation (CoDi) + SFID | 12.05s / 25 prompts|\n### Visualization of embedding translation by SFID\nWe include a visual aid for the embedding to indicate the bias-free representation achieved by SFID in **Figure 2 of our rebuttal PDF**. As SFID aims to obscure all the features in the important index by imputing ambiguous values, all the data points are gathered at a single point to mute the information related to the sensitive attributes, while maintaining the features in other indices as they are. It is important to note that although this translation is enforced to a single point, it still remains within the distribution of the original samples, as shown in Figure 3 of our main paper.\n### Limitation and Future work of SFID\nThank you for pointing out the limitations section. We have identified potential limitations and areas for future exploration in **our global rebuttal.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0oGpBfo5eW", "reviewer": "Reviewer_umE5", "summary": "this paper introduces Selective Feature Imputation for Debiasing (SFID), which integrates feature pruning and low confidence imputation (LCI) to effectively reduce biases in VLMs.", "questions": "About the working mechanism of the proposed method, the authors may dig deep how the identified features is correlated to the social biases, maybe with visualization tools.", "limitations": "see weaknesses and questions", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1.The proposed method utilize feature selection techniques such as RandomForest to identify gender-specific (or race) biases within the frozen representation, and subsequently replace bias-causing features with bias-free representation.\n2. SFID eliminates the need for costly retraining of pre-trained VLMs and it simply utilizes datasets with sensitive attributes in individual images or texts for debiasing.\n3.The experimental results demonstrate the efficacy of the proposed method in mitigating bias across 4 downstream tasks", "weaknesses": "1.the method is simple, utilizing Random Forest, which is pretty well-known in the machine learning community. However, the novelty is a little limited.\n2. how to define the \\Delta DP for attributes with more than two levels? For instance, race has white, asian, black, etc.\n3. what is the principle for choosing the number of important features? will it affect the performance?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Novelty of the proposed method\nWe appreciate the reviewer's opinion regarding the simplicity and novelty of utilizing RandomForest in our framework.\n\n\nWhile the RandomForest algorithm itself is well-known and simple, the novelty of our work lies not in the use of RandomForest alone, but in the innovative way we integrate it within our Selective Feature Imputation for Debiasing (SFID) framework. The SFID methodology introduces several novelties:\n- **Selective Feature Imputation:**  Our approach combines feature pruning with low-confidence imputation (LCI) to replace bias-causing features with bias-free representations. This novel technique maintains the semantic integrity of the embeddings while effectively reducing bias.\n- **Efficiency and Cost-Effectiveness**: SFID eliminates the need for costly retraining of pre-trained VLMs and does not require paired text-image datasets, making it a cost-effective and efficient solution for debiasing.\n- **Versatility Across Modalities and Tasks**: Unlike existing methods that focus on specific modalities or tasks, SFID is designed to be seamlessly integrated into various components of VLMs, including both encoders and decoders, across a wide range of tasks such as zero-shot classification, text-to-image retrieval, image captioning, and text-to-image generation. **We emphasize that none of the existing work can debias such a seamless number of downstream tasks without further training or fine-tuning.**\n- **Empirical Validation and Effectiveness**: Our experimental results demonstrate the practical effectiveness of SFID in mitigating biases across various tasks without compromising performance. This is evidenced by significant improvements in fairness metrics across multiple benchmarks. The comparative analysis with other debiasing methods, such as DeAR, CLIP-clip, and Prompt-debias (newly added), highlights t**he superior performance of SFID**, further validating the novelty and utility of our approach.\n\nWhile RandomForest is indeed a well-known algorithm, the novelty of our work lies in the innovative application and integration of this algorithm within the SFID framework. This novel approach offers a significant advancement in the field of debiasing VLMs, as demonstrated by our empirical results and comparative analysis. Furthermore, we highlight the necessity for a unified debiasing framework, such as SFID, to address the challenges posed by the emergence of diverse VLMs applications.\n\n### Multiple sensitive attributes in racial bias\nThanks for pointing out the details how to define $\\Delta DP$ in multiple attributes and how SFID mitigates such bias. \n\nSFID is free from the type of bias when the attribute labels are given such as gender and race in FairFace dataset, even in scenarios involving multiple attributes.\nWe define $\\Delta DP$ for multiple attributes by adopting the evaluation metric from [1] and [2], which measures the maximum discrepancy across the sensitive attributes for each class. This metric is defined as follows:\n\n\\begin{align*}\nDP_c =  \\max_{i \\in S} \\max_{j \\in S \\setminus \\{i\\}} \\Bigg( \\bigl| P(y=c \\mid a=i) - P(y=c \\mid a=j) \\bigr| \\Bigg)\n\\end{align*}\nwhere $i, j \\in S$, $c$ is a class, and $S$ denotes the set of multiple sensitive attributes. We consider the mean and maximum values of $DP_c$ to evaluate the bias across the classes. By applying this metric, we can effectively measure and demonstrate the reduction in bias achieved by SFID, despite the differences in sensitive attributes between the training and test sets.\n\nTo support the adaptability of SFID in addressing different types of bias with multiple attributes, we report experimental results for racial bias in our **global rebuttal**. The table in the global rebuttal demonstrates that SFID can effectively mitigate racial bias even when dealing with multiple sensitive attributes. This showcases SFID's versatility and robustness in reducing bias across various scenarios.\n\n[1] Foulds, James R., et al. \"An intersectional definition of fairness.\" 2020 IEEE 36th International Conference on Data Engineering (ICDE). IEEE, 2020.   \n[2] Denis, Christophe, et al. \"Fairness guarantee in multi-class classification.\" arXiv preprint arXiv:2109.13642 (2021).\n\n### Impact of the number of important features\nThank you for pointing out how to choose the number of important features. Indeed, the number of important features significantly affects performance, but we have an easy way to determine the optimal number, $k$, as shown in Appendix A.1 of the main paper.\n\nIn SFID, selecting $k$ is crucial for performance. As shown in Figure 5 of Appendix A.1, we determine an appropriate value for $k$ by identifying the elbow point in the feature importance curve, obtained by sorting the indices by their importance. According to Figure 5, $k=50$ is a reasonable elbow point.\n\nWe also analyze the impact of $k$ on performance. As expected, we achieve the best trade-off between utility and fairness when $k=50$, as shown in Figure 6 and Figure 7 in Appendix C of the main paper. \n\nOverall, we have a solid guideline for choosing the number of important features, $k$, and acknowledge its impact on performance. The observed trend aligns with our intuition.\n\n### Visualization: Correlation of identified features to social biases\nThank you for suggesting this valuable enhancement to our paper.\n\nWe visualize how the important features are correlated to social biases by showing GradCAM visualizations, as presented in **Figure 1 of our rebuttal PDF file**. For example, the more important features highlight human faces, while the least important features are correlated with the image background.\n\nSFID imputes the representations at important indices with the values from low-confidence samples, making the face-related features ambiguous. Despite this imputation, the replaced values remain within the distribution of the original samples, as shown in Figure 3 of our main paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xVet5ngD59", "reviewer": "Reviewer_Wrj8", "summary": "The authors propose a method for debiasing the representations of Vision-Language Models (VLMs) that can be applied at various layers of the image and text encoders/decoders, and can be used for a variety of downstream tasks such as image generation, 0-shot classification, text to image retrieval, and image captioning.", "questions": "- Is the improvement of the proposed approach over the compared methods significant when CI are considered, where these CI's are obtained from bootstrapping or some other way?\n- How does the proposed approach perform when compared against the method proposed in \"Debiasing vision-language models via biased prompts.\"?", "limitations": "I think the discussion on limitations could be improved. The authors point to Section ***5.2 Result Analysis** as their discussion of limitations. 5.2 covers the results of their experiments, and does include some discussion on how the experimental results on image generation point to room for improvement. However, I feel like a discussion of the limitations would be greatly improved by dedicating a separate section/subsection to it. I would expect the authors to discuss more fundamental limitations, rather than just experimental performance. For instance, I would include details on how the method assumes access to a validation set with labels for bias/protected attributes (e.g. a dataset with race/gender/etc labels).", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- This work's goal of debiasing VLMs is an important and timely problem that should be of interest to many in the NeurIPS community \n- The proposed approach is seemingly effective, lightweight and seems easy to apply\n- The method is finetuning-free, which mitigates issues such as catastrophic forgetting", "weaknesses": "**Minor issues:**\n\n- The writing could be improved in parts. There are minor grammatical errors in parts, such as \"For instance, Hirota et al. [18], Zhao et al. [42] investigated bias in image captioning, where specific genders or races are disproportionately represented **leading to generate** biased caption.\" I suggest another editing pass or two.\n- I find the structure of section 3 \"Bias Analysis in VLMs\" is a little odd. It mostly reads like background/preliminary material, but it also contains some experimental setup details (such as in line 165). This seems off, as this is before the proposed method section. I suggest moving these experimental details to section 5, with the rest of the experimental setup. \n- Figure 4 shows that DeAR fails to produce bias-free representations. Including a figure likes this naturally makes the reader want to see a figure showing that the proposed approach **does** succeed here. It feels incomplete as-is. \n\n**More significant issues:**\n\n- The experimental results don't include confidence intervals. The authors justify this by pointing out that the proposed approach is deterministic. However, CIs still can and should be calculated by an approach such as bootstrapping the test set. \n- Relatedly, the text to image generation experiment is **not** deterministic if the text2img generator's random seed is changed, so CIs should definitely be reported here. \n- The compared methods are limited to only 2 other debiasing approaches and the baseline model. \n- The proposed approach of 1) finding features most associated with the biased attributes and 2) replacing these features with an imputed value from low-confidence samples is, I think, very similar in practice to Chuang et al.'s approach [1] in \"Debiasing Vision-Language Models via Biased Prompts\". Chuang modifies embeddings by projecting them onto the subspace defined by the linear direction most associated with the biased attribute(s). Intuitively, the direction most associated with the biased attribute should be similar to the direction defined by the features identified in step 1) of the proposed approach. Not that I am **not** saying that this similarity alone is a limitation; I think the proposed approach is different enough to still have sufficient novelty. However, I think that Chuang's approach really needs to be compared against (for all experiments except maybe image captioning), in order to verify that these features are either finding a different direction in the embedding space or that the low confidence imputation performs better than just projecting onto the bias subspace. \n\n[1] Chuang, Ching-Yao, et al. \"Debiasing vision-language models via biased prompts.\" arXiv preprint arXiv:2302.00070 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Confidence interval in Text-to-Image Generation\nThank you for pointing out the lack of confidence intervals in our experimental results.\n\nWe acknowledge that text-to-image generation is not deterministic. This is why we conduct text-to-image generation 10 times with different seeds and use a unified evaluation metric to measure the skewed distribution across 10 different generations for each neutral prompt:\n\\begin{align*}\nSkew = \\frac{1}{|\\mathcal{P}|} \\sum_{p \\in \\mathcal{P}} \\frac{\\max(N_{p,m}, N_{p,f})}{10},\n\\end{align*}\nwhere $N_{p,m}$ and $N_{p,f}$ are the numbers of detected genders for each profession, $\\mathcal{P}$ is a profession set. For example, if a model generates the same gender for a class 9 times out of 10, the Skew value for this profession becomes 90%. Although this metric does not include a confidence interval, it **accounts for the randomness in text-to-image generation**.\n\nOn the other hand, a metric used in [1] measures gender discrepancy as follows:\n\\begin{align*}\nDiscrepancy=\\sqrt{\\Bigl(\\frac{N_{p,m}}{\\vert \\mathcal{P}\\vert}-0.5\\Bigr)^2 + \\Bigl(\\frac{N_{p,f}}{\\vert \\mathcal{P}\\vert}-0.5\\Bigr)^2},\n\\end{align*}\nThis metric can be used in a single generation, so it may produce a confidence interval when conducted on multiple seeds. However, this metric does not effectively reflect bias in text-to-image generation. For example, assume we have a set of four gender-dominated professions: {nurse, dancer, doctor, engineer}. If a biased text-to-image model always produces female images for nurse and dancer, and male images for doctor and engineer, the evaluation metric becomes 0, even though the bias is prevalent. This metric fails to demonstrate bias, even with a confidence interval over 10 runs. Therefore, our evaluation metric, which computes Skew over 10 runs, more effectively reflects the bias in text-to-image generation, accounting for randomness.\n\nOn the other hand, demonstrating a confidence interval for the mismatch rate for gender-prompt is possible as it can be measured by a single experiment. As we have already executed the experiment with 10 different seeds,  we report the confidence interval for mismatch rates in **Table 1 of the rebuttal PDF file**.\n\n[1] Chuang, Ching-Yao, et al. \"Debiasing vision-language models via biased prompts.\" arXiv preprint arXiv:2302.00070 (2023).\n### Confidence interval for all downstream tasks\nWe agree with the importance of including confidence intervals in experimental results, even when the model is deterministic, as is the case with pre-trained models used in zero-shot classification, text-to-image retrieval, and image captioning.\n\nFor zero-shot classification and image captioning, we conducted experiments on the full test dataset. We utilized the bootstrapping technique by generating 1000 datasets with replacement and reported the confidence intervals in **Table 1 of the rebuttal PDF file** for zero-shot classification and image captioning. \n\nIn the case of text-to-image retrieval, the test set is a subset of the entire Flickr30K dataset, created by randomly sampling 1000 samples. We extended the experiments by using 10 different test sets to obtain the confidence intervals. These results are also included in **Table 1 of the rebuttal PDF file**.\n\n#### Robust superiority of SFID\nOverall, when we include confidence intervals for all downstream tasks via bootstrapping or additional experiments, SFID consistently outperforms the comparison methods in all cases. **This robust performance across various scenarios highlights the effectiveness and reliability of SFID in mitigating bias.**\n\n### Comparison with Prompt-debias (Chuang et. al.)\nThank you for recognizing a similar approach and the novelty of our work. We agree that while their goal in debiasing VLM embeddings is analogous to ours, our methodologies differ significantly.\n\nWe include the experimental results of Prompt-debias in Table 1 of the rebuttal PDF file. It turns out that while Prompt-debias can mitigate bias in various downstream tasks, our method, **SFID, performs significantly better**. Moreover, in text-to-image generation, Prompt-debias works only with neutral prompts but deteriorates the bias in the case of mismatch rates for gender-prompts.\n\nThe reason our method works better than Prompt-debias is due to the bias amplification in the decoder or image encoder. Prompt-debias's application is limited to the text embedding, which is the output of the text encoder. In contrast, SFID mitigates bias in either the encoder or decoder output, or both, providing a more comprehensive approach to debiasing.\n\n### Visualization of embedding translation by SFID\nWe include a visual aid for the embedding to indicate the bias-free representation achieved by SFID in **Figure 2 of our rebuttal PDF**. As SFID aims to obscure all the features in the important index by imputing ambiguous values, all the data points are gathered at a single point to mute the information related to the sensitive attributes, while maintaining the features in other indices as they are. It is important to note that although this translation is enforced to a single point, it still remains within the distribution of the original samples, as shown in Figure 3 of our main paper.\n\n### Grammar and Structure\nThank you for pointing out the organization of our paper. We will address the minor grammatical errors and ensure clarity throughout. In Section 3, it is true that it serves both as preliminaries and motivation section. Experimental setups are included in this section to provide readers with context regarding each downstream task and experimental setting, ensuring consistency. We will revise and reorganize our content to enhance readability.\n\n### Limitation and Future work of SFID\nThank you for pointing out the limitations section. We have identified potential limitations and areas for future exploration in **our global rebuttal.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Thank you to the reviewers for the valuable feedback! Here is our global rebuttal, with a PDF file attached for figures and tables. Please refer to the individual rebuttals for specific details and additional information.

### SFID mitigates various types of bias, even in multi-attribute case
To address the reviewers' concerns regarding various types of bias in Vision-Language Models (VLM), especially in more complex cases involving multiple sensitive attributes, we conducted additional experiments focusing on racial bias with more than two sensitive attributes.

Firstly, we adopted the FairFace dataset for training the attribute classifier, as it contains seven racial attributes: East Asian, Indian, Black, White, Middle Eastern, Latino Hispanic, and Southeast Asian. Given that RandomForest can handle multiple classes, SFID is also applicable in this context. During the evaluation stage for zero-shot classification, we used the FACET dataset, which contains 'skin tone' labels instead of race. We categorized the skin tone attributes into three categories: 'lighter,' 'middle,' and 'darker.' In this setting, the biased zero-shot classifier tends to produce higher accuracy by associating certain skin tones with specific professions (e.g., bartender with lighter skin, trumpeter with darker skin).

To mitigate this bias, SFID demonstrates its effectiveness even though the attributes in the training set and test set do not exactly match, i.e. race in FairFace and skin tone in FACET. We adopted an evaluation metric inspired by [1] and [2], which measures the maximum discrepancy across the sensitive attributes for each class. This metric is defined as follows:

\begin{align*}
DP_c =  \max_{i \in S} \max_{j \in S \setminus \{i\}} \Bigg( \bigl| P(y=c \mid a=i) - P(y=c \mid a=j) \bigr| \Bigg)
\end{align*}
where $i, j \in S$, $c$ is a class, and $S$ denotes the set of multiple sensitive attributes. We consider the mean and maximum values of $DP_c$ to evaluate the bias across the classes. By applying this metric, we can effectively measure and demonstrate the reduction in bias achieved by SFID, despite the differences in sensitive attributes between the training (debiasing) and test sets.

| Method | Mean Acc. | Mean DP | Max DP |
| -------- | -------- | -------- | -------- |
| CLIP-RN50 (Baseline)     | 51.92     | 13.94     | 33.89     |
| CLIP-RN50 (SFID)     | 51.35     | **13.27**     | **32.56**     |
| CLIP-ViT-B/32 (Baseline)     | 52.48     | 13.54     | 44.62     |
| CLIP-ViT-B/32 (SFID)    | 51.97     | **13.31**     | **32.71**     |
| XVLM (Baseline)     | 56.61     | 14.85     | 45.30     |
| XVLM (SFID)    | 56.51     | **14.59**     | **43.00**     |

The results show that even with limited data and racial bias present only in the image dataset, SFID can effectively mitigate the racial bias. We expect even more advanced results if we have access to race-profession related text datasets as well. 

[1] Foulds, James R., et al. "An intersectional definition of fairness." 2020 IEEE 36th International Conference on Data Engineering (ICDE). IEEE, 2020.    
[2] Denis, Christophe, et al. "Fairness guarantee in multi-class classification." arXiv preprint arXiv:2109.13642 (2021).



### Limitation and future work of SFID
In response to the reviewer's concerns, we have identified some limitations of SFID and potential areas for future exploration.
##### Dependence on validation set
SFID assumes that labels in the debiasing dataset (training and validation set) are available. If the validation set is unavailable, a subset of the training set could be used as a validation set. A potential limitation is that the quality of low-confidence samples depends on the representativeness of the validation set. For example, if the samples in the validation set are out-of-distribution, the imputed values will follow this distribution, potentially worsening performance.
##### Compound bias
SFID can be applied to various types of bias, as shown in our response above. However, there is room for improvement in mitigating more than one type of bias simultaneously, such as gender and race together. As future work, we plan to extend our framework to address compound bias.

##### Diversity in Low Confidence Imputation
In the Low Confidence Imputation (LCI) process, we take the average of all low-confidence samples for each feature index. Although the imputed value remains in-distribution, the diversity of the imputed value could be limited as they are projected into a single imputation value. We will explore how the distribution of the imputation value affects diversity and how to properly distribute the imputation values to ensure diversity in the generation task while maintaining debiased results.

We appreciate the reviewer's insights and will address these limitations in our future work.

Meta-review: The reviewers find the problem of debiasing of VLMs being studied in this paper to be timely and important, the proposed method to be effective, lightweight and easy to apply, and the experimental results to be convincing. The reviewers appreciate the fine-tuning free nature of the proposed method and the fact that it works across multiple tasks and model components.

The reviewers had raised some concerns, but the rebuttal successfully addressed most of them and all reviewers recommend acceptance. The authors are recommended to improve the final paper version by following the reviewer recommendations.
Decision: Accept (spotlight)