A Unified Debiasing Approach for Vision-Language Models across Modalities and Tasks

Authors: hoin  jung
taeuk  jang
xiaoqian  wang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 06:07:24

Abstract: Recent advancements in Vision-Language Models (VLMs) have enabled complex multimodal tasks by processing text and image data simultaneously, significantly enhancing the field of artificial intelligence. However, these models often exhibit biases that can skew outputs towards societal stereotypes, thus necessitating debiasing strategies. Existing debiasing methods focus narrowly on specific modalities or tasks, and require extensive retraining. To address these limitations, this paper introduces Selective Feature Imputation for Debiasing (SFID), a novel methodology that integrates feature pruning and low confidence imputation (LCI) to effectively reduce biases in VLMs. SFID is versatile, maintaining the semantic integrity of outputs and costly effective by eliminating the need for retraining. Our experimental results demonstrate SFID's effectiveness across various VLMs tasks including zero-shot classification, text-to-image retrieval, image captioning, and text-to-image generation, by significantly reducing gender biases without compromising performance. This approach not only enhances the fairness of VLMs applications but also preserves their efficiency and utility across diverse scenarios.

Main paper: 
 introduction: 
 Vision-Language Models (VLMs) have revolutionized the way we handle multimodal tasks by enabling simultaneous processing of text and image data. Models such as CLIP [31] and XVLM [43] serve as foundation models [41] for various downstream tasks demonstrating the remarkable versatility of these systems such as zero-shot classification and text-to-image retrieval. Additionally, models like BLIP [25] and CoDi [37] enhance this spectrum by facilitating tasks such as image captioning and text-to-image generation. These examples highlight the diverse capabilities of VLMs in adapting to various specific multimodal interactions.Despite their remarkable capabilities, VLMs have a critical bias issues, often skewing the model outputs in ways that reflect societal stereotypes [21] such as gender [46] or racial [45] biases in assigning professions or describing scenarios. For example, studies have identified biases in multi-class zero-shot classification [17] , where models might disproportionately associate certain professions with specific genders. Similarly, biases in text-to-image retrieval [18, 39] can lead to the preferential retrieval of images that reinforce stereotypical narratives. The implications of these biases extend to image captioning [45, 20] and text-to-image generation [10] , where the descriptive and generative capacities of VLMs may perpetuate and even amplify existing societal prejudices. These issues highlight a critical need for effective debiasing strategies that can ensure the fairness of VLMs applications. Given the bias issues in the realm of VLMs, debiasing approaches have been proposed. However, the existing debiasing frameworks for VLMs tend to focus on specific modalities and tasks. For example, [20] and [23] are tailored to particular tasks such as image captioning or text-to-image generation, respectively. On the other hand, while some debiasing strategies are originally designed for mitigating bias in the image or text encoder's output for retrieval task, like DeAR [34] and CLIPclip [38] , they offer potential for broader application across various modalities and tasks. For instance, DeAR can extend the use of adversarial training to neutralize biases in the decoder's intermediate representations by aligning them near the decision boundary for sensitive attribute by deceiving the attribute classifier. Yet, this method's effectiveness is hampered by the intrinsic complexities and sensitivities of adversarial training techniques, which are highly dependent on precise hyperparameter tuning. Similarly, CLIP-clip could potentially adapt to various tasks by pruning features in the frozen representation that exhibit high mutual information with sensitive attributes. However, this approach does not consider dependencies between feature columns. Furthermore, the process of zeroing out pruned features might inadvertently distort the semantic meaning of embeddings, leading to a loss in the quality and relevance of the output. Moreover, Prompt-Debias [11] , which relies on pre-defined text prompts to debias the text encoder, is limited to text encoders and cannot be applied to text decoders or other components.To overcome these limitations, we propose Selective Feature Imputation for Debiasing (SFID) incorporating feature pruning and low confidence imputation (LCI). Unlike existing methods, this approach can be seamlessly integrated into various parts of VLMs, whether applied to encoders, decoders, or both, enhancing its utility in diverse contexts, while effecitvely maintain the dimensionalty and semantical meaning of debiased representation. In details, by utilizing feature selection techniques such as RandomForest [8] to identify gender-specific (or race) biases within the frozen representation, SFID prunes and subsequently replace bias-causing features with bias-free representation obtained from ambiguous samples identified by RandomForest, thereby maintaining the semantic content while effectively reducing bias. Furthermore, SFID eliminates the need for costly retraining of pre-trained VLMs and does not require paired text-image datasets. It simply utilizes datasets with sensitive attributes in individual images or texts for debiasing. For instance, to debias gender bias in VLMs, datasets like FairFace [22] for image inputs and Bias-in-Bios [12] for text inputs are employed. This approach not only maintains the utility and efficiency of VLMs but also broadens their applicability across varied scenarios, thereby enhancing fairness without compromising performance.The experimental results demonstrate the efficacy of the proposed method in mitigating bias across various downstream tasks, such as zero-shot classification, text-to-image retrieval, image captioning, and text-to-image generation. Our method consistently outperforms other debiasing methods without compromising the performance of the downstream tasks. Consequently, SFID enhances the fairness and utility of VLMs across a wide range of multimodal tasks, setting a new standard for debiasing strategies in this field. 
bias evaluation in vlms: 
 Recent research has focused significantly on identifying and evaluating bias in VLMs. Agarwal et al. [2] and Wolfe et al. [40] observed that CLIP [31] embeddings exhibit substantial racial and gender biases. Additionally, Chuang et al. [11] highlighted that text prompts can capture spurious features in visual representations, exacerbating the bias. Kim et al. [24] further investigated how the association between sensitive attributes and specific keywords contributes to bias issues in downstream tasks. These biases lead to unfair outcomes in zero-shot binary classification and text-to-image retrieval, as noted by [13, 18] . Moreover, Slyman et al. [35] extend the bias in zero-shot classification in multi-class setting using the FACET dataset [17] and its evaluation metric. Seth et al. [34] suggested a new metric for fairness in text-to-image retrieval, considering the gender distribution in the query set. Beyond zero-shot classification and text-to-image retrieval, other downstream tasks also reveal biases. For instance, Hirota et al. [20] and Zhao et al. [45] investigated bias in image captioning, where specific genders or races are disproportionately represented leading to generate biased caption. Similarly, Cho et al. [10] raised concerns about bias in text-to-image generation and suggested evaluation methods. Finally, Sathe et al. [32] emphasized the need for a unified evaluation approach for various downstream tasks to address these pervasive biases comprehensively.As bias issues have arisen, many debiasing methods have been proposed. Zhang and Ré [44] trained an adapter on frozen representations to debias spurious correlations in zero-shot binary classification. Additionally, Chuang et al. [11] , Adila et al. [1] , and Berg et al. [6] suggested methods to manipulate input prompts or text tokens for debiasing spurious correlations in VLMs' encoders, covering downstream tasks such as zero-shot binary classification and text-to-image retrieval. For image captioning, Hirota et al. [20] proposed a fine-tuning method for both the encoder and decoder to mitigate biases. For text-to-image generation, Kim et al. [23] recommended a de-stereotyping prompt design to address biases. Some methods, such as DeAR [34] and CLIP-clip [38] , manipulate frozen representations without training the entire model and can be generalized across various tasks, as discussed in Section 4. Dehdashtian et al. [13] also aimed to debias frozen representations from both the image and text encoders, though this approach requires class labels and text-image pair datasets, which are challenging to define and obtain across various downstream tasks. To address these limitations, we propose the first unified debiasing method for VLMs across various downstream tasks, which demonstrates outstanding performance compared to the extensions of other debiasing methods, such as DeAR and CLIP-clip.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: Stated in Section 5.2, result analysis, regarding the performance of the proposed method in text-to-image generation task. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
zero-shot classification: 
 Multi-class zero-shot classification leverages the capability of VLMs that train image and text encoders jointly. Specifically, the predicted class is determined by providing text prompts about classes to the text encoder and selecting the class with the highest cosine similarity to the encoded image. For instance, the text prompt "a photo of a/an [CLASS NAME]" is used for the zero-shot classification. Bias issues arise from the difference in accuracy between genders for a given class. For example, in the class Carpenter, the accuracy for male carpenter images and female carpenter images might differ, as VLMs are likely to associate the concept of "male" more strongly with "carpenter." Therefore, bias is defined by the average demographic disparity, which is determined by the gender disparity in recall for each class following [17] ,∆DP mean = 1 |K| k∈K P ( Ŷ = k|a = 1) -P ( Ŷ = k|a = 0) ,where Ŷ is the predicted class, and k ∈ K represents each class in the multi-class classification, and a ∈ {0, 1} is the sensitive attribute. The overall accuracy is used as an evaluation metric for the performance of VLMs. Lower ∆DP indicates fair classification across the sensitive attribute, while higher overall accuracy denotes higher classification ability. As a baseline for zero-shot classification, CLIP [31] with ResNet-50 [19] and ViT-B/32 [15] , and XVLM [43] are adopted. We utilize the FACET [17] dataset, which includes 49,551 images across 52 classes with gender sensitive attribute.Text-to-image retrieval leverages the matching ability of image and text encoders. For a given ground truth caption, images in the query set are retrieved by sorting them according to the cosine similarity between the image embeddings and the text embedding of the caption. Bias in retrieval arises when the gender distribution in the retrieved set is skewed to a certain gender. For example, for a gender-neutral caption such as "a person in a suit is hurrying across the street." VLMs retrieve male images more frequently than females by associating the concept of 'suit' and 'male'.The evaluation metric for fairness in text-to-image retrieval is defined as Skew@M , as suggested in [34] . Let p a = N a /N for a ∈ {0, 1}, where N a is the number of images for each gender in the original dataset and N is the total number of images. For M retrieved images, calculate pa = M a /M , where M a is the number of images for each sensitive attribute in the retrieved set. The metric is then defined as Skew a = log(p a /p a ) for each sensitive attribute a indicating if a particular gender is retrieved more frequently. The final evaluation metric, Skew, is defined by averaging the maximum Skew values over the set of text prompts T ,Skew = 1 |T | t∈T max a Skew t a .The performance metric for text-to-image retrieval is defined as Recall@K, which measures the probability of the ground truth image being among the top K, where K ∈ {1, 5, 10}. Thus, lower Skew and higher Recall are desired. We use the same baselines: CLIP ResNet-50, CLIP ViT-B/32, and XVLM. In the experiments, we utilize the Flickr30K [42] dataset which includes ground truth captions and gender attributes. After modifying the ground truth captions to be gender-neutral, we select 1,000 images for testing [38] and retrieve the top 100 images for each caption for bias measure.Image captioning aims to generate a caption given an image. Fairness issues arise from differences between the gender mentioned in the caption and the ground truth gender of the subjects in the image [20] , as VLMs may prefer certain genders for particular subjects. For example, as shown in Figure 1 , image captioning models tend to associate contexts and genders, such as (athlete, male) and (long hair, female).For the evaluation metric, we first detect the gender in the generated caption by its pronoun. Specifically, for an image in the query set, we measure the gender mismatch rate for a k-th image,I k = 1 if (original gender) ̸ = (detected gender) 0 if (original gender) = (detected gender) or (neutral detected gender)where the misclassification rate for each gender is defined asM R M = 1 |M| k∈M I k , M R F = 1 |F | k∈F I k , and M R O = 1|D| k∈D I k , with M , F , and O indicating male, female, and overall, respectively. Although the overall misclassification rate is used in [20] , it cannot perfectly reflect fairness. For example, in two different situations where (M R M , M R F , M R O ) are (3.0%, 3.0%, 3.0%) and (0.0%, 6.0%, 3.0%), respectively, the overall misclassification rates are the same, but the rates for each gender are not fair. To address this, we derive a Composite Misclassification Rate, defined asM R C = M R 2 O + (M R F -M R M ) 2, which can be minimized when both the overall misclassification rate and the disparity in misclassification rates between genders are low.On the other hand, the caption's quality is measure by METEOR [4] and SPICE [3] . METEOR measures the balance between precision and recall of n-grams in generated captions, incorporating synonyms, while SPICE focuses on the semantic content of captions by comparing sets of propositional semantic tuples extracted from candidate and reference captions. (See Appendix B for details.) Considering fair image captioning, the evaluation metric for caption quality should account for both Figure 2 : GradCAM visualization for feature indices sorted by their importance in predicting an attribute (e.g., gender). Highly important features (left) focus on attribute-related characteristics such as face in the image, while the least important features (right) are associated with the background. SFID not only identifies the crucial biased features but also imputes these biased features with ambiguous values derived from low-confidence samples.the ground truth caption and a neutral caption.MaxMETEOR = max (METEOR(T truth , T caption ), METEOR(T neutral , T caption )) , MaxSPICE = max (SPICE(T truth , T caption ), SPICE(T neutral , T caption )) .In image captioning, a lower M R C score and higher METEOR and SPICE scores are preferable.In the experiment, due to ambiguity in images containing more than one person, such images are excluded from the evaluation. We adopt two image captioning models, CLIP-cap [27] and BLIP [25] . The MS-COCO dataset [9] is used as the query dataset, which consists of 10,780 images with 5 ground truth captions for each image.VLMs can be used for text-to-image generation, creating an image given a text prompt. However, bias issues in text-to-image generation arise in two ways: generating an image with the wrong gender despite of a specific gender is given in the prompt, or generating a biased gendered image for a neutral text prompt. We utilize the input prompt from [10] to generate images for captions: "a photo of G who works as a/an P," where gender set G = {a man, a woman, a person} and profession set P = {accountant, engineer, • • • }. We use 83 professions as in [10] .For the evaluation, we first detect the gender of the generated image by utilizing BLIP-2 [26] by asking "Does the person look like a male or a female?". For gender-specific prompts, a fair text-toimage generation model should produce exact gender images irrespective of the profession. Thus, we measure the mismatch rate:1 |D| k∈D I k, where D is a set of prompts for either the male or female subgroup. We adapt the composite misclassification rate, M R C , as the same as the image captioning to verify a balanced outcome between mismatch rates for each gender. For neutral prompts, a fair model should produce an equal number of images of each gender across all professions. The bias metric thus measures the skewed distribution of the model:EQUATIONwhere N p,m and N p,f are the numbers of detected genders for each profession, and C = 10 is the number of generation for each prompt. In text-to-image generation, lower values of both M R C and Skew indicate fairness. We use SDXL [28] and CoDi [37] as baselines for text-to-image generation. 4 Proposed MethodQuestion: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Justification: Detailed information on the computational resources is provided in Appendix G.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
selective feature imputation for debiasing: 
 Let X D be a debiasing dataset with the sensitive attribute label y D . Let X Q be a query dataset that the user is interested in and wants to evaluate in debiased downstream tasks. For a frozen component in VLMs g, whether it is an encoder or decoder, or processes image or text, we obtain the frozen representations Z D = g(X D ) and Z Q = g(X Q ), respectively.For the debiasing representation Z D , Selective Feature Imputation for Debiasing (SFID) uses a RandomForest [8] f to predict the sensitive attribute y D . Given the interpretability of RandomForest, it is capable of providing feature importance for predictions, allowing us to identify which features in the frozen representation are relevant to the sensitive attribute. Additionally, RandomForest is known for not requiring hyperparameter tuning [29] and for its computational efficiency [7] , making it an easily implementable choice. As the objective of SFID is to lead VLMs' components to produce a fair outcome, free of bias regarding a sensitive attribute, SFID prunes the important features that show higher relevance to the sensitive attribute. This procedure is beneficial as it considers the dependency between features, whereas methods like CLIP-clip [38] extract feature importance by measuring the mutual information between each feature and the sensitive attribute, assuming each feature is independent.However, simply dropping features cannot maintain the dimensionality of the representation, which is crucial for using the embedding for generation tasks. For example, in CLIP-CAP [27] , the decoder (GPT-2 [30] ) takes the image representation from the CLIP ViT-B/32 [31] image encoder as input, and the input dimension of GPT-2 is fixed. Therefore, we must maintain the dimension of the representation after pruning to utilize the pre-trained decoder but approaches like filling with zerovalues or Gaussian noise may mislead the semantic meaning of the representation, as described in Figure 4 and ablation study in Section 5.3.To address this, SFID replaces important features with ambiguous features through Low Confidence Imputation (LCI). LCI is defined as the average of the features in low-confidence samples from the validation set as determined by RandomForest. RandomForest is known for its robustness against overfitting and can provide reliable confidence levels, identifying which samples are more ambiguous with low confidence. These low-confidence samples are likely to be 'hard to identify the sensitive attribute,' implying they are free of biased features.We visualize how the important features are correlated to social biases by showing GradCAM [33] visualizations, as presented in Figure 2 . For example, the more important features highlight human faces, while the least important features are correlated with the image background. SFID imputes In SFID, we utilize low-confidence samples to obtain imputation values that mute bias-related information within the embedding. However, in text-to-image generation, users may sometimes wish to specify the gender of the generated image through gender-specific prompts, such as "a photo of a man who works as a nurse." In such cases, rather than suppressing bias-related features, we impute features from samples that are classified with high confidence as belonging to the specified gender, using a RandomForest classifier, to retain the desired gender attributes. For text-to-image generation, we report results using both low-confidence (LC) and high-confidence (HC) imputation approaches.We extend our approaches to address more complex bias scenarios in VLMs, focusing on multiple sensitive attributes. Specifically, we conduct additional experiments that examine racial bias by considering more than two sensitive attributes, enabling us to capture a broader spectrum of bias patterns. Since RandomForest can accommodate multi-class classification, SFID is applicable in this context, as illustrated in Figure 2 , by employing multi-class classification with RandomForest.For training the attribute classifier, we use the FairFace dataset, which includes seven racial categories: East Asian, Indian, Black, White, Middle Eastern, Latino Hispanic, and Southeast Asian. A detailed analysis and the corresponding results on multiple sensitive attributes are provided in Appendix F.5 Experimental Result 
implementation detail: 
 We use the FairFace dataset [22] for image inputs and the Bias-in-Bios dataset [12] for text inputs as our debiasing datasets, denoted as X D . Each dataset is split into training and validation sets. A RandomForest classifier is applied to a 2D-shaped embedding representing all training samples, with low-confidence samples selected from the validation set. All hyperparameters and model settings for each baseline follow the default configurations provided in their respective open-source repositories. Detailed experimental settings, along with evaluation metrics and query datasets, are described in Section 3. We also adopt DeAR [34] , CLIP-clip [38] , and Prompt-Debias [11] as baseline comparison methods. A detailed discussion of these methods is provided in Appendix C.For a fair comparison, we conduct 10 experiments using different subsets and report the mean and standard deviation for the text-to-image retrieval task. For zero-shot classification and image captioning, we employ 1000 bootstrapping iterations to calculate the confidence intervals. For text-toimage generation, images are generated using 10 random seeds. The mismatch rates are reported as the mean and standard deviation over 10 runs, while the Skew for the Neutral prompt is reported as a single value based on 10 runs. Further analysis of this evaluation metric can be found in Appendix E. As shown in Table 2 , SFID effectively mitigates biases in CLIP RN50, CLIP ViT-B/32, and XVLM for multi-class zero-shot classification on the FACET dataset and text-to-image retrieval on the Flickr30K dataset. Specifically, the fairness metrics such as ∆DP in zero-shot classification and Skew@100 in text-to-image retrieval outperform existing debiasing methods, including DeAR, CLIP-clip, and Prompt-Debias without compromising performance metrics such as accuracy and recall.In Table 3 for the image captioning task, SFID consistently improves both the overall misclassification rate and the composite misclassification rate, outperforming other debiasing methods. This indicates that SFID not only reduces the likelihood of gender misclassification but also balances the misclassification rate across genders.As shown in Table 4 for text-to-image generation, SFID demonstrates improvements in both overall and composite mismatch rates. Notably, SFID with the high-confidence strategy (HC) shows a significant reduction in mismatch rates, though SFID with the low-confidence strategy (LC) also achieves a considerable level of improvement. This suggests that the debiased generative model adheres more closely to the intended gender specified by the user, rather than associating certain genders with professions in a biased manner, outperforming other methods. In particular, DeAR with SDXL tends to produce only one gender, resulting in a higher overall mismatch rate and an increase in Skew. Although there were improvements in Skew for neutral prompts, the score remains notably high, indicating that further refinements are necessary for this task.To verify the impact of the low confidence imputation and the hyperparameter τ , we conduct an ablation study using XVLM, one of the state-of-the-art foundation models. As shown in Table 5 , our low confidence imputation strategy demonstrates outstanding performance compared to zero and Gaussian imputation. Moreover, the hyperparameter τ shows optimal performance at τ = 0.7, thereby we set τ = 0.7 across the experiments. 
conclusion: 
 In conclusion, our study addresses the critical issue of bias in Vision-Language Models (VLMs) by introducing the Selective Feature Imputation for Debiasing (SFID) method. This approach effectively reduces biases across various downstream tasks, including zero-shot classification, text-toimage retrieval, image captioning, and text-to-image generation, without compromising performance. Additionally, SFID does not require extensive hyperparameter tuning or costly retraining, making it cost-efficient. By training a debiasing dataset separate from the test query set, SFID demonstrates its transferability and maintains zero-shot capability. Furthermore, SFID's ability to generalize across different VLM components, such as encoders and decoders, highlights its versatility in diverse multimodal contexts. Experimental results show that SFID outperforms existing debiasing methods, making it a versatile and efficient solution. This advancement paves the way for fairer and more reliable applications of VLMs in diverse multimodal scenarios, promoting both fairness and practicality in real-world applications. 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: The research primarily focuses on positive societal impacts and does not have any negative societal implications.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: The research utilizes publicly available datasets and pre-trained models, ensuring there is no risk of misuse. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: All code and datasets used in the research are properly credited, with the appropriate license (CC-BY 4.0) explicitly mentioned and respected. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL.• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: No new assets are introduced in this research.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: This research does not involve crowdsourcing or research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: This research does not involve crowdsourcing or research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: In SFID, the number of features to be pruned, denoted as k, is a critical hyperparameter. To determine an appropriate value for k, we analyzed the feature importance by plotting Figure 5 , which shows the sorted feature importance ranks. The plot reveals that the first few features have significantly higher importance, stabilizing around the top 100 features for all components. Therefore, we set k = 50. In some scenarios, embeddings may not be in a 2D shape, which poses challenges for further processing. For instance, in a decoder, outputs are tensors with shapes such as Z ∈ R N ×S×C for text decoder and Z ∈ R N ×C×H×W ,for image decoders, where N is the number of samples, S is the sequence length, C is the number of channels, H is the height of the feature, and W is the width of the feature. These non-2D shaped tensors are not suitable for extracting feature importance via RandomForest, which requires 2D data Z ∈ R N ×C . To address this, we transform the data into a 2D shape by averaging over the sequence length or applying global average pooling,Z ′ i,j = 1 S S k=1 Z i,k,j or Z ′ i,j = 1 H × W H k=1 W l=1 Z i,j,k,l .This averaging is applied solely for extracting feature importance indices Sand imputation values µ j .METEOR measures the balance between precision and recall of n-grams in generated captions, incorporating synonyms. Let P and R be the precision and recall of matches between the generated caption and ground truth, including exact, synonym, and paraphrase matches: METEOR = F mean • (1 -Pen) where F mean = 10•P •R R+9•P is a harmonic mean, and Pen = 0.5 × number of chunks number of matches 3 is a penalty term where a chunk is a set of contiguous words in the generated captions that are in the reference. SPICE [3] focuses on the semantic content of captions by comparing sets of propositional semantic tuples extracted from candidate and reference captions. SPICE is the F1 score of precision and recall between the tuples of generated captions and ground truth.As described in Section 1, two existing methods, DeAR [34] and CLIP-clip [38] , utilize debiasing datasets with gender labels and pre-trained models to debias embeddings. While DeAR is specifically designed for image encoders and CLIP-clip for both image and text encoders, their methodologies could potentially be extended to text and image decoders, as well as other encoders. In contrast, Prompt-Debias [11] is designed primarily for text encoders, using pre-defined text prompts to achieve debiasing. These three approaches are selected for comparison in our analysis.DeAR [34] involves a two-step training process. First, it trains an attribute classifier, h c using the frozen representation by minimizing the cross-entropy loss. Second, it trains an linear transformation h a for Z D to produce an additive feature to make a debiased representation Z a , i.e., Z a = h a (Z D ) + Z D . The objective function for optimizing h a arg minha λ 1 ∥Z a -Z D ∥ 2 2 + λ 2 max(Sof tmax(h c (Z a ))) -λ 3 CrossEntropy(h c (Z a ), y D ) ,indicating the need to maintain the representation's meaning while guiding the linear transformation h a to minimize the maximum probability of h c and maximize the cross-entropy loss of h c . The goal is for the additive feature to make the representation ambiguous, placing it near the attribute classifier's decision boundary. However, DeAR does not consider feature importance and applies adversarial training to all feature columns. Moreover, it requires several hyperparameters for the adversarial training, such as learning rate, weight decay, and the weights between loss functions (λ 1 , λ 2 , and λ 3 ). This complexity may lead to inefficiencies in adversarial training, which is sensitive to hyperparameter settings, resulting in less effective translation of the feature, as shown in Figure 6 .For DeAR training, all hyperparameters are set according to [34] .CLIP-clip [38] aims to identify features related to the sensitive attribute by measuring the mutual information between each column feature and the sensitive attribute. However, this approach assumes that each column independently contains meaningful information about the sensitive attribute, which can be overly restrictive. The dependency between feature maps in the neural networks' representation output is well-documented [5, 36] , supporting SFID's capability in more effective feature selection.Moreover, CLIP-clip is designed to clip the representation, reducing the dimension size of the embedding. The reduced embedding is not applicable as input for the decoder, which necessitates imputing the pruned features. Since CLIP-clip does not provide any suggestions for the imputed values, we adopted zero-value imputation across our experiments for CLIP-clip, where the number of pruned feature is set as k = 60, showing best accuracy-fairness trade-off described in Appendix D.On the other hand, a metric used in [11] measures gender discrepancy as follows:Discrepancy = N p,m |P| -0.5 2 + N p,f |P| -0.5 2 ,This metric can be used in a single generation, so it may produce a confidence interval when conducted on multiple seeds. However, this metric does not effectively reflect bias in text-to-image generation.For example, assume we have a set of four gender-dominated professions: nurse, dancer, doctor, engineer. If a biased text-to-image model always produces female images for nurse and dancer, and male images for doctor and engineer, the evaluation metric becomes 0, even though the bias is prevalent. This metric fails to demonstrate bias, even with a confidence interval over 10 runs. Therefore, our evaluation metric in Eq. ( 1), which computes Skew over 10 runs, more effectively reflects the bias in text-to-image generation, accounting for randomness.We extend our analysis to more complex bias scenarios in Vision-Language Models (VLMs), involving multiple sensitive attributes. Specifically, we conduct additional experiments that focus on racial bias, considering more than two sensitive attributes to capture a broader spectrum of bias patterns.Firstly, we adopted the FairFace dataset for training the attribute classifier, as it contains seven racial attributes: East Asian, Indian, Black, White, Middle Eastern, Latino Hispanic, and Southeast Asian.Given that RandomForest can handle multiple classes, SFID is also applicable in this context. During the evaluation stage for zero-shot classification, we used the FACET dataset, which contains 'skin tone' labels instead of race. We categorized the skin tone attributes into three categories: 'lighter,' 'middle,' and 'darker.' In this setting, the biased zero-shot classifier tends to produce higher accuracy by associating certain skin tones with specific professions (e.g., bartender with lighter skin, trumpeter with darker skin).To mitigate this bias, SFID demonstrates its effectiveness even though the attributes in the training set and test set do not exactly match, i.e. race in FairFace and skin tone in FACET. We adopted an evaluation metric inspired by [16] and [14] , which measures the maximum discrepancy across the sensitive attributes for each class. This metric is defined as follows:DP c = max i∈S max j∈S\{i} P (y = c | a = i) -P (y = c | a = j)where i, j ∈ S, c is a class, and S denotes the set of multiple sensitive attributes. We consider the mean and maximum values of DP c to evaluate the bias across the classes. By applying this metric, we can effectively measure and demonstrate the reduction in bias achieved by SFID, despite the differences in sensitive attributes between the training (debiasing) and test sets. The results in Table 6 show that even with limited data and racial bias present only in the image dataset, SFID can effectively mitigate the racial bias. We expect even more advanced results if we have access to race-profession related text datasets as well. Justification: The main claims are explicitly outlined in the abstract and introduction, as detailed in Section 1, particularly in the last two paragraphs. These sections clearly state the paper's contributions and scope. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.Justification: The research conducted in the paper conforms to the NeurIPS Code of Ethics.The study ensures scientific integrity through transparent methods and reproducibility, addresses societal impacts positively, and promotes fairness by mitigating biases.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 


Decision: The reviewers find the problem of debiasing of VLMs being studied in this paper to be timely and important, the proposed method to be effective, lightweight and easy to apply, and the experimental results to be convincing. The reviewers appreciate the fine-tuning free nature of the proposed method and the fact that it works across multiple tasks and model components.

The reviewers had raised some concerns, but the rebuttal successfully addressed most of them and all reviewers recommend acceptance. The authors are recommended to improve the final paper version by following the reviewer recommendations.

Reviews: review: paper_summary: paper_summary: The paper introduces a new method to reduce biases in VLMs, which works by using a random forest to identify the bias-related features in model representations and then imputes the values for those features with values from the low-confidence samples. The authors test it on tasks like zero-shot classification, image captioning, and text-to-image generation.
strength_weakness: strength_weakness: - The paper focuses primarily on gender bias without exploring how the method performs on more complex biases, or even biases on other attributes -- e.g. race or age attributes are provided in the FairFace dataset, it would be good to show how SFID can mitigate biases w.r.t these attributes. 
- The paper doesn't address potential limitations of using Random Forest for feature importance. There could be cases where complex, non-linear relationships between features and bias are missed by this approach.
- Insufficient comparison with a wider range of existing debiasing methods. The paper only compares SFID with DeAR and CLIP-clip, and is potentially missing out on how it compares to other methods e.g. https://arxiv.org/pdf/2302.00070.
- Notation in Section 3 could be simplified (especially 3.3)
questions: questions: - In line 188 the authors mention that “For a frozen component in VLMs g, whether it is an encoder or decoder, or processes image or text, we obtain the frozen representations”. Can the authors provide a breakdown of the computation cost when the encoder or the decoder is a language model that is large? 
- Would it be possible to generate embedding translation (figure 4) for SFID as well?
limitations: limitations: The authors have provided some limitations in section 5.2. The paper could benefit from an extended limitations section.
paper_summary: paper_summary: this paper introduces Selective Feature Imputation for Debiasing (SFID), which integrates feature pruning and low confidence imputation (LCI) to effectively reduce biases in VLMs.
strength_weakness: strength_weakness: 1.the method is simple, utilizing Random Forest, which is pretty well-known in the machine learning community. However, the novelty is a little limited.
2. how to define the \Delta DP for attributes with more than two levels? For instance, race has white, asian, black, etc.
3. what is the principle for choosing the number of important features? will it affect the performance?
questions: questions: About the working mechanism of the proposed method, the authors may dig deep how the identified features is correlated to the social biases, maybe with visualization tools.
limitations: limitations: see weaknesses and questions
paper_summary: paper_summary: The authors propose a method for debiasing the representations of Vision-Language Models (VLMs) that can be applied at various layers of the image and text encoders/decoders, and can be used for a variety of downstream tasks such as image generation, 0-shot classification, text to image retrieval, and image captioning.
strength_weakness: strength_weakness: **Minor issues:**

- The writing could be improved in parts. There are minor grammatical errors in parts, such as "For instance, Hirota et al. [18], Zhao et al. [42] investigated bias in image captioning, where specific genders or races are disproportionately represented **leading to generate** biased caption." I suggest another editing pass or two.
- I find the structure of section 3 "Bias Analysis in VLMs" is a little odd. It mostly reads like background/preliminary material, but it also contains some experimental setup details (such as in line 165). This seems off, as this is before the proposed method section. I suggest moving these experimental details to section 5, with the rest of the experimental setup. 
- Figure 4 shows that DeAR fails to produce bias-free representations. Including a figure likes this naturally makes the reader want to see a figure showing that the proposed approach **does** succeed here. It feels incomplete as-is. 

**More significant issues:**

- The experimental results don't include confidence intervals. The authors justify this by pointing out that the proposed approach is deterministic. However, CIs still can and should be calculated by an approach such as bootstrapping the test set. 
- Relatedly, the text to image generation experiment is **not** deterministic if the text2img generator's random seed is changed, so CIs should definitely be reported here. 
- The compared methods are limited to only 2 other debiasing approaches and the baseline model. 
- The proposed approach of 1) finding features most associated with the biased attributes and 2) replacing these features with an imputed value from low-confidence samples is, I think, very similar in practice to Chuang et al.'s approach [1] in "Debiasing Vision-Language Models via Biased Prompts". Chuang modifies embeddings by projecting them onto the subspace defined by the linear direction most associated with the biased attribute(s). Intuitively, the direction most associated with the biased attribute should be similar to the direction defined by the features identified in step 1) of the proposed approach. Not that I am **not** saying that this similarity alone is a limitation; I think the proposed approach is different enough to still have sufficient novelty. However, I think that Chuang's approach really needs to be compared against (for all experiments except maybe image captioning), in order to verify that these features are either finding a different direction in the embedding space or that the low confidence imputation performs better than just projecting onto the bias subspace. 

[1] Chuang, Ching-Yao, et al. "Debiasing vision-language models via biased prompts." arXiv preprint arXiv:2302.00070 (2023).
questions: questions: - Is the improvement of the proposed approach over the compared methods significant when CI are considered, where these CI's are obtained from bootstrapping or some other way?
- How does the proposed approach perform when compared against the method proposed in "Debiasing vision-language models via biased prompts."?
limitations: limitations: I think the discussion on limitations could be improved. The authors point to Section ***5.2 Result Analysis** as their discussion of limitations. 5.2 covers the results of their experiments, and does include some discussion on how the experimental results on image generation point to room for improvement. However, I feel like a discussion of the limitations would be greatly improved by dedicating a separate section/subsection to it. I would expect the authors to discuss more fundamental limitations, rather than just experimental performance. For instance, I would include details on how the method assumes access to a validation set with labels for bias/protected attributes (e.g. a dataset with race/gender/etc labels).

Comments: title: Please read the author rebuttal, other reviews and respond to the authors NOW!
comment: Thanks for the comprehensive response. I have raised my score accordingly.
I thank the authors for their comprehensive rebuttal, which has addressed my concerns. So I have raised my score accordingly. Please include the important analysis and definition in the main content.
Dear Reviewers,

Thanks to those of you who already responded to the authors acknowledging the rebuttal and asking follow-up questions if any.

Those who have not responded yet, please do the following ASAP: thoroughly read the rebuttal, the other reviews and respond to the authors about whether all your questions / concerns have been addressed or not. If not, please elaborate on which questions / concerns are still not addressed so that the authors have fair chance of addressing them before the author-reviewer discussion period ends in ~41 hours from now (August 13th, 11:59pm AoE).

Your AC
I thank the authors for their comprehensive rebuttal. I have raised my score accordingly.