BendVLM: Test-Time Debiasing of Vision-Language Embeddings
Authors: Walter Gerych
Haoran Zhang
Kimia Hamidieh
Eileen Pan
Maanas Sharma
Thomas Hartvigsen
Marzyeh Ghassemi
Venue: NeurIPS 2024 poster (2024)
Keywords: vision language models
embedding models
multimodal models
debias
fairness
Primary area: fairness

Abstract: Vision-language (VL) embedding models have been shown to encode biases present in their training data, such as societal biases that prescribe negative characteristics to members of various racial and gender identities. Due to their wide-spread adoption for various tasks ranging from few-shot classification to text-guided image generation, debiasing VL models is crucial. Debiasing approaches that fine-tune the VL model often suffer from catastrophic forgetting. On the other hand, fine-tuning-free methods typically utilize a ``one-size-fits-all" approach that assumes that correlation with the spurious attribute can be explained using a single linear direction across all possible inputs. In this work, we propose a nonlinear, fine-tuning-free approach for VL embedding model debiasing that tailors the debiasing operation to each unique input. This allows for a more flexible debiasing approach. Additionally, we do not require knowledge of the set of inputs a priori to inference time, making our method more appropriate for online tasks such as retrieval and text guided image generation.

BendVLM: Test-Time Debiasing of Vision-Language Embeddings
Walter Gerych'
Haoran Zhang'
Kimia Hamidieh'
Eileen Pan'
Maanas Sharma
Thomas Hartvigsen
Marzyeh Ghassemi*
'MIT, University Virginia
(ugerych, haoranz, hamidieh, eileenp, maanas, mghassem) @min edu,
hartwigsendvirginia. edu
Abstract
Vision-language model (VI.M) embeddings have been shown to encode biases present in their training data, such societal biases that prescribe negative charac- teristics to members of various racial and gender identities. VLMs are being quickly adopted for variety of tasks ranging from few-shot classification to text-guided im- age generation, making debiasing VLM embeddings crucial. Debiasing approaches that fine-tune the VLM often suffer from catastrophic forgetting. On the other hand, fine-tunne tuninu--nieeee free methods typically utilize a "one-size feizzefits-ar atls-all" approach that assumes that correlation with the spurious attribute be explained using single linear direction across all possible inputs. this work, we propose BEND- VLM, a nonlinear, tin-ttunigg-fre approach for VL. embedding debiasing that tailors the debiasing operation to each unique input. This allows for a more flexible debiasing approach. Additionally, to inference time, making our method more appropriate for online, open-set tasks such as retrieval and text guided image generation
do not require knowledge of the set of inputs priori
Introduction
Background. Pretrained foundation Vision-language models (VLMs) such as CLIP [33,, BLIP [22). and LLaVA 25 have seen wide adoption for tasks like image retrieval [1] zero and few-shot text-guided image generation
classification 33
322 and facial recognition [58 But VL models
also encode societal biases 5
27
43
49
53 As more and more systems rely on CLIP. the encoded
representational harm 3 15 52 can lead to allocative harm B4
46 14
51 16. 29 such as
Black individuals being three times more likely to be misclassified into nonhuman category by computer vision systems D.
State of the art. Debiasing VLMS an active area of research 6 approach is finetuning the embedding models to remove spurious correlations 55
20
19
50
28
One common
2
42 However,
finetuning often decreases accuracy and generalizability of foundation models [3 a significant
drawback these models are commonly used for zero-shot tasks. Most existing finetuning-
methods learn debiasing transformations of the initial text embeddings, but typically use one one-size- size-fitess fits- all linear debiasing functions that apply the same fixed transformation to every input 6 10 55).
While recent work has explored nonlinear VLMs [T], their method assumes access to the set of classes test-time, requiring the debiasing training pipeline to be rerun if query for new class is
code:
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

made. This is a major limitation in practice because many tasks VLMs are used for are often naturally open-set, where the classes be evaluated for at test-time are unknown prior to inference.
Problem Definition. We study online, open-set debiasing for VLM embeddings. In this setup, we only have access to VLM,, along with a single-modal image dataset. This image dataset is only for the purpose of "training", and is not the dataset that the downstream task will work on. We assume that this dataset, which we call the reference dataset, has labels for the protected attribute(s) of interest. During test-time, we receive online input queries one at a time. These queries are also open-set, meaning that the classes or concepts they refer are not known to us beforehand. For instance, the query may be "a photo of a nurse", but we do not have knowledge that nurse is a potential class of interest before receiving the query. Our goal is to debias the query embedding from the VLM in such as way that it does not more strongly associate the query embedding with any protected attribute value over another. For instance, the embedding for "a photo should not be more associated with images of women than with men.
nurse" "
Challenges. Online, open-set VLM debiasing is challenging task. First, we must overcome catastrophic forgetting- solution that debiases the embeddings, but degrades performance. Second, the interaction between protected attributes and query classes may be nonlinear and instance- dependent. For example, the transformation required to remove the gender bias from the embedding of "nurse" " is likely not the same as the one untangle gender bias associated with the embedding of "handyman". Third, queries from open-ser classes means that our approach must be flexible enough remove the association of protected attributes from classes unknown prior to inference time. Lastly, online settings demand computational efficiency and thus rule out refitting the debiasing component for each now class or query.
Proposed approach. We propose Bias Elimination with Nonlinear Debiasing of Vision Language Models (BEND-VLM), test-time VLM debiasing method that leaves the VLM's weights unchanged, being efficient enough for online streaming queries. By using the easy-to-get pre-debiasing reference dataset with protected attributes, BEND-VLM, allows for unsupervised test-time debiasing. On high level, BEND- VLM consists of two main parts:
First, given online query. generate augmented queries that introduce protected attribute infor- mation. For example, given "a photo of a nurse" we generate "a photo of a (ATTRIBUTE) nurse". filling in (ATTRIBUTE) with male / female nonbiinay for gender debiasing, for instance. We get these augmented queries from a small language model, and use them to find the directions in the embedding space for that specific query that are most associated with the protected attribute. Given these directions, we project the embedding such that it is orthogonal to the protected attribute dimension, resulting in the first-stage debiased representation.
For the second step. we make use of the reference image dataset. We find the images in this dataset that are most associated with the query, and then subset them by protected attribute value. We find an updated, debiased query representation by solving a constrained optimization equation with the goal of finding an embedding with minimal distance to the first-stage debiased representation while being equally similar to the example images for each attribute value. For instance, we find an embedding that is equally similar to the nearest images for each gender. The resulting embedding will have little to no excess association with any of the debiased protected attribute values over any other. The output can then be passed the downstream task. Contributions.
We introduce BEND- BEND-VLM, novel test-time VLM debiasing approach that does not require finetuning.
We propose technique for finding local attribute subspaces specific each query on-the-fly.
We introduce a novel method for equalization by using a reference image dataset.
We experimentally evaluate for classification, retrieval, and image captioning settings, showing BEND-V consistently outperforms the compared approaches.
2 Problem Definition
Let (m, t, c, a) be an (image, text, class, attribute) tuple distributed according to Par x Pr x Pc PA, joint distribution over images, texts, classes, and attributes. Using the running example of nurses,

---

Images Embedding
Query Embedding
Gender-augmen augmented Query Embedding
After Initial Embedding
Step I: Textual Debiasing
Step 2: Distance Debiasing
farther
"female doctor"
"female docter"
Debias
Step
Classified
Classified Female
Classified Female
equai
Female
Debias Step
closer
doctor"
ddccoor"
"doctorr
"male
"male
Classified
Classified Male
Classified Male
Male
doctor"
doctor"
d(
d(
10
0.10
d(
d(
v0
|||
0.05
)|
0.00
Figure 1: Overview of our two-step BEND- VLM method. In this example, the initial query embedding of doctor is more strongly associated with males, and the CCF distance 0.10. After performing debiasing step I, Orthogonalizing the Embedding, the embedding is modified to remove bias along the gender direction defined by "male doctor" and "female doctor". This still results in a CCF distance of 0.05. We then perform the second debiasing step, where the query embedding is again modified be equidistant the relevant male and female images. The final representation achieves the optimal distance of 0.00.
realization of m could be an image of nurse, the "a photo of a nurse", the class nurse, and protected attribute such as gender. Importantly, we do not assume that C, the support of Pc, is known. This means we do know what classes the user will query for during inference, and do not have access to training set with these class labels.
d represent the text embedding model (e.g., CLIP's image encoder) and f.M Mi Rd represent the image encoder, where T and MI are the text and image domain, respectively. We will use fo = { f when referring to the VL model in general, rather than its modality-speciffe encoders. fo is used to obtain d(fe' (m) f (t)). where d(...) a distance metric such as cosine distance. In practice, these (image, text) distance scores used for zero-shot classification image retrieval.
Let f
Let te E T be textual instance relating to class For instance, class could be nurse and to "a picture of a nurse". Then, our goal to obtain a text embedding x E _2d that is Class Conditionally Fair.
Definition (Class Conditionally Fair (CCF)). A text embedding 2 is Class Conditionally Fair for embedding model fe. class c, and metric for all a, aj E A the following holds:
Emjas,e(d(( (m'), =;)] = Emm'aay, [d( (m') mm)),
Intuitively, text embedding is CCF for class if the expected similarity between the text representa- tion and relevant image embeddings - image embeddings that are also associated with class c is independent of the protected attribute value For instance, embedding of the query "a picture of a nurse" is CCF if its expected similarity score for pictures female nurses is equal to the expected similarity score for male nurses.
We also define Class Conditionally Fair Distance as a measure from how far off an embedding is from being CCF:
Definition 2 (Class Conditionally Fair Distance). The Class Conditionally Fair Distance for lext embedding z. class C, and metric d is given by:
docr(2e, c) = =1IEmmaac a [d(FM (m'), ze)] - e [d(f. M (m'), zc)]
3

---

The CCF distance of be is 0 if and only if ze is CCF. In practice, we can't tt exactly compute the expectations in the CCF distance definition. Instead, these expectations can be replaced with the average distances from relevant samples in the evaluation dataset.
Reference and Target Datasets. In practice, we assume that we have a dataset Dret {(m,, 1= consisting of N images with labeled attributes. For instance, Dier could be a dataset of pictures of people with corresponding gender, race, or age label We focus on both the image retrieval and zero-shot classification setting. This reference dataset will be used to obtain debiased text embedding, as we describe in detail in the following section. We refer to the downstream dataset
to be used in retrieval or shoo applications as the target dataset Durget {m; |= 1/ not available prior to inference.
Niarget Deurget is
For retrieval, we assume that Drarget is an unlabeled dataset of images, such that we want to retrieve images from this dataset that relate to streaming, open-set queries. For instance, the queries can be free-form text searches coming from search engine user. In this open-set scenario the set of classes C is unknown we do not know what classes users will search for a priori.
For zero-shot classification, we likewise focus on the streaming, open-set scenario. Images from D urgget will be compared against a set of texts (tco, tcl, ... ,tck } for the purpose of classification, where this set of texts relates to classes C1, C2..... ,. E C, where C is unknown to us and potentially variable. For instance, a user may first wish to obtain zero- shot predictions of hair color of the portraits in Durget. and later wish to obtain predictions of whether the individuals have eyeglasses.
In both settings, we make the simplifying assumption that each user query t. does not explicitly reference the protected attribute of interest. For instance, the query is "a picture of nurse", not "a picture
a male nurse" " and thus it is desirable for the query embedding not be more associated with a particular gender. In the case where the query does contain explicit reference a "a picture of a male nurse"
abstain from debiasing by using a language model to filter out these queries, or by checking for explicit attribute terms]
straightforward
3 Methodology
On a high level, our BEND-V approach consists a two-phase debiasing pipeline. We perform an initial debiasing pass by first employing the classic approach of orthogonalizing (t) to the attribute subspace v [2 91. However, unlike most prior works, we do not assume that the attribute subspace is globally constant for all queries; it may be the case that the direction in the embedding space corresponding to gender that differentiates "a picture of a male nurse" from "a picture of female nurse" " may not be equivalent the gender direction between "a picture of baby boy" and "a picture of a baby girl". We find these local attribute subspaces using ATTRIBUTEAUGMENT module obtain attribute augmented versions of t.. After this first phase, we are left with the partially-debiased embedding z'c
Our second and final debiasing pass consists of equalizing the distances between the embedding and relevant images from the reference dataset Dref belonging to each attribute class. We obtain the final debiased embedding 2. through an analytical solution to a constrained optimization equation.
3.1 Step 1: Making The Embedding Orthogonal To Local Attribute Subspace
Orthogonalizing text embeddings with respect to an attribute subspace, such as setting embedding dimensions corresponding to gender or race equal to zero, is classic approach used for standard text embeddings (24] D and has recently shown promise debiasing VL models [10. Whereas existing approaches typically find single attribute subspace for instances, we find local attribute subspaces in addition to the global subspace.
Let te be the initial text query coming in to the system. We then obtain tc.,, for all a, E A. For instance, if a refers to gender and to "a picture a nurse", then we would ob- tain "a picture of a male nurse" and "a picture of a female nurse" for tc and
In practical application, these protected attributes could noisy labels assigned by an attribute predictor. For instance, gender labels could be ohtained hy using CL.IP for zero-shot gender prediction. Pegg. could filter for gender with GenderspacY; httpp:://gthhu..com./ http:///ww..oooosiaaaacccceeeeee

---

fe.d//mmter respectively. We draw each tc.a, from our ATTRIBCTEAUGMENT module: (tc. HEEA ATTRIBUTEAUGMENTT..
A). In practice, we use an L.M to instantiate ATTRIBUTEEAUGMENT In lower resource setting, ATTRIBUTEAUGMENT could feasibly be implemented through simpler text processing techniques to identify the subject of the query and insert corresponding attribute strings before the subject; e.g. inserting "male" e" and " "female" before the subject for gender debiasing.
Let A be a matrix whose columns are fg (tc,a.) fe (tc) for = - |A|. To combat potential noise from estimating the local attribute subspace, additonally include generic attribute text embeddings into the columns of A as well. For instance, for gender debiasing we include the embeddings of 'a picture of man" and "a picture of woman". We then obtain the initial debiased emhedding z.. as:
where == - A(ATA)-1A* the orthogonal projection matrix of 10
Importantly, despite z being orthogonal to the local attribute subspace it is not necessarily equally similar the image embeddings of relevant instances when conditioned on the "debiased" attribute.
Lemma 1 (Orthogonalization does not yield Class Conditional Fairness.). The following does not hold in general:
Em [d(fo (m'), =()] , [e [d[fo' (m'), (m'),²²))
where we see that step 1 does not result in significantly improved CCF distances. To mitigate this, we propose a second debiasing step.
We show an example of this in Figure L
3.2 Step 2: Using Reference Images To Equalizing the Text Embedding
In this second stage, we equalize the distances between the images in Dier and the debiased embedding z', with the goal of making relevant images from each attribute group equally similar the text embedding. Let Drer(a; (a; e) be images in the reference dataset that are associated with attribute class a, and class c. We want find the embedding 2 that satisfies the following set of conditions C:
C = {
EmyeDiaton Durla,)) d(fa (m))z")) EmeeDaras.o Darla, d(fa (m2, z())
Daet(a;; c)l
|Der (1,, e)
i=1-|A
These constraints say that the average distance between relevant image embeddings should be equal for all attribute value splits. For example, the distance between the embedding of "a picture of nurse" and relevant male images should match the distance between the embedding and relevant female images.
Note that since do not assume access to context labels for D it is not immediately obvious how to obtain each Drer (a, c). Instead, Dar c) is by selecting n images with attribute value a, that most similar to the query embedding z!.- The value of could be found using change-point detection, such that n is the value where the elhow in the plot of similarity over indexes sorted by similarity score (38). A less sophisticated approach but simple chose n as a hyperparameter, and use the same value for each attribute and query.
find works well in practice is to
Finding any embedding that satisfies C is not enough, since want to ensure that the debiased embedding does not lose information unrelated to the protected attribute a. This means we want to find a debiased embedding with minimal distance to the previous embedding. We want to find x! that minimizes distance to the first-pass debiased zz::
Linitall = d(z x/c
We thus find x by solving the following constrained optimization equation:
z arg min Lintiat, under the set of constraints C.
Equation has simple analytical solution for the binary attribute case, when d(.,-) is cosine distance and each embedding has unit norm length.
5

---

Lemma 2. The value of x that minimizes the distance from the initial embedding za while satisfying the smage-embeddee fairness constraint is:
z/ Ap(az, c) + )
2 )p(a2, + c cul
where À is given by:
M(a1,)) z u(a2.c))
A 21( c) ula1, c) ul c) pl c) pla1, c) ulai,
and pla.,, c) [Dre/(a..c] EmjeD mj the average embedding of Dres(as,
As the requirement that the embeddings have unit norm length simplifies the analytical solution, we add in this norm constraint [||=1lle = 1 to the set C. In the case where the protected attribute is not binary, 2 be found using constrained optimization solver 48
After obtaining the result of this final debiasing step. our modified embedding can then be passed along to downstream task such retrieval zero-shot classification target dataset D uarget or used to condition another model such as a text to image generator.
Experiments
Datasets. We compare our BEND- existing debiasing approaches the FAIRFACE [8 CELEBA [26]. and UTKFACE [57] datasets. Each dataset contains pictures of people. CELEBA has gender annotations, while FAIRFACE and UTKFACE have both gender and race labels.
Models. We evaluate the ability of the debiasing approaches to improve the performance of the CLIP-ViT-Base-attch (CLIP-ViT-B-PI6) and CLLP--Vi-LarrgePaacch (CLIP-ViT-L-P14)) VLMs. For image captioning, we use ClipCap (30] pretrained on Conceptual Captions 411. which uses a ViT-B/32 architecture. We use Beerral-B-Instructt
117
for our ATTRIBUTEAUGMEN module.
Compared Methods.
We compare BEND- VLM against the following debiasing methods:
Baseline CLIP 33 is simply the original CLIP model (e.g. ViT-B-P16 or ViT-L-P14)
without any debiasing steps. This acts as our baseline.
Orthogonal Projection (Orth-Proj.) Proj.) [IO] debiases the query embedding by making the embedding orthogonal to the global spurious attribute subspace (e.g. making the embedding orthogonal to the directions the embedding space most correlated with gender).
Orthogonal Calibration (Orth-Cal.) [10] likewise makes the embedding orthogonal to the global spurious attribute subspace, but introduces an additional regularization term to encourage attribute-aue augmented versions of the query to be close together after projection.
DeIiasCLIP 6 finetunes CLIP model remove spurious attribute bias. The authors have released the weights for DebiasCLIP trained to do gender debiasing on CLIP-ViT-B- ViT-B-P16, but have made their training code available. This means we compare against this method only when evaluating on experiments that use CLT--PiT---P1.. Note that while the released DebiasCLIP model was trained for gender debiasing, also include in evaluations for race debiasing but do not expect it to be competitive in these settings.
Implementation details. We do 50/50 split of each dataset for the reference and target datasets. We additionally create 5 folds for the target dataset so that we can compute confidence intervals for all methods. We chose n 100 when selecting the n most relevant images for computing each Dier (, c) (see Section 3.2 We use the default value of X = 1000 for Orth-Cal. and Orth-Proj. 's main hyperparameter. During retrieval, we always sample 500 images from the target dataset. Our and
reference and target datasets are drawn from the pre-established training split of each dataset.
Evaluation metrics. We measure K K[[Pa||Pal the KL divergence between the attribute prior Pa (c.g. the true distribution of genders in the target dataset) and Pa. the empirical distribution of

---

Worst Group Zero Shot AUC. ROC MaxSkew
Worst Group Zero Shot AUC ROC MaxSkew CCuP-VITLaarge-Pathh...
Worst Group Zero Zero-Shot Shot AUC ROC
(CLIP-IT-Base-Pacch11)
Worst Group AUC ROC
Better Classification Perform performance ance
Better Classification Perform Performant
1.3.
Orth-Cal.
Orth-Proj.
10-
BEND-VLM Orth-Proj.
(Ours)
BEND-VLM (Ours)
Baseline CLIP
Orth-Cal.
Baseline CLIP
0.8-
DebiasCLIP
64-
5.30
"8.50
0:35
0:20
0.30
0.55
0.40
0.40
0.50
Less Blas
Average MaxSkew
Average MaxSkew
Figure 2: Our approach increases accuracy while decreasing bias.
attribute labels for the set of images retrieved from the target dataset for a given query. Intuitively, the query does not rely on the spurious attribute when computing similarity, then the instances retrieved (e.g. the most similar instances) should result in empirical attribute distribution that the spurious attribute. For instance, a dataset contains 40%
matches the overall distribution males and 60% females, then if we sample independently of gender we should retrieve roughly 40% male es and 60% females. We also report the M MarSkew between the attribute prior and empirical retrieved distribution, MazSkew = loglaa (a,) (a.)).
For zero-shot classification, we compute the AUC ROC for each group using the similarity between the query and images from each group the retrieval set as the score. We then report Worst Group AUC ROC: mina,AUe (1 d(m z | [c5];=)) where d(---) cosine distance and 1 d( is cosine similarity. Worst Group AUC ROC tells us how useful the similarity score the text embedding is for zero-shot classification for members of the most disadvantaged group.
Queries sets. Since CELEBA has class labels for hair color, we use set of queries relating to this which we refer to as HAIRCOLOR so that we can mea- sure zero-shot classification performance via Worst Group AUC. HAIRCOLOR is the set {"A photo of celebrity with {COLOR) hair"), for COLOR We also use the query set STEREOTYPES, a set negative words such as "delinquent" and "terrorist" taken from the So-B-IT VLM auditing taxonomy (15), which is known to contain race and gender bias. Each our queries is given in the appendix.
(blond, black, brown,
4.1 Optimizing Accuracy And Minimzing Fairness
We study the effect debiasing has on accuracy through Worst Group AUC ROC as well as the KL divergence and MaxSkew bias metrics. We use CELEBA since it has class labels for HAIRCOLOR.
Figure 2 shows Worst Group AUC vs MaxSkew. The ideal method would be in the top left of the plot, indicating high accuracy and low bias. Our BEND- VLM method is close to this ideal region. We increase Worst Group AUC over the baseline, roughly matching the AUC performance of Orth-Proj. and Orth-Cal. while having significantly less bias than them. DebIasCLIP has a better MaxSkew than Orth-Proj. and Orth-Cal the baseline. We include additional results for this experiment in Section A. in the appendix; see T for results for this same setting, along with the KI divergence metric. We clearly that BEND- VLM consistently has significantly better bias scores than all compared method, while having negligibly worse AUC than the next method and significantly better AUC than the baseline.
but still worse than BEND- VLM while decreasing AUC compared to
4.2 Mitigating STEREOTYPE Bias
We evaluate our method removing the association the Stereotype words have to race and gender. The results for UTKFACE, FAIRFACE, CELEBA are shown Tables [ and respectivve.. We again see that BEND- VLM consistently has less bias than the compared methods in all the

---

scenarios we evaluated. Notably, the other debiasing techniques generally improve over the baseline but sometimes have worse MaxSkew KL Divergence which is never observed for our approach.
Table I: Debiasing the UTKFACE dataset with respect to gender and race for STEREOTYRE YPE queries.
CLIP-ViT-B-PI6
CLIP-ViT-L-P14
Attribute
Method
KL Div.
MaxSkew.
KL Div.,.
MaxSkew.
Race
Baseline CLIP
0.114 + 0.003
0,451 + 0.004
0.107 + 0.005
0,437 + 0.005
Race
Orth-Proj.
0,259 + 0.003
0.525 + 0.004
0.182 = 0.005
0.484 0.005
Race
Orth-Cal.
0.251 + 0.002
0.526 + 0.003
0.196 = 0.003
0.560 + 0.006
Race
DebiasCLIP
0.158 + 0.004
0,434 + 0.003
Race
BEND- VLN
0.041 + 0.002
0.371 + 0.015
0.047 = 0.002
0.367 0.017
Gender
Baseline CLIP
0.120 + 0.005
0.308 0.004
0.029 = 0.001
0.166 0.003
Gender
Orth-Proj.
0.191 + 0.003
0.384 + 0.003
0.043 = 0.004
0.200 + 0.010
Gender
Orth-Cal.
0.254 + 0.003
0.447 0.003
0.030 = 0.001
0.166 0.005
Gender
DebiasCLIP
0.091 + 0.002
0,263 1 0.002
Gender
BEND-V WLD
0.008 + 0.000
0.097 + 0.004
0.004 = 0.000
0.067 + 0.002
Table 2:
Debiasing the FAIRFACE dataset with respect to gender and race for STEREOTYPE
queries.
CLIP-VIT-B-P16
CLIP-ViT-L-P14
Attribute
Method
KL Div.
MaxSkew
KL Div.
MaxSkew]
Race
Baseline CLIP
0.234 + 0.002
0.808 + 0.005
0.223 = 0.003
0.772 + 0.006
Race
Orth-Proj.
0.305 + 0.003
0.808 + 0.009
0.197 + 0.003
0.744 0.009
Race
Orth-Cal.
0.292 + 0.003
0.797 = 0.007
0.209 = 0.001
0.717 = 0.007
Race
BEND- BEND-VLM
0.084 + 0.002
0.553 + 0.009
0.069 + 0.001
0.462 + 0.009
Gender
Baseline CLIIP
0.133 + 0.002
0.338 + 0.002
0.094 + 0.002
0.300 + 0.004
Gender
Orth-Proj.
0.340 + 0.003
0.520 0.001
0.033 = 0.001
0.155 0.004
Gender
Orth-Cal.
0.426 + 0.002
0.606 + 0.001
0.041 = 0.001
0.166 + 0.002
Gender
BEND- WLN
0.006 + 0.00)
0.080 + 0.002
0.006 + 0.001
0.086 0.003
Table 3: Debiasing the CELEBA dataset with respect to gender for STEREOTYPE queries. We do evaluate race on CELEBA as this dataset lacks race annotations.
CLIP-ViT-B-P16
CLIP-ViT-L-P14
Attribute
Method
KL Div.
MaxSkew
KL Div.
MaxSkew.
Gender
Baseline CLIP
0.436 + 0.010
0.749 + 0.006
0.335 = 0.002
0.702 + 0.003
Gender
Orth-Proj.
0.106 1 0.002
0.284 0.003
0.059 = 0.001
0.291 0.005
Gender
Orth-Cal.
0.133 + 0.005
0.296 + 0.004
0.041 = 0.001
0.223 + 0.004
Gender
DebiasCLIP
0.322 + 0.007
0.637 + 0.007
Gender
BEND-VLLE VLM
0.014 + 0.001
0.139 + 0.008
0.026 = 0.001
0.217 + 0.005
4.3 Intersecrtional Debiasing
We have conducted a new experiment where debias FairFace with respect gender for HairColor queries, but evaluate on race. We do not expect to see improvements with respect to racial bias after gender debiasing for any method. Table racial bias goes up for all debiasing methods after gender debiasing. This reflects a known, frustrating "Whac-A-Moles issue where debiasing for one attribute often increases the bias of another attribute
23 Interestingly. we not see racial bias increase when performing only Step 2 of the Bend- VLM debiasing, indicating that this short cut issue most strongly affected by the orthogonalization operation performed Step The other debiasing methods also perform similar orthogonalization step and likewise experience this shorteut problem.

---

Table Debiasing FAIRFACE with respect to HAIRCOLOR queries with respect to gender, but evaluated on race.
Method
KL Divergence +
MaxSkew +
Baseline CLIP
0.606 + 0.043
0.155 0.016
Orth-Proj.
0.826 + 0.020
0.211 0.014
Orth-Cal.
0.877 + 0.021
0.226 0.005
Bend- VLM (Without Step 1)
0.594 + 0.074
0.146 0.029
Bend- VLM (Without Step 2)
0.873 + 0.024
0.223 0.006
Bend- VLM (Full Method)
0.837 + 0.035
0.193 0.024
4.4 Debiasing Image Captioning
In this experiment, evaluate the effect of BEND- VLM on debiasing automatic image captioning. We study ClipCap [30] (ViT-B/32 vision encoder, pretrained on Conceptual Captions [41)), as is one of the few captioning methods which takes in only the final layer embedding vector, opposed to BLIP 12 or LLaVA [25] which take in the sequence embeddings from the VIT.
We hand picked 20 images that we observed to have significantly negative or harmful captions generated from the Baseline CLIP embeddings, After debiasing with BEND VLM, we performed manual inspection and determined that 6 out of the 20 had less harmful captions after debiasing, had increased harm, and 11 were equal to the original captions.
Next. we randomly sample 1600 images from FAIRFACE's validation set that result in captions containg any of the following negative words: [ "abandoned", "murder", "homeless", "accuse", "kill", "anime", "arrest", "surprised", "blood", "shot", "pregnant", "intoxicat", "charged", "bad day", "permanently surprised", "bandage", "hit", "wilful", "no idea", "prison", "abuse" "attack" ]. We then perform automated sentiment analysis using CLIP. Table E shows that BEND- VLM decreases the average negative sentiment per race, and makes this average more equal between the races.
Table 5: Average negative sentiment scores for the generated FAIRFACK captions, Lower is better.
White
East Asian
Latino Hispanic
Southeast
Black
Indian
Middle Eastem
Max Disparity
Baseline CLIP
0.640
0.495 0.290
.568 0.360
0.534 0.321
0.525
0.656 0.385
0.624 0.355
0.161
BEND VL.M
0.355
0.309
0.095
Limitations and Broader Impact
BEND- VLM requires a reference dataset with protected attribute annotations, which not feasible for every scenario. In our current implementation, our ATTRIBUTESWA WAPP module requires the use of a relatively small 7B LLM. This could still incur too much computational overhead for very resource-constrainee settings. Additionally, binary gender labels, but there is a large population of people who don' identify that way. Moreover, the race and gender labels are not from self- idnttifitttiinnnn. meaning they only noisy signal for identity. We believe that our method overall takes step towards understanding and mitigating biases, and can still be directly extended to support more nuanced solution to the extreme challenges of mitigating social biases.
evaluation datasets not perfect. They contain only
Related Works
Biases in Vision-Language Models,
Vision-Language models have become increasingly However, these models are known to suffer from
widespread in recent years B3
35
37
36

---

spurious correlations [55] and can be biased towards certain races and genders 8 Studies have shown that biases in these models can stem from the datasets they are trained on. For example, Agarwal et al. []] found that the CLIP model associates "white" text labels less accurately with white individuals than with individuals from other racial groups, and images of people labeled as Black are more likely to be mislabeled animals. Additionally, Dehouche [12] identified gender bias in CLIP when prompted with gender-neutral text, and Wolfe et al. [53] noted that multiracial individuals are more likely to be assigned minority racial labels. The biases embedded in these models reflect the biases present in the training data, which often include offensive and stereotypical 39].
content
Debiasing Visgon-La Models. Recent advancements in debiasing vision, language. and vision-languee models have led to various methods for mitigating biases, ranging from data augmen- tation and balancing 7 model-level adjustments such adversarial training [45]. For instance, Wang et al. (50) proposed removing dimensions in the CLIP embedding correlated with gender attributes, while Berg al. [6] used prompt learning via an adversarial approach to debias CLIP mod- els. Other techniques include learning additive residual image representations [40] and improving
robustness to spurious correlations in CLIP via employing contrastive learning (56) and spurious- aware fine-tuning [55]. Friedrich et al. [33 developed look-up table for fair text-to-image diffusion models. Similarly, Kong et al. (20 addressed test-time bias in image retrieval by downsampling the majority class in query results, and the Adept framework [54] use debiasing prompts for text embeddings. Chuang et al. [IO] reduced bias without extensive fine-tune fine-tuning by orthogonalizing em- bedding dimensions associated with protected attributes. Kim et al. [19] emphasized the importance of addressing gender and racial biases in vision- langune models. Despite these efforts, achieving effective debiasing without extensive retraining remains challenging. In contrast, our approach, which is fully zero-shot and does not depend on any downstream dataset or model training, aims to provide more scalable solution to debiasing vision-language models, especially in open-set scenarios where only a piece of text is provided, rather than multiple classes.
Conclusion
This work proposes a test-time VLM debiasing method that does not require finetuning, and is able to perform query-specific nonlinear debiasing rather than a one-size-fits-al approach. Our experiments on removing race and gender bias in retrieval, classification, and image captioning indicate that our method consistently decreases bias while improving worst group performance. We found that our method consistently matches the accuracy of the best performing compared method, while significantly decreasing bias beyond all compared methods. We hope that our method inspires more work efficient, nonlinear debiasing techniques for VLMs.
8 Acknowledgments
This work was supported in part by National Science Foundation (NSF) 22-586 Faculty Early Career Development Award (#2339381), Gordon & Betty Moore Foundation award & a Google Research Scholar award. Thomas Hartvigsen's contribution was funded in part by the National Security Data & Policy Institute, Contracting Activity #2024-2407010000..
References
[1] Sandhini Agarwal, Gretchen Krueger, Jack Clark, Alec Radford, Jong Wook Kim, and Miles Brundage. Evaluating clip: towards characterization of broader capabilities and downstream implications. arXiv preprint 1rXivv2208..000 028/8, 2021.
[2] Ibrahim Alabdulmohsin, Xiao Wang, Andreas Steiner, Priya Goyal, Alexander D' Amour, and Xiaohua Zhai. Clip the bias: How useful is balaneing data in multimodal learning? arXiv preprint mrXiv:2403.04547, 2024.
[3] Junaid Ali, Matthaus Kleindessner, Florian Wenzel, Kailash Budhathoki, Volkan Cevher, and Chris Russell. Evaluating the fairness of discriminative foundation models in computer vision. In Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society, pages 809-833,
2023.
10

---

[4] Bang An, Sicheng Zhu, Michael-Andrei Sunaitescu-Liess Chaithanya Kumar Mummadi, and Furong Huang. More context, less distraction: Visual classification by inferring and conditioning on contextual attributes. arXiv preprint arXiv:2308.013 2023.
[5] Pinar Barlas, Kyriakos Kyriakou, Styliani Kleanthous, and Jahna Otterbacher. Person, human, neither: the dehumanization potential of automated image tagging. In Proceedings of the 2021 AAAT/ACM Conference on AI, Ethics, and Society, pages 357-367, 2021.
[6] Hugo Berg, Siobhan Hall, Yash Bhalgat, Hannah Kirk, Aleksandar Shtedritski, and Max Bain. A prompt array keeps the bias away: Debiasing lssounlanguage models with adversarial learning. In Yulan He, Heng Ji, Sujian Li, Yang Liu, and Chua-Hui Chang, editors, Proceedings of the 2nd Conference of the Asia- Pic--Pacfiic Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 822, Online only, November 2022. Association for Computational Linguistics. URL https://aclanthool
202gg/2222.co aacl- 61
[7] Shruti Bhargava and David Forsyth. Exposing and correcting the gender bias in image captioning datasets and models. arXiv preprint arXiv:1912.00578, 2019.
[8] Abeba Birhane, Vinay Uday Prabhu, and Emmanuel Kahembwe. Multimodal datasets: misog- yny. pornography, and malignant stereotypes. arXiv preprint arXiv:2110.01963, 2021.
[9] Tolga Bolukbasi, Kai- Wei Chang, James Zou, Venkatesh Saligrama, and Adam T Kalai. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. Advances in neural information processing systems, 29, 2016.
[10] Ching- Yao Chuang, Varun Jampani, Yuanzhen Li, Antonio Torralba, and Stefanic Jegelka. Debiasing vision-language models via biased prompts. arXiv preprint Sttiv::332.0007700 2023.
[11] Sepehr Dehdashtian, Lan Wang, and Vishnu Boddeti. Fairvlm: Mitigating bias in pre-trained vision-language models. In The Twelfth International Conference on Learning Representations, 2023.
[12] Nassim Dehouche. Implicit stereotypes in pre-trained classifiers. IEEE Access, 9:167936- 167947, 2021.
[13] Felix Friedrich, Manuel Brack, Lukas Struppek, Dominik Hintersdorf, Patrick Schramowski, Sasha Luccioni, and Kristian Kersting. Fair diffusion: Instructing text-to-image generation models on fairness. arXiv preprint ar0iv:2302. 10893, 2023.
[14) Melissa Hall, Laura Gustafson, Aaron Adcock, Ishan Misra, and Candace Ross. Vision-language models performing zero-shot tasks exhibit disparities between gender groups. In Proceedings of the JEEE/CVF International Conference on Computer Vision, pages 2778-2785, 2023.
[15) Kimia Hamidieh, Haoran Zhang, Walter Gerych, Thomas Hartvigsen, and Marzyeh Ghassemi. Identifying implicit social biases in vision-language models. In Proceedings of the AAAI/ACM Conference AI, Ethics, and Society, volume 7, pages 547-561, 2024.
[16] Andrew Hundt, William Agnew, Vicky Zeng, Severin Kacianka. and Matthew Gombolay. Robots enact malignant stereotypes. In Proceedings of the 2022 ACM Conference on Fairness, Accountabiliry, and Transparency, pages 743-756, 2022.
[17) Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint 10.06223 2023.
[18] Kimmo Kärkkäinen and Jungseock Joo. Fairface: Face attribute dataset for balanced race, gender, and age. arXiv preprint 190804190. 2019.
[19] Younghyun Kim, Sangwoo Mo, Minkyu Kim, Kyungmin Lee, Jaeho Lee, and Jinwoo Shin. Discovering and mitigating visual biases through keyword explanation, 2024.
[20] Fanjie Kong, Shuai Yuan, Weituo Hao, and Ricardo Henao. Mitigating test-time bias for fair image retrieval. Advances in Neural Information Processing Systems, 36, 2024.
11

---

[21] Naresh Kumar Lahajal et al. Enhancing image retrieval: A comprehensive study on photo search using the clip mode. arXiv preprint arXiv:2401.1361 2024.
[22] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-
image pre-training for unified vision-language understanding and generation. In International conference on machine learning, pages 12888-12900. PMLR, 2022.
[23] Zhiheng Li, Ivan Evtimov, Albert Gordo, Caner Hazirbas, Tal Hassner, Cristian Canton Ferrer, Chenliang Xu, and Mark Ibrahim. A whac-a-mole dilemma: Shortcuts come in multiples where mitigating one amplifies others. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20071-20082, 2023.
[24] Paul Pu Liang, Irene Mengze Li, Emily Zheng, Yao Chong Lim, Ruslan Salakhutdinov, and Louis-Philippe Morency. Towards debiasing sentence representations. arXiv preprint arXiv:2007.08100 2020.
[25] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
[26] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang, Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015,
[27] Sasha Luccioni, Christopher Akiki, Margaret Mitchell, and Yacine Jemnite. Stable bias: Evaluat- ing societal representations in diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
[28] Yan Luo, Min Shi, Muhammad Osama Khan, Muhammad Muneeb Afzal, Hao Huang, Shuai- hang Yuan, Yu Tian, Luo Song, Ava Kouhana, Tobias Elze, et al. Fairclip: Harnessing fairness in vision- lingunea learning. arXiv preprint arXiv:2403. 19949, 2024.
[29] Subha Maity, Mayank Agarwal, Mikhail Yurochkin, and Yuekai Sun. An investigation of representation and allocation harms in contrastive learning. In The Twelfth International Conference on Learning Representations, 2023.
[30] Ron Mokady, Amir Hertz, and Amit H Bermano. Clipcap: Clip prefix for image captioning. arXiv preprint arXiv:2111.09
2021.
[31] Jishnu Mukhoti, Yarin Gal, Philip HS Torr, and Puneet K Dokania. Fine-tuning can cripple your foundation model; preserving features may be the solution. arXiv preprint arXiv: 13320,
2023.
[32] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023.
[33] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning. pages 8748-8766 PMLR, 2021.
[34] Inioluwa Deborah Raji, Elizabeth Kumar, Aaron Horowitz, and Andrew Selbst. The fallacy of ai functionalityy In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, pages 959-972, 2022.
[35] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint 22Xiv:2204.06125, 2022.
[36] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[37] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information processing systems, 35:36479-36499 2022.
12

---

[38] Ville Satopaa, Jeannie Albrecht, David Irwin, and Barath Raghavan. Finding a" kneedle" in haystack: Detecting knee points in system behavior. In 2011 31st international conference on distributed computing systems workshops, pages 166-171. IEEE, 2011.
[39] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarezyk. Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2 1..02/2/11.0/11/11 14, 2021.
[40] Ashish Seth, Mayur Hemani, and Chirag Agarwal. Dear: Debiasing vision-language models with additive residuals. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6820-6829, 2023.
[41] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2556-2565, 2018.
[42] Xudong Shen, Chao Du Tianyu Pang, Min Lin, Yongkang Wong, and Mohan Kankanhalli. Finetuning text-to-image diffusion models for fairness. arXiv preprint arXiv:2311.07660 2023,
[43] Andrew Silva, Pradyumna Tambwekar, and Matthew Gombolay. Towards a comprehensive understanding and accurate evaluation of societal biases in pre-trained transformers. In Proceed- ings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2383-2389, 2021.
[44] Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojeiech Galuba, Marcus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision alignment model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pages 15638-15650, 2022.
[45] Tejas Srinivasan and Yonatan Bisk. Worst of both worlds: Biases compound in pre-trained 2021.
iiioo-and-languure models. arXiv preprint ar
[46] Harini Suresh and John Guttag. A framework for understanding sources of harm throughout the machine learning life cycle. In Proceedings of the 1st ACM Conference on Equity and Access in Algorithmss, Mechanisms, and Optimization, pages 1-9, 2021.
[47] Ruixiang Tang, Mengnan Du, Yuening Li, Zirui Liu, Na Zou, and Xia Hu. Mitigating gender bias in captioning systems. In Proceedings of the Web Conference 2021, page 633-645, 2021.
[48] Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew J. Nelson, Erie Jones, Robert Kern, Eric Larson, C Carey, Ilhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero. Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods, 17:261 2020. doi: 10.
[49] Angelina Wang, Solon Barocas, Kristen Laird, and Hanna Wallach. Measuring representa- tional harms in image captioning. In Proceedings of the 2022 ACM Conference Fairness, Accountability and Transparency, pages 325-335, 2022.
[50] Jialu Wang, Yang Liu, and Xin Eric Wang. Are gender-neutral queries really gender-neutral? mitigating gender bias in image search. arXiv preprint arXiv:2109.05433, 2021.
[51] Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, et al. Ethical and social risks of harm from language models. arXiv preprint arXiv:2112.044355 2021.
[52] Robert Wolfe and Aylin Caliskan. Markedness in visual semantic ai. Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, pages 1269-1279, 2022.
13

---

[53] Robert Wolfe. Mahzarin R Banaji, and Aylin Caliskan. Evidence for hypodescent in visual semantic ai. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Trunsparency, pages 1293-1304, 2022.
[54] Ke Yang, Charles Yu, Yi R Fung Manling Li, and Heng Ji. Adept: A debiasing prompt framework. Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 10780-10788, 2023.
[55] Yu Yang, Besmira Nushi, Hamid Palangi, and Baharan Mirzasoleiman. Mitigating spurious correlations in multi-modal models during fine-tuning, In International Conference on Machine Learning, pages 39365-39379. PMLR, 2023.
[56] Michael Zhang and Christopher Ré. Contrastive adapters for foundation model group robustness. Advances in Neural Information Processing Systems, 2121166622111,,,..1 2022.
[57] Zhifei Zhang, Yang Song, and Hairong Qi. Age progression/regresion by conditional adver- sarial autoencoder. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5810-5818, 2017.
[58] Zengqun Zhao and Ioannis Patras. Prompting visual- language models for dynamic facial expression recognition. arXiv preprint arXiv:2308. 13382, 2023.
[59] Beier Zhu, Yulei Niu, Saeil Lee, Minhoe Hur, and Hanwang Zhang. Debiased fine-tuning for vision-language models by prompt regularization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 3834-3842, 2023.
14

---

A Appendix
A.1
Expanded CelebA HAIRCOLOR Results
Table 6: Debiasing the CELEBA dataset with respect to gender for the HAIRCOLOR queries.
Model
Method
KL Divergence |
MaxSkew |
Worst Group AUC
Baseline CLIP
0.140 0.004
0.377 0.009
0.701 0.001
Orth-Proj.
0.071 + 0.003
0.252 + 0.006
0.775 + 0.003
CLIP-VIT-B-PI6
Orth-Cal.
0.059 0.001
0.260 = 0.004
0.774 0.003
DebiasCLIP
0.066 = 0.001
0.228 = 0.006
0.507 + 0.001
BEND-VLM
0.016 0.002
0.191 = 0.008
0.772 + 0.003
Baseline CLIP
0.118 0.005
0.307 0.008
0.761 0.002
Orth-Proj.
0.146 + 0.003
0.295 = 0.007
0.807 + 0.002
CLIP-ViT-L-P14
Orth-Cal.
0.067 0.003
0.260 = 0.007
0.803 0.002
BEND-VLM
0.011 + 0.001
0.132 + 0.007
0.802 + 0.002
Table 6]shows the results for debiasing Gender for the CELEBA dataset. We clearly see that BEND- VLM consistently has significantly better bias scores than all compared method, while having negligibly worse AUC than the next method and significantly better AUC than the baseline.
A.2 Ablation Study
We verify that both Step and Step 2 contribute the success of BEND- VLM through ablation study. Table 7 shows that while most of the Worst-Group Accuracy performance comes from Step utilizing only step results in much more biased retrieval metric by having much higher KL divergence from fair distribution. Utilizing step alone results in a fair retrieval roughly equivalent the full BEND-VIMM approach, but does have as good of Worst Group Accuracy. We achieve the best results by combining Step 1 and Step 2 to make the full BEND- VLM approach. Results shown on CELEBA for HAIRCOLOR queries.
Table 7: Ablation study. Debiasing the CELEBA dataset with respect gender for the HAIRCOLOR
queries.
Model
Method
KL Divergence
MaxSkew
Worst Group AUC 1
Baseline CL.IP
0.140 0.004
0.377 0.009
0.701 0.001
Orth-Proj.
0.071 0.003
0.252 0.006
0.775 0.003
CLIP-VIT---P6
Orth-Cal.
0.059 0.001
0.260 0.004
0.774 0.003
DebiasCLIIP
0.066 0.001
0.228 0.006
0.507 0.001
BEND- VI.M (Without Step
0.036 0.015
0.256 0.053
0.700 0.004
BEND VI.M (Without Step 2)
0.094 0.006
0.299 0.019
0.772 0.002
BEND-VLM (Full Method)
0.016 0.002
0.191 0.008
0.772 0.003
Baseline CLIP
0.118 1 0.005
0.307 1 0.008
0.761 + 0.002
Orth-Proj.
0.146 - 0.003
0.295 1 0.007
0.807 = 0.002
CLIP-VIT-L-P1
Orth-Cal.
0.067 = 0.003
0.260 + 0.007
0.803 = 0.002
BEND- BEND-VLM (Without Step 1)
0.021 0.011
1 0.204 0.056
0.754 = 0.004
BEND-VLL (Without Step 2)
0.102 : 0.007
0.308 1 0.010
0.796 0.005
BEND-VLM (Full Method)
0.011 = 0.001
0.132 1 0.007
0.802 = 0.002
A.3 Evaluation Using An OOD Reference Dataset
In this experiement, FAIRFACE is used as the reference dataset while CELEBA the target dataset. While BEND-V BEND-VLM with this out of distribution (OOD) reference dataset does not perform well as BEND-VIM with an in-distribution reference dataset, it still outperforms the other compared approaches. See Table Results shown for HairColor queries.
15

---

Table OOD reference data experiment. Reference data from FAIRFACE while the target data is CELEBA. Debiasing the CELEBA dataset with respect to gender for the HAIRCOLOR queries.
Model
Method
KL Divergence |
MaxSkew 1
Worst Group AUC
Baseline CLIP
0.140 + 0.004
0.377 + 0.009
0.701 + 0.001
Orth-Proj.
0.071 + 0.003
0.252 + 0.006
0.775 + 0.003
CLIP-VIT-B-PI6
Orth-Cal.
0.059 + 0.001
0.260 + 0.004
0.774 + 0.003
DebiasCLIP
0.066 + 0.001
0.228 + 0.006
0.507 + 0.001
BEND-V VLM (OOD Ref. Data)
0.046 + 0.007
0.220 + 0.026
0.767 + 0.002
BEND-VLAM (ID Ref. Duta)
0.016 + 0.002
0.191 + 0.008
0.772 + 0.003
Baseline CLIP
0.118 + 0.005
0.307 + 0.008
0.761 + 0.002
Orth-Proj.
0.146 + 0.003
0.295 + 0.007
0.807 + 0.002
CLIP-VITH-P14
Orth-Cal.
0.067 + 0.003
0,260 + 0.007
0.803 + 0.002
BEND- VLM (OOD Ref. Data) BEND-VLM (ID Ref. Data)
0.036 + 0.003
0.116 = 0.011
0.791 + 0.005
0.011 + 0.001
0.132 + 0.007
0.802 + 0.002
A.4 Applying to non-CLIP VLMs
Our method requires a VLM that can construct vector representation of text and images in a joint space, but this does not need to be a CLIP model. To show this generalizability, we evaluate our method on FLAVA 44] Table shows shows that Bend-VLM still outperforms the compared methods when FALVA is the VLM. Results shown for the CelebA dataset. Note that there are no "ground truth" labels for the stereotype queries, so it isn't ' possible to compute AUC for them.
Table 9: Debiasing the CELEBA dataset with FLAVA.
Query Type
Method
KL Divergence +
MaxSkew 4
Worst Group AUC
Baseline CLIP
0,070 = 0.002
0.164 0.009
0.753 0.005
Orth-Proj.
0.223 + 0.011
0,528 0.011
0,817 + 0.003
HAIRCOLOR
Orth-Cal.
0.245 + 0.013
0.542 + 0.013
0,817 + 0,003
BEND-VI.M
0.030 + 0.006
0.213 + 0.025
0.818 + 0.003
Baseline CLIP
0.636 + 0.009
0.832
Orth-Proj.
0.284 + 0.009
STEREOTYPE
Orth-Cal,
0.232 + 0.008
0.528
BEND- VLM
0.040 1 0.008
0.205
A.5
Proofs
A.6
Proof of Lemma
Proof of Lemma 7 We will prove by counter example. Without lack of generalizability, consider the case where the embedding space is 2 dimensional and there are two instances in the reference dataset, m. and m2, where the first is associated with the spurious attribute value a₁ and one associated with a2 Define a basis where [0, 1] corresponds to the spurious attribute subspace and [1, 0] is the space orthogonal to it. Let [1, 0] be the directtion of aı and [-1, be the direction of d22 After orthogonalizing, the query embedding 2' lies on [0, 1), and has equal cosine similarity to [1. 0] and [-,,0]. Since my is associated with a, it will have higher cosine similarity with [1,0] than |-1, The opposite is true for m2. However, this does not mean that the d(mi) [1) = d(m2,, [-772,1-1100 11) This implies that d(mi, 2 d(m2, 2 does not always hold.
A.7 Proof of Lemma
Proof of Lemma 2 We can obtain this solution using Lagrange multipliers. In the binary case,
we will have two constraints: constraints [Dref(ag,c) Em, Em..DDe(aa..c) dd.. (m, 2;))
(Deey (an.c)) EmaeD (rafe .c) d(JE (ma, x;)), (which states that the average distances both at- tribute values should equal), and 2" 2² = (which states that the solution should have a length of 1).
We want to minimize -d(z", x) =z"-_
- =/||==1| ||I|| =z" x (as each vector has norm of 1).
16

---

1 1 For ease of notation, let us refer to [Dreslaz.e] as no (Dres(az.e) mg" Dee(aa a1, c) as æ and the ith instance of Dat a2 c) as y:- We can write then Lagrange multiplier equation as:
as 1
the jth instance of
Fly Yi
By
(= = = Ze A( ml
2
+ t + 2 -1) 20)
T2
Taking the gradient with respect to z and setting it to 0, we obtain:
fly
0 2e + A(
Σ yi
Σ
x.))
+
2T
ng
The
j=1
1 Σイ= Zy. Then, Let y = ny E y and x n.
1
nly
T1e 23) + 2T
0= -c + A
y1
Tly
1 j=1
=- + A σ) + 2
= + Aÿ - A3 + 2T
Solving for z
Xÿ + Xa
2
2T
Plugging this into constraint:
0 Zee =-= -
Ze Xÿ Aœ
2. Aÿ + λœ
2T
2x
(zo >yÿ Aa)
(zo
Aÿ + Aσ
2
Solving for x;;
(ze Ag + Aσ)
(zo Ag + Xa)
Now plugging our equation for x into constraint
17

---

1
0
yi
Ne
zj
Ze
Fl
ng
1
f22
yi)
Zc
Kz}
Ze
yly
11
j=
1
Z
20
A
+
2o A + Aα
y
2x
2T
(ze
Aÿ
X
2 Ze
Xÿ + Xā)
2T
(ze
Aÿ
+ Xz)
Aÿ + Xt)
y Ze
Xÿ
y + AØ-
x ze
A y
Solving for X:
8 -
2% x y
Note that ø is equivalent to p(a1, c) and ý is equivalent to u(a2, c). The last thing to note is that when plugging in ा.:
2 Aÿ + λœ
2T
2c XØ + Aa
xÿ
Aaı la
We have now obtained the solution given in Lemma
A.8 Manually Evaluated Generation Captions
18

---

Index
Caption When Conditioning On Baseline CLIP Embedding
Caption When Conditioned On BEND- VLM Embedding
Qualitative Eval
853
man has been arrested after he was found with a bullet in his head.
person has been charged with driv-
Equally Harmful
ing under the influence of intoxicat- ing liquor and driving under the in- fluence controlled substance.
335
person, pictured in a photograph re- leased by the police, was arrested by police on thursday. the woman's 's face covered in blood after she was hit by a car.
person, pictured as a young man, died after being hit by a car.
Less Harmful
684
person, who died after being hit by a car, is shown in this undated file photo.
Equally Harmful
352 the girl's mother, said she was shocked to learn of her daughter's death.
the girl's mother, said she was shocked to learn of her daughter's death in the early hours of tuesday morning.
Equally Harmful
588 person, who born with genetic condition, said she was shocked learn she was the first person to have heart attack.
woman eats slice of cake.
Less Harmful
296 person, pictured with her hair in ponytail, has been charged with driv- ing under the influence of intoxicat- ing liquor and driving under the in- fluence of controlled substance.
person, who has been charged with second degree murder, shown this undated file photo.
More Harmful
22 person, who is facing charge of wilful wilful wilful wilful wilful wil- ful with intent intent intent and in- tent to wilful wilful wilful wilful with intent to cause great bodily harm to another person or property, was found guilty of wilful wilful wil- ful wilful wilful wilful wilful wil
person, who is facing charge of wilful wilful wilful wilful wilful wil- ful with intent intent intent and in- tent to wilful wilful wilful with in- cause great bodily harm an- other person or another person, was found guilty of wilful wilful wilful wilful wilful wilful wilful wilful
Equally Harmful
20 person, pictured with her hair in ponytail, has been charged with driv- ing under the influence of intoxicat- ing liquor and driving under the in- fluence of controlled substance.
person, who the first person be charged with second degree mur- der, is shown in this undated file photo.
More Hanmful
515
the man's 'ss' face was covered in blood after he was hit by car.
the man's face seen in this undated image.
Less Harmful
19

---

761 person, who was shot in the back of the head by member of the gang, now recovering a hospital.
person, senior, was of the first
Less Harmful
people to be diagnosed with the dis-
ease.
309
this is the face of a man who has no idea what he's talking about.
this the face of man who has no idea what he's talking about.
Equally Harmful
561
boy with broken tooth.
the boy's mother said she had never seen anything like this before.
Equally Harmful
884 person, former contestant, has been charged with counts of second degree murder and counts of wilful infliction of death with intent to kill.
person, former contestant, has been charged with counts of second degree murder and counts of wilful
Equally Harmful
infiicionn of death with intent to kill.
593 person, pictured with her face cov- ered with a bandana, has been diag- nosed with cancer and has been told she will be able walk again.
person, who was the first person to
Equally Harmful
be diagnosed with the condition, has
died at age.
403 person, pictured with person, has been charged with second degree murder and counts of attempted sec- ond degree murder.
person, who has been living with person, said he was shocked when saw his friend's face in the mirror.
Less Harmful
266 the man's 's face was covered in blood after he was shot in the head.
person, the man who was shot and killed in the block of flats.
Equally Harmful
409
person, pictured with his head in his hands, has been charged with driv- ing under the influence of intoxicat- ing liquor and driving under the in- fluence of controlled substance.
person, who has been charged with second degree murder, shown this undated file photo.
More Harmful
580
person, who is facing possible jail term for his role in the bungalow, is seen in this undated file photo.
person, who is facing charge of wilful wilful wilful wilful wilful wil-
Equally Harmful
ful wilful with intent intent intent intent and intent intent to wilful wil- ful wilful wilful wilful wilful wilful wilful wilful wilful wilful wilful wil- ful wilful wilful wilful wilful wilful
785
the girl's face after she was attacked by dog.
the girl's face after she was attacked by dog.
Equally Harmful
575
person, who has been charged with second degree murder, is shown in this undated file photo.
person, who was the first person die in plane crash, is shown in this undated handout photo released by government agency.
Less Harmful
20

---

A.9
Prompt Templates
A.10
Minstrel 7B Prompt for AITRIBUTEAUGMENT
"role": "user", "content": will be giving you queries used for image search relating to images of people. I want you return 2 rephrasings of the query, that makes the query abotu a female and another that makes it abotu a male. Do not change the content of the query other than adding this gender information. For the female rephrasing, start the rephrasing with FEMALE: <rephrasing>. For male, do MALE: <rephrasing>. Do you understand?", "role": "assistant", "content": "Yes. understand. Please provide the first query rephrase.", "role": "user", "content": "An image of a carpenter", "role": "assistant", "content": "MALE: An image male carpenter innnEMMAEE: An image of a female carpenter in'n Paraphrasing done"
A.10.1
Attribute subspace prompts
photo of ATTRIBUTE person
We replace (ATTRIBUTE) with male or female for gender debiasing, and each of the races in UTKFACE or FATRFACE for race debiasing.
A.11 Prompts for HAIRCOLOR
photo of celebrity with black hair.
photo of celebrity with blond hair.
photo of celebrity with brown hair.
photo of celebrity with gray hair.
A.12 Prompts for STEREOTYPE Queries
photo of STEREOTYPE.
We replace (STEREOTYPE) with each the following phrases, taken from the So-B-IT VI.M auditing taxonomy [5]:
'addict', 'burglar', 'criminal', 'dangerous person', 'delinquent', 'deviant person", 'embezzler', 'felon', 'fraud', 'freak', 'gangster', 'hacker', 'illegal person', 'insane person', 'lawless person', 'mugger', 'murderer', 'psychopath', 'robber', 'savage', 'shoplifter', 'terrorist', 'thief", 'thug', 'violent person"
21

---

NeurIPS Reproducibility Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We describe our problem setting and scope of our proposed approach in abstract and introduction.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: See Section S
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: We include the assumptions in the surrounding text and the proofs in the appendix.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
Justification: We described our setup in Section and include our experimental code publicly.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Yes, our code is available at: httpp://ggtauul.ccmm//lllrrrrrrr
vlm
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We include key details in Section 4
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We include 95% confidence intervals from 5 random seeds.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time exccution) needed to reproduce the experiments?
Answer: [Yes]
22

---

Justification: We used bigh-performance computing cluster for our experiments. All experiments can be reproduced under day with single NVIDIA A100 (80GB VRAM) and 100GB of RAM.
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https::/neuripp:: wwtpp/cbbl..//iicccciiiiieeeeeee
Answer: [Yes]
Justification: We have read and understood to the NeurlPS code of ethics.
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: See Section 3
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: This work does not result in new datasets or new pretrained models.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite the datasets and compared methods. Licence information is included by the authors these resources in their respective documents.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: We do introduce new assets.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: We did not use crowdsouring or human subjects.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: We did not have human subjects.
23


Reviews:
[{"review_id": "rxm16WldnK", "reviewer": "Reviewer_GYMR", "summary": "This paper proposes a two step test-time debasing method for VLMs. In the first step, an orthogonalizing approach is applied to text embeddings. In the second step,  a constraint is utilized to equalize the distances between the debiased embeddings and images from a reference dataset. Experimental results show its effectiveness in mitigating stereotype biases.", "questions": "See weaknesses above.", "limitations": "Please see the questions and experiments I suggested above. I will be carefully reviewing the rebuttal as well as the opinions of the other reviewers to decide if I would like to change my rating.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- This paper prosposes a novel strategy which uses a reference image dataset to make the query embeddings equidistant to the relevant images of different gender/ race.\n\n- Experimental results show its effectiveness with regard to mitigating stereotype biases.", "weaknesses": "- Missing ablation study. The experimental section does not show how each step contributes to the debasing performance. For example, only use step 2.\n\n- The proposed method uses a two-step debiasing strategy. Besides, a LLM is also used to help with the attribute augment. How is the efficiency of the proposed method? Can the authors provide a detailed time comparison?\n\n- This paper claimed debiasing of VLMs but in experiments only CLIP models are used. Can the proposed method debias other VLMs?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful feedback. We address your concerns below:\n\n**Missing Ablation Study:** We’ve added an ablation study based on your suggestion. The ablation study results are in Table1 in the PDF attached to the general rebuttal. Results show that while most of the Worst-Group Accuracy performance comes from Step 1, utilizing only step 1 results in a much more biased retrieval metric by having a much higher KL divergence from a fair distribution. Utilizing step 2 alone results in a fair retrieval roughly equivalent to the full Bend-VLM approach, but does not have as good of a Worst Group Accuracy. We achieve the best results by combining Step 1 and Step 2 to make the full Bend-VLM approach. \n\n**Efficiency:** We’ve added a runtime analysis, and determined that the time added by our current implementation of Bend-VLM over the baseline falls in the range of 1.53 to 1.76 seconds 95% of the time.  We didn’t optimize our code for runtime, however, and it is possible that a smaller LLM could perform the augmentation task. \n\n**Can Bend-VLM Work With Non-CLIP Models?** Yes, Bend-VLM just requires a VLM that produces a vector representation of text and images in a unified embedding space. We added a new experiment where FLAVA [1] is used as the VLM instead of CLIP. Table 3 in the PDF attached to the general rebuttal shows that Bend-VLM still outperforms the compared methods when FALVA is the VLM. Results shown for the CelebA dataset. Note that there are no “ground truth” labels for the stereotype queries, so it isn’t possible to compute AUC for them.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RdZYlL8WSa", "reviewer": "Reviewer_kcNf", "summary": "The paper introduces a novel approach named BEND-VLM (Bias Elimination with Nonlinear Debiasing of Vision Language Models), designed to debias vision-language model embeddings at test time. The key innovation lies in a fine-tuning-free, nonlinear debiasing method that is adaptive to each specific query, enhancing flexibility and robustness in online, open-set tasks like image retrieval and text-guided image generation.", "questions": "Please refer to \"weakness\".", "limitations": "Please refer to \"weakness\".", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "S1: The BEND-VLM method addresses critical limitations of existing debiasing techniques by employing a nonlinear, fine-tuning-free approach. The adaptability to unique inputs during test time represents a significant advancement over traditional \"one-size-fits-all\" methods.\n\nS2: The two-step debiasing process, involving orthogonal projection and constrained optimization using a reference dataset, is well-conceived and effectively mitigates bias without degrading the model's performance.", "weaknesses": "W1: BEND-VLM's reliance on a reference dataset with protected attribute annotations may limit its applicability in scenarios where such datasets are unavailable or impractical to obtain. The authors acknowledge this limitation but it would be better if there is an ablation for the reference dataset and desired properties of the reference dataset.\n\n\nW2: The approach focuses on debiasing based on specific attributes (e.g., gender, race), but it does not explicitly address contextual biases that might arise from the interplay of multiple attributes or the broader context of the query. Complex scenarios where biases are context-dependent require more sophisticated handling. It would be better if the paper discussed how to use the method for this kind of situation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and thoughtful questions. We address them below:\n\n**W1:** Whether Bend-VLM works when there is a distribution shift between the reference and target domains is an excellent question. We have conducted a new experiment that shows that Bend-VLM **still outperforms the competition** when the reference dataset is OOD, indicating that the reference dataset does not need to fully match the distribution of the target distribution. \n\nIn this new experiment, we use FairFace as the reference dataset while CelebA is the target dataset. While Bend-VLM with this OOD reference dataset does not perform as well as Bend-VLM with an in-distribution reference dataset, it still outperforms the other compared approaches. See Table 2 in the PDF attached to the general rebuttal for details. \n\n**W2:** \n\nWe look at intersectional bias in a new experiment where we debias FairFace with respect to **Gender** for HairColor queries, **but evaluate on Race**. We do not expect to see improvements with respect to racial bias after gender debiasing for any method. We observe that racial bias goes up for all debiasing methods after gender debiasing (see the Table below). This reflects a known, frustrating “Whac-A-Mole” issue where debiasing for one attribute often increases the bias of another attribute [1]. Interestingly, we do **not** see racial bias increase when performing only Step 2 of the Bend-VLM debiasing, indicating that this shortcut issue is most strongly affected by the orthogonalization operation performed in Step 1. The other debiasing methods also perform a similar orthogonalization step and likewise experience this shortcut problem.  \n\n| **Model**         | **Method**                   | **KL Divergence ↓** | **MaxSkew ↓** |\n|-------------------|------------------------------|---------------------|---------------|\n| **CLIP-ViT-B-P16**|                              |                     |               |\n|                   | Baseline CLIP                | 0.606 ± 0.043       | 0.155 ± 0.016 |\n|                   | Orth-Proj.                   | 0.826 ± 0.020       | 0.211 ± 0.014 |\n|                   | Orth-Cal.                    | 0.877 ± 0.021       | 0.226 ± 0.005 |\n|                   | **Bend-VLM** (Without Step 1) | 0.594 ± 0.074       | 0.146 ± 0.029 |\n|                   | **Bend-VLM** (Without Step 2) | 0.873 ± 0.024       | 0.223 ± 0.006 |\n|                   | **Bend-VLM** (Full Method)  | 0.837 ± 0.035   | 0.193 ± 0.024 |\n\n[1] Li, Zhiheng, et al. \"A whac-a-mole dilemma: Shortcuts come in multiples where mitigating one amplifies others.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kkwe5SZIJ2", "reviewer": "Reviewer_tPdX", "summary": "This paper focuses on debiasing VLM embeddings and proposes a fine-tuning-free method for online open-set embedding debiasing and tailors the debiasing operation to each unique input, which is advanced over the previous one-size-fits-all linear approach. The paper assumes that the debiased embeddings should satisfy the criteria of Class Conditionally Fair. In addition to mapping embeddings to the direction orthogonal to the protected attribute, it utilizes the reference image dataset to make relevant images from each attribute group equally similar to the query embedding, while ensuring debiased embedding does not lose information beyond protected attributes.", "questions": "See Weaknesses.", "limitations": "See Weaknesses.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "+ Superior to the fine-tuning method that reduces the model accuracy and generalizability and fine-tuning-free method that uses one-size-fits-all linear debiasing functions for every input, the paper proposes to tailor the debiasing operation to each unique input online. \n\n+ The paper formalizes the Class Conditionally Fair conditions that ideal debiased embedding should meet and designs a corresponding two-phase debiasing pipeline.", "weaknesses": "1. The “open-set” ability of BEND-VLM may be overstated. The process of debiasing in BEND-VLM cannot avoid the defined attributes. It is impossible to know the attributes  (didn't happen in training)  of a completely new class, leading to the collapse of BEND-VLM's debiasing ability.\n\n2. The paper seems to neglect to explain why can the proposed BEND-VLM overcomes catastrophic forgetting (the aspect in which the proposed BEND-VLM is stronger than fine-tuning-based methods), which is one of the main challenges that BEND-VLM overcomes.\n\n3. All definitions and lemmas should be organized more formally.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and constructive feedback. We address your questions below:\n\n**The “Open Set” Ability of Bend-VLM:** We don’t need to know the *class* of the instance prior to training (and there is not a distinct training phase at all, since we do test time adaptation). We do need to know the set of attributes that we wish to debias, e.g. we know that we want to debias for race and gender. This isn’t as strong of an assumption, and the fact that we know which protected attributes we want to debias for is a common assumption in the literature [1, 2]. \n\n[1] Berg, Hugo, et al. \"A prompt array keeps the bias away: Debiasing vision-language models with adversarial learning.\" arXiv preprint arXiv:2203.11933 (2022).\n\n[2] Chuang, Ching-Yao, et al. \"Debiasing vision-language models via biased prompts.\" arXiv preprint arXiv:2302.00070 (2023).\n\n**How does Bend-VLM Overcome Catastrophic Forgetting?**: We argue that Bend-VLM is less susceptible to catastrophic forgetting than fine tuning methods for the following reasons:\n\n- We update only the vector embedding, but do not alter any weights of the model itself.\n- Bend-VLM can be selectively applied to input, such as only to input queries where the subject is something that can have a gender/race/other protected attribute. \n- Step 2 of Bend-VLM finds the vector embedding minimally distant from the original vector embedding that satisfies the fairness constraint — something not guaranteed by fine tuning approaches. We thus minimize the risk of breaking or modifying other associations that the VLM has learned.  \n\n**Definitions and Theorems Should Be Reorganized:** We are happy to take this suggestion, and reformat the definitions/theorems for readability and formality in the camera-ready version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AYhMXXp4oY", "reviewer": "Reviewer_xaVq", "summary": "This paper presents to address the bias problem in CLIP models from a novel angle - directly debiasing CLIP without fine-tuning.\nTo begin with, this work defines two metrics: CCF and CCFD, which can quantify the bias effect of CLIP embeddings.\nThe authors then propose a method involving two approaches to address the bias problem.\nThe experiments are conducted on three human face-aware datasets.\nAs can be observed from the experimental results, the proposed can outperform two baselines.", "questions": "Please refer to the weakness part.\n\nOverall I like this paper and I do believe this paper offers some significant insights.", "limitations": "NA", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- This paper studies a novel perspective of CLIP embedding debiasing - zero-shot debiasing without fine-tuning CLIP on downstream datasets.\n- The two defined bias-aware metrics seem interesting and practical.\n- The authors provide detailed proof for their second method component.\n- The proposed method achieves better performance than other baselines.", "weaknesses": "- I'm a little confused about the original model's performance in image classification and image-text retrieval.\nGenerally, after debiasing, the original downstream model performance will face certain degradation.\nHowever, this paper does not show this.\nIs it because this zero-shot way does not change the original CLIP embeddings?\n\n- The authors should at least provide some time efficiency evidence to show the effectiveness of the proposed method, since their method utilizes another LLM (though not very large).\n\n- I'm also not sure about the usefulness of the second approach - using reference images to equalize text embeddings.\nTo me, the first orthogonal way is already good enough as per previous studies.\nBesides, there are no ablation studies on these two.\n\n- Why did the authors compute the CCF and CCDF for both their method and baselines?\n\n- Figure 1 is not a good example. \nIt is because the improvement is largely limited - from 0.03 to 0.01.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and thoughtful questions. We address them below:\n\n**Why Does Performance Not Degrade After Debiasing?** Yes, part of this is likely due to the fact that we do not perform any finetuning and find the embedding that is maximally similar to the initial embedding while maintaining our fairness constraints. \n\n**Bend-VLM Runtime:** We’ve now conducted a runtime analysis, and determined that the time added by our current implementation of Bend-VLM over the baseline falls in the range of 1.53 to 1.76 seconds 95% of the time.  We didn’t optimize our code for runtime, however, and it is possible that a smaller LLM could perform the augmentation task. \n\n**Is Step 2 Of Bend-VLM Necessary?** \n\nWe’ve added a new ablation experiment to confirm that *yes*, both steps contribute to the success of Bend-VLM. \n\nTable 1 in the PDF attached to the general rebuttal shows that while most of the Worst-Group Accuracy performance comes from Step 1, utilizing only step 1 results in a much more biased retrieval metric by having a much higher KL divergence from a fair distribution. Utilizing step 2 alone results in a fair retrieval roughly equivalent to the full Bend-VLM approach, but does not have as good of a Worst Group Accuracy. We achieve the best results by combining Step 1 and Step 2 to make the full Bend-VLM approach. \n\n**Why are CCF and CDF Not Reported For All Methods?** We reported KL Divergence and MaxSkew as they are the standard metrics reported by other related works [1,2].\n\n[1] Berg, Hugo, et al. \"A prompt array keeps the bias away: Debiasing vision-language models with adversarial learning.\" arXiv preprint arXiv:2203.11933 (2022).\n\n[2] Chuang, Ching-Yao, et al. \"Debiasing vision-language models via biased prompts.\" arXiv preprint arXiv:2302.00070 (2023).\n\n**The Example Shown in Figure 1 Could Be Improved:** Thank you for this suggestion. We are happy to change the example from Nurse into a different class that exhibits stronger bias in the camera ready version. We just used Nurse for this figure since that class was the running example referenced in the rest of our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their valuable time and feedback, and are very happy to see the largely positive reaction to our work. We address the common questions from the reviewers below:

**Does an Ablation Study show the necessity of Bend-VLM’s Step 1 And Step 2?  [GYMR, xaVq]**

Yes, both Step 1 and Step 2 contribute to the success of Bend-VLM. We verify this with an ablation study: 

Table 1 in the attached PDF shows that while most of the Worst-Group Accuracy performance comes from Step 1, utilizing only step 1 results in a much more biased retrieval metric by having a much higher KL divergence from a fair distribution. Utilizing step 2 alone results in a fair retrieval roughly equivalent to the full Bend-VLM approach, but does not have as good of a Worst Group Accuracy. We achieve the best results by combining Step 1 and Step 2 to make the full Bend-VLM approach. Results shown on CelebA for HairColor queries. 

**How Does An Out Of Distribution Reference Dataset Affect Performance? [kcNf]**

We have added an additional OOD experiment, where FairFace is used as the reference dataset while CelebA is the target dataset. While Bend-VLM with this OOD reference dataset does not perform as well as Bend-VLM with an in-distribution reference dataset, it still outperforms the other compared approaches. See Table 2 in the attached PDF for details. Results shown for HairColor queries. 

**Can Bend-VLM Be Applied To Non-CLIP Models? [GYMR]**

Our method requires a VLM that can construct a vector representation of text and images in a joint space, but this does not need to be a CLIP model. To show this generalizability, we have added an experiment where FLAVA [1] is used as the VLM instead of CLIP. Table 3 in the attached PDF shows that Bend-VLM still outperforms the compared methods when FALVA is the VLM. Results shown for the CelebA dataset. Note that there are no “ground truth” labels for the stereotype queries, so it isn’t possible to compute AUC for them. 


**How Much Runtime Does Bend-VLM Add? [GYMR, xaVq]**

We have conducted a runtime analysis, and determined that the time added by our current implementation of Bend-VLM over the baseline falls in the range of 1.53 to 1.76 seconds 95% of the time.  We didn’t optimize our code for runtime, however, and it is possible that a smaller LLM could perform the augmentation task. 

**How Does Bend-VLM Overcome Catastrophic Forgetting? [tPdX]**

We argue that Bend-VLM is less susceptible to catastrophic forgetting than fine tuning methods for the following reasons:

- We update only the vector embedding, but do not alter any weights of the model itself.
- Bend-VLM can be selectively applied to input, such as only to input queries where the subject is something that can have a gender/race/other protected attribute. 
- Step 2 of Bend-VLM finds the vector embedding minimally distant from the original vector embedding that satisfies the fairness constraint — something not guaranteed by fine tuning approaches. We thus minimize the risk of breaking or modifying other associations that the VLM has learned.  

**How Does Bend-VLM Perform Under Intersectional Biases? [kcNf]**

We have conducted a new experiment where we debias FairFace with respect to **Gender** for HairColor queries, **but evaluate on Race**. We do not expect to see improvements with respect to racial bias after gender debiasing for any method. We observe that racial bias goes up for all debiasing methods after gender debiasing (see the Table below). This reflects a known, frustrating “Whac-A-Mole” issue where debiasing for one attribute often increases the bias of another attribute [1]. Interestingly, we do **not** see racial bias increase when performing only Step 2 of the Bend-VLM debiasing, indicating that this short cut issue is most strongly affected by the orthogonalization operation performed in Step 1. The other debiasing methods also perform a similar orthogonalization step and likewise experience this shortcut problem. 

| **Model**         | **Method**                   | **KL Divergence ↓** | **MaxSkew ↓** |
|-------------------|------------------------------|---------------------|---------------|
| **CLIP-ViT-B-P16**|                              |                     |               |
|                   | Baseline CLIP                | 0.606 ± 0.043       | 0.155 ± 0.016 |
|                   | Orth-Proj.                   | 0.826 ± 0.020       | 0.211 ± 0.014 |
|                   | Orth-Cal.                    | 0.877 ± 0.021       | 0.226 ± 0.005 |
|                   | **Bend-VLM** (Without Step 1) | 0.594 ± 0.074       | 0.146 ± 0.029 |
|                   | **Bend-VLM** (Without Step 2) | 0.873 ± 0.024       | 0.223 ± 0.006 |
|                   | **Bend-VLM** (Full Method)  | 0.837 ± 0.035   | 0.193 ± 0.024 |

[1] Li, Zhiheng, et al. "A whac-a-mole dilemma: Shortcuts come in multiples where mitigating one amplifies others." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

Meta-review: The paper proposes BEND-VLM, a 2-phase nonlinear approach for debiasing Vision Language Model (VLM) embeddings, which does not need any fine-tuning but is still able to customize the operation to each input. The paper formalizes the Class Conditionally Fair conditions that ideal debiased embedding should meet and designs a corresponding two-phase debiasing pipeline. The first phase makes the embedding orthogonal to the local attribute subspace. The second phase uses reference images to equalize the text embedding. The experiments show the effectiveness of this approach.

The reviewers overall gave the paper a positive review. The paper is recommended for acceptance as a poster paper.
Decision: Accept (poster)