Mitigating Test-Time Bias for Fair Image Retrieval
Authors: Fanjie Kong
Shuai Yuan
Weituo Hao
Ricardo Henao
Venue: NeurIPS 2023 poster (2023)
Keywords: Vision-language
Fairness
Text-based Image Retrieval
Deep Learning
Application
TL;DR: ensuring fairness in text-based image retrieval by mitigating test-time bias from target retrieval set.

Abstract: We address the challenge of generating fair and unbiased image retrieval results given neutral textual queries (with no explicit gender or race connotations), while maintaining the utility (performance) of the underlying vision-language (VL) model. Previous methods aim to disentangle learned representations of images and text queries from gender and racial characteristics. However, we show these are inadequate at alleviating bias for the desired equal representation result, as there usually exists test-time bias in the target retrieval set. So motivated, we introduce a straightforward technique, Post-hoc Bias Mitigation (PBM), that post-processes the outputs from the pre-trained vision-language model. We evaluate our algorithm on real-world image search datasets, Occupation 1 and 2, as well as two large-scale image-text datasets, MS-COCO and Flickr30k. Our approach achieves the lowest bias, compared with various existing bias-mitigation methods, in text-based image retrieval result while maintaining satisfactory retrieval performance. The source code is publicly available at \url{https://github.com/timqqt/Fair_Text_based_Image_Retrieval}.

Mitigating Test- Time Bias for Fair Image Retrieval
Fanjie Kong Duke University kong @ edu
Shuai Yuan Duke University shuai @ cc.cc.uu ds.duke.edu
Weituo Hao Tok Inc. weituohao @tiktok.com
Ricardo Henao
Duke University
KAUST
ricardo.henao duke.edu
Abstract
We address the challenge of generating fair and unbiased image retrieval results given neutral textual queries (with no explicit gender or race connotations), while maintaining the utility (performance) of the underlying vision- aasion-languge (VL) model. Previous methods aim to disentangle learned representations of images and text queries from gender and racial characteristics. However, we show these are inadequate at alleviating bias for the desired equal representation result, as there usually exists test-time bias in the target retrieval set. So motivated, we introduce a straightforward technique, Post-hoc Bias Mitigation (PBM), that post-processes the outputs from the pre-trained vision-language model. We evaluate our algorithm real-world image search datasets, Occupation and 2, as well two large- scale image-text datasets, MS-COCO and Flickr30k. Our approach achieves the lowest bias, compared with various existing bias- bias-mitigation methods, in text-based image retricval result while maintaining satisfactory retrieval performance. The source code is publicly available at athttps:/// htap::://githan github....... com/tinggt/aairrr.m..../o..//. Text_ based_ Imwae_mmearre Retrieval
Introduction
Image search on the web based on text-based image retrieval (TBIR) (Lew et al. 2006 involves interpreting user's (text) query and returning corresponding images that are considered relevant in terms of semantic meaning (Chen et al.) 2015). With recent advancements in multi-modal representation learning, Vision-Language (VL) models such as CLIP have been widely used to enhance the efficacy of text-based image retrieval (Radford et al.) 2021 Cao et al. (022). These models are usually trained on vast datasets that consist of millions text-image pairs scrapped from the web, which inevitably manifest societal biases especially for neutral queries (Wang et al. 202 Hall et al. 2023; Wang et al. 2021b), i.e., queries without explicit demographic (gender or race) connotations. In Figure CLIP and an unbiased alternative delivered by the proposed approach.
show image retrieval results for the neutral query "Bus Driver" using
In our work, we adhere to equal representation as our fairness objective (Mchrotra and Celis 2021 Kay et al. 2015), as an alternative to proportional representation (Jalal et 2021 |eerg al. 2022). The latter, which aligns the demographic proportions in the retrieval results with those in the dataset, which is susceptible biases during collection (Kay al. (015). Instead, equal representation for all demographic groups of interest attempts mitigate (obscure) the influence of any inherent biases.
Previous works have been dedicated developing unbiased VL models that promote fairness in
TBIR tasks (Wang et al. 202 Berg et al. 2022 Wang et al. 2022 Kim et al.) 2023
Chuang
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Unbiased??
Query: Bus Driver
Text-based image retrieval (TBIR) results of neutral query. Right: all returned images are male bus drivers, possibly pushing the message that only men (can) perform this job. Left: desirable unbiased result with equal gender representation obtained by the proposed PBM approach.
Figure I:
et als 2023; Parraga et al. 2022). Berg et al. (2022) leveraged adversarial training for debiasing in which a learnable prompt prepended to the input text queries, while an adversary seeks to discourage the image and text encoders from capturing gender or race information for retrieval. Alternatively, Wang et al. 2021a) discarded the components of the visual and text representations that highly correlated with gender, by estimating the mutual information between these components and gender attributes. Further. Wang et al. (2022) neutralized the gender contributions in image representations by enforcing equal contributions from gender (male and female) features using bias contrast loss, while maximizing the contribution of gender-irrelevant features. These studies largely rely demographic attributes that be somewhat visually perceived. For instance, gender attributes are derived from perceived characteristics of masculinity or femininity, leading to labels such "Male" or "Female". Similarly, race-related attributes are often characterized based skin tones and typically categorized into groups such as "Fair Skin" and "Dark Skin" (Kay et al.. 2015 Celis and Keswani 2020). However, these methods primarily focus debiasing the gender or race encoding within VL models, which may be insufficient due to bias or imbalance in the image retrieval candidate pool (test set). In contrast, though we also mitigate bias in TBIR using the same demographic attribute annotations as other works, the proposed solution is simpler in the sense that it forgoes the need for access to gradients or retraining the VL model.
In this paper, we start Section 3.1 by defining the text-based based image retrieval task in the context of fairness objective focused on equal representation. We then analyze the effectiveness of existing bias reduction methods in Section 3.44 and demonstrate how they fall short in achieving equal representation, mainly due imbalances the test-time image retrieval set. Based on this observation,
in Section 3.5 we propose a simple yet effective post-processing debiasing method called post-hoc bias mitigation (PBM) that creates fair retrieval subsets guided by predicted gender (or race) attributes obtained from either an off- off-the-shelf gender (or race) classifier or zero-shot inference using a pre- trained VL model. In Section we evaluate PBM on real-world web image search (Kay et al. 2015 Celis and Keswani] (020)) and large-scale image-text datasets such as MS-COCO (Chen et al. 2015 and Flickr30K (Plummer et al. 2015). By comparing our approach to various existing bias mitigatio techniques, we show that our method achieves the lowest bias while maintaining satisfactory retrieval performance, as evidenced by both quantitative and qualitative results. Importantly, PBM strives to provide more unbiased, fair and diverse visual representation of different groups in search results, ultimately promoting social welfare by challenging existing social perceptions of gender and race.
The summarized contributions of this work are:
We present an analysis of the effect of existing debiasing methods on VL models, and highlight their insufficiency in achieving equal representation for text-based image retrieval.
We propose PBM, a straightforward and efficient test-time post-procetd pest-processin debiasing algorithm that generates fair retrieval subsets, guided by predicted gender/race information obtained from an off-the-shelf classifier or inferred via zero-shot using the VL model itself.
We evaluate PBM on two real-world web image search and two large-scale image-text datasets for text-based image retrieval, and compare with existing bias-mitign mitigaaiinio techniques, demon- strating its effectiveness in achieving the lowest bias among all tested methods.

---

2 Related work
Text-based image retrieval Text-based image retrieval is the process of searching and retrieving images from large database using textual descriptions or keywords as queries. The task is usually tackled by image-text feature alignment (Cao et al., 2022), which embeds image and text inputs into shared feature space such that relevant image and text features are close each other. In recent years, substantial progress has been made due to the emergence of large-scale datasets, as well as the development of effective deep Saarinngbassed image-text models. Pioneering works include ViVE (Frome et al., (2013), which bridges the semantic gap between image content and textual descriptors by aligning CNN-based features of images with textual embeddings from ImageNet labels. Subsequent approaches, such as VSE++ (Faghri et al. (017) and SCAN (Lee et al. 2018). further refine these joint embeddings to improve retrieval performance. Recently, OpenAT's CLIP (Radford et al. 202T) has emerged as powerful approach for image retrieval based on similarity matching between extracted image and text features. CLIP leverages a pre-trained transformer model, jointly optimized for both image and text understanding, which allows it to effectively match images and textual deseriptions.
Fairness in machine learning Recent studies have highlighted numerous unfair behaviors in machine learning models (Angwin et al. 2016 Buullamwini and Gebru 2018). For example, a risk assessment algorithm used in the United States criminal justice system predicted that Black defendants were more likely commit future crimes than white defendants, even after controlling for criminal record (Angwin et al.) 2016). Moreover, individuals with different gender and skin-tones are likely to receive disparate treatment in commercial classification systems such as Face++, Microsoft, and IBM systems (Buolamwini and Gebru 2018). Consequently, there has been a surge in demand and interest for developing methods to mitigate bias, such as regularizing disparate impact (Zafar et al.] (2015) and disparate treatment (Hardt et al., 2016), promoting faimess through causal inference (Kusner et al. 2017). and incorporating fairness guarantees in recommendations and information retrieval (Beutel et al. 2019 Morik et al. 2020).
Fairness in vision-language models After some studies revealed the bias of using VL models in downstream tasks (Wang et al. 2021a Hall et al.] 2023 Wang et al. 2021b), efforts to address and mitigate these biases in VL models have gained increasing attention. Existing solutions for fair Dision-language models can be generally classified into pre- preccessss,ng in- un-processing, and post- processing methods. Pre- processiningggg techniques usually involve re- weighting adjusting the training data to counter imbalances across demographic attributes, while preserving the utility of the dataset for the target task (Friedler et al. 2014: Calmon et al., (017). In-processing methods focus on altering the training objective by incorporating fairness constraints, regularization terms or leveraging adversarial learning to obtain representations invariant to gender/race (Berg et al. 2022; Wang et al..
2023 Xu et al. 2021 Cotter et al.] (2019). Post-processing approaches achieve fairness by applying (pre-)trained model (Cheng et al. [02T Calmon al. 2017) or via feature
post-hoc corrections
clipping (Wang et al. 2021a) on the output of image-text encoders based on mutual information.
Our work lies in the post-processing category of debiasing methods that encourages equal repre- sentation of diverse demographics. We also identified the fair subset selection approach used in potential post-hoe debiasing method for TBIR. While Mehrotral and Celis (2021) shares our goal of ensuring equality of gender/race attributes in the set of results, their focus did not extend to the TBIR scenario with underlying VI. model nor detail an effective method for obtaining accurate demographic attributes for debiasing. More importantly, they assumed demographic attributes seen by their algorithm to be available ground truth labels subject to noise. This assumption creates difficulties when attempting to adapt their method to a real-world problem such as TBIR. Complementary, our approach is meant to address these deficiencies by providing a practical debiasing procedure, that includes acquiring demographic attributes.
Mehrotra and Celis (202T)
Gender/Racial Bias in Web Image Search Our research is closely related to studies in the Human Computer Interaction community that demonstrated gender inequality issues in current online image search systems (Kay et al. 2015; Noble 2018 Celis and Keswani (020). These studies revealed how gender bias in occupational image search results influences people's 's perceptions about the presence of men and women in various professions. Our work builds upon these findings by examining the gender and racial bias in image search algorithm and offering innovative solutions for reducing bias in popular image retrieval framework using pre- pre-trained VL model, such as CLIP (Radford et al. 2021
3

---

3 Method
3.1 Problem formulation
Image retrieval Suppose C the set of all text queries of interest (gender- defined below), and V N is database of N images, from which we retrieve images given query inputs. Each query input E C is relevant to a ground-truth set of images V S VV provided by human annotators.
The text-based image retrieval task aims at finding images from the database so they best match the text query inputs. Specifically, given any text query c E C and a fixed retrieval size K (K < N) as inputs, the goal is design an algorithm that returns a bag of K images Ve.K S V, where [Ve,xl K, such that Ve,x contains as many relevant images from Vi as possible.
For evaluation purposes, a top-K recall score ("Recall @ K"" is usually computed to quantify whether the most relevant images are included in the retrieval output. Specifically, we write
Recall K Eeec VV., nV
100%,
EcEc IV21
where selected specifically for each dataset so that |V[ K for most queries E C; this way, the recall can be realistically close to 100%.
Debiasing in image retrieval Using gender as a motivating scenario, we are interested gender debiasing in cases where queries relate to human characteristics with no gender connotations, which we refer to as gender-neutral queries. For example, queries could be occupations that are widely open to all individuals, irrespective of gender (Organization 2019), such as chef, nurse, social worker, etc.
We anticipate that gender-neutral queries should yield image retrieval sets that comprise equal representation of both male and female associated images, i.e., in a 1:1 ratio, which is consistent with the equal representation goal discussed by Wang al. (2021a) 2023); Mehrotra and Celis (202T): Mehrotra and Vishnoi (2022). Note that we use gender debiasing as example, but the same setting can be generalized to other debiasing issues such as racial discrimination, as we will demonstrate in the experiments.
Specifically, assume each image v V has gender attribute g(v) e (+1,-1}, which corresponds to the two genders manifested in the image, male and female, respectively. For each gender-neutral query c, the gender bias of the resulting retrieved bag of images Ve,k is defined the normalized absolute difference between the numbers of images of each gender, i.e.,
B(Vc.))
1 K
Σ VEVe,K
1(g(v) =
Σ L(g(v) vEV..k
Σ PEV..K
8(v)
(I)
which ranges from & (mod(K,)) 2) (minimum bias) (maximum bias). In our retrieval approach, we average B( B(Ve,x) across all gender-neutral queries c C as evaluation metric for fairness, ie.,
AbsBias@K
B(V..) =
Σ CEC
Σ g(v)
|C|
CC
K
CEC
NEVC
The goal is to minimize AbsBias K while maintaining satisfactory retrieval Recall @
3.2 Similarity-based image-text matching
Overall framework As in previous works (Singh al. 2003 Bai et al. 2014 Zaidi al. 2019
Cao et al. 2022] Mukhoti et al.) (022), we use a similarity matching approach to tackle the image retrieval task. Such an approach involves aligning image and text features from largely pre-trained vision and language models during training time. At inference time, we rank images based on whether their features are similar to the input query text features and use the top-ranked images as our retrieval output. Specifically, we use an image encoder network f.(-) and text encoder network fy (-) to embed both image and text inputs into shared d-dimensional feature space as fa(v), (c) Rd cosine similarity score S(v, c) is then computed to quantify the relevance between and e (Singh
al. 2003 Bai et al. 2014 Wang et al. 202

---

Training As in previous works (Chen et al. 2020 Radford et al. 2021), the training of the image and text encoders is achieved through optimizing an NT-Xent (Normalized Temperature Cross Entropy) loss with stochastic gradient descent on mini-batches.
Inference Given a new image database Viest for each new query input c Clest, we compute its similarity score S(V,, c) with every image Vi E syrest and then pick the top-K images that have the maximum similarity scores form our retrieved set Ve.K.
3.3 Fairness criterion for image retrieval
Fairness can be defined in numerous ways in favor of different situation (Saxena et al., 2020). In the context of fair image retrieval (Wang et al. 2021a Berg et al. 2022), define fairness objective as equal representation, implying that the retrieved image set through the retrieval algorithm should encompass an equal number of samples from each demographic group. We formally define our criterion as follows:
Definition 3.3.1 (Equal Representation) An image retrieval algorithm satisfies equal representa- lion with respect to binary demographic altributes g(v) if.
= ++]]]] (B.-P(EB(L(((()) Bv. [B. = -1]]]
where P represents the distribution of retrieved image set Ve corresponding to neutral queries c.
The above definition implies 3lB((V)lv.-- 0, i.e., the expected bias is zero for any retrieved image sets. Our definition of equal representation is equivalent to "Equal opportunity for ranking distributions" introduced by (Singh and Joachims (2017) when ranking position bias is constant.
3.4
Bias analysis
Using the image retrieval framework defined above as a basis, is important to incorporate strategies to reduce gender (and race) biases of the retrieval output BV., Below, we briefly analyze existing methods and discuss their limitations.
Existing methods Most methods address the gender fairness issue by enforcing model features to less dependent on gender information. This achieved by mainly two types of approaches, namely adversarial training (Edwards and Storkey, 2015 Berg et al.) 2022 Xu et al., 2021 and mutual information (MI) minimization (Wang et al. 2021a 2023). Specifically, adversarial training involves training separate (adversarial) classifier network by adding adversarial loss so that the adversarial network cannot distinguish gender given the encoded image features (Edwards and Storkey 2015; Berg al. 2002 Xu al., (021). Alternatively. MI minimization aims at reducing the MI between feature distribution and their gender by clipping feature dimensions highly correlated with gender (Wang et al.) 2021a). Both methods encourage the model to extract features that are independent of gender. However, we argue below that enforcing this kind of independence between image features and gender is not sufficient to effectively eliminate gender bias in image retrieval.
Illustrative example We start by showing debiasing results using mutual information minimization. Similar insights can be obtained via adversarial training, which is shown in the Supplementary Material (SM). Figure 2 shows an example for the query "engineer" comparing three methods, namely, CLIP (no debiasing), MI-based debiasing and the proposed PBM (described below). For each method, we first compute, sort and group the similarity scores of all images into 1% quantiles (~ 30 images each). Then, for each quantile, we compute the gender bias as defined in ( This analysis shows the relationship between gender bias and similarity scores. Note that for retrieval, we use samples with the largest similarity scores to form our retrieved bag of images, thus only the right-most data points (highlighted in the figure) are part of the retrieval set. We show percentiles in the figure to emphasize similarity rank rather than the similarity scores per se.
As shown in Figure 2(a) the original image retrieval algorithm with no debiasing based on CLIP tends to assign higher scores (at larger percentiles) to samples associated with male attributes, as evidenced by the high correlation between gender and the similarity scores. This makes the final retrieval output largely biased towards male samples. In contrast, Figure 2(b) shows the result of
debiasing via MI minimization (Wang et al. 202 Using the regression line as a visual guide,
5

---

1
Gender blas
Gender bias
Gender bias
-1
25
75
100
75
50
100
Similarity score percentile
Similarity score percentile
Similarity score percentile
(a) CLIP (no debiasing)
(b) MI debiasing
(c) PBM
Figure 2: Gender bias distribution for different methods using "engineer" as query. We compute similarity scores for all images from the test image set and plot them against gender bias in 1%
quantile increments. The red circle marks the top-K K window covering the final retrieval output Ve.x.
Top-100 gender bias
1.0
Top-1 100 gender bias
1.0
Top- 10-100 gender blas
1.0
0.0
0.0
0.0
-0.5
-0.5
-0.5
Enpiterr
Crg neur
Expreir
-1.0
-1.0
1..0
D.5
-1.0
DD.
0.D
1.0
0.5
Correlation
Correlation
Correlation
(a) CLIP (no debiasing)
(b) MI debiasing
(c) PBM
Figure 3: Comparing top-100 retrieval gender bias with full set similarity gender bias Spearman's correlation. For each query (occupation), we visualize the correlation between similarity score and gender bias against the top- 100 retrieval gender bias. Two typical examples: "engineer" and "administrative assistant" are highlighted for illustration.
see that the correlation between gender and similarity score is close to zero. However, the gender bias is consistently larger than zero across the range of similarity scores, the final retrieval is still biased. A desirable debiasing outcome is that for which the regression line aligns with the dotted line (zero gender bias across the similarity range), shown in Figure 2(c) This is accomplished by the proposed PBM (described below).
Example with multiple queries In order to get more generalizable understanding of the behavior shown in Figure we repeat the analysis with all 45 occupations from the dataset provided by Kay et al. (2015) and visualize the results in Figure 3 Specifically; for each query (occupation). calculate the Spearman's rank correlation between all (test-set) similarity scores and gender bias similar Figure 2 We use Spearman's 's rank correlation because is more appropriate quantify associations between (variable) rankings. Moreover, we also calculate the gender bias of the final retrieval bag (K 100) for each occupation. Consistent with the illustrative findings the "engineer" query, in Figure 3(a) we see large correlation between similarity scores and gender bias as well as a large retrieval-sse gender bias. MI minimization in Figure ((b) manages to reduce the correlation between similarity score and gender but suffers from considerable gender bias; mostly clustered around 0.4. The proposed PBM not only reduces the correlation between similarity scores and gender bias but also significantly pushes the bias of each retrieval result to zero, which is the sought target debiasing outcome.
Insights Our observations can be explained and understood from a theoretical perspective. Previous methods are mostly concerned with making their image features independent of gender information. For a fixed query c, suppose we sample a random image from the test image data distribution Vlest, its feature distribution (v) is independent of gender distribution g(v). Since the query feature (c) is constant, the similarity score S(v, c), defined in Section 3.2] is a deterministic function of fv( and is thus also independent of gender (P). This is why the results for MI-based unbiasing shown in Figure ((b) and B(b)sshow little correlation between similarity scores and gender bias. Similar results for debiasing with adversarial learning be found in the SM.
However, as shown in Figure2(5) there is still a consistent gender bias across similarity score values. This means that the gender distribution in each window tends to manifest the gender distribution of

---

Algorithm/ 1 Post-hoc Bias Mitigation (PBM).
Input: Text query c, retrieval size K, image database Viest similarity measure from pre-traines vision-language models S(,, ), and gender prediction model g(-).
Output: Image retrieval bag Ve,k-
1: Split glest into Vira V-1 and NNA using the gender prediction model g());
2: Let Ve.k = 2
3: while |Ve.xl < K do
4: V+1 arr max S(v,c); V-1 arg max S(v.c); VEV
VNIA arg max S(v,
vEV-I /2> S(VNIA, c) then
ve N/
5: if (((++I.))+((v111)1 +
6: Vax t Ve. (V+1,V-1);
Vir - Vi \ (V+1):
ViT - Vinr (v-1))
7: else
8: Vr.k - Va,k U (VNA):
V < N/A
\ (UNIA):
9:
end if
10:
end while
return Ve,k
the whole test image set Viest which may not be balanced. This is also confirmed in Figure ((b) as the final output bias of most occupations is clustered around 0.4, which is precisely the gender bias of the entire image set Vlest Therefore, ensuring independence. quantified here in terms of correlation, between image features and gender may not guarantee zero gender bias if the test image set itself is biased due to imbalance.
Two types of bias From the examples above, we can differentiate two types of bias, namely, the model bias from training and the bias from the test-time image distribution. The former can be quantified based on the correlation between similarity score and gender and only depends on the training data distribution and the way in which the model is trained. This has been previously described and studied (Caliskan et al. 2017 Zhao et al. 2017]. In comparison, the latter type of bias manifests in the test phase because the test image set (the database from which we retrieve images) does not necessarily have commensurate numbers of male and female samples. We refer this type of bias as test-time bias. In fact, such proportions are usually unknown for image databases.
These two sources of bias coexist and can be addressed separately during model training and inference. Previous methods (Edwards and Storkey 2015) Wang et al. 2021a| Berg al., 2022) have been fairly successful at addressing the first type of bias on the training side, but neglect the test-time bias that is specific to the test set. To tackle the test-time bias, it is necessary find strategy targeting the test image set, so that the retrieved image genders are balanced despite the gender imbalance in the training (source) set. This very insight motivates our approach called PBM, which we describe below. As previously shown in Figures[(c))aaa 2(c) [(c] PBM achieves substantially smaller retrieval gender bias than MI debiasing. Other existing approaches will be considered in the experiments.
3.5 Post-hoe Bias Mitigation (PBM)
To address the second type of bias induced by the imbalanced image test set, one simple idea is sub-sampling. Specifically, we could first sub-sample the image set to make sure its gender ratio is balanced before doing retrieval. However, a clear limitation of such an approach is that some highly relevant images may be dropped during sub-sampling, which may negatively affect retrieval quality. This problem is especially exacerbated if the test set has very large gender bias. Alternatively, our intuition is to sub-sample after computing and ranking similarity scores of all images using a post-hoe method to control gender bias while sampling from the image source set, which we call Post-hoc Bias Mitigation (PBM).
A general version of the PBM algorithm, which is straightforward, is presented in Algorithm For each image V; E gytest, we first predict its gender g(v;) E {+1,-1), which splits the images into two subsets, Vilstt and Viest We then rank images from the two subsets based on their similarity -
+1 scores separately. While forming the retrieval bag, we sample the top of both subsets together in pairs. Specifically, we select the top LK/2] samples from each subset. If K is odd, we randomly pick one of the subsets and select one more top sample from This method ensures low gender bias of the retrieval output as long as gender predictions g(v,) are accurate.
7

---

One extension of our method considers the case where gender may not be applicable to some images. For instance, our image dataset may contain cartoon characters that are not easily associated with any gender, or maybe the gender cannot be determined from the image due to body or facial coverings. In this case, we allow our gender predictions to take N/A values, yielding a NiA subset. Images from VNA do not inherently exhibit visually perceptible bias. Thus, they could be selected in favor of other male-female pairs if their similarity scores are higher and are exempt from bias measurement.
Gender prediction To predict gender g(v), we consider the following two methods for different scenarios. Note that all options are considered in the experiments.
Supervised gender classification: For scenarios where ground-truth gender attributes available, such as MS-COCO (Lin et al. 2014) Zhao et al., (021), we can train a complementary small gender classification network. Using our encoded image feature fa(r) as input, we train a 3-layer MLP to classify gender. For datasets where gender may not be applicable to some samples, we add a N/A
class to the set of labels for predictions.
Zero-shot inference using word embeddings or prompt: For scenarios where supervised training of gender is not possible, we infer gender in zero-shot manner (Radford et al.) 2021 Li
ct al. (022)) using the implicit knowledge embedded in the text features of large pre-trained vision- language models (Radford al. 2021 Li et al. 2022). Given that the semantics of gender and race attributes are already incorporated into the text encoder, we can extract them from word embeddings using words such "Man" and "Woman". Alternatively, we could use prompts prepended to the occupation search query. For example, we add gender-specific adjectives like "Male" or "Female" in front of query. Finally, we compute the (cosine) similarity score of each image v with them, i. e. S(v, "Male" " + c), S(v, "Female" + c). The plus operator (+) here refers to string concatenation. The gender prediction (vv) is then generated by comparing these two similarity scores.
4 Experiments
We first evaluate our image retrieval algorithm on two real-world web image search datasets, Occupa- tion 1 (Kay al. 2015) and Occupation 2 (Celis and Keswani 2020). We also test on two large-scale image-text datasets, MS MS-COCO (in et al.] 20144) and Flickr30k (Plummer et al. 2015 to further validate the effectiveness of PBM in handling more complex text-based image retrieval scenarios.
For comparison, we consider adversarial training (Edwards and Storkey 2015 Berg et al. 2022) and mutual information minimization (Wang et al. [221ap as baseline methods. We also include other types of debiasing methods such as FairSample (Wang et al.) (021a)), which balances the gender distribution of image-text pairs within training batch, and FairExpec (Mehrotra and Celis 2021 denoised selection algorithm designed to select fair subset based on noisy demographic attributes.
We use AbsBias @ KK and Recall @ KK as evaluation metrics as described in Section 3.1
Real-world web image search The first dataset, which we refer to as Occupation Kay et al. 2015), comprises the top 100 Google image search results for 45 nender-neutral occupation terms, such as "chef", "librarian", "primary school teacher", etc. Each image within this dataset is annotated with a crowd-sourced gender attribute (either "male" or "female") that characterizes the person depicted in the image. Occupation 2 (Celis and Keswani (2020). the second dataset, includes the top 100 Google image search results for 96 occupations, where both gender and race (represented as skin-tone: fair and dark skin) annotations are provided. Notably, the gender and race attributes also include a N/A category in Occupation 2, where the annotators have chosen the option of "Not applicable" or "Cannot determine" for the gender or skin-tone represented in the image. Consequently, we treat the images labeled with N/A as neutral examples that do not contribute to the bias of retrieval, since in principle, the users cannot perceive gender racial information from the image.
For these two datasets, we consider OpenAl's CLIP ViT-B/16 (Radford al. 202 as the VL model for all debiasing methods. The baselines for comparison are MI-clip from Wang et al. (2021a). adversarial training adapted from Edwards and Storkey| (2015), and Debias Prompt from Berg et al.) (2022). We have also tailored FairExpec (not originally proposed for TBIR) to our task by integrating with CLIP and our proposed gender predictor 8(). We refer to this adapted model as CLIP- FairExpec in our experiments. We test our PBM method with four variants of gender predictors g(-). as discussed in Section 3.5] These variants include a supervised classifier, zero-shot inference, and ground truth from annotators. The implementation details are provided in the SM.
8

---

Table 1: Results for debiased image retrieval from Occupation and 2 datascts.
Mechod
AbsBizs (2)
Occupation
Recall (00XT)
Gender
Abs (000])
Occuparion
Gender Recalli
Absilias 1000])
Occupation
Recall
Race
Sclection
.5370
31.5
4111
CLJ Oriiina [eatoodd etal.
3566
Mi.elp Wang KV4r
-3708
-2599
Advenamm
Lowwads and Stockey 2012
.2316
.2013
Prongt Pem
.6333
3564
4936
BIL3
CLIP-FailEppe pismoor céli 3621
.2498
2619
.2788
34.7
PEM Zero shot Emtcaing
.198/
49.8
.1500
42.1
3133
4012
PBM Zero shot Prongt
.0560
46.1
.0433
42.5
2271
36.0
PRM -Sigariied Clavaifirr
1404
50.3
1171
42.
1955
37.9
PBM - Grondd-nuth Gender and Skinsooe
.000
49.1
0000
42.4
0000
41.3
Table 2: Results for debiased image retrieval from MS-COCO and Flickr30k datasets.
Gender Bias
Recall
Dalaset
Method
Blas@ ())
((
Bias [O(]]
Recall I(f)
Recal((
Recal TOKT)
SCAN Lee 2018
1250
.044
2506
47.7
82.0
91.0
FairSaninle (ang al.
2021a
.140
1951
.2347
49.7
82.5
90.9
COCO-Ik
CLIP (Radford
2021
.9000
.2024
26488
48.2
77.9
88.0
MI-clip Wang el M
0670
.1541
.2057
46.1
75.2
86.0
Our PB
.0022
.091
.1082
37.3
73.6
84.8
SCAN lee ei al. 2018
1379
.1333
2484
25.4
54.1
67.3
FairSumple Waang
2021a
.1133
1916
.2588
26.8
55.3
68.5
COCO-5k
CLIP (Radiora F3 2041
0070
.1500
21311
28.7
53,9
64.7
MI-clip Wang
0672
.1774
.1611
27.3
50.8
62.0
Our PBAT
.492
.1006
.1212
22.3
50.5
61.9
SCAN (Ler
2018
1098
.3341
.3960
41.4
69.9
79.1
FairSantre Waag eral. 2021a
.0744
2699
.3537
35.8
67.5
77.7
Flick330K
CLIP (Radford di 2021
1150
3150
.3586
67.2
89.1
93.6
Mi-elip Wang el al. 20210
.9600
.2466
.2951
63.9
85.4
91.3
Our PBAY
.0360
.1527
.1640
41.2
85.3
92.6
We summarize the experimental results in Table Compared with other methods, PBM variants achieve significantly lower @bsBias@ 100 while maintaining comparable Recall @ 100 scores. The male:female ratio of images in Occupation 1 is approximately 61:49, thereby mitigating the first type of bias by MI-clip, adversarial training, debias prompt and random selection is not enough. The CL.IP- FairExpec cannot achieve better results since the FairExpec algorithm relies on the independence assumption, while the samples in our test dataset are correlated. This happens because images for occupations are collected from the same Google image search, which is biased. Further, FairExpec requires reliable probabilistic predictions for gender, however, in our setting the gender attributes provided by an off-the-shelf MLP predictor based on visual features. In such a scenario, the label estimates yielded by the off-the-sheif MLP may not be always trustworthy, the domain has shifted during inference. Consequently, these estimates could include misleading information, resulting in undesirable debiasing outcomes for CLIp-FairExpec. Moreover, it should be noted that Debias Prompt (Berg et al., (022) always achieves the highest Recall@ 100, since we utilize their publicly accessible model, which has been fine-tuned the Flickr30k dataset to enhance the performance of text-based image retrieval.
Large-scale text-based image retrieval We consider MS-COCO (Lin et al. 2014) and Flickr30k (Plummer et al.J 2015). Our setup aligns with Wang et al. 2021a), where the gen- der attributes are directly inferred from the text captions of images. We consider the same baseline models Wang et al. (2021a), which are SCAN (Lee al. 2018 and CLIP, as well as their proposed approach, FairSample and MI-clip. It should be noted that FairSample is specifically designed to debias SCAN, a specialized in in-domain VL model. We choose the best performing PBM model, which leverages the pre-trained classifier for gender attributes.
The performance metries in Table 2 are consistent with Wang et al. (2021a), which resemble our AbsBias@ @ K metric, with the omission of using absolute values in the sum This bias metric is computed as Bias @ K = [CT Exec & Evevy g(v). From the Table 2 we see that PBM maintains the bias to minimum even when dealing with intricate text queries or images of complex scenes.
tlasspeeffrmance trade-off analysis In Figure 4 we show the trade-off between retrieval perfor- mance and bias for MI-clip, adversarial training and the four PBM variants. We choose MI-clip and adversarial training for comparison, as these models offer implementation simplicity of adjusting trade-off between AbsBias @ 100 and Recall@ 100. For PBM, we introduce trade-off parameter through stochastic variable representing the probability of opting for fair subset at any given time, as opposed to merely selecting the image with the highest similarity score. Additional details about this experiment provided in the SM. Our results indicate that in range of relatively high

---

Adversorial Training
MI-clip
MbsBlaseaa 810
0.6
AbsBlas- 100
0.4
PBM Supervised Classifier
0.4
PBM Zero shot Embedding
Zero-shat Prompt
0.2
0.2
PBM Ground-truth Genders
Original CLIP
0.0
0.0
45%
48%
50%
52%
55%
58%
60%
0%
20%
30%
40%
50%
Recall@100
Recall@100
Figure 4: Trade-off between Recall@K and AbsBias@E K for debiasing gender attributes within Occupation 1 (Middle) and race attributes using Occupation 2 (Right).
AbsBias@ @ K values, MI-clip and adversarial training are still able to reduce bias while preserving Recall @ K. However, in terms of lowering @bsBias@ K, they struggle to maintain satisfactory TBIR performance. In contrast, the proposed PBM consistently succeeds sufficiently reducing bias and maintaining performance, provided that predictions of attributes reasonably accurate. Details of implementing this trade-off can be found in the S.M.
Discussion
In this study, examined gender and racial bias in text-based image retrieval (TBIR) for neutral text queries. In an attempt to identify bias in the test-time (inference) phase, we conducted an in-depth quantitative analysis on bias reduction, alongside existing debiasing methods and the proposed PBM. We concluded that solely addressing training- timen model-encoded bias is not sufficient for obtaining equal representation results, because test-time bias also exists due to imbalance in the test image set used during retrieval. So motivated, we proposed Post-hoc Bias Mitigation (PBM), a straightforward post-processing method that aims directly alleviate test- est-time bias. Experiments on multiple datasets show that our method can significantly reduce bias while maintaining satisfactory retrieval accuracy the same time.
Moreover, the potential impact of PBM extends far beyond the initial scope of text-based image retrieval systems. The core concept of our methodology can be seamlessly adapted wide variety of information retrieval systems, such as image-based text retrieval or enerr-by-example image retrieval, as long as the demographic information of the test is accessible or can be estimated, egg., via zero-shot inference. Overall, our approach is not limited to enhancing fairness in text-based image retrieval, thus can be extended a broad range of VL model applications.
Limitations
Some limitations of the proposed method are duly acknowledged. Firstly, the efficacy of our approach dependent upon the availability of sufficient number of examples for each category (gender or racial attribute) within the test image set. Our work currently does not consider any techniques, such as using synthetic samples, to mitigate the issues arising from insufficient representations of certain demographic groups. Secondly, the debiasing effect of PBM pertains to the predictability and accessibility of demographic attributes. Attempts at debiasing religious representation, or other socio-cultural factors identities, in images speech present significant challenges, because the predicting or securing annotations regarding religious information, can be exceptionally difficult. Thirdly,
prioritize the "equal representation", to ensure a retrieval bag that contains equal representations of each demographic group. This compromises fairness towards content providers in the image retrieval process. From the standpoint of content providers, fairness should imply that similar samples are treated similarly, regardless of the demographic group membership of their provided samples. Lastly, our work assumes all queries are neutral. We do not develop a technique to identify if query is neutral or not, thereby limiting the applicability of our method in hybrid text query retrieval where text query can be biased like "Male doctor". It is important to note, however, that the above challenges are not unique to our method but a common issue encountered in other debiasing approaches. A more comprehensive discussion on the limitations of our work is available in the SM.
retrieval results sacrifice recall performance
10

---

Acknowledgements
The authors would like to thank the anonymous reviewers for their insightful comments. This research was supported by ONR N00014-18-1-2888 100014-18-1-2881-P000020
References
Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. 2016. Machine bias. In Ethics of data and analytics. Auerbach Publications, Boca Raton, FL, USA, 254-264.
Yalong Bai, Wei Yu, Tianjun Xiao, Chang Xu, Kuiyuan Yang, Wei-Ying Ma, and Tiejun Zhao. 2014. Bag-of-words based deep neural network for image retrieval. In Proceedings of the 22nd ACM international conference on Multimedia. Association for Computing Machinery. New York, NY, USA, 229-232.
Hugo Berg, Siobhan Mackenzie Hall, Yash Bhalgat, Wonsuk Yang, Hannah Rose Kirk, Aleksandar Shtedritski, and Max Bain. 2022. A prompt array keeps the bias away: Debiasing vision- laiiun-a models with adversarial learning. arXiv preprint arXiv:2203. 1933 (2022).
Alex Beutel, Jilin Chen, Tulsee Doshi, Hai Qian, Li Wei, Yi Wu, Lukasz Heldt, Zhe Zhao, Lichan Hong, Ed H Chi, et al. 2019. Fairness recommendation ranking through pairwise comparisons. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining. Association for Computing Machinery, New York, NY, USA, 2212-2220.
Joy Buolamwini and Timnit Gebru. 2018. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on fairness, accountability and transparency. PMLR, 77-91.
Aylin Caliskan, Joanna Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from language corpora contain human-like biases. Science 356, 6334 (2017), 183-186.
Flavio Calmon, Dennis Wei, Bhanukiran Vinzamuri, Karthikeyan Natesan Ramamurthy, and Kush R Varshney. 2017. Optimized Pre-Processing for Discrimination Prevention. In Ad- vances in Neural Information Processing Systems, Guyon, U. Von Luxburg, S. Ben- gio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gameet (Eds.), Vol. 30. Curran Associates, Inc. https://proceedee http///roooedingg........ppppppp neurips..... cc/cc//c/p/// hcc/pappr_fiieee wwlrrffiessppppaee Seteeerfflee//ppee////// pptp:/ppp//////f/// (a44a222d44a4$ffffffeee444444433
Paper.
Min Cao. Shiping Li, Juntao Li, Liqiang Nie, and Min Zhang. 2022. Image-text retrieval: A survey on recent research and development. arXiv preprint arXiv:2203. 14713 (2022).
L Elisa Celis and Vijay Keswani. 2020. Implicit diversity in image summarization. Proceedings of the ACM on Human- Compute Interaction 4, CSCW2 (2020), 1-28.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for contrastive learning of visual representations. In International conference on machine learning.
PMLR, 1597-1607.
Xinlei Chen, Hao Fang. Tsung-Y Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Ziticck 2015, Microsoft coco captions: Data collection and evaluation server. arXiv preprint 0tt:::1554000332 (2015).
Pengyu Cheng. Weituo Hao, Siyang Yuan, Shijing Si, and Lawrence Carin. 2021. Fairfil: Contrastive neural debiasing method for pretrained text encoders. arXiv preprint arXiv:2103 103.040 13 (2021).
Ching-Yan Yao Chuang, Varun Jampani, Yuanzhen Li, Antonio Torralba, and Stefanie Jegelka. 2023. Debiasing vision-language models via biased prompts. arXiv preprint arXiv:2302.00077 (2023).
Andrew Cotter, Heinrich Jiang, Maya R Gupta, Serena Wang. Taman Narayan, Seungil You, and Karthik Sridharan. 2019. Optimization with Non-Differentiable Constraints with Applications to Fairness, Recall, Chum, and Other Goals. J. Mach. Learn. Res. 20, 172 (2019), 1-59.
11

---

Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn. Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1 1929 (2020).
Harrison Edwards and Amos Storkey. 2015. Censoring representations with an adversary. arXiv preprint wwt.iv:/5/I..0 (2015).
Fartash Faghri, David Fleet, Jamie Ryan Kiros, and Sanja Fidler. 2017. Vse++: Improving visual-semantic embeddings with hard negatives. arXiv preprint arXiv:1707.05617 (2017).
Sorelle Friedler, Carlos Scheidegger, and Suresh Venkatasubramanian. 2014. Certifying and removing disparate impact. arXiv preprint arXiv: 1412.3756 (2014).
Andrea Frome, Greg S Corrado, Jon Shlens, Samy Bengio, Jeff Dean, Marc' Aurelio Ranzato, and Tomas Mikolov. 2013. Devise: A deep risual-semantic embedding model. Advances in neural information processing systems 26 (2013).
Gurobi Optimization, LLC. 2023. Gurobi Optimizer Reference Manual. httpss:/www.gurrdt
comm
Melissa Hall, Laura Gustafson, Aaron Adcock, Ishan Misra, and Candace Ross. 2023. Vision- Language Models Performing Zero-Shot Tasks Exhibit Gender-based Disparities. arXiv preprint arXiv:2301.11100 (2023).
Moritz Hardt, Eric Price, and Nati Srebro. 2016. Equality of opportunity in supervised learning. Advances in neural information processing systems 29 (2016).
Lisa Anne Hendricks, Kaylee Burns, Kate Saenko, Trevor Darrell, and Anna Rohrbach. 2018. Women also snowboard: Overcoming bias in captioning models. In Proceedings of the European conference on computer vision (ECCV). 771-787.
Ajil Jalal, Sushrut Karmalkar, Jessica Hoffmann, Alex Dimakis, and Eric Price. 2021. Fairness for image generation with uncertain sensitive attributes. In International Conference Machine Learning. PMLR, 4721-4732.
Matthew Kay, Cynthia Matuszek, and Sean A Munson. 2015, Unequal representation and gender stereotypes in image search results for occupations. In Proceedings of the 33rd annual acm conference on human factors in computing systems. 3819-3828.
Jac Myung Kim, Koepke, Cordelia Schmid, and Zeynep Akata, 2023. Exposing and Mitigating Spurious Correlations for Cross-Modal Retrieval. arXiv preprint arXiv:2304.0339) (2023).
Matt J Kusner, Joshua Loftus, Chris Russell, and Ricardo Silva. 2017. Counterfactual faimesss
Advances in neural information processing systems 30 (2017).
Kuang-Huet Huei Lee, Xi Chen, Gang Hua, Houdong Hu, and Xiaodong He. 2018. Stacked cross attention for image-text matching. Proceedings of the European conference on computer vision (ECCV). 201-216.
Michael S Lew, Nicu Sebe, Chabane Djeraba, and Ramesh Jain. 2006. Content-based multimedia information retrieval: State of the art and challenges. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM) 2, (2006), 1-19.
Chunyuan Li, Haotian Liu, Liunian Li, Pengchuan Zhang, Jyoti Aneja, Jianwei Yang, Ping Jin, Houdong Hu, Zicheng Liu, Yong Jae Lee, et al. 2022. Elevater, A benchmark and toolkit for evaluating 35 (2022), 9287-9301.
visual models. Advances in Neural Information Processing Systems
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13. Springer, 740-755.
12

---

Anay Mehrotra and L Elisa Celis. 2021. Mitigating bias in set selection with noisy protected attributes. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency. 237- 248.
Anay Mehrotra and Nisheeth Vishnoi. 2022, Fair Ranking with Noisy Protected Attributes. Advances in Neural Information Processing Systems 35 (2022), 31711-31722
Marco Morik, Ashudeep Singh, Jessica Hong, and Thorsten Joachims. 2020. Controlling fairness and bias in dynamic learning-to-rank, In Proceedings of the 43rd international ACM SIGIR conference on research and development information retrieval. 429 438.
Jishnu Mukhoti, Tsung- Yu Lin, Omid Poursaeed, Rui Wang, Ashish Shah, Philip HS Torr, and Ser-Nam Lim. 2022. Open Vocabulary Semantic Segmentation with Patch Aligned Contrastive Learning. arXiv preprint wxXiv:221.04499 (2022).
Safiya Umoja Noble. 2018. Algorithms of oppression. In Algorithms of oppression. New York University Press.
International Labour Organization. 2019. quantum leap for gender equality: For a better future of work for all. (2019).
Otávio Parraga, Martin D More, Christian M Oliveira, Nathan S Gavenski, Lucas S Kupssinskü, Adilson Medronha, Luis V Moura, Gabriel S Simóes, and Rodrigo C Barros. 2022. Debiasing Methods for Fairer Neural Models in Vision and Language Research: A Survey. arXiv preprint arXiv:2211.05617 (2022).
Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. 2015. Flickr30k entities: Collecting region-t-phrass correspondences for richer mmage-to-sentence models. In Proceedings of the IEEE international conference on computer vision. 2641-2649.
Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learning. PMLR, 8748-8763.
Nripsuta Ani Saxena, Karen Huang, Evan DeFilippis, Goran Radanovic, David C Parkes, and Yang Liu. 2020. How do fairness definitions fare? Testing public attitudes towards three algorithmic definitions of fairness in loan allocations. Artificial Intelligence 283 (2020), 103238.
Ashudeep Singh and Thorsten Joachims. 2017. Equality of opportunity in rankings. In Workshop on Prioritizing Online Content (WPOC) at NIPS, Vol. 31.
Kulwinder Singh. Ming Ma, and Dong- Won Park. 2003. Content-based Image Retrieval using FFT & Cosine Similarity Coefficient.. In SIP. 315-319.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems. 1000-60110
Jialu Wang, Yang Liu, and Xin Eric Wang. 2021a. Are gender-neutral queries really gender-neutral? mitigating gender bias image search. arXiv preprint arXiv:2 109.05433 (2021).
Jialu Wang, Yang Liu, and Xin Eric Wang. 2021b. Assessing multilingual fairness in pre-trained multimodal representations. arXiv preprint arXiv:2106.00 06683 (2021).
Junyang Wang, Yi Zhang, and Jitao Sang. 2022. FairCLIP: Social Bias Elimination based on Attribute Prototype Learning and Representation Neutralization. arXiv preprint arXiv:2210. 14562 (2022).
Rui Wang, Pengyu Cheng, and Ricardo Henao. 2023. Toward Fairesss in Text Generation via Mutual Information Minimization based on Importance Sampling. arXiv preprint arXiv:2302. 13/36 (2023).
13

---

Han Xu, Xiaorui Liu, Yaxin Li, Anil Jain, and Jiliang Tang. 2021. To be robust or to be fair: Towards fairness in adversarial training. In International Conference on Machine Learning. PMLR, 11492-11501.
Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez Rodriguez, and Krishna P Gummadi. 2015. Learning fair classifiers. arXiv preprint arXiv:1507.05255 1, 2 (2015).
Syed Ali Jafar Zaidi, Attaullah Buriro, Mohammad Riaz, Athar Mahboob, and Mohammad Noman Riaz. 2019. Implementation and comparison of text-based image retrieval schemes. International Journal of Advanced Computer Science and Applications 10, (2019), 61 1-161.
Dora Zhao, Angelina Wang, and Olga Russakovsky. 2021. Understanding and evaluating racial biases in image captioning. In Proceedings of the IEEE/CVF International Conference Computer Vision. 14830-14840.
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2017. Men also like shopping: Reducing gender bias amplification using corpus-level constraints. arXiv preprint arXiv: (2017).
14

---

Appendix
A Impact of Demographic Group Classifier on Debiased Results
Table 3: Group sensitivities and sensitivity ratios (p) for demographic attributes predicted by different classifiers Occupation Gender and Occupation 2 Race.
Gender
Race
Method
Male Sensitivity
Fomale Sensitivity
P
Light skin Sensitivity
Dark skin Sensitivity
PBM - Supervised Learning
0.97
0.88
1.10
0.93
0.84
111
PBM Word Embedding
0.98
1.04
0.84
0.78
1,88
PBM Zero-shot Prompt
0.98
0.97
1.01
0.88
0.81
1.09
Original CUP Abs0las@100
AbsBios@100
Original CLIP AbsBlaz@110
Ratio p 1.12
0.2
0.8 F1 score (pertarmance)
0.9
0.7
2 Sansiiivity ratio
Relationship between the perfor- mance of the demographic group classifier (F1- score) and the retrieval bias (AbsBias @ 100) when utilizing PBM.
Figure 5:
Figure 6: Relationship between the bias of the demographic group classifier (ratio of male sen- sitivity to female sensitivity) and the retrieval bias (AbsBias 100) when utilizing PBM.
The demographic group classifier is an important module of our proposed method PBM. The debiasing result is intricately linked to the demographic group classifier's accuracy and prediction bias towards different demographic groups. Figure showcases the relationship between the demographic group classifier's 's performance and the ensuing retrieval bias, by artificially introducing noise to the demographic group (logit) predictions via Gaussian noise with a standard deviation ranging from (no noise) to 1. These results underscore that better group classifier performance yields lower bias, that bias converges to that of the original CLIP as the group classifier gets worse, and importantly, that the bias after PBM will be no worse than that of the original CLIP.
Further, Table 3 shows the individual demographic group sensitivities under three different scenarios, from which we can see that the group classifier is i) able to achieve good classification sensitivity (no lower than 0.81 and 0.90 in average), likely because demographic image attributes (gender and skin tone) are typically captured in images, and ii) that different scenarios exhibit different degrees of bias as measured by the group sensitivity ratio, which must be close to I for the model be unbiased. Table reveals sensitivity discrepancies among different attributes. To delve deeper into the influence of classifier bias on PBM outcomes, we present the retrieval bias as function of the sensitivity ratio in Figure 6 This is achieved by altering the gender classification threshold from 0 (maximizing male sensitivity) to 1 (minimizing male sensitivity). From Figure we can conclude that the classifier bias does affect retrieval bias, however, only severely for more extreme sensitivity ratios, which is fortunately not the case our results as shown in Table
B Bias-recall Trade-off Strategies
In Figure we exhibit the bias-recall trade-off curves for MI-clip, Adversarial Training, and various PBM methods. Here, outline the missing details to achieve these trade-offss.
For adversarial learning, the trade-off is controlled by adjusting the adversarial loss weights between 0 and 1.0. In MI-clip, we modify the clipped dimensions from 10 to 500 (CLIP output dimension 512). Regarding PBM methods, a trade-off parameter is introduced via stochastic variable which denotes the likelihood of choosing a fair subset at any given time, instead of simply opting for the image with the top similarity score. Each curve is plotted by interpolating 10 points of the corresponding trade-off parameters.
15

---

C Datasets
Occupation (Kay et al., 2015) Occupation comprises the top 100 Google Image Search results for 45 gender-neutral occupation terms, such as "chef", "librarian", "primary school teacher", etc. Each image within this dataset annotated with crowd-sourced gender attribute (either Male or Female) that characterizes the person depicted in the image. The entire Occupation 1 dataset is exclusively utilized for evaluating gender debiasing effect, as shown in T
Occupation 2 includes the top 100 Google Image Search results for 96 occupations. Each image in the dataset comes annotated with gender attribute and a race attribute (represented by skin-tone, namely, Fair Skin and Dark Skin). Notably, the gender attribute and race attribute also include a N/A category, where the annotators have chosen the option of "Not applicable" or "Cannot determine" for the gender or skin-tone depicted in the image. Consequently, we treat the image labeled with N/A as a neutral example that does not contribute to the bias of retrieval, since the user cannot perceive gender racial information from the image. Different from Occupation 1, gender attributes in Occupation are categorized as [Male, Female, N/A), while race attributes classified (Fair Skin, Dark Skin, N/A). The enitre Occupation dataset is only used for evaluation on mitigating gender and race bias, as the results shown in Table
Occupation 2 (Celis and Keswani, 2020)
MS-COCO (Lin et al., 2014) The first large-scale image-text dataset is MS-COCO captions dataset, which is partitioned into 113,287 training images, 5,000 validation images, and 5,000 test images. Each image is accompanied by five corresponding captions. Our experimental setup aligns with the methodology detailed by [aan et al. (202 Only the first caption of each image is used for evaluation. Further, they ensure all captions are gender-neutral by identifying and replacing or removing gender-specific words with corresponding neutral terms, with the help of predefined word banks (Zhao al. 2017 Hendricks al. 2018).
Flickr30k (Plummer et al. 2015) The second large-scale image-text dataset employed in our experiment is Flickr30K, which contains 31,000 images obtained from Flickr. Adhering to the partitioning scheme presented in Plummer et al. (2015), we allocate 1,000 images each for validation and testing, with the remaining images designated for training. We obtain the ground truth of gender attributes of images in Flickr30k in the same way as MS-COCO (Wang et al., 2021a), as we detect the gender-speeific words in the caption to determine the gender attributes of its corresponding image.
D Baseline Models
Random Select To simulate an ideal scenario, where image features bear no dependency to gender (and race) attributes, for a neutral query c, we randomly select K candidates from the true relevant image set V2, with replacement. . As for each query c, the size of the relevant image set [V= is most 100. Using sampling with replacement simulate the situation that the gender attribute distribution is fixed, and irrelevant retrieval algorithm. We report AbsBias@ K for reference. The Recall@ K is omitted since the value is meaningless, as we only sample from the true relevant image set V
CLIP (Radford et al., 2021 We consider OpenAl's CLIP ViT-B/16 (Radford et al. 2021 the VL model for all debiasing methods. Specifically, the image encoder fo( (-) is Vision Transformer (ViT) (Dosovitskiy et al. 2020)) comprising 12 transformer blocks of width 768, with 12 self-attention heads in each block. ViT processes images of size 224 x 224 by dividing them into 16 x 16 patches and outputs 5 -i-dimensional image features by linear projection. The text encoder f is a standard text transformer Vaasaani et al. (017)) with masked self-attention, consisting of 12 transformer blocks of width 512 and self-attention heads in each block, with a linear projection layer the end well. The CLIP ViT-B/16 model is loaded with pre-trained weights provided by OpenAl (Radford et al. 2021. All the following debiasing methods use this pre-trained CLIP VIT-B/16.
MI-clip (Wang et al., 2021a) MI-clip (Wang et al., 202 clips the fixed number of output dimensions of the image encoder in CLIP to reduce the mutual information between image features and demographic attribute distribution. For MI-clip in Table I we clip 312 dimensions of output image features. These 312 dimensions were chosen by examining the reduction in bias and while maintaining retrieval performance. We also show a trade-off between bias reduction and retrieval
16

---

performance of MI-clip with the number of clipped dimensions from 100 to 500 on the Occupation 1 dataset in Figure
Adversarial Training (Edwards and Storkey, 2015) For adversarial training, we use the same minimax problem setup as in (Edwards and [ The encoder is the original CLIP image encoder. The decoder is realized by a ViT with vision transformer blocks, and the adversarial predictor is 3-layer MLP. Training employs the same loss function in Edwards and Storkey (2013), where the final loss is the sum of the cost of reconstructing v from f& (V), a measure of dependence between (v(v) and g(v) and the error of target task (i.e., image-text aligning loss XNT- (Radford eet al. 202T)). We assign different weights to the loss of dependence measuring loss, in order to demonstrate the trade-off between bias reduction and retrieval performance. We report the adversarial learning results when the weight of measuring dependence loss is 0.7, and the weights for the other two losses are both 0.15. Additional weight combinations were considered and shown in Figure 4
Debias Prompt (Berg et al., 2022) The Debias Prompt method (Berg et al. 2022) also leverages an adversarial learning framework. However, instead of just fine-tuning the image and text encoders, they also prepended zero-initialized learnable prompts before inputting query tokens. Considering that their evaluate their pre-trained model (sourced from their github repository https: //github. com/ httd//eiiasviiiiinnoo lang) on the Occupation I and Occupation 2 datasets.
model is already debiased for gender and race attributes, we directly
Cxxp-FairExpee (Mehrotra and Celis 2021) We tailor FairExpec (not originally proposed for TBIR) to our task by integrating it with CLIP and our proposed gender predictor g(.). We refer to this adaptation CL.PP- Cl.I--airExpec in our experiments.
Using binary gender as an example for simplicity, our CLIP-FairExpec treats the image-text similarity output as the utility score for each image. Then, the objective of the optimization is to maximize the total similarity scores for selecting K images corresponding to a query c. The noise estimate in the original FairExpec is derived from the probability output of our attribute predictor (v). We use the probability output from the attribute predictor (v) as the noise estimate Also, there is a constraint the sum of the noise estimates q such that the sum is at least L - öK and at most U U-6K SK, where I. and U is the lower bound and upper bound for the sum of the noise estimate, respectively. 5 E (0. 1) is noise tolerance level, that controls how much the constraints can be violated due to the presence of noise. Since our fairness objective is equal representation for each gender attribute class, we wish the sum of noise estimate for each class of gender attribute equal. Hence, we set the L = U = k/2. In order to force our model to prioritize minimizing bias over maintaining performance, we choose very small 0.001. We select K images from V with respect neutral query based on the above constrained optimization problem. Further, each selection is solved by the GUROBI solver (Gurobi Optimization, LLC 2023). Upon solving for all selections for queries in C, we compute the AbsBias @ K and Recall @@ @ presented in Table
SCAN (Lee al. 2018) We consider the Stacked Cross Attention Network (SCAN) (Lee et al. 2018) as an alternative VL model to CLIP. SCAN is a specialized in-domain training model, so is trained on the MS-COCE training dataset and tested on the MS-COCO test dataset. Similarly, for the experiments with Flickr30k, the model is trained on the Flickr30k training set and then tested on the Flickr30k test set. We use official implementation of SCAN from https://githe:n com/ kuanghuei/Soe
To mitigate bias during the training of SCAN, we implement the FairSample approach as recommended by Wang et al. (202 We maintain the same hyperparameters settings as Lee et al. (2018). To address the bias arising from the unbalanced gender distribution within training batches, FairSample is proposed in the following way: for every positive image-ter text pair (v, c) within a training batch, we first identify if the query c is gender- or gender-specific. If the training query e gender-neutral, a negative image is sampled from either the male or female image sets, each with probability of 1/2. However, if the query is gender-specific, we maintain the original negative sampling strategy, thereby preserving the model's ability to generalize effectively on such queries.
FairSample (Wang et al. 2021a)
17

---

E Post-hoc Bias Mitigation (PBM)
E.1 Engineering Details
PBM Supervised Classifier We can determine the gender attributes with pre-trained image classifier. Here, the image classifier is pre-trained on MS-COCO training set with gender attribute annotations from Zhao et al. (2021) The image classifier is 3-layer multi-layer perceptron (MLP) as shown Table that takes the image representation from the original CLIP as input. We empirically show that the image classifier can be highly accurate even using light-weight classification MLP. The F1 score for gender attribute prediction is 92.8%
PBM Zero-Shot Embedding We describe the first of the two types of zero-shot inference described in Section 3.5 For zero-shot inference based on the embedding approach, we choose the text embeddings for ["Unknown Gender", "Man". "Woman") tokens to classify the gender attributes of images, and ("Unknown Skin", "Fair Skin", "Dark Skin") for categorizing race attributes. The gender or race attribute of image is determined by which text embedding has the maximum similarity score to the image representation (v)
PBM - Zero-Shot Prompt For the second zero-shot inference described in Section 3.5 namely, the prompt method, we prepend adjectives to the text query c. We use 1'T,, "Male", "Female"| for gender attributes and ( "Fair-skinned", "Dark-skinned") for race attributes. The gender attributes for each image retrieved by the query с determined by which prompted query has the maximum similarity score the image representation fv
PBM Ground- Truth Attribute (Gender or Skin-tone) We use the annotations in the dataset as the predicted attributes g(v) for reference. This shows the upper- bound performance of our method if all gender predictions are correct (known).
E.2
Additional PBM Results
In T. we showcase the results of applying PBM to CLIp models that has been debiased by other approaches, such as MI-clip, Adversarial Learning, and Debias Prompt. When PBM utilized in conjunction with other debiasing strategies, it exhibits a unique bias-recall trade-off, thus catering to a variety of application scenarios.
Table Results of applying PBM Supervised Learning on modified or fine-tuned CLIP.
Occupation Gender
Occupation 2 Race
Method
AbsBias 100 (L)
Recall 100(t)
AbsBias@ 100(L)
Recall 100(1)
PBM
.1404
50.3
.0955
37.9
MI-clip - PBM
.0780
42.1
.0737
29.1
Adversarial Training PBM
1000
39.6
.0997
35.7
Debias Prompt- PBM
.1711
52.1
.1035
40.6
F
Neural Network Architectures
We summarize the details of the neural networks employed in our experiments Table 5 For Image Encoder, the Patch Extraction (dimensions: 16, 16) extracts 196 non-overlapping 16 x 16 patches from the 224x224 image. These extracted patches are subsequently flattened. The subsequent Positional and Linear Embedding (768) maps these patch vectors onto 768-dimensional space and adds 2D positional embeddings patches to the 768-dimensional vectors. Next, 12 Vision Transformer Blocks (768, 12) processes the 768 168-dimensiionl embeddings. Each of these blocks features 12 eeff-attention heads. Lastly, the output embedding is obtained from a unique classification token ([CLS]) that add to the input sequence of patch embeddings. The output from [CLS] Token x 768 is then reduced from 768 dimensions to 512 dimensions using a Linear Projection (512).
Similarly in the Text Encoder, the initial phase involves Positional and Token Embedding (512). This step maps each token in the input text onto 51 512-dimensional vector space and integrates positional embeddings into these vectors. Following this, the text encoder employs 12 Transformer Blocks
18

---

Table 5: The architecture of each component of CLIP and the MLP used in our experiments.
ImageEncoder())
Layer
Type
Patch Extraction( 16, 16)
2
Positional and Linear Embedding(7666)
15
Vision Transformer Blocks(768, 12)
16
[CLS] Token I 768
Linear Projection (512)
TextEncoderr
Layer
Type
Positional and Token Embedding (512)
2- 13
Transformer Blocks (512, 8)
14
[CLS] Token 1 512
15
Linear Projection (512)
MLP(.)
Layer
Type
fc-512 + BatchNorm ReLU()
2
fc-512 + BatchNorm ReLUU
3
fc-512 + BatchNorm ReLUO
4
fc-n_class Softmax()
(512, 8) to process these 512-dimensional embeddings. Each of these blocks contains 8 self-attention heads. Finally, the output embedding is derived from [CLS] Token I 512. The subsequent Linear Projection (512) then maps the extracted text representation onto the multi-modal embedding space that aligns with the image embeddings.
G Computation Resources
All of our experiments
NVIDIA TITAN Xp 12GB GPU with CUDA version 11.5.
H Code and Data Availability
Occupation I dataset is available at http:p///tttttt.oo/////////eeeee
Occupation 2 dataset can be downloaded from 7IN-EE88-7RCC--25000LLLLL
MS-COCO dataset can be access through http:://oooodaaastt.oggooooo and its
crowd-sourced gender/racial annotations from http:://prinnttarissaaaaaa http::/wwwwaai.igguuu....u imagecaptioning- bbas/
Flickr30k dataset can be access via wathttts///shhnonn cs. illinois....... www.:ooiseecu//anatttttitooooooo eduu/DenttationGraph/ And the gender word banks to identify the gender attributes of Flickr30k's images is avaliable in the Appendix of the paper by Wang al. (2021a).
I Broader Impact
The recent years constituting what can be called the model architecture unifying era, witnessed seismic shift from small task-specific models foundation models containing billions of parameters, with numerous applications deployed based such large models. However, as artificial intelligence (AI) systems become more prevalent, the challenging question of fairness becomes more urgent. The concept of fairness in machine learning revolves around creating algorithms and models that DO NOT discriminate against certain groups based on gender, race, socioeconomic status, or any other potentially biasing factors. As machine learning algorithms increasingly used in decision-making makinn
19

---

processes, from job applications, college admissions, to criminal justice and healthcare, subsets of the population who represent minorities may see unfavoring model performance compared individuals in majority groups. Therefore it is imperative to develop unbiased machine learning systems such that decisions are made fairly and equitably. Our study concerning fair image retrieval, among many other fairness research works, can be used to inform policymakers about the potential risks and benefits of Al systems, potentially enacting new laws and regulations to ensure that these systems are utilized responsibly and ethically.
Specifically, the biased performance model is possibly caused by statistical skewness both in the training and testing sets. Existing methods mainly focus on enforcing independence between the model's output and sensitive attributes during training. However, much less effort has been made to mitigate bias during test- rest-time, a potentially vital component of the debiasing procedure. Many machine learning systems are deployed setting where the biased testing set is almost guaranteed, and as such, may suffer from fairness concerns. Importantly, PBM is able to dissociate the ranking similarity from sensitive/protected attributes (e.g., gender) thus reducing bias, meaning that image candidates share an equal chance to be retrieved even in unbalanced testing set. We do not claim that PBM guarantees fairness, and there is always the risk that may be misinterpreted or exploited, but we hope that PBM encourages more inclusive approach to Al development.
20


Reviews:
[{"review_id": "ShiY1zPUgj", "reviewer": "Reviewer_KCNe", "summary": "This paper proposes a post-hoc fairness method for text-to-image retrieval tasks. Instead of pre-processing the dataset or in-processing via fairness-derived regularization, this paper focuses on the post-processing of the retrieved items. The proposed post-hoc bias mitigation (PBM) utilizes a pre-trained bias estimator, e.g., a gender classifier. Using the gender classifier, PBM first classifies the gallery set (or test set) into male / female / NA sets. Then, it samples the final retrieval items from each set by balancing each gender for the final retrieval results. The experimental results show that the proposed PBM can show great bias mitigation while showing small performance drops for the target task.", "questions": "As my previous comment, I would like to expect more discussions related to the notion of fairness chosen in this paper. In my opinion, the notion of fairness can vary by situation.", "limitations": "I don't think this paper has a potential negative societal impact.\n\nAs my previous comment, the proposed method has fundamental performance limitations caused by (1) the need of an accurate gender classifier (2) the nature of the post-processing method (PBM does not update the original model, therefore, PBM cannot improve R@K).\n\nI think this paper is a generally okay paper, while its strengths slightly outweigh its flaws. In my opinion, this paper would become stronger if discussions of the target fairness scenario and other fairness scenarios when the listed limitation would be / not be critical.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper addresses an important problem in vision-language tasks, especially for text-to-image retrieval tasks.\n- The analysis of the source of the bias (\"Two Types of Bias\") is insightful and well-described.\n- The proposed method is easy-to-implement and practical. Moreover, as this method is a post-processing method that does not require a re-training of the model, this method can be readily applied to any text-to-image retrieval method.\n- Although the proposed method highly relies on the performance of the pre-trained gender classifier, this paper extensively studied many possible alternatives of the gender classifier, e.g., zero-shot embedding or zero-shot prompt. It is also pleasurable to see the \"oracle\", or the upper bound of the proposed method, by employing ground-truth gender labels. It shows the advantages and disadvantages of the proposed method more clearly.\n- This paper is easy-to-follow", "weaknesses": "- The proposed method needs a pre-trained gender (bias) classifier. As shown in the experiments, the performances of PBM heavily rely on the quality of the gender (bias) classifier. Although this paper shows a generalizable approach using zero-shot prompts, I think it is an unavoidable weakness of this paper.\n- Compared to in-processing methods, PBM shows worse target task performances. I think it depends on the target objective of the actual user: if one needs a highly performed text-to-image retrieval system (i.e., with high recall) while allowing somewhat moderate discrimination, in-processing methods would be better. On the other hand, if one needs a strictly fair text-to-image retrieval system while sacrificing recall performances, the proposed PBM would be better. I would expect related discussions in the future revision, if possible.\n- As another fundamental limitation, PBM cannot improve the original target task (e.g., Recall@K) because it is a post-processing method.\n- The notion of fairness can be written in multiple ways. For example, the target fairness problem in this paper is similar to \"demographic parity (DP)\" problem, rather than \"equalized odds (EO)\". As far as the reviewer understood, mutual information minimization (MI) is more focusing on EO, while the proposed method is focusing on DP. In my opinion, both EO and DP are valuable problems, while a user has to choose a better fairness notion by considering the user's application. I think the problem raised by this paper is very important, especially considering that it is a retrieval task, not a classification task, but in my opinion, it would be helpful for the readers to discuss the target fairness scenario and the benefit of the target fairness scenario.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **Response to Reviewer KCNe**\nThanks for your detailed review and constructive feedback. Before reading our dedicated response to your comments, we kindly suggest you start with our global response, which offers a broad overview and addresses some of the general concerns raised. \nBelow is our response to your comments,\n\n#### **\"The proposed method needs a pre-trained gender (bias) classifier. ...\"**\nWe agree with your observation that our PBM method's performance is contingent upon the quality of the demographic group classifier. In our global response, we argue that our goal is to mitigate perceivable bias in web image search system, which alleviates the chances of the group classifier for having poor performance because perceivable attributes are in principle easy to classify. Further, in the Table 1 (rebuttal), we empirically show that the group classifier works well across gender and race demographic attributes in Occupation 1 and 2 datasets. We have not tested our approach with demographic attributes that are difficult to predict, thus this will be discussed and emphasized as a limitation in the revision.\n\n#### **\"Compared to in-processing methods, PBM shows worse target task performances. ...\"**\nWe acknowledge that our PBM eliminates bias at the cost of causing retrieval performance drops that are more severe than other in-processing methods. However, we provide a simple bias/recall trade-off strategy by introducing $\\theta$, a parameter controlling the likelihood of obtaining a fair set. As a result, $\\theta$ can be used to prioritize recall as needed. We present the results of bias-recall trade-off curve in Figure 4 (paper). We will include a dedicated section in revision to discuss trade-off strategy, including how to choose $\\theta$ under different application context and pseudo-codes of PBM trade-off algorithm, to avoid confusions.\n\n#### **\"As another fundamental limitation, PBM cannot improve target task...\"**\nWe acknowledge that our PBM is unable to improve the target task performance with a fixed underlying image retrieval model. However, one can fine-tune the CLIP on the target task dataset. In Table 2 (rebuttal), we show the results of applying PBM on a fine-tuned or modified CLIP, including MI-clip, Adversarial Learning and Debias Prompt. The PBM - Debias Prompt slightly improve the recall since the Debias Prompt model is fine-tuned on additional data. We will emphasize this point in the limitation section of the revision.\n\n#### **”The notion of fairness can be written in multiple ways. ...“**\nWe acknowledge that fairness can be defined in a multitude of ways under different contexts. This will be emphasized in the revision. In our paper, fairness is defined as \"equal representation\", i.e., that the output retrieval bag should include an equal number of samples from each demographic group. We write down the mathematical definition of \"equal representation\" used in our paper as follows,\n\n$$\n\\mathbb{E}_{V_c \\sim P}[\\mathbb{E}_v[\\mathbb{1}(g(v)=+1)]]  \n$$\n\n$$\n\\qquad = \\mathbb{E}_{V_c \\sim P}[\\mathbb{E}_v[\\mathbb{1}(g(v)=-1)]]\n$$\n\n, where P is the image set distribution. Other notations follow from the paper.\n\nOur definition of equal representation is equivalent to the definition 3 of \"Equal opportunity for ranking distributions\" in [1] when all ranking position biases are equal. \n\nFurther, we would like to clarify that the fairness problems of \"EO\" and \"DP\" are for classification tasks. In retrieval tasks, we use \"equal representation\" as the goal of the fairness. Since \"EO\" and \"DP\" are defined in a classification setting, they cannot be directly applied to our retrieval setting, where the output is a bag of images. However, one can extend the notion of \"EO\" and \"DP\" to other tasks, similar to [1]. This will discussed in the introduction of the revised paper.\n\n[1] Singh, Ashudeep, and Thorsten Joachims. \"Equality of opportunity in rankings.\" Workshop on Prioritizing Online Content (WPOC) at NIPS. Vol. 31. 2017\n"}, {"review_id": "hJMVN9q8e9", "reviewer": "Reviewer_TTMH", "summary": "The paper proposes a method for mitigating test-time bias in image retrieval of vision-language models. More concretely, the authors demonstrate how prior work debiasing methods fail to ensure equal representation of different sensitive groups in the retrieval bag at test-time when the test set is imbalanced across different sensitive groups (for example, there are more male engineers at the test set than female engineers). To mitigate this issue, they propose a simple method where they train a sensitive group classifier (for example, a gender classifer) to split the test set across the sensitive groups and then perform the retrieval over each subset independently (if the task is to retrieve K samples, and the sensitive groups are: Male/Female then they would retrieve K/2 from the predicted Male group and K/2 from the predicted Female group). They show through experiments that their method reduces the disparity between the number of retrieved Male and Female samples at test time while maintaining “satisfactory” retrieval performance. \n", "questions": "Given the methodology’s weakness of relearning the bias through the sensitive group classifier, I am not confident in the validity of this method. In order to prove me otherwise, I encourage the authors to provide the following: \n\n1- Report the difference between the recall performance on different sensitive groups in the test set. For example, for each profession (doctor, for example), I would encourage the author to report the absolute difference between the recall of the Male samples versus Female samples annotated in the test set and then average the metric over all professions. Their method will have a high absolute difference compared to other methods, thus exacerbating the bias. \n\n2- Even if the results above are satisfactory, I would like to see a theoretical or methodological argument proving that the gender classifier will not demonstrate the abovementioned problem. \n\nFurthermore, I encourage the authors to include an experiment where prior work methods like MI-CLIP and FairSample but with explicit bias text in the prompt (e.g., “Male Engineer”) and then perform K/2 retrieval for each gender-based prompt. I expect this will result in a balanced retrieval bag at test time with minimal additional cost. \n", "limitations": "The issue above should be discussed in their limitations sections. I encourage the authors to expand their limitations section and include a more critical discussion of their use of sensitive group classification systems. Overall, I need more evidence about the validity of the contribution. However, I am happy to update my score in case I missed any central point that could be pointed out in the authors’ replies. \n", "rating": 6, "confidence": 4, "soundness": 1, "presentation": 2, "contribution": 3, "strengths": "1- The problem is well motivated; indeed, when the test set is imbalanced, it is clear that the prior work debiasing method, while not amplifying the bias of the test set, will nevertheless exhibit the proportional bias of the test set. \n\n2- The method is simple and easy to implement. \n\n3- The writing is clear, and the results are well explained. \n", "weaknesses": "This work's main weakness is using a sensitive group classifier (for example, gender classifier) to split the test set into different subgroups. An apparent problem with this approach is that the sensitive group classifier will likely be biased. This is because, similar to how VL models correlate class (for example, professions) with gender, it will also correlate gender with class. So, for example, if the dataset has more Male engineers, then a gender classifier based on the VL model features is more likely to correctly predict engineers as male than predicting engineers as Females. This apparent problem is not discussed in the paper not are solutions proposed to address it.\n\nFurthermore, the experimental results could explain why this might be an issue. More concretely, the proposed method consistently reduces the recall performance, often by significant margins. For example, in Table 2, Recall 1 is reduced from 67.2 to 41.2 on Flickr30k. This pattern might be directly related to the bias of the gender classifier. Indeed, if the classifier, for example, is not good at predicting Female Engineers as Female, then the test set predicted Female split might contain very few engineers, and thus, the K/2 extracted samples from the predicted split might contain very few relevant engineers and thus reducing the recall performance. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **Response to Reviewer TTMH**\nThanks for your detailed review and constructive feedback. Before reading our dedicated response to your comments, we kindly suggest you start with our global response, which offers a broad overview and addresses some of the general concerns raised. \nBelow is our response to your comments,\n\n#### **\"An apparent problem with this approach is that the sensitive group classifier will likely be biased.\" and \"Report the difference between the recall performance on different sensitive groups in the test set...\"**\nWe acknowledge your concern about the performance and fairness of the demographic group classifier. Regarding the performance concern, since our goal is to mitigate perceivable bias in web image search systems, the group classifier is unlikely to have poor performance because perceivable attributes are in principle easy to classify. Further, in Table 1 (rebuttal), we empirically show that the group classifier works well across different demographic attributes in Occupation 1 & 2 datasets. Please refer to the global response for a more detailed discussion on this issue.\n\nAs you requested, we report the absolute difference between the recall performance stratified across different sensitive groups in our test set, as shown in the Table 3 (rebuttal). The results show that PBM achieves the least absolute difference compared to in-processing methods and MI-clip.\n\n#### **\"I would like to see a theoretical or methodological argument proving that the gender classifier will not demonstrate the abovementioned problem.\"**\nIt is reasonable to suspect that the demographic group classifier may exacerbate retrieval bias. However, we argue that if we can get a classifier that has good performance when classifying perceivable attributes, thus the PBM classifier can ensure the retrieval bias no worse than retrieving images without any gender or race information. The empirical results of Figure 1 and Figure 2 (rebuttal) support this claim.\n\n##### **Theoretical analysis of PBM debiasing effect**\nIn this analysis, we simplify our focus to a scenario where the only attributes are male and female. The notations below are adopted from our main paper.  To understand the group classifier's influence, we investigate the likelihood of PBM creating a minimally biased image set comprising one male and one female. The probability of successfully selecting a male and female image pair can be expressed as:\n\n$$ \n\\mathbb{P}[g(v_1) \\neq g(v_2)] \n$$\n\n$$ \n\\qquad = \\mathbb{P}[g(v_1) = +1 \\wedge \\hat{g}(v_1) =1]\\mathbb{P}[g(v_2) = -1 \\wedge  \\hat{g}(v_2) = - 1] \n$$\n\n$$ \n\\qquad \\+ \\mathbb{P}[g(v_1) = -1 \\wedge  \\hat{g}(v_1) = + 1]\\mathbb{P}[g(v_2) = + 1 \\wedge  \\hat{g}(v_2) = - 1] .\n$$\n\nIn the above equation, $\\mathbb{P}[g(v_1) = +1 \\wedge \\hat{g}(v_1) = -1]$\nrepresents the error in misclassifying a man as a woman and \n$\\mathbb{P}[g(v_1) = -1 \\wedge \\hat{g}(v_1) = + 1]$ is the error in misclassifying a woman as a man. We denote them as ϵ1 and ϵ2 as follow,\n\n$$\\epsilon_1 =  \\mathbb{P}[g(v_1) = + 1 \\wedge \\hat{g}(v_1) = - 1],\n$$\n\n$$\n\\epsilon_2  = \\mathbb{P}[g(v_1) = -1 \\wedge \\hat{g}(v_1) = + 1].\n$$ \n\nThen, we can reframe our equation as:,\n\n$$\n \\mathbb{P}[g(v_1) \\neq g(v_2)] = (1 - \\epsilon_1) (1 - \\epsilon_2) + \\epsilon_1 \\epsilon_2 .\n$$\n\nSuppose the total error of misclassifying one gender as the other is given by,\n\n$$ \n\\epsilon_1 + \\epsilon_2  = \\epsilon. \n$$\n\nWe can further refine our equation:\n\n$$\n\\mathbb{P}[g(v_1) \\neq g(v_2)] = 1 - (\\epsilon_1 + \\epsilon_2) + \\frac{1}{2} ((\\epsilon_1 + \\epsilon_2)^2 - (\\epsilon_1 - \\epsilon_2)^2)   = 1 - \\epsilon + \\frac{1}{2}(\\epsilon^2 - (\\epsilon_1 - \\epsilon_2)^2) .\n$$\n \nSubsequently, we have an expression for the likelihood of forming a minimal fair image set in terms of the total error $\\epsilon$ and prediction disparity $\\epsilon_1 - \\epsilon_2 $. From the above equation, the lowest $\\mathbb{P}[g(v_1) \\neq g(v_2)] $ arises when $|\\epsilon_1 - \\epsilon_2 | = \\epsilon$, suggesting the most significant disparity in predicting male and female with a fixed error $\\epsilon$. \n\nConsider the worst case of binary classification is $\\epsilon  = 0.5$ and $|\\epsilon_1 - \\epsilon_2 | \\geq 0.5$. \nWe have, \n$$\\mathbb{P}[g(v_1) \\neq g(v_2)] \\geq 0.5$$\n\nRemarkably, the above inequality suggests the probability of choosing a fair subset via PBM is no worse than constructing a fair subset by randomly selecting 2 images from a gender-balanced dataset, given equal occurrence for both genders($\\mathbb{P}(g(v)=+1)=0.5$ and $\\mathbb{P}(g(v) = -1) = 0.5)$. \n\nTherefore, we can conclude that the retrieval bias using PBM can be no worse than choosing samples without any demographic information (which is the goal of in-processing methods), because PBM addresses the dataset bias at test-time. This claim is reinforced by the empirical results of Figure 1 and Figure 2 (rebuttal).\n\n##### **Empirical analysis of PBM debiasing effect on Occupation 1**\nWe show an empirical analysis that is closer to real-world case in Figure 2 (rebuttal). Specifically, we vary the sensitivity ratio between males and females representing the bias by changing the group classification threshold. We show that the resulting retrieval bias is lower than that of the original CLIP for a wide range of sensitivity ratios.\n\n#### **\"Furthermore, I encourage the authors to include an experiment where prior work methods like MI-CLIP and\"**\nPlease see Table 2 (rebuttal) for results of our methods combining with MI-clip and other fine-tuned CLIP.\n"}, {"review_id": "58UIzlOUKe", "reviewer": "Reviewer_VHZT", "summary": "The paper tackles the problem of bias in text-image retrieval. The goal is to generate fair and unbiased image retrieval results given neutral textual queries. As opposed to previous approaches that aim to separate the learned representations of images and text from characteristics such as gender and race, the paper claims that these methods are insufficient for achieving the objective of equal representation, primarily because bias often persists during the evaluation phase within the target retrieval set. In order to address this, the authors propose PBM, a straightforward and efficient test-time post-processing debiasing algorithm that generates fair retrieval subsets, guided by predicted gender/race information obtained from an off-the-shelf classifier or inferred via zero-shot using the VL model itself and evaluate their approach on various benchmarks.", "questions": "1. What architecture do you use for the text encoder and video encode? I think this information should be added in sec 3.2\n\n2. What is the performance of the gender prediction? (both zero-shot and supervised) Does the zero-shot prompt works better than supervised classifier? The results from Tab 1 seem to me to be in contradiction with Fig 4 (if I am reading it correctly) since AbsBias is always lower for supervised vs zero-shot, but in Tab 1 the results are different.\n\n3. Any insights on why the performance drop is so large in Tab 2? \n\nMinor - Can the method be applied to other types of biases except gender and race biases? Are there any other datasets that have other bias information? What is needed to adapt the method to other biases?", "limitations": "The limitations are discussed and a brief societal impact is also presented.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper tackles an important problem. I think that addressing bias is a very important direction and this paper can provide useful insights to the research community.", "weaknesses": "My main concern with the paper is that I don't understand if there is a link between the performance of the gender/race classifier and the bias. Since the performance of these classifiers directly affect the performance, I would have wished to see more ablations on this front.\n\nSome parts seem to be unpolished. For example Fig 4 is very hard to read, especially the last plot. Also, the colors are quite similar and it's very hard to assess which method is which.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **Response to Reviewer VHZT**\nThanks for your detailed review and constructive feedback. Before reading our dedicated response to your comments, we kindly suggest you start with our global response, which offers a broad overview and addresses some of the general concerns raised. \nBelow is our response to your comments,\n\n#### **\"...if there is a link between the performance of the gender/race classifier and the bias...\"**\nThank you for bringing up this critical point. We understand your concern regarding the relationship between the performance of the demographic group classifier and the overall bias of our system. In Table 1 (rebuttal), we show the performance (sensitivity) of the group classifier stratified across different demographic groups. The Zero-shot Prompt method achieves best sensitivities on gender, whereas Supervised Classifier is the best on race. Combining these results with the AbsBias@100 reported in the Table 1 (paper), we can conclude that the debiasing effect is consistent to the performance of the group classifier. \n\nMoreover, in Figure 1 and Figure 2 (rebuttal), we examine the relationship between the group classifier predictions and retrieval bias, by varying the performance and sensitivities of the group classifier. Figure 1 (rebuttal) demonstrate that better group classifier performance leads to reduced bias, that bias converges to that of the original CLIP as the group classifier gets worse, and importantly, that the bias after PBM will be no worse than that of the original CLIP. Meanwhile, Figure 2 reveals that the retrieval result will have more bias as the group classifier has more predictive disparity (i.e., sensitivity ratio approaches 0 or $\\infty$), and the lowest bias is achieved when the group classifier prediction is fairly balanced (i.e., sensitivity ratio close to 1). \n\nFurther, for a more theoretical justification, please refer to the response to reviewer TTMH.\n\n#### **\"Some parts seem to be unpolished...\"**\nThanks for pointing out the visual quality of our paper. The performance of different PBM methods is very similar in Figure 4. Hence, it is somewhat difficult to distinguish them. We improve the readability of Figure 4 by zooming into the bias-recall curves for PBM variants. Please refer to Figure 3 (rebuttal). We will include the better visualization in our revision.\n\n#### **\"What architecture do you use for the text encoder and video encode?”**\nThe image encoder is a ViT with 8 vision transformer blocks and the text encoder has 12 Transformer Blocks. Both image and text encoder are pre-trained by OpenAI [1]. The details of the network architecture is included in our Supplementary Material. We will certainly include this information into Section 3.2.\n\n[1] Radford, Alec, et al. \"Learning transferable visual models from natural language supervision.\" ICML (2021).\n\n#### **\"What is the performance of the gender prediction?...\"**\nWe include the sensitivity across different demographic groups in the Table 1 (rebuttal). For the Figure 4 (paper), the left plot is the recall/bias trade-off evaluated on debiasing race for Occupation 2 dataset, which should correspond to the third column in Table 1 (paper). In this case, the supervised classifier predicts more accurately than the zero-shot prompt method, so the lowest AbsBias@100 is achieved by the PBM - Supervised Classifier in Occupation 2 - Race. This will be emphasized in the caption of Figure 4 in revision to avoid confusion.\n\n#### **\"Any insights on why the performance drop is so large in Tab 2?\"**\nFor the COCO and Flickr30k benchmarks, the retrieval pools for each query are very limited, thus when we apply PBM, the recall performance suffers because there are usually not enough minor demographic groups to satisfy the PBM constraint.\n\n#### **\"Minor - Can the method be applied to other types of biases except gender and race biases?\"**\nYes. We believe our method can be applied to other types of bias as long as such bias are perceivable, which means they can be annotated by a human and predicted by a neural network. UTKface and Fairface are face datasets that are annotated with age, gender and race [2,3]. Our methods can definitely be applied to debias age when retrieving images from UTKface. We will include experiments with UTKface and Fairface in our revision (we were unable to get the results in time before the rebuttal deadline).\n\n[2] Zhang, Zhifei, Yang Song, and Hairong Qi. \"Age progression/regression by conditional adversarial autoencoder.\" CVPR (2017).\n\n[3] Karkkainen, Kimmo, and Jungseock Joo. \"Fairface: Face attribute dataset for balanced race, gender, and age for bias measurement and mitigation.\"  CVPR (2021)."}, {"review_id": "xRbaJXBpR8", "reviewer": "Reviewer_PMeK", "summary": "The paper considers fair text-based image retrieval, in particular when the target retrieval set is biased. The chosen fairness metric is _equal representation_, e.g., the number of retrieved male and female images should be identical. The authors conjecture that if the target retrieval set is biased, debiasing the (multimodal vision-language) model during training is not sufficient to guarantee fairness of the retrieved results in test-time. Therefore, they propose Post-hoc Bias Mitigation (PBM): first, they sort the most relevant candidate images from each group (e.g., male and female) separately and then take equal amount of images from the two groups. In their experiments they consider different ways of estimating the (unknown at test time) sensitive attributes: zero-shot embedding, prompting and supervised classification. Experiments are executed on 4 standard datasets: Occupation 1 and 2, MS-COCO and Flickr30k.", "questions": "Q1: I think it will be prudent to show the distribution of the sensitive attributes for the target ground-truth retrieval set(s) ($V_{c}^{\\*}$) as well as the whole test sets ($\\mathcal{V}$), e.g., the male:female ratio per occupation in the Occupation datasets. I would be interested to see the relationship between bias/recall and the distribution ratio (between male/female in $V_{c}^{\\*}$). For example, if the target retrieval set is relatively balanced, are the other baseline methods still biased?\n\nQ2: How do you control the bias-recall trade-off? Do you vary the probability (L348) from 0.1 to 0.9 on Figure 4? \\\nMinor: Are the results in Figure 4. for Occupation 1 and 2 datasets and gender as the sensitive attribute?\n\nQ3: How would you combine multiple sensitive attributes (e.g., if we want to be fair w.r.t. both gender and race in Occupation 2)?\n\nQ4: Is the problem formulation and the metrics standard/adapted from prior work or novel/custom? I see that you closely follow (Wang et al., 2021a), so might be good to mention this.\n\nQ5: Formal metric definitions. For Recall@K why don't you consider $\\frac{1}{|C|}\\sum_{c \\in C} \\frac{| V_{c,K} \\cap V_{c}^* |}{|V_{c}^{*}|}$ ? Or it does not matter as $ | V_{c}^{\\*} | = K$ is the same for all queries $c$? When computing the bias, why do you take the absolute value, instead of considering Bias@K, following (Wang et al., 2021a)? If you take the absolute values, how do you get negative gender bias in Figs. 2 and 3?\n\nQ6: When instantiating PBM, do you use the original CLIP as image/text encoders, or any of the other baselines MI-clip, Adversarial training, Debias Prompt? What is the effect if you instantiate PBM with any of these \"debiased\" VL models?\n\nQ7: What is the underlying reason for the recall to still be low? Are the VL models not good enough? What happens if we lower K and consider the precision, esp. for the Occupation datasets? Unless male:female ratio is 50:50, it is impossible to obtain 100% recall for the given query anyway, as we have to retrieve other images from the underrepresented group that do not correspond to the occupation for which we query. L320: should it be 61:39?", "limitations": "In general, I suggest trying to (partially) include the Limitations section in the main paper or at least refer to it.\n\nLim1: Fairness metric definition. Translated in the context of supervised classification, it would look like $p(g = 0 | \\hat{y} = 1) = p(g = 1 | \\hat{y} = 1)$, where the value of the sensitive attribute $g$ is unknown at test time and \"prediction\" of $\\hat{y} = 1$ corresponds to the image being retrieved. This does not look like any other \"standard\" (supervised classification) fairness metric. I appreciate that the authors have included motivation about this design choice, but maybe some further discussion is needed. I would like to point out that in some cases or other scenarios the chosen metric may lead to unfairness, e.g., for the content providers (if a sample with lower utility is preferred over another solely due to having the \"correct\"/\"desirable\" sensitive attribute).\n\nLim (comment) 2 [out of scope]: Depending on the setting, if there are insufficient number of samples with a particular sensitive attribute, synthetic samples might be generated (Tanjim et al., 2022).\n\nTanjim et al., Generating and Controlling Diversity in Image Search, WACV 2022\n\nLim (comment) 3 [out of scope]: An end-to-end pipeline would need to decide if the input query is gender-neutral or not. E.g., querying for a famous actor/actress is clearly not gender-neutral. Then, is gender neutrality a 0/1 binary property, or it can be scored?", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The paper is well-written and easy to follow.\n* The prior work is presented fairly and it is somewhat clear how this paper is positioned with respect to it. It touches on a relevant topic with potentially high social impact.\n* The presented solution to the proposed problem formulation is intuitive, straightforward and seems to work well in practice.", "weaknesses": "1. The idea of post-processing is not novel and the technical contribution of the proposed solution is limited and rather incremental, in my opinion. Hardt et al. (2016) and Calders and Verwer (2010) propose probability thresholding, while Karako and Manggala (2018) consider a similar procedure of iterative selection of retrieved images based on their relevance and diversity. I acknowledge that the problem settings in these works are sufficiently different from the current paper, but the idea that we can take an equal amount of images from the two demographic groups (sorted by relevance) if we want to equalize the number of images from both groups, is not particularly exciting or insightful, even if the demographic attributes are inferred in a zero-shot manner during test-time.\n\nCalders and Verwer, Three naive Bayes approaches for discrimination-free classification, 2010\\\nKarako and Manggala, Using Image Fairness Representations in Diversity-Based Re-ranking for Recommendations, 2018\n\n2. It seems to me that the key factors for the performance (in terms of fairness; for a given, fixed query) of the method are the distribution of male/female among the ground-truth samples in the target retrieval set as well as the accuracy of $\\hat{g}(\\cdot)$ - the sensitive attribute predictor. I think it may be informative to further explore what happens when the distribution of samples from each demographic group is varied or when $\\hat{g}(\\cdot)$ is not very accurate. For example, according to Ghosh et al. (2021), \"developers should not use inferred demographic data as input to fair ranking algorithms, unless the inferences are extremely accurate\".\n\nGhosh et al., When Fair Ranking Meets Uncertain Inference, SIGIR 2021\n\n3. The details how to control the fairness/utility trade-off seem to be missing from the supplementary material (L350). I am unable to find other details from the SM as well (e.g., adversarial learning results, L187).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **Response to Reviewer PMeK**\nThanks for your detailed review and constructive feedback. Before reading our dedicated response to your comments, we kindly suggest you start with our global response, which offers a broad overview and addresses some of the general concerns raised. \nBelow is our response to your comments,\n\n#### **\"The idea of post-processing is not novel and the technical contribution of the proposed solution is limited...\"**\nWhile we acknowledge that using post-processing method to reduce the output bias is not novel, our work contributes to leveraging a group classifier built by supervised learning, zero-shot word embedding and zero-shot prompts. Our approach effectively reduces the bias in retrieval, showcasing the utility of neural network predictions for gender and race attributes in addressing retrieval test-time bias. Calders and Verwer (2010) requires ground-truth gender attribute and Karako and Manggala (2018) obtain the gender labels from user feedback data. We will clarify more about our contribution and add these two papers in the related work of our revision.\n\n####  **\"It seems to me that the key factors for the performance (in terms of fairness; for a given, fixed query) ....\".**\nThe distribution of the retrieval pool among Occupation 1 & 2, COCO and Flikcr30k are different. We list the male rates for these datasets in Table 4 (rebuttal).\n\nThough Ghosh et al. (2021) states \"developers should not use inferred demographic data as input to fair ranking algorithms, unless the inferences are extremely accurate\", our work focuses on mitigating perceivable bias in web image search system. The performance of our demographic group classifier is less likely to be poor given that perceivable biases are in principle easy to classify. This is demonstrated in Table 1 (rebuttal). Moreover, our work argues that only the post-hoc group classifier can address the test-time bias in image retrieval, while none of other debiasing methods addressed the test-time bias scenario.\n\n#### **\"The details how to control the fairness/utility trade-off...\"**\nThanks for pointing it out. We will include these details in the revision. To control the trade-off of adversarial learning, we vary the weights of adversarial loss from 0 to 1.0. For MI-clip, we vary the clipped dimensions from 10 to 500 (CLIP output dimension is 512). We include these details in the Section C.2 of our Supplemental Material. We will include a dedicated section to describe trade-off strategies of PBM, MI-clip and adversarial learning, including pseudo-codes of trade-off strategies and how to choose $\\theta$ under different application context, to avoid confusions. \n\n#### **\"Q1: I think it will be prudent to show the distribution of the sensitive attributes \"**\nWe show the distribution of gender across Occupation 1 & 2, COCO-val and Flickr30k test in Table 4 (rebuttal). If the target retrieval set is balanced, we believe there is likely still bias in the retrieval set since sampling bias exists if the retrieval bag size is not large enough.\n\n#### **\"Q2: How do you control the bias-recall trade-off? \"**\nTo manage the bias-recall trade-off, we employ a parameter θ, that is the probability of choosing a fair subset at any given time, rather than solely selecting the image with the highest similarity score. We vary θ in increments of 0.1, ranging from 0.0 to 1.0. As illustrated in Figure 4, the left-side plot is for Occupation 2 - Race, while the right-side plot for Occupation 1 - Gender.\n\n#### **\"Q3: How would you combine multiple sensitive attributes ... \"**\nIn this case, we can define combinations of different attributes from which PBM can choose separately. For example, we can define the combination each time PBM has to choose: {\"Light skin male\", \"Dark skin female\"} or {\"Light skin female\", Dark skin male} or {\"Light skin male\", \"Dark skin female\", \"Light skin female\", Dark skin male}. Extending to combine multiple sensitive attributes is an intriguing direction which we consider for future work.\n\n#### **\"Q4: Is the problem formulation and the metrics standard/adapted from prior work or novel/custom?\"**\nIndeed, our work is inspired by (Wang et al., 2021a). In Table 2(paper), we use the same metric and setup as (Wang et al., 2021a). We do not use Bias@K in our main result(Table 1 of paper) because we contend that the Bias@K used in (Wang et al., 2021a) omits using the absolute value in the sum. The Bias@K metric can be misleading sometimes because the positive bias and negative bias cancel each other out in the summation. This will be emphasized in the Section 4 of revision.\n\n#### **\"Q5: Formal metric definitions. \"**\nFor Recall@K, we follow the Recall@K used in (Wang et al., 2021a). $\\frac{1}{|C|}\\sum_{c \\in C} \\frac{| V_{c,K} \\cap V_{c}^* |}{|V_{c}^{*}|} $ is also a valid metric but should be named as \"mean Recall@K\". We choose to use AbsBias@K in Table 1 (paper) because we argue that the Bias@K used in (Wang et al., 2021a) omits using the absolute value in the sum.\n\nIn Figure 2 and 3 (paper), the gender bias is a simple summation of gender attributes (male = +1 and female = -1) in different quantiles. We will clarify these points in the Section 3.3 of revision.\n\n#### **“Q6: When instantiating PBM, do you use the original CLIP as ...”**\nIn our reported results, we apply PBM on the original CLIP. We also report applying PBM on other fine-tuned models in Table 2 (rebuttal).\n\n#### **\"Q7: What is the underlying reason for the recall to still be low?\"**\nWe conduct a brief analysis of the low recall performance in COCO and Flickr30k in the global response. \n\nWe will acknowledge this point in the limitation section of our revision.\n\n"}]

Author rebuttal:
We sincerely appreciate your valuable feedback. We are grateful for the constructive comments and suggestions, which provided us with insights to improve the quality and clarity of our work. Below, we address the common concerns raised by the reviewers.

### **Concerns on the performance of group classifiers** 
We acknowledge that the performance of the proposed PBM is affected by the accuracy and bias of the demographic group classifier, which will be emphasized in the revision. Regarding the former, in Figure 1 (rebuttal) we demonstrate the relationship between the performance of group classifier and the resulting retrieval bias by artificially introducing noise to the demographic group (logit) predictions via Gaussian noise with standard deviation varying from 0 (no noise) to 1. These results illustrate that better group classifier performance yields lower bias, that bias converges to that of the original CLIP as the group classifier gets worse, and importantly, that the bias after PBM will be no worse than that of the original CLIP.

Further, Table 1 (rebuttal) shows the individual demographic group sensitivities under three different scenarios, from which we can see that the group classifier is i) able to achieve good classification sensitivity (no lower than 0.81 and 0.90 in average), likely because demographic image attributes (gender and skin tone) are typically captured in images, and ii) that different scenarios exhibit different degrees of bias as measured by the group sensitivity ratio, which must be close to 1 for the model to be unbiased. Note that bias in (binary) classification can be measured in many ways other than sensitivity ratios. This will be acknowledged in the revision.

For the concerns about the bias of the group classifier, we acknowledge that there are sensitivity discrepancies between different attributes, as shown in Table 1 (rebuttal). To further investigate the influence of the group classifier bias on the PBM results, we present the retrieval bias as a function of the sensitivity ratio in Figure 2 (rebuttal). This is achieved by changing the group classification threshold from 0 (max male sensitivity) to 1 (minimum male sensitivity). As pointed out by the reviewers, group classifier bias affects retrieval bias, however, only severely for more extreme sensitivity ratios, which is fortunately not the case in our results as shown in Table 1 (rebuttal).

To show the impact of group classification calibration on PBM, we leverage a simple post-hoc calibration method, histogram binning [4], on the worst performing method in Table 1 (rebuttal). We calibrate the group classifier on Occupation 2 - Gender and then test it on Occupation 1 - Gender. The calibrated group classifier achieves a sensitivity ratio of 1.01 (93.5\% male sensitivity and 92.4\% female sensitivity). The resulting AbsBias@100 using the calibrated group classifier is 0.1848 and the Recall@100 is 51.3\% on Occupation 1 - Gender, which results in worse retrieval bias and better recall relative to the uncalibrated group classifier in Table 1 (paper). We believe this is because the retrieval dataset (Occupation 1 - Gender) is imbalanced, which may have a negative impact on the resulting AbsBias@100 after PBM. This will be highlighted as a potential limitation of PBM.

Finally, we emphasize that the main contribution of our work is to identify the occurrence of test-time bias, which significantly influences image retrieval results and manifests as a result of the need for selecting samples from a test pool (the retrieval set), which are likely to be correlated, thus prone to introduce bias, more so if the test pool itself is biased. Consequently, we seek to analyze and raise awareness of the test-time bias issue, which is critical in image retrieval scenarios. Our work can be seen as a first step that can be greatly improved with additional research.

[1] Pleiss, Geoff, et al. "On fairness and calibration." NeurIPS 30 (2017).

[2] Zafar, Muhammad Bilal, et al. "Fairness constraints: A flexible approach for fair classification." JMLR (2019).

[3] Celis, L. Elisa, et al. "Classification with fairness constraints: A meta-algorithm with provable guarantees." ACM FAccT (2019).

[4] Zadrozny, Bianca, and Charles Elkan. "Obtaining calibrated probability estimates from decision trees and naive bayesian classifiers." ICML (2001).

### **Concerns about the recall performance drop using PBM** 
For the COCO and Flickr30k benchmarks, the retrieval pools for each query are very limited. When we apply PBM, the recall performance suffers because there are usually not enough samples in the minor demographic groups to satisfy the PBM constraint. For instance, there are queries mostly describing one gender like "A person wearing a dress..." in COCO, which make the debiasing problem very hard. Further, the performance drop in Occupation 2 race classification is much larger because there are simply no sufficient images for minor skin-tones in some occupations. For example, occupations "editor" and "photographer" have only 2 and 5 images annotated as "dark skin", respectively. This is a sample size limitation that will be emphasized in the revision.

There is a fundamental trade-off between bias and recall, and Table 1 and 2 (paper) only show results where we prioritize low bias rather than maintaining high recalls. For users who seek prioritizing retrieval performance, our method can be easily adjusted to prefer recall over bias, as illustrated in Figure 4 (paper) and discussed in the ``Bias-performance trade-off analysis'' paragraph in Section 4. Specifically, Figure 4 shows that for a fixed recall value, our method has significantly lower bias than other methods. This supports that our method is able to reduce bias while maintaining comparable recalls. We agree that reading the results in Table 1 and 2 alone could be misleading, thus we will clarify the results in Figure 4 in the revision.


Meta-review: All reviewers are supportive of this paper post-rebuttal. The identified weaknesses, like dependence on pre-trained gender-classifier, validity of method, and several limitations, have been adequately addressed in the rebuttal and should be incorporated in the cam-ready to further strengthen the work.
Decision: Accept (poster)