SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Insights Our observations can be explained and understood from a theoretical perspective. Previous methods are mostly concerned with making their image features independent of gender information. For a fixed query  $c$ , suppose we sample a random image from the test image data distribution  $\nu \sim \mathcal{V}^{\mathrm{test}}$ , its feature distribution  $f_{\phi}(\nu)$  is independent of gender distribution  $g(\nu)$ . Since the query feature  $f_{\psi}(c)$  is constant, the similarity score  $S(\nu, c)$ , defined in Section 3.2, is a deterministic function of  $f_{\phi}(\nu)$  and is thus also independent of gender  $g(\nu)$ . This is why the results for MI-based unbiasedness shown in Figure 2(b) and 3(b) show little correlation between similarity scores and gender bias. Similar results for debiasing with adversarial learning can be found in the SM.
However, as shown in Figure 2(b), there is still a consistent gender bias across similarity score values. This means that the gender distribution in each window tends to manifest the gender distribution of
Algorithm 1 Post-hoc Bias Mitigation (PBM).
Input: Text query  $c$ , retrieval size  $K$ , image database  $\mathcal{V}^{\mathrm{test}}$ , similarity measure from pre-trained vision-language models  $S(\cdot, \cdot)$ , and gender prediction model  $\hat{g}(\cdot)$ .
Output: Image retrieval bag  $V_{c,K}$ .
1: Split  $\mathcal{V}^{\mathrm{test}}$  into  $\mathcal{V}_{+1}^{\mathrm{test}}$ ,  $\mathcal{V}_{-1}^{\mathrm{test}}$  and  $\mathcal{V}_{\mathrm{N / A}}^{\mathrm{test}}$  using the gender prediction model  $\hat{g} (\cdot)$ ;
2: Let  $V_{c,K} = \varnothing$
3: while  $|V_{c,K}| < K$  do
4:  $v_{+1} = \underset {v\in \mathcal{V}_{+1}^{\mathrm{test}}}{\arg \max}S(v,c);$ $v_{-1} = \underset {v\in \mathcal{V}_{-1}^{\mathrm{test}}}{\arg \max}S(v,c);$ $v_{\mathrm{N / A}} = \underset {v\in \mathcal{V}_{\mathrm{N / A}}^{\mathrm{test}}}{{\arg \max}}S(v,c)$
5: if  $[S(\nu_{+1},c) + S(\nu_{-1},c)] / 2 > S(\nu_{\mathrm{N / A}},c)$  then
6:  $V_{c,K}\gets V_{c,K}\cup \{v_{+1},v_{-1}\} ;\quad \mathcal{V}_{+1}^{\mathrm{test}}\gets \mathcal{V}_{+1}^{\mathrm{test}}\setminus \{v_{+1}\} ;\quad \mathcal{V}_{-1}^{\mathrm{test}}\gets \mathcal{V}_{-1}^{\mathrm{test}}\setminus \{v_{-1}\} ;$
7: else
8:  $V_{c,K}\gets V_{c,K}\cup \{v_{\mathrm{N / A}}\} ;$ $\mathcal{V}_{\mathrm{N / A}}^{\mathrm{test}}\leftarrow \mathcal{V}_{\mathrm{N / A}}^{\mathrm{test}}\setminus \{v_{\mathrm{N / A}}\} ;$
9: end if
10: end while
11: return  $V_{c,K}$
the whole test image set  $\mathcal{V}^{\mathrm{test}}$ , which may not be balanced. This is also confirmed in Figure 3(b), as the final output bias of most occupations is clustered around 0.4, which is precisely the gender bias of the entire image set  $\mathcal{V}^{\mathrm{test}}$ . Therefore, ensuring independence, quantified here in terms of correlation, between image features and gender may not guarantee zero gender bias if the test image set itself is biased due to imbalance.
Two types of bias From the examples above, we can differentiate two types of bias, namely, the model bias from training and the bias from the test-time image distribution. The former can be quantified based on the correlation between similarity score and gender and only depends on the training data distribution and the way in which the model is trained. This has been previously described and studied (Caliskan et al., 2017; Zhao et al., 2017). In comparison, the latter type of bias manifests in the test phase because the test image set (the database from which we retrieve images) does not necessarily have commensurate numbers of male and female samples. We refer this type of bias as test-time bias. In fact, such proportions are usually unknown for image databases.
These two sources of bias coexist and can be addressed separately during model training and inference. Previous methods (Edwards and Storkey, 2015; Wang et al., 2021a; Berg et al., 2022) have been fairly successful at addressing the first type of bias on the training side, but neglect the test-time bias that is specific to the test set. To tackle the test-time bias, it is necessary to find a strategy targeting the test image set, so that the retrieved image genders are balanced despite the gender imbalance in the training (source) set. This very insight motivates our approach called PBM, which we describe below. As previously shown in Figures 2(c) and 3(c), PBM achieves substantially smaller retrieval gender bias than MI debiasing. Other existing approaches will be considered in the experiments.
3.5 Post-hoc Bias Mitigation (PBM)
To address the second type of bias induced by the imbalanced image test set, one simple idea is sub-sampling. Specifically, we could first sub-sample the image set to make sure its gender ratio is balanced before doing retrieval. However, a clear limitation of such an approach is that some highly relevant images may be dropped during sub-sampling, which may negatively affect retrieval quality. This problem is especially exacerbated if the test set has a very large gender bias. Alternatively, our intuition is to sub-sample after computing and ranking similarity scores of all images using a post-hoc method to control gender bias while sampling from the image source set, which we call Post-hoc Bias Mitigation (PBM).
A general version of the PBM algorithm, which is straightforward, is presented in Algorithm 1. For each image  $\nu_{i} \in \mathcal{V}^{\mathrm{test}}$ , we first predict its gender  $\hat{g}(\nu_{i}) \in \{+1, -1\}$ , which splits the images into two subsets,  $\mathcal{V}_{+1}^{\mathrm{test}}$  and  $\mathcal{V}_{-1}^{\mathrm{test}}$ . We then rank images from the two subsets based on their similarity scores separately. While forming the retrieval bag, we sample the top of both subsets together in pairs. Specifically, we select the top  $\lfloor K/2 \rfloor$  samples from each subset. If  $K$  is odd, we randomly pick one of the subsets and select one more top sample from it. This method ensures low gender bias of the retrieval output as long as gender predictions  $\hat{g}(\nu_{i})$  are accurate.
One extension of our method considers the case where gender may not be applicable to some images. For instance, our image dataset may contain cartoon characters that are not easily associated with any gender, or maybe the gender cannot be determined from the image due to body or facial coverings. In this case, we allow our gender predictions to take N/A values, yielding a  $\mathcal{V}_{\mathrm{N / A}}^{\mathrm{test}}$  subset. Images from  $\mathcal{V}_{\mathrm{N / A}}^{\mathrm{test}}$  do not inherently exhibit visually perceptible bias. Thus, they could be selected in favor of other male-female pairs if their similarity scores are higher and are exempt from bias measurement.
Gender prediction To predict gender  $\hat{g} (\nu)$ , we consider the following two methods for different scenarios. Note that all options are considered in the experiments.
Supervised gender classification: For scenarios where ground-truth gender attributes are available, such as MS-COCO (Lin et al., 2014; Zhao et al., 2021), we can train a complementary small gender classification network. Using our encoded image feature  $f_{\phi}(\nu)$  as input, we train a 3-layer MLP to classify gender. For datasets where gender may not be applicable to some samples, we add a N/A class to the set of labels for predictions.
Zero-shot inference using word embeddings or prompt: For scenarios where supervised training of gender is not possible, we can infer gender in a zero-shot manner (Radford et al., 2021; Li et al., 2022) using the implicit knowledge embedded in the text features of large pre-trained vision-language models (Radford et al., 2021; Li et al., 2022). Given that the semantics of gender and race attributes are already incorporated into the text encoder, we can extract them from word embeddings using words such as "Man" and "Woman". Alternatively, we could use prompts presupposed to the occupation search query. For example, we add gender-specific adjectives like "Male" or "Female" in front of a query. Finally, we compute the (cosine) similarity score of each image  $\nu$  with them,  $i.e., S(\nu, "Male" + c), S(\nu, "Female" + c)$ . The plus operator (+) here refers to string concatenation. The gender prediction  $\hat{g}(\nu)$  is then generated by comparing these two similarity scores.
We first evaluate our image retrieval algorithm on two real-world web image search datasets, Occupation 1 (Kay et al., 2015) and Occupation 2 (Celis and Keswani, 2020). We also test on two large-scale image-text datasets, MS-COCO (Lin et al., 2014) and Flickr30k (Plummer et al., 2015) to further validate the effectiveness of PBM in handling more complex text-based image retrieval scenarios.
For comparison, we consider adversarial training (Edwards and Storkey, 2015; Berg et al., 2022) and mutual information minimization (Wang et al., 2021a) as baseline methods. We also include other types of debiasing methods such as FairSample (Wang et al., 2021a), which balances the gender distribution of image-text pairs within a training batch, and FairExpec (Mehrotra and Celis, 2021), a denoised selection algorithm designed to select fair subset based on noisy demographic attributes. We use AbsBias@KK and Recall@KK as evaluation metrics as described in Section 3.1.
Real-world web image search The first dataset, which we refer to as Occupation 1 (Kay et al., 2015), comprises the top 100 Google image search results for 45 gender-neutral occupation terms, such as "chef", "librarian", "primary school teacher", etc. Each image within this dataset is annotated with a crowd-sourced gender attribute (either "male" or "female") that characterizes the person depicted in the image. Occupation 2 (Celis and Keswani, 2020), the second dataset, includes the top 100 Google image search results for 96 occupations, where both gender and race (represented as skin-tone: fair and dark skin) annotations are provided. Notably, the gender and race attributes also include a N/A category in Occupation 2, where the annotators have chosen the option of "Not applicable" or "Cannot determine" for the gender or skin-tone represented in the image. Consequently, we treat the images labeled with N/A as neutral examples that do not contribute to the bias of retrieval, since in principle, the users cannot perceive gender or racial information from the image.
For these two datasets, we consider OpenAI's CLIP ViT-B/16 (Radford et al., 2021) as the VL model for all debiasing methods. The baselines for comparison are MI-clip from Wang et al. (2021a), adversarial training adapted from Edwards and Storkey (2015), and Debias Prompt from Berg et al. (2022). We have also tailored FairExpec (not originally proposed for TBIR) to our task by integrating it with CLIP and our proposed gender predictor  $\hat{g}(\cdot)$ . We refer to this adapted model as CLIP-FairExpec in our experiments. We test our PBM method with four variants of gender predictors  $\hat{g}(\cdot)$ , as discussed in Section 3.5. These variants include a supervised classifier, zero-shot inference, and ground truth from annotators. The implementation details are provided in the SM.
Table 1: Results for debiased image retrieval from Occupation 1 and 2 datasets.
Method	Occupation 1 - Gender	Occupation 2 - Gender	Occupation 2 - Race
AbsBias@100(↓)	Recall@100(↑)	AbsBias@100(↓)	Recall@100(↑)	AbsBias@100(↓)	Recall@100(↑)
Random Selection	.6370	-	.3155	-	.4171	-
CLIP Original (Radford et al., 2021)	.6231	58.3	.3566	46.2	.5002	46.2
MI-clip (Wang et al., 2021a)	.3769	47.0	.2539	42.2	.4099	42.3
Adversarial Training (Edwards and Storkey, 2015)	.2316	44.0	.2603	37.8	.4880	43.3
Debias Prompt (Berg et al., 2022)	.6373	59.3	.3564	46.2	.4946	50.2
CLIP-FairExpec (Mehrotra and Celis, 2021)	.2498	47.0	.2619	44.2	.2788	34.7
PBM - Zero-shot Embedding	.0969	49.8	.1150	42.1	.3133	40.2
PBM - Zero-shot Prompt	.0560	46.1	.0443	42.5	.2571	36.0
PBM - Supervised Classifier	.1404	50.3	.1171	42.1	.0955	37.9
PBM - Ground-truth Gender and Skin-tone	.0000	49.1	.0000	42.4	.0000	41.3
Table 2: Results for debiased image retrieval from MS-COCO and Flickr30k datasets.
Dataset	Method	Gender Bias	Recall
Bias@1 (↓)	Bias@5 (↓)	Bias@10 (↓)	Recall@1 (↑)	Recall@5 (↑)	Recall@10 (↑)
COCO-1k	SCAN (Lee et al., 2018)	.1250	.2044	.2506	47.7	82.0	91.0
FairSample (Wang et al., 2021a)	.1140	.1951	.2347	49.7	82.5	90.9
CLIP (Radford et al., 2021)	.0900	.2024	.2648	48.2	77.9	88.0
MI-clip (Wang et al., 2021a)	.0670	.1541	.2057	46.1	75.2	86.0
Our PBM	.0402	.0961	.1082	37.3	73.6	84.8
COCO-5k	SCAN (Lee et al., 2018)	.1379	.2133	.2484	25.4	54.1	67.8
FairSample (Wang et al., 2021a)	.1133	.1916	.2288	26.8	55.3	68.5
CLIP (Radford et al., 2021)	.0770	.1750	.2131	28.7	53.9	64.7
MI-clip (Wang et al., 2021a)	.0672	.1474	.1611	27.3	50.8	62.0
Our PBM	.0492	.1006	.1212	22.3	50.5	61.9
Flickr30K	SCAN (Lee et al., 2018)	.1098	.3341	.3960	41.4	69.9	79.1
FairSample (Wang et al., 2021a)	.0744	.2699	.3537	35.8	67.5	77.7
CLIP (Radford et al., 2021)	.1150	.3150	.3586	67.2	89.1	93.6
MI-clip (Wang et al., 2021a)	.0960	.2746	.2951	63.9	85.4	91.3
Our PBM	.0360	.1527	.1640	41.2	85.3	92.6
We summarize the experimental results in Table 1. Compared with other methods, PBM variants achieve significantly lower AbsBias@100 while maintaining comparable Recall@100 scores. The male:female ratio of images in Occupation 1 is approximately 61:49, thereby mitigating the first type of bias by MI-clip, adversarial training, debias prompt and random selection is not enough. The CLIP-FairExpec cannot achieve better results since the FairExpec algorithm relies on the independence assumption, while the samples in our test dataset are correlated. This happens because images for occupations are collected from the same Google image search, which is biased. Further, FairExpec requires reliable probabilistic predictions for gender, however, in our setting the gender attributes are provided by an off-the-shelf MLP predictor based on visual features. In such a scenario, the label estimates yielded by the off-the-shelf MLP may not be always trustworthy, as the domain has shifted during inference. Consequently, these estimates could include misleading information, resulting in undesirable debiasing outcomes for CLIP-FairExpec. Moreover, it should be noted that Debias Prompt (Berg et al., 2022) always achieves the highest Recall@100, since we utilize their publicly accessible model, which has been fine-tuned on the Flickr30k dataset to enhance the performance of text-based image retrieval.
Large-scale text-based image retrieval We consider MS-COCO (Lin et al., 2014) and Flickr30k (Plummer et al., 2015). Our setup aligns with Wang et al. (2021a), where the gender attributes are directly inferred from the text captions of images. We consider the same baseline models as Wang et al. (2021a), which are SCAN (Lee et al., 2018) and CLIP, as well as their proposed approach, FairSample and MI-clip. It should be noted that FairSample is specifically designed to debias SCAN, a specialized in-domain VL model. We choose the best performing PBM model, which leverages the pre-trained classifier for gender attributes.
The performance metrics in Table 2 are consistent with Wang et al. (2021a), which resemble our AbsBias@K metric, with the omission of using absolute values in the sum. This bias metric is computed as Bias@K =  $\frac{1}{|C|}\sum_{c\in C}\frac{1}{K}\sum_{v\in V_K^c}g(v)$ . From the Table 2, we see that PBM maintains the bias to a minimum even when dealing with intricate text queries or images of complex scenes.
Bias-performance trade-off analysis In Figure 4, we show the trade-off between retrieval performance and bias for MI-clip, adversarial training and the four PBM variants. We choose MI-clip and adversarial training for comparison, as these models offer implementation simplicity of adjusting trade-off between AbsBias@100 and Recall@100. For PBM, we introduce a trade-off parameter through a stochastic variable representing the probability of opting for a fair subset at any given time, as opposed to merely selecting the image with the highest similarity score. Additional details about this experiment are provided in the SM. Our results indicate that in a range of relatively high
Figure 4: Trade-off between Recall@K and AbsBias@K for debiasing gender attributes within Occupation 1 (Middle) and race attributes using Occupation 2 (Right).
AbsBias@  $K$  values, MI-clip and adversarial training are still able to reduce bias while preserving Recall@  $K$ . However, in terms of lowering AbsBias@  $K$ , they struggle to maintain satisfactory TBIR performance. In contrast, the proposed PBM consistently succeeds in sufficiently reducing bias and maintaining performance, provided that predictions of attributes are reasonably accurate. Details of implementing this trade-off can be found in the SM.
In this study, we examined gender and racial bias in text-based image retrieval (TBIR) for neutral text queries. In an attempt to identify bias in the test-time (inference) phase, we conducted an in-depth quantitative analysis on bias reduction, alongside existing debiasing methods and the proposed PBM. We concluded that solely addressing training-time model-encoded bias is not sufficient for obtaining equal representation results, because test-time bias also exists due to imbalance in the test image set used during retrieval. So motivated, we proposed Post-hoc Bias Mitigation (PBM), a straightforward post-processing method that aims to directly alleviate test-time bias. Experiments on multiple datasets show that our method can significantly reduce bias while maintaining satisfactory retrieval accuracy at the same time.
Moreover, the potential impact of PBM extends far beyond the initial scope of text-based image retrieval systems. The core concept of our methodology can be seamlessly adapted to a wide variety of information retrieval systems, such as image-based text retrieval or query-by-example image retrieval, as long as the demographic information of the test set is accessible or can be estimated, e.g., via zero-shot inference. Overall, our approach is not limited to enhancing fairness in text-based image retrieval, thus can be extended to a broad range of VL model applications.
Some limitations of the proposed method are duly acknowledged. Firstly, the efficacy of our approach is dependent upon the availability of a sufficient number of examples for each category (gender or racial attribute) within the test image set. Our work currently does not consider any techniques, such as using synthetic samples, to mitigate the issues arising from insufficient representations of certain demographic groups. Secondly, the debiasing effect of PBM pertains to the predictability and accessibility of demographic attributes. Attempts at debiasing religious representation, or other socio-cultural factors or identities, in images or speech present significant challenges, because the predicting or securing annotations regarding religious information, can be exceptionally difficult. Thirdly, as we prioritize the "equal representation", our retrieval results sacrifice recall performance to ensure a retrieval bag that contains equal representations of each demographic group. This compromises fairness towards content providers in the image retrieval process. From the standpoint of content providers, fairness should imply that similar samples are treated similarly, regardless of the demographic group membership of their provided samples. Lastly, our work assumes all queries are neutral. We do not develop a technique to identify if a query is neutral or not, thereby limiting the applicability of our method in hybrid text query retrieval where text query can be biased like in "Male doctor". It is important to note, however, that the above challenges are not unique to our method but are a common issue encountered in other debiasing approaches. A more comprehensive discussion on the limitations of our work is available in the SM.
The authors would like to thank the anonymous reviewers for their insightful comments. This research was supported by ONR N00014-18-1-2871-P00002-3.
Han Xu, Xiaorui Liu, Yaxin Li, Anil Jain, and Jiliang Tang. 2021. To be robust or to be fair: Towards fairness in adversarial training. In International Conference on Machine Learning. PMLR, 11492-11501.
Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez Rodriguez, and Krishna P Gummadi. 2015. Learning fair classifiers. arXiv preprint arXiv:1507.05259 1, 2 (2015).
Syed Ali Jafar Zaidi, Attaullah Buriro, Mohammad Riaz, Athar Mahboob, and Mohammad Noman Riaz. 2019. Implementation and comparison of text-based image retrieval schemes. International Journal of Advanced Computer Science and Applications 10, 1 (2019), 611-618.
Dora Zhao, Angelina Wang, and Olga Russakovsky. 2021. Understanding and evaluating racial biases in image captioning. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 14830-14840.
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2017. Men also like shopping: Reducing gender bias amplification using corpus-level constraints. arXiv preprint arXiv:1707.09457 (2017).
A Impact of Demographic Group Classifier on Debiased Results
Table 3: Group sensitivities and sensitivity ratios  $(\rho)$  for demographic attributes predicted by different classifiers on Occupation 1 - Gender and Occupation 2 - Race.
Method	Gender	Race
Male Sensitivity	Female Sensitivity	ρ	Light skin Sensitivity	Dark skin Sensitivity	ρ
PBM - Supervised Learning	0.97	0.88	1.10	0.93	0.84	1.11
PBM - Word Embedding	0.98	0.94	1.04	0.84	0.78	1.08
PBM - Zero-shot Prompt	0.98	0.97	1.01	0.88	0.81	1.09
Figure 5: Relationship between the performance of the demographic group classifier (F1-score) and the retrieval bias (AbsBias@100) when utilizing PBM.
Figure 6: Relationship between the bias of the demographic group classifier (ratio of male sensitivity to female sensitivity) and the retrieval bias (AbsBias@100) when utilizing PBM.
The demographic group classifier is an important module of our proposed method PBM. The debiasing result is intricately linked to the demographic group classifier's accuracy and prediction bias towards different demographic groups. Figure 5 showcases the relationship between the demographic group classifier's performance and the ensuing retrieval bias, by artificially introducing noise to the demographic group (logit) predictions via Gaussian noise with a standard deviation ranging from 0 (no noise) to 1. These results underscore that better group classifier performance yields lower bias, that bias converges to that of the original CLIP as the group classifier gets worse, and importantly, that the bias after PBM will be no worse than that of the original CLIP.
Further, Table 3 shows the individual demographic group sensitivities under three different scenarios, from which we can see that the group classifier is i) able to achieve good classification sensitivity (no lower than 0.81 and 0.90 in average), likely because demographic image attributes (gender and skin tone) are typically captured in images, and ii) that different scenarios exhibit different degrees of bias as measured by the group sensitivity ratio, which must be close to 1 for the model to be unbiased. Table 3 reveals sensitivity discrepancies among different attributes. To delve deeper into the influence of classifier bias on PBM outcomes, we present the retrieval bias as a function of the sensitivity ratio in Figure 6. This is achieved by altering the gender classification threshold from 0 (maximizing male sensitivity) to 1 (minimizing male sensitivity). From Figure 6, we can conclude that the classifier bias does affect retrieval bias, however, only severely for more extreme sensitivity ratios, which is fortunately not the case in our results as shown in Table 3.
B Bias-recall Trade-off Strategies
In Figure 4, we exhibit the bias-recall trade-off curves for MI-clip, Adversarial Training, and various PBM methods. Here, we outline the missing details to achieve these trade-offs.