SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Fanjie Kong
fanjie.kong@duke.edu
Shuai Yuan
shuai@cs.duke.edu
Weituo Hao
TikTok Inc.
weituohao@tiktok.com
Ricardo Henao
ricardo.henao@duke.edu
We address the challenge of generating fair and unbiased image retrieval results given neutral textual queries (with no explicit gender or race connotations), while maintaining the utility (performance) of the underlying vision-language (VL) model. Previous methods aim to disentangle learned representations of images and text queries from gender and racial characteristics. However, we show these are inadequate at alleviating bias for the desired equal representation result, as there usually exists test-time bias in the target retrieval set. So motivated, we introduce a straightforward technique, Post-hoc Bias Mitigation (PBM), that post-processes the outputs from the pre-trained vision-language model. We evaluate our algorithm on real-world image search datasets, Occupation 1 and 2, as well as two large-scale image-text datasets, MS-COCO and Flickr30k. Our approach achieves the lowest bias, compared with various existing bias-mitigation methods, in text-based image retrieval result while maintaining satisfactory retrieval performance. The source code is publicly available at https://github.com/timqqt/Fair_Text_based/Image_Retrieval.
Image search on the web based on text-based image retrieval (TBIR) (Lew et al., 2006) involves interpreting a user's (text) query and returning corresponding images that are considered relevant in terms of semantic meaning (Chen et al., 2015). With recent advancements in multi-modal representation learning, Vision-Language (VL) models such as CLIP have been widely used to enhance the efficacy of text-based image retrieval (Radford et al., 2021; Cao et al., 2022). These models are usually trained on vast datasets that consist of millions of text-image pairs scrapped from the web, which inevitably manifest societal biases especially for neutral queries (Wang et al., 2021a; Hall et al., 2023; Wang et al., 2021b), i.e., queries without explicit demographic (gender or race) connotations. In Figure 1, we show image retrieval results for the neutral query "Bus Driver" using CLIP and an unbiased alternative delivered by the proposed approach.
In our work, we adhere to equal representation as our fairness objective (Mehrotra and Celis, 2021; Kay et al., 2015), as an alternative to proportional representation (Jalal et al., 2021; Berg et al., 2022). The latter, which aligns the demographic proportions in the retrieval results with those in the dataset, which is susceptible to biases during collection (Kay et al., 2015). Instead, equal representation for all demographic groups of interest attempts to mitigate (obscure) the influence of any inherent biases.
Previous works have been dedicated to developing unbiased VL models that promote fairness in TBIR tasks (Wang et al., 2021a; Berg et al., 2022; Wang et al., 2022; Kim et al., 2023; Chuang
Figure 1: Text-based image retrieval (TBIR) results of a neutral query. Right: all returned images are male bus drivers, possibly pushing the message that only men (can) perform this job. Left: desirable unbiased result with equal gender representation obtained by the proposed PBM approach.
et al., 2023; Parraga et al., 2022). Berg et al. (2022) leveraged adversarial training for debiasing in which a learnable prompt is prepended to the input text queries, while an adversary seeks to discourage the image and text encoders from capturing gender or race information for retrieval. Alternatively, Wang et al. (2021a) discarded the components of the visual and text representations that are highly correlated with gender, by estimating the mutual information between these components and gender attributes. Further, Wang et al. (2022) neutralized the gender contributions in image representations by enforcing equal contributions from gender (male and female) features using a bias contrast loss, while maximizing the contribution of gender-irrelevant features. These studies largely rely on demographic attributes that can be somewhat visually perceived. For instance, gender attributes are derived from perceived characteristics of masculinity or femininity, leading to labels such as "Male" or "Female". Similarly, race-related attributes are often characterized based on skin tones and typically categorized into groups such as "Fair Skin" and "Dark Skin" (Kay et al., 2015; Celis and Keswani, 2020). However, these methods primarily focus on debiasing the gender or race encoding within VL models, which may be insufficient due to bias or imbalance in the image retrieval candidate pool (test set). In contrast, though we also mitigate bias in TBIR using the same demographic attribute annotations as other works, the proposed solution is simpler in the sense that it forgoes the need for access to gradients or retraining the VL model.
In this paper, we start Section 3.1 by defining the text-based image retrieval task in the context of a fairness objective focused on equal representation. We then analyze the effectiveness of existing bias reduction methods in Section 3.4 and demonstrate how they fall short in achieving equal representation, mainly due to imbalances in the test-time image retrieval set. Based on this observation, in Section 3.5 we propose a simple yet effective post-processing debiasing method called post-hoc bias mitigation (PBM) that creates fair retrieval subsets guided by predicted gender (or race) attributes obtained from either an off-the-shelf gender (or race) classifier or zero-shot inference using a pretrained VL model. In Section 4, we evaluate PBM on real-world web image search (Kay et al., 2015; Celis and Keswani, 2020) and large-scale image-text datasets such as MS-COCO (Chen et al., 2015) and Flickr30K (Plummer et al., 2015). By comparing our approach to various existing bias-mitigation techniques, we show that our method achieves the lowest bias while maintaining satisfactory retrieval performance, as evidenced by both quantitative and qualitative results. Importantly, PBM strives to provide a more unbiased, fair and diverse visual representation of different groups in search results, ultimately promoting social welfare by challenging existing social perceptions of gender and race.
The summarized contributions of this work are:
- We present an analysis of the effect of existing debiasing methods on VL models, and highlight their insufficiency in achieving equal representation for text-based image retrieval.
- We propose PBM, a straightforward and efficient test-time post-processing debiasing algorithm that generates fair retrieval subsets, guided by predicted gender/race information obtained from an off-the-shelf classifier or inferred via zero-shot using the VL model itself.
- We evaluate PBM on two real-world web image search and two large-scale image-text datasets for text-based image retrieval, and compare with existing bias-mitigation techniques, demonstrating its effectiveness in achieving the lowest bias among all tested methods.
Text-based image retrieval Text-based image retrieval is the process of searching and retrieving images from a large database using textual descriptions or keywords as queries. The task is usually tackled by image-text feature alignment (Cao et al., 2022), which embeds image and text inputs into a shared feature space such that relevant image and text features are close to each other. In recent years, substantial progress has been made due to the emergence of large-scale datasets, as well as the development of effective deep learning-based image-text models. Pioneering works include DeViSE (Frome et al., 2013), which bridges the semantic gap between image content and textual descriptors by aligning CNN-based features of images with textual embeddings from ImageNet labels. Subsequent approaches, such as VSE++ (Faghri et al., 2017) and SCAN (Lee et al., 2018), further refine these joint embeddings to improve retrieval performance. Recently, OpenAI's CLIP (Radford et al., 2021) has emerged as a powerful approach for image retrieval based on similarity matching between extracted image and text features. CLIP leverages a pre-trained transformer model, jointly optimized for both image and text understanding, which allows it to effectively match images and textual descriptions.
Fairness in machine learning Recent studies have highlighted numerous unfair behaviors in machine learning models (Angwin et al., 2016; Buolamwini and Gebru, 2018). For example, a risk assessment algorithm used in the United States criminal justice system predicted that Black defendants were more likely to commit future crimes than white defendants, even after controlling for criminal record (Angwin et al., 2016). Moreover, individuals with different gender and skin-tones are likely to receive disparate treatment in commercial classification systems such as Face++, Microsoft, and IBM systems (Buolamwini and Gebru, 2018). Consequently, there has been a surge in demand and interest for developing methods to mitigate bias, such as regularizing disparate impact (Zafar et al., 2015) and disparate treatment (Hardt et al., 2016), promoting fairness through causal inference (Kusner et al., 2017), and incorporating fairness guarantees in recommendations and information retrieval (Beutel et al., 2019; Morik et al., 2020).
Fairness in vision-language models After some studies revealed the bias of using VL models in downstream tasks (Wang et al., 2021a; Hall et al., 2023; Wang et al., 2021b), efforts to address and mitigate these biases in VL models have gained increasing attention. Existing solutions for fair vision-language models can be generally classified into pre-processing, in-processing, and post-processing methods. Pre-processing techniques usually involve re-weighting or adjusting the training data to counter imbalances across demographic attributes, while preserving the utility of the dataset for the target task (Friedler et al., 2014; Calmon et al., 2017). In-processing methods focus on altering the training objective by incorporating fairness constraints, regularization terms or leveraging adversarial learning to obtain representations invariant to gender/race (Berg et al., 2022; Wang et al., 2023; Xu et al., 2021; Cotter et al., 2019). Post-processing approaches achieve fairness by applying post-hoc corrections to a (pre-)trained model (Cheng et al., 2021; Calmon et al., 2017) or via feature clipping (Wang et al., 2021a) on the output of image-text encoders based on mutual information.
Our work lies in the post-processing category of debiasing methods that encourages equal representation of diverse demographics. We also identified the fair subset selection approach used in Mehrotra and Celis (2021) as a potential post-hoc debiasing method for TBIR. While Mehrotra and Celis (2021) shares our goal of ensuring equality of gender/race attributes in the set of results, their focus did not extend to the TBIR scenario with an underlying VL model nor detail an effective method for obtaining accurate demographic attributes for debiasing. More importantly, they assumed demographic attributes seen by their algorithm to be available ground truth labels subject to noise. This assumption creates difficulties when attempting to adapt their method to a real-world problem such as TBIR. Complementary, our approach is meant to address these deficiencies by providing a practical debiasing procedure, that includes acquiring demographic attributes.
Gender/Racial Bias in Web Image Search Our research is closely related to studies in the Human Computer Interaction community that demonstrated gender inequality issues in current online image search systems (Kay et al., 2015; Noble, 2018; Celis and Keswani, 2020). These studies revealed how gender bias in occupational image search results influences people's perceptions about the presence of men and women in various professions. Our work builds upon these findings by examining the gender and racial bias in image search algorithm and offering innovative solutions for reducing bias in popular image retrieval framework using pre-trained VL model, such as CLIP (Radford et al., 2021).
Image retrieval Suppose  $C$  is the set of all text queries of interest (gender-neutral defined below), and  $\mathcal{V} = \{\nu_i\}_{i=1}^N$  is a database of  $N$  images, from which we retrieve images given query inputs. Each query input  $c \in C$  is relevant to a ground-truth set of images  $V_c^* \subseteq \mathcal{V}$  provided by human annotators.
The text-based image retrieval task aims at finding images from the database so they best match the text query inputs. Specifically, given any text query  $c \in C$  and a fixed retrieval size  $K$  ( $K \ll N$ ) as inputs, the goal is to design an algorithm that returns a bag of  $K$  images  $V_{c,K} \subseteq \mathcal{V}$ , where  $|V_{c,K}| = K$ , such that  $V_{c,K}$  contains as many relevant images from  $V_{c}^{*}$  as possible.
For evaluation purposes, a top- $K$  recall score ("Recall@ $K$ ) is usually computed to quantify whether the most relevant images are included in the retrieval output. Specifically, we write
$$
Recall@K = \frac{\sum_{c\in C}\left|V_{c,K}\cap V_{c}^{*}\right|}{\sum_{c\in C}\left|V_{c}^{*}\right|}\times 100\% ,
$$
where  $K$  is selected specifically for each dataset so that  $|V_{c}^{*}| \leq K$  for most queries  $c \in C$ ; this way, the recall can be realistically close to  $100\%$ .
Debiasing in image retrieval Using gender as a motivating scenario, we are interested in gender debiasing in cases where queries relate to human characteristics with no gender connotations, which we refer to as gender-neutral queries. For example, queries could be occupations that are widely open to all individuals, irrespective of gender (Organization, 2019), such as chef, nurse, social worker, etc.
We anticipate that gender-neutral queries should yield image retrieval sets that comprise equal representation of both male and female associated images, i.e., in a 1:1 ratio, which is consistent with the equal representation goal discussed by Wang et al. (2021a, 2023); Mehrotra and Celis (2021); Mehrotra and Vishnoi (2022). Note that we use gender debiasing as an example, but the same setting can be generalized to other debiasing issues such as racial discrimination, as we will demonstrate in the experiments.
Specifically, assume each image  $\nu \in \mathcal{V}$  has a gender attribute  $g(\nu)\in \{+1, - 1\}$ , which corresponds to the two genders manifested in the image, male and female, respectively. For each gender-neutral query  $c$ , the gender bias of the resulting retrieved bag of images  $V_{c,K}$  is defined as the normalized absolute difference between the numbers of images of each gender, i.e.,
$$
B \left(V _ {c, K}\right) = \frac {1}{K} \left| \sum_ {v \in V _ {c, K}} \mathbb {1} \{g (v) = + 1 \} - \sum_ {v \in V _ {c, K}} \mathbb {1} \{g (v) = - 1 \} \right| = \frac {1}{K} \left| \sum_ {v \in V _ {c, K}} g (v) \right|, \tag {1}
$$
which ranges from  $\frac{1}{K} \bmod (K, 2)$  (minimum bias) to 1 (maximum bias). In our retrieval approach, we average  $B(V_{c,K})$  across all gender-neutral queries  $c \in C$  as an evaluation metric for fairness, i.e.,
$$
\operatorname {A b s B i a s} @ K = \frac {1}{| C |} \sum_ {c \in C} B (V _ {c, K}) = \frac {1}{| C |} \sum_ {c \in C} \frac {1}{K} \left| \sum_ {v \in V _ {K} ^ {C}} g (v) \right|.
$$
The goal is to minimize AbsBias@  $K$  while maintaining a satisfactory retrieval Recall@  $K$ .
3.2 Similarity-based image-text matching
Overall framework As in previous works (Singh et al., 2003; Bai et al., 2014; Zaidi et al., 2019; Cao et al., 2022; Mukhoti et al., 2022), we use a similarity matching approach to tackle the image retrieval task. Such an approach involves aligning image and text features from largely pre-trained vision and language models during training time. At inference time, we rank images based on whether their features are similar to the input query text features and use the top-ranked images as our retrieval output. Specifically, we use an image encoder network  $f_{\phi}(\cdot)$  and a text encoder network  $f_{\psi}(\cdot)$  to embed both image  $\nu$  and text  $c$  inputs into a shared  $d$ -dimensional feature space as  $f_{\phi}(\nu), f_{\psi}(c) \in \mathbb{R}^{d}$ . A cosine similarity score  $S(\nu, c)$  is then computed to quantify the relevance between  $\nu$  and  $c$  (Singh et al., 2003; Bai et al., 2014; Wang et al., 2021a).
Training As in previous works (Chen et al., 2020; Radford et al., 2021), the training of the image and text encoders is achieved through optimizing an NT-Xent (Normalized Temperature-scaled Cross Entropy) loss with stochastic gradient descent on mini-batches.
Inference Given a new image database  $\mathcal{V}^{\mathrm{test}}$ , for each new query input  $c \in \mathcal{C}^{\mathrm{test}}$ , we compute its similarity score  $S(\nu_i, c)$  with every image  $\nu_i \in \mathcal{V}^{\mathrm{test}}$  and then pick the top- $K$  images that have the maximum similarity scores to form our retrieved set  $V_{c,K}$ .
3.3 Fairness criterion for image retrieval
Fairness can be defined in numerous ways in favor of different situations (Saxena et al., 2020). In the context of fair image retrieval (Wang et al., 2021a; Berg et al., 2022), we define our fairness objective as equal representation, implying that the retrieved image set through the retrieval algorithm should encompass an equal number of samples from each demographic group. We formally define our criterion as follows:
Definition 3.3.1 (Equal Representation) An image retrieval algorithm satisfies equal representation with respect to binary demographic attributes  $g(\nu)$  if,
$$
\mathbb {E} _ {V _ {c} \sim P} [ \mathbb {E} _ {v} [ \mathbb {1} (g (v) = + 1) ] ] = \mathbb {E} _ {V _ {c} \sim P} [ \mathbb {E} _ {v} [ \mathbb {1} (g (v) = - 1) ] ]
$$
where  $P$  represents the distribution of retrieved image set  $V_{c}$  corresponding to neutral queries  $c$ .
The above definition implies  $\mathbb{E}[B(V_c)]_{V_c\sim P} = 0$ , i.e., the expected bias is zero for any retrieved image sets. Our definition of equal representation is equivalent to "Equal opportunity for ranking distributions" introduced by Singh and Joachims (2017) when ranking position bias is constant.
3.4 Bias analysis
Using the image retrieval framework defined above as a basis, it is important to incorporate strategies to reduce gender (and race) biases of the retrieval output  $B(V_{c,K})$ . Below, we briefly analyze existing methods and discuss their limitations.
Existing methods Most methods address the gender fairness issue by enforcing model features to be less dependent on gender information. This is achieved by mainly two types of approaches, namely adversarial training (Edwards and Storkey, 2015; Berg et al., 2022; Xu et al., 2021) and mutual information (MI) minimization (Wang et al., 2021a, 2023). Specifically, adversarial training involves training a separate (adversarial) classifier network by adding an adversarial loss so that the adversarial network cannot distinguish gender given the encoded image features (Edwards and Storkey, 2015; Berg et al., 2022; Xu et al., 2021). Alternatively, MI minimization aims at reducing the MI between feature distribution and their gender by clipping feature dimensions highly correlated with gender (Wang et al., 2021a). Both methods encourage the model to extract features that are independent of gender. However, we argue below that enforcing this kind of independence between image features and gender is not sufficient to effectively eliminate gender bias in image retrieval.
Illustrative example We start by showing debiasing results using mutual information minimization. Similar insights can be obtained via adversarial training, which is shown in the Supplementary Material (SM). Figure 2 shows an example for the query "engineer" comparing three methods, namely, CLIP (no debiasing), MI-based debiasing and the proposed PBM (described below). For each method, we first compute, sort and group the similarity scores of all images into  $1\%$  quantiles ( $\sim 30$  images each). Then, for each quantile, we compute the gender bias as defined in (1). This analysis shows the relationship between gender bias and similarity scores. Note that for retrieval, we use samples with the largest similarity scores to form our retrieved bag of images, thus only the right-most data points (highlighted in the figure) are part of the retrieval set. We show percentiles in the figure to emphasize similarity rank rather than the similarity scores per se.
As shown in Figure 2(a), the original image retrieval algorithm with no debiasing based on CLIP tends to assign higher scores (at larger percentiles) to samples associated with male attributes, as evidenced by the high correlation between gender and the similarity scores. This makes the final retrieval output largely biased towards male samples. In contrast, Figure 2(b) shows the result of debiasing via MI minimization (Wang et al., 2021a). Using the regression line as a visual guide, we
(a) CLIP (no debiasing)
(b) MI debiasing
(c) PBM
Figure 2: Gender bias distribution for different methods using "engineer" as query. We compute similarity scores for all images from the test image set and plot them against gender bias in  $1\%$  quantile increments. The red circle marks the top- $K$  window covering the final retrieval output  $V_{c,K}$ .
(a) CLIP (no debiasing)
Figure 3: Comparing top-100 retrieval gender bias with full set similarity gender bias Spearman's correlation. For each query (occupation), we visualize the correlation between similarity score and gender bias against the top-100 retrieval gender bias. Two typical examples: "engineer" and "administrative assistant" are highlighted for illustration.
see that the correlation between gender and similarity score is close to zero. However, the gender bias is consistently larger than zero across the range of similarity scores, so the final retrieval is still biased. A desirable debiasing outcome is that for which the regression line aligns with the dotted line (zero gender bias across the similarity range), as shown in Figure 2(c). This is accomplished by the proposed PBM (described below).
Example with multiple queries In order to get a more generalizable understanding of the behavior shown in Figure 2, we repeat the analysis with all 45 occupations from the dataset provided by Kay et al. (2015) and visualize the results in Figure 3. Specifically, for each query (occupation), we calculate the Spearman's rank correlation between all (test-set) similarity scores and gender bias similar to Figure 2. We use Spearman's rank correlation because it is more appropriate to quantify associations between (variable) rankings. Moreover, we also calculate the gender bias of the final retrieval bag ( $K = 100$ ) for each occupation. Consistent with the illustrative findings on the "engineer" query, in Figure 3(a), we see a large correlation between similarity scores and gender bias as well as a large retrieval-set gender bias. MI minimization in Figure 3(b) manages to reduce the correlation between similarity score and gender but suffers from considerable gender bias; mostly clustered around 0.4. The proposed PBM not only reduces the correlation between similarity scores and gender bias but also significantly pushes the bias of each retrieval result to zero, which is the sought target debiasing outcome.