Evaluating Bias and Fairness in Gender-Neutral Pretrained Vision-and-Language Models
Authors: Laura Cabello
Emanuele Bugliarello
Stephanie Brandl
Desmond Elliott
Venue: EMNLP 2023 Main (2023)
Keywords: bias
fairness
multimodal

Abstract: Pretrained machine learning models are known to perpetuate and even amplify existing biases in data, which can result in unfair outcomes that ultimately impact user experience. Therefore, it is crucial to understand the mechanisms behind those prejudicial biases to ensure that model performance does not result in discriminatory behaviour toward certain groups or populations. In this work, we define gender bias as our case study. We quantify bias amplification in pretraining and after fine-tuning on three families of vision-and-language models. We investigate the connection, if any, between the two learning stages, and evaluate how bias amplification reflects on model performance. Overall, we find that bias amplification in pretraining and after fine-tuning are independent. We then examine the effect of continued pretraining on gender-neutral data, finding that this reduces group disparities, i.e., promotes fairness, on VQAv2 and retrieval tasks without significantly compromising task performance.

Evaluating Bias and Fairness in Gender-Neutral Pretrained Vision-and-Langgage Models
Laura Cabello Emanuele Bugliarello Stephanie Brandl Desmond Elliott
Department of Computer Science, University of Copenhagen, Denmark emanuele, brandl, de Jedi ku. dk
Abstract
Protraining Stage
Fine-Tuning Stage
Pretrained machine learning models are known to perpetuate and even amplify existing biases in data, which result in unfair outcomes that ultimately impact user experience. Therefore, it crucial to understand mechanisms behind those prejudicial biases ensure that model performance does not result in discriminatory behaviour toward certain groups or populations. In this work, we define gender bias our case study. We quantify bias amplification pre- training and after fine-tuning on three families of vision-and-land and-language models. We investigate the connection, if any, between the two Iearn- ing stages, and evaluate how bias amplification reflects on model performance. Overall, we find that bias amplification pretraining and after fine-tuning amine the effect of continued pretraining on gender-neutral data, finding that this reduces group disparities, i.e, promotes fairness, on VQAv2 and retrieval tasks without significantly compromising task performance.
MD
Extrinsic Bies
Intrinsic Performance
MD
Fairness
GENDER PRETRAINING
Figure I: V&L model pretrained on data 1) (Mp) further pretrained on gender-neutral multimodal data DN, resulting in gender neutral V&L model (M)). Both models can then be used in a two-phase analysis: 1) bias amplification is measured on the intrinsic bias of pretrained models, and 2) bias amplification, task performance and fairness are evaluated on the extrinsic performance of fine-tuned models.
independent. We then
and their context make predictions, and thus are susceptible to unwanted biases. However, these authors do not explore the broad landscape of V&L models and focus on biases in common visual datasets (Wang et al., 2022; Hirota et al., 2022a), only on pretrained models (Zhou et al., 2022) or only focus on one application, e.g., image caption- ing (Hendricks et al., 2018: Hirota al., 2022b) semantic segmentation (Lee et al., 2021).
1 Introduction
As shown by Mitchell (1980) and Montañez et al. (2019), inductive biases are essential for learning algorithms to outperform random guessing. These task-specific biases allow algorithms to generalize beyond training data but, necessarily, they should be conflated with prejudicial or unwanted bi- ases. Unwanted bias, such as bias against de- mographic groups, can found in many appli- cations, from computer vision systems to natural language processing (NLP). Vision-and-language (V&L) models lie at the intersection of these areas, where one of the key challenges is deploying robust models to perform high-level reasoning based on the multimodal context instead of exploiting biases in data (Zhao et al., 2017).
In this work, we investigate what extent the unwanted bias in V&L model caused by the pre- training data. To answer this question, we focus on one important aspect of bias encoded in V&L mod- els, namely bias amplification. Bias amplification occurs when model exacerbates unwanted biases from the training data and, unlike other forms of bias, it is not solely attributed to the data, yet it vary greatly during training (Hall al., 2022).
We explore bias amplification in two encoder- only V&L models: LXMERT (Tan and Bansal, 2019) and ALBEF (Li al., 2021), and the encoder-decoder model BLIP (Li et al., 2022). Specifically, we quantitatively and qualitatively
Multiple studies (Lee al., 2021; : Hirota al., 2022b; Zhou et al., 2022) have shown that V&L models leverage co-occurrences between objects

---

analyse the relationship between the bias encoded in pretrained models, and after fine-tuning on down- stream tasks including visual question answering, visual reasoning and image-text retrieval.
2 Related Work
Bias in language In general, bias can be defined as "undue prejudice" (Crawford, 2017). Studies tar- geting language models (Kurita al., 2019; Zhao et al., 2019) have shown that biases encoded in pre- trained models (intrinsic bias) can be transferred to downstream applications (extrinsic bias), but the relationship between these biases is unclear. There are several studies GolldfrbbTarranne et al.. 2021; Delobelle et al., 2021; Kaneko et al., 2022; Cao et al., 2022; Orgad al., 2022), showing that intrinsic bias in language models does not consis- tently correlate with bias measured extrinsically on downstream task or, similarly, with empirical fairness (Shen et al., 2022; Cabello et al., 2023). Contrarily, Jin et al. (2021) observed that the effects of intrinsic bias mitigation are indeed transferable in fine-tuning language models. To the best of our knowledge, are the first to investigate if the same holds for V&L models.
While bias can be studied with respect to any protected attribute, the majority of NLP research has focused (binary) gender (Sun et al., 2019; Stanczak and Augenstein, 2021; Shrestha and Das, 2022). We also use gender bias as our case study but different to previous work, we advocate for the inclusion of gender-neutral terms (Dev et al., 2021) and consider three gender categories based on visual appearance: male, female and gender- neutral (e.g., PERSON). The use of both visual and grammatical gender information across V&L tasks is needed for identifying the target of, for exam- ple, a question. But the demographies of the sub- ject should not solely influence the outcome of the model. Otherwise, the model may reinforce harm- ful stereotypes resulting in negative consequences for certain group identities (van Miltenburg, 2016).
Motivated by this argument, we investigate the effect of shifting the projection of gender-marking to a gender-neutral space by continued pretraining on gender-neutral multimodal data-a a form of do- main adaptation (Gururangan et al., 2020)-and how it reflects on task performance after fine-tuning, Figure depicts an overview of our full workflow.
Bias in vision & language Prior research ob- served the presence of gender disparities in visual datasets like COCO (Bhargava and Forsyth, 2019; Zhao al., 2021; Tang et al., 2021) and Flickr30k (van Miltenburg. 2016). Recent studies also re- vealed the presence of unwanted correlations V&L models. Prejudicial biases found in V&L models are not only attributed to one domain, i.e., vision or language, but they are compound (Wang et al., 2019), and this should be studied together. Srinivasan and Bisk (2021); Hirota et al. (2022b) and Zhou et al. (2022) show that different model architectures exhibit gender biases, often preferring to reinforce a stereotype over faithfully describing the visual scene. Bianchi et al. (2023) show the presence of stereotypes in image generation mod- els and discuss the challenges of the compounding nature of language vision biases. Another line of work addresses visual contextual bias (Choi et al., 2012; Zhu et al., 2018; Singh et al., 2020) and study a common failure of recognition models: an object fails to be recognized without its co-occurring con- text. So far, little work has investigated bias ampli- fication in pretrained V&L models. Our study is among the first to cast some light on the gender bias encoded in pretrained V&L models and evaluate
Contributions We examine whether bias ampli- fication measured on pretrained V&L models (in- trinsic bias) relates to bias amplification measured on downstream tasks (extrinsic bias). We show that biased pretrained model might not translate into biased performance on downstream task to a similar degree. Likewise, we measure model fair- ness through group disparity and show that it is not unequivocally related bias in a model. Fur- thermore, we empirically present a simple, viable approach to promote fairness in V&L models: per- forming an extra epoch of pretraining on unbiased (gender-neutral) data reduces fine-tuning variance and group disparity VQAv2 and retrieval tasks on the majority of models studied, without signifi- cantly compromising task performance.
We make our code publicly available to ensure reproducibility and foster future research.'
As first suggested by Goldfarb -arraatt-aaaatt--- et al. (2021). we can broudly categorize bias into intrinsic and extrinsic. There- fore, intrinsic metrics are applicd directly to word representa- tions and relate bias to the geometry of the emhedding space, whereas extrinsic metrics evaluate bius in downstream tasks.
'http:/ http:://gtthht com/coastalcph/ gender-nnutral-vl

---

how translates to downstream performance.
captions include word from a set of male-related tokens (e.g., BOY), and no caption includes word from the set of female-related tokens (e.g, GIRL); and vice-versa for 'Female'. Images are labelled as 'Neutral' if most the subjects are listed as gender-neutral (e.g., PERSON), or if there is no ma- jority gender mention in the texts. Finally, images are discarded from the analysis when the text men- tions both male and female entities, there are no people mentioned. This process can be applied to both pretraining data and downstream task data. See Appendix A for the complete word list.
Gender-neutral language Zhao et al. (2019) ex- amine the effect of learning gender-neutral embed- dings during training of static word embeddings like Gl (Pennington et al., 2014). Sun al. (2021) and Vanmassenhove et al. (2021) present rule-based and neural rewriting approaches to gen- erate gender-neutral alternatives in English texts. Brandl et al. (2022) find that upstream perplexity substantially increases and downstream task perfor- mance severely drops for some tasks when gender- neutral language is used in English, Danish and Swedish. Amend et al. (2021) show that the sub- stitution of gendered for gender-neutral terms on image captioning models poses viable approach for reducing gender bias. In our work, we go one step beyond and investigate the effect of contin- ued pretraining V&L models on in-domain data where gendered terms have been replaced by their gender-neutral counterparts (e.g., sister sibling).
Measuring Bias in V&L Models
4.1 Intrinsic Bias
When we measure the intrinsic bias of a model, we are interested in whether there are systematic dif- ferences in how phrases referring to demographic groups are encoded (Beukeboom et al., 2014). We can measure the intrinsic bias using the model's 's language modelling task, where the tokens related to grammatical gender are masked.
3 Problem Formulation
Let MD be a V&L model pretrained on cor- pora D. The masked words related to grammat- ical gender are categorised on N disjoint de- mographic groups
We characterize the gender bias encoded in V&L models in two-phase analysis:
i) Intrinsic bias: First, we investigate the bias encoded after the V&L pretraining phase.
(Male, Female, Neutral} based on reported visual appearance in the image. The gender associated with an image is consid- ered as the ground truth (see previous section for more details). Let J: for [1, be the cate- gorical random variable corresponding to the pres- ence of the group i. We investigate the gender- context distribution: the co-occurrence between attributes A (........ ||44) ]. e.g. gender terms, for a demographic group Js, and contextual words T = {ti,..., (T), e.g., objects that appear in a given text. This results in a co-occurrence ma- trix C. that captures how often pairs of attribute- context words occur in defined context S, e.g., image caption in a corpus C. Formally, for every demographic group gi, over the A, attributes and T objects, and all possible contexts in corpus C
ii Extrinsic bias and task performance: Second, fine-tune the models common down- stream tasks to further investigate how bias affects model performance.
These investigations will be performed using a set of original, pretrained models MD. and models that have been further pretrained on gender-neutral data MN in order to mitigate any biases learned during pretraining (64.4). We hypothesize that this bias mitigation technique will decrease both intrin- sic and extrinsic biases encoded in the models.
Data Our analysis relies on data where the gen- der of the main actor of the image is known. This is, to some degree, annotated in the crowdsourced text, é.g., image captions or questions." 3 Following Zhao et al. (2017) and Hendricks et al. (2018), im- ages are labelled 'Male' the majority of its
AA
T
C
S(a,, 4)) with é [1,
SEC j=1 k=1
(1) where S(aj, (x) 1 if the attribute and object co-occur, zero otherwise. Based on C standard
'Zhao et al. (2021) annotated samples from the COCO datasel (Lin et al., 2014) with the perceived attributes (gen- der and skin-tone) of the people in the images. However, their gender labels agree on 66.3% of the images compared to caption-derived annotations. To be consistent across all datasets used in our project, we will not use their human- collected annotations for analysing gender bias on COCO.
*We define gender correlations as our case study of rep- resentational hias, but note that our methodology can he ex- tended to analyse bias with regard to any protected attributc(s).

---

statistical metrics like precision, recall and Fl can be computed. In addition, we will quantify the bias amplification
T - SKATEBOARD in the model predictions, com- pared to the prior distribution from the dataset. In contrast, negative value indicates that model pre- dictions diminish the bias present in the dataset. A value of 0 implies that the model does not amplify the bias present in the dataset. Note that this does not imply that the model predictions are unbiased.
given model MD to better understand the degree of bias exacerbated by the model. We use the metric presented by Wang and Russakovsky (2021), which is described in more detail in the next section.
4.2 Bias Amplification
4.3 Extrinsic Bias & Fairness
We use the BiasAmp metric introduced by Wang and Russakovsky (2021), as it accounts for varying base rates of group membership and naturally de- couples the direction of bias amplification: While BiasAmpr->A measures the bias amplification due to the task influencing the protected attribute pre- diction, 5 BiasAmpA >T measures the bias ampli- fication due to the protected attribute influencing the task prediction. We give a concise treatise of BiasAmp here, and refer to Wang and Rus- sakovsky (2021) for further details.
The second phase of our analysis measures extrin- sic bias amplification: downstream performance and fairness (group disparity). A given model is fine-tuned on downstream tasks that require dif- ferent reasoning skills based on the image context. We evaluate model performance with respect to the three demographic groups defined in A and com- pare results in search of the more equitable system.
4,4 Gender-neutral Domain Adaptation
Motivated by the fact that models are known to acquire unwanted biases during pretraining (Hall et al.. 2022), we also investigate what happens if model MD is further pretrained for one additional epoch on gender-neutral data, with the goal of creat- ing a more gender- gender-neutral neutral model M. We hypothe- size that this may be sufficient to reduce the biases encoded in the original model. Given dataset D, new dataset DV is created by substituting gender-related tokens in the text for gender-neutral tokens. The substitution is based on hand-crafted lexicon, egg., woman or man may be substituted person. 7 The new model MN is used for both the intrinsic and extrinsic bias evaluations.
In our setup, the set of attributes a E A is given by A = (Male, Female, Neutral), and the set of tasks (or objects) t T are the most frequent nouns co-occurring with gendered terms in the training sets (see Appendix A for details). De- note by P(T, = 1) the probability that an example in the dataset belongs to class t. And, similarly, P(T, - 1) the probability that an example in the dataset is labelled as class t by the model. Wang and Russakovsky (2021) introduce two terms to dis- ambiguate the direction of bias amplification. The first term, Aat, quantifies the difference between the bias in the training data and the bias in model predictions.
Experimental setup
The second term, Yat, identifies the direction of correlation of A, with Tú: that is, Vot alters the sign of the Agt to correct for the fact that the bias can have two directions. Thereby,
5.1 Models
We take the LXMERT architecture (Tan and Bansal, 2019) as popular representative of V&L mod- cls, and build our controlled analysis on VOLTA (Bugliarello et al., 2021). VOLTA is an implementa- tion framework that provides a fair setup for com- paring V&L models pretrained under the same con- ditions, which enables us to compare the influence of diverse training data on representational bias. In this case, LXMERT, 180K refers to the original checkpoint and LXMERT 3m to the model trained on CC3M (Bugliarello et al., 2021). We also study
BiasAmp...r
Σ Aar (1 Mos) ) Dut (2)
[A[T]
BiasAmpA-IT will be positive the model pre- dictions amplify the prevalence of class label t T between groups A in the dataset. For instance, bias amplified if Ag MALE im- ages are more likely to appear in the presence of a
'We do not consider gender prediction as a tusk per se, as gender -or any other sensitive attribute- prediction entangles a complex categorization and moral debate (Keyes, 2012 18; Lrson, 2017). Instead, we use a ML. task proxy and ask the model to predict the subject of sentence given its context.
"See Appendix
"Note that when the pretraining data D is composed of multiple corpora, we argue that domain adaptation to a non- biased space should be performed only on clean data, and, therefore, | I S (D].

---

ALBÉF in two sizes and BLIP. Table 1 lists the models included in our analysis.
Model (MD)
Gender-neutral model (M)
LXMERT
LXMERT 1SOK
LXMERT
LX EXMERTS
5.2 Gender-neutral Data
3M
ALBEFAM
ALBE COCO, ALBEES 4M
JM ALBEE ALBEE 14M
As a natural extension to study representational gender bias, we want to evaluate to what extent gender-neutral data helps to mitigate gender bias. Amend al. (2021) showed that gender-neutral training might be viable approach for reducing gender bias in image captioning models. We study its effect in more generic pretrained V&L models.
ALBEFA4
14M
129M
BLIPN
129M
Table 1: Summary of the models. The subscript in the model name indicates the number images in the pretraining set. All gender-neutral models pretrained MANNNNN
with in-domain data (LXMERT 180K and BLIPI 129M on COCO; LXMERT 3M on CC3M). For models with more than one gender-neutral version, the superscript indicates the dataset used for gender-neutral pretraining.
The gender-neutral pretraining data is the result of substituting terms with grammatical gender for gender-neutral equivalents, e.g., "A woman walk- ing her dog" translates into "A person walking their dog." To this end, create a list of gender enti- ties by merging previous hand-curated lexicons used in similar context to ours, provided by An- toniak and Mimno (2021).²
2021; Srinivasan and Bisk, 2021; Agarwal et al.,
2021; Cho et al., 2022) and expose representational bias in a masked language modelling (MLM) task. The words masked are gendered terms given by the same lexicon used in $5.2. Personal pronouns (if any) are also masked to avoid leaking gender infor- mation into the model representation. For example, "A woman walking her dog" would be masked as "A [MASK] walking [MASK] dog". The image asso- ciated with each sentence is also input to the model, in setup that reflects the pretraining conditions.
Starting from a pretrained checkpoint, we per- form an extra epoch of pretraining. The training is done based on a linear function that increases the probability for model to learn from gender- neutral captions. The starting rate is p=0.15 and, as the training progresses, the probability of getting a gender-neutral caption increases to p=1.0 at the last step. Note that as the probability of getting a gender-neutral caption increases, the learning rate decreases. This methodology supports our intuition that starting with gender-neutral corpus would be too drastic for the model to adapt to, and instead cause catastrophic forgetting.
We investigate the intrinsic bias of the models as detailed in $3, ie.., we look at the co-occurrence of context words (e.g., car, ball) with particular word choices from the model (e.g., gender words like woman, child). Previous work (Sedoc and Un- gar, 2019; Antoniak and Mimno, 2021; Delobelle et al., 2021) showcases how the measure of bias can be heavily influenced by the choice of target seed words. To avoid misleading results from low frequency words, we define the set of target words to be the 100 most frequent common nouns that co-occur with the gender entities in the correspond- ing training data. Table 2 provides a summary of gender distribution.
Finally, we continue pretraining the original model checkpoints for an extra epoch without the gender-neutral alternative (i.e., p000). The evalu- ation on this new checkpoint will help us to draw conclusions on longer training, as well as ensure the correct implementation of our setup.
5.3 Evaluation Tasks
For evaluation of downstream tasks, we report task performance and analyse group disparities. Bias amplification reported on the validation splits.
To evaluate intrinsic bias, we do not look at the exact word prediction but instead consider two op- tions to annotate the gender of the predicted word. First, we can extract and sum the probabilities of all male, female and gender-neutral tokens within our set to select the most probable gender entity. However, given that the distributions of tokens fol- lows Zipf's Law, the probability mass computed for each gender group is nearly equal, yielding in- conclusive results. Therefore, we use the gender category of the most probable token. Then, the bias
We follow standard practice for assessing gender bias in V&L models (Zhao et al., 2017; Hen- dricks al., 2018; Wang et al., 2019; Tang al.,
MLM
*See Appendix A for the complete list.
"We deliberately omit tokens like 'actor' from the list if the female (or male) equivalent is not always used (people do not always use the word 'actress' when referring to a female and 'female" as we suspect that they are more often used on non-human entities.
character). We also discard 'male"

---

coCO
CC3M
VQAv2
GQA
NLVR2
F30R
report Recall @ on the Flickr30K (Plummer al., 2015) benchmark. Bias amplification is measured on the subset of data targeting people. In IR, we query the model with captions that include a word from the set of male-related or female-related to- kens and compare to the gender annotated in the image retrieved. In TR, we query the model with images annotated as 'Male' or 'Female' and com- pare the gendered terms in the caption retrieved. Captions with gender-neutral terms are treated as separate case to assess how often the models retrieve images from each group, yet the image retrieved could be potentially valid for any gender case. In both subtasks, we consider that the model does not amplify gender bias when the image caption retrieved has gender-neutral subject.
Image
Iniage
Question
Question
Seatence
Image
Male
725
901
20000
8265
91
345
Female
945
4860
207
Neutral
1187
1095
18549
4442
377
336
Total
2275
2941
48047
17567
567
889
Table 2: Gender distribution across validation splits each dataset. Note that for COCO, this refers to the minival split in (Tan and Bansal, 2019). COCO and F30K have five captions per image. Gender was inferred from image captions F30K. Gender was inferred from questions in VQAv2, GQA and from the sentence given in NLVR2.
COCO, CC3M and
present in model predictions is measured with the statistical and bias amplification metries presented in $4.2.
Results
Visual Question Answering VQA (Antol et al.. 2015) requires the model to predict an answer given an image and a question. LXMERT formulates VQA as multi-answer classification task, and AL- BEF and BLIP treat it as language generation task. We evaluate models on the VQAv2 (Goyal al., 2017) and GQA datasets (Hudson and Manning, 2019), and report performance as VQA-Score and accuracy, respectively.
6.1 Intrinsic Bias
We evaluate intrinsic bias in encoder-only models. Considering that bias varies as a function of the bias in a dataset, amongst other variables (Hall et al., 2022), we define our experiments with LXMERT variants as our control setup: the same model archi- tecture is trained with the same hyperparameters on disjoint corpora yielding two versions of the model, LXMERT 180K and LXMERT:M.
Bias amplification is measured on the subset of question-answer pairs targeting people. Gender is inferred from the question, considering all the gender entities presented in Appendix We filter any answer category whose answer does not occur with gender entities at least 50 times in the training set. Finally, numerical and yes/no question-answer pairs are also removed leaving total of 165 answer categories in VQAv2 and 214 in GQA.
Gender-neutral pretraining mitigates gen- dered outputs Figure 2 shows results for LXMERT 180K models; complete results are in Ap- pendix C. A model is penalised when it predicts token from the opposite gender, but we consider a gender-neutral term as a valid output." 10 The mod- els pretrained with gender-neutral data, have near perfect Fl performance as they learnt to predict gender-neutral tokens when their standard counter- parts, LXMERT 180K and LXMERT 3M. had low confidence on the most probable token."! We pre- sume these images where the visual appearance of the main subject is unclear. Interestingly, the trade-off between precision and recall has opposite directions for Female and Male groups vs Neutral in LXMERT 180K and LXMERT the models tend to output female- and male- tokens more often than neutral-related, even when the subject in the
Natural Language for Visual Reasoning NLVR2 (Suhr al., 2019) requires the model to predict whether a text describes a pair of images. The notion of bias amplification considered this project would require us to manually annotate the gender from all the images to be able to extract gender-context patterns from the training data. For this reason, we only evaluate the group disparity in NLVR2 through differences in performance, reported accuracy.
1S0K
Image-Text Retrieval This retrieval task con-
Predicting gender-neutral term shows that the model understands the depicted visual concept the generic level. 'The models do not forget to predict gender-related to- kens. LXMERT" ssxx predicts <37% of the time word from the set of neutral-related tokens (compared to ~20% in LXMERT ((0))
tains two subtasks: text-to-image retrieval (IR), where we query the model with a caption to retrieve an image, and image-to- imaggeto-textt retrieval (TR), where we use an image to retrieve a suitable caption. We

---

Precision COCO CC3M
Recall COCO
F1
provement observed in LXMERT Yo0K mainly due to the extra pretraining steps regardless of gender-neutral data. Our strategy for mitigating gender bias on pretrained models generally leads to lower task performance on NLVR2 and image retrieval, revealing a trade-off between bias miti- gation and task performance. The same trade-off has been observed in language models (He et al., 2022; Chen et al., 2023). However, gender-neutral models report similar or even superior performance on question answering and text retrieval tasks com- pared to their original versions.
CC3M
COCO
CC3M
LXMERT 1o0X LXMERT Leok
M
0.73
0.67
0.95
0.91
0.82
0.77
0.77
0.78
0.91
0.84
0.84
0.81
N
0.54
0.48
0.7
0.65
M
0.99
0.95
1
0.99
1
0.97
0.99
0.96
1
0.99
0.99
0.98
0.99
0.92
1
0.96
0.6
0.7
0.8
0.9
1.0
Figure 2: Statistical analysis of gender bias in ML. with gendered terms masked. Predicting token from the gender-neutral set is always considered correct (Pre- cision=1). Models report higher recall scores for Male (M) and Female (F) groups, showcasing the complete- ness of positive predictions; it is the opposite for Neutral (N) tokens.
models consistently reduce group disparity Group performance is depicted in Figure for a subset of models and tasks. Table 7 in Appendix D shows the complete results. We ob- serve that group disparity is consistently reduced on VQAv2 and retrieval tasks. An exception are LXMERT models, which show a minor, undesir- able increase in group disparity on VQAv2, GQA and text retrieval tasks. For instance, in question- answering tasks with LXMERT, we observe a re- duction in the min-max gap of 4.5 (LXMERT)) 180k) points in VQAv2, while the min-max gap increase in GQA is only of 0.4 points. Note that Tan and Bansal (2019) pretrained LXMERT on GQA train and validation data, which results in very high performance (~85.0 O for all groups) on the GQA validation set. We speculate that the gains in performance equality across groups could be due to shift of the final word representations to a more equidistant vector space between gendered terms and their context. That is, the conditional probabil- ity distribution of a gendered term given its context is smoother across different demographic groups. We leave exploration of this for future work. In re- cent work, Feng al. (2023) continued pretraining language models on partisan corpora and observed that these models do acquire (political) bias from said corpora. In our case, the continued pretraining could make the MN models more robust regarding gendered terms.
Gender-neutral
image was annotated gender neutral (low recall).
Pretrained models reflect training data biases Table shows the aggregated bias amplification measured in encoder-only model variants. Our bias mitigation strategy has the same consistent behaviour across LXMERT models and evalua- tion data (COCO or CC3M): models tend to re- flect the same degree of bias present in the data (BiasAmpT-AA closer to zero). ALBEIN-COCO and ALBEFN-CCOM models benefit from pretrain- ing on gender-neutral data differently, both de- crease the overall bias amplification. Wang and Russakovsky (2021) caution against solely report- ing the aggregated bias amplification value, as it could obscure attribute-task pairs that exhibit strong bias amplification. We report it here as relative metric to compare the overall amplified bias between the models, and should not be consid- ered in its own. See Appendix for results broken down by gender.
14M
We also investigated the equivalent to LXMERTN 3M but pretrained on gender-neutral data for a reduced number of steps to match those in LXMERTN 180K- We verified that more pretraining steps gender-neutral data equates to reduced bias amplification in absolute terms.
Gender-neutral training reduces fine-tuning variance Dodge et al. (2020) and Bugliarello et al. (2021) analysed the impact of random seeds in fine-tuning. We do this analysis our trol setup and observe that gender-neutral variants of LXMERT consistently report lower variance in performance on all tasks, except for NLVR2. We, however, observe a strong variance in the fne-tuning process for NLVR2 due to the random
6.2 Extrinsic Bias & Fairness
Trade-offs in task performance Downstream performance on the test sets shown Table LXMERT 180K may require more pretraining steps to converge, as we verify that the performance im-

---

LXMERT 180K
LXMERT 1sOK
LXMERT 3M
LXMERT
ALBET14M
ALBEF
ALBEPN.COM
COCO
-0359
-00008
-00617
-0014
-0072
-.0517
-0792
CC3M
-0346
-0002
-0007
-0000
-0018
-00367
-0050
Table 3: BiasAmpr-1a averaged over attributes (gender entities) and tasks (top-100 nouns) for LXMERT and ALM models. Light and dark backgrounds indicate bias amplification measured in-domain and out-of-domain data respectively. Negative values indicate an overall decrease of the bias model's predictions.
75
14
1/1
+P
0C
|XMERT
81.0P
(a) VQA22
(b) GQA
(c) F30K IR
(d) F30K TR
Figure 3: Validation-set results of selected models (o: LXMERT 1SOK, ALBEF 14M and BLIP 129M) and their gender- neutral version (3: LXMERT 180K ALBEFN-COCO and BLIP 129M O: ALBEF We report VQA-accuracy in VQAv2, accuracy GQA, and Recall @ in F30K by gender group: male (M), female (F), and neutral (N).
VQAv2
GQA test-dev
NLVR2
F30K
However, contrary to what we observed in pre- trained models (Table 3), there is no evidence that the gender-neutral pretraining influenced pos- itively (nor negatively) the extrinsic bias of the models: depends on the model, downstream task and gender group (see Appendix for results on BiasAmp, >r' fine-tuning variance). Figure 4 displays BiasAmp broken-down by gender category measured on GQA for a subset of mod- els. Whereas the degree of bias amplification is fairly consistent between model MD and MN in VQAv2 (see Appendix D), there is higher variance in GQA: ALBEFN.C C-COO reduces the bias ampli- fication compared ALBEF 14M, but we observe the opposite effect on BLIPN-COCO.
test-dev
test-P
test IR
test TR
LXMERT BOK
70.3
$9.4
74.5
53.0
61.1
LXMERT TROK
71.6
59.3
74.5
53,9
66.2
1M
67.2
55.4
71.5
54.4
59.5
LXMERTY
70.0
ALBEFM
72,9
$6.6
79.3
82.6
93.3
ALBE
72,9
56.3
77.1
82.5
ALBEFNOCIM
72.9
56.6
78.4
82.4
94.2
ALBEF
58.4
82.4
ALBEFN-CO00
74.1
57.3
12.3
85.5
95.4
ALBERN-COM
74.1
58.1
81.0
85.1
95.2
BLIP 120M
75.3
38.1
79.7
87.5
96.7
BLIP 139M
75.2
58.3
79.3
86.9
96.2
Table 4: Test results for model MD and its gender- neutral version MN. We report VQA-accuracy in VQAv2, accuracy in GQA and NLVR2, and Recall @ in F30K. Results for original models computed by us.
In retrieval tasks, we look into models' behavior when querying them with neutral instances. Re- gardless of the degree of intrinsic bias in the model. models exhibit the same trend: in IR, all models mostly retrieve images labeled as 'Neutral', but twice as much 'Male' images as 'Female'. We find similar results for TR, i.e., query images whose main actor is defined as Neutral, but, in this see- nario, only half of the captions retrieved relate to people. See Appendix D for detailed results.
weight initialisation of the classification layer. See Appendix E for specific results across 6 runs.
Intrinsic & extrinsic bias are independent We estimate bias amplification in VQA tasks by evalu- ating the fluctuations in models' predictions when they differ from the correct answer. Otherwise, the models are said to not amplify the bias from the data. We find that all model variants MD and MN reduce the gender bias across tasks.
7 Conclusion
This result is inexplicably low, despite fifteen attempts at fine-tuning with different random seods. We saw similar instabilities when fine-tuning the released 1. 1.MERTT models, but we found seeds that gave above-chance accuracy.
This paper presented a comprehensive analysis of gender bias amplification and fairness of encoder- only and decoder V&L models. The in-

---

0.05
Studying the effects of gender-neutral pretraining on V&L models with frozen language model, such as ClipCap (Mokady et al.. 2021) and BLIP- (Li et al., 2023), left as future work.
0.00
-0.05
Due to computational limitations, we restricted most of our analysis to single runs. We perform first analysis across multiple random seeds for LXMERT models Appendix E. There, we no- tice that gender-neutral models seem to have lower variance after fine-tuning. Yet, the cross-seed per- formance of a given model can fluctuate consider- ably for some tasks (e.g.. NLVR2), corroborating previous findings from Bugliarello et al. (2021). Likewise, bias amplification, along with other fair- ness metrics like group disparity, often fluctuates across runs. We report bias amplification variance in fine-tuning of LXMERT models, but the absence of confidence intervals for all models and tasks due to the same reason stated above- should be considered. We hope to motivate future work to address this issue.
BiusAmpa
-0.10
-0.15
-0.20
-0.25
freaale
reurra
0.30
LXMERT INI
ALBEFM
AM
7984
Figure 4: Bias amplification measured question- answering (GQA) broken down by gender group. M are gender-neutral pretrained on COCO.
trinsic bias analysis shows consistent results - in terms of bias mitigation in models trained on gender-neutral data, even if these models reflect biases present in data instead of diminishing them (as we observed with LXMERT). In line with previ- ous findings in language models (Goldfarb- Tarrant et al,,, Kaneko et al., 2022; Orgad et al., 2022), intrinsic bias in V&L models does not necessarily transfer to extrinsic bias on downstream tasks. Sim- ilarly, we find that the bias in a model and its empir- ical fairness -gooup disparity on task performance- are in fact independent matters, which is in line with the NLP literature (Shen et al., 2022; Cabello et al.. 2023). Intrinsic bias can potentially rein- force harmful biases, but these may not impact the treatment of groups (or individuals) downstream tasks. We believe that bias and fairness should al- ways be carefully evaluated as separate matters. One of they key findings of our work is that the extra pretraining steps on gender-neutral data are beneficial to reduce the group disparity in every model architecture tested on VQAv2, and in the ma- jority of models for both retrieval tasks. Crucially, there is no penalty to pay for this fair outcome: the overall task performance of gender-neutral models is similar or better than their original versions.
Moreover, despite the existence of multilingual multimodal datasets (Elliott al., 2016; Liu al., 2021; Bugliarello et al., 2022, inter-alia), our ex- perimental setup is limited to English datasets and models. Studies of (gender) bias using only English data are not complete and might yield inaccurate conclusions, albeit overcoming the structural per- vasiveness of gender specifications in grammatical gender languages such us German or Spanish is not trivial (Gabriel et al., 2018). Likewise, our work considers a single dimension of social bias (gen- der). Further research on analyzing social biases on V&L models should account for intersectionality: how different social dimensions, e.g., gender and race, can intersect and compound in ways that can potentially impact model performance on most dis- favoured groups, egg., Black Women discussed in Crenshaw (1989).
Ethics Statement
The models and datasets used in this study are pub- licly available, and we strictly follow the ethical implications of previous research related to the data sources. Our work is based on sensitive informa- tion such gender, based on reported visual ap- pearance in the image captions. We would like emphasize that we are not categorizing biological sex or gender identity, but rather using the given image captions as proxies to the outward gender appearance.
Limitations
The framework to characterize gender bias in V&L presented in this study is general and extensible to analyse other forms of bias in multimodal models. We consider three base architectures settle on the implementation. However, our work would benefit from analyzing a wider range of models.

---

Acknowledgments
Federico Bianchi, Pratyusha Kalluri, Esin Durmus, Faisal Ladhak, Myra Cheng. Debora Nozza, Tat- sunori Hashimoto, Dan Jurafsky, James Zou, and Aylin Caliskan. 2023. Easily accessible text-to- image generation amplities demographic stereotypes at large scale. In Proceedinggs of the 2023 ACM Conference on Fairness, Accountability, and Trans- parency, FAceT '23, page 1593-1504, New York. NY, USA. Association for Computing Machinery.
We are grateful to Benjamin Rotendahl and Rita Ramos for initial discussions about data and eval- uation. We also thank members of CoAStaL and LAMP groups for their valuable feedback. Laura Cabello is funded by the Novo Nordisk Founda- Emanuele Bugliarello is supported by the funding from the European Union's 's Horizon 2020 research and inno- vation programme under the Marie Sklodowska- Curie grant agreement No 801199. Stephanic Brandl is funded by the European Union under the Grant Agreement no. 10106555, FairER. Views and opinions expressed are those of the author(s) only and do not necessarily reflect those of the European Union or European Research Executive Agency (REA). Neither the European Union nor REA can be held responsible for them. This work was supported by research grant (VIL53122) from VILLUM FONDEN.
tion (grant NNF 20$A0066568).
Stephanie Brandl, Ruixiang Cui, and Anders Søgaard. 2022. How conservative are language models? adapt- ing to the introduction of gender-neutral pronouns. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computa- rional Linguistics: Human Language Technologies, pages
Seattle, United States. Association for Computational Linguistics.
Emanuele Bugliarello, Ryan Cotterell, Naoaki Okazaki, and Desmond Ellioot. 2021. Multimodal pretraining unmasked: A meta-analysis and unified framework of vision-and- lannuuugg BERT'. Transactions of the Association for Computational Linguistics, 9:978- 994.
Emanuele Bugliarello, Fangyu Liu, Jonas Pfeiffer, Siva Reddy, Desmond Elliott. Edoardo Maria Ponti, and Ivan Vulić. 2022. IGLUE: A benchmark for transfer learning across modalities, tasks, and languages. Proceedings of the 39th International Conference Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 2370-2392.
References
Sandhini Agarwal, Gretchen Krueger, Jack Clark, Alec Radford, Jong Wook Kim, and Miles Brundage. 2021. Evaluating clip: Towards characterization of broader capabilities and downstream implications.
PMLR.
Jack J Amend. Albatool Wazzan, and Richard Souvenir. 2021. Evaluating gender-neute neutral training data for automated image captioning. In 2021 IEEE Inter- national Conference on Big Data (Big Data), pages 1226-1235.
Laura Cabello, Anna Katrine Jørgensen, and Anders Søgaard. 2023. On the independence association bias and empirical fairness in language models. In Proceedings of the 2023 ACM Conference on Fair- ness, Accountability, and Transparency, FAceT '23, page 370-378, New York, NY, USA. Association for Computing Machinery.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Mar- garet Mitchell, Dhruv Batra, Lawrence Zitnick, and Devi Parikh. 2015. VQA: Visual question an- swering. In Proceedings of the IEEF/CVF Interna- tional Conference on Computer Vision (ICCV), pages 2425- 2433.
Yang Cao, Yada Pruksachatkun, Kai-Wei Chang, Rahul Gupta, Varun Kumar, Jwala Dhamala, and Aram Gal- styan. 2022. On the intrinsic and extrinsic fairness evaluation metrics for contextualized language repre- sentations. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 2: Short Papers), pages 561-570. Dublin, Ireland, Association for Computational Linguistics.
Maria Antoniak and David Mimno. 2021. Bad seeds: Evaluating lexical methods for bias measurement. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 1ith International Joint Conference on Natural Lan- guage Processing (Volume 1: Long Papers), pages 189--1904, Online. Association for Computational Linguistics.
Zhenpeng Chen, Jie M. Zhang, Federica Sarro, and Mark Harman. 2023. A comprehensive empirical study of bias mitigation methods for machine learn- ing classifiers.
Camiel J. Beukehoom, Vü, and Faculteit der So- ciale Wetenschappen. 2014. Mechanisms of linguis- tie bias: How words reflect and maintain stereotypic expectancies.
Jaemin Cho. Abhay Zala, and Mohit Bansal. 2022. Dall- eval: Probing the reasoning skills and social biases of text-to-image generative transformers.
Myung Jin Choi, Antonio Torralba, and Alan S. Willsky. 2012. Context models and out-of-context objects. Pattern Recognition Letters, 33(7):853 862. Special Issue on Awards from ICPR 2010.
Shruti Bhargava and David Forsyth. 2019. Exposing and correcting the gender bias in image captioning datasets and models.

---

Kate Crawford. 2017. The trouble with bias. In Con- ference on Neural Information Processing Systems, invited speaker.
Linguisties and the 11th International Joint Confer- ence on Natural Language Processing (Volume 1: Long Papers), pages 1926-1940, Online. Association for Computational Linguistics.
Kimberle Crenshaw. 1989. Demarginalizing the inter- section of race and sex: A black feminist critique of antidiscrimination doctrine, feminist theory and antiracist politics. The University of Chicago Legal Forum, 140:139-167.
Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017. Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering. In Conference on Computer Vision and Pattern Recognition (CVPR).
Pieter Delobelle, Ewoenam Kwaku Tokpo, Toon Calders, and Bettina Berendt. 2021. Measuring fairness with biased rulers: A survey on quantify- ing biases in pretrained language models. CoRR,
Suchin Gururangan, Ana Marasovié, Swabha Swayamdipta, Kyle Lo, Beltagy, Doug Downey, and Noah A. Smith. 2020. Don't stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8342 8360, Online. Association for Computational Linguisties.
Sunipa Dev, Masoud Monajatipoor, Anaelia Ovalle, Ar- jun Subramonian, Jeff Phillips, and Kai-Wei Chang. 2021. Harms of gender exclusivity and challenges in non-binary representation in language technologies. In Proceedings of the 2021 Conference on Empiri- cal Methods in Natural Language Processing, pages 1968-1994, Online and Punta Cana, Dominican Re- public. Association for Computational Linguistics.
Melissa Hall, Laurens van der Maaten, Laura Gustafson, Maxwell Jones, and Aaron Adcock. 2022. A system- atic study of bias amplification.
Zexue He, Yu Wang. Julian McAuley, and Bod- hisattwa Prasad Majumder. 2022. Controlling bias exposure for fair interpretable predictions. In Find- ings of the Association for Computational Linguistics: EMNLP 2022. pages 1854-5866, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.
Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah Smith. 2020. Fine-tuning pretrained language models: Weight ini- tializations, data orders, and early stopping.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghami. Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlshy. 2021. An image is worth 16x 16 words: Transformers for image recognition at scale. In International Conference on Learning Representations.
Lisa Anne Hendricks, Kaylee Burns, Kate Saenko, Trevor Darrell, and Anna Rohrbach. 2018. Women also snowboard: Overcoming bias in captioning mod- els. In Proceedings of the European Conference on Computer Vision (ECCV).
Yusuke Hirota, Yuta Nakashima, and Noa Garcia. 2022a. Gender and racial bias in visual question answering datasets. In 2022 ACM Conference on Fairness, Ac- countability, and Transparency, FAceT '22, page 1280-1292, New York, NY, USA. Association for Computing Machinery.
Desmond Elliott, Stella Frank, Khalil Sima'an an, and Lu- cia Specia. 2016. Multi30K: Multilingual English- German image descriptions. In Proceedings of the 5th Workshop on Vision and Language, pages 70- 74, Berlin, Germany. Association for Computational Linguistics.
Yusuke Hirota, Yuta Nakashima, and Noa Garcia. 2022b. Quantifying societal bias amplification in image cap- tioning. In Proc. IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR). 10 pages.
Shangbin Feng, Chan Young Park, Yuhan Liu, and Yulia Tsvetkov. 2023. From pretraining data to language models downstream tasks: Tracking the trails of political biases leading to unfair NLP models. In Proceedings of the 61st Annual Meeting of the As- sociation for Computational Linguistics ( (Volume 1: Long Papers), pages 11 11737-11766 162., Toronto, Canada. Association for Computational Linguistics.
Drew A. Hudson and Christopher D. Manning, 2019. Gqa: A new dataset for real-world visual reasoning and compositional question answering. Confer- ence on Computer Vision and Pattern Recognition (CVPR), pages 6700-6709.
Ute Gabriel, Pascal Mark Gygax, and Elisabeth A. Kuhn. 2018. Neutralising linguistic sexism: Promising but cumbersome? Group Processes & Intergroup Rela- tions, 21:844 858.
Xisen Jin, Francesco Barbieri, Brendan Kennedy, Aida Mostafazadeh Davani, Leonardo Neves, and Xiang Ren. 2021. On transferability of bias mitigation ef- fects language model fine-tuning, In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 3770-3783, Online. Association for Computational Linguistics.
Seraphina Goldfarb- Tarrant, Rebecca Marchant, Ri- cardo Muñoz Sánchez, Mugdha Pandya, and Adam Lopez. 2021. Intrinsic bias metries do not correlate with application bias. Proceedings of the 59th An- nual Meeting of the Association for Computational

---

Masahiro Kaneko, Danushka Bollegala, and Naoaki Okazaki. 2022. Debiasing isn't t enough! - on the effectiveness of debiasing MLMs and their social biases in downstream tasks. In Proceedings of the 29th International Conference on Computational Lin- guisticy, pages 1299-1310, Gyeongju, Republic of Korea. International Committee Computational Linguistics.
Punta Cana, Dominican Republic. Association for Computational Linguistics.
Tom Michael Mitchell. 1980. The need for biases in learning generalizations. In Rutgers CS tech report CBM-TR-117.
Ron Mokady, Amir Hertz, and Amit H Bermano. 2021. ClipCap: CLIP prefix for image captioning. arXiv preprint arXiv:211.00933.
Os Keyes. 2018. The misgendering machines: Trans/hci implications of automatic gender recognition. Proc. ACM Computt Interact., 2(CSCW).
George D. Montañez, Jonathan Hayase, Julius Lauw, Dominique Macias, Akshay Trikha, and Julia Vendemiatti. 2019. The futility of bias-free learn- ing and search. In Al 2019 Advances in Artificial Intelligence, pages 277- Cham. Springer Inter- national Publishing.
Keita Kurita, Nidhi Vyas, Ayush Pareek, Alan W Black, and Yulia Tsvetkov. 2019. Measuring bias in con- textualized word representations. In Proceedings of the First Workshop on Gender Bias in Natural Lan- guage Processing, pages 166-172, Florence, Italy. Association for Computational Linguistics.
Hadas Orgad, Seraphina Soldfarb-Tarranny and Yonatan Belinkov. 2022. How gender debiasing affects in- ternal model representations, and why it matters. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies. pages 2602-2628, Seattle, United States. Association for Computational Linguistics.
Brian Larson. 2017. Gender as a variable in natural-
language processing: Ethical considerations. In Pro- ceedings of the First ACI. Workshop on Ethics in Natural Language Processing, pages 1-11, Valencia, Spain. Association for Computational Linguistics.
Seungho Lee, Minhyun Lec, Jongwuk Lee, and Hyun- jung Shim. 2021. Railroad is not train: Saliency as pseudo-pixel supervision for weakly supervised semantic segmentation. 2021 IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5491-5501.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Confer- ence on Empirical Methods in Natural Language Pro- cessing (EMNLP), pages 1532-1543, Doha, Qatar. Association for Computational Linguisties.
Jumnan Li. Dongxu Li. Silvio Savarese, and Steven Hoi. 2023. BLIP-2: Bootstrapping eagguage-mmag pre- training with frozen image encoders and large lan- guage models. In Proceedings of the 40th Interna- tional Conference on Machine Learning, Proceedings of Machine Learning Research. PMLR
Bryan Plummer, Liwei Wang, Chris M. Cervantes, Juan C. Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. 2015. Flickr30k entities: Collecting region-to-phree correspondences for richer image- to-sentence models. In 2015 IEEE International Conference Computer Vision (ICCV), pages 264 1- 2649.
Junnan Li, Dongxu Li. Caiming Xiong, and Steven C. H. Hoi. 2022. Blip: Bootstrapping language- image pre- training for unified vasion-language understanding and generation. In International Conference on Ma- chine Learning.
João Sedoc and Lyle Ungar. 2019. The role of protected class word lists in bias identification of contextual- ized word representations. In Proceedings of the First Workshop on Gender Bias in Natural Language Processing, pages 55 61. Florence, Italy. Association for Computational Linguistics.
Junnan Li, Ramprasaath R. Selvaraju, Akhilesh Deepak Gotmare, Shafiq Joty, Caiming Xiong, and Steven Hoi. 2021. Align before fuse: Vision and language representation learning with momentum distillation. In Advances in Neural Information Processing Sys-
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. 2018. Conceptual captions: A cleaned. hypemymed, image alt-text dataset for automatic im- age captioning. In Proceedings ACL.
tems.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays. Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. 2014. Microsoft coco: Common objects context. In Computer Vision
Aili Shen, Xudong Han, Trevor Cohn, Timothy Bald- win, and Lea Frermann. 2022. Does representational fairness imply empirical fairness? In Findings of the Association for C Computational Linguistics: AACL- IJCNLP 2022, pages 81 95, Online only. Association for Computational Linguistics.
ECCV 2014, pages 740-755, Cham. Springer Inter- national Publishing.
Fangyu Liu, Emanuele Bugliarello, Edoardo Maria Ponti, Siva Reddy, Nigel Collier, and Desmond EI- liott. 2021. Visually grounded reasoning across lan- guages and cultures. In Proceedings of the 2021 Conference on Empirical Methods Natural Lan- guage Processing, pages 10467-10485, Online and
Sunny Shrestha and Sanchari Das. 2022. Exploring gender biases in ml and ai academic research through systematic literature review. Frontiers in Artificial Intelligence, 5.

---

Krishna Kumar Singh, Dhruv Kumar Mahajan, Kristen Grauman, Yong Jae Lee. Matt Feiszli, and Deepti Ghadiyaram. 2020. Don't judge an object by its con- text: Learning to overcome contextual bias. 2020 IEEE/CVF Conference on Computer Vision and Pat- lern Recognition (CVPR), pages 1067-11075
you need. In Advances in Neural Information Pro- cessing Systems, volume 30. Curran Associates, Inc.
Angelina Wang, Alexander Liu, Ryan Zhang, Anat Kleiman, Leslie Kim, Dora Zhao, Iroha Shirai, Arvind Narayanan, and Olga Russakovsky, 2022. Revise: A tool for measuring and mitigating bias in visual datasets. Int. J. Comput. Vision, 130(7):1790-138)
Tejas Srinivasan and Yonatan Bisk. 2021. Worst of both worlds: Biases compound in pre-trained vision-and- language models.
Angelina Wang and Olga Russakovsky. 2021. Direc- tional bias amplification. In ICML.
Karolina Stanczak and Isabelle Augenstein. 2021. A survey on gender bias in natural language processing.
Tianlu Wang, Jieyu Zhao, Mark Yatskar, Kai-Wei Chang, and Vicente Ordonez. 2019. Balanced datasets are not enough: Estimating and mitigating gender bias in deep image representations. In ICCV.
Alane Subr. Stephanie Zhou, Ally Zhang, Iris Zhang. Huajun Bai, and Yoav Artzi. 2019. A corpus for reasoning about natural language grounded in pho- tographs. In Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguistics, pages 6418-6428, Florence, Italy. Association for Computational Linguistics.
Dora Zhao, Angelina Wang, and Olga Russakovsky. 2021. Understanding and evaluating racial biases in image captioning. In International Conference on Computer Vision (ICCV).
Tony Sun, Andrew Gaut, Shirlyn Tang, Yuxin Huang, Mai ElSherief. Jieyu Zhao, Diba Mirza, Elizabeth Belding, Kai-Wei Chang, and William Yang Wang. 2019. Mitigating gender bias in natural language processing: Literature review. In Proceedings of the 57th Annual Meeting of the Association for Computa- tional Linguistics, pages 1630-1640, Florence, Italy. Association for Computational Linguistics.
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Ryan Cotterell, Vicente Ordonez, and Kai-Wei Chang. 2019. Gender bias contextualized word embeddings. In Proceed- ings of the 2019 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, Volume (Long and Short Papers), pages 629-634, Min- neapolis, Minnesota. Association for Computational Linguistics.
Tony Sun, Kellie Wehster, Apu Shah, William Yang Wang. and Melvin Johnson. 2021. They, them, theirs: Rewriting with gender-neutral english.
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Or- donez, and Kai-Wei Chang. 2017. Men also like shopping: Reducing gender bias amplification using corpus-level constraints. In Proceedings of the 2017 Conference on Empirical Methods in Natural Lan- guage Processing, pages 2979-2989, Copenhagen, Denmark. Association for Computational Linguis- tics.
Hao Tan and Mohit Bansal. 2019. LXMERT: Learning mrosstmodality encoder representations from trans- formers. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-IJCNLP), pages 5100-5111, Hong Kong, China. Association for Com- putational Linguistics.
Kankan Zhou, Eason Lai, and Jing Jiang. 2022. VI.- StereoSet: A study of stereotypical bias in pre-trained vision-language models. In Proceedings of the 2nd Conference of the Asia-Pucific Chupter of the Asso- ciation for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing ( (Volume I: Long Papers), pages 527-538. Online only. Association for Computational Linguis- tics.
Ruixiang Tang. Mengnan Du, Yuening Li. Zirui Liu, Na Zou, and Xia Hu. 2021. Mitigating gender bias in captioning systems. In Proceedings of the Web Con- ference 2021, WWW '21, page 633-645, New York, NY, USA. Association for Computing Machinery.
Emiel van Miltenburg. 2016. Stereotyping and bias in the flickr30k dataset. In Proceedings of Multimodal Corpora, pages 1-4.
Xizhou Zhu, Han Hu, Stephen Lin, and Jifeng Dai. 2018. Deformable convnets v2: More deformable, better results. 2019 IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9300- 9308.
Eva Vanmassenhove, Chris Emmery, and Dimitar Shteri- onov. 2021. NeuTTaa Rewriter: A rule-basee based and neu- ral approach to automatic rewriting into gender neu- tral alternatives. In Proceedings of the 2021 Confer- ence on Empirical Methods in Natural Language Pro- cessing, pages 8940-8948, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, L ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all

---

A
Seed words
Male Female
Neutral
boy
girl
child
Gender terms
boyfriend boys girls
girlfriend
partner
Female: aunt, bride, businesswoman, daugh- ter, daughters, fiancce, fiancée, gal, gals, girl, girlfriend, girls, grandmother, her, her- self, lady, landlady, mama, mom, mother, queen, she, sister, sisters, spokeswoman, wife, woman, women, womens.
kids
brother sister
sibling
brothers
sisters
siblings
businessman dad
businesswoman
businessperson
mom
parent
dude
gal
person
dudes
gals
people
Male: boy, boyfriend, boys, brother, brothers, businessman, dad, dude, dudes, father, fiance, fiancé, gentleman, grandfather, groom, guy, he, him, himself, his, husband, king. landlord, man, men, mens, papa, son, sons, spokesman, uncle.
father
mother
parent
fiance
fiancee
partner
fiancé
fiancée
partner
gentleman
lady
person
grandfather groom guy he him himself herself themself
grandmother
grandparent
bride
newlywed
gal
person
Neutral: businessperson, child, childs, grand- parent, kid, kids, landlord, monarch, new- lywed, parent, partner, pbling, people, per- son, sibling, siblings, someone, spokesperson, spouse, their, them, themself, they.
she
they
her
them
his husband wife
her
their
spouse
Gender-neutral mappings Using the gender terms listed above, we generate mappings from male and female neutral terms: see Table 5 for details. These mappings are used to continue pre- training on gender- neutral (debiased) data as ex- plained in $5.2.
king landlord
queen
monarch
landlady woman person
landlord
man
men
women
someone
mens
womens
people
papa
mama
parent
daughter
kid
Objects List of top-100 most frequent nouns co- occurring with gender terms in the training split in COCO (Lin et al., 2014) and Conceptual Captions (CC3M) (Sharma et al., 2018).
sons
daughters
childs
spokesman uncle
spokeswoman
spokesperson
aunt
pbling
Table Gender-neutre mappings used for continual pre-training in gender-neutral data as described in $5.2.
COCO: tennis, group, street, baseball, table, dog, front, ball, player, field, snow, game, beach, horse, skateboard, umbrella, water, phone, kite, hand, top, board, ski, couple, motorcycle, food, elephant, People, picture. pizza, surfboard, room, shirt, bench, wave, frisbee, court, park, air, cake, bed, laptop. train, cell, racket, bat, bus, kitchen, plate, glass, ocean, side, grass, giraffe, building, city, skier, road, car, suit, trick, cat, tie, tree, bike, photo, boat, hat, slope. baby, area, sign, sidewalk, computer, hill, head, surfer, moun- tain, video, skateboarder, soccer, truck, ba- nana, couch, camera, skate, crowd, lot, snow- board, background, wine, bear, day, back, lug- gage, cow, living, fence, ramp.
beach, car, photo, dog, event, street, home. ball, wedding, family, city, film, time, tree, award, goal, hair, front, night, water, baby, business, illustration, politician, sport, show, way, portrait, face, book, premiere, fan, room, head, friend, year, athlete, park, house, fash- ion, soccer, character, flower, country, style, field, side, party, festival, picture, stage, rock, eye, couple, world, shirt, vector, camera, pop. tv, ceremony, hat, glass, snow, horse, school, road, phone, arm, art, window, crowd, sea, ta- ble, part, boat, suit, basketball, model, top. birthday, star, student, view, tennis, smile, wall, celebrity, baseball.
chair,
CC3M: player, team, actor, football, game, artist, hand, day, match, background, dress,

---

B Models
Precision
Recall
COCO
OC3M
COCO
CC3M
COCO
CC3M
this section, we provide an overview on the mod- els we use in our evaluation. We refer to their original work for more details.
1JOMERT LIMERTT LXWERT Tour
0.17
0.70
0.91
0.04
0.84
6.01
0.34
0:40
377
0.45
0.95
0.99
0.97
LXMERT (Tan and Bansal, 2019) is a cross- modal architecture pretrained to learn vision-and language representations. It consists of three Trans- former (Vaswani al., 2017) encoders, where vi- sual and language inputs are encoded separately in two independent stacks of Transformer layers be- fore feeding them into the cross-modality encoder. The cross modalityy encoder uses bi bi-directional cross attention to exchange information and align the entities across the two modalities. LXMERT is trained with four objectives: masked language mod- elling (MLM), masked object prediction, image- text matching (ITM) and image question answer- ing.
0.99
0.92
0.90
0.66
0.85
0.94
0.91
0.77
0.68
0.71
0.87
0.89
0.91
0.79
0.89
LIMEAAT
0.99
0.99
1
0.99
0.99
0.99
0.99
0.99
0.6 0.7
0.8
Figure 5: Statistical analysis of gender bias found through masked language modelling with gendered terms masked. Prediction of token from the gender- neutral set is always considered correct (Precision=1) Models report higher recall scores for Male (M) and Female (F) groups, showcasing the completeness of pos- itive predictions, whereas opposite for Neutral- related (N) tokens.
Similar to LXMERT, ALBEF (Li et al., 2021) is a dual-stream encoder (Bugliarello et al., 2021) that first learns separate visual and textual embeddings using transformerrbasee image and text encoders; and then fuses them in cross-moda Transformer using image-text contrastive loss (ITC), which en- ables more grounded vision and language repre- sentation learning. The model pretrained with two other objectives: masked language modelling (MLM) and image-text matching (ITM) on the multimodal encoder. Unlike LXMERT, ALBEF does not rely on image features extracted from an off- the-shelf object detector, but directly feeds the image into Vision Transformer (Dosovitskiy al., 2021) BLIP (Li et al., 2022) is versatile model based on a multimodal mixture of encoder-decoder net- work, that can be applied to a wide range down- stream tasks. The authors introduce a novel boost- rapping method to generate synthetic captions and remove noisy pairs from large-scale web data. Un- like LXMERT and ALBEF, BLIP is trained with autoregressive language modelling objective that allows the generation of coherent captions given an image. The model is also pretrained using the unimodal image-text contrastive loss (ITC) and the cross-modal image-text matching (ITM) loss used by ALBEF.
setup.
MLM experiment broken down by gender Ta- ble 6 provides a more granular look at which gender groups actually ampliffing/ddccrasinng the bias in the pretrained models.
D Bias & Fairness in Downstream Tasks
Extrinsic Bias The following graphs comple- ment results shown in 6.2 for bias amplification measured downstream tasks: Figure 6 shows results GQA; Figure shows results VQAv2; Figure 8 and Figure 9 show the bias revealed on image-text retrieval tasks when querying the mod- els with gender-neutral caption (or image), re- spectively.
Task performance & Fairness We present gran- ular results task performance in validation in Table 7 and group disparity, defined the min- max difference between group performance (A).
C Bias in Pretrained Models
Intrinsic bias Figure complements Figure from the main paper showing statistical results mea- sured on the intrinsic bias analysis in our control

---

0.00
BlasAmpa
-0.0
0.15
COCO
Male
Female
Neutral
-0.20
0.25
male
LXMERT 180K
-0295
-0048
-0733
fanal
LXMERT
-00004
-0000
-0014
0.30
180K
M
LXMERT 3M
-00577
-0020
-.1062
LXMERTaM
ALBEFaM
ALBEFFu
LXMERT 3M
-0014
+.0001
-0028
LXMERT
-0082
-0009
-0109
Figure 6: Bias amplification measured question- answering (GQA) broken down by gender group. M are gender-neutral pretrained on CC3M.
180K
ALBEF4M
1.000
-00517
-108
ALBEFN-COMM
-0748
-11993
11529
AM
ALBEIN 4M
CCC3MM
-0754
-0037
1073
ALBEFFIM
-0418
11146
-066
ALBEFN-COCO
-00559
-0669
0822
14M
ALBEFN.CCM
-0556
-0983
0083
14M
0.00
0.01
CC3M
0.02
Male
Female
Neutral
BluxAmpp
-0.03
LXMERT 180K
-00281
-0276
-0482
LXMERTY
+.0008
00081
-0113
-0.4
180K
LXMERT 3M
-0043
-0030
+.0055
-0.0
LXMERT
+.0002
+.0004
-0011
male
0.06
LXMERTN 180K
-0011
+0003
-0012
uauzal
00007
M
ALBEF4M 4M
-00433
-0069
-0422
ALBEFN-COM -OCO
AM
-0329
-0514
-0152
LXMERT 180K
ALBRRFox
ALBEF 1MM
1290
ALBEFN ALBEEN-CC3M C-C3M
4M
-0295
-0497
-0313
(a) M are gender-ncutral models pretrained on COCO.
ALBEF1AR
+.0159
-0042
-0062
ALBEFN-COCO
-0290
-0250
-0561
4M
ALBEFN-CCH 14M
-0535
-0641
-0534
0.00
Table 6: BiasAmpr-HA (BA.) per gender group, aver- aged over tasks (top-100 nouns) for LXMERT and AL- BEF models, evaluated on validation splits COCO (top) and CC3M (bottom). Light and dark backgrounds indicate bias amplification measured within in-domain and of--omainn data respectively. A model amplifies the bias in the dataset value is positive. negative value indicates overall decrease the bias in model's predictions.
00.01
-000
BluxAmpA -T
-0.03
-0.04
0.05
molr
0.06
fauae
0.07
M
M"
LXMERT>M
ALBEFAM
ALBEF
(b) M are gender-neutral models pretrained on COCO.
Figure 7: Bias amplification measured on question- answering (VQAv2) broken down by gender group.

---

VQAv2
GQA
NLVR2
F30K
Acc.
A(1)
Acc.
A(L)
Acc.
A())
r@ IR
A(1)
r@
TR
A())
M
77.5
83.8
81.9
55,8
61.6
LXMERT 180K
F
76.3
4.8
84,9
1.1
75.0
6.9
58.5
10.1
62.8
7.5
72.7
84.8
81.1
48.4
55.3
M
79.6
83.9
79.3
56.1
66.4
LXMERT 180K
78.1
1.5
85.4
1.5
74.2
6.0
58.1
8.8
67.6
8.4
79.3
85.3
80.2
49.3
59.2
68.4
63.7
72.3
56.0
63.0
LXMERT 3M
F 66.5
6.0
65.6
1.9
64.0
14.8
59.4
8.2
63.7
8.7
N
62.4
64.5
78.8
51.2
55.0
M 70.3
64.6
79.3
51.4
56.2
LXMERTN 3M
F 67.9
2.4
66.8
2.3
67.7
11.6
52.3
5.0
61.4
9.0
N
70.1
64.5
78.8
47.3
52.4
M
75.1
60.0
87.9
83.1
94.5
ALBEFAM
73.7
1.4
61.5
2.5
76.8
11.1
87.9
10.1
98.1
7.0
74.3
62.5
79.6
77.8
91.1
M
75.0
60.5
85.7
83.7
94.2
ALBEFN.COCO
73.6
60.7
1.6
75.8
9.9
87.2
10.1
96.6
5.2
N
74.0
62.1
79.8
77.1
91.4
75.0
61.0
84.6
82.2
94.8
ALBEFN-CCMM
F 73.7
1.3
60.7
2.0
74.8
9.8
87.1
9.2
97.6
7.7
N
74.5
62.7
79.3
77.9
89.9
M
76.0
59.7
86.8
87.5
95.9
ALBEF
75.0
1.0
59.6
2.8
79.8
7.0
89.1
7.3
100.0
6.0
75.6
62.4
81.8
94.0
M
76.0
60.6
60.4
86.6
95.4
ALBEEN- COCO
74.9
60.3
2.4
52.5
7.9
87.8
5.9
99.0
5.0
14M
75.5
62.7
57.6
81.9
94.0
M
75.8
60.7
87.9
86.6
95.9
ALBEFN-CCOM 14M
75.0
61.9
2.3
10.1
89.8
7.8
98.1
4.6
N
75.5
63.0
81.7
82.0
93.5
M
76.5
60.3
82.4
88.7
97.1
BLIP 129M
F
75.1
1.4
60.3
2.0
77.8
6.0
90.6
6.2
99.0
3.8
75.6
62.3
83.8
84.4
95.2
M
76.4
60.1
84.6
88.1
98.3
BLIPN
75.2
1.2
59.7
2.6
73.7
10.9
89.6
5.8
99.0
4.1
129M
75.8
62.3
80.9
83.8
94.9
Table 7: Validation results per group: male (M), female (F), and neutral (N). We report VQA-accuracy in VQAv2, accuracy GQA and NLVR2, recall in F30k and group disparity (A) across tasks, Lower better.

---

80
70
femsle
acatral
30-
sestral
Sensle
25-
50
40
15
30
10
20-
10
0
LXMERTISX ALBEFM
ALBEF
129M
LXMERTIaI ALBEFM
ALBEF 1403
BL
(a) IR MM are gender-neutral models pretrained on COCO.
(a) TR- MM are gender-neutral models pretrained on COCO.
70
Samuk
aratral
sestnl
60
25-
50
15
30
10
20
10
0-
LXMERTSM ALBEFM
ALBEF
LXMERT3M
ALB
(b) IR- R are gender-neutral models pretrained on CC3M.
(b) TR- M are gender-neutral models pretrained on CC3M.
Figure 8: Extrinsic bias measured on text-to-image retrieval (IR) on Flickr30K. Bias is measured as the percentage of images retrieved from each group when querying the models with gender-neutral caption.
Figure 9: Extrinsic bias measured image-to-text
trieval (TR) (c)-(d) on Flickr30K. Bias is measured as the percentage of captions retrieved from each group when querying the models with gender-neutral image.

---

E Variance in fine-tuning
Table shows the mean and standard deviation in bias amplification when fine-tuning LXMERT models with different random seeds. The variance is due to random initialization. In line with what we observed in $6.2, there is no clear trend when comparing model M with it's gender-neutral pre- training counterpart, MD
VQAv2
GQA
mean:::
mean
male
-0311=00057
-0019:#0071
LXMERT 180K
female
-0497=00053
-0477710088
60
+.0000+.0030
-0252:1.0889
58
male -0301=.0011
-0227:20006
LXMERT 180K
female -0538 .034
-0528.1.0106
56-
neutral +0007+00014
-02455+0042
male 00169 0054
-0254:1.135
54
LXMERT 3M
female -0667-0004
-0935++0110
52
neutral -0557
-0269:1.0099
LXMERT LXMERTII
LXMERT
LNMERT
LXMERT LNMERTA
LXMERTS
LAMHERTS
male 0064--0038
-0188++0060
LXMERT 3M
female .0634=00046 -0971100111
neutral
00383=0005
-01941.0109
(a) VQAv2
(b) GQA
Table 8: BiasAmpa-7 fine-tuning variance of
70-
TR
LXMERT models across question answering tasks. Each model is fine-tuned 6 times on each task. report average VQA-accuracy in VQAv2 and average accuracy in GQA, together with its standard deviation.
50
40
Figure 10 shows violin plots of the distribution results when fine-tuning LXMERT models with different random seeds. The variance is due to random initialization. Gender-neutral models re- veal lower standard deviation across tasks. This finding reveals of the benefits perform tra steps of pretraining on gender-neutral data: to reduce variance in downstream performance. This observation aligns with the NLP literature showing that biases in a model are independent from model performance (Cabello et al., 2023).
LXMERT not LNMERTM LXMERIN LAMERTY
LXMERT
LAMERTS sM LXMERTN
de0 LAMERTTS
NIVR2
(d) F30K
Figure 10: Fine-tuning variance LXMERT models across tasks. On the left with white background, origi- nal models (Mp). On the right with darker background, models after gender-neutral pretraining (M²). Each model fine-tuned 6 times on each task. The dots represent experimental ohservations. We report av- erage VQA-accuracy in VQAv2, accuracy in GQA and NLVR2, and recall in F30k.


Reviews:
[{"review_id": "emEyIGm3E1", "reviewer": "Reviewer_mmdc", "summary": "", "questions": "Would it be possible for the authors to clarify how the intrinsic bias reflected in V&L models is different from the intrinsic biases in pretrained language models? Since the calculation of the intrinsic biases for V&L models is based on its language modeling capabilities, I wonder how much of it is due to the intrinsic bias from the language encoder and how much of it is from the cross-modality encoder. Is there a way to measure this? I believe this kind of analysis is informative and will also help ground V&L bias research in relation to existing language model bias research and vision models bias research. ", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive review and valuable feedback. We address your questions below.\n\n> Minor weakness: In the abstract the authors used the term \"causal link\" - this is slightly confusing. The results show that the intrinsic and extrinsic biases are independent (not correlated). \"not causal\" is a subset of \"not correlated\". I believe changing the word in the abstract makes the communication more accurate.\n\nWe agree with you and we will rephrase this sentence to convey the finding that intrinsic and extrinsic biases are independent.\n\n> Would it be possible for the authors to clarify how the intrinsic bias reflected in V&L models is different from the intrinsic biases in pretrained language models? [...]\n\nThank you for raising this question, we will better clarify this methodology when describing the evaluation tasks (section 5.3). The intrinsic bias reflected by V&L models in our paper have been analyzed considering both domains, image and language. Following previous work [1,2,3,4,5] we also use the MLM task for evaluating intrinsic bias, and keep the image associated with each text (caption) as its input, in a setup that reflects the pretraining conditions (see L358–369 in the manuscript). \n\nEvaluating the linguistic skills of V&L models isn’t a trivial matter. We refer to ​​[7] for a detailed study on how to adapt pretrained V&L models to a text-only input. Early on in the project we performed an ablation study with LXMERT, where we zeroed-out the input image (we simply input a constant array of zeros) but kept the text component. We observed a higher degree of gender bias in the model compared to using both visual and text components.   \n\nTherefore, we can say that the prejudicial biases found in V&L are not only attributed to one domain, but they are compound. For example, [5] studied how within- and cross-modality gender biases are expressed using a set of template-based data, and demonstrated that V&L models tend to reinforce a stereotype over faithfully describing the visual scene. In the field of image generation, [6] demonstrates the presence of stereotypes in image generation models and discuss the challenges of the compounding nature of language-vision biases.\n\n> Several works in bias research have addressed the issue of intersectionality. [...]\n\nYes, thank you for pointing out the issue of intersectionality. We will acknowledge the fact that intersectionality was not studied in the limitations of the paper. \n\nReferences: \n\n[1] Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2017. Men also like shopping: Reducing gender bias amplification using corpus-level constraints.   \n[2] Lisa Anne Hendricks, Kaylee Burns, Kate Saenko, Trevor Darrell, and Anna Rohrbach. 2018. Women also snowboard: Overcoming bias in captioning models.   \n[3] Ruixiang Tang, Mengnan Du, Yuening Li, Zirui Liu, Na Zou, and Xia Hu. 2021. Mitigating gender bias in captioning systems.  \n[4] Tejas Srinivasan and Yonatan Bisk. 2021. Worst of both worlds: Biases compound in pre-trained vision-and- language models.   \n[5] Tianlu Wang, Jieyu Zhao, Mark Yatskar, Kai-Wei Chang, and Vicente Ordonez. 2019. Balanced datasets are not enough: Estimating and mitigating gender bias in deep image representations.   \n[6] Federico Bianchi, Pratyusha Kalluri, Esin Durmus, Faisal Ladhak, Myra Cheng, Debora Nozza, Tatsunori Hashimoto, Dan Jurafsky, James Zou, and Aylin Caliskan. 2023. Easily Accessible Text-to-Image Generation Amplifies Demographic Stereotypes at Large Scale.   \n[7] Lovisa Hagström and Richard Johansson. 2022. How to Adapt Pre-trained Vision-and-Language Models to a Text-only Input?.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The main strengths I find from the paper:\n\n- The paper follows theoretically valid bias research framework and comprehensively examines bias from pretraining to downstream tasks. \n- The paper asks and addresses interesting questions on top of examining the biases: are there correlations between the intrinsic and extrinsic bias amplification \n- The paper also shows a viable solution to existing biases - adding extra training step with gender-neutral data and this comes with little costs of performance \n- The experiments and well-designed and clearly communicated ", "reasons_to_reject": "I do not find significant weakness of the paper as reasons to reject. If the authors address the questions below that would make the paper stronger.\n\nMinor weakness:\nIn the abstract the authors used the term \"causal link\" - this is slightly confusing. The results show that the intrinsic and extrinsic biases are independent (not correlated). \"not causal\" is a subset of \"not correlated\". I believe changing the word in the abstract makes the communication more accurate.", "questions_to_authors": "Would it be possible for the authors to clarify how the intrinsic bias reflected in V&L models is different from the intrinsic biases in pretrained language models? Since the calculation of the intrinsic biases for V&L models is based on its language modeling capabilities, I wonder how much of it is due to the intrinsic bias from the language encoder and how much of it is from the cross-modality encoder. Is there a way to measure this? I believe this kind of analysis is informative and will also help ground V&L bias research in relation to existing language model bias research and vision models bias research. ", "paper_topic_and_main_contributions": "The paper examined bias amplification in the pretraining and fine-tuning phase of V&L models and the possible connection between two training phases. The paper analyzed from intrinsic bias (bias encoded after the pretraining phase) and the extrinsic bias and task performance (bias encoded after the fine-tuning phase). The paper measures bias amplification on the intrinsic bias of the models and all three of bias amplification, task performance, and fairness on the extrinsic bias of fine-tuned models. The authors did not find causal connection between the intrinsic and extrinsic bias amplification and showed that extra steps of pretraining with gender neutral reduces group disparities. \n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Several works in bias research have addressed the issue of intersectionality. This work only examines gender bias but the bias can exhibit in different ways when gender overlaps with other dimensions of identities. For instance, gender and race, gender and age, etc. The authors mentioned in the Limitation section that this framework can be extended to examine other dimensions of bias. It would make the paper stronger if the authors also address the limitation on not examining overlapping dimensions of identity. A few references on intersectionality in bias research:\n\nIntersectionality: \nCrenshaw, Kimberlé Williams. “Mapping the margins: intersectionality, identity politics, and violence against women of color.” Stanford Law Review 43 (1991): 1241-1299.\n\nIntersectionality in bias research in NLP:\nField, A., Blodgett, S. L., Waseem, Z., & Tsvetkov, Y. (2021). A survey of race, racism, and anti-racism in NLP. arXiv preprint arXiv:2106.11410.\n\nTan, Yi Chern and Elisa Celis. “Assessing Social and Intersectional Biases in Contextualized Word Representations.” ArXiv abs/1911.01485 (2019): n. Pag.\n\nDavidson, Thomas et al. “Racial Bias in Hate Speech and Abusive Language Detection Datasets.” ArXiv abs/1905.12516 (2019): n. Pag.\n\nLin, Inna Wanyin et al. “Gendered Mental Health Stigma in Masked Language Models.” Conference on Empirical Methods in Natural Language Processing (2022).\n"}}, {"review_id": "u2wVEpLqQ3", "reviewer": "Reviewer_HKVB", "summary": "", "questions": "1. It is exciting to see the finding that there is no evidence that the gender-neutral pertaining influenced positively (nor negatively) the extrinsic bias of the models. However, I am confused about how to summarize the results from Appendix E and Figure 4. Can you provide more illustrations or a clearer explanation?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive review and valuable feedback. We address your questions below.\n\n> [...] I am confused about how to summarize the results from Appendix E and Figure 4. Can you provide more illustrations or a clearer explanation?\n\nWhile it is true that “there is no evidence that the gender-neutral pertaining influenced positively (nor negatively) the extrinsic bias of the models”, it does not mean that the gender-neutral pretraining did not influence the extrinsic bias at all. In fact, as we can see in Figure 4 for GQA, there is a variation on the bias amplification measured on models pretrained on gender neutral data (M^N) compared to the original models (M), but this variance does not always go in the same direction: it is model and group dependant. In addition, the variance observed on M^N models also depends on the downstream task (see Appendix D). Therefore, we cannot establish a strong relationship between intrinsic and extrinsic biases (nor on the effect of gender-neutral pertaining) but conclude that they are independent. \n\nRegarding the results from Appendix E, we report variance of bias amplification (as recommended by [1]) in Table 8, where we can see that results are in line with what we explained earlier, i.e., there is no clear trend when comparing a model M with it’s gender-neutral pretraining counterpart, M^N. What’s more, the difference in the degree of bias amplification is minimal when taking into account the standard deviation. In addition, we also report fine-tuning variance in model performance across tasks (Figure 10). Here, we do observe a common trend: gender-neutral models reveal lower standard deviation across tasks. This finding indicates that it is beneficial to perform extra steps of pretraining on gender-neutral data to reduce variance in downstream performance. This observation aligns with the NLP literature showing that biases in a model are independent from model performance [2,3].\n\nWe will clarify these points further when we revise the paper.\n\nReferences:\n\n[1] Angelina Wang and Olga Russakovsky. 2021. Directional bias amplification.  \n[2] Aili Shen, Xudong Han, Trevor Cohn, Timothy Bald- 926 win, and Lea Frermann. 2022. Does representational fairness imply empirical fairness?  \n[3] Laura Cabello, Anna Katrine Jørgensen, and Anders Søgaard. 2023. On the independence of association bias and empirical fairness in language models.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper is the first to investigate whether the biases encoded in pretrained models can be transferred to downstream applications or not on V&L models.\n2. This paper leverages bias amplification broadly examined in language models to understand to what extent the pertaining data cause the bias in a V&L model.\n3. This paper further investigates the effect of continued pretraining V&L models on in-domain data where generated terms have been replaced by their gender-neutral counterparts.\n4. This paper provides a clear demonstration of understanding bias in V&L models: (1) using gender-context distribution to define intrinsic bias; (2) leveraging BiasAmp metric to decouple the direction of bias amplification; (3) understanding extrinsic bias and fairness by evaluating model performance concerning the three demographic groups and comparing results in search of the more equitable system; (4) applying gender-neutral domain adaptation to reduce intrinsic and extrinsic biases.\n5. This paper conducts comprehensive experiments on three models (LXMERT, ALBEF, and BLIP) under VOLTA framework among four evaluation tasks (MLM, VQA, NLVR, and image-text retrieval.) These experimental results strongly support the arguments and the detailed explanations help readers better understand.\n6. Overall, this paper presents a comprehensive analysis of gender bias amplification and fairness of encoder-only and encoder-decoder V&L models, showing a promising direction for the community to leverage the evaluation methods from language-only to the vision-and-language area.\n\n", "reasons_to_reject": "In my opinion, there are no significant reasons to reject this paper. The only thing I would worry about is the novelty. But it is reasonable to follow the wide-acknowledged methods in the community to explore the new area.", "questions_to_authors": "1. It is exciting to see the finding that there is no evidence that the gender-neutral pertaining influenced positively (nor negatively) the extrinsic bias of the models. However, I am confused about how to summarize the results from Appendix E and Figure 4. Can you provide more illustrations or a clearer explanation?", "paper_topic_and_main_contributions": "This paper investigates bias amplification in pre-trained and fine-tuned vision-and-language models across three families. The study examines how intrinsic bias in pre-trained models relates to extrinsic bias in downstream tasks. Results suggest that a biased pre-trained model may not always result in biased performance in downstream tasks to the same extent. The study also shows that additional pre-training on gender-neutral data can reduce group disparities without significantly impacting task performance.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Sl3y40elHi", "reviewer": "Reviewer_xmLY", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review and valuable feedback. We address your concerns below.\n\n> The paper's novelty is not convincing, as it neither introduces new bias metrics nor proposes novel bias-mitigation methods\n\nWe acknowledge that our paper does not introduce a new bias metric nor a novel bias-mitigation method. Instead, the novelty lies in leveraging the use of common bias and fairness metrics in language models to understand the interaction between the two, bias and fairness, in V&L models. This is understudied in the era of large-scale pretrained multimodal models and it hardly hurts the scientific discourse to apply methodologies to different domains. In addition, we complement this analysis with a comprehensive study on the effects of continued pretraining on in-domain debiased data (gender-neutral data) which “shows a viable solution to existing biases”, as mentioned by Reviewer mmdc. Finally, Reviewer HKVB also highlights that we are “the first to investigate whether biases encoded in pretrained models can be transferred to downstream applications or not on V&L models”.\n\n> Several important notations lack clarity\n\nThank you for pointing out unclear notations used in the paper; we hope that the following explanations will improve the clarity of our writing in the final copy.\n\nWe will include further explanation regarding g_i (section 4.1) to clarify the definition given in lines 218–220. More concretely, g_i denotes the presence of one of the 3 demographic groups considered in the analysis (male, female, neutral). To make this clearer, we could rename g_i to ease the connection with the set of demographic groups defined in L214.\n\nRegarding the meaning of T in the second paragraph of section 4.2, it refers to the same T defined earlier in section 4.1; similarly, the specific tasks (or classes) that we refer to are the same defined in section 4.1. The specific words used for these tasks are listed in Appendix A. We will add a footnote to better reflect this association.\n\n> The paper lacks discussion regarding the reasons behind the observed benefits of \"extra pretraining steps on gender-neutral data for reducing group disparity in all model architectures tested on VQAv2, and in the majority of models for both retrieval tasks.\"\n\nWe agree the paper would benefit from a more extensive discussion of our findings. The effects of gender-neutral pretraining on the models’ group disparity are shown in Figure 3 in the main paper, and ​​Table 7 in Appendix 7. These results are discussed in L518–534. It is true that the current version of the paper does not elaborate much on the specific reason behind the benefits observed in terms of disparity reduction. Our intuition is that this could be due to shifting the final word representations to a more equidistant vector space between gendered terms and their context: the conditional probability distribution of a gendered term given its context is smoother across different demographic groups. This could make the pretrained model more robust before being fine-tuned. Our intuition goes in line with studies in language models such as [1]. In [1], authors continue pretraining LM on partisan data to study the political leanings of the models: they observed that LMs do acquire (political) bias from pretraining corpora, yet “left-leaning corpora generally resulted in a left/liberal shift [...]”. \n\nReferences:\n\n[1] Shangbin Feng, Chan Young Park, Yuhan Liu, Yulia Tsvetkov. 2023. From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Models.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper introduces and clearly defines two types of gender biases: intrinsic bias and extrinsic bias.\n2. The case study evaluates three vision and language models, providing valuable insights.\n3. The limitation section is well-written, effectively highlighting most of the limitations of this paper.", "reasons_to_reject": "1. The paper's novelty is not convincing, as it neither introduces new bias metrics nor proposes novel bias-mitigation methods.\n2. Several important notations lack clarity. For instance, the definition of g_i in the second paragraph of 4.1 Intrinsic Bias needs further explanation. Additionally, the meaning of T in the second paragraph of 4.2 Bias Amplification is not well-defined. It is also unclear what specific tasks or classes are being referred to.\n3. The paper lacks discussion regarding the reasons behind the observed benefits of \"extra pretraining steps on gender-neutral data for reducing group disparity in all model architectures tested on VQAv2, and in the majority of models for both retrieval tasks.\"", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper assesses gender bias and fairness in pretrained vision & language models. The findings suggest that there is no causal link between bias amplification during pretraining and after fine-tuning. Furthermore, the study investigates whether pretrained models can mitigate group disparities when fine-tuned on gender-neutral datasets.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "mSmeLTSSEH", "reviewer": "Reviewer_cgUh", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and valuable feedback. We address your concerns below.\n\n> Similar research has been done for vision or language models separately in previous work, making the findings less impressive. [...] It is not clear enough why we expect that a V&L model should differ from a V or L model in terms of bias and fairness, i.e., the motivation of this paper. [...]\n\nWhile it is true that similar research has been done independently in vision or language models, little work has addressed the presence of social biases and/or fairness in V&L models. Prejudicial biases found in V&L are not only attributed to one domain, but they are compound and need to be studied jointly. For example, [1] studied how within- and cross-modality gender biases are expressed using a set of template-based data, and demonstrated that V&L models tend to reinforce a stereotype over faithfully describing the visual scene. In the field of image generation, [2] demonstrates the presence of stereotypes in image generation models and discuss the challenges of the compounding nature of language-vision biases.\n\nReferences:\n\n[1] Tianlu Wang, Jieyu Zhao, Mark Yatskar, Kai-Wei Chang, and Vicente Ordonez. 2019. Balanced datasets are not enough: Estimating and mitigating gender bias in deep image representations.   \n[2] Federico Bianchi, Pratyusha Kalluri, Esin Durmus, Faisal Ladhak, Myra Cheng, Debora Nozza, Tatsunori Hashimoto, Dan Jurafsky, James Zou, and Aylin Caliskan. 2023. Easily Accessible Text-to-Image Generation Amplifies Demographic Stereotypes at Large Scale.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is well-written and easy to follow. \n- The paper provides a comprehensive literature review of related work.\n- Experimental settings are clearly introduced.", "reasons_to_reject": "- Similar research has been done for vision or language models separately in previous work, making the findings less impressive. For example, as has already been cited in the paper, Wang et al. (2019) investigated bias in vision, Zhao et al. (2019) evaluated bias in language, and Shen et al. (2022) investigated the relationship between intrinsic bias and extrinsic bias. It is not clear enough why we expect that a V&L model should differ from a V or L model in terms of bias and fairness, i.e., the motivation of this paper. As for the findings in this paper, such as 'gender-neutral pretraining mitigates gendered outputs', 'pretrained models reflect training data biases', and 'trade-offs in task performance' are consistent as findings for V or L models in the previous, which are not surprising.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper extensively evaluates both the intrinsic bias and extrinsic bias in V&L models and examines the effectiveness of fine-tuning over gender-neutral data. Experiments show that V&L models suffer from bias, and there is no clear causal link between intrinsic bias and extrinsic bias. Moreover, gender-neutral fine-tuning could improve fairness.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper analyzes different types (intrinsic and extrinsic) of gender bias amplification correlating to different stages (pre-training and fine-tuning) of training vision and language models. The paper also proposes a method for addressing bias amplification through an additional pre-training step. Reviewers suggest including a limitation in the work that it focuses on a single dimension of social bias (gender), and doesn't consider how different dimensions of identity may intersect with one another in nuanced ways (e.g., race and gender). Other suggestions that I encourage the authors to make include clarifying some of the notation and expanding some of the discussion on related work.
Decision: Accept-Main