SlowGuess/NAACL_2021_OCR huggingface dataset:
Overall, the results are consistent with prior work on biases in both humans and with ungrounded models such as BERT. Following Tan and Celis (2019), each experiment examines the bias in three types of embeddings: word embeddings, sentence embeddings, and contextualized word embeddings. While there is broad agreement between these different ways of using embeddings, they are not identical in terms of which biases are discovered. It is unclear which of these methods is more sensitive, and which finds biases that are more consequential in predicting the results of a larger system constructed from these models. Methods to mitigate biases will hopefully address all three embedding types and all of the three questions we restate below.
Do joint embeddings encode social biases? See Experiment 1, section 4.1. The results presented in table 3 and table 6 clearly indicate that the answer is yes. Numerous biases are uncovered with results that are broadly compatible with May et al. (2019) and Tan and Celis (2019). It appears that more pronounced social biases exist in grounded compared to ungrounded embeddings.
Can grounded evidence that counters a stereotype alleviate biases? See Experiment 2, section 4.2. The results presented in table 4 and table 6 indicate that the answer is no. Biases are somewhat attenuated when models are shown evidence against them, but overall, preconceptions about biases tend to overrule direct visual evidence to the contrary. This is worrisome for the applications of
VisualBERT [CLS] TOK0 ... TOK_CONTEXTUAL ... TOKN [SEP] [IMG] IMG0 ... IMGN
VL-BERT [CLS] TOK0 ... TOK_CONTEXTUAL ... TOKN [SEP] IMG0 IMG1 ... IMGN [END]
ViLBERT [CLS] TOK0 ... TOK_CONTEXTUAL ... TOKN [SEP]
LXMert [CLS] TOK0 ... TOK_CONTEXTUAL ... TOKN [SEP] [CROSS_MODEL]
2Some pretraining images for VL-BERT are from the Visual Genome.
Gender	Level	VJamBERT	LVMER	VJHERT	Race	Level	VJamBERT	LVMER
Google	Google	Google	Google	Google	Google	Google	Google	Google	Google
W	0.57	1.04	0.55	1.61	C3: EA/AA, Pleasant	S	0.23	0.31	-0.16	1.37
C6: M/W, Career/Fam	S	-0.18	0.98	0.69	-0.02	C	0.31	0.25	0.19	0.93	0.93
C	-0.61	0.76	0.17	0.46	C	-0.01	-0.29	0.44	0.68	0.68
C8: Science/Arts, M/W	W	0.77	0.59	0.43	-0.29	C12: EA/AA, Career/Family	S	-0.29	0.04	-0.04	-1.45
S	0.62	0.26	-	0.19	C	-0.54	0.05	-0.32	-0.96	0.08
C	0.30	-0.32	0.13	0.26	C	0.36	0.92	0.88	0.08	0.08
C11: M/W, Pleasant	W	-0.66	-0.91	-0.08	-1.20	C13: EA/AA, Science/Arts	S	0.04	0.61	0.58	-1.44
S	-0.74	-1.08	-0.20	0.01	C	0.12	0.35	0.16	0.98	0.98
C	0.42	-0.62	0.25	-0.18	Double Bind: EA/AA, Competent	S	0.58	1.09	0.92	0.90
Competent: M/W, Competent	W	-0.23	-0.57	-1.18	-1.28	C	0.75	1.28	0.98	1.44	1.44
S	-0.28	-0.29	-0.55	-1.35	C	1	1.14	1.30	1.48	1.48
C	-0.67	0.20	-0.48	0.31	C	1.10	1.19	1.46	1.54	1.54
Likeable: M/W, Likeable	W	-1.24	-1.26	-1.10	-0.91	Double Bind: EA/AA, Likeable	S	-0.25	0.41	0.93	0.87
S	0.10	-0.12	0.60	-0.03	C	-0.09	0.73	-0.04	1.01	1.01
C	-0.42	1.25	-0.83	-0.19	Occupation: EA/AA, Occupation	S	-0.15	-0.41	-0.71	1.38
Occupation: M/W, Occupation	W	0.02	0.86	1.56	1	C	-0.26	-0.26	-0.40	-0.06	0.12
S	0.77	0.95	1.32	-0	C	-0.70	-0.37	-1.11	0.12	0.12
C	0.98	1.53	0.52	0.11	Angry Black Woman Stereotype	S	-0.07	0.41	-1.31	1.59
C	-0.50	0.46	-0.12	-0.48
C	0.71	0.66	1.27	-0.13
Table 3: The results for all bias classes on Experiment 1 using Google Images that asks Do joint embeddings encode social biases? Numbers represent effect sizes and  $p$ -values for the permutation test described in section 4. They are highlighted in blue when  $p$ -values are below 0.05. Each bias type and model are tested three times against (W) word embeddings, (S) sentence embeddings, and (C) contextualized word embeddings. The answer to the question clearly appears to be yes. All models are biased. Note that out of domain, biases appear to be amplified.
Gender	Level	Vocabulary Google	Vocabulary Google	L-Mer Google	L-Mer Google	Race	Level	Vocabulary Google	Vocabulary Google	L-Mer Google	L-Mer Google
W	1.05	1.09	-0.20	1.97	C3: EA/AA, Pleasant	W	1.55	1.03	0.60	1.34
C6: M/W, Career/Fam	S	-0.57	1.34	0.78	1.57	S	1.54	0.85	0.84	-0.08
C	-0.86	0.65	0.21	0.44	C	0.26	-0.14	0.58	0.76
C8: Science/Arts, M/W	W	0.77	0.59	0.43	-0.29	W	-0.04	0.88	0.93	-1.49
S	0.62	0.26	-	0.19	S	0.36	0.81	0.33	-1.27
C	0.30	-0.32	0.13	0.26	C	0.84	1.02	0.98	0.18
C11: M/W, Pleasant	W	-1.48	-1.33	-0.13	-0.77	W	-1.74	1.27	-0.38	-1.51
S	-1.13	-1.17	-0.55	-0.21	S	-0.08	1.04	-0.13	0.95
C	-0.15	-0.46	0.38	-0.17	C	1	1.39	0.97	0.96
Competent: M/W, Competent	W	0.23	0.23	-1.37	1.50	W	1.13	1.56	1.06	1.41
S	-0.12	-0.35	-0.98	-1.14	S	1.25	1.45	1.25	1.45
C	-0.60	-0.08	-1.11	0.44	C	1.11	1.20	1.46	1.57
Likeable: M/W, Likeable	W	-1.31	-0.61	-0.93	-1.98	W	0.29	1.13	1.29	0.90
S	1.76	-0.16	-0.81	1.99	S	0.42	1.04	0.43	1.29
C	-0.11	1.31	-1	-0.12	C	0.93	1.12	1.40	0.06
Occupation: M/W, Occupation	W	-0.77	0.05	1.33	-1.74	W	-0.04	-0.48	-0.33	-1.40
S	0.33	0.22	0.58	-0.20	S	0.15	-0.18	0.22	-0.03
C	0.90	1.46	0.34	0.16	C	-0.57	-0.19	-1.10	0.10
W	0.34	-0.28	-0.27	1.67
S	0.49	-0.53	0.31	0.03
C	1.71	1.44	1.34	-0.21
Table 4: The results for all bias classes on Experiment 2 using Google Images that asks Can joint embeddings be shown grounded evidence that a bias does not apply? Numbers represent effect sizes and  $p$ -values for the permutation test described in section 4. They are highlighted in blue when  $p$ -values are below 0.05. Each bias type and model are tested three times against (W) word embeddings, (S) sentence embeddings, and (C) contextualized word embeddings. The answer to the question appears to be no, although fewer tests are statistically significant compared to table 3 showing that visual evidence is helpful.
such models. In particular, using such models to search or filter data in the service of creating new datasets may well introduce new biases.
To what degree are encoded biases in joint embeddings from language or vision? See Experiment 3, section 4.3. The results for the second variant of Experiment 3 which is performed by masking the input text or image are presented in table 5 and table 6 are generally significant, more
so for language than vision. We report results for the sentence-level encoding and observed similar results for the word-level encoding. We did not measure contextual encodings as they would include the encoding for the [MASK] token. This indicates that biases arise from both modalities, but this does differ by model architecture. For VLBERT language appears to dominate. The results for the first variant of Experiment 3 congruent with
Gender	Mask	VishNBERT	LXMent
Google	Google	Google	Google
C6	T	0.14	1	1.18	-0
I	-	0.87	0.69	-0.03
C8	T	0.46	0.41	0.11	0.27
I	-	0.39	0.04	0.18
C11	T	-0.47	-1.21	-1.33	0.03
I	-	-1.11	-0.22	0.02
Competent	T	-0.06	-0.40	-0.21	-1.99
I	-	-0.35	-0.55	-1.05
Likeable	T	-0.07	-0.18	0.28	-1.99
I	-	-0.11	0.72	0.64
Occupation	T	0.05	1.08	0.92	-0.17
I	-	0.91	1.32	0
Race	Mask	VisualBERT	L-5Mert
Google	Google	Google	Google
C3	T	0.33	0.34	0.33	-0.01
I	-	0.31	0.21	0.95
C12	T	-0.52	0.05	-0.39	0
I	-	0.08	-0.36	-1.06
C13	T	-0	0.33	-0.10	-0
I	-	0.33	0.17	0.95
Competent	T	-0.44	1.10	1.33	-1.99
I	-	1.15	1.29	1.45
Likeable	T	-0.68	0.58	0.11	-1.99
I	-	0.73	-0.14	1.06
Occupation	T	-0.27	-0.24	-0.65	-0.17
I	-	-0.30	-0.38	-0.25
ABW	T	0.76	0.54	-0.01	-0.42
I	-	0.43	-0.13	-0.08
Table 5: The results for all bias classes on Experiment 3, using the second masking variant of the experiment, with Google Images asking the question To what degree are biases encoded in grounded word embeddings from language or vision? Numbers represent effect sizes and  $p$ -values for the permutation test described in section 4. All numbers were measured over sentence-level encodings. They are highlighted in blue when  $p$ -values are below 0.05. Biases are measured for masked tokens (T) and masked image regions (I). This answer appears to be that both vision and language play a significant role, but this differs across model architectures.
Gender	Level	Experiment 1	Experiment 2	Experiment 3
VisualBERT-COCO	VisualBERT-Consp	L-Mer-COCO	VisualBERT-Consp	VisualBERT-COCO	VisualBERT-Consp	L-Mer-COCO	VisualBERT-Consp	Mask	VisualBERT-COCO	VisualBERT-Consp	L-Mer-COCO	VisualBERT-Consp
C6	W	0.13	0.94	0.92	-0.14	0.15	0.95	0.61	1.98	T	-	1.15	0.01	0
S	0.28	1.11	1.32	0	0.41	0.83	1.16	-1.17	I	-	1.09	1.32	-0
C	-0.20	0.80	1.53	0.61	-0.99	0.58	1.46	0
Occupation	W	-0.07	0.75	0.39	-0.31	-0.64	-0.52	-0.66	1.99	T	-	0.74	-0.07	0
S	-0.23	0.73	-0.18	-0.01	0.09	-0.30	-1.14	0.69	I	-	0.71	-0.17	-0
C	-0.32	0.58	-0.14	0.01	-0.35	1.96	-0.70	0.90
Table 6: The results for two classes of bias on all three experiments using COCO and Conceptual Captions. Images for other bias classes could not be found in these datasets. These results are generally consistent with results on the Google Images dataset.
Number of statistically significant tests out of 6 total gender bias tests
Level	Experiment 1	Experiment 2	Experiment 3
VisualBERT Google	VLBERT Google	LxMert Google	VLBERT Google	VisualBERT Google	VLBERT Google	LxMert Google	VLBERT Google	Mask	VisualBERT Google	VLBERT Google	LxMert Google	VLBERT Google
W	2	2	3	3	2	2	2	2	T	-	1	3	4
S	2	1	3	3	3	3	2	2	I	-	2	3	3
C	3	3	3	4	2	3	3	4
Number of statistically significant tests out of 7 total race bias tests
Level	Experiment 1	Experiment 2	Experiment 3
VisualBERT Google	VLBERT Google	LxMert Google	VLBERT Google	VisualBERT Google	VLBERT Google	LxMert Google	VLBERT Google	Mask	VisualBERT Google	VLBERT Google	LxMert Google	VLBERT Google
Table 7: A summary of all previous results on the new image dataset derived from Google searches showing the number of significant bias test partitioned by the type of test. There are a total of 6 gender bias tests and 7 race bias test. Experiments 1 and 2 show no strong differences between models while in Experiment 3 ViLBERT stands out.
these results, with, large effect sizes ( $s = 0.42$  for ViLBERT and  $s = 0.467$  for VisualBERT with  $12\%$  of tests being statistically significant) demonstrating that language contributes more than vision. It could be that the biases in language are so powerful
that vision does not contribute to them given that in any one example it appears unable to override the existing biases (experiment 2). It is encouraging that models do consider vision, but the differing biases in vision and text do not appear to help.
Visually grounded embeddings have biases similar to ungrounded embeddings and vision does not appear to help eliminate them. At test time, vision has difficulty overcoming biases, even when presented counter-stereotypical evidence. This is worrisome for deployed systems that use such embeddings, as it indicates that they ignore visual evidence that a bias does not hold for a particular interaction. Overall, language and vision each contribute to encoded bias, yet the means of using vision to mitigate is not immediately clear. We enumerated the combinations of inputs possible in the grounded setting and selected three interpretable questions that we answered above. Other questions could potentially be asked using the dataset we developed, although we did not find any others that were intuitive or non-redundant.
While we discuss joint vision and language embeddings, the methods introduced here apply to any grounded embeddings, such as joint audio and language embeddings (Kiela and Clark, 2015; Torabi et al., 2016). Measuring bias in such data would require collecting a new dataset, but could use our metrics, Grounded-WEAT and Grounded-SEAT, to answer the same three questions.
Many joint models are transferred to a new dataset without fine-tuning. We demonstrate that going out-of-domain into a new dataset amplifies biases. This need not be so: out-of-domain models have worse performance which might result in fewer biases. We did not test task-specific fine-tuned models, but intend to do so in the future.
Humans clearly have biases, not just machines. Although, initial evidence indicates that when faced with examples that go against prejudices, i.e., counter-stereotypepinq, there is a significant reduction in human biases (Peck et al., 2013; Columb and Plant, 2016). Straightforward applications of this idea are far from trivial, as Wang et al. (2019) show that merely balancing a dataset by a certain attribute is not enough to eliminate bias. Perhaps artificially manipulating visual datasets can debias shared embeddings. We hope that these datasets and metrics will lead to understanding human biases in grounded settings as well as the development of new methods to debias representations.
This work was supported by the Center for Brains, Minds and Machines, NSF STC award 1231216,
the Toyota Research Institute, the MIT CSAIL Systems that Learn Initiative, the NSF Graduate Research Fellowship, the DARPA GAILA program, the United States Air Force Research Laboratory under Cooperative Agreement Number FA8750-19-2-1000, and the Office of Naval Research under Award Number N00014-20-1-2589 and Award Number N00014-20-1-2643. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.
We would like to urge subsequent work to avoid a common ethical problem we have noticed while reviewing the literature on bias in NLP. Much prior work refers to gender as "male" and "female", thereby conflating gender and sex. Recent work in psychology has disentangled these two concepts, and conflating them both blinds us to a type of bias while actively causing harm.
Our approach studies societal biases in models. These biases are inherently unjust, predisposing models toward judging people by skin color, age, etc. They are also practically damaging; they can result in real-world consequences. As part of large systems these biases may not be apparent as the source of discrimination, and it may not even be apparent that systems are treating individuals differently. People may even acclimatize to being treated differently or may interpret a machine discriminating based on race or gender as an inevitable but fair consequence of using a particular algorithm. We vehemently disagree. All systems and algorithm choices are made by humans, all data is curated by humans, and ultimately humans decide what to do with and when to use models. All unequal outcomes are a deliberate choice; engineers should not be able to hide behind the excuse of a black-box or a complex algorithm. We believe that by revealing biases, by providing tests for biases that are as focused as possible on the smallest units of systems, we can both assist the development of better models and allow the auditing of models to ascertain their fairness.
Data was collected in an ethical manner approved by the institution IRB board. No crowd-
sourced workers were employed. Instead we used a top  $k$  keyword search on Google Images. Because we collected images from the web, there is no straightforward way to use self-identified characteristics for gender and race. We expect biases and preconceived notions of identity to have some bearing on label accuracy. The dataset includes images available for free on the web and simple captions, e.g., Here is a man.
The biases we evaluate in this paper are based on various theories and works in psychology, such as the trope of the angry Black woman. Of course, that literature itself is limited; there are many biases which affect billions of people but do not appear in any available test, e.g., for almost any ethnic group there are those who will believe they do not work hard, but there are virtually no ethnic-group-specific tests. There are also likely biases which we have not yet articulated. Unfortunately, at present there is no coherent theory of biases to generate an exhaustive list and test them.
Marianne Bertrand and Sendhil Mullainathan. 2004. Are emily and greg more employable than lakisha and jamal? a field experiment on labor market discrimination. American Economic Review, 94(4):991-1013.
Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hann Wallach. 2020. Language (technology) is power: A critical survey of "bias" in nlp. In Proceedings of ACL.
Aylin Caliskan, Joanna J Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from language corpora contain human-like biases. Science, 356(6334):183-186.
Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollar, and C Lawrence Zitnick. 2015. Microsoft coco captions: Data collection and evaluation server. arXiv:1504.00325.
Patricia Hill Collins. 2004. Black sexual politics: African Americans, gender, and the new racism. Routledge.
Corey Columb and E Ashby Plant. 2016. The obama effect six years later: The effect of exposure to obama on implicit anti-black evaluative bias and implicit racial stereotyping. Social Cognition, 34(6):523-543.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv:1810.04805.
Aleksandr Drozd, Anna Gladkova, and Satoshi Matsuoka. 2016. Word embeddings, analogies, and machine learning: Beyond king-man+ woman= queen. In Proceedings of coling 2016, the 26th international conference on computational linguistics: Technical papers, pages 3519-3530.
Anthony G Greenwald, Debbie E McGhee, and Jordan LK Schwartz. 1998. Measuring individual differences in implicit cognition: the implicit association test. Journal of personality and social psychology, 74(6):1464.
Ronghang Hu, Amanpreet Singh, Trevor Darrell, and Marcus Rohrbach. 2019. Iterative answer prediction with pointer-augmented multimodal transformers for textvqa. arXiv:1911.06258.
Douwe Kiela and Stephen Clark. 2015. Multi-and cross-modal semantics beyond vision: Grounding in auditory perception. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2461-2470.
Lianian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. 2019. VisualBERT: A simple and performant baseline for vision and language. arXiv:1908.03557.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. ViLBERT: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In Advances in Neural Information Processing Systems, pages 13-23.
Chandler May, Alex Wang, Shikha Bordia, Samuel R Bowman, and Rachel Rudinger. 2019. On measuring social biases in sentence encoders. arXiv:1903.10561.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111-3119.
Vishvak Murahari, Dhruv Batra, Devi Parikh, and Abhishek Das. 2019. Large-scale pretraining for visual dialog: A simple state-of-the-art baseline. arXiv:1912.02379.
Tabitha C Peck, Sofia Seinfeld, Salvatore M Aglioti, and Mel Slater. 2013. Putting yourself in the skin of a black avatar reduces implicit racial bias. Consciousness and cognition, 22(3):779-787.
Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. arXiv:1802.05365.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. OpenAI preprint.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. 2018. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of ACL.
Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. 2019. Vl-bert: Pre-training of generic visual-linguistic representations. arXiv preprint arXiv:1908.08530.
Hao Tan and Mohit Bansal. 2019. Lxmert: Learning cross-modality encoder representations from transformers. arXiv preprint arXiv:1908.07490.
Yi Chern Tan and L Elisa Celis. 2019. Assessing social and intersectional biases in contextualized word representations. In Advances in Neural Information Processing Systems, pages 13209-13220.
Atousa Torabi, Niket Tandon, and Leonid Sigal. 2016. Learning language-visual embedding for movie understanding with natural-language. arXiv:1609.08124.
Tianlu Wang, Jieyu Zhao, Mark Yatskar, Kai-Wei Chang, and Vicente Ordonez. 2019. Balanced datasets are not enough: Estimating and mitigating gender bias in deep image representations. In Proceedings of the IEEE International Conference on Computer Vision, pages 5310-5319.
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Ryan Coterell, Vicente Ordonez, and Kai-Wei Chang. 2019. Gender bias in contextualized word embeddings. arXiv:1904.03310.
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2018. Gender bias in coreference resolution: Evaluation and debiasing methods. arXiv:1804.06876.