SlowGuess/ACL_2020_OCR huggingface dataset:
models. In European Conference on Computer Vision, pages 793-811. Springer.
Joseph Henrich, Steven J Heine, and Ara Norenzayan. 2010. The weirdest people in the world? Behavioral and brain sciences, 33(2-3):61-83.
Miguel A Hernán, Sonia Hernández-Díaz, and James M Robins. 2004. A structural approach to selection bias. Epidemiology, pages 615-625.
Yasmeen Hitti, Eunbee Jang, Ines Moreno, and Car- olyne Pelletier. 2019. Proposed taxonomy for gender bias in text; a filtering methodology for the gender generalization subtype. In Proceedings of the First Workshop on Gender Bias in Natural Language Processing, pages 8-17, Florence, Italy. Association for Computational Linguistics.
Dirk Hovy. 2015. Demographic factors improve classification performance. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), volume 1, pages 752-762.
Dirk Hovy. 2018. The social and the neural network: How to make natural language processing about people again. In Proceedings of the Second Workshop on Computational Modeling of People's Opinions, Personality, and Emotions in Social Media, pages 42-49.
Dirk Hovy, Taylor Berg-Kirkpatrick, Ashish Vaswani, and Eduard Hovy. 2013. Learning whom to trust with mace. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1120-1130.
Dirk Hovy, Federico Bianchi, and Tommaso Fornacciari. 2020. Can you translate that into man? commercial machine translation systems include stylistic biases. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Seattle, WA, USA. Association for Computational Linguistics.
Dirk Hovy and Anders Søgaard. 2015. Tagging performance correlates with author age. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), volume 2, pages 483-488.
Dirk Hovy and Shannon L Spruit. 2016. The social impact of natural language processing. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), volume 2, pages 591-598.
Jing Jiang and ChengXiang Zhai. 2007. Instance weighting for domain adaptation in nlp. In Proceedings of the 45th annual meeting of the association of computational linguistics, pages 264-271.
Anna Jørgensen, Dirk Hovy, and Anders Søgaard. 2015. Challenges of studying and processing dialects in social media. In Proceedings of the Workshop on Noisy User-generated Text, pages 9-18.
Kenneth Joseph, Lisa Friedland, William Hobbs, Oren Tsur, and David Lazer. 2017. Constance: Modeling annotation contexts to improve stance classification. arXiv preprint arXiv:1708.06309.
David Jurgens. 2013. That's what friends are for: Inferring location in online social media platforms based on social relationships. In Seventh International AAAI Conference on Weblogs and Social Media.
David Jurgens, Yulia Tsvetkov, and Dan Jurafsky. 2017. Incorporating dialectal variability for socially equitable language identification. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 51-57.
ML Kern, G Park, JC Eichstaedt, HA Schwartz, M Sap, LK Smith, and LH Ungar. 2016. Gaining insights from social media language: Methodologies and challenges. Psychological methods, 21(4):507-525.
Svetlana Kiritchenko and Saif Mohammad. 2018. Examining gender and race bias in two hundred sentiment analysis systems. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, pages 43-53.
Jon Kleinberg, Jens Ludwig, Sendhil Mullainathan, and Cass R Sunstein. 2018. Discrimination in the age of algorithms. Journal of Legal Analysis, 10.
Austin C Kozlowski, Matt Taddy, and James A Evans. 2018. The geometry of culture: Analyzing meaning through word embeddings. arXiv preprint arXiv:1803.09288.
Keita Kurita, Nidhi Vyas, Ayush Pareek, Alan W. Black, and Yulia Tsvetkov. 2019. Measuring bias in contextualized word representations. arXiv preprint arXiv:1906.07337.
Anne Lauscher, Goran Glavaš, Simone Paolo Ponzetto, and Ivan Vulić. 2020. A general framework for implicit and explicit debiasing of distributional word vector spaces.
Yitong Li, Timothy Baldwin, and Trevor Cohn. 2018. Towards robust and privacy-preserving text representations. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), volume 2, pages 25-30.
Veronica E. Lynn, Youngseo Son, Vivek Kulkarni, Niranjan Balasubramanian, and H. Andrew Schwartz. 2017. Human centered nlp with user-factor adaptation. In Empirical Methods in Natural Language Processing.
Thomas Manzini, Yao Chong Lim, Yulia Tsvetkov, and Alan W Black. 2019. Black is to criminal as Caucasian is to police: Detecting and removing multiclass bias in word embeddings. arXiv preprint arXiv:1904.04047.
Robert R. McCrae and Paul T. Costa Jr. 1989. Reinterpreting the Myers-Briggs type indicator from the perspective of the five-factor model of personality. Journal of Personality, 57(1):17-40.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. In Proceedings of ICLR.
Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. 2019. Model cards for model reporting. In Proceedings of the Conference on Fairness, Accountability, and Transparency, pages 220-229. ACM.
Ehsan Mohammady and Aron Culotta. 2014. Using county demographics to infer attributes of twitter users. In Proceedings of the joint workshop on social dynamics and personal attributes in social media, pages 7-16.
Malvina Nissim, Rik van Noord, and Rob van der Goot. 2019. Fair is better than sensational: Man is to doctor as woman is to doctor. arXiv preprint arXiv:1905.09866.
David K Park, Andrew Gelman, and Joseph Bafumi. 2006. State-level opinions from national surveys: Poststratification using multilevel logistic regression. Public opinion in state politics, pages 209-28.
Rebecca J Passonneau and Bob Carpenter. 2014. The benefits of a model of annotation. Transactions of the Association for Computational Linguistics, 2:311-326.
Silviu Paun, Bob Carpenter, Jon Chamberlain, Dirk Hovy, Udo Kruschwitz, and Massimo Poesio. 2018. Comparing bayesian models of annotation. Transactions of the Association for Computational Linguistics, 6:571-585.
Ellie Pavlick, Matt Post, Ann Irvine, Dmitry Kachaev, and Chris Callison-Burch. 2014. The language demographics of amazon mechanical turk. Transactions of the Association for Computational Linguistics, 2:79-92.
James W Pennebaker. 2011. The secret life of pronouns. New Scientist, 211(2828):42-45.
James W. Pennebaker and Lori D. Stone. 2003. Words of wisdom: Language use over the life span. Journal of Personality and Social Psychology, 85(2):291.
Barbara Plank, Dirk Hovy, and Anders Søgaard. 2014. Learning part-of-speech taggers with inter-annotator
agreement loss. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 742-751.
Marcelo OR Prates, Pedro H Avelar, and Luís C Lamb. 2018. Assessing gender bias in machine translation: a case study with google translate. *Neural Computing and Applications*, pages 1-19.
Daniel Preotciuc-Pietro, Johannes Eichstaedt, Gregory Park, Maarten Sap, Laura Smith, Victoria Tobolsky, H Andrew Schwartz, and Lyle Ungar. 2015. The role of personality, age, and gender in tweeting about mental illness. In Proceedings of the 2nd workshop on computational linguistics and clinical psychology: From linguistic signal to clinical reality, pages 21-30.
Daniel Preotiuc-Pietro, Maarten Sap, H Andrew Schwartz, and Lyle Ungar. 2015. Mental illness detection at the world well-being project for the clpsych 2015 shared task. In Proceedings of the Workshop on Computational Linguistics and Clinical Psychology: From Linguistic Signal to Clinical Reality, NAACL.
Iyad Rahwan, Manuel Cebrian, Nick Obradovich, Josh Bongard, Jean-François Bonnefon, Cynthia Breazeal, Jacob W Crandall, Nicholas A Christakis, Iain D Couzin, Matthew O Jackson, et al. 2019. Machine behaviour. Nature, 568(7753):477.
Alexey Romanov, Maria De-Arteaga, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alexandra Chouldechova, Sahin Geyik, Krishnamaram Kenthapadi, Anna Rumshisky, and Adam Tauman Kalai. 2019. What's in a name? reducing bias in bios without access to protected attributes. arXiv preprint arXiv:1904.05233.
Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. 2018. Gender bias in coreference resolution. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), volume 2, pages 8-14.
Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A. Smith. 2019. The risk of racial bias in hate speech detection. In Proceedings of the 57th Conference of the Association for Computational Linguistics, pages 1668-1678, Florence, Italy. Association for Computational Linguistics.
H Andrew Schwartz, Johannes C Eichstaedt, Margaret L Kern, Lukasz Dziurzynski, Megha Agrawal, Gregory J Park, Shrinidhi K Lakshmikanth, Sneha Jha, Martin EP Seligman, Lyle Ungar, et al. 2013. Characterizing geographic variation in well-being using tweets. In Seventh International AAAAI Conference on Weblogs and Social Media (ICWSM 2013).
Gabriel Stanovsky, Noah A. Smith, and Luke Zettlemoyer. 2019. Evaluating gender bias in machine
translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1679-1684, Florence, Italy. Association for Computational Linguistics.
Tony Sun, Andrew Gaut, Shirlyn Tang, Yuxin Huang, Mai ElSherief, Jieyu Zhao, Diba Mirza, Elizabeth Belding, Kai-Wei Chang, and William Yang Wang. 2019. Mitigating gender bias in natural language processing: Literature review. In Proceedings of the 57th Conference of the Association for Computational Linguistics, pages 1630–1640, Florence, Italy. Association for Computational Linguistics.
Harini Suresh and John V Guttag. 2019. A framework for understanding unintended consequences of machine learning. arXiv preprint arXiv:1901.10002.
Chris Sweeney and Maryam Najafian. 2019. A transparent framework for evaluating unintended demographic bias in word embeddings. In Proceedings of the 57th Conference of the Association for Computational Linguistics, pages 1662-1667, Florence, Italy. Association for Computational Linguistics.
Spencer S Swinton. 1981. Predictive bias in graduate admissions tests. ETS Research Report Series, 1981(1):i-53.
Peter Trudgill. 2000. Sociolinguistics: An introduction to language and society. Penguin UK.
Amos Tversky and Daniel Kahneman. 1973. Availability: A heuristic for judging frequency and probability. Cognitive psychology, 5(2):207-232.
Svitlana Volkova, Theresa Wilson, and David Yarowsky. 2013. Exploring demographic language variations to improve multilingual sentiment analysis in social media. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1815-1827.
Kellie Webster, Marta Recasens, Vera Axelrod, and Jason Baldridge. 2018. Mind the gap: A balanced corpus of gendered ambiguous pronouns. Transactions of the Association for Computational Linguistics, 6:605-617.
Thomas A. Widiger and Douglas B. Samuel. 2005. Diagnostic categories or dimensions? A question for the Diagnostic and Statistical Manual of Mental Disorders—Fifth Edition. Journal of Abnormal Psychology, 114(4):494.
Diyi Yang, Robert E Kraut, Tenbroeck Smith, Elijah Mayfield, and Dan Jurafsky. 2019. Seekers, providers, newcomers, and storytellers: Modeling social roles in online health communities. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, pages 1-14.
Mark Yatskar, Luke Zettlemoyer, and Ali Farhadi. 2016. Situation recognition: Visual semantic role labeling for image understanding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5534-5542.
Emilio Zagheni and Ingmar Weber. 2015. Demographic research with non-representative internet data. International Journal of Manpower, 36(1):13-25.
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2017. Men also like shopping: Reducing gender bias amplification using corpus-level constraints. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2979-2989.
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2018. Gender bias in coreference resolution: Evaluation and debiasing methods. arXiv preprint arXiv:1804.06876.
Ran Zmigrod, Sebastian J. Mielke, Hanna Wallach, and Ryan Cotterell. 2019. Counterfactual data augmentation for mitigating gender stereotypes in languages with rich morphology. In Proceedings of the 57th Conference of the Association for Computational Linguistics, pages 1651-1661.
A.1 Related Work in Other Fields
We survey the literature of adjacent fields and outline different streams related to our framework. These examples illustrate the ubiquity and complexity of bias, and highlight its understanding in different disciplines over time.
Bias became a crucial topic in social science following the seminal work of Tversky and Kahneman, showing that human thinking was subject to systematic errors (Tversky and Kahneman, 1973). Human logic was seemingly separate from the principles of probability calculus. "Bias" here is interpreted as the result of psychological heuristics, i.e., mental "shortcuts" to help us react faster to situations. While many of these heuristics can be useful in critical situations, their indiscriminate application in everyday life can have adverse effects and cause bias. This line of work has spawned an entire field of study in psychology (decision making).
The focus of Tversky and Kahneman (1973) (and a whole field of decision making that followed) was human behavior. Still, the same basic principles of systematic differences in decision making apply to machines as well. However, algorithms also provide systematic ways to reduce bias, and some see the mitigation of bias in algorithm decisions as a potential opportunity to move the needle positively (Kleinberg et al., 2018). Thus, we can apply frameworks of contemporaries in human behavior to machines (Rahwan et al., 2019), and perhaps benefit from a more scalable experimentation process. Costello and Watts (2014) studies human judgment under uncertain conditions, and proposes that we can algorithmically account for observed human bias, provided there is sufficient random noise in the probabilistic model. This view suggests bias within the model itself, what we have called overamplification.
Still, most works on bias in decision making assume working with unbiased data, even though social science has long battled selection bias. Most commonly, data selection is heavily skewed towards the students found on western university campuses (Henrich et al., 2010). Attempts to remedy selection bias in a scalable fashion use online populations, which in turn are skewed by unequal access to the Internet, but which we can mitigate through reweighting schemes (Couper, 2013).
In some cases, algorithmic bias has helped understand society better. For example, semantic bias in word embeddings has been leveraged to track trends in societal attitudes concerning gender roles and ethnic stereotypes. Garg et al. (2018); Kozlowski et al. (2018) measure the distance between certain sets of words in different decades to track this change. This use of biased embeddings illustrates an interesting distinction between normative and descriptive ethics. When used in predictive models, semantic bias is something to be avoided (Bolukbasi et al., 2016). I.e., it is nominatively wrong for many applications (e.g., we ideally would want all genders or ethnicities equally associated with all jobs). However, the works by Garg et al. (2018) and Kozlowski et al. (2018) show that it is precisely this bias of word embeddings that reflects societal attitudes. Here, the presence of bias is descriptively correct. Similarly, Bhatia (2017) uses this property of word embeddings to measure people's psychological biases and attitudes towards making individual decisions.
A.2 Discussion: Example Case Studies
Part of Speech Taggers and Parsing. The works by Hovy and Søgaard (2015); Jørgensen et al. (2015) outline the effect of selection bias on syntactic tools. The language of demographic groups systematically differs from each other for syntactic attributes. Therefore, models trained on samples whose demographic composition (e.g., age and ethnicity) differs from the target perform significantly worse. Within the predictive bias framework, the consequence of this selection bias is an error disparity -  $Q(\epsilon_{D = \text{general}} | A = \text{age}, \text{ethnicity}) \propto \text{Uniform}$ , the error of the model across a general domain ( $D$ ) is not uniform with respect to attributes ( $A$ ) age and ethnicity. Li et al. (2018) shows that this consequence of selection bias can be addressed by adversarial learning, removing the age gap and significantly reducing the performance difference between ethnolects (even if it was not trained with that objective). Garimella et al. (2019) quantifies this bias further by studying the effect of different gender compositions of the training data on tagging and parsing, supporting the claim that debiased samples benefit performance.
Image Captions. Hendricks et al. (2018) shows the presence of gender bias in image captioning, overamplifying differences present in the train
ing data. Prior work focused on context (e.g., it is easier to predict "mouse" when there is a computer present). This bias manifests in ignoring people present in the image. The gender bias is not only influenced by the images, but also by biased language models. The primary consequence is an outcome disparity  $-Q(\hat{Y}_D|gender) \approx P(Y_D|gender)$ , the distribution of outcomes (i.e. caption words and phrases) produced from the model  $Q(\hat{Y}_D|gender)$  overselects particular phrases beyond the distribution observed in reality: (i.e.  $P(Y_D|gender)$ ; this is true even when the source and target are the same:  $D = source = target$ ).
To overcome the bias and to increase performance, Hendricks et al. (2018) introduce an equalizer model with two loss-terms: Appearance Confusion Loss (ACL) and Confident Loss (Conf). ACL increases the gender confusion when gender information is not present in the image, making it difficult to predict an accurately gendered word. Confident loss increases the confidence of the predicted gendered word when gender information is present in the image. Both loss terms have the effect of decreasing the difference between  $Q(\hat{Y}_D|gender)$  and  $P(\hat{Y}_D|gender)$ . In the end, the Equalizer model performs better in predicting a woman while still misclassifying a man as a woman, but decreasing error disparity overall.
Sentiment Analysis. Kiritchenko and Mohammad (2018) show the issues of both semantic bias and overamplification. They assess scoring differences in 219 sentiment analysis systems by switching out names and pronouns. (They switch between male and female pronouns, and between prototypical white and black American first names based on name registers.) The results show that male pronouns are associated with higher scores for negative polarity, and prototypical black names with higher scores for negative emotions. The consequence of the semantic bias and overamplification are outcome disparities:  $Q(\hat{Y}_D|gender) \propto P(Y_D|gender)$  and  $Q(\hat{Y}_D|race) \propto P(Y_D|race)$ . This finding again demonstrates a case of descriptive vs. normative ethics. We could argue that because aggression is more often associated with male protagonists, the models reflect a descriptively correct (if morally objectionable) societal fact. However, if the model score changes based on ethnicity, the difference likely reflects (and amplifies) societal ethnic stereotypes. Those stereo
types, though, are both normatively and descriptively wrong.
Differential Diagnosis in Mental Health. In the clinical community, differentiating a subject with post-traumatic stress disorder (PTSD) from someone with depression is known to be difficult. It was, therefore, surprising when early work on this task produced AUCs greater than 0.85 (this and similar tasks were part of the CLPsych2015 Shared task; (Coppersmith et al., 2015)). Labels of depression and PTSD had been automatically derived from a convenience sample of individuals who had publicly stated their diagnosis in their profile. The task included a 50/50 split from each category. However, Preotiuc-Pietro et al. (2015) show that these classifiers primarily picked up on differences in age or gender – subjects with PTSD were more likely to be older than those with depression. While age and gender themselves are valid information for mental health diagnosis, the design yielded classifiers that predicted nearly all older individuals to have PTSD, and those younger to have depression, despite the 50/50 split. These classifiers resulted in outcome disparity, because older individuals were much less likely to be labeled depressed than the target population (and younger less likely for PTSD:  $Q(\hat{Y}_D|A = age) \propto P(Y_D|A = age)$ ). In the end, the task organizers mitigated the issue by using matched controls – adding another  $50\%$  samples for each class such that the age and gender distributions of both groups matched. Recently, Benton et al. (2017) showed that accounting for demographic attributes in the model could leverage this correlation while controlling for the confounds.
Assessing Demographic Variance in Language. A final case study in applying our framework demonstrates how inferring user demographics can mitigate bias. Consider the task of producing population measurements from readily available (but biased) community corpora. E.g., assessing representative US county life satisfaction from tweets (Schwartz et al., 2013). Unlike our other examples, the outcomes of the source training data (i.e., surveys) are expected to be representative, while the features come with biases. The source feature distributions with respect to human attributes are dissimilar from the ideal distribu
${}^{6}$  A convenience sample, a term from social science, is a set of data selected because it is available rather than designed for the given task.
tion, while the source outcomes match that target outcomes  $(Q(X_{source}|A)\approx P(X_{target}|A)$  but  $Q(Y_{source}|A)\sim P(Y_{target}|A))$
In this case, the effectiveness of countermeasures preventing selection and semantic biases (for  $X_{source}$  and  $X_{target}$ ) should result in increased predictive performance against a representative community outcome. Indeed, Giorgi et al. (2019) adjust the feature estimates,  $X$ , to match representative demographics and socio-economics by using inferred user attributes, and find improved predictions for the life satisfaction of a Twitter community.