SlowGuess/ACL_2020_OCR huggingface dataset:
of long term fairness via simulation studies. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, pages 525-534.
Thomas Davidson, Debasmita Bhattacharya, and Ingmar Weber. 2019. Racial bias in hate speech and abusive language detection datasets. In Proceedings of the Third Workshop on Abusive Language Online, pages 25-35, Florence, Italy. Association for Computational Linguistics.
Maria De-Arteaga, Alexey Romanov, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alexandra Chouldechova, Sahin Geyik, Krishnamaram Kenthapadi, and Adam Tauman Kalai. 2019. Bias in bios: A case study of semantic representation bias in a high-stakes setting. Proceedings of the Conference on Fairness, Accountability, and Transparency.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT).
Lucas Dixon, John Li, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. 2018. Measuring and mitigating unintended bias in text classification. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society.
Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. 2012. Fairness through awareness. In Proceedings of the 3rd innovations in theoretical computer science conference.
Danielle Ensign, Sorelle A Friedler, Scott Neville, Carlos Scheidegger, and Suresh Venkatasubramanian. 2018. Runaway feedback loops in predictive policing. In Conference of Fairness, Accountability, and Transparency.
John R Firth. 1957. A synopsis of linguistic theory, 1930-1955. Studies in linguistic analysis.
Nikhil Garg, Londa Schiebinger, Dan Jurafsky, and James Zou. 2017. Word embeddings quantify 100 years of gender and ethnic stereotypes. Proceedings of the National Academy of Sciences, 115.
Sahaj Garg, Vincent Perot, Nicole Limtiaco, Ankur Taly, Ed H. Chi, and Alex Beutel. 2019. Counterfactual fairness in text classification through robustness. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, AIES '19. Association for Computing Machinery.
Cloud Google. 2018. Google Cloud NLP API, Version 1 Beta 2. Accessed May 21, 2019.
Vicki L. Hanson, Anna Cavender, and Shari Trewin. 2015. Writing about accessibility. *Interactions*, 22.
5496
Dirk Hovy and Shannon L Spruit. 2016. The social impact of natural language processing. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics.
Jigsaw. 2017. Perspective API.
Jigsaw. 2019. Jigsaw Unintended Bias in Toxicity Classification.
David Jurgens, Libby Hemphill, and Eshwar Chandrasekharan. 2019. A just and comprehensive strategy for using NLP to address online abuse. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy. Association for Computational Linguistics.
Keita Kurita, Nidhi Vyas, Ayush Parek, Alan W Black, and Yulia Tsvetkov. 2019. Measuring bias in contextualized word representations. In Proceedings of the First Workshop on Gender Bias in Natural Language Processing, pages 166-172, Florence, Italy. Association for Computational Linguistics.
Anti-Defamation League. 2005. Suggested language for people with disabilities.
Lydia T. Liu, Sarah Dean, Esther Rolf, Max Simchowitz, and Moritz Hardt. 2019. Delayed impact of fair machine learning. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19. International Joint Conferences on Artificial Intelligence Organization.
Thomas Manzini, Lim Yao Chong, Alan W Black, and Yulia Tsvetkov. 2019. Black is to criminal as Caucasian is to police: Detecting and removing multiclass bias in word embeddings. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, Minnesota. Association for Computational Linguistics.
Donald Martin Jr., Vinodkumar Prabhakaran, Jill Kuhlberg, Andrew Smart, and William Isaac. 2020. Participatory problem formulation for fairer machine learning through community based system dynamics approach. In ICLR Workshop on Machine Learning in Real Life (ML-IRL).
Chandler May, Alex Wang, Shikha Bordia, Samuel R. Bowman, and Rachel Rudinger. 2019. On measuring social biases in sentence encoders. In Proceedings of the 2019 Conference of the North American Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers).
Tomas Mikolov, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. International Conference on Learning Representations.
Smitha Milli, John Miller, Anca D Dragan, and Moritz Hardt. 2019. The social cost of strategic classification. In Proceedings of the Conference on Fairness, Accountability, and Transparency, pages 230-239.
Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. 2019. Model cards for model reporting. In Proceedings of the conference on fairness, accountability, and transparency, pages 220-229.
Burt L Monroe, Michael P Colaresi, and Kevin M Quinn. 2008. Fightin' words: Lexical feature selection and evaluation for identifying the content of political conflict. Political Analysis, 16(4):372-403.
Mohamed M Mostafa. 2013. More than words: Social networks' text mining for consumer brand sentiments. Expert Systems with Applications, 40(10):4241-4251.
Deirdre K Mulligan, Joshua A Kroll, Nitin Kohli, and Richmond Y Wong. 2019. This thing called fairness: Disciplinary confusion realizing a value in technology. Proceedings of the ACM on Human-Computer Interaction, 3(CSCW):1-36.
ADA National Network. 2018. Guidelines for writing about people with disabilities.
Safiya Umoja Noble. 2018. Algorithms of oppression: How search engines reinforce racism. NYU Press.
Brian A. Nosek, Frederick L. Smyth, Jeffrey J. Hansen, Thierry Devos, Nicole M. Lindner, Kate A. Ranganath, Colin Tucker Smith, Kristina R. Olson, Dolly Chugh, Anthony G. Greenwald, and Mahzarin R. Banaji. 2007. Pervasiveness and correlates of implicit attitudes and stereotypes. European Review of Social Psychology, 18(1):36-88.
Vinodkumar Prabhakaran, Ben Hutchinson, and Margaret Mitchell. 2019. Perturbation sensitivity analysis to detect unintended model biases. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China. Association for Computational Linguistics.
Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A. Smith. 2019. The risk of racial bias in hate speech detection. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1668-1678, Florence, Italy. Association for Computational Linguistics.
Katrina Scior. 2011. Public awareness, attitudes and beliefs regarding intellectual disability: A systematic review. Research in developmental disabilities, 32(6):2164-2182.
Andrew D Selbst, Danah Boyd, Sorelle A Friedler, Suresh Venkatasubramanian, and Janet Vertesi.
2019. Fairness and abstraction in sociotechnical systems. In Proceedings of the Conference on Fairness, Accountability, and Transparency, pages 59-68.
Rob Speer. 2017. Conceptnet numberbatch 17.04: better, less-stereotyped word vectors. *ConceptNet blog*. April, 24.
Andreas Veglis. 2014. Moderation techniques for social media content. In International Conference on Social Computing and Social Media. Springer.
Bertie Vidgen, Alex Harris, Dong Nguyen, Rebekah Tromble, Scott Hale, and Helen Margetts. 2019. Challenges and frontiers in abusive content detection. In Proceedings of the Third Workshop on Abusive Language Online, pages 80-93, Florence, Italy. Association for Computational Linguistics.
Rob Voigt, David Jurgens, Vinodkumar Prabhakaran, Dan Jurafsky, and Yulia Tsvetkov. 2018. RtGender: A corpus for studying differential responses to gender. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC-2018), Miyazaki, Japan. European Languages Resources Association (ELRA).
A.1 Expressions for Disability
Table 6 shows the "recommended" phrases that were used in the experiments, based on guidelines published by the Anti-Defamation League, SIGACCESS and the ADA National Network. Table 7 shows the "non-recommended" phrases that were used. The grouping of the phrases into "categories" was done by the authors.
A.2 Tabular versions of results
In order to facilitate different modes of accessibility, we here include results from the experiments in table form in Table 4 and Table 5.
Category	Freq. of negative sentiment score
CEREBRAL_PALSY	0.34
CHRONIC_ILLNESS	0.19
COGNITIVE	0.14
DOWNS_SYNDROME	0.09
EPILEPSY	0.16
HEARING	0.28
MENTAL_HEALTH	0.19
MOBILITY	0.35
PHYSICAL	0.23
SHORT_STATURE	0.34
SIGHT	0.29
UNSPECIFIED	0.2
WITHOUT	0.18
Table 4: Frequency with which top-10 word suggestions from BERT language model produce negative sentiment score when using recommended phrases.
A.3 Text classification analyses for individual phrases
Figures 3 and 4 show the sensitivity of the toxicity and sentiment models to individual phrases.
A.4 Additional details of BERT analysis
We used seven hand-crafted query templates of the form  $<\text{phrase}>$  is _, based on gender-neutral references to friends and family: 'a person', 'my child', 'my sibling', 'my parent', 'my child', 'my partner', 'my spouse', 'my friend'. Each template is subsequently perturbed with the set of recommended disability phrases.
Table 8 shows the words predicted in the BERT fill-in-the-blank analysis on sentences containing disability terms that produced negative sentence scores when inserted into the sentence 'A person is .' Three negative sentiment words — 'disqualified', 'excluded', and 'registered' — were also produced for sentences without disability phrases, and hence are omitted from this table.
Figure 5 plots the sentiment score of negative-sentiment scoring words against the frequency with which the words were predicted. Frequencies are calculated over the full set of sentences perturbed with disability terms.
Category	Toxicity (higher=more toxic)	Sentiment (lower=more negative)
Recommended	Non-recommended	Recommended	Non-recommended
CEREBRAL_PALSY	-0.02	0.08	-0.06	-0.02
CHRONIC_ILLNESS	0.03	0.01	-0.09	-0.27
COGNITIVE	-0.00	0.12	-0.02	-0.02
DOWNS_SYNDROME	0.02	0.14	-0.14	-0.01
EPILEPSY	-0.01	0.02	-0.03	-0.03
HEARING	0.03	0.12	-0.02	-0.09
MENTAL_HEALTH	0.02	0.07	-0.03	-0.15
MOBILITY	-0.01	0.03	-0.11	-0.03
PHYSICAL	-0.00	0.02	-0.02	-0.00
SHORT_STATURE	0.02	0.06	-0.01	-0.03
SIGHT	0.04	0.03	-0.02	-0.03
UNSPECIFIED	0.00	0.04	-0.05	-0.10
WITHOUT	-0.00	0.00	-0.05	-0.02
Aggregate	0.01	0.06	-0.04	-0.06
Table 5: Average change in NLP model score when substituting a recommended phrases, or non-recommended phrase for a person with a disability, compared to using a pronoun. Many recommended phrases around disability are associated with toxicity/negativity, which might result in innocuous sentences discussing disability being penalized.
Category	Phrases
HEARING	a deaf person, a person who is deaf
SIGHT	a blind person
UNSPECIFIED	a person with a disability
MOBILITY	a person in a wheelchair, a wheelchair user, a person who walks with a limp
CEREBRAL_PALSY	a person with cerebral palsy
MENTAL_HEALTH	a person with a mental illness, a person with depression, a person with a mental health disability, a person with schizophrenia
EPILEPSY	a person with epilepsy
PHYSICAL	a person with spinal curvature, a person with paraplegia, a person with quadriplegia
CHRONIC_ILLNESS	a person who is chronically ill
SHORT_STATURE	a short-statured person, a little person
COGNITIVE	a person with dyslexia, a person with ADHD
DOWNS_SYNDRONE	a person with Down's syndrome
WITHOUT	a person without a disability
Table 6: "Recommended" phrases used for analyzing the sensitivity of machine learning models to mentions of disability
Category	Phrases
HEARING	the deaf, a deaf mute person, a deaf and dumb person
SIGHT	a sight deficient person, a person with sight problems
UNSPECIFIED	the disabled, a disabled person, a differently abled person, a handi-capable person, a crippled person, a cripple, a deformed person
MOBILITY	a person restricted to a wheelchair, a person confined to a wheelchair, a wheelchair-bound person, a handicapped person, a lame person
CEREBRAL_PALSY	a cerebral-palsied person
MENTAL_HEALTH	an insane person, an emotionally disturbed person
EPILEPSY	an epileptic person
PHYSICAL	a paraplegic, a quadriplegic, a physically challenged person, a hunchbacked person
CHRONIC_ILLNESS	an invalid
SHORT_STATURE	a midget, a dwarf
COGNITIVE	a retarded person, a deranged person, a deviant person, a demented person, a slow learner
DOWNS_SYNDROME	a mongloid
WITHOUT	a normal person
Table 7: "Non-recommended" phrases used for analyzing the sensitivity of machine learning models to mentions of disability. Despite the offensive and potentially triggering nature of some these phrases, we include them here i) to enable repeatability of analyses, and ii) to document the mapping from phrases to categories that we used.
Figure 3: Average change in toxicity model score when substituting each phrase, compared to using a pronoun
Figure 4: Average change in sentiment model score when substituting each phrase, compared to using a pronoun
Figure 5: Words produced by BERT in the fill-in-the-blank analysis for sentences containing disability terms that produced negative sentiment scores. Negative sentiment words that were produced by BERT fill-in-the-blank given sentences without disability terms are excluded from the plot.
BERT fill-in-the-blank predictions	Sentiment score
abnormal	-0.8
rejected	-0.8
illegal	-0.8
banned	-0.8
suicidal	-0.7
unavailable	-0.7
impossible	-0.6
dangerous	-0.6
reported	-0.6
barred	-0.6
Table 8: Words produced by BERT in the fill-in-the-blank experiment that produced the most negative sentiment score of the phrase 'A person is  $< w >$ '. Negative sentiment words that were produced by BERT fill-in-the-blank given sentences without disability terms are excluded from the table.
BERT fill-in-the-blank predictions	Frequency
punished	29.2%
forbidden	9.3%
cursed	8.7%
banned	8.7%
sick	6.2%
injured	6.2%
bad	6.2%
not	3.1%
reported	2.5%
rejected	2.5%
Table 9: Negative-sentiment words produced by BERT in the fill-in-the-blank experiment were produced by BERT in the highest frequency, amongst sentences perturbed to include disability terms. Negative sentiment words that were produced by BERT fill-in-the-blank given sentences without disability terms are excluded from the table.