SlowGuess/NAACL_2021_OCR huggingface dataset:
In this paper, we have argued for considering language variation and social meaning when building representations. However, such research could potentially also support the development of applications that can cause harm. Long-standing research in sociolinguistics has shown rich connections between language variation and social attributes, including sensitive attributes such as gender and ethnicity (e.g. Eckert (2012)). One may take that as a motivation to build automatic profiling systems. However, as discussed in Section 2, sociolinguists have emphasised the highly contextual nature of social meaning (the same linguistic feature can have different social meanings) and the agency of speakers (language is not just a reflection of someone's identity, but can be actively used as a resource for identity construction). Profiling systems tend to impose categories on people based on broad stereotypical associations. They fail to recognise the rich
local identities and agency of individuals. Besides privacy concerns, misclassifications by such systems can cause severe harms.
Another ethical consideration is the training data. Data with colloquial language will likely offer richer signals for training, which could be augmented with information about the social and communicative context. Online sources such as Twitter and Reddit may be attractive given their size and availability of fine-grained social metadata. However, the use of large-scale online datasets (even though it is 'public') raises privacy and ethical concerns. We recommend following guidelines and discussions surrounding the use of online data in social media research—not only regarding collecting and storing data, but also how such data is shared, and how analyses based on such data are reported and disseminated (Fiesler and Proferes, 2018; Zook et al., 2017; Fiesler et al., 2020). One key step is documenting the datasets (Bender and Friedman, 2018; Gebru et al., 2018). In addition, social biases in these datasets can propagate into the learned representations (Bolukbasi et al., 2016; Caliskan et al., 2017), which may impact downstream applications that make use of these representations.
This work is part of the research programme Veni with project number VI.Veni.192.130, which is (partly) financed by the Dutch Research Council (NWO). We also like to thank Kees van Deemter for useful feedback.
Reina Akama, Kento Watanabe, Sho Yokoi, Sosuke Kobayashi, and Kentaro Inui. 2018. Unsupervised learning of style-sensitive word vectors. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 572-578, Melbourne, Australia.
Jannis K. Androutsopoulos. 2000. Non-standard spellings in media texts: The case of German fanzines. Journal of Sociolinguistics, 4(4):514-533.
David Bamman, Chris Dyer, and Noah A. Smith. 2014a. Distributed representations of geographically situated language. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 828-834, Baltimore, Maryland.
David Bamman, Jacob Eisenstein, and Tyler Schnoebelen. 2014b. Gender identity and lexical varia
tion in social media. Journal of Sociolinguistics, 18(2):135-160.
Marco Baroni, Georgiana Dinu, and German Kruszewski. 2014. Don't count, predict! A systematic comparison of context-counting vs. context-predicting semantic vectors. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 238-247, Baltimore, Maryland.
Emily M. Bender and Batya Friedman. 2018. Data statements for natural language processing: Toward mitigating system bias and enabling better science. Transactions of the Association for Computational Linguistics, 6:587-604.
Emily M. Bender and Alex Lascarides. 2019. Linguistic fundamentals for natural language processing II: 100 essentials from semantics and pragmatics. Synthesis Lectures on Human Language Technologies, 12(3):1-268.
Douglas Biber. 1988. Variation across Speech and Writing. Cambridge University Press.
Douglas Biber and Susan Conrad. 2009. Register, Genre, and Style. Cambridge Textbooks in Linguistics. Cambridge University Press.
Yonatan Bisk, Ari Holtzman, Jesse Thomason, Jacob Andreas, Yoshua Bengio, Joyce Chai, Mirella Lapata, Angeliki Lazaridou, Jonathan May, Aleksandr Nisnevich, Nicolas Pinto, and Joseph Turian. 2020. Experience grounds language. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8718-8735, Online.
Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020. Language (technology) is power: A critical survey of "bias" in NLP. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5454-5476, Online. Association for Computational Linguistics.
Su Lin Blodgett, Lisa Green, and Brendan O'Connor. 2016. Demographic dialectal variation in social media: A case study of African-American English. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1119-1130, Austin, Texas.
Tolga Bolukbasi, Kai-Wei Chang, James Zou, Venkatesh Saligrama, and Adam Kalai. 2016. Man is to computer programmer as woman is to homemaker? Debiasing word embeddings. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS'16, page 4356-4364.
Aylin Caliskan, Joanna J. Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from language corpora contain human-like biases. Science, 356(6334):183-186.
Kathryn Campbell-Kibler. 2007. Accent, (ING), and the social logic of listener perceptions. American Speech, 82(1):32-64.
Kathryn Campbell-Kibler. 2009. The nature of sociolinguistic perception. Language Variation and Change, 21(01):135-156.
Kathryn Campbell-Kibler. 2010. The effect of speaker information on attitudes toward (ING). Journal of Language and Social Psychology, 29(2):214-223.
Jack Chambers. 2003. *Sociolinguistic Theory: Linguistic Variation and Its Social Significance*. Oxford: Blackwell.
Anne Cocos and Chris Callison-Burch. 2017. The language of place: Semantic value from geospatial context. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 99-104, Valencia, Spain.
Alexis Conneau, German Kruszewski, Guillaume Lample, Loic Barrault, and Marco Baroni. 2018. What you can cram into a single  $\$ \& !\#$ * vector: Probing sentence embeddings for linguistic properties. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2126-2136, Melbourne, Australia.
Erika Darics. 2013. Non-verbal signalling in digital discourse: The case of letter repetition. Discourse, Context & Media, 2(3):141-148.
Thomas Davidson, Debasmita Bhattacharya, and Ingmar Weber. 2019. Racial bias in hate speech and abusive language detection datasets. In Proceedings of the Third Workshop on Abusive Language Online, pages 25-35, Florence, Italy.
Marco Del Tredici and Raquel Fernandez. 2017. Semantic variation in online communities of practice. In IWCS 2017 - 12th International Conference on Computational Semantics - Long papers.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota.
Penelope Eckert. 2008. Variation and the indexical field. Journal of Sociolinguistics, 12(4):453-476.
Penelope Eckert. 2012. Three waves of variation study: The emergence of meaning in the study of sociolinguistic variation. Annual Review of Anthropology, 41(1):87-100.
Penelope Eckert and John R. Rickford, editors. 2002. Style and Sociolinguistic Variation. Cambridge University Press.
Philip Edmonds and Graeme Hirst. 2002. Near-synonymy and lexical choice. Computational linguistics, 28(2):105-144.
Jacob Eisenstein. 2013. What to do about bad language on the internet. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 359-369, Atlanta, Georgia.
Jacob Eisenstein. 2015. Systematic patterning in phonologically-motivated orthographic variation. Journal of Sociolinguistics, 19(2):161-188.
Guy Emerson. 2020. What are the goals of distributional semantics? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7436-7453, Online.
Jessica Ficler and Yoav Goldberg. 2017. Controlling linguistic style aspects in neural language generation. In Proceedings of the Workshop on Stylistic Variation, pages 94-104, Copenhagen, Denmark.
Casey Fiesler, Nathan Beard, and Brian C. Keegan. 2020. No robots, spiders, or scrapers: Legal and ethical regulation of data collection methods in social media terms of service. In Proceedings of the International AAAI Conference on Web and Social Media, pages 187-196.
Casey Fiesler and Nicholas Proferes. 2018. "Participant" perceptions of Twitter research ethics. *Social Media + Society*, 4(1).
Edward Finegan and Douglas Biber. 2002. Register variation and social dialect variation: the Register Axiom, page 235-267. Cambridge University Press.
Lucie Flek. 2020. Returning the N to NLP: Towards contextually personalized classification models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7828-7838, Online. Association for Computational Linguistics.
Aparna Garimella, Carmen Banea, and Rada Mihalcea. 2017. Demographic-aware word associations. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2285-2295, Copenhagen, Denmark.
Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, and Kate Crawford. 2018. Datasheets for datasets. CoRR, abs/1803.09010.
Anna Gladkova, Aleksandr Drozd, and Satoshi Matsuoka. 2016. Analogy-based detection of morphological and semantic relations with word embeddings: What works and what doesn't. In Proceedings of the NAACL-HLT SRW, pages 47-54, San Diego, California, June 12-17, 2016.
William L. Hamilton, Jure Leskovec, and Dan Jurafsky. 2016. Diachronic word embeddings reveal statistical laws of semantic change. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1489-1501, Berlin, Germany.
Bo Han and Timothy Baldwin. 2011. Lexical normalisation of short text messages: Makn sens a #twitter. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 368-378, Portland, Oregon, USA.
Susan C. Herring and Asta Zelenkauskaite. 2009. Symbolic capital in a virtual heterosexual market: Abbreviation and insertion in Italian iTV SMS. Written Communication, 26(1):5-31.
Dirk Hovy. 2018. The social and the neural network: How to make natural language processing about people again. In Proceedings of the Second Workshop on Computational Modeling of People's Opinions, Personality, and Emotions in Social Media, pages 42-49, New Orleans, Louisiana, USA.
Dirk Hovy and Christoph Purschke. 2018. Capturing regional variation with distributed place representations and geographic retrofitting. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4383-4394, Brussels, Belgium.
Dirk Hovy and Anders Søgaard. 2015. Tagging performance correlates with author age. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 483-488, Beijing, China.
Eduard H. Hovy. 1990. Pragmatics and natural language generation. Artificial Intelligence, 43(2):153-197.
Christian Ilbury. 2020. "Sassy Queens": Stylistic orthographic variation in Twitter and the enregisterment of AAVE. Journal of Sociolinguistics, 24(2):245-264.
William Labov. 1972. Sociolinguistic patterns. 4. University of Pennsylvania Press.
David Lazer, Alex Pentland, Lada Adamic, Sinan Aral, Albert-Laszlo Barabasi, Devon Brewer, Nicholas Christakis, Noshir Contractor, James Fowler, Myron Gutmann, Tony Jebara, Gary King, Michael Macy, Deb Roy, and Marshall Van Alstyne. 2009. Computational social science. Science, 323(5915):721-723.
Daisy Leigh. 2018. Expecting a performance: Listener expectations of social meaning in social media. Presented at New Ways of Analyzing Variation (NWAV) 47.
Chang Li, Aldo Porco, and Dan Goldwasser. 2018. Structured representation learning for online debate stance prediction. In Proceedings of the 27th International Conference on Computational Linguistics, pages 3728-3739, Santa Fe, New Mexico, USA.
Jiwei Li, Michel Galley, Chris Brockett, Georgios Spithourakis, Jianfeng Gao, and Bill Dolan. 2016. A persona-based neural conversation model. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 994-1003, Berlin, Germany.
Fei Liu, Fuliang Weng, Bingqing Wang, and Yang Liu. 2011. Insertion, deletion, or substitution? Normalizing text messages without pre-categorization nor supervision. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 71-76, Portland, Oregon, USA.
Hui Liu, Yongzheng Zhang, Yipeng Wang, Zheng Lin, and Yige Chen. 2020. Joint character-level word embedding and adversarial stability training to defend adversarial text. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 8384-8391.
Li Lucy and David Bamman. 2021. Characterizing English variation across social media communities with BERT. In To appear in TACL.
Veronica Lynn, Youngseo Son, Vivek Kulkarni, Niranjan Balasubramanian, and H. Andrew Schwartz. 2017. Human centered NLP with user-factor adaptation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1146-1155, Copenhagen, Denmark.
François Mairesse and Marilyn A. Walker. 2011. Controlling user perceptions of linguistic style: Trainable generation of personality traits. Computational Linguistics, 37(3):455-488.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S. Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111-3119.
Dong Nguyen, A. Seza Dogruoz, Carolyn P. Rose, and Franciska de Jong. 2016. Computational sociolinguistics: A Survey. Computational Linguistics, 42(3):537-593.
Dong Nguyen and Jacob Eisenstein. 2017. A kernel independence test for geographical language variation. Computational Linguistics, 43(3):567-592.
Dong Nguyen and Jack Grieve. 2020. Do word embeddings capture spelling variation? In Proceedings of the 28th International Conference on Computational Linguistics, pages 870-881, Barcelona, Spain (Online).
Dong Nguyen, Maria Liakata, Simon DeDeo, Jacob Eisenstein, David Mimno, Rebekah Tromble, and Jane Winters. 2020. How we do things with words: Analyzing text as social and cultural data. Frontiers in Artificial Intelligence, 3:62.
Andrea Nini, George Bailey, Diansheng Guo, and Jack Grieve. 2020. The graphical representation of phonetic dialect features of the North of England on social media, pages 266-296. Edinburgh University Press, United Kingdom.
Xing Niu and Marine Carpuat. 2017. Discovering stylistic variations in distributional vector space models via lexical paraphrases. In Proceedings of the Workshop on Stylistic Variation, pages 20-27, Copenhagen, Denmark.
Umashanthi Pavalanathan, Jim Fitzpatrick, Scott Kiesling, and Jacob Eisenstein. 2017. A multidimensional lexicon for interpersonal stancetaking. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 884-895, Vancouver, Canada.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543, Doha, Qatar.
Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2227-2237, New Orleans, Louisiana.
Aleksandra Piktus, Necati Bora Edizel, Piotr Bojanowski, Edouard Grave, Rui Ferreira, and Fabrizio Silvestri. 2019. Misspelling oblivious word embeddings. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3226-3234, Minneapolis, Minnesota.
Alexey Romanov, Anna Rumshisky, Anna Rogers, and David Donahue. 2019. Adversarial decomposition of text representation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 815-825, Minneapolis, Minnesota.
Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A. Smith. 2019. The risk of racial bias in hate speech detection. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1668-1678, Florence, Italy.
Mark Sebba. 2007. Spelling and society: The culture and politics of orthography around the world. Cambridge University Press.
Philippa Shoemark, James Kirby, and Sharon Goldwater. 2018. Inducing a lexicon of sociolinguistic variables from code-mixed text. In Proceedings of the 2018 EMNLP Workshop W-NUT: The 4th Workshop on Noisy User-generated Text, pages 1-6, Brussels, Belgium.
Philippa Shoemark, Debnil Sur, Luke Shrimpton, Iain Murray, and Sharon Goldwater. 2017. Aye or naw, whit dae ye hint? Scottish independence and linguistic identity on social media. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 1239-1248, Valencia, Spain. Association for Computational Linguistics.
Michael Silverstein. 2003. Indexical order and the dialectics of sociolinguistic life. Language & Communication, 23(3):193 - 229. Words and Beyond: Linguistic and Semiotic Studies of Sociocultural Order.
Ian Stewart, Yuval Pinter, and Jacob Eisenstein. 2018. Si O no, que penses? Catalanian independence and linguistic identity on social media. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 136-141, New Orleans, Louisiana. Association for Computational Linguistics.
Caroline Tagg. 2009. A corpus linguistics study of SMS text messaging. Ph.D. thesis, University of Birmingham.
Sali A. Tagliamonte. 2015. Making waves: The story of variationist sociolinguistics. John Wiley & Sons.
Rachael Tatman, Leo Stewart, Amandalynne Paullada, and Emma Spiro. 2017. Non-lexical features encode political affiliation on Twitter. In Proceedings of the Second Workshop on NLP and Computational Social Science, pages 63–67, Vancouver, Canada. Association for Computational Linguistics.
Sean Trott, Tiago Timponi Torrent, Nancy Chang, and Nathan Schneider. 2020. (Re) construing meaning in NLP. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5170-5184, Online.
Johanna Vaattovaara and Elizabeth Peterson. 2019. Same old paska or new shit? On the stylistic boundaries and social meaning potentials of swearing loanwords in Finnish. *Ampersand*, 6:100057.
Abby Walker, Christina García, Yomi Cortés, and Kathryn Campbell-Kibler. 2014. Comparing social meanings across listener and speaker groups: The indexical field of Spanish /s/. Language Variation and Change, 26(02):169-189.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353-355, Brussels, Belgium.
Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. 2020. BLiMP: The benchmark of linguistic minimal pairs for English. Transactions of the Association for Computational Linguistics, 8:377-392.
Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. 2019. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625-641.
Albert Webson, Zhizhong Chen, Carsten Eickhoff, and Ellie Pavlick. 2020. Do "undocumented workers" == "illegal aliens"? Differentiating denotation and connotation in vector spaces. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4090-4105, Online.
Yi Yang and Jacob Eisenstein. 2017. Overcoming language variation in sentiment analysis with social attention. Transactions of the Association for Computational Linguistics, 5:295-307.
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In The IEEE International Conference on Computer Vision (ICCV), pages 19-27.
Matthew Zook, Solon Barocas, danah boyd, Kate Crawford, Emily Keller, Seeta Peña Gangadharan, Alyssa Goodman, Rachelle Hollander, Barbara A. Koenig, Jacob Metcalf, Arvind Narayanan, Alondra Nelson, and Frank Pasquale. 2017. Ten simple rules for responsible big data research. PLOS Computational Biology, 13(3):1-10.