Classist Tools: Social Class Correlates with Performance in NLP (2024)

Abstract: The field of sociolinguistics has studied factors affecting language use for the last century. Labov (1964) and Bernstein (1960) showed that socioeconomic class strongly influences our accents, syntax and lexicon. However, despite growing concerns surrounding fairness and bias in Natural Language Processing (NLP), there is a dearth of studies delving into the effects it may have on NLP systems. We show empirically that NLP systems’ performance is affected by speakers’ SES, potentially disadvantaging less-privileged socioeconomic groups. We annotate a corpus of 95K utterances from movies with social class, ethnicity and geographical language variety and measure the performance of NLP systems on three tasks: language modelling, automatic speech recognition, and grammar error correction. We find significant performance disparities that can be attributed to socioeconomic status as well as ethnicity and geographical differences.1 With NLP technologies becoming ever more ubiquitous and quotidian, they must accommodate all language varieties to avoid disadvantaging already marginalised groups. We argue for the inclusion of socioeconomic class in future language technologies.

heading: 1 Introduction
text: The relationship between social class and language was first examined by Labov (1964). He discovered that New Yorkers with greater socioeconomic status pronounce the /R/ sound after vowels, whereas individuals with lower socioeconomic position drop it. He quantified this observation by asking employees in various department shops (a proxy for socioeconomic rank) for things found on
1We make our artifacts available on our Github repository.
the fourth floor. Then he kept track of how many Rs were dropped in those two words. He discovered a clear anti-correlation between the store’s (and presumably the speakers’) socioeconomic standing and the quantity of dropped Rs: the higher the status, the fewer dropped Rs.
Since Labov’s (1964) analysis of social stratification and language, linguistics has made concerted efforts to understand how different sociodemographic factors influence language production and perception, and how speakers use them to create identity (Rickford, 1986; Bucholtz and Hall, 2005; Eckert, 2012); using both naturally produced language and language from media like TV shows and films (Stamou, 2014) to study sociolects. Despite a considerable body of evidence demonstrating links between language and demographic characteristics (the “first wave” of sociolinguistic variation studies), comparatively few socio-demographic factors have been studied in the context of Natural Language Processing (NLP) technology (Cercas Curry et al., 2024). But NLP is a product and a tool for understanding other phe-
12643
nomena and must represent all language varieties. Existing research on socio-demographic characteristics has primarily concentrated on the signaling effect of certain linguistic variables like age, ethnicity, regional origin, and gender (Johannsen et al., 2015). Most of these publications focus on any bias toward the particular variable they study. However, NLP hardly interacts with the second and third waves of sociolinguistics, i.e., how variation 1) shapes local identity and 2) drives language development. To narrow this gap, we use NLP to focus on socioeconomic position.
Using a dataset of 95K utterances from Englishlanguage television episodes and movies, we empirically investigate performance as a function of socioeconomic class. We study its effect on vocabulary, Automated Speech Recognition (ASR), language modelling, and grammatical error correction. In all applications, we see strong correlations. Movie characters are usually typecast to present as a certain class, providing a representative sample without the privacy issues associated with regular subjects. The authenticity of these representations is backed by work in sociolinguistics such as McHoul (1987) and Quaglio (2008).
We discover that the performance of NLP tools is associated with socioeconomic class, geographic variety (US vs. UK), and race across all tasks. Our findings highlight an important lack of flexibility of NLP tools, and a more fundamental issue: What does it mean if NLP technologies only reliably work for a limited segment of society?
Contributions
• We construct a corpus of 95K utterances from television shows and movie scripts, coded for socioeconomic status, geography, and race.
• We empirically show that socioeconomic status measurably impacts the performance of NLP systems across four measures.

heading: 2 Social Class and Language
text: Social stratification is the grouping of people based on their socioeconomic status (SES), which is determined by factors such as income, education, wealth, and other characteristics. Different groups are distinguished in terms of power and prestige. There are various social stratification systems, such as the Indian caste system, indigenous American clans or tribes, and the Western hierarchical class system.
The exact number of social strata is unknown and varies by country and culture. However, at
least three strata are commonly used to refer to different groups in Western societies: upper, middle, and lower-class people. Other systems distinguish between blue collar and white collar jobs. Recently, the Great British Class Survey (GBCS, Savage et al., 2013) has taken an empirical approach to understanding the different social strata, and they propose a seven-level system for the United Kingdom. Their stratification is based on economic, social, and cultural capital: elite, established middle-class, technical middle-class, new affluent workers, traditional working class, emergent service workers, and precariat. They derive these classes from a survey conducted over British citizens that received more than 160K responses.
Social class influences people’s daily lives by granting or limiting access to resources. Beyond power and prestige, social stratification has a significant impact on people. For example, lower socioeconomic status has been linked to poorer health outcomes and higher mortality. (Saydah et al., 2013).
SES also affects language use from the very early stages of development. Bernstein (1960) theorised that language takes on a different role in families of different socioeconomic status, where middleclass parents encourage language learning to describe more abstract thinking leading to broader vocabularies. In working-class families, parents are limited to more concrete and descriptive concepts. Parents from lower SES tend to interact less with their children, with fewer open-ended questions than parents from higher SES, which shapes language development (Clark and Casillas, 2015). While Usategui Basozábal et al. (1992) show that education reduces this language gap, education has traditionally been one of the most important factors in determining social class and potential for upward social mobility.
Given the well-documented effects of socioeconomic status on language development and use, it stands to reason that social class should be carefully considered as a variable in NLP.

heading: 3 Dataset
text: Here, we want to assess the impact a user’s social class has on the performance of NLP systems. Although there have been concerted efforts to document wide varieties of English, existing datasets do not currently capture all social strata or other sociodemographic aspects sufficiently. For example, the National Speech Corpus (Clopper and Pisoni, 2006) provides education information
for the talker and their parents but consists of almost exclusively white and educated speakers;2 the CORAAL corpus (Kendall and Farrington, 2018) on the other hand is focused on African American English (AAE), not capturing other ethnicities. To the best of our knowledge, no datasets are available that would suit our purposes: a large enough sample that is annotated with the speakers’ socieconomic background. We thus collect our own.
We identify an initial sample of English language television series by selecting the 250 highestrated series on the International Movie Database (IMDB).3 We annotate each show according to attributes,4 i.e., class, race, gender, dialect, and geography of the main characters, as well as the genre and time period of the show. Three annotators familiar with the shows agree on the labels unanimously. We only consider shows that are scripted to ensure that the language production of each character is controlled by the script-writers and actors to produce a particular vernacular and to avoid collecting highly sensitive information about private individuals. We therefore exclude cartoons, documentaries. We further exclude science fiction and historical fiction, as these are likely to operate without strict adherence to existing social structures and language variation. After our filtering and selection, 63 shows remain in consideration. Of those, 34 are predominantly about white men, and 29 feature a mixture of white women and men. Only three predominantly feature female characters, and none black or lower-class female characters. Based on this initial sample, we identified additional shows to augment gaps (e.g., the lack of Black stories) in class and dialect. Because we want to measure the effect of class separately from gender, race and regional accent, we select the TV shows from the list representing each category.
In our final selection of movies and television shows, we include a total of 19 shows that are based in the United States of America and the United Kingdom. To ensure racial diversity, we emphasise movies and shows that predominantly represent Black characters in addition to movies and
2Education is one aspect of social class. For a further discussion see Saunders (1990) and Savage et al. (2013).
3IMDB top 250 TV shows. 4Note that we are using ascribed characteristics rather than self-reported since we are dealing with fictional characters, however these two are found to be highly correlated based on linguistic features (Weirich and Simpson, 2018)
shows that represent white characters.5 We include the U.S.-based shows The Wire and When They See Us (predominantly lower-class Black characters), Fargo (middle-class Black characters), The Fresh Prince of Bel-Air (upper-class Black characters). For the shows’ centre white characters, we include Trailer Park Boys (lower-class white), The Sopranos (lower-class white), Breaking Bad (middle-class white), and Arrested Development (upper-class white). The list of shows above, however, predominantly feature male characters. For this reason, we also include shows whose characters represent a wider array of genders. The shows that we include are Pose (lower-class Black and Latinx), Big Little Lies (middle-class white), and Sex and the City (upper middle-class white). Finally, to compare with other regional varieties of English, we include movies and shows from different regions in the United Kingdom. The movies and shows that we include are Train Spotting and T2 (lower-class white Scottish, predominantly male characters), Derry Girls (lower-class white Irish, primarily female), Downton Abbey (English), The IT Crowd (white, middle-class) and The Crown (upper-class white English). We report full details in Table 8, Appendix A.
We collect the first season for each of the shows on our list, except for Fargo and The Wire. For Fargo, we collect the fourth season, which includes Black characters. For The Wire, we collect seasons 1 and 3, which feature Black characters but have a minority representation of white characters.
We deliberately chose this approach to data selection from “artificial sources” since collecting class information is highly sensitive and can introduce privacy risks. Fictional characters, however, are often scripted into particular roles with explicit SES. A drawback of this methodology is the question of authenticity - the extent to which actors are representative of particular sociolects. Actors reportedly undergo significant training to learn different lects.6 By selecting highly rated shows, we hope to capture good, realistic performances. Moreover, TV show and film data is increasingly used in sociolinguistics (Stamou, 2014) and Quaglio (2008) found important similarities between real and TV show dialogue. In NLP, media data is broadly used
5At least one of the authors is familiar with every show and movie. We annotated them for race and class based on the setup and main characters.
6See for example, https://www.newyorker.com/ magazine/2009/11/09/talk-this-way
in tasks like emotion analysis (Plaza-del Arco et al., 2024) and dialogue (Lison and Tiedemann, 2016). We therefore have reason to believe our dataset provides a reasonable proxy for real speakers.
In total, our dataset contains 95K utterances, text and speech, from 19 TV shows and movies (see Table 1 for dataset statistics).

heading: 3.1 Lexical Analysis
text: Bernstein (1960) showed that children from working-class families had significantly smaller vocabularies even when general IQ was controlled for, and Flekova et al. (2016) showed that there is significant lexical and stylistic variation between different social strata, with lexical features being stronger predictors of income than even age.
We first assess whether our dataset has sufficient linguistic cues to capture lexical differences. We generally follow the methodology set out in Flekova et al. (2016). However, we model class as a categorical value. We calculate:
Surface: Turn length, mean word length per turn, ratio of words longer than five letters, type-token ratio. Flekova et al. (2016) also measure the use of emojis, but this is not relevant to our analysis.
Readability Metrics: Readability metrics aim to measure the complexity of a text generally based on the number of syllables per word and the number of words per sentence. Using Textstat,7 we calculate the Automatic Readability Index (ARI) (Senter and Smith, 1967), the Flesch-Kincaid Grade Level (FKG) (Kincaid et al., 1975), the Coleman-Liau Index (CLI) (Coleman and Liau, 1975), the Flesch Reading Ease (FRE) (Flesch, 1948), the LIX Index (LIX) (Anderson, 1983), and the Gunning-Fog Index (FOG) (Gunning, 1968). We normalise the text before calculating these metrics by lower-casing
7https://github.com/textstat/textstat
and removing punctuating except for periods since these are used by CLI. Note that these were designed to evaluate long-form text.
Syntax: Texts with more nouns and articles as opposed to pronouns and adverbs are considered more formal. We measure the ratio of each POS per turn using Stanza (Qi et al., 2020).
Style: Finally, to capture some notion of speaker style, we measure the number of abstract words8 , the ratio of hapax legomena (HL), and the number of named entities (NER).
We run a multi-class logistic regression to understand whether the features are correlated to class. After normalisation, we run a logistic regression with social class as the predicted class and the
8https://onlymyenglish.com/ list-of-abstract-nouns/
features above as the predictive features. In contrast with Flekova et al. (2016), we find our set of features only mildly predictive in this dataset. Although CLI and the number of characters have strong coefficients, the overall performance is very low. Our best model achieves a macro F1 of 0.16, only slightly better than a most-frequent label baseline (macro-F1 of 0.07). In addition, we run the same regression for the other sociodemographic aspects: gender, race, and geographical origin. Overall, we find these less predictive with F1 scores similar to the baseline, suggesting the features better capture differences caused by SES than other sociodemographic factors. Table 2 shows the coefficients for each sociodemographic factor.
Lexical features do not capture strong differences in SES indicating one of two things: the features are better suited for written text (for example, many of them capture formality), and/or our dataset does not accurately capture linguistic differences. To assess whether linguistic cues in our dataset might help us differentiate between speakers of SES which the previous study did not capture, we experiment with alternative text encodings such as TF-IDF, and sentence embeddings (Reimers and Gurevych, 2019).9 The results are shown in Table 3. Logistic regression based on TF-IDF achieves an F1 of 0.52 (x1.8 lexical features). We hypothesise two reasons for the improvement over the lexical features: (1) there are linguistic differences that the metrics do not capture, or (2) the logistic regression model can better differentiate between TV shows. Given the existing literature in support of linguistic differences and the (albeit small) predictive power of the metrics, we ascribe the improvement to a mixture of both. With this in mind, we continue our experiments with user-facing NLP systems.

heading: 4 Social Class and Speech
text: Beyond written language, our dataset affords testing whether speech models understand and process accent unevenly. Therefore, we test disparities in ASR word error rates across social classes. We calculate the WER for all shows and movies to measure how well ASR technologies capture different variants of English in our sample.
Models: We use Wav2Vec2 (Baevski et al., 2020), XLS-R (Babu et al., 2021), and Whispermedium (Radford et al., 2023) as implemented in
9We use scikit-learn’s TfidfVectorizer collecting uniand bi-grams for TF-IDF, and https://huggingface.co/ sentence-transformers/all-mpnet-base-v2.
transformers (Wolf et al., 2020).
Processing Pipeline: We use movies and TV shows which are available online. We first collect the audio and subtitles from OpenSubtitles10 and use them as gold reference.11 Then, we used one ASR model to transcribe the audio.
The timestamps from the model-generated transcription and the subtitles often do not match, and neither source provides information about which character is speaking. We therefore group all utterances from a single episode or movie together into one long string and calculate the WER using Jiwer (Morris et al., 2004). Ideally, we would like to separate by character to get a more fine-grained analysis. However, this approach would require us to keep mappings from utterance to character consistent across episodes. Despite several approaches, we were not able to achieve this goal with the computational power we had. Our analysis is therefore group-based, rather than character-based.
Results: Figure 2 shows the differences in worderror-rate by race and class in US TV shows. We find clear trends across models, with effects from both race and class: lower ASR error rates are associated with higher SES and with whiteness. These trends are stronger for the Wav2Vec2 model.

heading: 5 Language Modelling
text: Language modelling plays an essential role in downstream applications in NLP, so it is imperative that language models can accurately and equally represent different speakers’ lects. Language models are evaluated through perplexity, a measure of how expected a given sentence is: the higher the perplexity, the more unexpected. Perplexity can give us an indication of how well a model might perform in downstream tasks: Gonen et al. (2023) show that the lower the perplexity of a prompt, the better the prompt can perform the task. Here, we calculate perplexity as a measure of linguistic acceptability, that is, how ‘expectable’ a particular language variant might be to a given language model. Should models display differing perplexities for different groups, they would put such groups at a disadvantage.
Models: We experiment with two state-of-theart base models, Llama 2 (Touvron et al., 2023)
10https://www.opensubtitles.org/ 11Here, we manually checked a random sample to verify the quality. We henceforth assume that the subtitles are an accurate transcription of what is being said.
and Mistral-7B (Jiang et al., 2023). Moreover, we include Zephyr-7B (Tunstall et al., 2023), a Mistral-7B-based chat model to test the effect of alignment.12
We calculate perplexity for each sentence in our dataset after lower-casing them and removing numbers and punctuation. We exclude turns shorter than five tokens as they generally do not differentiate between classes.
Results: Table 4 shows the mean perplexity and standard deviation for each model and each class for the entire dataset. We do not find significant differences based on geographical location (U.K. vs U.S.A) across the models. We see small differences across groups based on class alone, however, because language is affected by other sociodemographic besides class, we also consider the effect of race and geographic variety.
We calculate mean perplexity by SES in the U.K. and the U.S.A. Figure 3 shows that class and per-
12HuggingFace: meta-llama/Llama-2-7b, mistralai/Mistral7B-v0.1, and HuggingFaceH4/zephyr-7b-beta
plexity are correlated and we find similar patterns across geographical dialects: higher SES leads to lower perplexity also in U.S.A.-based shows.
Next, we consider U.S.-based shows. The breakdown of by class and race are shown in Table 5. Once both factors are taken into account, a clearer picture emerges: lower SES leads to higher perplexity. To confirm these differences are significant, we run a one-tailed student’s t-test and find significant differences between classes, particularly for white speakers for both Mistral-7B and Zephyr-7B. In the case of Mistral, lower-class and lower-middle class speakers have significantly higher perplexities than Mid-Upper (∗) and Middle class speakers (∗). In the case of Zephyr, the model shows higher perplexities for lower classes than it does for Middle (∗), Mid-Upper (∗∗) and Upper class speakers (∗∗).13 In sum, models show significantly higher perplexities for Lower class speakers than for higher prestige groups.

heading: 6 Grammar Correction
text: Finally, we consider the grammaticality of each sentence. We suspect that grammar features correlate strongly with class, as “correct” grammar is typically a hallmark of signalling higher SES. We calculate the edit distance between the original sentence and the corrected one. Lower edit distance means higher grammaticality.
Models: Since we are concerned with potential end-user experience, we choose to evaluate the four most downloaded models on Huggingface since they will reach the largest audience. We use the following models for grammar correction:
13Significance: ∗: p < 0.05, ∗∗: p < 0.01.
Perplexity
Black Low 328.848 Middle, Upper 259.673
White Low 275.337 Low, middle 342.865 Middle 229.278 Middle, Upper 205.585 Upper 302.057
Results: Figure 4 shows the percentage of utterances that each model corrected, grouped by geographical variation. The Flan-T5-based model generates significantly more corrections than the other models, followed by CoEdit-large. Overall, U.K. utterances are corrected slightly more often by the models. Table 6 shows the mean edit distances between the original sentence extracted from the subtitles and the models’ proposed corrections. We find little difference in grammar error correction for most models, even when controlling for geographical and racial differences. However, on further analysis, we find that models generally produce corrections for relatively few utterances.
Instead, we consider whether models are more likely to produce edits for different classes. In Figure 5 we plot the count of utterances with corrections per model and per class. From this, we can see a clear pattern: models produce corrections more frequently for those of lower SES.
In addition, we notice that often some of these corrections are performed on in-group slang or regional linguistic phenomena, see Table 7.
19Skag refers to heroin in Scottish slang.

heading: 7 Discussion
text: Following established work in sociolinguistics, our results support our hypothesis that NLP systems display biases towards different sociolects, beyond geographical and ethnic differences.
On the difference between life and art: The use of media (if not synthetic) data as a stand-in for real-world examples is not uncommon in NLP (e.g. Plaza-del Arco et al., 2024; Lison and Tiedemann, 2016). Our work presents a starting point to study an aspect of linguistic variation that has thus far received little attention in the NLP literature. To substantiate our call to action, the characters’ language need not be an exact replica but a good approximation of a given sociolect. However, we do not claim that all media portrayals are accurate representations: TV shows and films are written by certain people, for specific audiences, and the quality of acting varies. Through our selection process, we selected shows that would not caricature certain groups, including TV shows like The Wire and Pose, which feature a cast representative of the characters. Nevertheless, these results should be validated with real speakers.
NLP as an enforcer of “standard” language: Native speakers of any language have different registers available to them, including when interacting with NLP systems. While this may help mitigate some of the potential harms shown in this paper, our position is that humans should not have to adapt their language to interact with technology, but rather technology should adapt to humans. By dint of their use, NLP systems are setting a standard of language: NLP systems are becoming more common, not only in social contexts calling for formal (i.e. standardised) language, or in fields like Social Science (Ziems et al., 2024), but in everyday sce-
narios. For example one of the proposed use cases for the Gramformer is correcting text messages as a built-in feature for messaging apps, language models inform systems like automatic correction. In Section 6, we show grammar-correction models erase particularities of certain language varieties like regional slang or norms. Likewise, LLMs like chatGPT now boast hundreds millions of users. Keeping in mind the findings from Gonen et al. (2023) and ours in Section 5, we may find users of lower SES have worse user experience with such systems. This widespread interaction with language technologies reinforces the lect of middle class, white English speakers as the standard even in quotidian activities.
Going forward While reporting on the socioeconomic background of study participants and dataset contributors is more complicated than other factors such as age or ethnicity (as it is not a one-point measurement and it is culture-dependent), we encourage researchers to report on this to draw a more accurate picture of the language varieties NLP will capture and to better benchmark systems across sociolects. Given the limitations of our study (most notably, restriction to English and group rather than individual scores, see the Limitations section below), we envision several possible avenues for
future research in this area: (1) the findings of this study should be validated for other languages where such differences have been observed; (2) the collection of a (possibly larger) corpus of unaggregated speakers; and (3) while we find support from previous research such as Flekova et al. (2016) for social media data, the findings should also be validated for other data forms such as long-form written texts.

heading: 8 Related Work
text: Works in NLP considering social class in the context of language use are few and far between (Cercas Curry et al., 2024), mostly focusing on predicting social class from texts (e.g. Basile et al., 2019; Lampos et al., 2014). Nguyen et al. (2016) ascribe the dearth of work in bias and social class to the lack of self-reported explicit labels in online user profiles. The difficulty in obtaining such data hinders efforts in addressing this issue. More recently, some work has focused on representational harms with regards to social class: Curto et al. (2022) investigate bias against the poor in NLP in terms of the language that surrounds poor people, considering the associations of ‘poor’ and ‘rich’ in embeddings and language models; Kiritchenko et al. (2023) instead focus on aporophobia, classist attitude stigmatising the poor, in the context of toxic language. To the best of our knowledge, we are the first to investigate how NLP systems perform with speakers from different social strata.

heading: 9 Conclusion
text: Social class plays an important role in people’s identity construction, and is consequently strongly reflected in their language use. Despite its central status as a variable in sociolinguistics, there is so far little work in NLP engaging with social class and its impact. Our contributions are twofold: following methods from sociolinguistics, we use a data set of 95K movie transcripts that we annotate for portrayed class, race, and geographical variety (UK vs. US). We use this data to show that linguistic class markers greatly affect the performance of various NLP tools. Our findings suggest that speakers of low-prestige sociolects experience lower application performance on a range of tasks. These results suggest that we should actively incorporate social class as a variable in NLP systems if we want to make them more equitable and fair. Our data set also provides a starting point for more explorations of social class in NLP.
Limitations
Our study presents several limitations: First, we are limited to the study of class, region, and race in English. TV shows and films are biased in their representations of different identities—not all groups are (equally) represented and those that are may be stereotyped into certain roles. For example, in the initial top-200 TV shows list, we did not find any TV shows about lower-class women, Latinxs, upper-class Non-English Britons, or non-white Britons. For this reason, we excluded a gender-based analysis. Moreover, in this list, the majority of TV shows about Black Americans revolve around the drug trade.
Second, characters are not necessarily written by, nor perfectly depict the group they represent and this is potentially reflected in their language. To ensure high quality portrayals, we sourced highly rated and realistic shows, ensuring characters were not caricatures of the groups they portrayed. Having said that, a sociolinguistics review found that TV show and film data is increasingly common as data for the study of sociolects, particularly with regards to race and class (Stamou, 2014). More specifically, work in sociolinguistics has also defended the use of film data in conversation analysis (McHoul, 1987; Quaglio, 2008). Furthermore, TV shows like The Wire have been praised for their linguistic authenticity (Lopez and Bucholtz, 2017), reportedly primarily hiring Baltimore natives. For these reasons, we believe our study sufficiently captures the relevant linguistic phenomena.
Despite our best efforts, we were unable to reliably separate the utterances by character across episodes. Instead of introducing more sources of potential mix-ups, we therefore estimate the metrics over all characters in a show. Although this is a limitation in terms of the accuracy of the results, we expect that character-level annotations would strengthen our findings rather than negate them. Generally speaking, each TV show focuses on a single social stratum. A single character belonging to a different stratum would add noise and make the classes less clearly defined, so we suspect our results would be more marked with character-level annotations, not less. We leave the character-based separation for future work.
Finally, though our dataset is large in terms of utterances, it is limited in the number of characters. However, we present a case study hoping to motivate further research that may validate our findings.
Ethical Considerations
Our paper is arguing for fairness in the services provided by NLP systems for people of all socioeconomic backgrounds. However, we note that by showing measurable differences in language use across groups, we run the risk of profiling speakers. In our work, we used fictional characters to avoid this, but in order to ensure fairness of service all peoples must be represented. Future work should ensure that this is done respectfully and with consent from any participants.
Furthermore, measuring socioeconomic status is not a straightforward process. The class system used in this paper is fuzzy and tied to Western social structures that are not valid across the world. Future work should focus on measuring in the most appropriate way by following established metrics and guidelines from economics and other fields, such as Savage et al. (2013). We urge any future work to follow ethical guidelines when it comes to dataset and system development.

heading: Acknowledgements
text: A special thanks to Federico Bianchi and Lindsay Schaffer and the reviewers for their helpful comments. Amanda Cercas Curry and Dirk Hovy were supported by the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation program (grant agreement No. 949944, INTEGRATOR). They are members of the MilaNLP group and the Data and Marketing Insights Unit of the Bocconi Institute for Data Science and Analysis.
Giuseppe Attanasio was supported by the Portuguese Recovery and Resilience Plan through project C645008882-00000055 (Center for Responsible AI) and by Fundação para a Ciência e Tecnologia through contract UIDB/50008/2020. He conducted part of the work as a member of the MilaNLP group at Bocconi University, Milan.

heading: A Annotations per show
text: Table 8 shows the annotated demographics for each season/film.

heading: Future Work
text: We leave the character-based separation for future work. Finally, though our dataset is large in terms of utterances, it is limited in the number of characters. However, we present a case study hoping to motivate further research that may validate our findings. Ethical Considerations
Our paper is arguing for fairness in the services provided by NLP systems for people of all socioeconomic backgrounds. However, we note that by showing measurable differences in language use across groups, we run the risk of profiling speakers. In our work, we used fictional characters to avoid this, but in order to ensure fairness of service all peoples must be represented. Future work should ensure that this is done respectfully and with consent from any participants. Furthermore, measuring socioeconomic status is not a straightforward process. The class system used in this paper is fuzzy and tied to Western social structures that are not valid across the world. Future work should focus on measuring in the most appropriate way by following established metrics and guidelines from economics and other fields, such as Savage et al. (2013). We urge any future work to follow ethical guidelines when it comes to dataset and system development