A Framework for the Computational Linguistic Analysis of Dehumanization

Dehumanization is a pernicious psychological process that often leads to extreme intergroup bias, hate speech, and violence aimed at targeted social groups. Despite these serious consequences and the wealth of available data, dehumanization has not yet been computationally studied on a large scale. Drawing upon social psychology research, we create a computational linguistic framework for analyzing dehumanizing language by identifying linguistic correlates of salient components of dehumanization. We then apply this framework to analyze discussions of LGBTQ people in the New York Times from 1986 to 2015. Overall, we find increasingly humanizing descriptions of LGBTQ people over time. However, we find that the label homosexual has emerged to be much more strongly associated with dehumanizing attitudes than other labels, such as gay. Our proposed techniques highlight processes of linguistic variation and change in discourses surrounding marginalized groups. Furthermore, the ability to analyze dehumanizing language at a large scale has implications for automatically detecting and understanding media bias as well as abusive language online.

section_name: Keywords:
paragraphs: computational sociolinguistics, dehumanization, lexical variation, language change, media, New York Times, LGBTQ
Warning: this paper contains material that some may find offensive or upsetting.
section_name: Introduction
paragraphs: Despite the American public's increasing acceptance of LGBTQ people and recent legal successes, LGBTQ individuals frequently remain the targets of hate and violence BIBREF0, BIBREF1, BIBREF2. At the core of this issue is dehumanization, “the act of perceiving or treating people as less than human” BIBREF3, a process that heavily contributes to extreme intergroup bias BIBREF4. Language is central to studying this phenomenon; like other forms of bias BIBREF5, BIBREF6, BIBREF7, BIBREF8, BIBREF9, dehumanizing attitudes are expressed through subtle linguistic manipulations, even in carefully-edited texts. It is crucial to understand the use of such linguistic signals in mainstream media, as the media's representation of marginalized social groups has far-reaching implications for social acceptance, policy, and safety.
While small-scale studies of dehumanization and media representation of marginalized communities provide valuable insights (e.g. BIBREF10), there exist no known large-scale analyses, likely due to difficulties in quantifying such a subjective and multidimensional psychological process. However, the ability to do large-scale analysis is crucial for understanding how dehumanizing attitudes have evolved over long periods of time. Furthermore, by being able to account for a greater amount of media discourse at once, large-scale techniques can provide a more complete view of the media environment to which the public is exposed.
Linguistics and computer science offer valuable methods and insights on which such large-scale techniques might be developed for the study of dehumanization. By leveraging more information about the contexts in which marginalized groups are discussed, computational linguistic methods not only enable large-scale study of a complex psychological phenomenon, but can even reveal linguistic variations and changes that are not easily identifiable through qualitative analysis alone.
In this work, we develop a computational linguistic framework for analyzing dehumanizing language, with a focus on lexical signals of dehumanization. Social psychologists have identified numerous components of dehumanization, such as negative evaluations of a target group, denial of agency, moral disgust, and likening members of a target group to non-human entities such as vermin. Drawing upon this rich body of literature, we first identify clear linguistic analogs for these components and propose several computational techniques to measure these linguistic correlates. We then apply this general framework to explore changing representations of LGBTQ groups in the New York Times over the span of three decades. We additionally use this lens of dehumanization to investigate differences in social meaning between the denotationally-similar labels gay and homosexual.
This paper aims to bridge the gaps between computational modeling, sociolinguistics, and dehumanization research with implications for several disciplines. In addition to enabling large-scale studies of dehumanizing language and media representation of marginalized social groups, these techniques can be built into systems that seek to capture both conscious and unconscious biases in text. Furthermore, this work has implications for improving machines' abilities to automatically detect hate speech and abusive language online, which are typically underpinned by dehumanizing language. Finally, our case study demonstrates that such computational analyses of discussions about marginalized groups can provide unique insights into language variation and change within sensitive sociopolitical contexts, and help us understand how people (and institutions) use language to express their ideologies and attitudes towards certain social groups.
section_name: Background ::: Dehumanization
paragraphs: Our lexical semantic analysis involves quantifying linguistic correlates of component psychological processes that contribute to dehumanization. Our approaches are informed by social psychology research on dehumanization, which is briefly summarized here. Prior work has identified numerous related processes that comprise dehumanization BIBREF4. One such component is likening members of the target group to non-human entities, such as machines or animals BIBREF4, BIBREF11, BIBREF12. By perceiving members of a target group to be non-human, they are “outside the boundary in which moral values, rules, and considerations of fairness apply" BIBREF13, which thus leads to violence and other forms of abuse. Metaphors and imagery relating target groups to vermin are particularly insidious and played a prominent role in the genocide of Jews in Nazi Germany and Tutsis in Rwanda BIBREF14. More recently, the vermin metaphor has been invoked by the media to discuss terrorists and political leaders of majority-Muslim countries after September 11 BIBREF15. According to BIBREF16, the vermin metaphor is particularly powerful because it conceptualizes the target group as “engaged in threatening behavior, but devoid of thought or emotional desire".
Disgust underlies the dehumanizing nature of these metaphors and is itself another important element of dehumanization. Disgust contributes to members of target groups being perceived as less-than-human and of negative social value BIBREF17. It is often evoked (both in real life and experimental settings) through likening a target group to animals. BIBREF18 find that priming participants to feel disgust facilitates “moral exclusion of out-groups". Experiments by BIBREF17 and BIBREF19 similarly find that disgust is a predictor of dehumanizing perceptions of a target group. Both moral disgust towards a particular social group and the invocation of non-human metaphors are facilitated by essentialist beliefs about groups, which BIBREF4 presents as a necessary component of dehumanization. In order to distinguish between human and non-human, dehumanization requires an exaggerated perception of intergroup differences. Essentialist thinking thus contributes to dehumanization by leading to the perception of social groups as categorically distinct, which in turn emphasizes intergroup differences BIBREF4
According to BIBREF4, much of the prior research describes “extremely negative evaluations of others" as a major component of dehumanization. This is especially pronounced in BIBREF20's account of delegitimization, which involves using negative characteristics to categorize groups that are “excluded from the realm of acceptable norms and values" BIBREF20. While BIBREF20 defines delegitimization as a distinct process, he considers dehumanization to be one means of delegitimization. BIBREF13 also discusses broader processes of moral exclusion, one of which is dehumanization. A closely related process is psychological distancing, in which one perceives others to be objects or nonexistent BIBREF13. BIBREF21 identifies elements that contribute to the objectification (and thus dehumanization) of women, one of which is denial of subjectivity, or the habitual neglect of one's experiences, emotions, and feelings.
Another component of dehumanization is the denial of agency to members of the target group BIBREF4. According to BIBREF16, there are three types of agency: "the ability to (1) experience emotion and feel pain (affective mental states), (2) act and produce an effect on their environment (behavioral potential), and (3) think and hold beliefs (cognitive mental states) BIBREF16. Dehumanization typically involves the denial of one or more of these types of agency BIBREF16.
In Section SECREF4, we introduce computational linguistic methods to quantify several of these components of dehumanization.
section_name: Background ::: Related Computational Work
paragraphs: While this is the first known computational work that focuses on dehumanization, we draw upon a growing body of literature at the intersection of natural language processing and social science research. We are particularly inspired by the area of automatically detecting subjective language, which was largely pioneered by Janyce Wiebe and colleagues, who developed novel lexical resources and algorithms for this task based on word and phrase distributions BIBREF5. These resources have been used as linguistically-informed features in machine learning classification of biased language BIBREF7. Other work has expanded this lexicon-based approach to account for the role of syntactic form for identifying the writer's perspective towards different entities mentioned in a text BIBREF6, BIBREF22.
These methods have been used and expanded to analyze pernicious, but often implicit social biases BIBREF23. For example, BIBREF8 analyzed racial bias in police transcripts by training machine learning classifiers with linguistic features informed by politeness theory BIBREF8, and BIBREF24 investigated historical racial biases in mass media through changing word embeddings BIBREF24. Other work has looked at words in particular syntactic contexts to analyze power and agency, and how these concepts relate to gender bias in movies BIBREF25 and in mass media articles about the #MeToo movement BIBREF26. There has also been a growing focus on the analysis of subtle manifestations of social biases, such as condescension BIBREF27, microagressions BIBREF9, and “othering" language BIBREF28, BIBREF29. In addition, our focus on dehumanization is closely related to the detection and analysis of hate speech and abusive language BIBREF30, BIBREF31.
Gender and racial bias have also been identified within widely-deployed NLP systems, for tasks including toxicity detection BIBREF32, sentiment analysis BIBREF33, coreference resolution BIBREF34, language identification BIBREF35, and in many other areas BIBREF36. Given the biases captured, reproduced, and perpetuated in NLP systems, there is also a growing interest in mitigating subjective biases BIBREF36, with approaches including modifying embedding spaces BIBREF37, BIBREF38, augmenting datasets BIBREF39, and adapting natural language generation methods to “neutralize" text BIBREF40.
A related line of research has developed computational approaches to investigate language use and variation in discourse about complex sociopolitical issues in the media. For example, some work has drawn upon political communication theory to automatically detect the framing of an issue BIBREF41, BIBREF42, BIBREF43 through both supervised classification BIBREF44, BIBREF45 and unsupervised methods, such as topic modeling and lexicon induction BIBREF46, BIBREF47, BIBREF48. Scholars have also developed computational methods to identify lexical cues of partisan political speech, political slant in mass media, and polarization in social media BIBREF49, BIBREF50, BIBREF48.
section_name: Background ::: Attitudes Towards LGBTQ Communities in the United States
paragraphs: Because we apply our framework to study dehumanizing language in discussions about LGBTQ communities in the New York Times, some background is necessary. Overall, the American public has become more accepting of LGBTQ people and supportive of their rights. In 1977, equal percentages of respondents (43%) agreed and disagreed with the statement that gay or lesbian relations between consenting adults should be legal BIBREF2. Approval of gay and lesbian relations then decreased in the 1980s; in 1986, only 32% of respondents believed they should be legal. According to Gallup, attitudes have become increasingly positive since the 1990s, and in 2019, 73% responded that gay or lesbian relations should be legal. The Pew Research center began surveying Americans about their beliefs about same-sex marriage in 2001 and found similar trends BIBREF51. Between 2001 and 2019, support for same-sex marriage among respondents jumped from 35% to 61%.
In addition to the public's overall attitudes, it is important to consider variation and change in the specific words used to refer to LGBTQ people. Because these labels potentially convey many different social meanings and have different relationships with dehumanization in the media, a primary focus of this study involves comparing different LGBTQ labels, specifically gay and homosexual. The Gallup survey asked for opinions on legality of “homosexual relations" until 2008, but then changed the wording to “gay and lesbian relations". This was likely because many people who identify as gay and lesbian find the word homosexual to be outdated and derogatory. According to the LGBTQ media monitoring organization GLAAD, homosexual's offensiveness originates in the word's dehumanizing clinical history, which had falsely suggested that “people attracted to the same sex are somehow diseased or psychologically/emotionally disordered" . Beyond its outdated clinical associations, some argue that the word homosexual is more closely associated with sex and all of its negative connotations simply by virtue of containing the word sex, while terms such as gay and lesbian avoid such connotations BIBREF52. Most newspapers, including the New York Times, almost exclusively used the word homosexual in articles about gay and lesbian people until the late 1980s BIBREF53. The New York Times began using the word gay in non-quoted text in 1987. Many major newspapers began restricting the use of the word homosexual in 2006 BIBREF52. As of 2013, the New York Times has confined the use of homosexual to specific references to sexual activity or clinical orientation, in addition to direct quotes and paraphrases .
Beyond differences in how LGBTQ people perceive the terms gay or lesbian relative to homosexual, it has been shown that the specific choice of label can also have an effect on attitudes towards LGBTQ people. In 2012, BIBREF54 asked survey respondents about either “gay and lesbian rights" or “homosexual rights". Respondents who read the word “homosexual" tended to show less support for LGBTQ rights. This effect was primarily driven by high authoritarians, people who show high sensitivity to intergroup distinctions. The authors posit that homosexual makes social group distinctions more blatant than gay or lesbian, which then leads to a greater degree of psychological distancing, thus enabling participants to further remove LGBTQ people from their realm of moral consideration BIBREF54. Based on prior research and evolving media guidelines, we expect our computational analysis to show that homosexual occurs in more dehumanizing contexts than other LGBTQ labels, such as gay.
section_name: Operationalizing Dehumanization
paragraphs: In Section SECREF1, we briefly discussed multiple elements of dehumanization that have been identified in social psychology literature. Here we introduce and quantify lexical correlates to operationalize four of these components: negative evaluations of a target group, denial of agency, moral disgust, and use of non-human metaphors (particularly vermin).
section_name: Operationalizing Dehumanization ::: Negative Evaluation of a Target Group
paragraphs: One prominent aspect of dehumanization is extremely negative evaluations of members of a target group BIBREF4. Attribution of negative characteristics to members of a target group in order to exclude that group from “the realm of acceptable norms and values" is specifically the key component of delegitimization, a process of moral exclusion closely related to dehumanization. We hypothesize that this negative evaluation of a target group can be realized by words and phrases whose connotations have extremely low valence, where the valence of a word refers to the dimension of meaning corresponding to positive/negative (or pleasure/displeasure) BIBREF55, BIBREF56. Thus, we propose several valence lexicon-based approaches to measure this component: paragraph-level valence analysis, Connotation Frames, and word embedding neighbor valence. Each of these techniques has different advantages and disadvantages regarding precision and interpretability.
section_name: Operationalizing Dehumanization ::: Negative Evaluation of a Target Group ::: Paragraph-level Valence Analysis
paragraphs: As we mentioned above, one important dimension of affective meaning is valence, which corresponds to an individual's evaluation of an event or concept, ranging from negative/unpleasant to positive/pleasant BIBREF55, BIBREF57. A straightforward lexical approach to quantify negative evaluations of a target group is thus to measure the average valence over all words in corpora containing discussions of the target group. We quantify this by using the valence dimension from the NRC VAD lexicon, which contains real-valued scores ranging from zero to one for valence, arousal and dominance for 20,000 English words, where a score of zero is the lowest valence (most negative emotion) and a score of one is the highest possible valence (most positive emotion) BIBREF56. Words with the highest valence in the NRC VAD Lexicon include love, happy, and happily, while words with the lowest valence include toxic, nightmare, and shit.
In our case study, we qualitatively found that paragraphs were the optimal level of context for analysis; full articles often focused on topics unrelated to LGBTQ communities, even if an LGBTQ label was mentioned somewhere in the article, while single sentences did not provide enough discussion to get a sense of how LGBTQ groups are represented by the newspaper. Thus, we calculate paragraph-level scores by taking the average valence score over all words in the paragraph that appear (or whose lemmas appear) in the NRC VAD lexicon.
section_name: Operationalizing Dehumanization ::: Negative Evaluation of a Target Group ::: Connotation Frames
paragraphs: While paragraph-level valence analysis is quick and simple, it is sometimes too coarse because we aim to understand the sentiment directed towards the target group, not just nearby in the text. For example, suppose the target group is named “B". A sentence such as “A violently attacked B" would likely have extremely negative valence, but the writer may not feel negatively towards the victim, “B".
We address this by using BIBREF22's Connotation Frames Lexicon, which contains rich annotations for 900 English verbs (BIBREF22). Among other things, for each verb, the Connotation Frames Lexicon provides scores (ranging from -0.87 to 0.8) for the writer's perspective towards the verb's subject and object. In the example above for the verb attack, the lexicon lists the writer's perspective towards the subject “A", the attacker, as -0.6 (strongly negative) and the object “B" as 0.23 (weakly positive).
We extract all subject-verb-object tuples containing at least one target group label using the Spacy dependency parser . For each subject and object, we capture the noun and the modifying adjectives, as group labels (such as gay) can often take either nominal or adjectival forms. For each tuple, we use the connotation frame lexicon to determine the writer's perspective either towards the subject if the group label appears in the subject noun phrase, or perspective towards the object if the label appears in the object noun phrase. We then average perspective scores over all tuples.
section_name: Operationalizing Dehumanization ::: Negative Evaluation of a Target Group ::: Word Embedding Neighbor Valence
paragraphs: While a Connotation Frames approach can be more precise than word-counting valence analysis, it limits us to analyzing SVO triples, which excludes a large portion of the available data about the target groups. This reveals a conundrum: broader context can provide valuable insights into the implicit evaluations of a social group, but we also want to directly probe attitudes towards the group itself.
We address this tension by training vector space models to represent the data, in which each unique word in a large corpus is represented by a vector (embedding) in high-dimensional space. The geometry of the resulting vector space captures many semantic relations between words. Furthermore, prior work has shown that vector space models trained on corpora from different time periods can capture semantic change BIBREF58, BIBREF59. For example, diachronic word embeddings reveal that the word gay meant “cheerful" or “dapper" in the early 20th century, but shifted to its current meaning of sexual orientation by the 1970s. Because word embeddings are created from real-world data, they contain real-world biases. For example, BIBREF37 demonstrated that gender stereotypes are deeply ingrained in these systems. Though problematic for the widespread use of these models in computational systems, these revealed biases indicate that word embeddings can actually be used to identify stereotypes about social groups and understand how they change over time BIBREF24.
This technique can similarly be applied to understand how a social group is negatively evaluated within a large text corpus. If the vector corresponding to a social group label is located in the semantic embedding space near negative words or terms with clearly negative evaluations, that group is likely negatively evaluated (and possibly dehumanized) in the text. These vector space models can be used in a diachronic study to capture subtle changes in the connotations of a group label over time, even when the denotational meaning of the label has remained constant. Furthermore, comparing differences in what words are most closely associated with related group labels can reveal meaning variation between these labels, even if they refer to similar populations.
We first preprocess the data by lowercasing, removing numbers, and removing punctuation. We then use the word2vec skip-gram model to create word embeddings BIBREF60. The resulting vector spaces have 100 dimensions, and we specify the following additional parameters to the model: context window of size 10, minimum frequency of 5 for a word to be included, 10 negative samples, 10 iterations, and a sampling rate of $10^{-4}$. We then use cosine similarity to determine a group label's nearest neighbors in the vector space. For our diachronic analysis, we first train word2vec on the entire corpus, and then use the resulting vectors to initialize word2vec models for each year of data in order to encourage coherence and stability across years. After training word2vec, we zero-center and normalize all embeddings to alleviate the hubness problem BIBREF61.
We then identify vectors for group labels by taking the centroid of all morphological forms of the label, weighted by frequency. For example, the vector representation for the label gay is actually the weighted centroid of the words gay and gays. This enables us to simultaneously account for adjectival, singular nominal, and plural nominal forms for each social group label with a single vector. Finally, we estimate the valence for each group label by identifying its 1000 nearest neighbors via cosine similarity (1000 was chosen arbitrarily), and calculating the average valence of all neighbors that appear in the NRC VAD valence lexicon.
We also tried using a group label's vector representation to induce a valence score directly, instead of using nearest neighbors as a proxy, by adapting the regression-based sentiment prediction from BIBREF62 for word embeddings. We found that this approach yielded similar results as analyzing nearest neighbor valence but was difficult to qualitatively interpret. More details for and results from this technique can be found in the Appendix.
section_name: Operationalizing Dehumanization ::: Denial of Agency
paragraphs: Denial of agency refers to the lack of attributing a target group member with the ability to control their own actions or decisions BIBREF16. Automatically detecting the extent to which a writer attributes cognitive abilities to a target group member is an extraordinarily challenging computational task. Fortunately, the same lexicons used to operationalize negative evaluations provide resources for measuring lexical signals of denial of agency.
section_name: Operationalizing Dehumanization ::: Denial of Agency ::: Connotation Frames
paragraphs: As in Section SECREF6, we use Connotation Frames to quantify the amount of agency attributed to a target group. We use BIBREF25's extension of Connotation Frames for agency BIBREF25. Following BIBREF25's interpretation, entities with high agency exert a high degree of control over their own decisions and are active decision-makers, while entities with low agency are more passive BIBREF25. This contrast is particularly apparent in example sentences such as X searched for Y and X waited for Y, where the verb searched gives X high agency and waited gives X low agency BIBREF25. Additionally, BIBREF25's released lexicon for agency indicates that the subjects of verbs such as attack and praise are given high agency, while the subjects of doubts, needs, and owes are given low agency BIBREF25.
This lexicon considers the agency attributed to subjects of nearly 2000 transitive and intransitive verbs. To use this lexicon to quantify denial of agency in our corpus, we extract all sentences' head verbs and their corresponding subjects, where the subject noun phrase contains a target group label. Unlike BIBREF22's real-valued Connotation Frames lexicon for perspective, the agency lexicon only provides binary labels, so we calculate the fraction of subject-verb pairs where a subject containing a group label was given high agency by its head verb.
section_name: Operationalizing Dehumanization ::: Denial of Agency ::: Word Embedding Neighbor Dominance
paragraphs: The NRC VAD lexicon's dominance annotations provide another resource for quantifying dehumanization BIBREF56. As with valence, the NRC VAD lexicon's dominance dimension contains real-valued scores between zero and one for 20,000 English words. However, it is important to note that the dominance lexicon primarily captures power, which is distinct from but closely related to agency. While power refers to one's control over others, agency refers to one's control over their own self. While this lexicon is a proxy, it qualitatively appears to capture signals of denial of agency; the highest dominance words are powerful, leadership, success, and govern, while the lowest dominance words are weak, frail, empty, and penniless. We thus take the same approach as in Section SECREF10 with the same vector space models. However, we now calculate the average dominance score of the 1000 nearest neighbors to each group label vector representation.
As in Section SECREF10, we also induced a dominance score directly from a group label's vector representation by adapting the regression-based sentiment prediction from BIBREF62 for word embeddings. More details and results for this technique can be found in the Appendix.
section_name: Operationalizing Dehumanization ::: Moral Disgust
paragraphs: In order to operationalize the moral disgust component of dehumanization with lexical techniques, we draw inspiration from Moral Foundations theory, which postulates that there are five dimensions of moral intuitions: care, fairness/proportionality, loyalty/ingroup, authority/respect, and sanctity/purity BIBREF63. The negative end of the sanctity/purity dimension corresponds to moral disgust. While we do not directly incorporate Moral Foundations Theory in our framework for dehumanization, we utilize lexicons created by BIBREF64 corresponding to each moral foundation. The dictionary for moral disgust includes over thirty words and stems, including disgust*, sin, pervert, and obscen* (the asterisks indicate that the dictionary includes all words containing the preceding prefix, such as both obscene and obscenity).
We opt for a vector approach instead of counting raw frequencies of moral disgust-related words because words that explicitly invoke moral disgust are very sparse in our news corpus. Furthermore, vectors are capable of capturing associations with the social group label itself, while counts in paragraphs and full news articles would not directly capture such associations. Using the word embeddings from Section SECREF10, we construct a vector to represent the concept of moral disgust, which is the average of the corresponding vectors for all words in the “Moral Disgust" dictionary, weighted by word frequency. This method of creating a vector from the Moral Foundations dictionary resembles that used by BIBREF65. We can then analyze implicit associations between a social group and moral disgust by calculating the cosine similarity between the group label's vector and the Moral Disgust concept vector, where a higher similarity suggests closer associations between the social group and moral disgust. As previously, each group label's vector is the average of its morphological variants' vectors, weighted by frequency.
section_name: Operationalizing Dehumanization ::: Vermin as a Dehumanizing Metaphor
paragraphs: Metaphors comparing humans to vermin have been especially prominent in dehumanizing groups throughout history BIBREF4, BIBREF15. Even if a marginalized social group is not directly equated to vermin in the press, this metaphor may be invoked in more subtle ways, such as through the use of verbs that are also associated with vermin (like scurry as opposed to the more neutral hurry) BIBREF66. While there is some natural language processing work on the complex task of metaphor detection (e.g. BIBREF67, BIBREF67), such systems would not be able to easily quantify such indirect associations.
We thus quantify the metaphorical relationship between a social group and vermin by calculating similarities between these concepts in a distributional semantic vector space, as we did with moral disgust in Section SECREF14. We create a Vermin concept vector by calculating the average of the following vermin words' vectors, weighted by frequency: vermin, rodent(s), rat(s) mice, cockroaches, termite(s), bedbug(s), fleas . We do not include the singular mouse or flea because non-vermin senses of those words were more frequent (computer mouse and flea market, respectively), and word2vec does not account for polysemy. We calculate cosine similarity between each social group label and the Vermin concept vector. A large cosine similarity between a label and the Vermin vector suggests that the social group is closely associated with vermin.
Table TABREF18 provides an overview of the four elements of dehumanization that we study and the lexical techniques used to quantify them.
section_name: Data
paragraphs: The data for our case study spans over thirty years of articles from the New York Times, from January 1986 to December 2015, and was originally collected by BIBREF68 BIBREF68. The articles come from all sections of the newspaper, such as “World", “New York & Region", “Opinion", “Style", and “Sports". Our distributional semantic methods rely on all of the available data in order to obtain the most fine-grained understanding of the relationships between words possible. For the other techniques, we extract paragraphs containing any word from a predetermined list of LGTBQ terms (shown in Table TABREF19).
Each acronym label was matched insensitive to case and punctuation. Some currently prominent LGBTQ terms, such as queer and trans were not included in this study, as other senses of these words were considerably more frequent in earlier years. We further filtered out paragraphs from sections that typically did not pertain to news, such as “Arts", “Theater", and “Movies". While these sections could provide valuable information, we decided to focus on representation of LGBTQ groups in more news-related contexts. Furthermore, due to the original newspaper layout and subsequent digital processing, Arts sections often contained extremely long paragraphs containing numerous listings and schedules. Removing these sections substantially simplified data processing.
A challenging question when analyzing mass media for subjective attitudes is deciding whose perspective we want to capture: an individual reporter, the institution, or society at large? In this case study, we aim to identify the institution's dehumanizing attitudes towards LGBTQ people. We represent the New York Times institution as a combination of the journalists' words in news articles, direct quotes, paraphrases from interviews, and published opinion articles. Therefore, despite our news focus, we include data from “Opinion" sections; while opinion articles are stylistically different from traditional journalistic reporting due to more overt biases and arguments, these articles are important in constructing the institution's perspective. In addition, we consider all text in each relevant paragraph, including quotes and paraphrases, because they are important to a newspaper's framing of an issue, as particular quotes representing specific stances are intentionally included or excluded from any given article BIBREF69. However, this approach is sometimes problematic when the newspaper quotes clearly extreme positions that they are unlikely to agree with, and we discuss this issue throughout our qualitative analysis.
We refer to the remaining subset of the New York Times data after filtering as the LGBTQ corpus. The LGBTQ corpus consists of 93,977 paragraphs and 7.36 million tokens. However, a large increase in reporting on LGBTQ-related issues has led to a highly skewed distribution in the amount of data over years, with 1986 containing the least data (1144 paragraphs and 73,549 tokens) and 2012 containing the most (5924 paragraphs and 465,254 tokens).
For all experiments, we also include results for the terms American and Americans. Our primary motivation for including American(s) is to contrast changes in LGBTQ labels' representation in the New York Times with another social group label that appears in sufficiently different contexts. This ensures that the changes we find in dehumanizing language towards LGBTQ groups do not apply uniformly to all social groups, and are thus not merely an artifact of the publication's overall language change. While a natural “control" variable would be labels such as straight or heterosexual, we found that these terms only occurred within discussions of LGBTQ communities, likely because they name socially unmarked categories. We also considered comparing LGBTQ labels to person/people, but we found that the word embedding-based experiments were particularly sensitive to syntactic forms, so we selected a word that behaves more similarly syntactically to gay and homosexual, particularly by having both nominal and adjectival uses. Nevertheless, it is important to note that American(s) is by no means a neutral control variable. Because of its in-group status for the New York Times (a U.S. institution), we expect our measurements to show that American(s) appears in more humanizing contexts than LGBTQ group labels; however, we do not expect to find substantial changes in the use of American(s) over time.
Figure FIGREF20 shows the counts of group labels for each year in the New York Times from 1986 to 2015. For visualization purposes, only words with a total count greater than 1000 are shown. Notably, the relative frequency of homosexual decreased substantially over time, while gay, lesbian, and bisexual are more frequent in later years. We also see the emergence of the terms lgbt and transgender after 2000. Aggregated counts over all years for each LGBTQ label can be found in the Appendix.
section_name: Results ::: Word Embeddings
paragraphs: Using all of the New York Times data available, we created vector space models for each year using the methods described in Section SECREF10. Table TABREF22 shows the ten nearest neighboring words (by cosine similarity) to our vector representation of all LGBTQ terms, which is the weighted average of the embeddings of the LGBTQ terms from Table TABREF19. For visual convenience, words that occurred fewer then ten times and proper names are not shown. We also filtered out other LGBTQ labels and forms of the word heterosexual from these lists, as these words were commom neighbors for all terms across all years.
Table TABREF22 shows that in 1986, LGBTQ groups were most highly associated with words that often convey a sense of sexual deviancy, including promiscuity, promiscuous, polygamy, bestiality, and pornography. These associations suggest that LGBTQ people were dehumanized to some extent at this time, and their identities were not fully recognized or valued. This shifted by 2000, where we no longer see associations between LGBTQ groups and ideas that evoke moral disgust. Instead, the 2000 vector space shows that LGBTQ people have become more associated with civil rights issues (suggested by interracial, homophobia, and discrimination). The words ordination and ordaining likely appear due to major controversies that arose at this time about whether LGBTQ people should be permitted to be ordained. We also see some indications of self-identification with the term openly. Finally, we see a slight shift towards associations with identity in 2015, with nearby words including nontransgender, closeted, equality, and sexuality. Curiously, the word abortion is a nearby term for all three years. Perhaps this is because opinions towards abortion and LGBTQ rights seem to be divided along similar partisan lines.
Table TABREF23 shows the ten nearest neighboring words to our representations of gay and homosexual (which are weighted averages of their singular/adjectival and plural forms' vectors). Proper names, words appearing less than ten times that year, other LGBTQ terms, and forms of heterosexual are again filtered out for convenience of visualization and interpretation. Table TABREF23 reveals variation in social meaning between gay and homosexual from as early as 1986 despite denotational similarity, and these differences intensify over time. In 1986, gay is associated with terms of discrimination, civil rights and activism, such as homophobia, feminist, suffrage, sexism, and a.c.l.u (American Civil Liberties Union, a nonprofit organization). On the other hand, homosexual is primarily associated with words related to sexual activity (e.g. promiscuity, anal, intercourse, consenting).
In 1986, this pattern may be likely because of discussions of sexual activity in the context of the AIDS epidemic, but we continue to observe the pejoration of the word homosexual over time and its increased use in dehumanizing contexts. In 2000, nearest neighbors of the word homosexual include deviant, immoral, and criminalizing. While gay becomes more associated with issues related to marriage equality and identity in 2015, homosexual becomes extremely associated with moral disgust and illicit activity, with nearest neighbors including bestiality, pedophilia, adultery, infanticide, and abhorrent.
This qualitative analysis of word embedding neighbors reveals significant variation and change in the social meanings associated with LGBTQ group labels, with clear relationships to dehumanizing language. We will now present our quantitative results for measuring each component of dehumanization.
section_name: Results ::: Negative Evaluation Towards Target Group ::: Paragraph-level Valence Analysis ::: Quantitative Results
paragraphs: Figure FIGREF27 shows the average valence for paragraphs containing LGBTQ labels (and American(s) for comparison), where a paragraph's valence is simply the average valence over its words (or lemmas) that appear in the NRC VAD Valence Lexicon. The NRC VAD lexicons actually contain several LGBTQ terms, which all have lower than the average valence score of 0.5: transsexual (0.264), homosexual (0.333), lesbian (0.385), gay (0.388), and bisexual (0.438). These values contrast starkly with more positively-valenced entries in the lexicon, such as heterosexual (0.561), person (0.646), human (0.767), man (0.688), and woman (0.865). These disparities likely reveal biases among the human annotators whose judgments were used to construct the NRC VAD lexicon BIBREF56. While the lexicon may itself be an interesting artifact of dehumanizing attitudes towards LGBTQ people, we removed these terms from the lexicon before calculating average valence over each paragraph in the LGBTQ corpus in order to isolate linguistic signals in the New York Times data from annotation biases. Without this preprocessing step, the temporal trends and relative differences between all LGBTQ terms, gay, and homosexual remain roughly the same, but each of the LGBTQ labels would occur in significantly more negative paragraphs than American.
Figure FIGREF27 shows the paragraph averages over five-year ranges for visualization purposes primarily due to data sparsity in later years for homosexual (there were just 208 paragraphs containing homosexual in 2014, relative to 3669 containing gay in the same year). Analysis of overlapping confidence intervals and Wilcoxon signed-rank tests over the means for each of the thirty years indicates that gay and all LGBTQ terms occur in significantly more positive paragraphs than homosexual ($p < 0.0001$). However, American actually occurs in more negative paragraphs than all LGBTQ terms or gay ($p < 0.05$). A linear regression analysis over all years reveals that all LGBTQ terms, gay, and homosexual all significantly increase in paragraph-level valence over time ($p < 0.0001$). However, when considering just the last 15 years, gay still significantly increases in paragraph-level valence, while homosexual may be trending downward, although this trend does not reach significance in our data ($p = 0.078$).
The paragraph-level valence analysis shown in Figure FIGREF27 suggests that LGBTQ groups have become increasingly positively evaluated over time, and have thus likely been less dehumanized in the New York Times. However, the lack of a statistically significant increase (rather, a slight downward trend) in average valence for paragraphs containing homosexual between 2001 and 2015 suggests that attitudes towards and evaluations of people described as homosexual have not improved in the same way as those described by other LGBTQ labels.
Finally, this measurement does not support our initial hypothesis that LGBTQ groups have been more negatively evaluated than the other social group American(s), but does reveal that the observed trend for LGBTQ labels is not merely an artifact of changing reporting styles, since paragraphs containing American(s) show a very different pattern. Overall, this result demonstrates substantial language change in the New York Times's discussion of LGBTQ people as well as variation in the contexts where different group labels appear, particularly homosexual.
section_name: Results ::: Negative Evaluation Towards Target Group ::: Paragraph-level Valence Analysis ::: Qualitative Analysis
paragraphs: How well does paragraph-level valence analysis capture negative evaluations of a target group, and thus a major component of dehumanization? To facilitate a qualitative evaluation of this technique, we identify several hundred paragraphs with the highest and lowest average valence. It appears that most paragraphs with high valence scores do express positive evaluations of LGBTQ individuals, and those with low scores express negative evaluations.
Table TABREF29 contain examples with extremely high and low valence. We identify several major themes from these results, which are included in the Interpretation column of Table TABREF29. Most paragraphs with high valence scores emphasize equal rights, while some focus on the activities of advocacy organizations. On the other end, numerous paragraphs with extremely low valence tend to focus on violence against LGBTQ people, discussions of disease (especially AIDS), and LGBTQ issues internationally. Other themes that emerged in low-valence paragraphs include reports on (and direct quotes from) public figures who dehumanized people as well as portrayals of LGBTQ people as reckless, irresponsible, and angry.
While this technique accurately captures the valence for many paragraphs, we also identify several shortcomings. Some of the extreme outliers are extremely short paragraphs, including subtitles within articles which are included as paragraphs in the data. Table TABREF30 shows several examples that were mischaracterized by our paragraph-level valence analysis technique. In addition to the short subtitles, there are several paragraphs with highly positive average valence that seem to express more negative evaluations of LGBTQ people. In the third paragraph in Table TABREF30, the valence is skewed by the positive words supported (0.853) and marriage (0.844) even though the paragraph is actually discussing low support for gay marriage. While the fourth paragraph argues that gay couples would be subpar parents relative to straight couples, it uses positive terms such as love and ideal. Furthermore, kinship terms tend to be assigned highly positive values in the NRC VAD Valence Lexicon, including child (0.912), children (0.857), and family (0.968). Similarly, even though the final example describes discrimination based on sexual orientation, the paragraph's average valence is impacted by positive kinship terms such as father (0.812) and mother (0.931) .
Overall, our qualitative analysis of paragraph-level valence scores shows that highly positive valence often accompanies expressions of positive evaluation towards LGBTQ groups, and low valence often accompanies expressions of negative evaluation. However, paragraph-level valence scores are also impacted by specific words cued by various topics; paragraphs about same-sex marriage tend to be more positive because words like marriage, marry, and couple have high valence scores (greater than 0.8), while paragraphs reporting on hate crimes tend to be more negative because they contain more extremely low-valence words related to crime, violence, injury, and death. Furthermore, we have found that this simple lexicon-based method has difficulties in disentangling perspectives within the text. For example, even though there are linguistic signals of dehumanization expressed in articles reporting on anti-LGBTQ violence and homophobic speech, these dehumanizing attitudes are not necessarily from the viewpoint of the journalist or the institution. Nevertheless, there could be an overall dehumanizing effect if the media's discussions of a marginalized social group largely emphasizes events that harm people, such as violent crimes and disease. It is possible that repeated associations between LGBTQ identity labels and such negative contexts contribute to negative evaluations of LGBTQ groups.
section_name: Results ::: Negative Evaluation Towards Target Group ::: Connotation Frames of Perspective ::: Quantitative Results
paragraphs: Figure FIGREF34 shows the writer's average perspective (valence) towards noun phrases containing either any LGBTQ labels, gay(s), homosexual(s), or the comparison group American(s) using the Connotation Frames lexicon BIBREF22. Note that the wide variation, particularly for homosexual, is likely due to sparsity, as limiting the connotation frames analysis to verbs' immediate subject and direct object noun phrase dependents (consisting of only determiners, adjectives, and nouns) greatly reduced the amount of data for each year. For example, there were only 39 triples for homosexual in 2015. We thus show results aggregated over five-year intervals, as in Figure FIGREF27.
As with paragraph-level valence, the writer's perspective towards the label homosexual is significantly more negative than towards gay ($ p < 0.001$). Linear regression indicates that perspectives towards noun phrases named by any LGBTQ term, gay, and American have all significantly increased over time ($p < 0.01$). However, the trends are still quite different, as the slopes for gay and all LGBTQ terms are an order of magnitude greater than American ($m =(1.1\pm 0.39)\times 10^{-4}$ for American, $m=(1.4\pm 0.18)\times 10^{-3}$ for all LGBTQ terms, and $m=(1.1\pm 0.22)\times 10^{-3}$ for gay). Furthermore, the writer's perspective towards noun phrases containing homosexual have significantly decreased over time ($p < 0.0001$).
Overall, connotation frames' perspective scores reveal a similar pattern as the paragraph-level valence analysis, where LGBTQ groups overall appear to be more positively evaluated in the New York Times over time. Unlike gay and the aggregated all LGBTQ terms, the label homosexual undergoes pejoration, as homosexual becomes increasingly used when (implicitly) expressing negative attitudes towards LGBTQ people.
section_name: Results ::: Negative Evaluation Towards Target Group ::: Connotation Frames of Perspective ::: Qualitative Analysis
paragraphs: To qualitatively analyze how well the connotation frames' lexicon capture the negative evaluation of a target group component of dehumanization, we identify subject-verb-object tuples where the verb indicates that the writer has extremely positive or negative perspective towards either the subject or object. The first two paragraphs in Table TABREF36 were identified among those with the most negative writers' perspectives towards phrases containing LGBTQ labels. The first paragraph (within a quote) uses the phrase any homosexual act as the direct object to the verb committed, which has the effect of framing homosexuality as a crime or other illicit activity. By deeming gay candidates unworthy of the priesthood, the speaker clearly negatively evaluates LGBTQ people. On the opposite end, many of the paragraphs labeled by our method as containing extremely positive perspectives towards phrases containing LGBTQ labels do appear to have positive evaluations of these groups. The second and third paragraphs of Table TABREF36 illustrate this, where the gays are viewed positively for having saved a town, and gay rights advocates are praised for their work.
However, we found several instances where paragraphs seem to have been mislabeled, which are shown in Table TABREF37. In the first paragraph of Table TABREF37, our technique identifies gay marriage as the subject of dependent of the negative-perspective verb harmed, but does not account for the preceding text, which actually reveals that the paragraph contradicts the premise that gay marriage causes harm, and thus does not contain overtly negative evaluations of LGBTQ groups (although this particular example reveals the difficulty of operationalizing this component because ProtectMarriage groups strongly oppose same-sex marriage and may itself have negative evaluations of LGBTQ people). The second example similarly shows that this simple method does not adequately account for various forms of negation, as the positive-perspective verb protect is actually negated. The last example in Table TABREF37 presents a complex case, and it is even qualitatively challenging to determine the writer's perspective towards LGBTQ people. Our method identifies gays as the subject of the verb strengthen, even though the subject should be the gerund allowing gays (into the military), and the lexicon's entry for the writer's perspective towards the subject of of strengthen is a highly positive 0.7. However, the object of this verb is the terrorist organization Al Qaeda; our background knowledge suggests that the capacity to strengthen Al Qaeda would reflect negative perspectives. However, this additional context provided by the rest of the paragraph indicates that the writer is being sarcastic and considers the proposition that gays have any impact on strengthening Al Qaeda to be ridiculous. Finally, the writer emphasizes their own stance in opposition to the Missouri congressional candidate by calling upon common stereotypes of gay people being good at dancing and accessorizing.
Measuring the connotation frames' lexicon perspective scores over verbs' subjects and direct objects cannot leverage as much context or data as measuring valence over paragraphs using the NRC VAD lexicon labeled for 20,000 words. However, this technique can make more fine-grained distinctions regarding the writer's (and institution's) attitudes directed towards LGBTQ people and is not as dramatically impacted by the emotional valence of the topic discussed. However, with both techniques presented, we have difficulties disentangling the journalist's perspective from those expressed by others and simply reported by the journalist. While removing direct quotations may partially address this issue, we deliberately did not remove text from direct quotes or paraphrases. The journalists and newspaper make intentional decisions about what text to include and exclude from quotations, which could still meaningfully represent their perspectives and values BIBREF69.
section_name: Results ::: Negative Evaluation Towards Target Group ::: Word Embedding Neighbor Valence ::: Quantitative Results
paragraphs: Figure FIGREF40 shows the average valence scores of the 1000 nearest neighbors to the vector representations of gay, homosexual, all LGBTQ terms, and American for each year. Consistent with other results for negative evaluations of a target group, the vector representations for all LGBTQ terms and gay show similar patterns, although the LGBTQ vector more closely resembles the homosexual vector in earlier years since homosexual was more frequently used in all discussions of LGBTQ communities at that time. In contrast to our other techniques to quantify negative evaluations of a target group, this measurement notably shows that the valence of American's neighboring words is significantly greater than any of the LGBTQ group representations' neighbors over all years (Wilcoxon's signed-rank test, $p < 0.0001$), indicating that American is used in more positive contexts than LGBTQ terms. Furthermore, all LGBTQ vectors' neighbors have an average valence below the neutral 0.5. However, the average valence of the aggregated all LGBTQ terms representation's neighbors significantly increases over time ($p < 0.0001$), suggesting some increasing humanization in the language used in discussions of LGBTQ people.
Figure FIGREF40 also reveals dramatic connotational differences between gay and homosexual. As shown by non-overlapping confidence intervals and a Wilcoxon signed-rank test, the average valence for homosexual's neighbors is significantly lower than gay's neighbors over all years ($p < 0.0001$). Gay's average neighbor valence slightly increases over time, but does not reach significance ($p = 0.060$), but homosexual's neighboring words become significantly more negative over time ($p < 0.0001$). Analyzing the valence of the nearest neighbors indicates that homosexual has long been used in more negative (and potentially dehumanizing) contexts than gay, and that these words' meanings have further diverged as the label homosexual has been used in increasingly negative contexts over time.
section_name: Results ::: Negative Evaluation Towards Target Group ::: Word Embedding Neighbor Valence ::: Qualitative Analysis
paragraphs: Compared to the previous methods, one limitation of using word embeddings to quantify negative evaluations of a target group is that the embeddings are not easily interpretable by analyzing a small sample of data. Instead, we assess this technique by identifying LGBTQ terms' nearest neighbors in several outlier years. To facilitate this qualitative analysis, we identify a set of unique nearest neighbors for each LGBTQ label in each outlier year, where a word is a unique nearest neighbor for a given LGBTQ term and year if it is not in that term's top 1000 nearest neighbors in any other year.
Table TABREF42 contains several example paragraphs that illustrate overarching themes for the outlier years 1993, 1999, and 2014. In the year 1999, gay, homosexual and the aggregated representation all LGBTQ terms were all more more closely associated with low-valence words than in almost any other year. We connected this finding to a period of intense reporting in the months following the October 1998 murder of a gay Wyoming college student, Matthew Shepard, which drew national attention to and condemnation of anti-LGBTQ violence. Because many occurrences of LGBTQ group labels appeared in similar contexts as content about this incident, terms directly related to Matthew Shepard's case have closer vector representations to LGBTQ terms in this year. For example, gay and all LGBTQ terms's 1000 nearest neighbors include wyoming in 1998 and 1999 and shepard from the years 1998-2001. Unique nearest neighbors for gay in 1999 include other terms that could be connected to this incident, including imprisoned, slanderous, disgraceful, unethical, and unjustifiable. Not only was Matthew Shepard's tragic murder deeply rooted in the dehumanization of LGBTQ people, but Shepard was further dehumanized in the media's emphasis on the gruesome details of his death BIBREF71. BIBREF71 argue that the media's framing of this case actually further socially stigmatized LGBTQ people.
Surprisingly, measuring a group label's valence by the valence of its nearest word embedding neighbors reveals that the most negative year for gay and all LGBTQ terms since 1999 was 2014, the second most-recent year of data. Unique nearest neighbors for gay in 2014 include harassing, intimidation, and stigma. We identified several major themes in 2014 that co-occurred with LGBTQ group labels and possibly led to this distributional semantic pattern, primarily reporting on anti-LGBTQ laws and attitudes in Uganda and Russia (particularly in light of the 2014 Winter Olympics in Sochi). We also found that the terms athletes and winterolympics appeared in both gay and all LGBTQ terms's nearest 1000 neighbors in 2014 (athletes only appeared as a nearby neighbor for two other years: 1994 and 2013). In addition, the terms Uganda and Ugandan are among all LGBTQ terms and gay's nearest 1000 neighbors in 2014, but no other year.
Unlike in 1999 and 2014, LGBTQ terms in the year 1993 seem to be associated with higher-valence words, especially homosexual, whose nearby words had the highest average valence in 1993 out of all years. Homosexual's unique nearest neighbors in 1993 include the high-valence words morale, pledge, civilian, compromise, force, and readiness. This words appear to be connected with numerous stories in 1993 covering the controversy over whether or not LGBTQ people should be allowed to serve in the military.
section_name: Results ::: Negative Evaluation Towards Target Group ::: Denial of Agency ::: Quantitative Results
paragraphs: Figure FIGREF45 shows the agency of each group label based on its head verb's entry in the Connotation Frames lexicon for agency and the group label's position relative to that verb BIBREF25. As in Figure FIGREF34 for perspective, there is large variance for each year due to data sparsity when using the Connotation Frames lexicon, particularly for homosexual, which is considerably less frequent than gay or other LGBTQ terms in later years. In extracting subject-verb pairs for this analysis, we decided to maximize precision by extracting only the nouns and their immediate adjectival modifiers, which limited the amount of data. For visualization purposes, we thus show average agency over five-year intervals.
Overall, Wilcoxon signed-rank tests on the means for all group labels for each of the 30 years indicates that gay occurs in contexts with significantly higher agency than homosexual ($p < 0.0001$). All four group labels actually significantly decrease in agency over time according to linear regressions over all 30 years ($p < 0.001$), but the slope for homosexual is much greater ( $m=(-7.9\pm 1.3)\times 10^{-3}$ for homosexual, compared to $m=(-3.9\pm .55)\times 10^{-3}$ for gay, and $m=(-1.5\pm .46)\times 10^{-3}$ for all LGBTQ terms). Furthermore in the most recent 15 years, gay and all LGBTQ terms show no significant change ($p = 0.097$ for gay and $p = 0.14$ for all LGBTQ terms), but homosexual still decreases significantly in agency ($p < 0.05$).
Figure FIGREF45 suggests that LGBTQ groups experience greater denial of agency in the New York Times than the institution's in-group identifier American. Furthermore, people described as homosexual experience even more denial of agency than people who are described as gay, illustrating how these two terms differ with respect to dehumanization. Unlike the improving attitudes indicated by our analysis of negative evaluations of a target group, it appears that denial of agency increased over time for all LGBTQ groups. However, the relatively rapid change in agency for homosexual is consistent with our other results of this label's pejoration.
section_name: Results ::: Negative Evaluation Towards Target Group ::: Denial of Agency ::: Qualitative Analysis
paragraphs: We again qualitatively investigate the labels assigned by this technique for a sample of paragraphs. In general, the binary labels for positive and negative agency appear to be reasonably accurate, as shown by the first four example in Table TABREF47. Verbs that attribute high agency to the subject include develop and endorse, suggesting that the LGBTQ-aligned subjects are in control and actively making their own decisions. On the other end, LGBTQ people are given low agency when presented as the subjects of passive verbs such as face and acknowledge.
The Connotation Frames Lexicon labels for agency seem to appropriately describe LGBTQ people's agency especially accurately for low agency; we could not readily identify counterexamples in our sample where LGBTQ people actually were portrayed with high agency but were labeled with low agency. However, we found several mischaracterizations in which LGBTQ people were labeled as having a high degree of agency but do not qualitatively seem to be portrayed as particularly agentive or in control of their own actions. This Connotation Frames technique considered the example below to attribute high agency to LGBTQ people because the word homosexual appeared in the subject of the high-agency verb violate; however, homosexual actually modifies relationships, not people themselves. Furthermore, this debate within religion appears to be devoid of input from LGBTQ people and thus does not portray them as particularly agentive.
At the same time, it underscored a stark division in Judaism over the place of homosexuals in society. Orthodox rabbinical groups believe that homosexual relationships violate Jewish law... (1996)
section_name: Results ::: Negative Evaluation Towards Target Group ::: Word Embedding Neighbor Dominance ::: Quantitative Results
paragraphs: Figure FIGREF51 shows the average dominance of each group label's word2vec representation's 1000 nearest neighbors according to the NRC VAD Dominance Lexicon BIBREF56, which we use as an estimate of the dominance associated with each group label. American is clearly significantly associated with greater dominance than gay, homosexual, and all LGBTQ terms (Wilcoxon signed-rank test; $p < 0.0001$), and gay has significantly higher dominance than homosexual ($p < 0.0001$). While the dominance associated with gay and all LGBTQ terms significantly increased over the three decades ($p < 0.01$), dominance significantly decreased for homosexual ($p < 0.0001$). Furthermore, while the increase in average nearest neighbor dominance does not reach significance for gay and all LGBTQ terms in the most recent 15 years ($p = 0.47$ for all LGBTQ terms and $p = 0.066$ for gay), homosexual significantly decreases in that time span ($p < 0.0001$).
Even though dominance may more directly encode power rather than agency, the NRC VAD Dominance Lexicon is a useful resource for operationalizing denial of agency because of the close relationship between these concepts. As with the Connotation Frames' agency measurements shown in Figure FIGREF45, our analysis of the dominance of each label's nearest word2vec neighbors suggest that LGBTQ groups experience greater denial of agency than the New York Times's in-group American. Both techniques show differences between the labels gay and homosexual, where homosexual is consistently associated with lower agency than gay and further decreases over time. However, these two measurements suggest slightly different temporal dynamics for the denial of agency of LGBTQ people in the earliest half of the data; we observe a decrease in agency for LGBTQ groups with Connotation Frames, but a slight increase with nearest neighbor dominance.
section_name: Results ::: Negative Evaluation Towards Target Group ::: Word Embedding Neighbor Dominance ::: Qualitative Analysis
paragraphs: Using the VAD Dominance Lexicon to calculate average dominance of each social group label appears to correspond well to our notion of denial of agency. Because gay's nearest neighbors have a much higher average dominance than homosexual's for most years, we qualitatively evaluate this technique by comparing words that appear as a nearby neighbor for gay and not homosexual for a given year, and vice versa. Words that appear within the 1000 words nearest to gay and not homosexual in the 2015 word2vec space include high-agency words such as brave, freedom, and advocate, which have dominance scores of 0.917, 0.905, and 0.818 respectively according to the VAD lexicon. On the other hand, words nearby to homosexual and not gay in 2015 include low-agency words such as submissive (0.173), unprotected (0.197), and ignorant (0.223). We find similar results comparing gay and homosexual in the word2vec model trained on 1986 data; high-agency words such as liberation (0.857) and mobilization (0.787) are nearby neighbors for gay and not homosexual, while low-agency words such as depressive (0.202) and involuntary (0.231) are nearby neighbors for homosexual and not gay.
We additionally qualitatively investigate the vector space models corresponding to several outlier years. In 1993, homosexual's neighbors have by far the highest average dominance compared to any other year for this label. Similar to what we saw with neighbor valence, this is likely due to a large volume of discourse about the military in debates surrounding the “Don't Ask, Don't Tell" legislation. Some high-dominance words that are unique to homosexual's nearest neighbors in 1993 include directive (0.918), force (0.905), enforce (0.836) and troops (0.804). Also similar to the neighbor valence results, gay's neighbors' in 1999 have the lowest average dominance than any other year. This could also likely be connected to Matthew Shepard's death and the subsequent outrage; unique neighbors for gay in 1999 include disgraceful (0.240), irresponsible (0.241), unethical (0.269), imprisoned (0.302), and slanderous (0.337).
section_name: Results ::: Moral Disgust ::: Quantitative Results
paragraphs: Figure FIGREF55 shows the changing relationships between all LGBTQ terms, gay, homosexual and the dehumanizing concept of Moral Disgust. Because the cosine distance between American and Moral Disgust is significantly greater over all years than any LGBTQ representation (Wilcoxon signed-rank test; $p < 0.0001$), American is the group label least associated with Moral Disgust. Furthermore, the cosine distance between gay and Moral Disgust is significantly greater than the distance between homosexual and Moral Disgust for every year ($p < 0.0001$), indicating that homosexual is more closely associated with Moral Disgust than gay is. Linear regression analyses show that American significantly decreases in cosine distance over the thirty years ($p < 0.0001$), while all LGBTQ terms and gay significantly increase in cosine distance ($p < 0.0001$), indicated weakening associations between LGBTQ people and moral disgust over time. Unlike the aggregated representation for all LGBTQ labels and gay, the increasing cosine distance between homosexual and Moral Disgust does not change significantly over time ($p = 0.54$), but does significantly decrease in the last 15 years ($p < 0.05$).
Overall, these measurements of associations between LGBTQ people and Moral Disgust are largely consistent with the other operationalizations of dehumanization. All LGBTQ group labels are more closely associated with Moral Disgust than the newspaper's in-group term American, but these associations weaken over time which suggests increased humanization. Notably, the term homosexual has always been more associated with Moral Disgust than the denotationally-similar term gay, and homosexual actually becomes more closely associated with this dehumanizing concept in recent years.
section_name: Results ::: Moral Disgust ::: Qualitative Analysis
paragraphs: From investigating word embedding neighbors in Table TABREF23, we saw that homosexual has become increasingly associated with immoral concepts, suggesting that moral disgust is a mechanism by which LGBTQ people are dehumanized. Although rarely implicitly invoked, the connection between LGBTQ people and disgust is supported by several examples from the data, as shown in Table TABREF57. The most dramatic change over time occurs in the late 1980s and early 1990s, when gay, homosexual, and all LGBTQ terms become rapidly less associated with Moral Disgust based on their increasing cosine distance. Because a lot of disease-related words are included in the Moral Disgust component of the Moral Foundations Dictionary BIBREF64, this is likely due to decreasing attention to the AIDS epidemic.
section_name: Results ::: Vermin as a Dehumanizing Metaphor ::: Quantitative Results
paragraphs: Figure FIGREF60 shows the relationships between LGBTQ labels (and American) and the dehumanizing vermin metaphor, quantified as the cosine distance between the labels' word2vec vectors and a Vermin concept vector representation, calculated as the centroid of multiple vermin-related words. As with Moral Disgust in Figure FIGREF55, the in-group term American is significantly further away from Vermin over all years than any of the LGBTQ representations (Wilcoxon signed-rank test; $p < 0.0001$). The cosine distance between gay and Vermin is also significantly greater than the distance between homosexual and Vermin ($p < 0.0001$), indicating that homosexual is more closely associated with the dehumanizing vermin metaphor than gay is. Furthermore, while all LGBTQ terms and gay significantly increase in cosine distance (become less associated with Vermin) over time (linear regression; $p < 0.0001$), homosexual slightly decreases in cosine distance (greater association with Vermin), but this decrease does not reach significance ($p = 0.13$).
Our measurement of the implicit vermin metaphor shows a similar pattern as our measurements for moral disgust, denial of agency, and negative evaluations of the target group. Overall, LGBTQ groups are more implicitly associated with vermin than the comparison social group American, but this association is weakening over time, suggesting increased humanization. Unlike the labels gay and the aggregated representation of all LGBTQ terms, the association between vermin and the label homosexual does not significantly change over time.
section_name: Results ::: Vermin as a Dehumanizing Metaphor ::: Qualitative Analysis
paragraphs: Metaphors comparing humans to vermin have been especially prominent in dehumanizing groups throughout history BIBREF4, BIBREF15. Although no New York Times writers directly compare LGBTQ people to vermin, this metaphor may be invoked in more subtle ways. Furthermore, several paragraphs in the data demonstrate that people explicitly mention the metaphor in order to argue against it. These examples, shown in Table TABREF62, further point to the existence of LGBTQ people-as-vermin metaphors, even if they are not frequently directly invoked.
section_name: Discussion
paragraphs: Our framework for the computational linguistic analysis of dehumanization involves identifying major dimensions of dehumanization from social psychology literature, proposing linguistic correlates for each dimension, and developing robust and interpretable computational methods to quantify these correlates. We applied this framework to study the dehumanization of LGBTQ people in the New York Times over the course of 30 years, from 1986 to 2015. We measured four dimensions of dehumanization: negative evaluations of the target group, denial of agency, moral disgust, and (implicit) invocations of vermin metaphors. Overall, we found increasingly humanizing descriptions of LGBTQ people over time. LGBTQ people have become more associated with positive emotional language, suggesting that negative evaluations of the target group have diminished. LGBTQ people have become more associated with higher-dominance words, suggesting decreasing denial of agency, although this finding was not replicated with the verb-centric “Connotation Frames" measurement. Furthermore, labels for LGBTQ people have moved further away from the concepts of moral disgust and vermin within distributional semantic representations, suggesting that harmful associations between LGBTQ people and these dehumanizing concepts have weakened over time.
Despite these broad trends, we found that the labels gay and homosexual exhibit strikingly different patterns among these four components of dehumanization. Homosexual is associated with more negative (low valence) language than gay, suggesting that there are more negative evaluations of people described as homosexual than gay. The label homosexual is also associated with greater denial of agency, and has stronger connections to moral disgust and vermin than gay does. Furthermore, unlike for LGBTQ labels as a whole, discussions of homosexual people do not seem to have become more humanizing over time. On the contrary, several techniques suggested that language used alongside the label homosexual has actually become more dehumanizing after about 2000. Through its repeated use in these contexts, the use of the word homosexual appears to have emerged as an index of more dehumanizing attitudes towards LGBTQ people than other labels. Despite the denotational similarity between homosexual and gay, these computational techniques can track the stark differences and divergences in social meanings.
In this work, we restricted our analysis to the lexical level for ease of interpretability, and leveraged a diverse array of existing resources, including the NRC VAD lexicon BIBREF56, Connotation Frames lexicons BIBREF22, BIBREF25, and the Moral Foundations Dictionary BIBREF64. For negative evaluations of a target group and denial of agency, we propose multiple different techniques that vary in accuracy and interpretability. While word-counting methods are often inaccurate due to their simplicity, their results can be easily understood, while embedding-based methods tend to suffer the opposite problem. We do not claim to have found the optimal balance of model quality and interpretability in this study, but carefully considering this tradeoff is especially important in work that seeks to characterize complex and sensitive social phenomena such as dehumanization.
section_name: Discussion ::: Limitations and Future Work
paragraphs: As the first attempt to computationally analyze dehumanization, this work has many limitations. While our qualitative analysis demonstrates that the proposed techniques capture some linguistics signals of dehumanization, other aspects of the data erroneously impact our measurements as well. For example, we still cannot entirely disentangle a journalist's perspective from the topics and events in the stories they discuss. These techniques would predict that articles emphasizing anti-LGBTQ violence are dehumanizing; in reality, these stories may express dehumanizing attitudes but those views are not necessarily held by the journalist. An exciting area of future work could involve developing more sophisticated computational methods that can better disambiguate the writer's attitudes, attitudes of people mentioned in the text, and topics and events, while recognizing that each of these could contribute to the overall representation of marginalized groups in the media. Furthermore, the present work uses word2vec since all known affective lexicons are type-level, but contextualized embedding-based methods have great potential for more nuanced analyses of dehumanizing language by leveraging token-level representations BIBREF72, BIBREF73.
Another limitation of this study is the lack of ground truth labels for what content is dehumanizing, which has prevented us from developing supervised methods and quantitative evaluations. While it is challenging to directly solicit opinions from human annotators about such a complex and subjective concept such as dehumanization, annotation strategies such as best-worst scaling can be promising approaches BIBREF74, particularly if they primarily consider opinions from members of the social groups of interest. These ground-truth labels could enable future work to develop models to unify all of these distinct components of dehumanization into a single measurement for a given text.
Our framework could also be greatly expanded to include more insights from dehumanization theory. Beyond the four elements of dehumanization that we discuss in this article, social psychology research has identified many other cognitive processes that contribute to dehumanization, such as psychological distancing, essentialism (the perception that the target group has some essence that makes them categorically and fundamentally different), and denial of subjectivity (neglect of a target group member's personal feelings and experiences). BIBREF75, BIBREF21, BIBREF76, BIBREF3. Scholars also often differentiate between two forms of dehumanization, animalistic (likening humans to animals) and mechanistic (likening humans to inanimate objects or machines), and the linguistic signals underpinning each of these forms could substantially differ from one another BIBREF4.
For simplicity and ease of interpretation, we focus on quantifying lexical cues of dehumanization. However, our understanding of the language of dehumanization would be enriched by considering linguistic features beyond the lexicon. For example, BIBREF77 has shown that definite plurals in English (e.g. the gays) have a unique social and pragmatic effect compared to bare plurals (e.g. gays) by packaging individual entities into one monolith and setting this group apart from the speaker. Indexing a speaker's non-membership in the group being discussed creates social distance between the speaker and group BIBREF77, which makes it likely that using definite plurals to label marginalized social groups plays an important role in dehumanization. Similarly, examining non-lexical signals could help us capture elements of dehumanization that we could not easily identify only through lexical resources. For example, the part of speech of a group label (such as gay as a noun or adjective) could have implications for essentialism, as adjectives simply name attributes to some entity, while nouns explicitly state the entity's category membership and encapsulates other stereotypes associated with that category BIBREF78, BIBREF79, BIBREF76, BIBREF80. We furthermore believe that incorporating discourse-level analysis, such as examining the role of direct quotes in an article and who is being quoted, could provide informative insights that could address some limitations discussed earlier.
We supported our proposed framework with a case study of LGBTQ representation in the New York Times. This case study had a number of limitations as an analysis of the dehumanization of LGBTQ people in the media. We only looked at one data source, which may have its own biases and does not capture the entirety of media discourse about LGBTQ people. Furthermore, we only studied newspaper articles written in (Standard) American English. Future work could assess how well these linguistic cues we measure generalize to other languages and focus on cross-linguistic comparisons of dehumanizing language. Finally, most of our case study involved analysis of an aggregated LGBTQ representation,with a specific focus on the labels gay and homosexual, primarily due to the amount of data available. However, as a result, we still have little understanding about the nuanced differences and changes in the representation of LGBTQ people who don't identify with (or aren't identified as) these labels.
section_name: Discussion ::: Ethical Implications
paragraphs: Our hope in pursuing this work is to draw attention to issues of media representation of marginalized groups and to eventually help make the online world a safer and kinder place. An important part of this mission is to acknowledge the ethical implications and potential issues of our own work.
The methods that we use to quantify dehumanization are themselves biased and potentially harmful. For example, we showed in Section SECREF25 that the lexicon that used to measure valence contains its own anti-LGBTQ biases by considering LGBTQ group labels to be primarily negative/unpleasant. We also trained word2vec models on New York Times data, which presents biases. While it is typically concerning when such models are trained on biased data due to harmful downstream effects BIBREF37, we leverage this data as a useful resource for uncovering human biases and understanding how biases emerge in media discourse about LGBTQ people.
Another concern of this work is that we used computational methods to represent human beings in order to analyze dehumanization on a large scale. However, representing people as sequences of numbers (especially in our vector-based experiments) is inherently dehumanizing them. While we hope that this work will humanize and empower marginalized groups, we acknowledge that it also has this opposite effect.
Other ethical implications of this project appear within our case study. Due to the time scale of our study (beginning in 1986), we did not include LGBTQ labels such as queer or trans, which often had different meanings and were often found in contexts unrelated to LGBTQ identities in earlier years. Furthermore, much of our analysis focused on an aggregated representation for LGBTQ people, which unintentionally minimized the vast diversity of social identities encompassed within this umbrella. We highlighted striking temporal changes and differences between the labels gay and homosexual, which were primarily chosen because these labels were well-represented throughout all years of data. However, emphasizing these labels at the expense of others may contribute to the continued erasure of people who are marginalized even within LGBTQ communities.
section_name: Conclusion
paragraphs: This work is the first known computational linguistic study of dehumanization, and provides contributions to multiple fields. The proposed framework and techniques to quantify salient components of dehumanization can shed light on linguistic variation and change in discourses surrounding marginalized groups. Furthermore, these tools for large-scale analysis have potential to complement smaller-scale psychological studies of dehumanization. Finally, this work has implications for automatically detecting and understanding media bias and abusive language online.
section_name: Acknowledgements
paragraphs: We would like to thank Rob Voigt, Chris Potts, Penny Eckert, David Jurgens, Ceren Budak, Anna-Marie Sprenger, and Ismael Mendoza for their insightful contributions and feedback. We would also like to thank our audience at NWAV-46, who provided helpful feedback on an earlier version of this project.
section_name: Appendix ::: Valence prediction results
paragraphs: In addition to quantifying the negative evaluation of a target group by calculating the average valence score of a group label's vector representation's nearest neighbors according to the NRC VAD lexicon, we also directly induced a valence score from the vector representation itself. We use the zero-centered, normalized, word embeddings created for each year as features in a regression model to directly predict valence (Field et al., 2019). Specifically, we train ridge regression models for each year, where each year's Word2Vec representation for words from the NRC VAD lexicon are features and the lexicon's valence scores are labels. 85% of words from the VAD lexicon were kept as the training set, and the remaining 15% was used as a test set to evaluate performance. We then use this set of regression models to predict a group label's valence from its vector representation.
Figure FIGREF67 shows the directly-induced valence score for each set of group labels from the ridge regression fit to the NRC VAD valence lexicon. Because we trained a different Word2Vec model for each year, we trained a different ridge regression model for each year. Over all thirty years, the Pearson correlation between predicted valence and actual valence on the test set ranged from 0.617 to 0.675, and $R^2$ values ranged from 0.423 to 0.451.The predicted scores show similar trends to the average neighbor valence. Homosexual has the most negative valence for every year, followed by gay and the aggregate over all LGBTQ terms, followed by American with the most positive valence. American is significantly more positive than all LGBTQ labels over all years (Wilcoxon signed-rank test; $p < 0.0001$), and gay is significantly more positive than homosexual for every year (Wilcoxon signed-rank test; $p < 0.0001$). Despite the stability in differences between these terms across experiments, the regression analysis suggests different temporal dynamics. Figure FIGREF67 shows that the predicted valence for gay, homosexual, and all LGBTQ terms all increase over time, but homosexual's predicted valence decreases from 2001 to 2015 ($p < 0.01$). This result is consistent with the other findings in this article in illustrating the pejoration of homosexual in recent years.
section_name: Appendix ::: Agency prediction results
paragraphs: Because we use the NRC VAD dominance lexicon to quantify denial of agency in the same way we use the valence lexicon to quantify negative evaluations of a target group, we again directly induce scores directly from target group label representation. We train another set of ridge regression models with word embedding features for each year, but now we fit the model to the NRC VAD dominance lexicon's scores rather than the valence scores.
Figure FIGREF69 shows the predicted dominance for each group label, which is calculated by fitting ridge regression models to the NRC VAD Dominance Lexicon using the lexicon's words' Word2Vec representations as features for each year. Pearson correlations between predicted and actual dominance scores for all regression models ranged from 0.561 to 0.614 on the test set, and $R^2$ values range from 0.338 to 0.361. Consistent with the average neighbor dominance approach , American has significantly greater dominance than any of the other LGBTQ terms (Wilcoxon signed-rank test; $p < 0.0001$). However a Wilcoxon signed-rank test over each year's means shows that there is no significant difference between the terms gay and homosexual ($p = 0.21$). The predicted dominance of all LGBTQ terms and gay significantly decrease ($p < 0.0001$), but not in the last 15 years ($p = 0.85$ for all LGBTQ terms and $p = 0.51$ for gay). Homosexual does not significantly change in predicted dominance in either the full 30 years ($p = 0.96$) or in the last 15 years ($p = 0.89$).
Why do gay and homosexual show such different patterns in directly-induced predicted dominance from the regression than average dominance based on their neighbors' entries in the NRC VAD lexicon? While the average dominance over the nearest neighbors showed significant differences, they were small in magnitude (often corresponding to differences of less than 0.025 points on a scale from 0 to 1). Perhaps because the word2vec features could only predict just over a third of the variance in dominance scores, they were not able to capture subtle semantic distinctions that could characterize differences in dominance scores

Figures and tables:
caption: Table 1: Overview of linguistic correlates and our operationalizations for four elements of dehumanization.
caption: Table 2: List of LGBTQ terms
caption: Figure 1: Counts for the six most frequent LGBTQ labels in each year of the New York Times data.
caption: Table 3: Nearest words to weighted average of all LGBTQ terms’ vectors in 1986, 2000, and 2015
caption: Table 4: Nearest words to vector representations of gay and homosexual in 1986, 2000, and 2015
caption: Figure 2: Average paragraph-level valence for paragraphs containing gay, homosexual, any LGBTQ term, and American, grouped into 5-year intervals. The shaded bands represent 95% confidence intervals. Paragraph-level scores are calculated as the average valence over all words that appear in the NRC VAD lexicon, which range from 0 to 1. Paragraphs containing LGBTQ labels become more positive over time. Paragraphs containing homosexual are significantly more negative than those containing other LGBTQ labels.
caption: Table 5: Example paragraphs with extremely high and low valence scores, along with an interpretation of the patterns we find. Words with extremely high valence scores (greater than 0.85) appear in blue, and somewhat high-valence words (scores between 0.7 and 0.85) appear in light blue. Words with extremely low valence scores (less than 0.15) appear in red, and somewhat low-valence words (scores between 0.15 and 0.3) appear in pink.
caption: Table 6: Examples mischaracterized by paragraph-level valence analysis. Words with extremely high valence scores (greater than 0.85) appear in blue, and somewhat high-valence words (scores between 0.7 and 0.85) appear in light blue. Words with extremely low valence scores (less than 0.15) appear in red, and somewhat low-valence words (scores between 0.15 and 0.3) appear in pink.
caption: Figure 3: Average connotation frame perspective scores for gay(s), homosexual(s), all LGBTQ terms, and American(s) over five-year intervals with 95% confidence intervals. Scores are calculated for each subject-verb-object triple containing these group labels as the writer’s perspective towards the subject or object (depending on the group label’s position) based on the head verb’s entry in the Connotation Frames lexicon (Rashkin et al., 2016).
caption: Table 7: Examples of paragraphs where the writer’s expresses highly positive and negative perspective towards LGBTQ groups, according to the Connotation Frames lexicon.
caption: Table 8: Examples of paragraphs where the writer’s perspective is mischaracterized by the Connotation Frames lexicon.
caption: Figure 4: Average valence of 1000 nearest words to our representations of gay, homosexual, all LGBTQ terms, and American, averaged over 10 word2vec models trained on New York Times data from each year. The shaded bands represent 95% confidence intervals, and the solid lines are Lowess curves for visualization purposes. Valence scores for each word come from the word’s (or its lemma’s) entry in the NRC VADValence Lexicon (Mohammad, 2018), which range from 0 (most negative) to 1 (most positive).
caption: Table 9: Example paragraphs from years where LGBTQ terms’ nearest neighbors had exceptionally high and low valence.
caption: Figure 5: Agency of gay, homosexual, all LGBTQ terms, and American using the Connotation Frames lexicon for agency (Sap et al., 2017) over all subject-verb-object (SVO) tuples containing at least one group label. An SVO tuple received a score of 1 if the label appears in a positive agency position relative to its head verb and 0 if it does not. The translucent bands represent 95% confidence intervals, and agency is calculated over five-year intervals.
caption: Table 10: Examples where the writer attributes high and low agency towards LGBTQ groups, according to the Connotation Frames lexicon for agency.
caption: Figure 6: Average dominance of 1000 nearest words to our representations of gay, homosexual, all LGBTQ terms, and American, averaged over 10 word2vec models trained on New York Times data for each year. Dominance scores for each word come from the word’s (or its lemma’s) entry in the NRC VAD Dominance Lexicon (Mohammad, 2018), which range from 0 (least dominance) to 1 (most dominance). The shaded bands represent 95% confidence intervals and the solid lines are Lowess curves for visualization purposes.
caption: Figure 7: Cosine distance between our representations of gay, homosexual, all LGBTQ terms, and American and the vector representation of theMoral Disgust concept, averaged over 10 word2vec models trained on New York Times data for each year. Increases in cosine distance indicate decreases in Moral Disgust; possible values range from 0 (most closely associated with Moral Disgust) to 1 (least associated with Moral Disgust). Shaded bands represent 95% confidence intervals and the solid lines are Lowess curves for visualization purposes.
caption: Table 11: Example paragraphs from the data that illustrate the connection between LGBTQ people and moral disgust. The bold words appear in the Moral Foundations Dictionary’s lexicon corresponding to moral disgust (Graham et al., 2009).
caption: Figure 8: Cosine distance between our representations of gay, homosexual, all LGBTQ terms, and American and the vector representation of the Vermin concept, averaged over 10 word2vec models trained on New York Times data for each year. Possible values for cosine distance range from 0 (most closely associated with Vermin) to 1 (least associated with Vermin). Shaded bands represent 95% confidence intervals, and the solid lines are Lowess curves for visualization purposes.
caption: Table 12: The only three paragraphs from the New York Times data that explicitly form a connection between LGBTQ people and vermin.
caption: Table 13: Overall counts for all LGBTQ terms in the New York Times from 1986-2015. Each label includes its morphological and orthographic variants.
caption: Figure 9: Predicted valence of our representations of gay, homosexual,all LGBTQ terms, and American directly induced by fitting ridge regression models to the NRC VAD Valence Lexicon with the lexicon’s words’Word2Vec vectors as features for each year. Results are averaged over 10Word2Vecmodels trained over each year’s data. Shaded bands represent 95% confidence intervals and the solid lines are Lowess curves as visual aids. Higher scores represent more positive valence.
caption: Figure 10: Predicted dominance of our representations of gay, homosexual, all LGBTQ terms, and American directly induced by fitting ridge regression models to the NRC VAD Dominance Lexicon with the lexicon’s words’ Word2Vec representations as features for each year. Results are averaged over 10 Word2Vec models trained over each year, shaded bands represent 95% confidence intervals and the solid lines are Lowess curves for visualization purposes. Higher scores represent greater predicted dominance.

Questions and answers:
question: Do they model semantics 
answers: answer: unanswerable: no
yes_no: yes
answer: unanswerable: no
yes_no: yes
question: How do they identify discussions of LGBTQ people in the New York Times?
answers: answer: unanswerable: no
extractive_spans: act paragraphs containing any word from a predetermined list of LGTBQ terms 
question: Do they analyze specific derogatory words?
answers: answer: unanswerable: no
yes_no: yes
answer: unanswerable: no
yes_no: yes