From Dogwhistles to Bullhorns: Unveiling Coded Rhetoric with Language Models
Year: 2023

Abstract:
Warning: content in this paper may be upsetting or offensive to some readers. Dogwhistles are coded expressions that simultaneously convey one meaning to a broad audience and a second one, often hateful or provocative, to a narrow in-group; they are deployed to evade both political repercussions and algorithmic content moderation. For example, in the sentence “we need to end the cosmopolitan experiment,” the word “cosmopolitan” likely means “worldly” to many, but secretly means “Jewish” to a select few. We present the first large-scale computational investigation of dogwhistles. We develop a typology of dogwhistles, curate the largest-to-date glossary of over 300 dogwhistles with rich contextual information and examples, and analyze their usage in historical U.S. politicians’ speeches. We then assess whether a large language model (GPT3) can identify dogwhistles and their meanings, and find that GPT-3’s performance varies widely across types of dogwhistles and targeted groups. Finally, we show that harmful content containing dogwhistles avoids toxicity detection, highlighting online risks of such coded language. This work sheds light on the theoretical and applied importance of dogwhistles in both NLP and computational social science, and provides resources for future research in modeling dogwhistles and mitigating their online harms.

heading: 1 Introduction
text: The cosmopolitan elite look down on the com-
mon affections that once bound this nation to-
gether: things like place and national feeling and
religious faith. . . The cosmopolitan agenda has
driven both Left and Right. . . It’s time we ended
the cosmopolitan experiment and recovered the
promise of the republic.
–Josh Hawley (R-MO), 2019 ∗Work done while interning at the Allen Institute for AI.
Figure 1: Schematic of how dogwhistles work, based on Henderson and McCready (2018) with the example of cosmopolitan. First, a speaker simultaneously communicates the dogwhistle message and their persona (identity). The in-group recovers both the message content and speaker persona, enabling them to arrive at the coded meaning (e.g. Jewish). The out-group only recognizes the message’s content and thus interprets it literally. This literal meaning also provides the speaker with plausible deniability; if confronted, the speaker can claim that they solely intended the literal meaning.
We have got this tailspin of culture, in our
inner cities in particular, of men not working and
just generations of men not even thinking about
working or learning to value the culture of work.
–Paul Ryan (R-WI), 2014
Cosmopolitan and inner city are examples of dogwhistles, expressions that “send one message to an out-group and a second (often taboo, controversial, or inflammatory) message to an in-group” (Henderson and McCready, 2018). Many listeners would believe that Hawley is simply criticizing well-traveled or worldly people, but others recognize it as an attack on the Jewish people. Similarly, many assume that Ryan is discussing issues within a geographic location, but others hear a pernicious stereotype of Black men as lazy. Crucially, Hawley and Ryan can avoid alienating the out-group by maintaining plausible deniability: they never explicitly say “Jewish” or “Black”, so they can reject accusations of racism (Haney-López, 2014).
Because dogwhistles can bolster support for par-
15162
ticular policies or politicians among the in-group while avoiding social or political backlash from the out-group, they are a powerful mechanism of political influence (Mendelberg, 2001; Goodin and Saward, 2005). For example, racist dogwhistles such as states’ rights and law and order were part of the post-Civil Rights Republican Southern Strategy to appeal to white Southerners, a historically Democratic bloc (Haney-López, 2014). Despite polarization and technology that enables message targeting to different audiences, dogwhistles are still widely used by politicians (Haney-López, 2014; Tilley et al., 2020) and civilians in online conversations (Bhat and Klein, 2020; Åkerlund, 2021).
Beyond political science, research on dogwhistles is urgent and essential for NLP, but they remain a challenge to study. Dogwhistles are actively and intentionally deployed to evade automated content moderation, especially hate speech detection systems (Magu et al., 2017). They may also have harmful unseen impacts in other NLP systems by infiltrating data used for pretraining language models. However, researchers face many difficulties. First, unless they are a part of the in-group, researchers may be completely unaware of a dogwhistle’s existence. Second, dogwhistles’ meanings cannot be determined by form alone, unlike most overt hateful or toxic language. Rather, their interpretation relies on complex interplay of different factors (context, personae, content, audience identities, etc.; Khoo, 2017; Henderson and McCready, 2018, 2019; Lee and Kosse, 2020), as illustrated in Figure 1. Third, since their power is derived from the differences between in-group and out-group interpretations, dogwhistles continuously evolve in order to avoid being noticed by the out-group.
We establish foundations for large-scale computational study of dogwhistles by developing theory, providing resources, and empirically analyzing dogwhistles in several NLP systems. Prior work largely focuses on underlying mechanisms or political effects of dogwhistle communication (Albertson, 2015; Henderson and McCready, 2018) and typically considers a very small number of dogwhistles (often just one). To aid larger-scale efforts, we first create a new taxonomy that highlights both the systematicity and wide variation in kinds of dogwhistles (§2.1). This taxonomy characterizes dogwhistles based on their covert meanings, style and register, and the personae signaled by their users. We then compile a glossary of 340 dogwhis-
tles, each of which is labeled with our taxonomy, rich contextual information, explanations, and realworld examples with source links (§2.2-2.3). As this glossary is the first of its kind, we highlight its value with a case study of racial dogwhistles in historical U.S. Congressional Speeches (§3).
We then apply our taxonomy and glossary to investigate how dogwhistles interact with existing NLP systems (§4). Specifically, we evaluate the ability of large language models (i.e. GPT-3) to retrieve potential dogwhistles and identify their covert meanings. We find that GPT-3 has a limited capacity to recognize dogwhistles, and performance varies widely based on taxonomic features and prompt constructions; for example, GPT-3 is much worse at recognizing transphobic dogwhistles than racist ones. Finally, we show that hateful messages with standard group labels (e.g. Jewish) replaced with dogwhistles (e.g. cosmopolitan) are consistently rated as far less toxic by a commercially deployed toxicity detection system (Perspective API), and such vulnerabilities can exacerbate online harms against marginalized groups (§5).
This work highlights the significance of dogwhistles for NLP and computational social science, and offers resources for further research in recognizing dogwhistles and reducing their harmful impacts. Our glossary, code, results, GPT3 outputs, and a form for adding new dogwhistles to our glossary are all available at: https: //dogwhistles.allen.ai.

heading: 2 Curating a dogwhistle glossary

heading: 2.1 Taxonomy
text: Based on prior work and our own investigations, we craft a new taxonomy (Figure 2). We categorize dogwhistles by register, type, and persona.
Register We label all dogwhistles as either part of a formal/offline or informal/online register. Formal/offline dogwhistles originated in offline contexts or are likely to appear in statements by mainstream political elites (e.g. family values). The informal/online register includes dogwhistles that originated on the internet and are unlikely to be used in political speech (e.g. cuckservative).
Type I Henderson and McCready (2018) distinguish dogwhistles into two types: Type I dogwhistles covertly signal the speaker’s persona but do not alter the implicatures of the message itself, while Type II dogwhistles additionally alter the
message’s implied meaning. We extend this typology to highlight the wide variety of dogwhistles, which has important consequences for building a theory of dogwhistles as well as future computational modeling. We identify three subcategories of “only persona-signaling” (Type I) dogwhistles: symbols (including emojis, abbreviations, and imagery), self-referential terms for members of the in-group, and dogwhistles that require specialized knowledge from a shared in-group culture.
Type II Dogwhistles with an “added message meaning” (Type II) tend to fall into two subcategories: they name a concept or serve as a substitute for a target group label. We further divide concepts into policies (titles for initiatives with covert implications, such as law and order), values that the in-group purports to uphold, expressions whose covert meanings are grounded in in-group humor, and other concepts, which are often coded names for entities that are not group labels (e.g. the New World Order conspiracy theory is antisemitic but does not name or describe Jewish people).
Dogwhistles serve as target group labels in three ways. Many are stereotype-based, whose interpretations rely on pre-existing associations between the dogwhistle and target group; we separate these into stereotype-based target group labels, which directly name the target group (e.g. cosmopolitan), while stereotype-based descriptors are less direct but still refer to the target group (e.g. innercity). Others have an arbitrary or phonetic relationship to the group label; these are commonly used to evade content moderation, such as “Operation Google” terms invented by white supremacists on 4chan to replace various slurs (Magu et al., 2017; Bhat and Klein, 2020). The final subcategory, Bogeyman, includes names of people or
institutions taken to represent the target group (e.g. George Soros↔Jewish, or Willie Horton↔Black).
Persona Persona refers to the in-group identity signalled by the dogwhistle. Figure 2 lists some personae, but this is an open class with many potential in-groups. There is considerable overlap in membership of listed in-groups (e.g. white supremacists are often antisemitic), so we label persona based directly on explanations from sources referenced in our glossary (as described in 2.2). Drawing upon third-wave sociolinguistics, personae are not static labels or stereotypes; rather, people actively construct and communicate personae through linguistic resources, such as dogwhistles (Eckert, 2008).

heading: 2.2 Gathering dogwhistles
text: We draw from academic literature, media coverage, blogs, and community-sourced wikis about dogwhistles, implicit appeals, and coded language. Since academic literature tends to focus on a small set of examples, we expanded our search to media coverage that identifies dogwhistles in recent political campaigns and speeches (e.g. Burack, 2020) or attempts to expose code words in hateful online communities (e.g. Caffier, 2017). During our search, we found several community-sourced wikis that provided numerous examples of dogwhistles, particularly the RationalWiki “Alt-right glossary”, “TERF glossary”, and “Code word” pages.1

heading: 2.3 Glossary contents
text: Our glossary contains 340 English-language dogwhistles and over 1,000 surface forms (morphological variants and closely-related terms), mostly from the U.S. context. Each dogwhistle is labeled
1rationalwiki.org/wiki/{Alt-right_ glossary,TERF_glossary,Code_word}
Dogwhistle Sex-based rights
In-group meaning
Trans people threaten cis women’s rights
Persona Transphobic
Type Concept: Value
Register Formal
Explanation Many anti-transgender people [claim that] women’s “sex-based rights” are somehow being threatened, removed, weakened, eroded, or erased by transgender rights. . . “Sex-based rights”, by the plain English meaning of those words, cannot exist in a country that has equality law. . . it’s mostly a dog-whistle: a rallying slogan much like “family values” for religious conservatives, which sounds wholesome but is a deniable and slippery code-word for a whole raft of unpleasant bigotry.
Source Medium post by David Allsopp
Example
When so-called leftists like @lloyd_rm demand that we give up our hard-won sex-based rights, they align themselves squarely with men’s rights activists. To both groups, female trauma is white noise, an irrelevance, or else exaggerated or invented.
Context Tweet by J.K. Rowling on June 28, 2020
Table 1: Example glossary entry for the transphobic dogwhistle sex-based rights
with its register, type, and signaled persona, an explanation from a linked source, and at least one example with linguistic, speaker, situational, and temporal context included, as well as a link to the example text. Table 1 shows one glossary entry for the transphobic dogwhistle sex-based rights.
Antisemitic, transphobic, and racist (mostly antiBlack but sometimes generally against people of color) dogwhistles are the most common, with over 70 entries for each persona. The glossary includes dogwhistles with other personae, such as homophobic, anti-Latinx, Islamophobic, anti-vax, and religious. See Table A.1 in the Appendix for glossary statistics across register, type, and persona. Because dogwhistles continuously evolve, we intend for this resource to be a living glossary and invite the public to submit new entries or examples.

heading: 3 Case study: racial dogwhistles in historical U.S. Congressional speeches
text: We showcase the usefulness of our glossary, with a diachronic case study of racial dogwhistles in politicians’ speeches from the U.S. Congressional Record (Gentzkow et al., 2019; Card et al., 2022) to analyze the frequency of speeches containing racist dogwhistles from 1920-2020. For this case study, we simply identify glossary terms based on regular expressions and do not distinguish between covert and literal meanings of the same expressions. We also measure how ideologies of speakers using dogwhistles changed over time using DW-NOMINATE (Poole and Rosenthal, 1985), a scaling procedure
that places politicians on a two dimensional map based on roll call voting records, such that ideologically similar politicians are located near each other (Carroll et al., 2009; Lewis et al., 2023). We consider the first dimension of DW-NOMINATE, which corresponds to a liberal-conservative axis.2
As shown in Figure 3, dogwhistle use began to increase during the Civil Rights Era, following the 1954 Brown vs. Board of Education Supreme Court decision mandating racial integration of public schools. This aligns with qualitative accounts of the Republican Southern Strategy: because explicit racism was no longer acceptable, politicians turned to dogwhistles to make the same appeals implicitly, particularly aiming to gain the support of white voters in the Southern United States (Mendelberg, 2001). Their frequency continued to increase from the 1970s through the 1990s, paralleling HaneyLópez (2014)’s account of dogwhistles during the Nixon, Reagan, Bush Sr., and Clinton presidencies. Since the 1990s, the frequency of racial dogwhistles has fluctuated but remained high. Like Haney-López (2014), we qualitatively observe that the dogwhistles invoked post-9/11 have shifted towards being more Islamophobic and anti-Latinx rather than exclusively anti-Black. We caution that this case study and Figure 3 do not make novel claims; rather, our goal is to show that even a naive application of our glossary illustrates qualitatively well-established historical patterns in U.S. politics.
Figure 4 shows how the average ideologies of speakers who use particular dogwhistles (property rights, thug, welfare reform, hardworking Americans, and Willie Horton) have shifted over time, and reveals interesting insights into the evolution and lifecycle of dogwhistles. Most racial dogwhistles in the U.S. Congressional Speeches have become increasingly associated with more conservative speakers over time. However, the inflection point when speaker ideologies shift varies across dogwhistles, suggesting that they emerged as dogwhistles at different points. For example, property rights became increasingly associated with more conservative speakers since the 1960s, while the average ideology of speakers using welfare reform
2The second dimension captures salient cross-cutting issues, and some argue that this dimension primarily captures race relations (Poole and Rosenthal, 1985). However, the second dimension’s interpretation is less clear as the vast majority of voting variation is along the first dimension, and is often ignored by political scientists (Bateman and Lapinski, 2016). We thus restrict this case study to the first dimension though future work may opt to consider the second dimension as well.
did not change until the 1990s. Willie Horton presents an interesting example. In his 1988 presidential campaign, George Bush ran a television advertisement featuring Willie Horton, a Black man convicted of rape and murder while on prison furlough (Mendelberg, 2001). The ad was so powerful among white voters that it propelled Bush to victory, but shortly afterwards was explicitly called out as racist (Haney-López, 2014). We see this pattern in Figure 4: in 1988, Willie Horton was associated with extremely conservative speakers, but quickly became more liberal, and Willie Horton no longer functioned as a successful dogwhistle.

heading: 4 Recognition of dogwhistles in GPT-3
text: We conduct two experiments to assess if a large language model, GPT-3 (Brown et al., 2020), can recognize dogwhistles. First, we interrogate whether GPT-3 is able to identify covert meanings of dogwhistles from our glossary, an ability that would be instrumental in understanding the breadth of online bigotry. Second, we measure GPT-3’s ability to surface dogwhistles, motivated by the fact that dogwhistles are often intentionally obscured from researchers which makes it impossible to ensure that a manual search is complete or comprehensive. Since GPT-3 is trained on large portions of internet data, it may be able to reveal a more comprehensive and diverse set of dogwhistles than manual efforts. Finally, we present a small manual analysis of in-context dogwhistle recognition using our glossary examples and GPT-4 as a proof of concept
for future work.

heading: 4.1 Identifying covert meanings
text: To gauge which types of in-group language GPT-3 might have seen during training, we examine if it is able to identify dogwhistles’ covert meanings.
Experimental setup We create prompts that vary in (1) which of the five definitions of dogwhistles is provided, if any, and (2) if the prompt includes a secret cue. For each input term, we construct 12 prompts (6 definitions including none, and each with or without a secret cue). Consider the example below, with a definition, secret cue, dogwhistle and GPT-3’s response . Prompts with no secret clue simply exclude the word secretly. We generate responses for each dogwhistle in the glossary and consider multiple forms for some, such as actual emojis as well as descriptions. We thus test 480 variants with 12 prompts for each, leading to 5,760 generations.3
A dogwhistle is the use of coded or suggestive
language in political messaging to garner support
from a particular group without provoking opposi-
tion. For example, “cosmopolitan” secretly means
“Jewish to many anti-Semitic people .
3We use the text-davinci-002 GPT-3 model with max_tokens=256, temperature=0.0 (more deterministic outputs) and num_outputs=1.
Results The first author manually evaluated whether or not each of the 5,760 GPT-3 generations contains the covert meaning for each dogwhistle. 80.3% of dogwhistles had their covert meanings identified in at least one generation. Overall, 56.0% generations contained the correct covert meaning for dogwhistles that are part of the formal/offline register, but just 29.4% for dogwhistles in the informal/online register. We refer readers to Appendix A.2 (Figure A.2) for more details about registerbased variation and examples of dogwhistles for which GPT-3 performed particularly well or poorly.
The specific prompt form strongly impacts GPT3’s ability to generate covert meanings (Table 2). Without a definition or secret cue, covert meanings are identified in just 8.5% of generations. Including both a definition and secret cue improves GPT-3’s performance over 5-fold, with dogwhistles’ covert meanings identified in 54.3% of generations.
We observe wide variation in GPT-3’s ability to identify covert meanings across personae. Among the most represented personae in our glossary (at least 100 generations for each), GPT-3 has the lowest recognition of transphobic dogwhistles, the highest recognition of homophobic and Islamophobic dogwhistles, with antisemitic, white supremacist, and racist dogwhistles in the middle (Appendix Table A.3). There is also variation in performance by dogwhistle type and the specific definition provided; we refer the reader to Appendix A.2 and Figure A.3 for more details.

heading: 4.2 Surfacing dogwhistles
text: In addition to evaluating if GPT-3 can identify dogwhistles’ covert meanings, we assess GPT-3’s ability to surface dogwhistles in text generation.
Experimental setup We construct a series of prompts that begin with one of five definitions of dogwhistles from prior work (Table A.2). The definition is followed by a question or request for examples (see Appendix A.1 for more prompting details). In the following example, the definition is
marked in blue, the request in purple, and GPT-3’s response is highlighted in yellow .
A dogwhistle is the use of coded or suggestive
language in political messaging to garner support
from a particular group without provoking oppo-
sition. What are examples of dogwhistles?
1. “Law and order”
2. “The silent majority”
3. “Family values”
4. “Welfare queens”
5. “Illegal aliens”
Evaluation We use our glossary as a proxy to measure precision and recall of GPT-3’s ability to surface dogwhistles because an exhaustive groundtruth set of dogwhistles does not exist. We calculate recall as the proportion of dogwhistles in our glossary that were also surfaced at least once by GPT-3. For precision, the authors manually inspect candidates appearing in at least 4% of GPT3 text generations for generic, white supremacist, racist, antisemitic, Islamophobic, and transphobic prompt types. Because our glossary is not exhaustive, this method yields conservative estimates (see Appendix A.1 for more evaluation details).
Precision Results We find that GPT-3 does have the ability to surface dogwhistles when prompted to do so, but caution that such results are imperfect and require manual verification. The most common errors involve explicit mentions of groups in stereotypes or conspiracy theories (Jews are behind the 9/11 attacks) or phrases that may accompany dogwhistles but are not dogwhistles themselves (I’m not racist but...). Precision in dogwhistle surfacing varies across prompt types; while the average precision over all six prompt types is 66.8%, scores range from just 50% for transphobic dogwhistle prompts to 91.3% for generic prompts (Figure A.1).
Recall Results GPT-3 surfaced 153 of 340 dogwhistles in our glossary (45%). We observe significant differences by register: GPT-3 surfaced 69.4% of formal/offline dogwhistles but just 12.9% of informal/online dogwhistles. Despite its ability to generate emojis and other symbols, GPT-3 did not surface any symbols or emojis from our glossary except for the antisemitic triple parentheses “((()))”.
Figure 5 shows GPT-3 surfacing recall results by both register and in-group personae. We show results for the five most-frequent personae represented in our glossary. Recall of dogwhistles in the
informal/online register is low across the board. For the formal/offline register, recall is considerably higher although it varies widely across personae. As with precision, GPT-3 has the lowest performance for transphobic dogwhistles, surfacing just 44.8% of formal/offline transphobic dogwhistles. For formal/offline antisemitic dogwhistles, recall is considerably higher but far from perfect at 71.7%. GPT-3 has 80.3% and 83.3% recall of racist and white supremacist dogwhistles, respectively, and full 100% recall of Islamophobic dogwhistles.

heading: 4.3 Identifying dogwhistles in context
text: Our experiments show that LLMs have some ability to identify covert meanings of a given dogwhistle specified in the prompt (Section 4.1) and generate dogwhistles (Section 4.2). We now consider a natural follow-up question: can LLMs correctly identify dogwhistles in real-world texts?
Fully addressing this question would require a larger dataset including a variety of linguistic contexts per dogwhistle, documents containing no dogwhistles, and documents with dogwhistle expressions that do not carry the dogwhistle meaning. We leave such a rigorous analysis for future work. For now, we present a small manual analysis with ChatGPT/GPT-4 (OpenAI, 2023), using several real-world examples from our glossary as a proof of concept for in-context dogwhistle recognition.
We prompt GPT-4 with the instructions below. Appendix Table A.5 shows the exact glossary example and GPT-4 output text for the dogwhistles cosmopolitan (elite) (antisemitic), inner-city (racist),
#IStandWithJKRowling, (transphobic), and did you see Kyle? (white supremacist).
A dogwhistle is the use of coded or suggestive
language in political messaging to garner support
from a particular group without provoking oppo-
sition. Identify the dogwhistle in the following
message: [GLOSSARY EXAMPLE].
Using the examples presented in Section 1, GPT4 correctly identifies the dogwhistles cosmopolitan (elite) and inner city, and generates a correct explanation for their covert meanings. However, the model does not work as well for the other examples. For the example containing #IStandWithJKRowling, GPT-4 correctly identifies that text covertly signals transphobia through support of JK Rowling, but does not select this hashtag as the dogwhistle. On the other hand, GPT-4 correctly identifies the dogwhistle in a tweet from JK Rowling, and correctly relates this symbol to the women’s suffrage movement, but does not capture the appropriation of this symbol to covertly communicate transphobia. Finally, GPT-4 misses both the dogwhistle and the precise covert meaning for did you see Kyle? (“see Kyle” sounds similar to the Nazi slogan “Sieg Heil”); while the model still ultimately identifies covert white supremacy, it generates a false explanation connecting the glossary example to this persona.

heading: 5 Dogwhistles and toxicity detection
text: Beyond evaluating language models’ ability to recognize dogwhistles, we seek to understand how dogwhistles affect the decisions that NLP systems make, and how this has downstream implications for content moderation and online safety. We begin to address this with a study of how dogwhistles are handled by a widely-deployed toxic language detection system, Google/Jigsaw’s Perspective API.4 Perspective API scores a text between 0 and 1 for a range of attributes (e.g. toxicity, identity attack, profanity), representing the estimated probability that a reader would perceive the text to contain that attribute. Perspective API’s models are multilingual BERT-based models distilled into singlelanguage convolutional neural networks for faster inference, and are trained on annotated data from online forums. We refer readers to the Perspective API Model Cards for more details.5
4https://perspectiveapi.com/ 5https://developers.perspectiveapi.
com/s/about-the-api-model-cards
Experimental setup We consider 237 hateful sentence templates from HateCheck (Röttger et al., 2021), a test suite for bias in hate speech detection, that contain placeholders for identity terms (group referents) in either adjectival, singular nominal, or plural nominal forms. We fill filled with a standard group label, a slur, or a dogwhistle in the corresponding grammatical form requested by the template. For this experiment, we consider racist (mostly anti-Black), antisemitic, and transphobic terms, as these personae are the most common in our glossary (see Tables A.7 and A.8 for a sample of sentence templates and group label terms, respectively). We feed our resulting 7,665 sentences to Perspective API to get scores for toxicity, severe toxicity, and identity attack.
Results Hateful sentences are rated as less toxic, less severely toxic, and less identity-attacking when dogwhistles are used instead of standard group labels or slurs (Figure 6). This pattern holds for all three personae (Appendix Figure A.4).
Interestingly, mean toxicity scores for slurs are lower than for standard group labels, especially for antisemitic slurs. We observe relatively wide variation in Perspective API’s ratings depending on the specific choice of slur. For example, sentences containing the N-word are almost always rated as more toxic than the same sentences containing Black or Black people. Lower toxicity ratings for other slurs, such as the highly derogatory antisemitic K-word6 may be because, similar to dogwhistles, Perspective API does not recognize that these terms refer
6https://ajc.org/translatehate/kike
to identity groups. However, deeper analysis of slurs is outside the scope of the current work.

heading: 6 Discussion & Conclusion
text: We lay the groundwork for NLP and computational social science research on dogwhistles by developing a new taxonomy and glossary with rich contextual information and examples. We demonstrate our glossary’s utility in a case study of historical U.S. Congressional speeches, where our quantitative analysis aligns closely with historical accounts. We further use our glossary to show that GPT-3 has some, but limited, ability to retrieve dogwhistles and recognize their covert meanings. Finally, we verify that dogwhistles readily evade PerspectiveAPI’s toxicity detection. We now turn to several implications of this work, highlighting potential future directions across disciplines.
Dogwhistles and toxic language Dogwhistles are closely related to other forms of subtle biases studied in NLP, such as implicit hate speech and symbols (Magu et al., 2017; Magu and Luo, 2018; ElSherief et al., 2018, 2021; Qian et al., 2019; Caselli et al., 2020; Menini et al., 2021; Arviv et al., 2021; Botelho et al., 2021; Wiegand et al., 2021a,b; Hartvigsen et al., 2022), microaggressions (Breitfeller et al., 2019), dehumanization (Mendelsohn et al., 2020), propaganda (Da San Martino et al., 2020), condescension (Pérez-Almendros et al., 2020), and stereotypes (Nangia et al., 2020; Sap et al., 2020; Nadeem et al., 2021).
However, dogwhistles are distinct from toxic language in several important ways. First, although often implicitly abusive, they are not exclusively hateful; for example, wonder-working power covertly signals the speaker’s Evangelical Christian identity (Albertson, 2015). Second, dogwhistles are characterized by dual meanings, wherein different sub-audiences interpret the exact same message differently (Henderson and McCready, 2018). Third, dogwhistles’ true meanings are intentionally hidden from the out-group (Saul, 2018). Nevertheless, because dogwhistles are often deployed specifically to avoid hate speech detection and other content moderation tools, NLP researchers should consider how dogwhistles highlight a vulnerability in extant language technologies, which ultimately puts people’s safety and well-being at risk.
We show that hateful speech using dogwhistles evades toxicity detection, and is one way that NLP systems (unintentionally) perpetuate harms against
marginalized groups. This finding is not surprising, as prior work shows that toxicity detection often fails on subtle language (Han and Tsvetkov, 2020; Hartvigsen et al., 2022), but underscores the need for toxicity and hate speech detection models to be able to flag hateful dogwhistles. One potential approach to improve such models could be to train them to recognize dogwhistles in naturallyoccurring in-group contexts (starting with modeling contextual factors; Zhou et al., 2023). More broadly, content moderation pipelines should take context into account and consider mechanisms to identify when a dogwhistle has potentially negative consequences. Beyond toxicity detection, future work ought to consider the impact of dogwhistles in a broader range of NLP tasks, such as bias mitigation or story generation.
How do LLMs know about dogwhistles? Our findings regarding GPT-3’s ability to surface and identify dogwhistles’ covert meanings are probably driven by the contents of the training data. GPT-3’s training data likely includes right-wing extremist content, as has been shown with its predecessor GPT-2 (Gehman et al., 2020), which may result in high performance for dogwhistles from these in-groups. Or perhaps the model is simply memorizing articles or social media posts that explicitly call out certain expressions as dogwhistles. Future work could evaluate if large language models can learn dogwhistles’ covert meanings from incontext usage alone by experimentally controlling for whether or not these terms are explicitly exposed as dogwhistles in the training data.
Moreover, we find that GPT-3’s performance varies widely across target groups. Transphobic dogwhistles are notably difficult for GPT-3 to surface and identify. Perhaps this is because the model is trained on fewer data from transphobic communities compared to other in-groups considered in this work. Furthermore, transphobic dogwhistles may be less frequent in the training data because many have emerged relatively recently. Another reason may be formatting: transphobic dogwhistles are often emoji-based and appear in social media screen names and profile bios rather than in posts themselves. We hope that future work will investigate the links between language models’ knowledge of dogwhistles and training data.
Potential of LLMs for dogwhistle research Beyond the risks presented by current NLP technologies, we wish to highlight the potential benefits of
using NLP to advance dogwhistle research. Even though LLMs’ performance is likely due to vast training data, and even then, their outputs require manual verification, our experiments with GPT-3 demonstrate that LLMs have some ability to surface dogwhistles and explain their covert meanings. This is particularly valuable as dogwhistles are intentionally hidden from out-group members, and out-group researchers may have no other way to access this information. There is thus a unique opportunity for LLMs to assist dogwhistle research, and political content analysis more broadly.
Bridging large-scale analysis and mathematical models Our work builds foundations for largescale computational analysis of dogwhistles in realworld political discourse. We diverge from prior quantitative dogwhistle research, which focuses on mathematically modeling the process underlying dogwhistle communication using probabilistic, game-theoretic, deep learning, and networkbased approaches on simulation data (Smaldino et al., 2018; Dénigot and Burnett, 2020; Henderson and McCready, 2020; Breitholtz and Cooper, 2021; Smaldino and Turner, 2021; Xu et al., 2021; Hertzberg et al., 2022; van der Does et al., 2022). We are optimistic about future research synthesizing these two strands of work to address many of the challenges presented by dogwhistles. For example, future work could use our resources along with these mathematical models to develop systems that can automatically detect dogwhistle usages, emergence of new dogwhistles, or decline of older terms as dogwhistles due to out-group awareness.
Implications for social science research Understanding dogwhistles at scale has vast implications across disciplines, so we develop resources useful for both NLP and social science researchers. We provide the most comprehensive-to-date glossary of dogwhistles and demonstrate through our case study how this resource can be used to analyze political speeches and other corpora, such as social media posts and newspaper articles. Dogwhistles have mostly been studied using primarily qualitative methods (Moshin, 2018; Åkerlund, 2021) and experiments (Albertson, 2015; Wetts and Willer, 2019; Thompson and Busby, 2021), and we hope that by facilitating quantitative content analysis, our resources can add to dogwhistle researchers’ methodological repertoires.

heading: 7 Limitations
text: This work represents an initial push to bring dogwhistles to the forefront of NLP and computational social science research, and as such, has many limitations. Our glossary is the most comprehensive resource to date (to the best of our knowledge) but aims to document a moving target, as dogwhistles continuously emerge or fall out of use due to outgroup awareness. We aim to make this resource a “living glossary” and encourage others to submit new entries or examples. We further encourage future research to develop models to automatically detect the emergence of new dogwhistles.
Another major limitation in this work is that we identify as out-group members for nearly all dogwhistles in the glossary and have an adversarial relationship with many of the communities studied (e.g. white supremacists). Although our work would ideally be validated by members of the ingroups, they have very little incentive to share this information, as that would damage the dogwhistle’s utility as a tool for covert in-group communication.
This work, like most prior work, is limited in that we operationalize dogwhistles as a static binary; we assume each term either does or does not have a dogwhistle interpretation and is categorically included or excluded from our glossary and analyses. In reality, dogwhistles are far more complicated constructs. For example, Lee and Kosse (2020) characterize dogwhistles along two dimensions: the size of their in-group and the degree to which their usage is conventionalized. Other axes of variation may include the level of out-group awareness, and the social and political risks of backlash to the communicator if the dogwhistle interpretation is exposed. It is even possible that audience members who hear a dogwhistle further recirculate it even if they themselves do not recognize the covert meaning (Saul, 2018). We hope future work will consider multifaceted and continuous measures of “dogwhistleness" that account for such nuances.
Finally, the current work is limited in the scope of dogwhistles considered: they are all in English with the vast majority coming from the U.S. political and cultural contexts. However, dogwhistles are prominent across cultures (Pal et al., 2018; Åkerlund, 2021) and we hope that future work will consider other languages and cultures, especially involving researchers who have high awareness of or expertise in non-U.S political environments.

heading: 8 Ethical Implications
text: We caution readers about several potential ethical risks of this work. First is the risk of readers misusing or misunderstanding our glossary. We emphasize that dogwhistles are extremely contextdependent, and most terms in the glossary have benign literal meanings that may be more common than the covert dogwhistle meanings. For example, many entities from the financial sector have been used as antisemitic dogwhistles (e.g. the Federal Reserve, bankers) but their primary usage has no antisemitic connotations.
Relatedly, some glossary entries include terms that originate from the target group but were appropriated by the dogwhistles’ in-group. Examples include the appropriation of goy (a Yiddish word for non-Jewish people) as an antisemitic in-group signal, and baby mama (originally from African American English) as a racist dogwhistle. As with hate speech detection (Sap et al., 2019), there is a risk of social bias in dogwhistle detection.
As we have discussed throughout this work, dogwhistle researchers face a challenge with no exhaustive ground truth and an unknown search space. We anticipate our glossary being a helpful resource for this reason, but because we also lack such exhaustive ground truth, there are bound to be biases in the representation of dogwhistles in our glossary. The current version of the glossary may exclude groups and thus lead to worse performance in dogwhistle detection, toxic language detection, and other downstream NLP tasks.
Our glossary also includes real-world examples of how each dogwhistle is used. This presents a privacy risk, which we mitigate by prioritizing examples from public figures or examples from anonymous social media accounts whenever possible. We do not release personal information of any speaker who is not a well-known public figure.
Finally, we do not pursue any computational modeling or prediction of dogwhistle usages in this work, but see it as a natural direction for future work. However, we caution researchers to consider dual-use issues in doing so. Many people use coded language in order to avoid censorship from authoritarian regimes (Yang, 2016) and marginalized groups may also use coded language for their own safety (Queen, 2007). When building computational models, we urge researchers to mitigate this dual-use risk as much as possible.

heading: Acknowledgements
text: We thank Ceren Budak, Yulia Tsvetkov, and audiences at Text as Data 2022 (TADA) and New Ways of Analyzing Variation 50 (NWAV) for their helpful feedback on an earlier version of this work. We also thank the anonymous reviewers for their comments and suggestions. J.M. gratefully acknowledges support from the Google PhD Fellowship.

heading: A Appendix
text: A.1 Details for dogwhistle surfacing We create 51 total request formulations that ask for generic examples of dogwhistles (n=17), dogwhistles that target specific social groups (n=25), and dogwhistles that are used by certain personae/ingroups (n=9). For each prompt, we also consider three spelling variations of “dogwhistle”: dogwhistle, dog-whistle, and dog whistle. Exact prompt text can be found in our project repository.
To encourage GPT-3 to generate a list, we conclude all prompts with a newline token followed by “1.”. All prompts were provided to a GPT-3 Instruct model (text-davinci-002) with de-
fault hyperparameters except for max_tokens=256, temperature=0.7, and num_outputs=5 (5 generations per prompt). The resulting texts are strings that take the form of an enumerated list. To aggregate and compare surfaced dogwhistles across each text completion, we post-process by: splitting by newline characters, removing enumeration and other punctuation, converting all outputs to lowercase, lemmatizing each surfaced term with SpaCy, and removing definite articles that precede generated dogwhistles. We then aggregate over all generations to determine how often each dogwhistle is surfaced for each in-group.
In calculating precision of dogwhistle surfacing, we mark each of the 154 candidate terms as true positives if they appear in the glossary. Some surfaced dogwhistles were marked as “correct” if they were closely related to a dogwhistle entry in our glossary, even if the exact term did not appear. Examples include national security, identity politics, the swamp, tax relief, and patriot. However, this is still a conservative estimate because our glossary is not exhaustive. GPT-3 surfaces a number of terms that potentially have dogwhistle usages but were not covered by our glossary, and thus not included in our precision estimates. Examples of these terms include names of Muslim political organizations (Hezbollah, Hamas, Muslim Brotherhood) and Second Amendment rights. Figure A.1 shows variation in precision of dogwhistle surfacing across prompt types (in-groups and generic prompting).
A.2 Details for identifying covert meaning
Variation across registers We identify variation in GPT-3’s ability to identify dogwhistles’ covert meanings based on prompt features, dog-
https://www.merriam-webster.com/words-at-play/dog-whistle-political-meaning https://en.wikipedia.org/wiki/Dog_whistle_(politics)
whistle register, and the interaction between the two. Figure A.2 shows that including the definition in prompts consistently improves GPT-3’s covert meaning identification for both formal and informal dogwhistles. However, including the secret cue has minimal effect for informal dogwhistles, and only leads to substantial improvement for identifying formal dogwhistles’ covert meanings.
Variation across personae There is significant variation in GPT-3’s performance across personae, as can be seen in Table A.3.
Variation across dogwhistle types GPT-3’s performance varies widely across dogwhistle types in our taxonomy (§2.1; Fig. 2). GPT-3 has the lowest performance for humor-based and arbitrary target group label dogwhistles, and the highest performance for representatives (Bogeymen), phoneticbased target group labels, and policies (Table A.4).
anti-liberal 0.292 0.106 transphobic 0.229 0.024
Variation across dogwhistle definitions Only 19.1% of GPT-3 generations include the correct
covert meaning when prompted with no dogwhistle definition. Prompting GPT-3 with any of the five dogwhistle definitions greatly improved performance over no definition provided, but the extent varied, with the Merriam-Webster definition yielding the lowest improvement (43.8%) and Wikipedia yielding the highest (54.3%) (Table A.6). The boost in performance by adding the secret cue depends on the specific definition used; the secret cue has a bigger effect when using the Merriam-Webster and Albertson (2015) definitions (Figure A.3).
Where does GPT-3 perform poorly? Most unrecognized dogwhistles are part of the informal register, especially symbols (e.g. the transphobic spiderweb or cherry emojis). Other unrecognized dogwhistles include “Operation Google” terms (e.g. Skype, Yahoo), more recent terms (e.g. Let’s Go Brandon), and several antisemitic and transphobic dogwhistles whose covert meanings are especially context-dependent (e.g. adult human female, XX, (Wikipedia) early life, fellow white people). Unrecognized formal dogwhistles tend to be extremely subtle and nuanced (e.g. Dred Scott as a conservative anti-abortion dogwhistle) or are highlyconventionalized phrases that may be far more commonly used without the covert implicature (e.g. the antisemitic dogwhistle poisoning the well).
Where does GPT-3 perform well? GPT-3 readily identifies Islamophobic dogwhistles (e.g. radical Islam, Barack Hussein Obama), many antisemitic conspiracy theories (e.g. Khazars, Kalergi Plan, Israel Lobby), some racist dogwhistles whose covert meanings are more widely discussed among the out-group (e.g. inner-city, ghetto, thug, war on drugs, black-on-black crime), some alt-right memes (e.g. Pepe the Frog), and conservative political rhetoric (e.g. balance the budget, judicial activism, job creators).
A.3 Details for toxicity detection Table A.7 shows a sample of the 237 hateful sentence templates used from HateCheck (Röttger et al., 2021). We used the subset of HateCheck templates labeled as hateful and containing a placeholder for an identity term in either adjectival, singular nominal, or plural nominal form. We filled in these placeholders with either a standard group label, dogwhistle, or slur for three personae: antisemitic, racist, and transphobic. Specific terms used can be found in Table A.8.
For all personae and toxicity measures provided by Perspective API, dogwhistles are rated as less toxic than slurs and standard group labels (Figure A.4. Interestingly, except for the N-word, Perspective seems to not recognize the extreme offensiveness of most slurs, and thus toxicity scores for hateful sentences containing slurs are similar to or lower than scores for the same hateful sentences containing standard group labels.
Template So sick of being around [IDENTITY_P] like you. [IDENTITY_P] are disgusting. You [IDENTITY_P] are all dishonest. [IDENTITY_P] are human garbage. Your [IDENTITY_A] lives are worthless. Never trust a single word said by a [IDENTITY_S]. I will cut their [IDENTITY_A] throats. I would love to see those [IDENTITY_P] suffer. All [IDENTITY_P] should fuck off forever. Of course that dumbass was a [IDENTITY_S]. [IDENTITY_P] think they’re special. I hate them. Isn’t it typical of a [IDENTITY_S] to be that stupid?
Table A.7: Sample of template hateful sentences from HateCheck (n=237). We create inputs for Perspective API by replacing identity term placeholders with a standard group label, dogwhistle, or slur in either adjectival (IDENTITY_A), singular nominal (IDENTITY_S), or plural nominal form (IDENTITY_P).
Racist Antisemitic Transphobic African-American Jewish transgender Black Jew trans inner-city cosmopolitan AGP welfare queen cosmopolitan elite autogynephile thug coastal elite AAP dindu globalist autoandrophile gangbanger international banker clownfish ghetto cultural Marxist troon urban Hollywood elite TIF superpredator Khazar TIM n****r k**e t****y c**n h**b s*****e
Table A.8: Racist, antisemitic, and transphobic terms used for toxicity analysis. We substitute identity placeholders in HateCheck templates (Röttger et al., 2021) with these terms to create inputs to Perspective API.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Section 7
3 A2. Did you discuss any potential risks of your work? Section 8
3 A3. Do the abstract and introduction summarize the paper’s main claims? Sections 0 and 1
3 A4. Have you used AI writing assistants when working on this paper? Yes, we used ChatGPT to rephrase a single sentence from the abstract to repeat the same point at the end of the introduction.
B 3 Did you use or create scientific artifacts? Section 2
3 B1. Did you cite the creators of artifacts you used? Congressional records data (Section 3), Glossary sources (Section 2), GPT-3 (Section 4), and Hate speech templates + models (Section 5)
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Our glossary and taxonomy will be open and available to the public (Section 2)
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Yes, in Sections 2, 5, and 8
3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Data released is anonymized with no identifying information
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Section 2 and Section 8
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 2
C 3 Did you run computational experiments? Section 4
C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Not applicable. We did not train our own model, and just used GPT-3 with the OpenAI API
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Yes, section 4 and Appendix A.1 and A.2
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Yes, Sections 3-5, with additional statistics in A.1-A.3 in the appendix
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Yes, Section 4 and Appendix A.1-A.2
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: Finally, we do not pursue any computational modeling or prediction of dogwhistle usages in this work, but see it as a natural direction for future work. However, we caution researchers to consider dual-use issues in doing so. Many people use coded language in order to avoid censorship from authoritarian regimes (Yang, 2016) and marginalized groups may also use coded language for their own safety (Queen, 2007). When building computational models, we urge researchers to mitigate this dual-use risk as much as possible