Modeling Cross-Cultural Pragmatic Inference with Codenames Duet
Year: 2023

Abstract:
Pragmatic reference enables efficient interpersonal communication. Prior work uses simple reference games to test models of pragmatic reasoning, often with unidentified speakers and listeners. In practice, however, speakers’ sociocultural background shapes their pragmatic assumptions. For example, readers of this paper assume NLP refers to “Natural Language Processing,” and not “Neuro-linguistic Programming.” This work introduces the CULTURAL CODES dataset, which operationalizes sociocultural pragmatic inference in a simple word reference game. CULTURAL CODES is based on the multi-turn collaborative two-player game, Codenames Duet. Our dataset consists of 794 games with 7,703 turns, distributed across 153 unique players. Alongside gameplay, we collect information about players’ personalities, values, and demographics. Utilizing theories of communication and pragmatics, we predict each player’s actions via joint modeling of their sociocultural priors and the game context. Our experiments show that accounting for background characteristics significantly improves model performance for tasks related to both clue giving and guessing, indicating that sociocultural priors play a vital role in gameplay decisions.

heading: 1 Introduction
text: “Most of our misunderstandings of other people are not due to any inability to... understand their words... [but that] we so often fail to understand a speaker’s intention.”
— George Armitage Miller (1974)
Certain pragmatic inferences can only be interpreted by individuals with shared backgrounds.
⋆Equal contribution.
For example, what researchers call fun may not be fun for kindergartners. Theories from sociolinguistics, pragmatics, and communication aim to explain how sociocultual background affects interpersonal interaction (Schramm, 1954)— especially since variation occurs across several dimensions: class (Bernstein, 2003; Thomas, 1983), age (Labov, 2011), gender (Eckert and McConnellGinet, 2013), race (Green, 2002), and more.
Rigorously modeling how culture affects pragmatic inference on all axes is understandably challenging. The board game Codenames Duet offers a more restricted setting of turn-based word reference between two players. In each round, THE CLUE GIVER provides a single-word clue;
6550
then THE GUESSER must interpret this clue to select the intended word references on the game board. Ideal inferences come from the players’ common ground—the set of shared beliefs between them (Clark, 1996). In practice, however, a player’s behavior can be idiosyncratic. Each player has knowledge and experience that shape how they interpret clues and make guesses. When players’ backgrounds differ, they may be more likely to misinterpret their partner, as seen in Figure 1.
Inspired by the above, we model the role of sociocultural factors in pragmatic inference with a new task and a series of ablation experiments. First, we describe the CULTURAL CODES dataset of cross-cultural Codenames Duet gameplay, with relevant background information from the players’ demographics, personalities, and political and moral values (§3). Then, we deconstruct each action in a game into a distinct modeling task, taking inspiration from work on cross-cultural pragmatics (§4). Finally, we model each task with/without sociocultural priors, and highlight how player background improves model performance (§6). Our dataset and code is released publicly at https: //github.com/SALT-NLP/codenames

heading: 2 Related Work
text: Cross-Cultural Pragmatics and NLP Pragmatics describes the nonliteral meaning that comes from context and social inference (Purpura, 2004; Thomas, 1983; Hatch et al., 1992). Although some pragmatic categories are universal (e.g., politeness), they can be expressed differently in sociocultural contexts (Taguchi, 2012; Shoshana et al., 1989; Gudykunst and Kim, 1984). When an intended meaning is misinterpreted, this is known as ‘pragmatic failure’ (Thomas, 1983)—often the result of misaligned reference frames or differences in common ground (Stadler, 2012; Crawford et al., 2017). Especially relevant to Codenames are communal lexicons, where common ground manifests in shared community vocabulary (Clark, 1998). Another axis of difference is between low/highcontext cultures (Hofstede, 2001); high-context cultures rely more on shared background. Pragmatics also differs by age (Saryazdi et al., 2022), region, ethnicity, politics, and class (Thomas, 1983), as does theory of mind (Fiske and Cox, 1979; Miller, 1984; Shweder, 1984; Lillard, 1998, 1999).
Outside of work on politeness (Sperlich et al., 2016; Fu et al., 2020), sarcasm (Joshi et al., 2016),
and irony (Karoui et al., 2017), the NLP subfield has not closely considered cross-cultural pragmatics. While there is work on understanding the role of individual culture—for example, learning demographic word vectors (Garimella et al., 2017), identifying deception/depression (Soldner et al., 2019; Loveys et al., 2018), or improving translation (Specia et al., 2016)—modeling cross-cultural pragmatic inference in communication remains a challenge (Hershcovich et al., 2022).
Still, a culture-free pragmatics has played a central role in various NLP tasks, from instructionfollowing (Fried et al., 2018), image captioning (Andreas and Klein, 2016), persona-consistent dialogue (Kim et al., 2020), and summarization (Shen et al., 2019). Much of this work is grounded in Bayesian models of cognition (Griffiths et al., 2008), with models like Bayesian Teaching (Eaves Jr et al., 2016), Naive Utility Calculus (Jara-Ettinger et al., 2016; Jern et al., 2017), and the Rational Speech Acts (RSA) model (Goodman and Frank, 2016; Franke and Jäger, 2016) that integrate language, world knowledge, and context to explain ideal pragmatic reasoning (Noveck, 2018) and grounded reference (Monroe et al., 2017). Instead of modeling socioculture in isolation, we model pragmatic inference, highlighting the role of culture in general interpersonal interaction.
Games as Testbeds for AI A significant body of work focuses on modeling optimal strategy across a wide set of games, including Go (Silver et al., 2016), Chess (Schrittwieser et al., 2020), Poker (Brown and Sandholm, 2017), Diplomacy (, FAIR), D&D (Callison-Burch et al., 2022; Zhou et al., 2022), and Mafia (Ibraheem et al., 2022). Reference games are growing in popularity as testbeds for AI. Tests for artificial pragmatic reasoning often rely on sequential language games, where two players leverage private knowledge either to compete Yao et al. (2021) or coordinate towards a common goal (Potts, 2012; Khani et al., 2018; Hawkins et al., 2015). In this vein, recent works have considered Codenames (Koyyalagunta et al., 2021; Kim et al., 2019; Jaramillo et al., 2020), Connector (Ashok Kumar et al., 2021; Kumar et al., 2021; Kovacs et al., 2022) InfoJigsaw (Khani et al., 2018), and image-based games (Bao et al., 2022). Word association games have been used in psychology to study semantic associations in cultural (Korshuk, 2007) and religious (Tikhonova, 2014) contexts. We utilize games to model the effect of
cross-cultural interactions on pragmatic inference.

heading: 3 The CULTURAL CODES Dataset
text: This study has been approved by the Institutional Review Board (IRB) at the authors’ institution. The purpose of the CULTURAL CODES dataset is to understand how measurable social factors influence dyadic communication in English. By collecting relevant participant background information, we aim to understand how these factors affect linguistic reasoning in a collaborative reference game.
3.1 Codenames Duet Game Overview
Codenames Duet is a collaborative variant of Codenames (Vlaada, 2015) designed for 2 players. The players share a 5 × 5 board of 25 common words. Each player has a distinct (but sometimes partially overlapping) map from words on the board to the following objectives: goal, neutral, and avoid. One player’s map is hidden from the opposing player. The objective of the game is for both players to guess all of their partner’s goal words without guessing any of their partner’s avoid words, as doing so results in an immediate loss.
CULTURAL CODES uses an adapted version of Codenames Duet. With each turn, players alternate between the THE CLUE GIVER and THE GUESSER roles. To begin the turn, THE CLUE GIVER (1) selects one or more associated goal words as targets. Next, THE CLUE GIVER (2) provides a single word clue that relates to the associated target(s). This clue is displayed to THE GUESSER, along with the number of targets she should find. The THE CLUE GIVER also (3) provides a justifying rationale for the clue, describing the relationship between the clue and the target(s). This rationale is not displayed to the partner. Using the clue and the number of target words THE GUESSER (4) guesses targeted words. For each guess, THE GUESSER (5) provides a justifying rationale for the guess. After ending the turn, players alternate roles and continue until all goal words are selected for both sides, or players are eliminated for guessing an avoid word. An overview of roles is illustrated in Figure 1. In §4, we formalize actions (1)-(4) as distinct modeling tasks.

heading: 3.2 Selecting Board Game Words
text: All experiments are run on a strategically filtered subset of the 400 words from Codenames Duet. We select the 100 most abstract and semantically
ambiguous board game words to elicit diverse responses from players. Since the polysemy (Ravin and Leacock, 2000) of a word—the number of related senses it includes—predicts the expected diversity of player responses, we retain only nouns with two or more senses in WordNet (Miller, 1992). Next, we rank polysemous words with Brysbaert et al. (2014)’s concreteness list, selecting the 100 most abstract words according to the mean of their human concreteness scores (finalized list can be found in Appendix A.)
When a player starts a game, we initialize the board with a random subset of 25 words from the filtered 100. For each player, 9 words are randomly mapped to goal, 3 are avoid, and 13 are neutral.

heading: 3.3 Gameplay Data
text: To collect gameplay data, we modified an opensource implementation of Codenames Duet,1 automatically pairing individuals who visited the game website. To source players, we relied on Amazon’s Mechanical Turk. We provided MTurkers with an initial instruction video detailing rules and how to play. To be eligible for the task, Turkers had to get ≥ 80% questions right on a qualifying quiz about Codenames rules and gameplay (Appendix D.1). Average game length was around 17.4 minutes, and MTurkers were paid $2.50 for every game.
Gameplay Attributes For each completed turn, we collected the following game state information from THE CLUE GIVER. Elements marked in gray were hidden from THE GUESSER.
Clue: THE CLUE GIVER’s clue c (e.g. c could be “transport” for the target “car”).
Target Word(s): (Hidden) The target words tn (e.g. “car”) that THE CLUE GIVER intended THE GUESSER to guess.
Target Word(s) Rationale(s): (Hidden) A free-text phrase rn, that describes the relationship between each target word tn and the clue c (e.g. “a car is a mode of transport”).
To summarize, each turn from THE CLUE GIVER results in a clue c and at least one target-rationale pair (tn, rn). On the other hand, we collect the following for THE GUESSER.
Guesses: The guesses gn that THE GUESSER selected for THE CLUE GIVER’s clue c.
1https://github.com/jbowens/ codenamesgreen
Rationale for Each Guess: A free-text phrase rn that relates the guess gn to the clue c
Manual inspection revealed a wide range of rationales. To prevent models from exploiting variance, we instructed GPT-3 to normalize text, removing pronouns and determiners.2 We provided few-shot examples of reformatted rationales and manually inspected normalized outputs. Additional preprocessing information can be found in Appendix B.

heading: 3.4 Sociocultural Priors and Worker Diversity
text: Because we aim to understand the role of sociocultural priors on gameplay, we asked Turkers to complete the standardized surveys below, which cover three broad dimensions: demography, personality, and morality.
Demographic Data (Figure 2) comes from both the annotation UI and in the task’s qualifying questionnaires. In the UI, we asked Turkers for their numeric age, their country of origin, and whether English is their native language. These were required features, so we will denote them as DemoReq. In the qualifier, we included an extended demographic survey with age range, level of education, marital status, and native language (Appendix D.2.1), which we will denote as DemoAll. We find that our annotator demographics are moderately diverse, mirroring Moss et al. (2020). Reported gender across annotators are evenly split: 53% identify as women, 47% identify as men, and 0% as other. Additional details are in Figure 2 and Appendix D.2.1.
Personality (Figure 3) surveys also offer insight into interpersonal interactions. We administer the Big 5 Personality Test (John et al., 1991), measuring a range of personality dimensions on a 5 point
2We use the text-davinci-003 variant from OpenAI. Without GPT-3 normalization, we find that model performance is artificially inflated.
Likert Scale. Features include Openness, Conscientiousness, Extraversion, Agreeableness, and Neuroticism. Definitions are in Appendix D.2.2.
Moral and Political Leaning (Figure 4) also influences decision making processes. Therefore, we asked annotators to self-report their political leaning (liberal, conservative, libertarian, etc). While political leaning captures broad elements of annotator values, Haidt and Graham (2007)’s widely adopted Moral Foundations Theory (MFT) deconstructs values into individual foundations (Care/Harm, Fairness/Cheating, Loyalty/Betrayal, Authority/Subversion, and Sanctity/Degradation). Differences in each foundation can stem from cultural variation (Haidt, 2012). To record annotator leaning on MFT, we administer an abridged version of the Moral Foundations Questionnaire (Graham et al., 2008), which reports each dimension on a 5 point Likert scale (see Appendix D.2.3). Later, we refer to all recorded features as Morality.
Agent Task Description Input Output N
CLUE GIVER (1) Target Words Generate, from the goal pi words, a subset of targets ti. Targets are used to generate a single clue word. {goal} = {BOS, p1, p2, ..., pn,EOS}
{targets} = {BOS, t1, t2, ..., tm,EOS}
7,961
(2) Generating a Clue Generate a one word clue c1 that relates selected target words while avoiding avoid ai and neutral ni words. {avoid, neutral, targets} = {BOS,AVO, a1, a2, . . . , ao, NEU, n1, ..., nn TGT, t1, t2, . . . , tm,EOS}
{clue} = {BOS, ci,EOS}
7,703
(3) Framing a Clue Generate reasoning r that frames a candidate clue word ci w.r.t. a target ti word from the set of targets. {targets, clue, target} = {BOS,TGTS, t1, ..., tn, CLUE, ci,TGT, ti,EOS}
{rationale} = {BOS, r,EOS}
9,519
GUESSER (4) Selecting Guess Words Generate a series of guesses {g1, ..., gm} from the unselected words given a clue ci. {unselected, clue} = {BOS,UN, u1, ..., un, CLUE, ci,EOS}
{guesses} = {BOS, g1, g2, ..., gm,EOS}
7,703
(5) Framing Guesses Generate reasoning r that frames a guess gi (from all guesses) w.r.t. clue ci {guesses, clue, guess} = {BOS,GUESSES, g1, ..., gn, CLUE, ci,GUESS, gi,EOS}
{rationale} = {BOS, r,EOS}
9,382

heading: 3.5 General Dataset Statistics
text: In total, we collect 794 games, with a total of 199 wins and 595 losses.3 Games lasted an average of 9.7 turns, resulting in 7,703 total turns across all games. THE CLUE GIVER targeted an average of 1.24 words per turn. For all collected games, both players provided DemoReq. For 54% of games, both players completed all background surveys; for the remaining 46% of games, at least one player completed all surveys. There were no games with no background information.

heading: 4 Tasks and Modeling
text: To investigate the role of sociocultural factors in pragmatic inference, we propose a set of tasks (Table 1) associated with THE CLUE GIVER (§4.1) and THE GUESSER (§4.2) roles. Concretely, we formalize each action into a conditional generation problem instead of classification, since outputs in
3Some players went inactive before a game was completed. We only collect games that are reasonably long: greater than the 90th percentile of incomplete games, or ≥ 7 turns.
CULTURAL CODES are unconstrained: actions and outputs depend on a changing board state.

heading: 4.1 Modeling THE CLUE GIVER

heading: 4.1.1 Selecting Target Words
text: To start, THE CLUE GIVER identifies target word(s) (1) on a board, which are later used to construct a target clue for the inference. Clues will target salient words, where salience is at least partially determined by the speaker’s cultural background (Wolff and Holmes, 2011). Each set of targets is a subset of the remaining goal words for a given turn (targets ⊆ goal)—we enforce this restriction in our annotation UI.

heading: 4.1.2 Giving a Clue
text: After selecting target words, THE CLUE GIVER must generate a common clue word across the targets (2). Here, THE CLUE GIVER must select a prototypical word across the targets. Because cultural background plays a role in inference (Thomas, 1983), a clue should lie in players’ common ground. Furthermore, the clue word should not lead the
guesser to pick a avoid ni or neutral ei word, since these words can end the game or turn (see §3.1). Therefore, we also include avoid and remaining neutral words in our input.

heading: 4.1.3 Framing the Target Rationales
text: The relationship between the target and clue word plays a critical role in communication—how information is framed with respect to common ground can influence pragmatic success (Crawford et al., 2017). To this end, we model THE CLUE GIVER’s framing of the rationale r for a specific target word t (3), connecting the target t to the clue (c.f., §3.3). Because the framing is constructed in relation to every target word (if multiple are provided), we also encode all targets in the input.

heading: 4.2 Modeling THE GUESSER

heading: 4.2.1 Selected Guesses
text: With the clue word, the THE GUESSER pragmatically infers THE CLUE GIVER’s targets, selecting a sequence of corresponding guesses (4). For this task, we model the sequence of all selected guesses, regardless of correctness. We input all unselected4
4Note that goal/avoid/neutral words differ across players. A goal word for one player can be avoid for another; game states are asymmetric. A clue from THE CLUE GIVER may also target a goal word for the THE GUESSER. As long as one does not guess a avoid word from the opposing player, the
words at the start of each turn for THE GUESSER, along with the provided clue. Like with Target Word Selection, guesses must be a subset of the unselected words (guesses ⊆ unselected); we enforce this during annotation.

heading: 4.2.2 Framing Guess Choice
text: Finally, THE GUESSER also provides framing rationale for their respective guesses, framing clues with respect to their guess (5).

heading: 4.3 Predicting Pragmatic Success
text: So far, our tasks focus on replicating elements of a game turn: the Selected Guesses task (§4.2.1), for example, models both incorrect and correct guesses. However, we also wish to understand if an entire turn sequence results in a successful inference; differences in cross-cultural inferences can result in pragmatic failures (Thomas, 1983). We formulate this as binary classification.
Importantly, we only consider a guess correct if it is intentional. A guess is intentional if and only if the clue giver listed it as a target. If THE GUESSER selects a goal word that is not a target word, we count it as “incorrect.” Like with guess generation, we encode unselected words in the input. Because we are not predicting the guess itself, we include
game continues. See §3.1.
target and rationale from THE CLUE GIVER.

heading: 4.4 Augmenting with Sociocultural Priors
text: We hypothesize that players’ backgrounds influence Codenames gameplay. To this end, we encode background player information for each task. For each dimension described in §3.4, we encode an attribute/answer pair (e.g. age: 22) for each survey question. Then, we prepend all attributes to the encoded strings for each outlined task (§4), using a unique token to delimit attributes for THE CLUE GIVER and THE GUESSER.
insocio = {BOS,GIVER,Clue GiverAttr:A, GUESSER, GuesserAttr:A}+ in
If a player did not respond to a specific attribute, we replace the attribute/answer pair with None. From our sociocultural priors (§3.4), we have 5 ablations: DemoReq, DemoAll, Personality, Morality, and All (concatenating and modeling all ablations). We additionally use no priors as a baseline, using in instead of insocio to test our hypothesis.

heading: 5 Experiment Setup
text: Baselines and Dataset Splits For generation baselines, we use two Seq2Seq models: T5 (Raffel et al., 2020) and BART (Lewis et al., 2020). We optimize the associated language modeling objective across our tasks. Additionally, we experiment with two retrieval baselines for all generation tasks: (1) randomly selecting a generation from the train set and (2) selecting the nearest k-N inputs using pretrained SentenceBERT (Reimers and Gurevych, 2020) or fastText (Bojanowski et al., 2017). Retrieval baselines yield insight into how well offthe-shelf pretrained models capture sociocultural diversity. For classification, we experiment with BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), and XLNet (Yang et al., 2019). Models are base variants, and results are averaged over 5 runs.
For each task, we split clue givers into 80-10-10 train/val/test, since all tasks depend on initial clue giver choices. Importantly, a single clue giver’s
data is not distributed across splits, since clue givers may reuse clues/strategies.
Evaluation Metrics We use a range of metrics to generation tasks. Rationale generation tasks (Target §4.1.3 & Guess §4.2.2) output entire sentences; therefore, we report F-1 scores from ROUGE-(1, 2, L) (Lin, 2004), BLEU (Papineni et al., 2002), and BERTScore (Zhang et al., 2020). For tasks that generate a single or set of words where order does not matter, (Guess Selection §4.2.1; Clue Generation §4.1.2), we report only ROUGE-1 and averaged word vector (fastText) cosine similarity.

heading: 6 Generation Results & Discussion
text: Including cultural priors improves modeling performance across all tasks. For generation problems, T5 generally outperforms BART, and our retrieval baselines lag behind more complex models. Finally, we conduct a qualitative analysis of 20 random samples from each task.
Picking Targets and Guesses From our results (Table 2), we find that selecting guesses is an easier modeling task than picking target words, likely because the input for selecting a guess contains the clue word. Intuitively, selecting target words is more arbitrary than selecting a guess from a clue— especially since our generation task does not enforce guess correctness. Our models reflect this observation. Guess Selection has R-1 scores that are, on average, twice as good as Target Word Selection (Target 34 vs. Guess 66). Furthermore, Guess Selection only requires demographics (DemoReq) to maximize performance, unlike Morality for Target Words. Regardless, both tasks see R-1 increase by ≈ 2 points over no prior baselines. Looking at model outputs between the None and Morality, we observe that models generate words like Well/Grace instead of Death/Poison and vice versa, depending on player background.
Generating a Clue for Targets Moving to our clue generation models, we again find that including sociocultural priors improves model performance (Table 3). Highest R-1 scores (26.54) occur when using Morality as a prior, resulting in a ≈ 2 pt. R-1 and 4 pt. cos-similarity increase when compared to a no prior baseline. We also suspect that selecting target words and generating a hint are interrelated processes: annotators are likely thinking about clues/targets in parallel. Therefore, the same Morality prior results in maximized performance.
While there are themes related to Morality in clue differences for a target word (accident → death vs. lucifer; or fair → equal vs. good), we also find that generations are more specific given sociocultural priors. Consider these generated target → clue pairs ✓ with and ✗ without priors:
• match → ✗ game ✓ cricket • bond → ✗ connection ✓ james • undertaker → ✗ funeral ✓ wrestler Each ✓ example generates a clue that relies on shared cultural background: specifically, knowing that cricket is a sport; that James Bond is a popular character; and that the Undertaker is a wrestler. More details can be found in Appendix C, Table 6.
Clue Generation Errors Across Sociocultural Subtypes Despite jointly modeling cross-cultural information, our performance is far from perfect. Generating successful clues is a core element of Codenames; however, our exact match accuracy on clue generation is only ≈ 26%. To understand errors, we sample 100 generated clues from the Clue Generation Task, and identify errors and differences between (socioculturally) generated clues and the ground truth label.
For 43 samples, we notice that sociocultural priors have no effect on clue generation; the output is identical to the no prior model for the given target word. In these instances, we suspect that our models fail to exploit common ground between a giver/guesser, yielding the same clue as without sociocultural priors. Upon further analysis, we observe that these errors occur frequently (37 samples) when both the clue giver and guesser are white or from North America. Because these demographics are already over-represented in our dataset, we suspect that the model simply ignores over-informative sociocultural priors.
Errors also occur because clues are over (20 instances, e.g. “guevera” instead of “overthrow”) or underspecified (13 instances, e.g. “supernatural” instead of “monster”) compared to the gold clue. In 21/33 of these instances, there is a demographic mismatch between the clue-giver and guesser: the clue-giver and guesser do not share race/country demographics. In contrast to having no effect, we suspect that models mispredict the common ground between guesser/giver. We also judge 18 generation errors to be of similar specificity to the target word—prefixes/suffixes of the gold label—or completely unrelated to the gold clue (6 instances).
Rationalizing Targets and Guesses Beyond generating target words and guesses, we ask models to explain how a target or guess is related to a clue word (e.g. James Bond is a movie character). Again, we find that providing contextual priors improves performance (Table 4). For Target Rationale Generation, models see maximized performance when all priors are included, while Guess Rationale generation sees improvements for Morality.
Like with Clue Generation, we find that improvements in Guess Rationale are from increased specificity (e.g. “actors are cast” → “actors are part of a cast”; “money is center” → “money is the center of everything”). While qualitative differences are clear for Guess Rationale, Target Rationale results are more subtle: improvements stem from minor variations in the type of framing ("a kind of" vs. "a type of") used by the annotator. Additional generations can be found in Appendix C, Table 7.
Classifying Pragmatic Failure We find that classification performance across each architecture is maximized when using sociocultural priors during training (Table 5). While BERT sees reduced improvement (an increase of only +0.02 F-1 over a no-prior baseline), XLNet and RoBERTa see maximum increases of +0.07 and +0.10 respectively. Both XLNet and RoBERTa see these improvements across the same Personality setting. Sociocultural priors improve performance across mirroring and evaluating pragmatic inference.
A Word on Word Vector Baselines Surprisingly, retrieving nearest words using a word vector approach (fastText) performs poorly for both Clue and Guess Generation (Tables 2 & 3). We suspect that pretrained vectors fail to capture sociocultural inference in word association tasks.

heading: 7 Conclusion
text: Language is grounded in rich sociocultural context. To underscore this context, we propose a setting that captures the diversity of pragmatic inference across sociocultural backgrounds. With our Codenames Duet dataset (7K turns across 156 players), we operationalize cross-cultural pragmatic inference. Across our experiments, we detail improvements in mirroring/evaluating inferences when using sociocultural priors. Our work highlights how integrating these priors can align models toward more socially relevant behavior.

heading: 8 Limitations
text: Cross-Cultural Inference Beyond Codenames Our work explores sociocultural pragmatic inference in a very limited setting, using a core vocabulary of just 100 words. Despite this limitation, we find significant diversity in our dataset; furthermore, our models successfully capture these diverse inferences. While a limitation of our work is its focus on a single setting, we expect domains outside of Codenames to see similar variance. Understanding and highlighting miscommunication in dialog—due to culture-dependent misinterpretation—is one such extension. These domains are likely much nosier than Codenames; we urge future work to further investigate them.
Spurious Correlations across Sociocultural Factors Across all tasks but one (Target Rationale Generation §4.1.3), jointly modeling all sociocultural priors does not result in the highest performing model. Because our sociocultural factors already correlate with each other (§3.4), we suspect that modeling all features may be redundant, adding spurious correlations and resulting in overfitting. Improved modeling methodology and careful regularization may address these issues; we leave these experiments for future work.
Bigger Models and Task Specific Modeling Currently, we evaluate small Seq2Seq models due to computational constraints; however, evaluation of 0-shot and few-shot performance on larger language models (e.g. GPT-3) is necessary. Given the changing state of the Codenames board—along with evidence that LLMs struggle with theory-ofmind-esque perspective taking (Sap et al., 2022)— our dataset can serve as a challenging benchmark for sociocultural understanding. However, successfully encoding game state into prompts for LLMs may require experimentation.
Finally, our current task formulation and modeling setup are straightforward: we simply encode all information in-context and do not assume recursive reasoning like in RSA (Goodman and Frank, 2016). Future work can explore these directions.
Human Evaluations Our evaluation is limited to automatic metrics and qualitative analysis. Evaluating cross cultural generation depends on the evaluator’s own culture. Each generation depends on the player’s sociocultural background; finding evaluators who match the player may be prohibitive.

heading: 9 Ethics
text: Broadly, our work models user background to determine the choices they make. While we focus on a fairly harmless setting (Codenames), our operationalization can be used in harmful ways (e.g. tracking and modeling user behavior without consent). Future work that uses sociocultural information should only be applied to settings where there is no foreseeable harm to end-users.
Furthermore, learning sociocultural associations can introduce positive and negative stereotypes; documenting and reducing harmful stereotypes is an important avenue for future work. Finally, we emphasize that our work is not evidence for linguistic determinism: sociocultural variation in language can influence but not determine thought.

heading: Acknowledgements
text: We are thankful to the members of SALT Lab for their helpful feedback on the draft. We are also thankful for the helpful feedback from Jing Huang and Rishi Bommasani. Caleb Ziems is supported by the NSF Graduate Research Fellowship under Grant No. DGE-2039655. This research was supported, in part, by MURI-ONR-N00014-20-S-F003 on Persuasion, Identity, and Morality in SocialCyber Environments, as well as a DARPA grant HR00112290103/HR0011260656.

heading: A Finalized Codenames Word List
text: We sample from the following list of 100 words: luck, grace, soul, fair, life, pass, revolution, change, charge, degree, force, code, genius, compound, time, wake, plot, draft, ghost, play, part, spell, well, point, link, mass, disease, sub, state, alien, space, mine, ray, millionaire, agent, bond, unicorn, figure, war, cycle, boom, sound, trip, centaur, death, club, crash, angel, cold, center, spring, round, date, press, cast, day, row, wind, fighter, embassy, beat, leprechaun, comic, pitch, mount, march, fall, undertaker, green, switch, strike, king, superhero, capital, slip, lead, check, lap, mammoth, air, match, spy, roulette, contract, witch, stock, light, drop, spot, novel, vacuum, cover, scientist, tag, conductor, field, racket, poison, ninja, opera.

heading: B Reformatting Rationales using GPT-3
text: Some annotators wrote verbose rationales (I think fall happens after you slip), while other annotators were more succinct (fall after slip). To prevent models from learning grammar variation across annotators, we normalize our text using GPT-3. We use the following prompt, using hand-written few-shot examples. Some of the examples are unchanged— we include them in the prompt to demonstrate positive examples to the model.
Normalize the text, removing determiners like “the” and “a” at the start of a sentence, along with any pronouns. Correct spelling and grammar mistakes. If possible, the final text should be formatted with the clue first and the target last or the target first and the clue last.
clue: “sub” target: “sandwich” text: “you can make a sub, which is a type of sanwich” output: “sub is a type of sandwich”
clue: "die" target: "cliff" text: "you may die if you fall off a cliff" output: "die if fall off a cliff"
clue: "explosion" target: "boom" text: "it makes sound" output: "explosion makes boom"
clue: "superman" target: "superhero" text: "most famous superhero" output: "superman is most famous superhero"
clue: "night" target: "club" text: "i love night club" output: "night club is a kind of club"
clue: "horn" target: "air" text: "an air horn is a type of horn" output: "air horn is a type of horn"
clue: "ivy" target: "poison" text: "poison ivy is a well known plant" output: "poison ivy is a well known plant"
clue: "month" target: "march" text: "march is a month" output: "march is a month"
clue: "{clue}" target: "{target}" text: "{text}" output: "

heading: C Example Generations
text: Here, we include example generations for a subset of our tasks, illustrating the influence of sociocultural factors on generated Codenames gameplay.
C.1 Clue Generation
Below, we highlight more clues generated with/without sociocultural priors. Note how some of the without generations are euro-centric: space →
nasa, {revolution, king} → war; adding priors creates more specific clues. However, this isn’t always true: target words {pass, check} → leads to poker instead of overtake when conditioned on priors. We suspect that the average player in our pool is not aware of how {pass, check} are associated with poker, resulting in a more generic generation.
C.2 Clue Framing
Additional generations can be found in Table 7. Again, we observe that adding sociocultural priors increases relation specificity.

heading: D Annotation Task Details
text: D.1 Qualification Test
To qualify for the HIT, workers were required to complete a consent form detailing dataset collection and release; and were expected to watch an instructional video outlining game rules.
Then they had to pass the following qualifying test, answering at least 6 out of 7 questions correctly.
1. True or False: "angry dog" is an example of a clue you could give. [Answer: False]
2. True or False: you and your partner have different lists of black (assassin) words. [Answer: True]
3. True or False: it is possible to skip a turn without guessing. [Answer: False]
4. True or False: the tan “down” arrow indicates that you guessed the word wrong, while the tan “up” arrow indicates that your partner guessed it wrong. [Answer: True]
5. Multiple Choice: Which of the following kinds of phrases does not follow from our list of target rationales types? [Answer: (b)]
(a) “a computer has a mouse” (b) “a doctor is smart” (c) “a dog is a kind of animal” (d) “a disease causes people to be sick”
6. Multiple Choice: How many guesses do you get (assuming there are still more words left to guess) [Answer: (d)]
(a) you get three guesses each turn (b) the number of guesses you get is the
same as the number of target words your partner’s clue
(c) as long as you keep picking green words, you can keep guessing, up to the number of target words in your partner’s clue
(d) as long as you keep picking green words, you can keep guessing without any limit, even if you guess more than the number of target words in your partner’s clue
7. Multiple Choice: During the 8th timer token in the video, it looked like my grid froze and I couldn’t make any more guesses. Why did this happen? [Answer: (b)]
(a) I guessed an assassin word (b) I already guessed all my partner’s words
correctly (c) I clicked the “end game” button (d) My partner left the game
D.2 Demographic, Personality, and Moral Questionnaires
Before starting any HITs, workers also had to complete three standardized surveys about their moral foundations, personality, and demographic information. The survey questions and worker statistics are given as follows.
D.2.1 Worker Demographics Questionnaire. Please answer these 8 questions about yourself.
1. With what gender do you identify? {Woman, Man, Transgender, Non-binary / nonconforming, Other}
2. What is your age? {0-17 years old, 18-22 years old, 22-30 years old, 30-45 years old, 45+}
Target Clue Without With Gold
explode boom explode causes boom bomb explodes with a boom explosions make a boom sound
horse unicorn a unicorn is a horse unicorn is a type of horse unicorns are similar to horses racket tennis tennis has racket a racket is used in tennis tennis uses a racket
day month day is month month has many days 30 days in a month
Results. Of the 153 unique players, 124 are from the U.S, 12 are from India, 8 are from Brazil, 3 from the U.K, 2 from Canada, and the rest are single players from the following 7 countries: Indonesia, Costa Rica, France, South Africa, Germany, and Portugal.
D.2.2 Worker Personality Big 5 Personality Questionnaire. Please answer these 10 questions about yourself on the following scale: [-2] Strongly Disagree; [-1] Disagree; [0] Neutral; [1] Agree; [2] Strongly Agree.
1. I see myself as someone who does a thorough job.
D.2.3 Moral Foundations And Political Leaning.
Moral Foundations Theory. Following Haidt and Graham (2007), we use the five-foundation theory of moral reasoning to understand our players’ values and leanings. This theory does not give explicit definitions for the five foundations, but following recent work by Ziems et al. (2022), we can assume the following definition sketches:
1. Care: wanting someone or something to be safe, healthy, and happy. Harm: wanting someone or something to suffer physically, emotionally, socially, intellectually, or spiritually.
2. Fairness: wanting to see individuals or groups treated equally or equitably Cheating: wanting to see unfairness, injustice, bias, exclusion, or discrimination.
3. Loyalty: wanting unity and seeing people keep promises or obligations to an in-group. Betrayal: wanting to see people lie, abandon an in-group, or become isolated and divided.
4. Authority: wanting to respect social roles, duties, privacy, peace, and order. Subversion: wanting to see people disrespect, disobey or cause disorder, challenge the statusquo, and do what they do not have permission to do.
5. Sanctity: wanting people and things to be clean, pure, innocent, and holy. Degradation: wanting people to follow selfish or crude desires and do things that make them or others dirty, corrupt, sick, repulsive, or perverted.
Moral Foundations Questionnaire We use the associated Moral Foundations Questionnaire, which we shortened to 12 questions as follows.
Please answer 12 questions about “right” and “wrong.” The prompts are the same in each case, but the considerations are different.
1. When you decide whether something is right or wrong, to what extent are the following considerations relevant to your thinking? Use the following scale: [0] Not at all relevant (It has nothing to do with my judgments of right and wrong); [1] Not very relevant; [2] Slightly relevant; [3] Somewhat relevant; [4] Very relevant; [5] Extremely relevant (It is one of the most important factors when I judge right and wrong)
(a) Whether or not someone suffered emotionally.
(b) Whether or not some people were treated differently than others.
(c) Whether or not someone’s action showed love for his or her country.
(d) Whether or not someone showed a lack of respect for authority.
(e) Whether or not someone violated standards of purity and decency.
(f) Whether or not someone was good at math.
(g) Whether or not someone cared for someone weak or vulnerable.
(h) Whether or not someone acted unfairly. (i) Whether or not someone did something
to betray his or her group. (j) Whether or not someone conformed to
the traditions of society.
2. Which of the following best describes your political views?
(a) Liberal (b) Moderate Liberal (c) Moderate Conservative (d) Conservative (e) Libertarian
D.3 Instructions for Writing Rationales We explain that rationales should use at least 3 words to describe the connection between the clue and the target. Annotators were encouraged to be creative while trying to use one of the structures below. We imposed these structures for the sake of regularity.
1. MERONYM x has y
(a) a dog has a tail (b) the pacific ocean has water
2. HYPERNYM x is a kind of y
(a) bunkbed is a kind of bed (b) whisper is a kind of communication
3. SYNONYM x means the same thing as y
(a) car means the same thing as automobile (b) sluggish means the same thing as slow
4. ANTONYM x means the opposite of y
(a) civilian means the opposite of soldier (b) fast means the opposite of slow
5. ADJECTIVE x describes y
(a) brave describes a firefighter (b) scary describes a clown
6. AGENT x does y
(a) a star does twinkle police do make an arrest
7. CAUSE x causes y
(a) a bed causes people to sleep (b) an oven causes food to bake (c) a disease causes people to be sick
8. PATIENT x acts on y
(a) a wrench acts on a bolt (b) a doctor acts on a patient
9. LOCATION x has an environment y
(a) a star has an environment firmament

heading: E Training and Hyperparameters
text: For our generation tasks, we perform use 5e-5 as our initial learning rate and perform a hyperparameter search over {1...20} epochs. For classification, we use the same splits and perform a hyperparameter sweep over learning rates ({1e-4, 5e-4, 1e-5, 5e-5, 1e-6, 5e-6}) and epochs ({1...15}). All models were trained on an NVIDIA A100 GPU. Across all experiments, GPU compute time was around 4-5 days.

heading: F Artifact Details
text: We use several models in our paper for their intended retrieval or generation task. Each model has its own license and number of parameters, listed below:
1. T5 (Raffel et al., 2020), 220M parameters, is under the Apache 2.0 License.
2. BART (Lewis et al., 2020), 140M, is under the Apache 2.0 License.
3. fastText (Bojanowski et al., 2017) is under the MIT License.
4. SentenceBERT (Reimers and Gurevych, 2020), 33M variant, is under the Apache 2.0 License.
5. BERT (Devlin et al., 2019) base, 110M, is under the Apache 2.0 License.
6. XLNet (Yang et al., 2019) base, 110M, is under the Apache 2.0 License.
7. RoBERTAa (Liu et al., 2019) base, 123M, is under the Apache License 2.0.
We plan on releasing CULTURAL CODES and corresponding code under Creative Commons Attribution Share Alike 4.0 International. While our
released dataset has extensive demographic information, we do not collect any identifiers that can uniquely isolate a person (e.g. name, MTurk ID, etc.)
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Section 8

heading: 3 A2. Did you discuss any potential risks of your work?
text: Section 9

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract + Introduction
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 3 for our introduced dataset, and we cite all baseline models (Section 5)

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 5

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: Appendix Section E

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Yes, Section 9

heading: 3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any
text: information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Appendix E

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Section 3
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 3 and Section 5
C 3 Did you run computational experiments? Yes, Section 5

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Appendix D and E
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 5 and Appendix D

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Our results are averaged across 5 runs; Section 5

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 5
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Yes, Section 3.3

heading: 3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots,
text: disclaimers of any risks to participants or annotators, etc.? Yes, Section 3.3 and Appendix C

heading: 3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Section 3.3

heading: 3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to
text: crowdworkers explain how the data would be used? Appendix C.1

heading: 3 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board?
text: Section 3

heading: 3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data?
text: Section 3.4

heading: Future Work
text: Future work that uses sociocultural information should only be applied to settings where there is no foreseeable harm to end-users. Furthermore, learning sociocultural associations can introduce positive and negative stereotypes; documenting and reducing harmful stereotypes is an important avenue for future work. Finally, we emphasize that our work is not evidence for linguistic determinism: sociocultural variation in language can influence but not determine thought