Venue: NAACL 2025

# Does Liking Yellow Imply Driving a School Bus?

Semantic Leakage in Language Models

Hila Gonen\({}^{1}\) Terra Blevins\({}^{1}\) Alisa Liu\({}^{1}\) Luke Zettlemoyer\({}^{1}\) Noah A. Smith\({}^{1,2}\)

\({}^{1}\)Paul G. Allen School of Computer Science & Engineering, University of Washington

\({}^{2}\)Allen Institute for Artificial Intelligence

hilagnn@gmail.com

{blvns,alisliu,lsz,nasmith}@cs.washington.edu

###### Abstract

Despite their wide adoption, the biases and unintended behaviors of language models remain poorly understood. In this paper, we identify and characterize a phenomenon never discussed before, which we call _semantic leakage_, where models leak irrelevant information from the prompt into the generation in unexpected ways. We propose an evaluation setting to detect semantic leakage both by humans and automatically, curate a diverse test suite for diagnosing this behavior, and measure significant semantic leakage in 13 flagship models. We also show that models exhibit semantic leakage in languages besides English and across different settings and generation scenarios. This discovery highlights yet another type of bias in language models that affects their generation patterns and behaviour.

## 1 Introduction

As language models (LMs) become more prevalent [45, 69, 70], we are steadily learning more about their peculiarities and the unique and often unexpected properties of their behavior. Phenomena ranging from hallucinations [8] to sycophancy [25] and many types of biases [24] have been revealed in these models' outputs. Each such discovery leads to a cycle of in-depth study and development of new methods to mitigate these behaviors as much as possible.

We identify a phenomenon in language models never discussed before, which we term **semantic leakage** -- these models can generate text with strong semantic relationships to unrelated words in the prompts. For example, when given the prompt _"He likes yellow. He works as a"_, Gpt4o1 generates the output _"school bus driver"_ (Figure 1). Here we say that the word _yellow_ has leaked into the generation in a way that unintentionally influences the generated occupation. Figure 1 shows three additional examples of prompt-generation pairs (using Gpt4o). In each example, the leakage from the semantic meaning of the underlined word in the prompt is apparent in the generation.

Footnote 1: [https://platform.openai.com/docs/models/grt-4o](https://platform.openai.com/docs/models/grt-4o)

We define semantic leakage in a generation as an undue influence of semantic features from words in the prompt on the generation, "undue" in the sense that the semantic relatedness between the prompt and the generation is stronger than would be expected in natural distributions. Often semantic leaks read as forced, overwrought, even non-sensical generations, like those found in children's stories.

In this paper, we introduce an evaluation metric for measuring semantic leakage. We examine semantic leakage with 109 examples of different semantic categories (animals, food, music, etc.) and demonstrate that it exists across 13 models and

Figure 1: Examples of semantic leakage in Gpt4o. The leaking concept is underlined.



4 temperature sampling values, as well as in additional generation settings (e.g., open-ended generation and multilingual settings). Our analysis shows that finetuned/instruction-tuned models tend to leak _more_, and that semantic leakage also happens across languages.

Semantic leakage is closely related to different types of biases models exhibit, ranging from gender, racial and cultural biases Bolukbasi et al. (2016); Caliskan et al. (2017); Gonen and Goldberg (2019); Nadeem et al. (2021) to cognitive and psychological biases Jones and Steinhardt (2022); Macmillan-Scott and Musolesi (2024); Hagendorff et al. (2023), in which associations between different concepts are learned by the model during training and exposed as bias during generation Gonen and Webster (2020); Schick et al. (2021). As an example, consider the prompt _"**She works at the hospital as a"_, and the prompt _"He works at the hospital as a"_. Given the generations _"nurse"_ and _"doctor"_, respectively, which is a typical biased behavior, we can think of the word "she" as the concept that leaks the property of the female gender into the generation _"nurse"_, a stereotypically female occupation. While still not fully understood, we suspect that much documented and discussed gender bias and other types of previously documented biases Navigli et al. (2023) are instances of associations that get learned and influence in a broader way, which is partially reflected as semantic leakage. Specifically, here we are looking at larger semantic classes (i.e., compared to gender or race), and we seek to quantify and study learned associations and their effect on model generation settings.

Given the nature of models in learning associations during training, whether semantic leakage is surprising or not is a point of contention. In addition, the implications of this behaviour depend on the user and application context, and are not necessarily either good or bad. At the same time, we strongly believe that it is an interesting and important behavior to investigate as it may be a broad enough umbrella to encompass many other associations that are studied as more specific cases. Studying the broader family may be easier and more beneficial, as the more general class may include cases more resistant to mitigation strategies and more prevalent in model outputs.

Finally, in addition to typical and previously discussed biases of different types that fall under this broad class of semantic leakage, we point to additional potential ramifications of this phenomenon. First, semantic leakage might hinder the performance of models via the overshadowing mechanism, introduced in Zhang et al. (2024), in which strong associations in a question override more important and relevant parts of the question to a point where the latter are ignored, resulting in a wrong answer. Semantic leakage might also be used in adversarial settings, similar to attacks presented in Wallace et al. (2019), where specific words that are added to the prompt negatively influence the output of the model. Other potential implications include hindering performance in creative writing Ippolito et al. (2022), where diversity and originality are especially important, but semantic leakage might significantly hurt the output quality in unexpected ways.

Our contributions in this paper can be summarized as follows: (1) we identify and define the phenomenon of _semantic leakage_ in language model generation (Section 2); (2) we build a test suite for detecting semantic leakage in language models (Section 2); (3) we evaluate 13 models with varying sizes using this test suite, uncovering consistent cross-model trends, and validate this automatic evaluation with human judgments (Section 4 and 5); (4) we show that models also exhibit semantic leakage in languages beyond English Chinese and Hebrew) as well as in crosslinkingal settings (Section 6) and in more open-ended generation (Section 7). By characterizing semantic leakage, we demonstrate yet another property language models exhibit in text generation, and highlight how choices in prompt construction can inadvertently affect model output.

## 2 Semantic Leakage

### Overview and Definitions

When producing text, language models can draw on semantic associations with words from the input, or prompt, that are not required or expected, and sometimes even violate rules of logic or common sense. For example, given the prompt _"He likes Roalsa. His favorite food is"_Gpt4q generates the output _"eucalyptus leaves"_. Here, we say that the semantic association with "koalas" and the foods they eat "leaks" into the generation, despite the fact that a person's favorite food and their opinion on koalas are unrelated in the real world. We call this phenomenon **semantic leakage**.

A related phenomenon of conceptual leakage has 

been reported in image generation (Rassin et al., 2022). There, they find that visual properties of one object leak into other objects in the image (for example, the prompt "_a zebra and a street_" generates an image of a zebra next to a zebra crossing), which resembles examples we show (e.g., yellow leaking into the occupation through "school bus driver", see first example in Figure 1). While this behaviour might be seen as subjective, the cases we consider in this paper are, we believe, beyond debate.

Semantic leakage in text generation can also manifest in more subtle ways: for the prompt _"He likes green__. He works as a"_, Gpt4o generates _"landscape architect"_ as a response. In other cases, the model may leak semantics that are not even used in the prompt: For example, when prompted with an idiom, a model can leak the literal semantic meaning of that phrase (that is not actually being used): for instance, when prompted with _"She gave him the green light for the new project. A day later he sent an invitation to everyone by mail, with an envelope colored"_, GPT3.5 generates the response _"bright green to match the theme of the project."_ This is similar to another observation made by Rassin et al. (2022) where the authors demonstrate that sense-ambiguous words are hard for the model to isolate, and the generated images often exhibit the unintended sense together with the intended one.

### Operationalizing the Measurement of Semantic Leakage

We define the different elements of semantic leakage as follows. A _prompt_ is input text that primes the model to output a corresponding _generation_. We consider two types of prompts: _control_ prompts, which do not include any spurious semantic signal (_"His favorite food is"_), and _test_ prompts (_"He likes boalas_. _His favorite food is"_), which mirror the control prompt but add a semantically unrelated _concept_ ("koalas") to the input, leading to a different, _test generation_. While it is known that changing the surface form of the prompt often alters model output (Gonen et al., 2023; Sclar et al., 2024), these new _test_ generations are frequently much more semantically similar to the _concept_ than the _control_ generations (Section 4).

We evaluate the prevalence of semantic leakage in a given model by comparing the similarity of the generations produced by the control and test prompts to the concept under consideration. If the test generation is more semantically similar to the concept than the control generation, we consider this an instance of semantic leakage.

To quantify the prevalence of semantic leakage, we design an evaluation setting that is motivated by the definition of semantic leakage, as having stronger connections of the concept to the test generation. The goal is to compare the similarity of the generations produced by the control and test prompts to the concept, while making sure no other factors are taken into consideration. We use the following similarities to then derive the evaluation metric detailed below:

\[\begin{split}\text{sim}_{\text{control}}&=similarity (\text{concept},\text{control})\\ \text{sim}_{\text{test}}&=similarity(\text{concept}, \text{test})\end{split}\]

Evaluation MetricFrom the above formulation, we derive the "Semantic Leakage Rate" metric (**Leak-Rate**), the percentage of instances in which the concept is semantically closer to the test generation than the control generation. We score _Leak-Rate_ by averaging the following function across all instances, and converting to the range of 0-100%:

\[\begin{split}\text{Leak-Rate}(\text{test},\text{control})=\begin{cases} 1&\text{sim}_{\text{test}}>\text{sim}_{\text{control}}\\ 0&\text{sim}_{\text{test}}<\text{sim}_{\text{control}}\\ 0.5&\text{sim}_{\text{test}}=\text{sim}_{\text{control}}\end{cases}\end{split} \tag{1}\]

This evaluation setting can be implemented as automatic evaluation (Section 4) or as human evaluation (Section 5), with no required adjustments. For automatic evaluation, we embed the texts (_concept, control generation, test generation_) with an embedding method (see Section 3.3) and compute similarity measures; for human evaluation, we ask annotators to judge which of the two pairs is more semantically similar.

When the model does not exhibit semantic leakage at all, we expect a Leak-Rate of 50%, i.e., an even split between test vs. control having higher similarity, for those that show a difference. We expect Leak-Rate higher than 50% when the model exhibits semantic leakage. The Leak-Rate scores are comparable across models for a given embedding method and test set of prompt pairs.

### Building a Test Suite

We build a test set of prompts containing categories and concepts with clear semantic associations. We also focus on categories and prompt settings that 

are likely to allow uncontroversial evaluation (by priming for short model outputs), though later we also consider open-ended generation (Section7). In addition, a subset of the test suite considers idioms, which have both literal and figurative interpretations, as concepts in the test prompts.

We manually create 109 prompts with concepts from categories such as colors, food, animals, songs, occupations and more. A few examples are listed in Table1.2 Each prompt in our test suite is matched with a control prompt as explained above. When designing the control prompts, we prioritized keeping them as basic and simple as possible to avoid confounding factors. In some cases, more elaborate prompts required us to include richer control prompts that are more similar to the test prompts in their length and structure.3 Both types of control prompts are included in our evaluation.

Footnote 2: The full list can be found at [https://gonenhila.github.io/files/semantic_leakage_prompts.txt](https://gonenhila.github.io/files/semantic_leakage_prompts.txt)

Footnote 3: An example is the prompt “_She went on a field trip to the zoo. Her sister got a gift from a friend: a shirt with a print of”_ for which we design the control prompt: “_She went on a field trip. Her sister got a gift from a friend: a shirt with a print of”_.

## 3 Experimental Setup

We present the experimental setup for automatically evaluating semantic leakage in a wide range of models and model sizes.

### Experimental Details

We evaluate semantic leakage in multiple language models from two families: GPT4 and Llama models Touvron et al. (2023), as detailed below. For all models, we explore several temperature values (0, 0.5, 1, 1.5), and run each prompt 10 times to get variation in the generations, when possible.

Footnote 4: [https://platform.openai.com/docs/models](https://platform.openai.com/docs/models)

Before evaluation, for cases where the prompt is repeated in the model generation, we remove the repeated prompt. We also truncate the generations after the first period since the main piece of information is generated before it, and because Llama models tend to generate unrelated sentences or phrases that might interfere with the evaluation.

### Models

We experiment with 13 models of two different state-of-the-art model families to explore semantic leakage in a diverse inventory of models.

GPT modelsWe use OpenAI's API and send requests to GPT models by calling Gpt3.5, Gpt4Achiam et al. (2023), and Gpt4o.5 With GPT models, for sentence completion prompts we prepend "Complete the sentence:", as we find the model performs the task better this way.

Llama modelsWe run all Llama variations using Huggingface Wolf et al. (2019).6 We cap the generation in Llama models at 100 tokens (300 tokens for open generation, Section7).

Footnote 5: gpt-3.5-turbo-0125, gpt-4-turbo-2024-04-09 and gpt-4o-2024-05-13, respectively.

Footnote 6: Llama2: 7B, 7B-chat. 13B, 13B-chat, 7OB, 7OB-chat. Llama3: 8B, 8B-Instrot, 70B, 7DB-Instrct.

### Embedding Methods

For automatic evaluation, we aim to use basic embedding methods that are able to detect and reflect semantic similarities, and are ideally detached from the models we evaluate to avoid confounding factors. We consider the following embedding methods. BERT-score directly provides a similarity score, for the others we apply cosine-similarity.

BERT-score (BS)BERT-scoreZhang et al. (2020) is an automatic evaluation metric for text generation, that computes a similarity score for each token in the candidate sentence with each token in the reference sentence, where token similarity is computed using contextual embeddings. We use the distilbert-base-uncased model.7

Footnote 7: [https://huggingface.co/efederici/sentence-bert-base](https://huggingface.co/efederici/sentence-bert-base)

For the multilingual experiments, where we expect generations in non-English languages (Section6), we use the respective models: bert-base-chinese for Chinese, and bert-base-multilingual-cased for Hebrew. Crosslingual settings still use the English model as the generations there are mainly in English.

SentenceBERTembeddings (SB)SentenceBERTReimers and Gurevych (2019) is a modification of BERTDevlin et al. (2019) that uses Siamese and triplet network structures to derive semantically meaningful sentence embeddings that can be compared using cosine-similarity. We use the huggingface implementation.8

Footnote 8: [https://huggingface.co/efederici/sentence-bert-base](https://huggingface.co/efederici/sentence-bert-base)

OpenAI embeddings (OAI)We use text-embedding-3-large,9 OpenAI's best 

performing embeddings. There is no public documentation of the model training.

In addition to these metrics, we validate our experiments with a manual evaluation of semantic leakage on a subset of the models (Section 5). This also serves as a validation of our automatic metrics.

## 4 Results

Significant semantic leakage across various use cases and models.Table 2 depicts the average leakage for each model, across multiple samplings and temperature values, as detailed in Section 3.1. We see that semantic leakage is exhibited by all model variations, and is detected by all embedding models we use. Leak-Rate values are all well above the 50% random mark and statistically significant10 with \(p<10^{-100}\), validating the semantic relatedness of the prompt and the test generation.

Footnote 10: Using a _t_-test for the lists of the Leak-Rate values to test that their mean is significantly greater than 50%.

Table 1 lists a few examples of semantic leakage from the different models, showcasing leakage in diverse use cases and styles and with respect to a variety of leaking concepts. In many cases we explore, the generations do not make sense in the context, or are very limited and focused on the leaking concept from the prompt.

Leakage is more pronounced in certain model variations.The results in Table 2 show that certain model variations tend to exhibit more semantic leakage than their counterparts. We now analyze the differences within the 2 model families.

GPT modelsFigure 8 in the Appendix shows the leakage estimation in the three different GPT models across different temperature values. Gpt4o

\begin{table}
\begin{tabular}{l|l|l} \hline \multicolumn{1}{c}{**Model**} & \multicolumn{1}{c}{**Prompt and generation**} \\ \hline Llama2 & _John likes dolphins. John’s father is working as a_ marine biologist \\
7b-chat & _John likes tigers. John’s father is working as a_ tiger in a zoo \\ \hline Llama2 & _He likes pandas. His favorite food is bamboo_ \\
13b-chat & _He likes mice. His favorite food is cheese_ \\ \hline Llama2 & _His name is Cedar. His friend lives in_ a treehouse \\
70b-chat & _His name is Ocean. His friend lives in_ a houseboat \\ \hline Llama3 & _He watched a polka concert. For dinner he had_ a plate of pierogies and a side of sauerkraut \\
8b-ins & _He watched a flamenco concert. For dinner he had_ a delicious paella \\ \hline Llama3 & _Mercedes has a friend. The friend's hobby is_ collecting Mercedes cars because he loves the Mercedes brand \\
70b-ins & _Forest has a friend. The friend's hobby ls_ gardening, to her home in a way which makes it a forest \\ \hline Gpt3.5 & _He is a firefighter. His favorite song is_ “Ring of Fine” Jy Johnny Cash \\ \cline{2-2}  & _He is a postman. His favorite songs is_ “Signed. Sealed, Delivered? By Steve Wonder \\ \hline Gpt4 & _The woman was as cold as ice. Afterwards, I went to the library and read_ about the fascinating science of cryogenics, which explores the process of freezing and preserving organic matter at extremely low temperatures \\ \cline{2-2}  & _It was her turn to speak and she got cold feet. A day later she stayed at home because she got hurt in her_ ankle during a morning jog \\ \hline Gpt4o & _She is a music lover. Her nephew was accused of_ stealing her prized vinyl collection \\ \cline{2-2}  & _She is a cinema tover. Her nephew was accused of_ pirating movies \\ \hline \end{tabular}
\end{table}
Table 1: Examples of instances that lead to semantic leakage, taken from different models. The prompt is in green (_italic_) and the generation is in blue.

\begin{table}
\begin{tabular}{l|r|l|l} \multicolumn{1}{c}{} & \multicolumn{3}{c}{**Leak-Rate**} \\ \multirow{2}{*}{**Model**} & BS & SB & OAI \\ \hline \hline GPT3.5 & 74.3 & 68.6 & **85.5** \\ GPT4 & 70.8 & 61.2 & 84.4 \\ GPT4o & **76.9** & **70.4** & 85.0 \\ \hline
2-7b & 66.8 & 64.9 & 72.8 \\
2-7b-chat & 72.6 & **71.7** & 77.8 \\
2-13b & 70.4 & 65.1 & 73.6 \\
2-13b-chat & 71.5 & 65.2 & **78.4** \\
2-70b & 71.9 & 63.2 & 75.9 \\
2-70b-chat & **75.4** & 66.8 & 78.2 \\ \hline
3-8b & 69.6 & 65.9 & 75.5 \\
3-8b-ins & **78.1** & 68.8 & **81.5** \\
3-70b & 71.6 & 68.1 & 75.2 \\
3-70b-ins & 76.3 & **71.2** & 77.3 \\ \hline \end{tabular}
\end{table}
Table 2: Semantic Leak-Rate averaged across 10 samples for each of 4 temperature values. No semantic leakage would correspond to a Leak-Rate of 50% (random guessing), with higher values indicating more leakage. The bottom sections present the Llama2 and Llama3 models, respectively. The model showing the most leakage in each ¡model family, metric¿ setting is **bolded**.

 

consistently leaks more than Gpt4 and Gpt-3.5.11

Footnote 10: This is not true for the OpenAI embedding model, which may be due to confounding model training factors; however, how the embeddings are constructed is not publicly available.

Llama modelsFor Llama models we consistently see that the instruction-tuned models (chat version in Llama2 and instruct version in Llama3) leak more than their pretrained-only counterparts. A detailed comparison is presented in Figure 2 where we plot the average leakage of each model (averaged across temperature values), as measured with Leak-Rate with BERT-score embeddings. All the differences are statistically significant11 with \(p<0.002\) except for Llama-2-13b. We see similar trends with all other metrics as well.

Footnote 11: \(t=0\).

Leakage across different temperatures.We now inspect the way sampling temperature affects semantic leakage. For the GPT models, we see no clear trends (Figure 8 in the Appendix).12

Footnote 12: \(t\)-tests on the Leak-Rate values show that the mean in finetuned models is significantly higher than vanilla versions.

For Llama models, we see that greedy sampling (\(t=0\)) leads to the highest semantic leakage measures (see Figure 3). Generally, lower temperature values lead to more leakage--this is consistent for most models and across all metrics.

## 5 Human Evaluation

We now perform a manual evaluation of semantic leakage, which will also validate our automatic metrics and experiments.

Human Evaluation SetupHuman evaluation is based on the same setting as automatic evaluation, described in Section 2.2.

We recruit two native English speakers who are not involved with the project. These annotators are not provided with the objective of this experiment, but instead are given the following annotation guidelines: _Consider the word or phrase X. Which of the following texts (A or B) is more **semantically related** to X? (A/B/Neither)._ We map the annotator's choice of (A/B/Neither) to (test, control, neither) and then score Leak-Rate (Eq. 1).

Using these guidelines, the annotators are asked to label 109 test-control generation pairs from each model (ordered randomly). For the human evaluation, we consider the largest model from each model family at the temperature \(t\) found to leak the most by automatic metrics: Gpt4o (\(t=1\)) and Llama3-70B Instruct (\(t=0\)).

AnalysisFigure 4 compares the semantic leakage detected by the human evaluation for Gpt4o against the automatic metric. The values in the human evaluation row are percentages for each category: test is more similar to the concept, control is more similar to the concept, or neither.

We expect humans to have higher tolerance for similar scores, i.e., more cases falling under \(\text{sim}_{\text{test}}=\text{sim}_{\text{control}}\) in the human evaluation than in the automatic evaluation, where it occurs almost solely when the test and control generations are the same (see Equation 1). To visually account for this difference we plot the automatic metric results by using colored gradient to depict the difference in similarity values \(\text{sim}_{\text{test}}-\text{sim}_{\text{control}}\), with positive values (shades of blue

Figure 3: Semantic leakage in Llama at different temperatures (measured with Leak-Rate using BERT-score).

Figure 2: Semantic leakage in Llama models, averaged across temperature values (measured with Leak-Rate using BERT-score).

 

semantic leakage. The results account for all ten generations sampled from each model.

We find similar trends on Llama3-70B Instruct with an average human-annotated Leak-Rate of 66.7 and automatic evaluation of Leak-Rate that range from 71.2 to 77.3 across the different embedding types.

We also calculate Kendall's \(\tau\) on the human annotations. We find high interannotator agreement between the human annotators (\(\tau=0.68\)), indicating that humans generally agree on the cases that constitute semantic leakage. We also compute Kendall's \(\tau\) on the human evaluation vs. similarity differences calculated using BERTScore embeddings to evaluate how well these methods correlate,14 and get a moderate correlation of \(\tau=0.39\) when averaged between the two annotators.15

Footnote 14: We introduce a slack variable of \(\epsilon\) = 0.03 to account for the different levels of similarity tolerance between humans and embedding-based similarity.

Footnote 15: While we use all model generations per example when obtaining label distributions and computing Leak-Rate, to calculate \(\tau\) we use the generations shown to the human annotators, as this requires example-level alignments.

## 6 Multilingual and Crosslingual Semantic Leakage

In the previous sections we established that semantic leakage is exhibited in various scenarios in English, across different models and model sizes. We now inspect semantic leakage in languages outside of English, as well as in crosslingual settings, by focusing on Hebrew and Chinese, Hebrew-English and Chinese-English. As in the previous section, we consider the largest model from each model family at the temperature \(t\) found to leak the most: Gpt4o (\(t=1\)) and Llama3-70B Instruct (\(t=0\)). Since we found that Llama is struggling with Hebrew, especially with Hebrew generation, we exclude these results.

Multilingual Semantic LeakageWe translate all the prompts into Chinese and Hebrew (see examples in Table 4 in the Appendix, rows 1, 3) with the help of native speakers (including the authors). The translation is mostly straightforward with the exception of idioms and names. For English idioms that do not exist in the target language, we write a new example using an idiom in that language (together with a matching context and a control prompt). For names, in Chinese we choose a plausible Chinese name with that meaning (e.g., \(~{}\gamma\))\(~{}\mathbb{Z}\), meaning _little wheat_, for Rye). In Hebrew, we make an attempt to translate to an existing frequent name in Hebrew, or to substitute it with a name with a related meaning (e.g., Lake instead of River). If that is not an option (e.g., Rye), we come up with a different name that has a clear semantic field.

Crosslingual Semantic LeakageWe create mixed prompts that begin with Chinese/Hebrew and end with English (see examples in Table 4 in the Appendix, rows 2, 4). We use the translations to Chinese and Hebrew, and mix with English: for the first part of the prompt we use the translation to Chinese/Hebrew, and for the rest of it we use the original part in English. For evaluation purposes, we use the English concepts, since we expect the generation to be in English. For names or idioms that are used in Chinese/Hebrew parts of the prompt, we use their literal translations into English.

ResultsFigure 5 shows the human evaluation results for the multilingual and crosslingual experiments (with Gpt4o). We can see significant semantic leakage in multilingual and crosslingual settings, with Leak-Rate values that range from 70.6 to 78.4 for the 4 different settings, similar to the Leak-Rate values we got for English.16

Footnote 16: In 6 examples of the crosslingual prompts (Zh-En), the generations are just translations because of the way the prompt is designed - we annotate those cases as “neutral”.

The generation quality with the Llama model is generally much lower, and we get Leak-Rates of 66.5 and 61.5 for Chinese and Chinese-English, respectively, according to human annotation. As noted above, the quality of generation in Hebrew and Hebrew-English did not allow for evaluation of semantic leakage.

Table 3 shows the automatic evaluation of the multilingual and crosslingual experiments. The results for the multilingual settings are not as reliable as in the English setting since we cannot determine

Figure 4: Human detection of semantic leakage compared to automatic methods. Leak-Rate is reported on the right for each method.

 

the quality of the underlying embedding methods for Hebrew and Chinese.17 In the crosslingual setting we mainly compare English generations with English concepts, thus the results are more reliable.

Footnote 16: To the best of our knowledge, the model underlying sentenceBERT was trained (predominantly) on English. This is why the metric detects more leakage in cross-lingual settings (with evaluations on English) compared to other languages.

## 7 Open-Ended Generation

We also examine semantic leakage in open-ended scenarios that encourage the model to generate multiple sentences or paragraphs. It is less clear how to quantify the semantic leakage in this setting, as it can manifest in more ways within the longer output; we therefore rely on more qualitative analysis in addition to the automatic metrics. This section focuses on generations from Gpt4o with \(t=1\), as this model with this temperature was most prone to leakage in previous experiments (Section 4).

StoriesA popular setting for open-ended text generation is storytelling: we ask the model to write a story about a child, with the child's name being the concept we expect to leak. Figure 6 gives the story that Gpt4o generates when prompted to create a story about a child named Coral (and also Melody in Figure 9 in the Appendix). These generated stories contain many words and elements directly related to the semantics of each name, and the narrative of each story is tailored to those meanings. When measuring semantic leakage in this setting automatically for 23 different names,18 we get Leak-Rates of 78.7, 60.9 and 76.1 with BS, SB and OAI embeddings respectively, suggesting similar rates of leakage to the other examples.

Footnote 18: We remove any instance of the name from the generation, even when that instance is used as a common noun.

RecipesWe also consider open-ended recipe generation. In Figure 7, we present Gpt4o's generation when asked to suggest a recipe for use with a _blue_ pan (or a _white_ pan in Figure 10 in the Appendix), where we consider each color to be the potentially leaking concept. The model suggests to make blueberry pancakes and chicken alfredo, respectively, with the foods matching the color of the pan. When prompted with a red pan, ingredients like peppers, tomatoes and paprika are far more frequent, same as with asparagus in the case of a green pan and lemon with a yellow pan.

We experiment with 5 different colors (10 samples each) that are likely to be related to types of food (blue, green, red, white, and yellow). Automatic evaluation in this setting is more challenging since the concept might appear in the generation in a leaky way ("red pepper") or just as part of the answer, with no attributed leakage ("red pan") and these two cases are difficult to differentiate automatically. We decide to remove all occurrences of _"<color> pan"_ from the generations when using automatic metrics, and get Leak-Rates

Figure 5: Human detection of semantic leakage in multilingual and crosslingual settings.

Figure 6: Example of story generation, exhibiting semantic leakage based on the name in the prompt.

\begin{table}
\begin{tabular}{l|c|c|c|c}  & \multicolumn{3}{c}{**Leak-Rate**} \\
**Model** & **Language** & BS & SB & OAI \\ \hline \hline GPT4o & Heb & 60.6 & 53.5 & 67.2 \\  & Heb-En & 62.1 & 58.2 & 74.9 \\  & Zh & 67.6 & 48.9 & 80.8 \\  & Zh-En & 61.6 & 60.5 & 71.3 \\ \hline
3-70b-ins & Zh & 73.4 & 54.1 & 82.8 \\  & Zh-En & 79.4 & 81.2 & 85.6 \\ \hline \end{tabular}
\end{table}
Table 3: Semantic leakage scores for multilingual and crosslingual setting, averaged across 10 samplings and measured by Leak-Rate.

 

of 74.0/58.0/74.0 with BS/SB/OAI embeddings, again suggesting semantic leakage here as well.

Footnote 1: [https://github.com/facebook/2023-69-01-01](https://github.com/facebook/2023-69-01-01)

## 8 Related Work

Rassin et al. (2022) show similar conceptual leakage in image generation with Dalle-2 Ramesh et al. (2022), a text-to-image model by OpenAI. They find that sense-ambiguous words are hard for the model to isolate--similar to our observation that idioms in the prompt can leak the literal meaning into the generation. Rassin et al. (2024) try to mitigate this issue by encouraging the cross-attention maps to agree with the linguistic binding reflected by the prompt syntax. Leakage of properties in DALLE-2 has also been discussed by Ramesh et al. (2022) and Leviada et al. (2023).

As models have improved, a related line of research has asked whether they match human behavior, particularly with respect to cognitive biases and priming. Prior work compares models' behavior to documented behavior of humans in psychological experiments, highlighting some similarities as well as differences from human response to given information Jones and Steinhardt (2022); Macmillan-Scott and Musolesi (2024); Hagendorff et al. (2023). A known psychological phenomenon that is very related to models' semantic leakage as we present it here, is that of priming, and specifically semantic priming Meyer and Schvanveldt (1971); Neely (1976); Tulving et al. (1982); Tulving and Schaacter (1990), where decisions and reactions about specific words change according to given semantic stimuli provided before the target word. We plan to investigate this relationship further in future work.

Biases in Language ModelsThere is an extensive line of work investigating bias and stereotypes of different demographic groups in NLP models, ranging from word embeddings Bolukbasi et al. (2016); Caliskan et al. (2017); Gonen and Goldberg (2019) all the way to LMs Nadeem et al. (2021); Kotek et al. (2023); Cheng et al. (2023), sometimes focusing on specific end tasks Gonen and Webster (2020); Stanovsky et al. (2019); Kiritchenko and Mohammad (2018); Zhao et al. (2018). Existing works in this space usually investigate specific bias types with respect to concrete socio-demographic groups Kirk et al. (2021); Venkit et al. (2023, 2022); Navigli et al. (2023), as well as toxicity in model generation Gehman et al. (2020); Nozza et al. (2021). Some works also point to specific input words that cause harmful generation, such as first names De-Arteaga et al. (2019); Maudslay et al. (2019) among others Rotter et al. (2024); Attanasio et al. (2022). In this context, semantic leakage serves as a broader class of association bias, potentially pointing to the underlying mechanism that causes biases in model generation by latching onto concrete properties in the input, resulting in toxic or stereotyped generation.

## 9 Conclusion

We identified a phenomenon in generative language models never discussed before, which we term _semantic leakage_: language models are prone to generating text that reflects unrelated semantic information from the prompt, leading to peculiar and sometimes unreasonable outputs. We defined and measured semantic leakage in a range of models, and together with human evaluation show that it is prevalent and consistent across all models we test. We also found that semantic leakage occurs in many different generation settings, including multilingual and crosslinking ones.

This leakage reflects associations learned by the model, similar to how different types of biases are learned; therefore, our characterization of semantic leakage broadens the scope we should consider with respect to potential ramifications of learned associations. While the implications of semantic leakage are not fully clear, it is a consistent behavior pattern in LMs that merits further study.

The finding that instruction-tuned models leak more is of special interest, given that they are the most popular and best performing model variations currently used. We hypothesize that semantic leakage is more dominant in these models because the leaking generations are less generic and seem to provide more information/content, which might be a property that is incentivized under these fine-tuning processes. We plan to explore this hypothesis more formally in future work.

Figure 7: Example of recipe generation, exhibiting semantic leakage based on the color in the prompt.

 



### Limitations

While our experimental setup spans 13 models of different types and sizes, and explores different sampling temperature values, the scale of the prompts in our test suite remains limited due to the difficulty of manually creating prompts that are likely to leak in a way that we can detect and evaluate. This is also often the case when measuring other language model biases. In addition, though the results are consistent across all models and languages we experiment with, the trends might be different with other models or languages we have not tested.

Finally, we cannot guarantee that the automatic evaluations do not include noise: in some cases, even after our automatic post-processing of the generations, our automatic metrics might consider non-leaking instance as leaking (e.g., in cases of occurrences of the concept in the generation due to a partial repetition of the prompt). However, the human evaluation we conduct generally agrees with the findings we get with the automatic detection, supporting their reliability.

## Ethical Considerations

Semantic leakage is a broad umbrella that may encompass many types of biases, and it broadens our perception and understanding about association biases that are being learned by models and then exhibited in generation. As such, semantic leakage might manifest at times as a concrete type of bias, e.g., cultural, gender or racial biases. This depends on the specific setting and the chosen concept that leaks; while our prompts and concepts are fairly general, these biases and stereotypes could potentially surface in our evaluation.19

Footnote 19: For example, some concepts in the evaluation involve food and music associated with specific countries or ethnicities.

However, our main goal in this paper is not to focus on specific types of biases (or biases that are particularly harmful) but to characterize and quantify the broader class of semantic leakage. We believe that a better understanding of the greater picture will in turn arm us with better tools to combat the subset of biases and behaviors that do have harmful consequences in practice.

Apart from that, in this paper the annotations and translations were done by several volunteers that were willing to help for the project. The load of this work was very low.

## Acknowledgements

This research received support through Schmidt Sciences, LLC. We thank Shauli Ravfogel and Ido Levin for helpful discussions and ideas. We also thank the translators and annotators for their contribution to the paper (Weijia Shi, Jacqueline He, Jacob Schreiber) and the ARK lab members for their valuable feedback. Finally, we would like to express our gratitude to the reviewers of this paper for their useful comments and suggestions.

## References

* Achiam et al. [2023] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774. Cited by: SS5.
* A. Arunaga, D. Nozza, D. Hovy, and E. Baralis [2022]R. Anil, S. Borgeaud, Y. Wu, J. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805. Cited by: SS6.
* C. Attanasio, P. Nozza, D. Hovy, and E. Baralis [2022]Cited by: SS1.
* T. Bolukbasi, K. Chang, J. Y. Zou, V. Saligrama, and A. T. Kalai [2016]M. Cherea, E. Durmus, and D. Jurafsky [2023]M. Cheng, E. Dumus, and D. Jurafsky [2023]Marked personas: using natural language prompts to measure stereotypes in language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Toronto, Canada, pp. 1504-1532. External Links: Document, Link Cited by: SS1.
* M. De-Arteaga, A. Romanov, H. Wallach, J. Chayes, C. Borgs, A. Chouldechova, S. Gevik, K. Kenthapadi, and A. Tauman Kalai [2019]Bias in bios: a case study of semantic representation bias in a high-stakes setting. In proceedings of the Conference on Fairness, Accountability, and Transparency, Vol. , pp. 120-128. External Links: Document, Link Cited by: SS1.
* J. Devlin, M. Chang, K. Lee, and K. Toutanova [2019]Bert: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the

North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_.
* Gehman et al. [2020] Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. 2020. RealToxicityPrompts: Evaluating neural toxic degeneration in language models. In _Findings of the Association for Computational Linguistics: EMNLP 2020_, pages 3356-3369, Online. Association for Computational Linguistics.
* Gonen and Goldberg [2019] Hila Gonen and Yoav Goldberg. 2019. Lipstick on a pig: Debiasing methods cover up systematic gender biases in word embeddings but do not remove them. In _Proceedings of NAACL-HLT_.
* Gonen et al. [2023] Hila Gonen, Srini Iyer, Terra Blevins, Noah A Smith, and Luke Zettlemoyer. 2023. Demystifying prompts in language models via perplexity estimation. In _Findings of the Association for Computational Linguistics: EMNLP 2023_, pages 10136-10148.
* Gonen and Webster [2020] Hila Gonen and Kellie Webster. 2020. Automatically identifying gender issues in machine translation using perturbations. In _Findings of the Association for Computational Linguistics: EMNLP 2020_, pages 1991-1995.
* Hagendorff et al. [2023] Thilo Hagendorff, Sarah Fabi, and Michal Kosinski. 2023. Human-like intuitive behavior and reasoning biases emerged in large language models but disappeared in chatgpt. _Nature Computational Science_, 3(10):833-838.
* Ippolito et al. [2022] Daphne Ippolito, Ann Yuan, Andy Coenen, and Sehmon Burnam. 2022. Creative writing with an ai-powered writing assistant: Perspectives from professional writers. _arXiv preprint arXiv:2211.05030_.
* Ji et al. [2023] Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of hallucination in natural language generation. _ACM Computing Surveys_, 55(12):1-38.
* Jones and Steinhardt [2022] Erik Jones and Jacob Steinhardt. 2022. Capturing failures of large language models via human cognitive biases. _Advances in Neural Information Processing Systems_, 35:11785-11799.
* Kiritchenko and Mohammad [2018] Svetlana Kiritchenko and Saif Mohammad. 2018. Examining gender and race bias in two hundred sentiment analysis systems. In _Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics_, pages 43-53, New Orleans, Louisiana. Association for Computational Linguistics.
* Kirk et al. [2021] Hannah Rose Kirk, Yennie Jun, Filippo Volpin, Haider Iqbal, Elias Benussi, Frederic Dreyer, Aleksandar Shtedritski, and Yuki Asano. 2021. Bias out-of-the-box: An empirical analysis of intersectional occupational biases in popular generative language models. _Advances in neural information processing systems_, 34:2611-2624.
* Kotek et al. [2023] Hadas Kotek, Rickker Dockum, and David Sun. 2023. Gender bias and stereotypes in large language models. In _Proceedings of the ACM collective intelligence conference_, pages 12-24.
* Leivada et al. [2023] Evelina Leivada, Elliot Murphy, and Gary Marcus. 2023. Dall: e 2 fails to reliably capture common syntactic processes. _Social Sciences & Humanities Open_, 8(1):100648.
* Macmillan-Scott and Musolesi [2024] Olivia Macmillan-Scott and Mirco Musolesi. 2024. (ir) rationality and cognitive biases in large language models. _Royal Society Open Science_, 11(6):240255.
* Maudslay et al. [2019] Rowan Hall Maudslay, Hila Gonen, Ryan Cotterell, and Simone Teufel. 2019. It's all in the name: Mitigating gender bias with name-based counterfactual data substitution. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 5267-5275, Hong Kong, China. Association for Computational Linguistics.
* Meyer and Schvaneveltdt [1971] David E Meyer and Roger W Schvaneveltdt. 1971. Facilitation in recognizing pairs of words: evidence of a dependence between retrieval operations. _Journal of experimental psychology_, 90(2):227.
* Nadeem et al. [2021] Moin Nadeem, Anna Bethke, and Siva Reddy. 2021. Stereoset: Measuring stereotypical bias in pretrained language models. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 5356-5371.
* Navigli et al. [2023] Roberto Navigli, Simone Conia, and Bjorn Ross. 2023. Biases in large language models: origins, inventory, and discussion. _ACM Journal of Data and Information Quality_.
* Neely [1976] James H Neely. 1976. Semantic priming and retrieval from lexical memory: Evidence for facilitatory and inhibitory processes. _Memory & cognition_, 4(5):648-654.
* Nozza et al. [2021] Debora Nozza, Federico Bianchi, and Dirk Hovy. 2021. HONEST: Measuring hurtful sentence completion in language models. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 2398-2406, Online. Association for Computational Linguistics.
* Ramesh et al. [2022] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. 2022. Hierarchical text-conditional image generation with clip latents. _arXiv preprint arXiv:2204.06125_.
* Rassin et al. [2024] Royj Rassin, Eran Hirsch, Daniel Glickman, Shauli Ravfogel, Yoav Goldberg, and Gal Chechik. 2024. Linguistic binding in diffusion models: Enhancing attribute correspondence through attention map alignment. _Advances in Neural Information Processing Systems_, 36 

Royi Rassin, Shauli Ravfogel, and Yoav Goldberg. 2022. Dalle-2 is seeing double: Flaws in word-to-concept mapping in text2image models. In Proceedings of the Fifth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP_, pages 335-345.
* Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing.
* Rottger et al. (2024) Paul Rottger, Hannah Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, and Dirk Hovy. 2024. XSTest: A test suite for identifying exaggerated safety behaviours in large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), Mexico City, Mexico. Association for Computational Linguistics.
* Le Scao et al. (2022) Teven Le Scao, Angela Fan, Christopher Akiki, Elie Pavlick, Suzana Ilic, Daniel Hesslow, Roman Castagne, Alexandra Sasha Luccioni, Francois Yvon, et al. 2022. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100.
* Schick et al. (2021) Timo Schick, Sahana Udupa, and Hinrich Schutze. 2021. Self-diagnosis and self-debiasing: A proposal for reducing corpus-based bias in nlp. Transactions of the Association for Computational Linguistics, 9:1408-1424.
* Sclar et al. (2024) Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. 2024. Quantifying language models' sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting. In Proceedings of International Conference on Learning Representations.
* Sharma et al. (2024) Minank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R Bowman, Newton Cheng, Esin Durmus, Zac Hatfield-Dodds, Scott R Johnston, et al. 2024. Towards understanding sycophancy in language models. In Proceedings of International Conference on Learning Representations.
* Stanovsky et al. (2019) Gabriel Stanovsky, Noah A. Smith, and Luke Zettlemoyer. 2019. Evaluating gender bias in machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy. Association for Computational Linguistics.
* Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almhairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajwal Bhargava, Shruti Bhosale, et al. 2023. LIama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
* Tulving and Schacter (1990) Endel Tulving and Daniel L Schacter. 1990. Priming and human memory systems. Science247(4940):301-306.
* Tulving et al. (1982) Endel Tulving, Daniel L Schacter, and Heather A Stark. 1982. Priming effects in word-fragment completion are independent of recognition memory. Journal of experimental psychology: learning, memory, and cognition8(4):336.
* Venkit et al. (2023) Pranav Narayanan Venkit, Sanjana Gautam, Ruchi Panchanadikar, Ting-HaoKenneth Huang, and Shomir Wilson. 2023. Nationality bias in text generation. arXiv preprint arXiv:2302.02463.
* Venkit et al. (2022) Pranav Narayanan Venkit, Mukund Srinath, and Shomir Wilson. 2022. A study of implicit bias in pretrained language models against people with disabilities. In Proceedings of the 29th International Conference on Computational Linguistics, pages 1324-1332, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
* Wallace et al. (2019) Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. 2019. Universal adversarial triggers for attacking and analyzing NLP. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China. Association for Computational Linguistics.
* Wolf et al. (2019) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, et al. 2019. Huggingface's transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771.
* Zhang et al. (2020) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. In Proceedings of International Conference on Learning Representations.
* Zhang et al. (2024) Yuji Zhang, Sha Li, Jiateng Liu, Pengfei Yu, Yi R Fung, Jing Li, Manling Li, and Heng Ji. 2024. Knowledge overshadowing causes amalgamated hallucination in large language models. arXiv preprint arXiv:2407.08039.
* Zhao et al. (2018) Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2018. Gender bias in coreference resolution: Evaluation and debiasing methods. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), New Orleans, Louisiana. Association for Computational Linguistics 

Semantic Leakage in GPT Models

Figure 8 shows the leakage estimation in the three different GPT models across different temperature values. Gpt4o consistently leaks more than Gpt4 and Gpt-3.5.

## Appendix B Multilingual and Crosslingual Semantic Leakage

Table 4 shows example multilingual and crosslingual prompts and generations.

## Appendix C Open-Ended Generation

Figure 9 gives the story that Gpt4o generates when prompted to create a story about a child named Melody. The generated story contains many words and elements directly related to the semantics of the name, and the narrative of the story is tailored to that meaning.

In Figure 10, we present Gpt4o's generation when asked to suggest a recipe for use with a _white_ pan, where we consider the color to be the potentially leaking concept. The model suggests to make chicken alfredo, matching the color of the pan.

Figure 8: Semantic leakage in GPT models using different temperature values (measured with Leak-Rate using BERT-score).

Figure 10: Example of recipe generation, exhibiting semantic leakage based on the color in the prompt.

Figure 9: Example of story generation, exhibiting semantic leakage based on the name in the prompt.

 



\begin{table}
\begin{tabular}{l|c|c} Language & Prompt & Generation \\ \hline \hline Heb & NNTN