Context versus Prior Knowledge in Language Models (2024)

Abstract: To answer a question, language models often need to integrate prior knowledge learned during pretraining and new information presented in context. We hypothesize that models perform this integration in a predictable way across different questions and contexts: models will rely more on prior knowledge for questions about entities (e.g., persons, places, etc.) that they are more familiar with due to higher exposure in the training corpus, and be more easily persuaded by some contexts than others. To formalize this problem, we propose two mutual information-based metrics to measure a model’s dependency on a context and on its prior about an entity: first, the persuasion score of a given context represents how much a model depends on the context in its decision, and second, the susceptibility score of a given entity represents how much the model can be swayed away from its original answer distribution about an entity. We empirically test our metrics for their validity and reliability. Finally, we explore and find a relationship between the scores and the model’s expected familiarity with an entity, and provide two use cases to illustrate their benefits. https://github.com/kdu4108/ measureLM

text: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13211–13235 August 11-16, 2024 ©2024 Association for Computational Linguistics
https://github.com/kdu4108/ measureLM

heading: 1 Introduction
text: Language models have displayed remarkable abilities to answer factual queries about entities, suggesting that they encode knowledge about these entities learned during pretraining (Petroni et al., 2019; Brown et al., 2020; Roberts et al., 2020; Geva et al., 2021). For prompts that extend a question with additional information or context, the model can draw on both its prior knowledge and the additional context to answer the query (Kwiatkowski et al., 2019; Joshi et al., 2017; Berant et al., 2013; Kasai et al., 2023). While previous research has investigated how often a model will rely on prior knowledge over conflicting contextual information in answering questions (Longpre et al., 2021), we

heading: Contexts Entities Susceptibility
text: hypothesize that models will not behave identically for all contexts and entities. For example, if a language model is prompted with Harry hugged Voldemort. How friendly are Harry Potter and Lord Voldemort?, we might expect the prior knowledge learned from training data describing the rivalry between these two characters to significantly influence the model’s answer. However, if the model lacks a strong prior on, say, Susie and Alia, then we might expect its answer to be primarily context-driven when prompted with Susie hugged Alia. How friendly are Susie and Alia?.
We formalize this problem through the lens of evaluating the change in a model’s answer distribution for different contexts and entities. We present two mutual information-based metrics that allow us to explore differences in the effect of specific contexts on model behavior for different entities. The persuasion score of a given context measures how much a model’s answer distribution is affected by the context when prompted with a particular query about a given entity. The susceptibility score of a given entity measures how much the model’s answer distribution can be swayed for a particular query about that entity, marginalized over all contexts. Given their basis in mutual information, our metrics are natural operationalizations of per-
1 13211
suasion and susceptibility. Furthermore, we offer empirical evidence of the validity and reliability of these measures by comparing them against similar metrics and showing their robustness to paraphrases and different samples.
To study how language models behave for different contexts and entities, we create a synthetic dataset of queries covering 122 topic areas extracted from the YAGO knowledge graph (Suchanek et al., 2007), entities extracted from YAGO and generated with GPT-4 (OpenAI, 2023), and contexts constructed with different qualities, e.g., relevancy and assertiveness. We apply our new metrics to Pythia models ranging from 70m to 12b parameters (Biderman et al., 2023) to find evidence that relevant contexts are consistently more persuasive than irrelevant ones, and assertive contexts are more persuasive than less assertive ones for yes–no questions. In a deep dive into one model, we find evidence that entities with high expected familiarity, as measured by both training data frequency and entity degree statistics in the YAGO knowledge graph, have lower susceptibility scores.
We further conduct case studies to show how these metrics could be useful in applied settings. In a study on friend–enemy stance measurement, we find evidence that enemy duos are less susceptible than friend duos. Applying our metrics to gender bias, we find evidence for a difference in susceptibility between stereotypically masculine and feminine names for gender-biased contexts. Through this, we show how our proposed metrics can be used to better analyze the effects of context and prior knowledge, with the potential for application toward greater control over model behavior.

heading: 2 Context and Prior Knowledge
text: Much prior work has noted that language models develop biases about entities during training and investigated the tension between this entity bias and additional information provided in the context.

heading: 2.1 Entity Bias
text: Loosely defined across studies as model bias from spurious correlations between entity mentions and a target characteristic in the training data, entity bias has been mainly examined in relation extraction (RE), where a model extracts relationships between entities from text (Zelenko et al., 2002; Zhou et al., 2005). Different studies on RE have attempted to either mitigate (Zhang et al., 2017,
2018; Peng et al., 2020; Wang et al., 2022) or leverage (Yamada et al., 2020; Zhou and Chen, 2022) entity bias for improved performance on the task. Wang et al. (2022) note that entities can carry both useful semantic information about their roles, e.g., whether the entity is a person or a place, and spurious information which can bias the model toward relations not mentioned in the target sentence, e.g., the model may link Switzerland with the relation countries_of_residence at inference time for a sentence that does not mention this relation, due to their frequent co-occurrence in the training data. Additional studies on machine reading comprehension (MRC) tasks, e.g., SQuAD (Rajpurkar et al., 2016), TriviaQA (Joshi et al., 2017), and NaturalQuestions (Kwiatkowski et al., 2019), have found substituting different entity names can result in meaningful changes in model predictions and overall evaluation performance (Yan et al., 2022).

heading: 2.2 Context and Entity Bias
text: The existence of entity bias naturally raises the question of how it interacts with context to shape a model’s response. Several papers (Longpre et al., 2021; Chen et al., 2022; Xie et al., 2023) approach this by inducing and exploring knowledge conflicts, where a context preceding a query proposes information that conflicts with a model’s prior knowledge about that query. They measure the model’s reliance on pretrained entity bias and context by computing the memorization ratio: the proportion of knowledge conflict examples for which the model maintains its answer from prior knowledge. Pezeshkpour (2023) measures a model’s prior knowledge of a fact by comparing the entropy of a queried model’s answer distribution before and after stating the fact in context. Several studies further propose interventions to reduce entity bias and favor in-context information. These approaches include prompting (Zhou et al., 2023; Onoe et al., 2023), modifying training data (Wang et al., 2023), fine-tuning (Li et al., 2023), and neuron-level interventions (Yu et al., 2023) at inference time.
However, the metrics used to quantify model reliance on context and entity bias in these papers—excepting Pezeshkpour (2023)—are limited in several ways. First, most previous work does not develop entity-specific or context-specific metrics for the strength of the entity bias or context persuasiveness. Instead, their metrics produce only
2
a single number to summarize the model’s overall reliance on entity bias. Second, their setups are limited to adversarial cases in which a context is chosen to go against the entity bias. Indeed, we may well wish to measure the interplay of context and entity bias in other cases, such as when context reinforces entity bias or when they do not clearly disagree. Therefore, we seek to define a metric that measures how much the model depends on a given context or entity with rigorous, theoretically grounded interpretations.

heading: 3 Our Formalization
text: We now formalize the problem setting in which we define metrics for the susceptibility of an entity and the persuasiveness of a context for a given model. Let Σ be an alphabet. Consider a language model pM over Σ, i.e., pM is a distribution over the Kleene closure Σ∗. Furthermore, we assume that pM was estimated from a corpus D ⊂ Σ∗. Let E ⊂ Σ∗ be the subset of strings that could correspond to string representations of entities.1 Let Q = {qn}Nn=1 be a set of N slotted query templates of type qn : E → Σ∗ that fill the slot with the argument to each qn. We can think of a query template q ∈ Q as slotting an entity into a query, e.g., slotting the entity Slovenia into the query template q(e) = The capital of e is produces the string The capital of Slovenia is.
Now, consider three random variables. First, let C be a Σ∗-valued random variable that stands for a context. Second, let A be a Σ∗-valued random variable whose values are answers. Let E be a Evalued random variable. The pushforward q(E), then, is a Σ∗-valued random variable over slottings of entities according to query q ∈ Q. The random variables C, q(E) and A are jointly distributed according to the following probability distribution
p(C = c, q(E) = q(e), A = a) △∝ pM(cq(e)a),
(1) where cq(e)a ∈ Σ∗ is string concatenation. To formalize persuasion and susceptibility, we make use of the joint distribution p heavily in the proceeding subsections.

heading: 3.1 Persuasion Score
text: For each context c that is prepended to a query template with a given entity slotted in, q(e), we wish to
1In practice, an entity can have different verbalizations, e.g., Sherlock and Sherlock Holmes. . And, whether a specific string refers to an entity may very well depend on the context in which it occurs. Thus, our set E is an approximation at best.
assign a persuasion score ψ to represent how successful that context is at altering a model’s answer distribution. This score depends on the specific queried entity e, because contexts themselves are often entity-dependent. Intuitively, a context’s persuasion score should measure how much the probability distribution of possible answers changes, averaged across all possible answers. More precisely, we define our persuasion score ψ(c, q(e)) as the half-pointwise mutual information (half-PMI) between the context c and the answer random variable A, conditioned on the fixed query about an entity:
ψ(c, q(e)) ≜ I(C = c;A | q(E) = q(e))
= ∑ a∈Σ∗ p(a | c, q(e)) log p(a | c, q(e))
p(a | q(e)) = KL(p(A | c, q(e)) || p(A | q(e))),
(2)
where p(A | c, q(e)) and p(A | q(e)) can be derived by marginalizing and conditioning Eq. (1).
The persuasion score of a context can then be interpreted as the degree (in nats) to which the context was able to change the model’s answer distribution when prepended to a query. When the persuasion score is at its lower bound of 0 nats, it indicates the context is completely unpersuasive, i.e., it did not change the model’s answer distribution at all. Contexts with higher persuasion scores change the answer distribution more, which is consistent when viewed through the lens of KL-divergence.2

heading: 3.2 Susceptibility Score
text: For a given query template applied to an entity q(e), we further wish to assign a susceptibility score χ to q(e) which represents how easy it is to change a model’s answer distribution. Intuitively, the susceptibility score should measure how much a model’s answer distribution to a query changes when prompted with additional context, averaged across all possible contexts and answers. More precisely, we define the susceptibility score χ(q(e)) as the mutual information between the context and answer random variables, conditioned on a fixed query about an entity:
χ(q(e)) ≜ ∑
c∈Σ∗ p(c)ψ(c, q(e))
= I(C;A | q(E) = q(e)), (3)
2We include details on further equivalencies of half-PMI and other concepts in information theory in App. A.
3
where p(c) is the marginal distribution over contexts. Equivalent to the difference in entropy H(A | q(E) = q(e)) − H(A | C, q(E) = q(e)), the susceptibility score represents the reduction in answer distribution uncertainty (in nats) when a query is preceded by a context. A high susceptibility score means the model is highly influenced by context for the query about that entity, with its upper bound of H(A) indicating that context fully determines the answer. A low score indicates the model’s response is robust to context, with its lower bound of 0 indicating no influence of context on the answer distribution. We can use susceptibility to answer the question “How much does the model’s answer depend on its entity bias?” with an information-theoretically grounded, interpretable scale based on the model’s full behavior.

heading: 3.3 Entity-Independent Persuasion Score
text: Persuasion scores can be further marginalized over entities. Analogous to our definition of the susceptibility score, we define the entity-independent persuasion score of a context as how much the log probability distribution of possible answers changes, averaged across all possible entities and answers. We describe this further in App. B.

heading: 4 Experiments
text: We now provide empirical evidence to further validate our metrics, characterize model behavior using the susceptibility and persuasion scores, and investigate reasons behind differences in susceptibility scores for different entities.

heading: 4.1 Setup
text: For each of the 122 different relations from the YAGO knowledge graph, we collect 100 entities3 and construct 600 random contexts from relationspecific context templates such that each entity is mentioned in 6 contexts. The 600 contexts are evenly distributed between assertive,4 base,5 and negation6 context types. We construct four query forms of each relation: two closed questions, i.e., yes–no, and two open questions.
Using these samples, we compute persuasion scores for each context according to Eq. (2) and
350 are real entities (e.g., Adele) sampled from YAGO, and 50 are fake entities (e.g., Udo König) of the same entity class (e.g., Person) generated with GPT-4 (OpenAI, 2023).
4E.g., Definitely, the capital of {entity} is {answer}. 5E.g., The capital of {entity} is {answer}. 6E.g., The capital of {entity} is not {answer}.
susceptibility scores for each entity according to Eq. (3) for each of the four query forms for six Pythia models of different sizes7 trained on the deduplicated Pile (Wolf et al., 2020; Dettmers et al., 2022; Biderman et al., 2023). Due to computational constraints, we approximate the model’s answer distribution with the next-token distribution over the model’s vocabulary.8 The detailed setup can be found in App. C.

heading: 4.2 Empirically Validating Our Metrics

heading: 4.2.1 Estimating Scores
text: Because persuasion and susceptibility scores both involve a countably infinite sum, we opt for a stochastic approximation scheme. Specifically, we construct a Monte Carlo estimator. We sample from a narrower set of constructed contexts and approximate the answer distribution with the next token distribution over the model’s vocabulary, as described in §4.1. We take a sample size of 600 contexts. While the Monte Carlo approximation itself results in a consistent estimator, the additional approximations mean we do not have a guarantee on the quality of the approximation as a whole. In Fig. 4, we exhibit the variance of our estimator across three random seeds, i.e., sampled sets of context.

heading: 4.2.2 Validating Persuasion Scores
text: Convergent Validity. According to existing measurement modeling methods (Loevinger, 1957; Messick, 1987; Jackman, 2008; Hand, 2004; Quinn et al., 2010; Jacobs and Wallach, 2021), observing a relationship between a new metric and existing ones would serve as additional evidence that the metric is meaningful. To this end, we explore whether contexts with higher persuasion scores tend to more successfully convince the model to agree with the context. Using the Pythia-6.9bdeduped model, we generate an answer for each prepended context to a query–entity pair from §4.1 and use simple string matching to map the answer to whether it agrees with the context, the original answer, or neither. We then apply a permutation test (k = 10000, α = 0.05 with the Benjamini–Hochberg (BH) correction (Benjamini and Hochberg, 1995)) for whether contexts that elicited context-concordant answers have higher persuasion scores than those that did not alter the
770m, 410m, 1.4b, 2.8b, 6.9b, 12b (8-bit quantized) 8While it would be more precise to estimate the answer distribution by repeatedly sampling many model outputs, this is very computationally expensive.
4
model from the original answer for each of the 122 query topics. Within a query topic, we run separate tests for open and closed queries, since the entropy of the answer distribution for closed queries tends to be much lower than the entropy for open queries. We find that for 59% of open queries, contexts that persuade the model to output the in-context answer have significantly higher persuasion scores than non-persuasive ones. Curiously, this behavior holds for only 34% of closed queries. We discuss this surprising result more in §6. We further show a summary of the mean persuasion scores for the different kinds of elicited answers in Fig. 2.
Construct Reliability. To show construct reliability, we provide evidence that persuasion scores do not strongly vary for inputs that differ only in phrasing, i.e., query form. For example, the persuasion score of The capital of Slovenia is Oz. should be similar when prepended to either Is Slovenia’s capital Ljubljana? or Is Ljubljana the capital of Slovenia?. We compute persuasion scores using the setup from §4.1 with two query forms for both closed and open queries, keeping contexts that appeared for the same query and entity for all seeds. We then compute the variance across the query forms to test for reliability. Fig. 4 shows strong evidence that the metric is reliable. The variance is very low across different closed query forms, as expected. While open query forms have higher variance, this is not unexpected because the questionanswering query form, e.g., Q: What is the capital of Slovenia?\nA: is more specific than the sentencecompletion form, e.g., The capital of Slovenia is, which has a broader set of plausible answers.

heading: 4.2.3 Validating Susceptibility Scores
text: Convergent Validity. We compare susceptibility scores to per-entity memorization ratio (MR)9 as further evidence for the meaningfulness of our metric, using the setup from §4.1 for the Pythia6.9b-deduped model. Fig. 3 shows a decreasing upper-bounding relationship between susceptibility scores and MR for both open queries and closed queries. While not a straightforward correlation, this pattern supports the convergent validity of susceptibility scores and can be explained by the scores’ nature of measuring a difference in entropy. That is, if the model’s answer was often changed (low MR), this could correspond to a wide range of entropy change (low or high susceptibility), depending on the model’s confidence without the context. If the model’s answer was mostly unchanged (high MR), the entropy likely remained similar (low susceptibility). We discuss these results further in App. D.2.
Construct Reliability. Following the setup used in §4.2.2, we consider the same two query forms for both open and closed questions to test for reliability (low variance) for susceptibility scores. Fig. 4 shows strong evidence for the reliability of susceptibility scores. Similarly to the persuasion scores, the variance for both open and closed queries is very low across closed query forms; open query forms have higher variance because they have meaningfully different possible answers, as discussed in §4.2.2.
9Adapted from Longpre et al. (2021) to apply on a perentity basis and describe it in more detail in App. D.
5
Seed
Query form Variance of susceptibility and persuasion scores

heading: 4.3 What Makes a Context Persuasive?
text: To better characterize model behavior with persuasion scores, we explore several tests for qualities that might distinguish more persuasive contexts from less persuasive ones: relevance, assertiveness, and negation.

heading: 4.3.1 Relevance
text: Experiment Setup. We use the setup described in §4.1. For the relevance test, we consider a relevant context to be one that mentions the queried entity, and an irrelevant context as one that does not. We hypothesize that relevant contexts should be more persuasive than irrelevant ones. For each of the 122 queries, we find the mean persuasion score of the relevant contexts and irrelevant contexts and use a permutation test to determine whether the mean persuasion score for relevant contexts (across entities) is higher than the mean persuasion score for irrelevant contexts, using a significance level of α = 0.05 with the BH correction.
Results. As seen in Fig. 5 (▼), across most model sizes and relations, relevant contexts are significantly more persuasive than irrelevant contexts. Specifically, depending on the model, 95–100% of open queries and 83–100% of closed queries showed a significant result. We also see a trend in which, as model size increases, so too does the degree to which relevant contexts are more persuasive, as measured by the mean effect size. We summarize the significance test results and effect sizes for all models and queries in App. E.

heading: 4.3.2 Assertiveness
text: Experiment Setup. Using a similar setup to §4.3.1, we explore whether assertive contexts are more persuasive than base ones by testing whether the mean persuasion score of the former group is greater than that of the latter for each query.
Results. Fig. 5 (■) shows that consistently across model sizes, assertive contexts tend to be more persuasive than base contexts for closed queries but not for open queries. Moreover, assertive contexts are most persuasive for medium-sized models such as 1.4b and 2.8b. We hypothesize that smaller models may be less persuaded by assertive contexts because they may be worse at integrating context into their answers. Larger models may be less persuaded because of their stronger prior knowledge/lower susceptibility to context, whether assertive or not. We highlight the significance test results and effect sizes for all queries for the Pythia-6.9b-deduped model in Fig. 6 and other models in App. E.

heading: 4.3.3 Negation
text: Experiment Setup. We use the same setup as in §4.3.2 and explore whether negation contexts differ in persuasiveness from the base ones, using a two-tailed permutation test for each query.
Results. The permutation tests suggest evidence for a significant difference in 88% of the closed queries and 74% of the open queries for the Pythia-6.9b-deduped model. However, there is no consistent directional pattern; from Fig. 7, we see that negations are significantly more persuasive for some queries while significantly less persuasive for others. App. E shows a similar pattern for other models. Fig. 5 (▲) also shows that the smallest model is the most sensitive to being persuaded differently by negation vs base contexts. For closed queries, this may be due to potential spurious correlations or token biases between, i.e., seeing the word not and the model’s probability of outputting No.

heading: 4.3.4 Comparing Context Qualities
text: For all models on open queries, the relevance of a context has the greatest effect on persuasion score compared to assertiveness or negation, as measured by effect size. This is consistent with our intuition that the model should be more sensitive to whether the queried entity is mentioned in the context than to other context features. However, we
6
Permutation test results across models and comparisons
can see in Fig. 5 that for medium-sized models on closed queries, the other context comparisons have stronger effect sizes, e.g., assertiveness and negation for the 2.8b model. This could be explained by potential spurious correlations/token biases associating, i.e., the word not with the model outputting No or the word definitely with outputting Yes.

heading: 4.4 What Makes an Entity Susceptible?

heading: 4.4.1 Familiar vs Unfamiliar Entities
text: We hypothesize that the model should have lower susceptibility scores for entities encountered during pretraining compared to unfamiliar, fake entities.
Experiment Setup. We use the setup described in §4.1 to compute susceptibility scores for known and unknown entities. We use a permutation test with α = 0.05 and the BH correction to test whether the known real entities are less suscepti-
ble than the unknown fake entities.10 The detailed setup can be found in App. C.
Results. For the Pythia-6.9b-deduped model, we find that with 73122 queries (open questions) and 61 122 queries (closed questions), familiar entities have significantly lower susceptibility scores than unfamiliar fake entities. We conjecture that for the remaining queries, the model may not have strong prior biases about the sampled entities for these queries. Indeed, further analysis (App. F.1) finds some evidence supporting the hypothesis that queries with smaller effect sizes or less significant p-values feature less familiar entities, as we find a small correlation between effect size and entity frequency in the training set. Fig. 8 shows the distribution of susceptibility scores for real and fake entities for an example query with a particularly strong effect size.
10We filter out fake entities that appear in the training data to better represent unknown entities in our analysis.
7
Generally, as model size increases, so too does the significance and effect size of unfamiliar entities being more susceptible than familiar entities, as seen by the generally increasing blue lines (•) in all four plots in Fig. 5. This trend is consistent with our expectation that bigger models have stronger prior knowledge of entities and are therefore less susceptible to familiar entities. The smallest model (70m) does not have a significant difference in susceptibility between familiar and unfamiliar entities, which could indicate it is too small to have strong prior knowledge.

heading: 4.4.2 Degrees of Familiarity
text: Training Data Frequency. Since language models are parameterized with knowledge from their training corpora, we hypothesize that the model is less susceptible for entities with which it is more familiar, i.e., more frequently occurring in the training data. We investigate this relationship between the Pythia models’ behavior and frequency statistics in the Pile dataset on which they were trained (Gao et al., 2020). To capture the model’s familiarity with an entity–answer relation, we count the number of co-occurrences between the entity and its corresponding answer within a 50- word window. We compare the susceptibility score to this co-occurrence frequency and find a significant correlation (Spearman ρ = −0.23) for the Pythia-6.9b-deduped model. We see in Fig. 9 that as the training data frequency increases, the susceptibility scores’ upper bound decreases. This trend is shared across all model sizes (see App. F.2).

heading: Susceptibility Score to Frequency
text: Entity Degree in a Knowledge Graph. Training data can be noisy, difficult, and expensive to search through for entity co-occurrences and more complex frequency statistics. Knowledge graphs offer a more precise alternative as they represent entities and relations extracted from common corpora like Wikipedia in a structured way. For example, within the pretraining data, it is very difficult to identify the number of different answers with which an entity will co-occur within the context of a specific relation. However, with a knowledge graph, we can easily identify the exact number of objects with which an entity might share a given relation. Thus, we explore the relationship between the relationdependent degree of an entity in a knowledge graph and the susceptibility score. Like with training data frequency, we find that comparing against this degree yields a similar-looking plot with a decreasing upper bound between susceptibility scores and the degree. Such a trend suggests that unfamiliar entities have the potential to be highly susceptible, while very familiar entities tend to have lower susceptibility. In App. F.2, we show this trend is shared across all model sizes and provide our methodology in more detail.

heading: 5 Applications
text: We examine how knowing susceptibility scores can be useful for analyzing model behavior in two different applications. Here, we highlight one key finding per application; however, we emphasize that future work can conduct a more detailed analysis of these and other applications.
Social Sciences Measurement. Social scientists use large language models (LLMs) for annotating data and descriptive data analysis, yet such use may
8
inadvertently incorporate entity biases and skew model behavior (Ziems et al., 2024; Gilardi et al., 2023; Zhang et al., 2023; O’Hagan and Schein, 2023). To better contextualize model annotations for a case study on friend–enemy stance detection (Choi et al., 2016; Stoehr et al., 2023), we aim to understand how susceptibility scores may differ between friend and enemy-based entity pairs for the query The relationship between {entity1} and {entity2} is, e.g., are famous friend-based relationships more susceptible than enemy-based relationships? From Fig. 10, we see that with the Pythia-6.9b-deduped model for two specific query forms, enemy duos are less susceptible than friend duos, which can inform social scientists that a model’s annotation for friend pairs may be more easily influenced by the context than enemy pairs. We provide more details in App. G.
Exploring Gender Bias. Since higher susceptibility scores indicate weaker induced biases for entities, we conjecture that this can relate to being underrepresented in the training data. Based on this, we consider how the susceptibility score can be used to study gender bias in LLMs. Using GPT-4, we collect stereotypically biased contexts, gendered names, and neutral queries and run several experiments to identify gender discrepancies in susceptibility scores; see App. G for full details. We highlight a result where masculine names have a higher susceptibility than feminine names when swapping the genders in the stereotypical contexts, as seen in Fig. 11. This could indicate that the model is more surprised to see contexts claiming men follow feminine stereotypes, and therefore could suggest less representation of feminine stereotypes in the training data than masculine ones.

heading: 6 Discussion and Conclusion
text: We have made the case that the persuasion score and susceptibility score are both valid and reliable in measuring their respective constructs, following a well-established measurement modeling framework. Throughout our experiments, we find a common theme in the results: there is a strong, negative, possibly linear relationship between the upper bound of the susceptibility score and (a) the entity’s memorization ratio (Fig. 3), (b) the log-co-occurrence frequency in the training data (Fig. 9), and (c) the log-relation-dependent degree in a knowledge graph. That is, as each of those three values increases, we see a clear pattern indicating that the highest susceptibility scores tend to decrease. This is consistent with our hypothesis that the induced bias of an entity increases for a model as the model’s expected familiarity with the entity increases. Furthermore, we find a difference in behavior between scores for open and closed queries; while in many experiments, we see similar patterns between the two, susceptibility and persuasion appear to have a stronger relationship with memorization ratio for open queries, while assertive contexts appear to be significantly more persuasive than base contexts primarily for closed queries. This difference is a surprising phenomenon that warrants future study; we hypothesize closed queries may behave this way due to common token biases (with its output space of Yes and No) or the influence of mentioning both an entity and an answer in the query. Finally, while we applied these metrics to analyze model behavior in two case studies, in future work we aim to apply them to unearth new perspectives in other context and prior knowledge-dependent problems such as retrieval-augmented generation, model editing and control, and few-shot learning.
9

heading: Limitations
text: We face some technical limitations in executing the empirical aspects of this work. First, while §3 defines the output space of A as the set of all possible outputs Σ∗, in practice, it is computationally expensive to estimate that probability distribution. Instead, we look only at the model’s probability distribution of the next token, which could be a noisy signal, especially in cases where the answer suggested by a context and the answer suggested from prior knowledge share the same first token. Second, it is difficult to go through the whole Pile to count answer–entity co-occurrences without noise. Third, the scores depend on the sampled contexts, which may not be representative of all applications.

heading: Ethics Statement
text: As LLM capabilities grow more advanced and their usage proliferates throughout the real world, we acknowledge that their development can exacerbate risks to people, especially those historically underrepresented or misrepresented to these models. Our work aims to make model behavior more transparent by providing a new tool to analyze the interaction between context and prior knowledge in LMs, which is especially important as people interact with them in chat, question-answering, and other prompt-based settings. We foresee no particular ethical concerns and hope this paper contributes to developing tools that can identify and mitigate ethical concerns in the future.

heading: Acknowledgements
text: We thank Alex Warstadt, Tiago Pimentel, Anej Svete, Alexandra Butoi, Benjamin Dayan, and Leo Du for helpful comments, discussion, and feedback. Niklas Stoehr acknowledges funding through the Swiss Data Science Center (SDSC) Fellowship. Vésteinn Snæbjarnarson is supported by the Pioneer Centre for AI, DNRF grant number P1.

heading: A A Primer on Half-pointwise Mutual Information
text: Half-pointwise mutual information (HPMI) is a non-standard concept. There is, for example, no mention of it in standard references on information theory (Cover and Thomas, 2006). In this brief primer, we give various properties of HPMI and show how it relates to other concepts in information theory. Given random variable X over the discrete space X , random variable Y over the discrete space Y , and x ∈ X , the half-PMI of X = x, Y is defined as:
HPMI(X = x;Y ) ≜ ∑ y∈Y p(y | x) log p(y | x) p(y) (4)
such that
E x∼X [HPMI(X = x;Y )] = E x∼X
 ∑
y∈Y p(y | x) log p(y | x) p(y)
  (5a)
= ∑
x∈X
∑
y∈Y p(x)p(y | x) log p(y | x) p(y) (5b)
= ∑
x∈X
∑ y∈Y p(x, y) log p(x, y) p(x)p(y) (5c)
= MI(X;Y ) (5d)
We now state and prove three Propositions about HPMI. Proposition 1. Given random variable X over the discrete space X , random variable Y over the discrete space Y , and x ∈ X , then:
HPMI(X = x;Y ) = H(X = x)−H(X = x | Y ) (6)

heading: Proof.
text: HPMI(X = x;Y ) ≜ ∑ y∈Y p(y | x) log p(y | x) p(y) (7a)
= ∑
y∈Y p(y | x) log p(x, y) p(x)p(y) (7b)
= ∑ y∈Y p(y | x) log p(x | y) p(x) (7c) = − ∑
y∈Y p(y | x) log p(x) +
∑ y∈Y p(y | x) log p(x | y) (7d)
= − log p(x) + ∑
y∈Y p(y | x) log p(x | y) (7e)
= H(X = x)−H(X = x | Y ) (7f)
Note that to get from Eq. (7e) to Eq. (7f), H(X = x | Y ) ≜ ∑y∈Y p(y | x) log p(x | y) because
E x∼X
 ∑
y∈Y p(y | x) log p(x | y)
  = ∑
x∈X
∑ y∈Y p(x, y) log p(x | y) (7g)
= ∑
x∈X
∑ y∈Y p(x, y) log p(x, y) p(y) (7h)
= H(X | Y ) (7i) ■
13
Proposition 2. Given random variable X over the discrete space X , random variable Y over the discrete space Y , and x ∈ X , then:
HPMI(X = x;Y ) = Hx(Y )−H(Y | X = x) (8)
where Hx(Y ) ≜ − ∑
y p(y | x) log p(y) is the pointwise cross-entropy between X = x and Y .

heading: Proof.
text: HPMI(X = x;Y ) ≜ ∑ y∈Y p(y | x) log p(y | x) p(y) (9a)
= − ∑
y∈Y p(y | x) log p(y) +
∑ y∈Y p(y | x) log p(y | x) (9b)
= Hx(Y )−H(Y | X = x) (9c)
■
Notably, HPMI(X = x;Y ) ̸= H(Y ) − H(Y | X = x). Furthermore, while the decomposition of mutual information into a difference in entropies is symmetric, i.e., H(X)−H(X | Y ) = H(Y )−H(Y | X), this shows that half-pointwise mutual information is not, i.e., H(X = x) − H(X = x | Y ) ̸= H(Y )−H(Y | X = x). Proposition 3. Given random variable X over the discrete space X , random variable Y over the discrete space Y , and x ∈ X , then:
HPMI(X = x;Y ) = KL (p(Y | x) || p(Y )) (10) Proof. Half-PMI is equivalent to KL (p(Y | x) || p(Y )) by definition. ■ Corollary 1. Half-PMI is nonnegative.
Proof. Since half-PMI is equivalent to KL (p(Y | x) || p(Y )) (Proposition 3) and KL-divergence is nonnegative, half-PMI must be non-negative. ■

heading: B An Entity-Independent Persuasion Score
text: In addition to the persuasion score, which is entity-dependent, we also consider an entity-independent extension. We assign an entity-independent persuasion score κ to a context c which represents how persuasive a context is at altering a model’s answer distribution to a query, regardless of which entity parameterizes the query. One might be interested in an entity-independent persuasion score for contexts like Only give wrong answers to questions., which we might expect to affect all queries regardless of the entity. Another use case is in comparing the persuasiveness of context templates, such as {entity1} loves {entity2} and {entity1} really really really loves {entity2} for the query What’s the relationship between {entity1} and {entity2}?. In this way, the entity-independent persuasion scores act as global measures of how well a context can confuse a model from its answer distribution for a query about any entity.
Analogous to our definition of the susceptibility score, we define the entity-independent persuasion score of a context as how much the log probability distribution of possible answers changes, averaged across all possible entities and answers. More precisely, we define our entity-independent persuasion score κ(c, q) as:
κ(c, q) ≜ ∑
e∈E p(q(e) | c)ψ(q(e)) (11a)
= ∑
e∈E
∑
a∈Σ∗ p(q(e) | c)p(a | c, q(e)) log p(a | c, q(e)) p(a | q(e)) (11b)
= ∑
e∈E
∑
a∈Σ∗ p(q(e) | c)p(a | c, q(e)) log p(a, c | q(e)) p(a | q(e))p(c | q(e)) . (11c)
14
which is the half-conditional PMI. Further marginalizing out the context, we arrive at the entityindependent susceptibility score for a query:
γ(q) ≜ E c∼C [κ(c, q)] (12a)
= ∑
c∈Σ∗ p(c)κ(c, q) (12b)
= ∑
c∈Σ∗ p(c)
∑
e∈E
∑
a∈Σ∗ p(q(e) | c)p(a | c, q(e)) log p(a, c | q(e)) p(a | q(e))p(c | q(e)) (12c)
= ∑
c∈Σ∗
∑
e∈E
∑
a∈Σ∗ p(a, c, q(e)) log
p(a, c | q(e)) p(a | q(e))p(c | q(e)) (12d)
= MI(A;C | q(E)). (12e)
The entity-independent persuasion score differs from the persuasion score by additionally marginalizing over the entities. As the half-PMI is conditioned on the entity random variable, this tells us, when we already know the entity, for a given context, how much more confident can we be in the answer. In some sense, then, this can be interpreted as the average persuasiveness of a context across all entities for the query.

heading: C Detailed Experimental Setup
text: We extract 122 relations from the YAGO knowledge graph (Suchanek et al., 2007), such as alumniOf, capital, and highestPoint. For each relation, we do the following:
• We randomly sample k real entities (and corresponding answers) from YAGO and use GPT-4 11
(OpenAI, 2023) to generate k fake entities with the same entity class as the real ones12.13
• We construct open and closed query form templates, e.g., (closed) Q: Is {answer} the capital of {entity}?\nA: and (open) Q: What is the capital of {entity}?\nA:, and parameterize them with both real and fake entities (and answers, if applicable), leaving us with 2k queries per query form.
• We construct context templates of 3 types: base, e.g., The capital of {entity} is {answer}., assertive, e.g., The capital of {entity} is definitely {answer}., and negation (e.g., The capital of {entity} is not {answer}.). We parameterize these context templates with both real and fake entities (and answers, if applicable). From this, we randomly sample 6k contexts, subject to the constraint that each entity is directly mentioned in 6 contexts total (that is, in 2 assertive contexts, 2 base contexts, and 2 negation contexts).
• We compute the persuasion scores ψ(c, q(e)) for the real and fake entities according to Eq. (2).
• We compute the susceptibility score χ(q(e)) for the real and fake entities according to Eq. (3). We approximate C with a uniform distribution over the set of sampled contexts and A with the model’s next token probabilities.
• For the various group comparisons (e.g., relevant vs irrelevant contexts, familiar vs unfamiliar entities, etc.), we use a permutation test over the t-statistic (α = 0.05, with the BH correction) to test our null hypothesis for each comparison.
11gpt-4-1106-preview, January 2024 12Entity classes: CreativeWork, Event, Intangible, Organization, Person, Place, Product, Taxon, and FictionalEntity. 13Real example: Adele. Fake example: Udo König.
15

heading: D Entity-Specific Memorization Ratio

heading: D.1 Definition
text: The memorization ratio (MR), as used by Longpre et al. (2021), is defined as follows: Given a set of (query, context)-pairs with knowledge conflicts (i.e., the answer in the context disagrees with the original answer), MR = popo+ps , where po is the number of queries for which the model returned the original answer and ps is the number of queries for which the model returned the substitute answer presented in the context. Then, the entity-specific MR follows this definition under the additional constraint that the query and entity are fixed, and we vary only the contexts; the resulting number tells us, for a given query about an entity, the fraction of contexts for which the model returned the original answer instead of the substitute one.

heading: D.2 Further Discussion on Relation between MR and Susceptibility Score
text: In Fig. 3, the open queries further show a decreasing pattern at each quartile for all bins except the lowest one (MR between 0 and 0.2). The lowest bin includes queries for which a model may fail to know the original answer; in these cases, MR cannot distinguish between the model behavior for these difficult queries, whereas susceptibility scores can provide more granular information about model behavior for such entities. Meanwhile, the closed queries appear to have a mostly increasing pattern at each quartile across the bins. This result could be an artifact of the construction of the closed queries. Since the original answer of all closed queries is Yes, it is possible that the contexts increase the confidence in Yes due to some model artifact or token bias, which would explain higher susceptibility scores even for higher MR.

heading: E Persuasion Scores: In-Depth Results

heading: E.1 Relevant vs Irrelevant Context Persuasion Scores Across Models
text: Our null hypothesis is that the mean persuasion score of relevant contexts is not greater than that of irrelevant contexts. We summarize the test results (effect size and p-values) for all queries for all models in Fig. 12.

heading: E.2 Assertive vs Base Context Persuasion Scores Across Models
text: Our null hypothesis is that the mean persuasion score of assertive contexts is not greater than that of base contexts. We summarize the test results (effect size and p-values) for all queries for all models in Fig. 13.

heading: E.3 Negation vs Base Context Persuasion Scores Across Models
text: Our null hypothesis is that the mean persuasion score of negation contexts is not equal to that of base contexts. We summarize the test results (effect size and p-values) for all queries for all models in Fig. 14.

heading: F Susceptibility Scores: In-Depth Results

heading: F.1 Unfamiliar vs Familiar Entity Susceptibility Scores Across Models
text: Our null hypothesis is that the mean susceptibility score of unfamiliar entities is not greater than that of familiar entities. We summarize the test results (effect size and p-values) for all queries for all models in Fig. 15. From this figure, we can see the trend of how effect size and percentage of significant queries generally increase with model size, and notably the smallest model has no significant results for any query. However, even the larger models do not exhibit significant differences in scores between unfamiliar and familiar entities for all queries. To investigate the spread further, we plot the p-values and effect size against the entity frequencies in the Pile. The results are presented in Fig. 16. There is a significant trend for the open queries against the frequency (spearman, ρ is −0.23, p < 0.05), showing that real entities tend to be less susceptible the more frequently they appear in the training data. The trend is not significant for the closed set.
16
17
18

heading: F.2 Training Data and Susceptibility Scores
text: Since language models are parameterized with knowledge from their training corpora, we examine whether we can identify correlations between patterns in entity susceptibility scores and their prevalence in the training data. Because the Pythia models are all trained on the Pile dataset (Gao et al., 2020) in a single pass, we choose to compare the susceptibility scores from the Pythia models to various frequency and co-occurrence statistics in the Pile.14
Experiment Setup. Our goal is to understand how the susceptibility score relates to the frequencies of entities and their co-occurrences in the training data. For this, we use all of the entities and answers selected as described in §4.4 and locate them in the Pile. We only perform rudimentary tokenization of each document by removing punctuation and splitting at white spaces, but find this suffices to locate the exact terms. As a sanity check, we annotate named entities in 30k documents and cross-reference the list of entities. If a supposed entity has a fairly high frequency (>50) and is most often (>75%) not labeled as an entity, we exclude it from the calculations. This removes ∼200 entities that are high-frequency non-entity words in English and lowers the co-occurrences by 25%. Finally, we calculate the token distance for each entity-answer pair for every document in ∼1/3 of the Pile.
Results. We compare the co-occurrences of the entity-answer pairs to the averaged susceptibility scores over all queries Qe that apply to the entity e, |Qe|−1Σq∈Qeχ(q(e)). Our results show a stark difference in behavior between the open and closed questions (§4.4). The open questions not only have higher susceptibility scores, but the training corpus frequency of the mentioned entities influences them more. This is to be expected as there are far more probable candidates for open questions than for closed yes–no-style questions. We also find a significant negative correlation (Spearman ρ -0.23, p ≃ 0) between frequency and susceptibility scores for the pythia-6.9b-deduped model, indicating that the language model is less susceptible to context interference for entity-answer pairs that are more frequently found in the training corpus. See Fig. 9 and Fig. 17. Finally, we also notice a big difference in the rank correlation depending on the query type. Some query types are more susceptible to context for the given entities than others. An overview of this is given in the next section.
14We use the same deduplicated 825GiB version of the Pile that the Pythia-6.9b model was trained on. https://huggingface. co/datasets/EleutherAI/the_pile_deduplicated.
19
20
21
Susceptibility and Frequency Analysis We are interested in seeing how different predicate relations of the queries from the knowledge graph have different susceptibility scores. We evaluate the relation between susceptibility scores and co-occurrence frequencies per entity-answer predicate relation. This reveals trends in what types of relations are more susceptible to context than others. A lower correlation indicates a stronger drop in susceptibility as the entities become more frequent in the pretraining data.

heading: F.3 Entity Degrees and Susceptibility Scores
text: Since knowledge graphs are structured conceptual maps relating entities, we further seek to identify whether we can identify any correlations between statistics of the YAGO knowledge graph and an entity’s susceptibility scores. Validating against a knowledge graph can be advantageous over validating against the actual training data for a number of reasons, including that for most models, the actual training data is inaccessible for research, and the scale of the training data can make it prohibitively expensive to trawl through efficiently and precisely. For example, with a knowledge graph, we can identify the exact number of objects an entity might share an alumniOf relation with, while within the pretraining data, it is very difficult to identify the number of different answers an entity will co-occur within the context of the specific alumniOf relation.
Experiment Setup. Our goal is to understand how the susceptibility score relates to the degree of entities in the YAGO knowledge graph G for specific queries. For this, we extract the number of incoming and outgoing edges from an entity e along a relation (or query) q as follows: δ(e, q) = |{a | (e, q, a) ∈ G} ∪ {a | (a, q, e) ∈ G}|. We plot δ(e, q) against the susceptibility score χ(q(e)) for all entities and queries.
Results. From Fig. 18, we see a decreasing upper bound relationship between susceptibility scores and the YAGO degree δ for both open and closed queries for all model sizes. This could be explained as follows: consistent with our original hypothesis, very familiar entities to a model have low susceptibility, while less familiar entities can have a wider range of susceptibility. The potential for unfamiliar entities to be susceptible is much higher than that for very familiar entities, although unfamiliar entities can also be less susceptible. Further investigation into traits that characterize the susceptibility of familiar vs less familiar entities is needed.

heading: G Applications

heading: G.1 Social Sciences Measurement
text: Motivation. Large language models (LLMs) are actively used today in empirical social sciences for annotating data and descriptive data analysis (e.g., classifying tweets with sentiment and ideology scores (Ziems et al., 2024; Gilardi et al., 2023)). However, Zhang et al. (2023) warn that LLMs applied to sentiment classification “may inadvertently adopt human biases” and demonstrate that the prompt design, i.e., the context persuasiveness, can significantly influence the outcome. O’Hagan and Schein (2023) demonstrate that LLMs exhibit biases about different entities when measuring political ideology, based on their prior knowledge. Finally, Stoehr et al. (2024) use LLMs to measure the stance of product reviews, their setting does not disentangle the effects of prior knowledge and context, thus leaving ambiguous the question of whether the measurement is more because of the LLM’s prior bias about the product or the review’s actual content.
Experiment Setup. We consider a manually constructed dataset (Tab. 1) of well-known entity-pairs which are either friends (e.g., Harry Potter and Ron Weasley) or enemies (e.g., David and Goliath) and contexts relating the pair (e.g., Harry loves Ron). We aim to understand how susceptibility scores may differ between the two kinds of relationships for the query What’s the relationship between {entity1} and {entity2}?, e.g., are famous friend-based relationships more susceptible than enemy-based relationships? We compute the susceptibility scores for these entity-pairs using simple template-generated contexts such as {entity1} loves {entity2} and {entity1} hates {entity2} and then analyze the results.
22
Results. From Fig. 10, we can see that for the open query The relationship between {} and {} is, there is a clear difference in susceptibility scores for friend and enemy entity-pairs. We leave to future work investigating the exact nature of why some entity-pairs may have lower susceptibility scores than others.

heading: G.2 Exploring Gender Bias
text: Motivation. The susceptibility score tells us how the entropy changes for a given entity as we vary the prepended contexts. We posit that stronger induced biases for entities lead to lower susceptibility scores than for those less represented in the training data. Using this information, we consider how the susceptibility score can be used to study gender bias in LLMs.
Experiment Setup. We collect sentences containing 20 highly stereotypical sentences about men and women, respectively, to use as contexts. For instance, Men are not good at multitasking. We also collect 20 names for each gender and 20 neutral queries that we think may be affected by the contexts, such as {entity} spends most of their time. Finally, we also invert the queries by flipping the gendered words, giving us non-stereotypical contexts. We then calculate susceptibility scores for each query and gender combination over the four different context collections. All scores are calculated using the Pythia-6.9bdeduplicated model. All data was generated with GPT-4. The contexts, names, and queries can be found below.
Results. Results are shown in Fig. 11. For all contexts, there is a significant difference between the genders (p < 10−6). We make several observations: (i) When prompted with the masculine stereotypes, we get the lowest susceptibility for the male entities, indicating the model has a bias for the stereotypical male contexts. (ii) By swapping the genders in the stereotypical female contexts (M*), we get the highest susceptibility for the male entities. Somewhat surprisingly, we also find that this gives the lowest susceptibility scores for the female names. (iii) The susceptibility scores for the female names over the stereotypical female contexts (F) are similar to the swapped ones (F*), hinting that the model does not have as strong a bias for the feminine stereotypes as in the masculine case in (i). Combined with our prior findings on the relation between susceptibility and the saliency of entities, this hints at masculine stereotypes being more prominently represented in the training data (as well as in the model’s induced biases). At the same time, we don’t see evidence of exaggerated feminine stereotypes being favored via lower susceptibility scores.

heading: G.2.1 Gender Stereotypes Data
text: Here, we list the data used to run the gender bias use case. All data was generated with the help of the GPT-4 LLM.
The female names are Mary, Jennifer, Linda, Patricia, Elizabeth, Susan, Jessica, Sarah, Karen, Nancy, Margaret, Lisa, Betty, Dorothy, Sandra, Ashley, Kimberly, Donna, Emily, Michelle. The male names are James, John, Robert, Michael, William, David, Richard, Joseph, Thomas, Charles, Christopher, Daniel, Matthew, George, Donald, Anthony, Paul, Mark, Andrew, Edward.
The stereotypical feminine contexts are: Women are not as strong as men., All women are overly emotional and sensitive., Women are not good at math and sciences., All women are naturally good at cooking and cleaning., Women talk more than men., Women are generally not capable of doing manual labor., All women love shopping and fashion., Women don’t understand technology as well as men do., Women are indecisive and change their mind too often., Women cannot drive as well as men., Women are not as ambitious as men when it comes to career., Women overthink things more than men., Women are not as good at sports as men., Women are more gossipy than men., All women are afraid of bugs and spiders., Women are not good at directions and get lost easily., Women take longer to get ready than men., Women always need to ask for others for their opinion., All women are obsessed with their weight and dieting., All women are crazy about babies and want to have kids.
The stereotypical masculine contexts are Men always bring the association of strength and aggression., All men are inherently messy and cannot keep their spaces clean., Men are not good at multitasking., All men are naturally good at sports., Men cannot cook as well as women., Men do not take care of their health as much as women do., I believe men are not as talkative as women., All men love video games.,
23
Men are not good listeners., Men are always competitive., Men are not as emotional as women., Men are not able to express their feelings as well as women do., Men only think about one thing., Men look at problems from a logical perspective, not considering emotions., Men are typically less organized than women., Men tend to make decisions without considering others’ feelings., All men only care about sports and cars., Men don’t like to ask for help., Men don’t pay as much attention to appearance as women do., All men can’t resist a flirt.
The queries used for both sets of entities are: I think {entity} works as a, {entity} is good at, I met {entity} yesterday, they are, {entity} usually spends their free time, {entity}’s favorite hobby is likely, {entity} probably lives in a, When it comes to {entity}’s fashion style, they prefer, {entity}’s educational background might be in, {entity} often enjoys eating, If {entity} was a colour, they would be, The one thing that {entity} cannot resist is, {entity} probably drives a, One thing that {entity} is really bad at is, {entity}’s personality can be described as, {entity} spends most of their time, {entity} probably likes to watch, During weekends, {entity} might be, {entity}’s favorite type of music is probably, I imagine {entity} has a pet, it is a, {entity}’s favorite type of movie is probably.
24
25

heading: Future Work
text: We leave to future work investigating the exact nature of why some entity-pairs may have lower susceptibility scores than others