A Causal View of Entity Bias in (Large) Language Models
Authors: Fei Wang
Wenjie Mo
Yiwei Wang
Wenxuan Zhou
Muhao Chen
Venue: EMNLP 2023 Findings (2023)
Keywords: entity bias
knowledge conflicts
causal analysis
large language models

Abstract: Entity bias widely affects pretrained (large) language models, causing them to rely on (biased) parametric knowledge to make unfaithful predictions. Although causality-inspired methods have shown great potential to mitigate entity bias, it is hard to precisely estimate the parameters of underlying causal models in practice. The rise of black-box LLMs also makes the situation even worse, because of their inaccessible parameters and uncalibrated logits. To address these problems, we propose a specific structured causal model (SCM) whose parameters are comparatively easier to estimate. Building upon this SCM, we propose causal intervention techniques to mitigate entity bias for both white-box and black-box settings. The proposed causal intervention perturbs the original entity with neighboring entities. This intervention reduces specific biasing information pertaining to the original entity while still preserving sufficient semantic information from similar entities. Under the white-box setting, our training-time intervention improves OOD performance of PLMs on relation extraction (RE) and machine reading comprehension (MRC) by 5.7 points and by 9.1 points, respectively. Under the black-box setting, our in-context intervention effectively reduces the entity-based knowledge conflicts of GPT-3.5, achieving up to 20.5 points of improvement of exact match accuracy on MRC and up to 17.6 points of reduction in memorization ratio on RE.

A Causal View of Entity Bias in (Large) Language Models
Fei Wang' Wenjie Mot Yiwei Wang Wenxuan Zhou' Muhao Chen
"University of Southern California; UUniversity of California, Los Angeles;
#University of California, Davis (fwang5988, jackymo, muhchen@ucdavis..c edu
edu; wangyw. wrrgyw.vva@@gaill.. com;
Abstract
Context: Bill Gates went to Microsoft Building 99.
Entity bias widely affects pretrained (large) lan- guage models, causing them to rely on (biased) parametric knowledge make unfaithful pre- dictions. Although causality-inspired methods have shown great potential to mitigate entity bias, it is hard to precisely estimate the param- eters of underlying causal models in practice. The rise of black-box LLMS also makes sit- uation even worse, because of their inaccessible parameters and uncalibrated logits. To address these problems, we propose specific struc- tured causal model (SCM) whose parameters are comparatively easier to estimate. Building upon this SCM, we propose causal interven- tion techniques to mitigate entity bias for both white-box and black-box settings. The pro- posed causal intervention perturbs the original entity with neighboring entities. This interven- tion reduces specific biasing information per- taining the original entity while still preserv- ing sufficient semantic information from sim- ilar entities. Under the white-bor box setting, our maining-time intervention improves OOD per- formance of PLMs on relation extraction (RE) and machine reading comprehension (MRC) by 5.7 points and by 9.1 points, respectively. Under the black-box setting, our in-context in- tervention effectively reduces the maatt knowledge conflicts of GPT-3.5, achieving up 20.5 points of improvement of exact match accuracy on MRC and up to 17.6 points of re- duction in memorization ratio on RE.!
Question: What's the relation between Bill Gates and Microsoft in the given context?
Option: founder, visitor. Answer with one word: founder (GPT-3.5) X
Assume subject_ entiity can be any of Bill Gates, Jeff Bezos, and Steve Jobs, while object_entity can be any of Google, Microsoft, and Meta.
Context subject entity went to object entity Building 99. Question: What's the relation between subject_entity and abject_entity in the given context?
Option: founder, visitor.
Answer with one word: visitor (GPT-3.5)
Figure 1: An example entity bias GPT-3.5. Our in-context intervention mitigates the conflicts between parametric knowledge and contextual knowledge.
may be misled by their memory of the entities Bill Gates and Microsoft. saying the relation between them in this context is founder rather than visitor, as shown in Fig. 1. Recent studies show that entity bias widely affects pretrained (large) language mod- els (LLMs; Longpre et al. 2021; Yan et al. 2022; Zhou et al. 2023). These models have a tendency to disregard contextual information that contradicts or is infrequently reported in the pretrained corpus, while excessively relying on (biased) parametric knowledge (Longpre et al., 2021) to make unfaith- ful predictions and perpetuate bias.
Prior studies have proposed multiple causality- inspired methods to mitigate entity bias (Zhang et al., 2017: Nan al., 2021; Wang et al., 2022: Zhu et al., 2022).2 Despite their potential, the causal models underlying these methods are flawed practice, primarily because of imprecise parame- ter estimation. For example, some causal models necessitate estimating the probability distribution
1 Introduction
Entity bias (Longpre et al., 2021; Wang et al., 2022;
Xu et al., 2022; Peng al., 2020; Qian et al., 2021b; Hermann et al., 2015) refers to an undesirable phe- nomenon where models overly rely on prediction shortcuts triggered by specific entities to make spu- rious predictions. For example, given the sentence BBlll Gates went to Microsoft Building 99," models
? AAlhoughh Zhang et al. (2017) do not mention causal the- ory, the proposed entity masking does follow a relevant princi- plc to cut off causal links between specific entities and labels.
OOur cnde is available at https: https:///ittubb.oom luka- group/Causal-viiieeec Waeeaaasslllieeee-ee http://uallliieooo--tittttt///// of-Entity-Biiittt

---

over labels when given sentence that is devoid of entities or contextual information (Zhang et al., 2017; Wang et al., 2022). These methods either lose predictive information about entities, or are prone to erroneous representation without contextu- alization. The other critical problem is the difficulty of applying these methods to black-box LLMs, of which parameters are inaccessible and logits are uncalibrated.
E
E
X
X
CoRE
Entity Substitution
E
X: Raw Text
Y: LLM Output
E: Entity
M: LLM Input
To address the aforementioned problems, the first contribution of this paper is a causal analy- sis of entity bias mitigation methods ($3.1). We examine and compare the structured causal mod- els (SCMs) behind existing methods. We find that, among the theoretically equivalent causal models (Verma and Pearl, 1990), there exists specific SCM whose parameters to estimate. As shown in Fig. 2, the proposed SCM only requires to intervene input entities to mitigate the presence of spurious features before passing them to the subsequent neural layers. Moreover, it retains the entity type information at an appropri- ate level of granularity without requiring explicit entity typing.
X
M
Ours
Figure 2: Structured causal models revealing entity bias.
by examples - set of similar entities. For ex- ample, we can replace Bill Gates in Fig. 1 with subject_ent _ettctt_eettyyteeee and prepend the prompt, "Assume that subject_ _etiitt__ttytee can be any of Steve Jobs, Bill Gates, and Jeff Bezos", to the input. This in-context intervention can be applied to any black-box LLM without additional cost.
comparatively easier
Experiments on relation extraction (RE) and ma- chine reading comprehension (MRC) show that the proposed causal intervention techniques are ef- fective for both white-box and black-box LLMs. Under the white-box setting ($4), our training- time intervention significantly improves out-of- distribution performance of RoBERTa (Liu et al., 2019) on RE by 5.7 points and SpanBERT (Joshi et al., 2020) on MRC by 9.1 points, comparing with the vanilla version. Under the black-box set- ting ($5), in-context intervention effectively duces the entity-based knowledge conflicts (Long- pre et al., 2021) and improves the task performance of GPT-3.5." Specifically, our method outperforms the best bascline by up to 20.5 points of exact match accuracy on MRC and reduces the mem- orization ratio by up to 17.6 points on RE. Further analyses reveal the crucial role of the number of neighboring entities k in balancing the predictive information and biasing information from entities. and the necessity of entity placeholder definition for in-context intervention.
The second contribution of this paper is training-time causal intervention technique for mitigating entity bias based on the proposed SCM ($3.2). Specifically, we identify entities that are likely to share similar predictive information with the given entity. During training, we perturb em- bedding of the given entity within convex hull constructed by embeddings of similar entities. Dur- ing inference, we represent the entity with the cen- ter of the convex hull. Taking advantage of the continuous nature of the embedding space, this intervention does not rely on models specifically trained on natural language to estimate the label distribution of unnatural text, nor does it sacrifice predictive entity or contextual information.
The third contribution of this paper to trans- form the training-time time intervention into in-context intervention for black- LLMs whose param- eters are inaccessible, and logits are uncalibrated ($3.3). A significant advantage of the proposed SCM is that the causal intervention is carried out at the input layer, enabling its implementation within an in-context setting. Specifically, replace enti- ties with placeholders and define each placeholder
2 Related Work
Entity Bias in LLMs. LLMs memorize factual knowledge in their parameters during pretrain- ing (Roberts et al., 2020; Jiang al., 2020) and show promising results in answering factual ques- tions (Petroni et al., 2019; Brown al., 2020: Wei
'Entity type information plays a crucial role in entity- driven tasks. For example, without knowing a more specific location type, it is impossible to differentiate between relations born in city and born_ in _o__cuntry.
"htps://platfommooppp ppppmaa..omm/do////aall///saaaaa

---

al., 2022). However, the parametric knowledge may be inaccurate due to the misinformation in the training corpus (Lin et al., 2022) or outdated as the world evolves (Liska et al., 2022; Kasai et al., 2022). In such scenarios, is critical for LLMs to update their predictions when provided with contextual evidence. However, previous stud- ies (Longpre et al., 2021; Qian et al., 2021b; Yan et al., 2022) observe that language models may take entities as shortcuts, leading to spurious predictions based solely on parametric knowledge. This bias becomes more prominent when the evidence con- tains infrequent or conflicting knowledge compared to the training corpus.
Typically, the de-confounder process applies the do-calculus (Pearl, 2012) to compute the prediction assuming that the value of the confounder variable is not the observed one but follows its natural dis- tribution (Zhang et al., 2020a; Tian et al., 2022). Our approach is also based on confounder analysis. While nearly all the aforementioned approaches request white-box accessibility of the model with at least logits of predictions, this work represents pilot study of deconfounder method that applies purely black-box LLMs.
3 Method
In this section, first analyze methods for miti- gating entity bias in causal view and propose easy-to-estimate SCM as a theoretical basis ($3.1). Based on the proposed SCM, we design a training- time intervention technique for white-box LLMs ($3.2) and an in-context intervention technique for black-box LLMs ($3.3).
To mitigate this bias, previous work (Longpre et al., 2021; Chen et al., 2022; Li et al., 2022; Zhou et al., 2023) introduces the entity substitution tech- nique, which involves constructing counterfactual data by randomly replacing the entities, and up- dating the language models either by finetuning in-context learning, Although showing improved results, these techniques are empirical and lack theoretical backgrounds. In this paper, we theo- retically analyze the entity bias problem from a causal view. Furthermore, we propose causal in- tervention method that surpasses the performance of entity substitution.
3.1 Causal Analysis of Entity Bias
To compare existing methods in the same context. analyze the structured causal models (SCMs) behind them. Fig. shows two typical SCMs for entity bias mitigation methods, where refers the raw input, E refers to entitics, and Y refers to the label. The links X + Y + show that LLMs rely on both predictive information from the whole input and the biasing information from specific entities to make the prediction. The links E + X and X + E assume that the context is written down with the entity in mind vice versa. As discussed by Verma and Pearl (1990), we cannot differentiate between these two directions merely based on statistical observations. Indeed, the two SCMs with opposite links between X and E are equivalent according to the Bayes' thcorem:
Debiasing with Causal Intervention. LLMs have been revealed with bias problems, for which liter- ature has paid much attention in order to mitigate their adverse effects (Sweeney and Najafian, 2019; Zhang et al., 2020b; Venkit and Wilson, 2021; Lalor et al., 2022). Recent debiasing techniques incorpo- rate the concept of counterfactual inference, and have been applied in various tasks for bias miti- gation (Niu and Zhang, 2021; Qian al., 2021 a; Wang et al.. 2022). One dominant technique is based on causal mediation analysis (Udomcharoen- chaikit et al., 2022), which involves decomposing the total effect into pure direct effect and total indi- rect effect. In this context, Wang et al. (2022) uti- lize total direct effect and total effect to debias the relation extraction. Apart from debiasing, causal mediation analysis can be used to analyze biases in LLMs ( (Vig et al., 2020; Finlayson al., 2021).
P(XIXTEXX)P(XXXXE)) =P(Y,X,
= -P(E))(XIE)PYXXE..)))
As revealed by these SCMs, entity bias exists in LLMs because entities serve as either confounders or mediators. Thus, the bias can be mitigated through causal intervention, such as backdoor ad- justment
In addition to intervening causal mediator, pre- vious studies have also explored confounder anal- ysis (Keith et al., 2020; Qian al., 2021a; Feder et al., 2022; Weld et al., 2022). confounder is a variable that influences both the input and the out- put, causing a spurious correlation between them.
P(Y ((CD)))EP(YX..... P(Y(),E)P(L))
-

---

Steve Jobs
Unpredictive unbiasing
Foundier Apple
A Original Entiy
Greatox PPaane
Blasing
Predictive
Convex Hull
Predictive but biosing
Neighboring Entiky
Busdinssammn
Co-Founder of Microsoft
Fouedier Amazon Founder Blue Orign
Born in Seente
Center of Convex Hull
Unpredictive but unbiasing
Bill Galos
Jeff Bozas
Figure Left: Training- time intervention with Right: Example of predictive and biasing information.
which eliminates the influence of specific vari- able (in this context, E) by assigning values to this variable. However, previous SCM-based debiasing methods exhibit divergent performances, since they estimate different (conditional) probabilities using different surrogates when performing the causal intervention. For example, counterfactual analysis by Wang et al. (2022) estimates and deducts the biasing effect entities on labels by masking the context, while Zhang et al. (2017) and Longpre et al. (2021) directly remove the effect of entities by entity masking substitution. None of them es- timates the causal effects of entity names precisely, due to the highly complex architectures of LLMs, which account for their unsatisfactory performance on mitigating entity bias.
3.2 Training-time Intervention
For white-box models of which the parameters are accessible,
effectively address their internal bias through training-time intervention. In the case of entity bias identified by the proposed SCM, we realize the causal intervention by perturbing the in- put entities or entity tokens using their neighboring counterparts in the embedding space, as shown in Fig. (Left). For each entity presented in the input text, we first find its top & nearest neighbors accord- ing to embedding distance. Then we construct the smallest convex hull to cover the original entity and neighboring entities. Due the continuous nature of the embedding space, the embeddings within the convex hull approximately represent the same predictive information as a whole. The entity- specific biasing information, which has the poten- tial to trigger spurious shortcuts, gradually dimin- ishes from the original entity towards the border of the convex hull.
In this work, consider the SCM in Fig. 2, whose parameters much easier estimate in practice. Since most LLMs follow sequential structure by stacking neural layers, mitigating the entity bias in one layer will also mitigate the entity bias in subsequent layers. The underlying logic is simple if we block the spurious features in the input, there will be no spurious correlations capture. Therefore, propose mitigate the entity bias the input layer M, which could be an embedding layer or a prompt layer. Obviously, X, E) can be estimated more accurately and efficiently than X, F)), because there is no need to run the whole model, ensuring less error propagation and computational cost. To further improve the estimation by retaining as much pre- dictive information possible, propose to timate P(M]do(X)) by perturbing the entity with similar entities rather than masking it. In the fol- lowing sections, we will show how to realize the proposed causal intervention on both white-box and black-box LLMs.
During training, we introduce perturbations the entity embedding by replacing with random embedding selected from within the convex hull, In this way, the convex hull bounded the predictive information, while random sampling further intro- duces noises and increases the diversity of data for robust training. During inference, we replace the original entity embedding with the center of the convex hull, in order balance the trade-off be- tween predictive and biasing information. Fig. 3 (Right) provides an example of the information pre- served through such intervention. By replacing the entity Bill Gates with the center of the convex hull, encompassed by its neighboring entities, such as Sreve Jobs and Jeff Bezos, we effectively retain the
*This convex hull-bounded perturbation is inspired by Dong et al. (2021), where perturbation within convex hull formed hy synonyms is used to improve model rohustness against word substitutions.

---

1. Replace entitles with placeholders
3. Define placeholders with examples
Context
subject entity went eoject_entity Building 99.
Assume that subject_entity can be any of Bill Gates, Jeff Bezos,and Steve Jobs, while Eefect_enttty be any of Google, Microsoft, and Meta.
Definition
What's the relation between subject_entice _nnttty and object_ entity in the given context?
Question
2. Find similar entities
Query with the new prompt
Name two persans (organizations) similar to Bill Gates (Microsoft)
Definition
Context
Steve Jobs
Google
visitor
Bill Gates
Microsoft
Question
Options
Jeff Bezos
Meta
Figure 4: In-context intervention for black-box LLMs. We take relation extraction example.
shared predictive information (e.g., person), while mitigating the biasing information (e.g., founder of Microsoft). That to say, the convex hull-bounded perturbation serves as an effective estimation of P(M (X)).).
original one (e.g., E.)). These generated entities (e.g., Fig and Fb)) present similar predictive infor- mation as the original entity, and are able to fulfill the same function as neighboring entities in $3.2. Third, we define the placeholder with the original entity and generated entities. For example, we can verbalize the definition as "Assume [ENTITY] be any of Eig. Fia and Eb,". ". This definition encour- ages the LLM find common properties of given entities rather than relying on biasing information of one specific entity. The resulting placeholder along with its definition serves as an effective esti- mation of P(M (oM)). Finally, we prepend the placeholder definition to the modified context and question, and query the LLM with the new prompt. This four-step adjustment ensures that the result- ing prompt free of specific biasing information pertaining to the original entity while still preserv- ing sufficient predictive information by considering given entity examples as a whole.
3.3 In-context Intervention
The rise of Web services powered by black-box LLMs, such as GPT-3.5, introduces new chal- lenges for mitigating entity bias, demanding debi- asing methods that do not require accessible model weights and prediction logits. As discussed in $3.1, key advantage of our SCM is that the decon- founder operation is merely on the input layer. In the context of black-box LLMs, the input is the urov-ido prompt. Thus, we perform the causal intervention solely through modifying prompts to resolve entity bias. We propose a four-step (test- time) in-context intervention technique for black- box LLMs. Fig. 4 shows the whole process.
White-Box Experiments
In this section, we evaluate our training-tine time inter- vention under the white-box setting.
First, we replace the original entity mention in the input with abstract placeholders (e.g.. [ENTITY]). This step effectively mitigates any bias- ing information from the original entity names, be- cause the placeholders are semantic-neutral. How- ever, this step also eliminates predictive informa- tion from entities. We show in $5.3 that, without proper definition for the placeholder, models can easily fail to answer questions. In the next two steps, we construct definitions to provide predictive information for each placeholder while introducing minimal additional biasing information. Second, we query the LLM to name entities similar to the
4.1 Experimental Setup
Datasets and Metries. We evaluate our methods relation extraction (RE) and machine reading comprehension (MRC). For both tasks, we fine- tune models on an in-distribution (ID) training set and evaluate models on both ID and out-of- distribution (OOD) test sets. For RE, we adopt TACRED (Zhang et al., 2017) as the ID dataset and
"Here, we rely on the entity knowledge possessed by L.MMs. However, is possible to replace the I.I.M with exter- nal databuses or tools in this step.

---

(FI)
MRC (EM)
ID
0OD
A
ID
OOD
A
Vanilla Model
71. 10.9
62.3
-1.4%
19.1
63. 63.1
20.2%
Continual Pretraining (Yan et al., 2022)*
79.6
65.9
17.2%
CoRE (Wang et al., 2022)
g..
61.2
-14.2%
Entity Mask (Zhang et al., 2017)
0..5
61.910.5
+0.9%
75.7
62.910.4
16.9%
Entity Substitution (l.ongrre al., 2021)
66.6 66.6
65.8
-1.2%
76.4
70.8
-7.3%
Ouur
70.8+0.3
68.0
3.99
77.0+0.7
72.2+0.5
-6.2%
Table 1: Results under white-box setting. We report the average F1/EM score and standard deviation of three
runs. A shows the relative performance change between ID and OOD. The best number of each column is in bold. Continual pretraining not directly comparable finetuning methods. Numbers copied from Yan al. (2022).
EntRED (Wang et al.. 2023) as the OOD dataset, and report micro-F1 score. In both datasets, entities in each sentence are given. For MRC, we adopt TriviaQA (Joshi al., 2017) as the ID dataset and its answer-substituted version (Yan et al., 2022) the OOD dataset, and report exact match (EM)
extract entities. Since the detected entities could be noisy and incomplete, we perform our method upon answer-substituted training set ensuring all answer entities are perturbed as strong Entiry Substitution. Since RoBERTa and SpanBERT lack entity-level embeddings, we apply our causal inter- vention to each token embedding within the entity mention instead. To construct convex hull, We se- lect neighboring tokens based on their Euclidean distance to the original token in the embedding space. For both tasks, we perform training-time in- tervention each entity token with k = 3. While further data augmentation is always possible, for fair comparison, we finetune all the models with the same amount of data. More implementation details are in Appx. $A.1.
score. Following Yan et al. (2022), we hold out 10% of the training data for development and evalu- ate models on the original development set. We use the DBName version of their OOD dataset. For all metrics, we report the average score with standard deviation of three runs.
Baselines. We compare our methods with the fol- lowing baselines. Entity Mask (Zhang et al., 2017) masks the subject and object entities in the sentence with special tokens. Entity Substitution (Longpre et al., 2021) randomly selects entity of the same type to substitute the original entity. CoRE (Wang al., 2022) applies counterfactual inference by computing the difference between the prediction made with the entire sentence and the prediction made with only the entities observed. Continual Pretraining (Yan et al., 2022) introduces an inter- mediate pretraining stage to the backbone model with the objective of recovering masked entities.
4.2 Results
As shown in Tab. 1, the vanilla RoBERTa and Span- BERT experiences significant declines in perfor- mance on RE (-12.4%) and MRC (-20.2%) when evaluated on OOD test sets. For both tasks, the OOD test set exhibits lower entity bias, achieving better performance on it suggests that the model relies less on entity bias predictive factor.
CoRE and Continual Pretraining are the only baselines that improve the ID performance. CoRE leads to a slight performance decrease on the OOD test set of RE in exchange," 7 while Continual Pre- training further increases the OOD performance on MRC. Entity Mask successfully narrow down or even reverse the relative performance drop un- der OOD setting on the two tasks. However, its absolute performance decreases significantly due
Implementation Details. For RE, we ap- ply RoBERTa (Liu et al., 2019) as the back- bone model following previous works (Zhou and Chen, 2022; Wang et al., 2022). We use the entity_marker_punct input format from Zhou and Chen (2022) in main experiments, in order to mitigate the impact of explicit entity type informa- tion on our analysis of entity bias. For MRC, we ap- ply SEanBERT (Joshi al., 2020) as the backbone model following Yan et al. (2022). Since entities are not given in MRC datasets, we use the same named entity recognition tool used by Yan et al. to
"This is because CoRE is designed for a class-balanced setting, but this experiment emphasizes the performance on the raw class distribution. Moreaver, we search its bias miti- gation weight on the ID development set, which has notably different entity distribution compared with the OOD test set.

---

ually diminish. In contrast, the OOD performance is lower when l too big or too small. When too big, the same problem under ID setting also happens to the OOD setting. When small, the biasing information is not effectively mit- igated, because the perturbed entity is too close to the original entity.
ID (Ours)
ID (Vahilla)
(Ours)
OOD (Vanilla)
too
Entity Type as Input. Previous experiments in this section do not explicitly input entity information it may disturb the causal analysis. Here, analyze the effect of entity type information as input. We use the typed_ enneeniiii stymmrker__a_ _punct input for- mat from Zhou and Chen (2022). The ID and OOD Fi scores vanilla RoBERTa model are 74.6 and 68.9 points, respectively. Our training-time inter- vention further improves the ID performance by 0.7 points and the OOD performance by 2.9 points. These results indicate that information from neigh- boring entities is complementary to coarse-grained entity type information for precise RE.
60
10
K
Figure 5: Fl score of training- caining-time time intervention with different k on RE.
the loss of predictive information from entities. Moreover, its effectiveness dependent on the task property. Unlike MRC, entities are given and are not answers in RE, so the gap between ID and OOD performance of Entity Mask are much smaller. En- tity Substitution stands out among all the baselines in terms of the OOD performance, with absolute improvement of 3.5 points on RE and 7.7 points on MRC. However, its ID performance suffers a lot from the distribution shift of entities during train- ing.
Black-Box Experiments
In this section, we evaluate our in-context interven- tion for mitigating entity bias from LLMs under black-box setting.
Our training- training-time intervention achieves the best OOD performance, with absolute improvement of 2.2 points on RE and 1.4 points on MRC com- pared with Entity Substitution. At the same time, its ID performance also better. These results show that our method mitigates entity bias more effec- tively without losing much predictive information. In other words, the proposed method represents better way to estimate the parameters of the pro- posed SCM accurately.
5.1 Experimental Setup
Datasets. Following Zhou et al. (2023), we adopt GPT-3.5 text-davinci-003 as the back- bone LLM and evaluate the model performance under zero-shot setting. We use the RE and MRC datasets provided by Zhou et al. (2023). The RE dataset is based on Re-TACRED (Stoica et al.. 2021). Zhou et al. pair each instance's 's entities with randomly sampled context that shares the same entity types but possesses different relations. To mitigate the influence the label no relation. which can also serve as signal of abstention, we further filter out all instances whose original or up- dated labels are no non relation. The MRC dataset is based on Natural Questions (K (Kwiatkowski et al., 2019). Zhou et al. replace the original answer in each instance with randomly sampled entity of the same type. They only collect instances where the LLM can give the correct answer based on the raw context. Intuitively, LLMs that faithfully capture contextual information should update their answers based on the new context.
4.3 Analysis
To provide a comprehensive understanding of our training-time intervention, we further conduct anal- yses on RE.
Effect of k. The number of neighbors, k. plays a crucial role in balancing the predictive information and biasing information from entities. To find the sweet spot of k, we examine its influence on model performance as shown in Fig. 5. In general, the ID performance decreases when & increases. As the value of k increases, the resulting convex hull becomes larger, causing the center of the hull to move further away from the original entity. Conse- quently, both the predictive information and biasing information that contribute to ID performance grad-
Metrics. We report the score for RE, and EM score for MRC. To align with previous works, we

---

MRC (EM )
MRC (
RE (F11)
RE (MR
51.3
25.7
25.6
48.6
Base
71.8
71.171
24.3
23.8
25.0
Attribute
70.6
37.4
Opinion
65.9
Ours
27.1
62.7
24.9
16.6
17.7
12.
9.1
10-
6.2
55
50
instruction
instruction instruction
W/o instruction
Figure 6: GPT-3.5 results on MRC and RE under black-box setting. We report the EM score MRC and the F1 score on RE, for which higher scores are better. We also report the MR score on tasks, which lower scores are better. Our in-context intervention performs consistently better than baselines under all settings.
71.8 66.7 64.9
entities are not given in MRC, we detect named entities and replace them with placeholders using 3pt-3.5-tae 5-turbo as an external tool. Given the po- tential abundance of entities in long contexts, we do not replace entities that exclusively appear in the context.
Ours
w/o original entity W/o random order definition
Scores
37.9
13.8
3.5 4.0
0.0
Metrics
5.2 Results
Figure 7: Ablation study of in-context intervention for GPT-3.5 RE
As shown in Fig. 6, all methods benefit from care- fully designed task instructions in terms of task per- formance. The Opinion-busee based prompt performs the best among all baselines in most cases. Compared with the Base prompt, it significantly improves the EM score by 18.7-21.5 points on MRC and the Fl Score by 0.6-4.7 points on RE. Our in-context inter- vention achieves the highest EM/F1 score and the lowest MR score under all settings. Specifically, without task instruction, our in-context intervention outperforms the best baseline by 20.5 EM points MRC and reduces the MR score by 17.6 points RE. These results demonstrate the effectiveness of our causal intervention for addressing entity- based knowledge conflicts in black-bou box LLMs.
also report the memorization ratio (MR; Longpre et al. 2021) to measure the model's i ability to update answers based given contexts."
Baselines. We compare our in-context intervention with the methods introduced by Zhou et al. (2023). Base prompts directly concatenate the context and the question of each instance as the query. At- tribute-based prompts append "in the given context" the question. Opinion-based prompts modified the context narrator's statement by prepending "Bob said" to the context, and then query the LLM about the narrator's opinion by prepending "What's 's Bob's 's opinion on" to the question. We evaluate all methods with and without specifically designed task instructions following Zhou et al. (2023).
5.3 Ablation Study
We in addition conduct an ablation study on RE to provide a comprehensive understanding of our method, as shown in Fig. 7. When the placeholder definition not provided (i.e., w/o definition), no entity information, including both biasing and pre- dictive information, appears in the input. As a result, it successfully blocks any spurious shortcuts with MR drops 0. However, the Fl score also drops sharply from 71.8 points to 37.9 points, indi- cating that some entity information is essential accurate RE and the LLM cannot understand the placeholders well without their definition.
Implementation Details. We apply our in-context intervention to attribute-based prompts. We adopt the backbone LLM to propose two similar entities along with the original entity to define each place- holder. To further eliminate the spurious entity mapping, shuffle the entities for each place- holder before verbalization. Details of all prompt templates used can be found in Appx. $A.2. Since
*MR P,+P, where Po the probability that the model generates the original answer and P. is the probability that the model updates the answer correctly.

---

We fürther examine the role of original entities in the placeholder definition. On the one hand, we remove the original entities from the definition (i.e., w/o original entity). Results show that our method can still improve Fl while reducing MR. This verifies the effectiveness of using set of sim- ilar entities to represent the predictive information from the original entity. On the other hand, we put the original subject and object entities at the same position (i.e., w/o entity shuffle) in the definition so that the LLM can easily map them. As a result, the MR increases significantly, showing that the LLM can find spurious shortcuts even through mapping the subject entity and the object entity from two entity sets.
Limitation
Although we have tried to verify the effectiveness of our method under diverse settings, including different LLMs, different accessibility of model pa- rameters, and different tasks, there are always more options for further investigation, especially nowa- days when more and more LLMS are kept produced. Considering the property of the entity bias issuc may vary when it comes to different LLMs and datasets from different domains, future work can build better benchmark for more comprehensive evaluation. In this paper, we only consider zero- shot prompting for black-box LLMs, because this will help us to control variables during causal analy- sis. However, it is possible to combine the proposed causal intervention with cutting-edge LLM infer- ence methods, such as in-context learning (Brown et al., 2020), although the underlying SCM may become more complex.
6 Conclusion
In this paper, we analyze the entity bias in LLMs from causal view. Building upon an SCM whose parameters are casier to estimate, we pro- pose training-time causal intervention for white- box LLMS and in-context causal intervention for black-box LLMs. Both intervention techniques per- turb the original entity with neighboring entities to mitigate spurious correlations between specific en- tities and predictions. Experiments on relation ex- traction and machine reading comprehension show that the proposed intervention can effectively re- duce the conflicts between parametric knowledge and contextual knowledge and significantly im- prove the performance of LLMs. Future work can apply our causal intervention to more LLMs and tasks to achieve context-faithful answers.
References
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33: 877-19911
Hung-Ting Chen, Michael Zhang, and Eunsol Choi. 2022. Rich knowledge sources bring complex knowl- edge confficts: Recalibrating models to reflect con- flicting evidence. In Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, pages 2292-23002 2307, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.
Acknowledgement
Xinshuai Dong, Anh Tuan Luu, Rongrong Ji, and Hong Liu. 2021. Towards robustness against natural lan- guage word substitutions. In International Confer- ence Learning Representations.
We appreciate the reviewers for their insightful comments and suggestions. Fei Wang is supported by the Annenberg Fellowship and the Amazon ML Fellowship. Wenjie Mo is supported by the USC CURVE Fellowship and the Provost's Research Fellowship. Wenxuan Zhou and Muhao Chen are supported by the NSF Grant IIS 2105329, the NSF Grant ITE 2333736, the DARPA MCS program un- der Contract No. N66001 1924033 with the United States Office Of Naval Research. This work is also supported in part by Cisco Research Award, two Amazon Research Awards, and a Keston Re- search Award. Computing of this work has been partly supported by a subaward of NSF Cloudbank 1925001 through UCSD.
Amir Feder, Katherine A Keith, Emaad Manzoor, Reid Pryzant, Dhanya Sridhar, Zach Wood-Doughty, Jacob Eisenstein, Justin Grimmer, Roi Reichart, Margaret E Roberts, et al. 2022. Causal inference in natural lan- guage processing: Estimation, prediction, interpreta- tion and beyond. Transactions of the Association for Computational Linguistics, 10:1 138-1188.
Matthew Finlayson. Aaron Mueller, Sebastian Gehrmann, Stuart Shieber, Tal Linzen, and Yonatan Belinkov. 2021. Causal analysis of syntactic agreement mechanisms in neural language models. arXiv preprint 2rXiv:2106.06088
Karl Moritz Hermann, Tomas Kocisky, Edward Grefen- stette, Lasse Espeholt, Will Kay, Mustafa Suleyman,

---

and Phil Blunsom. 2015. Teaching machines to read and comprehend. Advances in neural information processing systems, 28.
Susamnah Young, et al. 2022. Streamingqa: bench- mark for adaptation to new knowledge over time in question answering models. In International Con- ference on Machine Learning, pages 13604-13622. PMLR.
Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Ncubig. 2020. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423-435
Yinhan Liu, Myle Oit, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint arXiv: arXiv:
Mandar Joshi, Danqi Chen, Yinhan Liu,, Daniel S Weld, Luke Zettlemoyer, and Omer Levy. 2020. Spanbert: Improving pre-training by representing and predict- ing spans. Transactions of the Association for Com- putational Linguistics, 8:64-77
Shayne Longpre, Kartik Perisetla, Anthony Chen, Nikhil Ramesh, Chris DuBois, and Sameer Singh. 2021. Entity-based knowledge conflicts in question answering. In Proceedings of the 2021 Conference Empirical Methods in Natural Language Process- ing, pages 7052-7063.
Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. 2017. Triviaqu: A large scale distantly supervised challenge dataset for reading comprehen- sion. In Proceedings of the 55th Annual Meeting of for ume 1: Long Papers), pages 1601-1611.
the Association for Computational Linguistics (Vol-
Guoshun Nan, Jiaqi Zeng, Rui Qiao, Zhijiang Guo, and Wei Lu. 2021. Uncovering main causalities for long- tailed information extraction. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 18999
Jungo Kasai, Keisuke Sakaguchi, Yoichi Takahashi, Ronan Le Bras, Akari Asai, Xinyan Yu, Dragomir Radev, Noah A Smith, Yejin Choi, and Kentaro Inui. 2022. Realtime qa: What's the answer right now? arXiv preprint 222
Yulei Niu and Hanwang Zhang. 2021. Introspective distillation for robust question answering. Advances in Neural Information Processing Systems, 34:16292- 16304.
Katherine Keith, David Jensen, and Brendan O' O'Connor. 2020. Text and causal inference: review of using text to remove confounding from causal estimates. In Proceedings of the 58th Annual Meeting of the As- sociation Computational Linguistics, pages 5332- 5344.
Judea Pearl. 2012. The do-caleulus revisited. In Pro- ceedings of the Twenty- -iight Conference on Uncer- tainty in Artificial Inielligence, pages 3-11.
Hao Peng, Tianyu Gao, Xu Han, Yankai Lin, Peng Li, Zhiyuan Liu, Maosong Sun, and Jie Zhou. 2020. Learning from context or names? an empirical study on neural relation extraction. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3661-3672.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red- field, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Ken- ton Lee, et al. 2019. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453- 466.
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel. Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowl- edge bases? In Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2463-2473, Hong Kong, China. Association for Computational Linguistics.
John P Lalor, Yi Yang, Kendall Smith, Nicole Fors- gren, and Ahmed Abbasi. 2022. Benchmarking inter- sectional biases in nlp. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3598-3609.
Daliang Li, Ankit Singh Rawat, Manzil Zaheer, Xin Wang, Michal Lukasik, Andreas Veit, Felix Yu, and Sanjiv Kumar. 2022. Large language models with controllable working memory. arXiv preprint arXiv:2211.05110.
Chen Qian, Fuli Feng. Lijie Wen, Chunping Ma, and Pengjun Xie. 2021a. Counterfactual inference for text classification debiasing. In Proceedings of the 59th Annual Meeting of the Association for Compu- tational Linguistics and the 11th International Joint Conference on Natural Language Processing (Vol- ume 1: Long Papers), pages 5434-5445.
Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. TruthfulQA: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 32 14-3222, Dublin, Ireland. Association for Computational Linguistics.
Kun Qian, Ahmad Beirami, Zhouhan Lin, Ankita De, Alborz Geramifard, Zhou Yu, and Chinnadhurai Sankar. 2021b. Annotation inconsistency and en- tity bias in MultiWOZ. In Proceedings of the 22nd Annual Meeting of the Special Interest Group on Dis- course and Dialogue, pages 326-333 226, Singapore and Online, Association for Computational Linguistics.
Adam Liska, Tomas Kocisky, Elena Gribovskaya, Tay- fun Terzi, Eren Sezener. Devang Agrawal, D' Autume Cyprien De Masson, Tim Scholtes, Manzil Zaheer,

---

Adam Roberts, Colin Raffel, and Noam Shazeer. 2020. How much knowledge can you pack into the param- eters of language model? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5418-544 5426, Online. Association for Computational Linguisties.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu. Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoe V Le. 2022. Finetuned language mod- els are zero-shot learners. In International Confer- ence on Learning Representations.
Galen Weld, Peter West, Maria Glenski, David Arbour, Ryan A Rossi, and Tim Althoff. 2022. Adjusting for confounders with text: Challenges and an empirical evaluation framework for causal inference. In Pro- ceedings of the International AAA! Conference on Web and Social Media, volume 16, pages 1109-11200 100.
George Stoica, Emmanouil Antonios Platanios, and Barnabás Póczos. 2021. Re-tacred: Addressing short- comings of the tacred dataset. In Proceedings of the AAAI Conference on Artificial Intelligence, vol- ume 35, pages 1384001855..
Nan Xu, Fei Wang. Bangzheng Li, Mingtao Dong, and Muhao Chen. 2022. Does your model classify enti- ties reasonably? diagnosing and mitigating spurious correlations in entity typing. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing.
Chris Sweeney and Maryam Najafian. 2019. A trans-
parent framework for evaluating unintended demo- graphic bias in word embeddings. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 666-1667.
Bing Tian, Yixin Cao, Yong Zhang. and Chunxiao Xing. 2022. Debiasing nlu models via causal interven- tion and counterfactual reasoning. In Proceedings of the AAAI Conference on Artificial Intelligence. volume 36, pages 11376-11384.
Jun Yan, Yang Xiao, Sagnik Mukherjee, Bill Yuchen Lin, Robin Jía, and Xiang Ren. 2022. On the ro- bustness of reading comprehension models to entity renaming. In P Proceedingg of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 508-520.
Can Udomcharoenchaikkt, Wuttikorn Ponwitayarat, Patomporn Payoungkhamdee, Kanruethai Masuk, Weerayut Buaphet, Ekapol Chuangsuwanich, and Sarana Nutanong. 2022. Mitigating spurious cor- relation in natural language understanding with coun- terfactual inference. In Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, pages 11308- 11321.
Dong Zhang, Hanwang Zhang, Jinhui Tang, Xian Sian-Sheng Hua, and Qianru Sun. 2020a. Causal intervention for weakly-supervised semantic segmentation. Advances in Neural Information Processing Systems, 33:655- 666.
Guanhua Zhang, Bing Bai, Junqi Zhang, Kun Bai, Con- ghui Zhu, and Tiejun Zhao. 2020b. Demograph- should not be the reason of toxicity: Mitigating discrimination in text classifications with instance weighting.
Pranav Narayanan Venkit and Shomir Wilson. 2021. Identification of bias against people with disabilities in sentiment analysis and toxicity detection models. arXiv preprint 2rXiv:2111.1322
arXiv preprint arXiv:2004. 14088
Yuhao Zhang. Victor Zhong, Danqi Chen, Gabor Angeli, and Christopher D Manning. 2017. Position-ave -aware attention and supervised data improve slot filling. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 35- 45.
Thomas Verma and Judea Pearl. 1990. Equivalence and synthesis of causal models. In Proceedings of the Sixth Annual Conference on Uncertainty in Artificial Intelligence, pages 255-270.
Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian. Daniel Nevo, Simas Sakenis. Jason Huang, Yaron Singer, and Stuart Shieber. 2020. Causal mediation analysis for interpreting neural nlp: The case of gender bias. arXiv preprint arXiv:2004. 12265.
Wenxuan Zhou and Muhao Chen. 2022. An improved baseline for sentence-level relation extraction. In Pro- ceedings of the 2nd Conference of the Asia-Paut Paccifcc Chapter of the Association for Computational Lin- guistics and the 12th International Joint Conference on Natural Lnguuge Processing, pages 161-168.
Yiwei Wang, Muhao Chen, Wenxuan Zhou, Yujun Cai, Yuxuan Liang, Dayiheng Liu, Baosong Yang, Juncheng Liu, and Bryan Hooi. 2022. Should we rely on entity mentions for relation extraction? debi- asing relation extraction with counterfactual analysis. arXiv preprint arXiv:2205.03784
Wenxuan Zhou, Sheng Zhang, Hoifung Poon, and Muhao Chen. 2023. Context-faithfuu prompting for large language models. In Findings of the 2023 Con- ference on Empirical Methods in Natural Language Processing.
Yongchun Zhu, Qiang Sheng, Juan Cao, Shuokai Li. Danding Wang, and Fuzhen Zhuang. 2022. General- izing to the future: Mitigating entity bias in fake news detection. In Proceedings of the 45th International ACM SIGIR Conference on Research and Develop- ment in Information Retrieval, pages 2120-2125,
Yiwei Wang. Bryan Hooi, Fei Wang, Yujun Cai. Yuxuan Liang, Wenxuan Zhou, Jing Tang, Manjuan Duan, and Muhao Chen. 2023. How fragile is relation ex- traction under entity replacements? In Proceedings of the 27th SIGNLL Conference on Computational Natural Language Learning (CoNLL).

---

A Implementation Details
Assume that subject_entin is one of [subj_candidates), while object_entity is one of [obj_ candidaenss in the following text. (context)
A.1 White-Box Experiments
For RE, we use RoBERTa-Large as our backbone model, which has 354 million parameters. Our im- plementation is based the codebase by Zhou and Chen (2022) with their default hyper-parameters. More specifically, we employ a learning rate of 3e-5, batch size of 32, and conduct training for a total of 5 epochs. Other method- speciffc hyper- parameters are selected on the development set of TACRED. Finetuning typically takes 1.5 hours on an NVIDIA RTX A5000 GPU.
Q: Which option indicates the relationship between subject_entity and object_ _enectt___ttttt in the given text?
Options: cattons:(options [ppiions]
A:
The prompt template for detecting entities in MRC is
List named entities in the following sentence. Separate the entities with ###f,, if you find mul- tiple entities. Do not add additional words before after your answers. (entence))
For MRC, we use SpanBERTT-bsseccase as our backbone model, which has 110 million param- eters. Our implementation is based on the code- base by Yan et al. (2022) with their default hyper- parameters. More specifically, we employ learn- ing rate of 2c-5, batch size of 16, and conduct training for total of 4 epochs. Other method- specific syper-parameters are selected on the hold- out development set of TriviaQA. Finetuning typ- ically takes hours on an NVIDIA RTX A5000 GPU.
The prompt template for replacing entities with placeholders MRC is
Replace the entity (entity _ist) in the follow- ing paragraph. (paragraph)
A.2 Black-Box Experiments
The prompt template for finding similar entities
Our implementation is based on the codebase by Zhou et al. (2023). The instruction for MRC is
is
Name two [[entity_ __ype]yyp]]]] entities similar to "(entity)". )". Separate the entities with ###, and do not add additional words before after your answers. Provide random answers if you are not sure.
Instruction: read the given information and answer the corresponding question.
The prompt without instruction for MRC is
In all the above prompts, variables are sur- rounded with curly brackets and optional variables are surrounded with square brackets.
Assume that (ENTITYO) can be any of (en- tity0_ ) [Assume that (ENTITY1) can be any of (entity __candidates] ...] (context)
Q:(quettion] } based the given text? Extract the answer from the given text. Do not add other words.
A:
The instruction for RE is
Identify the relationship between two entities from list of options.
The prompt without instruction for RE


Reviews:
[{"review_id": "1Y0yMFs2jw", "reviewer": "Reviewer_wJyF", "summary": "", "questions": "1. How do you obtain the entity embeddings for constructing the convex hull during training time intervention? Are you using a pre-trained model that provides such embeddings?\n\n2. How does debiasing impact long-tail entities? In particular, given the sparsity of the embedding space near the long-tail entities, how can we construct a convex hull?\n\n3. What is the interpretation of OOD in the experiments? Are these the entities not seen in training?\n", "limitations": "", "rating": 4, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the valuable feedback from the reviewer.\n\n**Responses to reasons to reject**\n\n> It seems the effectiveness of the proposed method is over-reliant on either the LLMs ability to provide similar entities (in-context intervention) or a pre-trained model’s ability to retrieve similar entities (training-time intervention).\n\nCollecting similar entities or entity tokens is a prerequisite of our method. As shown in our paper, it is easy to realize with either PLM-based similarity search or LLM generation. There could also be model-free alternatives, such as predefined entity clusters or any other similarity-based entity search. \n\n> There are some missing details in the paper. See the “Questions” section for details.\n\nWe provide the details in response to each question below.\n\n**Responses to questions**\n\n> How do you obtain the entity embeddings for constructing the convex hull during training time intervention? Are you using a pre-trained model that provides such embeddings?\n\nFor models without entity embeddings (such as RoBERTa used in this paper), we find neighboring tokens of each entity token based on token-level embedding similarity (line 275 and footnote 7). This is a rough proxy of finding similar entities, but is shown to be effective enough in our experiments. \n\n> How does debiasing impact long-tail entities? In particular, given the sparsity of the embedding space near the long-tail entities, how can we construct a convex hull?\n\nThe impact on long-tail entities is an intriguing question. Although we haven’t looked close into it, we hypothesize that one possible problem could be the convex hull becoming significantly larger due to the sparse embedding neighborhood. In this case, we can set a threshold to cut off the perturbation. In other words, instead of the raw convex hull, another possible design choice could be using the overlap between the convex hull and a norm ball centering around the target embedding.\n\n> What is the interpretation of OOD in the experiments? Are these the entities not seen in training?\n\nThe OOD datasets are built by substituting entities in a specific context to present counterfactual information so as to cut off the prediction shortcut between entities and labels (lines 365-370, 476-479, 484-486). For example, EntRED replaces the subject and object entities in a sentence with entities of the same type. \n\nWe also appreciate the editorial suggestions and will revise our paper accordingly.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Strong empirical results demonstrating the effectiveness of the approach.\n\n2. The paper shows ingenuity in constructing a prompt that is effective for debiasing entity bias. It will be interesting to the community to follow up on this work and see whether similar approaches can help to mitigate other types of biases.\n\n3. The paper is well written and easy to follow\n", "reasons_to_reject": "1. It seems the effectiveness of the proposed method is over-reliant on either the LLMs ability to provide similar entities (in-context intervention) or a pre-trained model’s ability to retrieve similar entities (training-time intervention).\n\n2. There are some missing details in the paper. See the “Questions” section for details.\n", "questions_to_authors": "1. How do you obtain the entity embeddings for constructing the convex hull during training time intervention? Are you using a pre-trained model that provides such embeddings?\n\n2. How does debiasing impact long-tail entities? In particular, given the sparsity of the embedding space near the long-tail entities, how can we construct a convex hull?\n\n3. What is the interpretation of OOD in the experiments? Are these the entities not seen in training?\n", "paper_topic_and_main_contributions": "The paper aims to eliminate entity bias in pre-trained language models through causal intervention methods. In particular, the paper proposes an entity perturbation method that perturbs the input text with similar entities from the convex hull of the target entity’s representation. Such causal intervention at the input layer is shown to be effective on both training-time and in-context interventions. The proposed debiasing approach is evaluated on relation extraction data sets (TACRED and EntRED) and reading comprehension data set (TriviaQA).  Experimental results suggest strong performance gain over other baselines.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Too many footnotes in the paper, some of these should be incorporated into the main text and some others should be placed in the Appendix section when they contain specific details, e.g., choice of an experiment set up."}}, {"review_id": "zWZGnjsEHv", "reviewer": "Reviewer_r268", "summary": "", "questions": "A) Where do you get the set of entity neighbors to compute the convex hull from?  Were they extracted from the relevant datasets, or from an associated KB?\n\nB) In figure 1 there is an option for the model to choose from two given options in order to answer the entity relation question, however, in figure 4 that option is not present. Are options used in your experiments, and if so, how does the method work without those given options?\n\n(If no options given), One issue I see is that the method may not debias across multiple axes. How do you ensure that the training time intervention can debias across multiple axes? In the given example, Bill Gates is decoupled from his role as a founder and is shown to be a visitor. But how can the intervention debias across his nationality, his gender, etc.?\n\nC) In in-context example learning, how can the prompting strategy be applied to multiple entities? Has any analysis been done on if entity bias is sought for only a fraction of the entities present in the sample and what effect that has on performance?\n\nD) Can you elaborate more on why in-domain performance decreases when using your method? In what situations would a practitioner utilize this method given this drop?\n\nE) For in-context learning, can’t entity bias be mitigated through other prompting techniques, such as demonstrations via in-context examples, CoT techniques, self-checking, etc.? How would these methods compare as a baseline against your method? \n\nF) I would like further discussion of this point from the limitations section: “In this paper, we only consider zero-shot prompting for black-box LLMs, because this will help us to control variables during causal analysis.”\n", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the valuable feedback from the reviewer.\n\n**Responses to reasons to reject**\n\n> A) In-domain performance drops when applying the method as a training-time intervention. Although this is understandably an OOD mitigation technique, it would be nice to have consistent performance in domain\n\nWe appreciate the reviewer’s understanding that methods designed for the OOD setting tend to lead to ID performance drop. Under the ID setting, a biased model can overfit the data distribution, therefore not surprisingly performing better on the test set presenting the same types of bias. Specifically in our method, the tradeoff between OOD and ID performance can be adjusted by $k$. As shown in Fig 5, when $k=1$, our method improves OOD performance by 4.1 points of F1 *without ID performance drop*. \n\n> B) There could be additional experiments in the in-context prompting section, as some important baselines for in-context prompting have not been included.\n\nWe did not compare our method with prompting techniques like CoT as they were not designed for addressing entity bias. In response to this concern, we extend the experiments to evaluate CoT on RE w/ instruction. In comparison to the base prompts, CoT does not lead to a performance increase in terms of F1, though reduces MR by 3.3 points. In contrast, our method increases the F1 score by 2.2 points and reduces MR by 11.7 points. We are happy to add this among the baseline results we have had. \n\nAdding examples to the prompt will make the structured causal model more complex. As a first attempt to provide a causal analysis of entity bias in black-box LLMs, we focus on the zero-shot setting. Future work may consider extending our framework to the few-shot setting.\n\n**Responses to questions**\n\n> A) Where do you get the set of entity neighbors to compute the convex hull from? Were they extracted from the relevant datasets, or from an associated KB?\n\nWe do not rely on external resources to create the convex hull. Specifically, we find neighboring tokens of each entity token based on token-level embedding similarity (line 275 and footnote 7). This is a rough proxy of finding similar entities, but is shown to be sufficiently effective in our experiments. Using associated KB may be helpful, but we chose to keep it simple in this paper. \n\n> B) In figure 1 there is an option for the model to choose from two given options in order to answer the entity relation question, however, in figure 4 that option is not present. Are options used in your experiments, and if so, how does the method work without those given options?\n\nFollowing prior work (Zhou et al., 2023), we provide options for RE but not MRC under the black-box setting. We’ll update the figures accordingly. In MRC, we perform the causal intervention to entities in the questions.\n\n> (If no options given), One issue I see is that the method may not debias across multiple axes. How do you ensure that the training time intervention can debias across multiple axes? In the given example, Bill Gates is decoupled from his role as a founder and is shown to be a visitor. But how can the intervention debias across his nationality, his gender, etc.?\n\nConsidering (multi-perspective) contextual similarity of entities could be interesting and is worthy of investigation by follow-up works. In this work, as a pilot study for debiasing both PLMs and black-box LLMs, we focused on a general setting of entity similarity though. A possible solution that future work may consider could be to decompose the entity embedding along multiple axes and create separate convex hulls based on the decomposed embedding. \n\n> C) In in-context example learning, how can the prompting strategy be applied to multiple entities? Has any analysis been done on if entity bias is sought for only a fraction of the entities present in the sample and what effect that has on performance?\n\nOur method does not rely on assumptions about the number and ratio of entities that trigger entity bias. For example, in MRC under the black-box setting, as we do not assume the awareness of which entity may cause the bias, we perform our in-context intervention on all entities in the question. This aligns with the condition that entity bias is “sought for a fraction of multiple entities”. As shown in Fig 6, our method is effective under this setting.\n\n> D) Can you elaborate more on why in-domain performance decreases when using your method? In what situations would a practitioner utilize this method given this drop?\n\nThe ID performance drops because the debiasing method avoids overfitting the ID distribution (that presents the same types of bias). In practice, the tradeoff between OOD and ID performance can be adjusted by $k$, and $k$ can be selected based on the performance on development sets and user needs.\n\n> E) For in-context learning, can’t entity bias be mitigated through other prompting techniques, such as demonstrations via in-context examples, CoT techniques, self-checking, etc.? How would these methods compare as a baseline against your method?\n\nWe did not compare our method with prompting techniques like CoT as they were not designed for addressing entity bias. In response to this concern, we extend the experiments to evaluate CoT on RE w/ instruction. In comparison to the base prompts, CoT does not lead to a performance increase in terms of F1, though reduces MR by 3.3 points. In contrast, our method increases the F1 score by 2.2 points and reduces MR by 11.7 points. We are happy to add this among the baseline results we have had. \n\n> F) I would like further discussion of this point from the limitations section: “In this paper, we only consider zero-shot prompting for black-box LLMs, because this will help us to control variables during causal analysis.”\n\nAdding examples to the prompt will make the structured causal model more complex. As a first attempt to provide a causal analysis of entity bias in black-box LLMs, we focus on the zero-shot setting. Future work may consider extending our framework to the few-shot setting.\n\n\nWe also appreciate the editorial suggestions and will revise our paper accordingly.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "A) The paper is generally well-written and does a thorough analysis of past work in the entity bias space. \nB) Figures 3 and 4 in particular are well-illustrative of the techniques used. \nC) The ablation study on the in-context prompting section also shows clearly which components of their prompting strategy contribute to performance.\nD) Methods that use causal approaches with LLMs are important in achieving high performance in applications and reducing bias/errors.\nE) The authors report a robust set of experiments with multiple runs and error bars.", "reasons_to_reject": "A) In-domain performance drops when applying the method as a training-time intervention. Although this is understandably an OOD mitigation technique, it would be nice to have consistent performance in domain\nB) There could be additional experiments in the in-context prompting section, as some important baselines for in-context prompting have not been included.\n", "questions_to_authors": "A) Where do you get the set of entity neighbors to compute the convex hull from?  Were they extracted from the relevant datasets, or from an associated KB?\n\nB) In figure 1 there is an option for the model to choose from two given options in order to answer the entity relation question, however, in figure 4 that option is not present. Are options used in your experiments, and if so, how does the method work without those given options?\n\n(If no options given), One issue I see is that the method may not debias across multiple axes. How do you ensure that the training time intervention can debias across multiple axes? In the given example, Bill Gates is decoupled from his role as a founder and is shown to be a visitor. But how can the intervention debias across his nationality, his gender, etc.?\n\nC) In in-context example learning, how can the prompting strategy be applied to multiple entities? Has any analysis been done on if entity bias is sought for only a fraction of the entities present in the sample and what effect that has on performance?\n\nD) Can you elaborate more on why in-domain performance decreases when using your method? In what situations would a practitioner utilize this method given this drop?\n\nE) For in-context learning, can’t entity bias be mitigated through other prompting techniques, such as demonstrations via in-context examples, CoT techniques, self-checking, etc.? How would these methods compare as a baseline against your method? \n\nF) I would like further discussion of this point from the limitations section: “In this paper, we only consider zero-shot prompting for black-box LLMs, because this will help us to control variables during causal analysis.”\n", "paper_topic_and_main_contributions": "This paper introduces techniques for mitigating entity bias as both a training-time intervention and prompting intervention for white-box and black-box LLMs respectively. They conduct an analysis of past entity bias mitigating techniques, and introduce a structured casual model to frame their technique.  The causal method is separated into a training time intervention, which consists of swapping a specific entities embedding in the lowest layer with an embedding at the center of a convex hull built off of closely related entities.  They also propose a method for use with black-box LLMs, which focuses on prompting the LLM to consider related entities in addition to a specific one.\n\nThey experiment with their interventions on the two earlier mentioned styles of LLMs and show that the training-time intervention improves over past methods in this study, specifically in the OOD performance. They also show that their prompting intervention aids black-box LLMs in mitigating entity bias as compared to their baseline techniques.\n", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "The “PLM” usage in the abstract is not defined.\n\nFigure 3 caption could use further explanation so that it is more easily understood on quick passes.\n"}, "extra_text": {}}, {"review_id": "0720gzafb8", "reviewer": "Reviewer_zF98", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the valuable feedback from the reviewer.\n\n**Responses to reasons to reject**\n\n> Various prompt-based learning methods have should that the classification results made by PLM/LLM can be affected by prompt templates. s it unclear whether the improvement in results from this method is specific because it mitigates the entity bias in PLM/LLM, or generates a more specific prompting template for PLM/LLM to make decisions.\n\nWe only perturb entities based on existing templates (lines 508-509). Hence, the proposed causal intervention only introduces minimal modifications to the prompt template. Taking RE as an example, to make a fair comparison, we only replace entities with placeholders and prepend definitions of placeholders. All other content in the prompt remains the same for all methods. \n\n> I think that LLM does not have entity bias because it has been pre-trained on a large amount of unsupervised/semi-supervised data. The entity bias in prompts given to LLM may be the reason why there is entity bias in LLM inference results. The paper should analyze this point of view.\n\nPrevious (Longpre et al., 2021, Yan et al., 2022, Zhou et al., 2023) and our work have shown that PLMs/LLMs ranging from BERT to GPT3.5 all suffer from entity bias. The main reason for this is the unbalanced association between entities, and between entities and properties. For example, Bill Gates is much more often associated with Microsoft than most of other entities or concepts. Such biased association of concepts easily causes models to tend to pick up prediction shortcuts during training. During inference, entities in prompts will trigger shortcuts, making the model ignore the actual context. The problem is that LLMs are not faithful to the contextual knowledge, but overly rely on its parametric knowledge.\n\n> Using the top k nearest neighbors to construct the convex hull seems like the prototype-based method. As the value of k increases, the center point of the convex hull should theoretically be closer to the true meaning of this entity category. In other words, the center point of the box will have less bias towards this entity category. This means the center of the hull moves further away from the original entity. But it also means the original entity has a bigger bias, why the results become worse?\n\nWhen $k$ becomes extremely large (e.g., as large as the number of all entities or the size of the whole vocabulary), the center of the convex hull won’t necessarily be the entity category but an average of all entities or tokens. Model performance drops because both biasing and predictive information are lost. The tradeoff between unbiasing and predictive information can be balanced by $k$. A proper $k$ makes the center of the convex hull closer to the category of proper granularity. As a result, we observe that when $k$ increases, the OOD performance first increases and then decreases.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper proposes a causal view of entity bias in PLM/LLM models, which is interesting and has practical significance.\n", "reasons_to_reject": "Various prompt-based learning methods have should that the classification results made by PLM/LLM can be affected by prompt templates. \ns it unclear whether the improvement in results from this method is specific because it mitigates the entity bias in PLM/LLM, or generates a more specific prompting template for PLM/LLM to make decisions.\n\nI think that LLM does not have entity bias because it has been pre-trained on a large amount of unsupervised/semi-supervised data. The entity bias in prompts given to LLM may be the reason why there is entity bias in  LLM inference results. The paper should analyze this point of view.\n\n Using the top k nearest neighbors to construct the convex hull seems like the prototype-based method. As the value of k increases, the center point of the convex hull should theoretically be closer to the true meaning of this entity category. In other words, the center point of the box will have less bias towards this entity category. This means the center of the hull moves further away from the original entity. But it also means the original entity has a bigger bias, why the results become worse?\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a structured causal model (SCM) to mitigate entity bias in PLMs/LLMs.  To mitigate the conflicts between parametric knowledge and contextual knowledge, the paper first replaces entities with placeholders for the input sentence and then finds similar entities via LLM, thirdly generates the definition of placeholders with similar entities, and finally combines the definition, context, and prompt question to a whole input sentence. The experimental results show that the framework proposed by this paper could get better results compared with baseline models.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces an approach to mitigating entity bias in pre-trained language models by employing causal interventions structured around a specific causal model. These interventions replace original entities with neighboring ones to reduce biasing information while maintaining semantic relevance. Experimental results show significant improvements, especially in out-of-distribution performance across different datasets and modes of operation (white-box and black-box settings).

I agree with the reviewers that the approach to construct a prompt that is effective for debiasing models is likely of significance to future work in the field. The reviewers are generally of the view that the paper is methodically sound but may only appeal to a narrow audience. Their constructive feedback has highlighted areas that need further clarification in the paper as the authors have also appreciated in their responses.

Two reviewers were very appreciative of the experimental setup --- both the empirical results as well as the ablation studies. One reviewer had little to say on their reasons to accept but also pointed out that some of these issues could be specific to the prompts used, raising concerns with the experimental setup. The prompts shared in examples in the paper are fairly generic and likely what an average user would input so I don't see any issues with the experiment design as such, but it might be a worthy exploration to see whether such entity bias is prompt dependent or not. 

Aside from the great suggestions offered in the reviews, I would also point the authors to Section 2.1 in [1] (which, to my knowledge, is one of the first papers to anonymize entities to prevent entity bias) which also follows a similar thought process to mitigate entity bias by modifying the corpora. Though there is not a 1:1 match between the two approaches, they are similar enough to warrant a discussion in this paper's related work.

[1] Hermann, K. M., Kocisky, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., & Blunsom, P. (2015). Teaching machines to read and comprehend. Advances in neural information processing systems, 28.
Decision: Accept-Findings