DeCoT: Debiasing Chain-of-Thought for Knowledge-Intensive Tasks in Large Language Models via Causal Intervention (2024)

Abstract: Large language models (LLMs) often require task-relevant knowledge to augment their internal knowledge through prompts. However, simply injecting external knowledge into prompts does not guarantee that LLMs can identify and use relevant information in the prompts to conduct chain-of-thought reasoning, especially when the LLM’s internal knowledge is derived from biased information on the pretraining data. In this paper, we propose a novel causal view to formally explain the internal knowledge bias of LLMs via a Structural Causal Model (SCM). We review the chain-of-thought (CoT) prompting from a causal perspective, and discover that the biased information from pretrained models can impair LLMs’ reasoning abilities. When the CoT reasoning paths are misled by irrelevant information from prompts and are logically incorrect, simply editing factual information is insufficient to reach the correct answer. To estimate the confounding effect on CoT reasoning in LLMs, we use external knowledge as an instrumental variable. We further introduce CoT as a mediator to conduct front-door adjustment and generate logically correct CoTs where the spurious correlation between LLMs’ pretrained knowledge and task queries is reduced. With extensive experiments, we validate that our approach enables more accurate CoT reasoning and enhances LLM generation on knowledge-intensive tasks.

heading: 1 Introduction
text: For knowledge-intensive tasks (Petroni et al., 2021; Hu et al., 2023; Sun et al., 2023b), specific knowledge is required to obtain an accurate response, which can be out of the distribution of LLMs’ internal knowledge (Yao et al., 2023; Yuan et al., 2023c). Since frequently fine-tuning LLMs can be highly expensive and inefficient (Zhai et al., 2023), the LLM’s internal knowledge can also be outdated and cause knowledge bias problems in LLMs (Zhang et al., 2023b; Wu et al., 2023; Zhang et al., 2023c).
To efficiently incorporate external knowledge (i.e., context), methods are proposed to retrieve taskrelevant language evidence (Liu et al., 2023; He et al., 2022; Zhu et al., 2023b; Shao et al., 2023; Trivedi et al., 2022a). Additionally, external knowledge bases can also directly augment and edit the knowledge-injected prompts (Wen et al., 2023; Sun et al., 2023a; Baek et al., 2023; Zhao et al., 2023b). However, simply injecting external knowledge in prompts does not guarantee that LLMs can identify and use relevant information in the prompts (Shi et al., 2023a; Weston and Sukhbaatar, 2023), especially when the LLM learns biased information in pretraining data (Zhang et al., 2023b). The knowledge bias in LLMs can further cause knowledge conflict or misunderstanding between exter-
14073
nal knowledge and the model’s internal knowledge (Mallen et al., 2023; Wang et al., 2023f,a). In such cases, LLMs may use irrelevant information from prompts and generate incorrect and unexpected responses (Li et al., 2023c; Xie et al., 2023).
When the LLM relies on chain-of-thought (CoT) reasoning for complex tasks, the biased information from pretrained models further impairs LLMs’ reasoning abilities. Many works propose to verify and post-edit the generated reasoning paths before prompt again (Zhao et al., 2023b; Peng et al., 2023; Wang et al., 2023c) to eliminate the factual errors in the generated CoT paths. However, logical reasoning errors can not be easily detected or corrected, as the effectiveness of factual verification and postediting reasoning chains can be limited to simply injecting more knowledge. For example in Figure 1, given the query (e.g., “"Judgment Night" was collaborated by Onyx and the heavy metal band formed in which city?”) and context which provides external knowledge (Lewis et al., 2020), the LLM may generate logically incorrect CoT (e.g., CoT 2), in which the last chain deviates from the reasoning paths (e.g., instead of the origin of “Biohazard”, some arbitrary band mentioned in the context). Such logical incoherence can be caused by the spurious correlation between the query (e.g., the concept “the heavy metal band formed in”) and the LLM’s internal knowledge understanding. Thus, the spurious correlation can lead the LLM to find some arbitrary evidence in the context regardless of its logical connection to the previous chain, as long as it contains the exact phrase. In such cases, factual verification methods cannot detect logical reasoning errors, and the answer can still be incorrect even with the facts verified as correct.
In this work, we propose a novel causal view via a Structural Causal Model (SCM) (Pearl et al., 2016) to formally explain the internal knowledge bias of LLMs. To measure spurious correlation, we propose to use external knowledge as an instrumental variable (Morgan and Winship, 2015) to estimate the Average Causal Effect (ACE) of CoT reasoning paths in LLMs through causal intervention. Based on the measurement of ACE, we can further introduce a CoT sampling method to find the best CoT as a mediator and conduct frontdoor adjustment (Pearl, 2009). In this approach, the spurious correlation between LLMs’ internal knowledge and task queries can be reduced, which ensures correct CoT reasoning and LLM generation. We summarize our contributions as follows:
• We discover that the bias from LLMs can trigger the usage of irrelevant information in the prompts, and cause the LLM to generate incoherent reasoning chains that impair the model’s reasoning ability.
• To formally understand the bias affecting CoT reasoning abilities, we propose a novel causal view introducing the external knowledge in prompts as an instrumental variable. This causal view uncovers the spurious correlation between queries and LLMs’ internal knowledge understanding.
• To alleviate the bias and ensure correct CoT reasoning, we estimate the average causal effect (ACE) between the CoT and the answer, and further propose a CoT sampling method to conduct the front-door adjustment.
• We conduct multiple experiments on various knowledge-intensive tasks as well as numbers of LLM backbone models, which demonstrates the effectiveness of our method.

heading: 2 Related Work
text: LLMs in Knowledge-intensive Tasks. In knowledge-intensive tasks (Petroni et al., 2021; Hu et al., 2023; Yang et al., 2018; Welbl et al., 2018) the LLM is asked to respond based on the provided context and its intrinsic knowledge. Retrievalaugmented prompting methods focus on identifying accurate and comprehensive evidence from support documents (Hoshi et al., 2023; Qian et al., 2023), in-context examples (Press et al., 2022; Khattab et al., 2022), knowledge bases (Trivedi et al., 2022a; Xu et al., 2023; Wang et al., 2023c; Feng et al., 2023; Zhu et al., 2023a), knowledge graphs (Wen et al., 2023; Sun et al., 2023a; Salnikov et al., 2023; Zhang et al., 2023a) and human feedback (Zhang et al., 2023b). However, extensive knowledge-injected prompts can introduce irrelevant information to distract LLMs (Shi et al., 2023a; Wang et al., 2023g) and cause LLMs’ unpredictable behaviors (Li et al., 2023c; Chen et al., 2023b). Instead of focusing on how to identify the best knowledge evidence, we investigate how to find logically correct CoT reasoning paths. Chain-of-thought Prompting. Chain-of-thought prompting has shown great potential in explaining LLMs’ thinking process (Yuan et al., 2023a; Li and Du, 2023) and answering multi-hop questions (Wang et al., 2023d; Ma et al., 2022) in complex
reasoning tasks (Fu et al., 2023). However, further works also mention issues of faithfulness and self-consistency in LLMs (Lanham et al., 2023; Turpin et al., 2023). To improve the faithfulness of intermediate chains, several works propose to verify and edit (He et al., 2022; Wang et al., 2023e; Zhao et al., 2023b) the factual errors in unfaithful chains, and (Radhakrishnan et al., 2023; Zhu et al., 2023a) propose to decompose complex questions and answer them individually. We argue that logically incorrect reasoning paths can lead the LLM astray from the right direction of finding the answer, even with the chains factually correct. Similar to previous works, our method first generates some candidate chain-of-thought reasoning paths, which makes our method use similar numbers of API calls (or inference times) per sample. Causal Intervention in Language Models. Causal intervention methods in language models focus on entity-level spurious correlation (Wang et al., 2023b; Zeng et al., 2020; Yang et al., 2023) or sentence-level selection bias (McMilin, 2022). Since LLMs are black-box models (Gat et al., 2023; Cheng et al., 2023), direct methods of causal-aware model reparameterization methods are limited in usage. With causal explanability extracted from the models (Wu et al., 2021; Zhao et al., 2023a), further studies introduce human-in-the-loop debiasing methods (Zhang et al., 2023b; Wu et al., 2022, 2021). However, human effort is normally more expensive, and involving humans in the loop may reduce the efficiency of the system. Instead, our method uses counterfactual context to automatically measure the causal effect and find better CoT to improve the performance of LLMs.

heading: 3 A Causal View
text: To understand the causal relationships in knowledge-intensive tasks, we introduce a Structural Causal Model (SCM) (Pearl et al., 2000) and identify the internal knowledge understanding of the LLMs (Z) as the confounder. In Figure 2a and Figure 2b, we formulate two types of conventional knowledge injection methods as two SCMs respectively. With the SCMs, we explain the effectiveness of conventional knowledge injection methods and their limitations. We further present the SCM of our method, the debiasing chain-of-thought (DeCoT), in Figure 2c. The formulation of our method, DeCoT is illustrated in Figure 2d and detailed in Section 5.
AQ
Z
E
(a) SCM for External Knowledge
E
C AQ
Z
(b) SCM for Chainof-thought
E
C AQ
Z
(c) SCM for DeCoT

heading: 3.1 SCM for External Knowledge
text: In Figure 2a, the causal path E → Q ← Z represents the knowledge injection process, where E denotes the external knowledge, Q denotes queries in the inference stage and Z denotes the LLM’s internal knowledge. Ideally, the query (Q) alleviates spurious correlations as a collider, influenced by the external knowledge (E) and LLM’s internal knowledge (Z), when E and Z are causally irrelevant (Pearl et al., 2000). However, most knowledge injection methods (Baek et al., 2023; Li et al., 2023b) incorporate the external knowledge as the context which is prefixed to the input prompt. Thus, the causal influence of the external knowledge on the query is also determined by the LLM, which makes E and Z dependent variables, and the spurious correlation between Q and Z remains.

heading: 3.2 SCM for Chain-of-thought
text: Chain-of-thought (as in Figure 2b) is introduced (Li et al., 2023a; Wei et al., 2022; Fu et al., 2023) to make the LLM explain and follow the reasoning
path before giving the final answer. The causal path Q → C → A shows that the CoT (C) can serve as the mediator between the query (Q) and the answer (A). However, since the CoT is also prompted from the LLM (Wei et al., 2022; Fu et al., 2023), it can also be causally dependent on LLM’s internal knowledge and thus forms the spurious correlation between C and Z. Notably, knowledge editing methods (Zhao et al., 2023b; Li et al., 2023a; Peng et al., 2023) can correct the factual errors in the context and the reasoning paths, while the reasoning logic remains incorrect.

heading: 4 Preliminaries

heading: 4.1 Task Formulation
text: For knowledge-intensive question-answering tasks, the model is prompted with a query Q = [q1, q2, . . . , qn] and a passage of context E = [e1, e2, . . . , el], i.e., external knowledge (Lewis et al., 2020). Given the query Q and the context E, the model θ is prompted to recurrently generate the response Y by sampling from the conditional probability distribution,
yt ∼ pθ (y|E,Q, y<t) .
Following the previous setting (Welbl et al., 2017; Trivedi et al., 2022b), we make no assumptions on the context more than what is available in the context of the data samples from their datasets. In this setting, there are irrelevant contexts (Tu et al., 2020; Yang et al., 2018), which lead to the spurious correlation identified in our paper. As illustrated and explained in Figure 2a of Section 3.1, the model directly generates the answer A = [a1, a2, . . . , am] without providing the intermediate reasoning process (i.e., A = Y ). Chain-of-thought Prompting. Following (Wei et al., 2022), we add the additional instruction to ask the model to generate its reasoning paths C by explaining step-by-step, before generating the final answer A (i.e., Y = [C,A]). By sampling N different CoTs C = [C1, C2, . . . , CN ] conditioned on the query Q and the context E, we can further condition the generation process of the answer A,
Ci ∼ pθ (C|E,Q) , (1) Ai,r ∼ pθ (A|E,Q,Ci) . (2)
In Eq. (1), since CoTs (C) are also generated from the LLM in which the pretrained internal knowledge (Z) can also confound on the generation process. As explained in Section 3.2, the confounding
effect can not only affect the factual accuracy of the generated CoTs but also lead to incorrect reasoning logic. Thus knowledge editing and verification methods (Zhao et al., 2023b; Li et al., 2023a; Peng et al., 2023) which solve the former problem, are limited in correcting logical errors in CoTs.

heading: 5 DeCoT: Debiasing Chain-of-thought

heading: 5.1 SCM for DeCoT
text: In Figure 2c, the CoT (C) is a mediator between the query (Q) and the answer (A). Based on the frontdoor criterion (Pearl et al., 2000), a mediator (C) should be causally independent of the confounder (Z), to enable front-door adjustment. However, in practice, CoTs are also generated by LLMs, which suggests potential spurious correlations between CoTs (C) and the LLM’s internal knowledge (Z). Thus, to track the bias from the unobserved confounder Z, we introduce the external knowledge as an instrumental variable (IV) (Kawakami et al., 2023; Kilbertus et al., 2020; Yuan et al., 2023b). By changing the value of the instrumental variable E (i.e., the external knowledge), we estimate the true causal relationship between C and A (Yuan et al., 2023b). For example, in Figure 2d, two pieces of counterfactual external knowledge (e.g. “Biohazard formed in Chicago” and “Judegment Night was by Acrassicauda formed in Iraq”) are introduced in the same example of Figure 1, where the average causal effect (ACE) is calculated as in (4). Due to the spurious correlation in the third chain of thoughts of “CoT 2”, responses generated from “CoT 2” remain unchanged (i.e., ACE = 0), while responses generated from the correct reasoning path “CoT 1” change corresponding to counterfactual evidence (i.e., ACE > 0).

heading: 5.2 External Knowledge as an Instrumental Variable
text: We model the external knowledge as an instrumental variable E to understand the causal relationship between the CoT C and the answer A (Yuan et al., 2023b). Due to the limitation of directly controlling the generation process of CoTs, we perform the causal treatment by including counterfactual knowledge through the instrumental variable E. Specifically, we query the LLM to extract T factual entities V = [v1, v2, . . . , vT ] which correspond to T counterfactual context E∗1 , E ∗ 2 , . . . , E ∗ T (prompt design explained in Appendix B). In each sample
E∗j = [e1, e2, . . . , vj , . . . , el] ,
the corresponding factual entity vj is to be replaced by counterfactual entities. Then, the LLM is further prompted to propose P counterfactual entities V ∗j = [v ∗ j,1, v ∗ j,2, . . . , v ∗ j,P ] to each extracted entity vj ∈ V (prompt design explained in Appendix B). P counterfactual context samples
E∗j,k(v ∗ j,k) = [e1, e2, . . . , v ∗ j,k, . . . , el], k ≤ P (3)
are constructed, by replacing the corresponding factual entity vj in each sample E∗j . In this approach, we estimate the average causal effect (ACE) corresponding to each CoT reasoning path Ci,
ACE(Ci, vj) = E (A|do(E), Q,Ci)− (4) E ( A|do(E∗j ), Q,Ci )
= Ev∗j,k∈V ∗j [pθ (A|E,Q,Ci)− pθ ( A|E∗j,k(v∗j,k), Q,Ci ) ],
in which the average causal effect measures the decreased confidence in the answer (measured as in (2)) with counterfactual context as the evidence. We observe that the average causal effect of different factual entities can vary to the context, queries, and CoTs, which we further conduct analysis experiments in Section 6.5. To consider the overall causal effect of the external context on each CoT, we propose to measure the average causal effect of all the intervened entities,
ACE(Ci) = Evj∈V ACE(Ci, vj), (5)
where the intervened entities vj are sampled from a uniform distribution of the external context E.

heading: 5.3 Average Causal Effect Guided Chain-of-thought Sampling
text: With the measured ACE scores, we develop an efficient sampling approach to obtain high-quality CoTs with more coherent reasoning chains, without additional deconfounding layers (Zhang et al., 2020; Wu et al., 2022) for LLM finetuning. Since LLMs are black-box models (Gat et al., 2023; Cheng et al., 2023), direct causal intervention methods on the parameterization of the input query Q and the context E are limited. Thus, we propose to use the sampled CoTs C as the mediator variable to conduct the front-door adjustment.
Based on the measured average causal effect (ACE), we construct importance scores in terms of how the final answer A reacts to different CoTs C intervened by the context E,
C∗ ∼ softmax [pθ (Ci|E,Q) · ACE(Ci)] , (6)
and the front-door adjustment can be realized by introducing the mediator C∗ sampled with the largest average causal effect in the reasoning path,
A∗ ∼ P (A|E,Q, do(C)) (7) ∝ pθ (A|E,Q,C∗) .
The causal effect on the sampled answer A∗ is mediated by the sampled CoT reasoning path C∗, whose mediator-outcome confounding effect is controlled and alleviated. We summarize our algorithm DeCoT in Algorithm 1 (Appendix D).

heading: 6 Experiments

heading: 6.1 Dataset and Evaluation
text: Knowledge-intensive tasks commonly require each question to be paired with a paragraph of context as support evidence (Li et al., 2023a; Zhu et al., 2023b; Su et al., 2023; Jang et al., 2023). Our method focuses on the reasoning errors caused by the spurious correlation from LLM’s pretrained knowledge. We follow the setting (Welbl et al., 2017, 2018; Yang et al., 2018; Trivedi et al., 2022b; Shi et al., 2023b; Wei et al., 2022) and use the original supporting contexts from the datasets without any additional data processing. We follow the evaluation protocols in (Yang et al., 2018) and conduct our experiments on datasets as follows: HotpotQA (Yang et al., 2018) contains questions that require multi-step reasoning over multiple support contexts. For each question, support documents are provided in the dataset, which are used as the context in our experiments. MuSiQue (Trivedi et al., 2022b) is another multistep question answering dataset. Similar to previous work (Ramesh et al., 2023), we conduct our experiment on the challenging part of the dataset, in which questions are annotated as ≥ 4 hops. WikiHop (Welbl et al., 2018) is a multi-choice multi-hop reasoning dataset. We use the queries in the dataset as the questions (Tu et al., 2019) in our setting. For baselines and our method, the models are prompted to generate the answers free-form instead of retrieving them from the candidate list. SciQ (Welbl et al., 2017) is a domain-specific question-answering task that contains only scientific questions. We evaluate baselines and our method on test samples with supporting evidence.
For datasets that lack test labels, we follow the same evaluation protocol as (Press et al., 2022; Shao et al., 2023; Chen et al., 2023a) and use the development sets as our test set. We use the Exact
Match (EM) and F1 proposed in (Yang et al., 2018) as our evaluation metrics.

heading: 6.2 Baseline and Backbone Model
text: Following (Shi et al., 2023b; Su et al., 2023; Trivedi et al., 2022a), we have applied our method to different pretrained LLMs: Flan-T5-XXL (Chung et al., 2022) which has 11B model parameters, LlaMA2-7B (Touvron et al., 2023) and GPT-3.5 Turbo (Brown et al., 2020). For LlaMA-2-7B model, we choose the finetuned versions from human feedback (Christiano et al., 2017), which can generally yield more stable chain-of-thought reasoning paths.
For baselines, we compare our method with a conventional chain-of-thought prompting method (CoT) (Wei et al., 2022) and context-aware contrastive decoding method CAD (Shi et al., 2023b). We also include the baseline which devises conventional chain-of-thought prompting methods without context (CoT w/o ctx) (Wei et al., 2022), to investigate the effect of context in different datasets. Implementation is detailed in Appendix A. Notably, compared to the baselines (Shi et al., 2023b; Wei et al., 2022), our approach does not rely on any additional assumptions and does not require any further processing on the data, to get the performance improvement over these baselines. That is, the inputs of our approach are the same as the baselines (Shi et al., 2023b; Wei et al., 2022).

heading: 6.3 Main Results
text: Table 1 presents evaluation results on the four datasets with three LLM backbone models. Comparison with Baselines. As we expected, for
all LLMs the performance is significantly lower when the context of supporting evidence is absent. Because of the poor performance of the direct prompting method, the context-aware contrastive decoding (CAD) baseline can use its answer distribution as the negative penalty on the positive distribution which is obtained by prompting with both the query and context. However, since the negative answer is not supported by either internal or external knowledge, it can have a more random distribution and limits the effectiveness of contrastive decoding methods. On the other hand, our method DeCoT achieves generally higher improvements on regular CoT compared with CAD by detecting logically incorrect CoTs and penalizing them. Instead of simply contrasting the distributions of positive and negative answers, we use counterfactual context to examine the answer distribution changes, which provides a more fine-grained measurement of the causal effect on LLMs’ internal knowledge bias. The consistent performance improvements suggest DeCoT can more accurately detect incorrect CoTs by the measurement, and perform targeted causal intervention.
Logical Reasoning Performance Understanding. We also observe that DeCoT gains relatively better F1 improvements on the SciQ dataset, which reach 18.58%, 46.37% and 14.01% for Flan-T5xxl, LlaMA-2-7B, and GPT-3.5 respectively. It suggests that accurate logical reasoning paths are more strictly required for scientific questions, and the correctness of the generated CoTs is more crucial. Thus, DeCoT’s better performance on the SciQ dataset suggests DeCoT is more effective in
debiasing LLMs’ logical reasoning ability.

heading: 6.4 Improving ReAct by DeCoT
text: Since our main purpose is to find potential spurious correlation and correct reasoning errors with counterfactual debiasing, our approach is compatible with and can improve other model reasoning method variant. To evaluate the generalizability of DeCoT, we apply the proposed counterfactual reasoning method on ReAct (Yao et al., 2022). Similar to (Wei et al., 2022), we use two knowledgeintensive tasks, HotpotQA (Yang et al., 2018), a question-answering task and FEVER (Thorne et al., 2018), fact verification task. In the evaluation, instead of using context from datasets, ReAct can generate a sequence of reasoning paths with knowledge retrieval queries to find relevant information from external knowledge bases (Yao et al., 2022). From Table 2, we observe that by incorporating ReAct generated context as reasoning evidence, DeCoT further improves ReAct’s performance by counterfactual reasoning.

heading: 6.5 Impact of the Selected Entities
text: We evaluate the impact of the number of the selected entities on the MuSiQue dataset based on the backbone model GPT-3.5. Since annotations in MuSiQue guarantee the minimal number of chains of thought is 4 hops (Ramesh et al., 2023), more factual evidence is required to support the final answer, which makes the impact of the selected entities higher in this case.
To illustrate the trend, we only conduct experiments on a number of the selected entities T with these representative values considering the expensive GPT API costs. In Figure 3a, we show the F1 and EM performance w.r.t. the different number of selected entities T = 0, 1, 3, 5. We include the result of T = 0 which indicates the regular CoT prompting method. With a larger T , it has
a higher probability for DeCoT to find more important entities for causal intervention. However, more causal intervention experiments require more counterfactual prompting, which is at the expense of more API calls or inference time. We observe that we can accurately find good factual entities by selecting the most popular entities.

heading: 6.6 Impact of the Alternative Entities
text: In Figure 3b, we also show the performance w.r.t. the different number of alternative entities P = 0, 1, 3. For the same reason of computational cost, we only choose a small number of alternative entities. Since the alternative entities are counterfactual, normally the negative causal effect should be similar among the alternative entities. However, practically the alternative entities may further introduce unexpected causal relationships to the context. Thus, by calculating the average causal effect of multiple alternative entities, such unexpected causal effects can be averaged out.

heading: 6.7 Case Study
text: We present case study results to understand the impact of context quality on the generated CoTs (more details of qualitative examples are in Appendix E). In Table 3, we show the CoT reasoning paths prompted by DeCoT and the following methods:
• CoTs w/ Context: regular CoT prompting with context (Wei et al., 2022) (i.e., CoT in Section 6.2.).
• CoTs w/o Context: regular CoT prompting with only task queries (i.e., CoT w/o ctx in Section 6.2.).
• CoT w/ Post-Editing: CoT with knowledge post-edit (Zhao et al., 2023b), when factual contradictions exist in the context.
For all the methods, the CoTs are prompted from the backbone GPT-3.5 model. Case Study of CoTs with Post-Editing. We observe that CoTs generated without context are likely to contain incorrect knowledge, which can further mislead the reasoning paths. For example, the directly generated CoTs of the question in the WikiHop dataset say “Terra Brasilis is a song by Caetano Veloso”, which is factually incorrect (highlighted in a red font). Due to this incorrect assumption made from LLMs’ hallucination, the following reasoning paths are misled to talk about irrelevant information (e.g., “Caetano Veloso’s album”), and thus the answer is wrong even with factual edit (highlighted in a blue font).
Case Study of CoTs with Context. Compared to CoTs generated without context, we observe that the CoTs prompted with context can be factually more faithful. However, the logical reasoning of these CoTs can still be wrong. For example, the CoTs generated with the context in the HotpotQA dataset correctly locate “Leucippus and Democritus” as the two “Pluralist school” members (highlighted in a red font). However, instead of answering with the one who created the school, the LLM
mistakenly chooses the wrong answer “Democritus”. We highlight the correct reasoning paths in green font to show that the key point in answering this question is by identifying the “creator”.

heading: 7 Conclusion
text: In this paper, we formally examine the LLM’s internal knowledge bias and identify it as the confounder by a structural causal model (SCM). We revisit the irrelevant contexts issue and further discover the spurious correlation between the LLM and task queries, which can further impair the LLM’s CoT reasoning abilities. Then, we propose DeCoT, a debiasing chain-of-thought prompting method in knowledge-intensive tasks, which alleviates the spurious correlation and enables the LLM to find more accurate and logically sound responses. Following the previous evaluation setting (Welbl et al., 2017; Trivedi et al., 2022b; Shi et al., 2023b; Wei et al., 2022), and using the same inputs as the baselines, extensive experimental results and case studies validate the effectiveness of our method DeCoT.

heading: 8 Limitations
text: Since DeCoT is an inference-stage causal intervention method, the improvement on LLMs’ reasoning abilities is attributed to alleviating the bias, but can be limited to the upper bound of the LLM’s capacity. To alleviate the causal effect of knowledge bias on LLMs’ reasoning abilities, future works can incorporate unbiased causal learning methods in the model pretraining or instruction tuning stage, which may enable more robust CoT reasoning. It is also interesting to study the theoretical causal foundation of CoT prompting’s mediator role in LLMs, which can be beneficial to better interpretability of black-box LLMs.

heading: 9 Ethics Statement
text: Our study on mitigating bias in Large Language Models (LLMs) recognizes the ethical implications of data-driven biases in AI, specifically addressing how these biases affect reasoning processes. We propose a novel approach to reduce bias impact, emphasizing the responsible and ethical advancement of AI technology. The datasets we used in our experiments are all publicly available. No personal information was gathered from our human participants, and they were not exposed to any harmful model outputs.

heading: B Prompt Design
text: B.1 Factual Entity Extraction
To extract the most relevant factual entities V in the context E (Section 5.2),
vj ∼ pθ (V |E, Instructent) , (8) E∗j = [e1, e2, . . . , vj , . . . , el] ,
in which Instructent is the explicit prompt instruction shown in the following.
Context Example 1: The Ritz-Carlton Jakarta is a hotel and skyscraper in Jakarta, Indonesia and 14th Tallest building in Jakarta. It is located in city center of Jakarta, near Mega Kuningan, adjacent to the sister JW Marriott Hotel. It is operated by The Ritz-Carlton Hotel Company. The complex has two towers that comprises a hotel and the Airlangga Apartment respectively. Nakuul Mehta, Kunal Jaisingh and Leenesh Mattoo respectively portray Shivaay, Omkara and Rudra, the three heirs of the Oberoi family.
Instruction Example 1: Extract the top 5 most frequently appeared entities in the context and provide in the format of a list: [Ritz-Carlton, Jakarta, Indonesia, Airlangga Apartment, Nakuul Mehta]
Context Example 2: Lisa Marie Simpson is a fictional character in the animated television series "The Simpsons". She is the middle child and most intelligent of the Simpson family. Voiced by Yeardley Smith, Lisa first appeared on television in "The Tracey Ullman Show" short "Good Night" on April 19, 1987. Cartoonist Matt Groening created and designed her while waiting to meet James L. Brooks. Groening had been invited to pitch a series of shorts based on his comic "Life in Hell", but instead decided to create a new set of characters. He named the elder Simpson daughter after his younger sister Lisa Groening.
Instruction Example 2: Extract the top 5 most frequently appeared entities in the context and provide in the format of a list: [Lisa Marie Simpson, The Simpsons, Yeardley Smith, The Tracey Ullman Show, Lisa Groening]
B.2 Alternative Entity Proposal
To ask the LLM to propose P counterfactual entities E∗j,1, E ∗ j,2, . . . , E ∗ j,P to the extracted entity vj (Section 5.2),
v∗j,k ∼ pθ (V |vj , Instructalt) , (9) E∗j,k(v ∗ j,k) = [ e1, e2, . . . , v ∗ j,k, . . . , el ] ,
in which Instructalt is the explicit prompt instruction shown in the following.
Example 1 Provide 3 most similar entities to "America", which are from the same entity type and have similar meanings. Provide the answers as a list: [Britain, France, Germany].
Example 2 Provide 3 most similar entities to "Florida", which are from the same entity type and have similar meanings. Provide the answers as a list: [New York, California, Arizona].
Example 3 Provide 3 most similar entities to "Beethoven", which are from the same entity type and have similar meanings. Provide the answers as a list: [Mozart, Mahler, Wagner].
C In-context Learning Examples
We design several CoT prompting examples to guide the LLM to generate in an expected format as following:
Example 1 Context: The Ritz-Carlton Jakarta is a hotel and skyscraper in Jakarta, Indonesia and 14th Tallest building in Jakarta. Question: The Oberoi family is part of a hotel company that has a head office in what city?
Based on the context, answer the question step by step and provide the final answer in the end. 1. The Oberoi family is an Indian family known for their involvement in hotel management 2. The Oberoi Group is the hotel company founded by the family Therefore, the final answer is [Delhi].
Example 2 Context: Lisa Marie Simpson is a fictional character in the animated television series "The Simpsons". Question: Musician Allie Goertz wrote a song about the "The Simpsons" character Milhouse, who Matt Groening named after who?
Based on the context, answer the question step by step and provide the final answer in the end. 1. The character Milhouse from the TV show "The Simpsons" was named by Matt Groening 2. Groening named the character after President Richard Nixon’s middle name Therefore, the final answer is [Richard Nixon].
Example 3 Context: Moloch: or, This Gentile World is a semi-autobiographical novel written by Henry Miller in 1927-28, initially under the guise of a novel written by his wife, June. Question: What nationality was James Henry Miller’s wife? Based on the context, answer the question step by step and provide the final answer at the end. 1. James Henry Miller (25 January 1915 – 22 October 1989) was an English folk singer and songwriter 2. His second wife was Margaret "Peggy" Seeger (born June 17, 1935), who is an American folksinger Therefore, the final answer is [American].

heading: D Pseudo Code for DeCoT
text: We illustrate our method as the pseudo-code in Algorithm 1.
Algorithm 1: DeCoT Input: Task query Q, Context E, the LLM
with model parameters θ 1 Init: Sample chain-of- thought reasoning
paths C = [C1, C2, . . . , CN ] by Eq. 1.
2 Extract T factual entities V = [v1, v2, . . . , vT ] from E by Eq. 8 ; 3 while i < N do 4 while j < T do 5 Propose P counterfactual entities {v∗j,1, v∗j,2, . . . , v∗j,P } by Eq. 9 ; 6 while k < P do 7 Construct counterfactual context
E∗j,k(v ∗ j,k) by Eq. 3 ;
8 end 9 Estimate ACE(Ci, vj), i < N for
each entity vj by Eq. 4 ; 10 end 11 Estimate ACE(Ci) by Eq. 5 ; 12 end 13 Sample CoT by Eq. 6 ; 14 Sample the answer by Eq. 7 ;

heading: E Qualitative Examples
text: We present more qualitative examples from HotpotQA (Table 4), MuSiQue (Table 5), SciQ (Table 6) and WikiHop (Table 7).

heading: Future Work
text: To alleviate the causal effect of knowledge bias on LLMs’ reasoning abilities, future works can incorporate unbiased causal learning methods in the model pretraining or instruction tuning stage, which may enable more robust CoT reasoning. It is also interesting to study the theoretical causal foundation of CoT prompting’s mediator role in LLMs, which can be beneficial to better interpretability of black-box LLMs