Focus on Your Question! Interpreting and Mitigating Toxic CoT Problems in Commonsense Reasoning (2024)

Abstract: August 11-16, 2024 ©2024 Association for Computational Linguistics Focus on Your Question! Interpreting and Mitigating Toxic CoT Problems in Commonsense Reasoning Jiachun Li1,2, Pengfei Cao1,2, Chenhao Wang1,2, Zhuoran Jin1,2, Yubo Chen1,2,* Daojian Zeng3, Kang Liu1,2, Jun Zhao1,2,* 1School of Artificial Intelligence, University of Chinese Academy of Sciences 2The Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences 3Hunan Normal University

heading: 1 Introduction
text: With the increase in scale, large language models (LLMs) have demonstrated outstanding performance in different tasks (Li et al., 2023; Wen et al., 2023; Sun et al., 2024; Jin et al., 2024a), among them, commonsense reasoning has received significant attention due to its importance for general intelligence (Wang et al., 2022, 2024; Liu et al., 2024). In this task, researchers have proposed a series of chain-of-thought (CoT) like techniques to elicit models’ potential abilities (e.g. SelfConsistency (Wang et al., 2023c), Least-to-Most (Zhou et al., 2023), Reflexion (Shinn et al., 2023)).
*Corresponding authors.
Through them, LLMs can generate reasonable rationales and improve their reasoning performance.
While these works have made great progress, we notice an overlooked problem in them, which we define as Toxic CoT — Sometimes LLMs can directly provide correct answers to questions, but after applying CoT-like methods, it brings extra reasoning paths to models, causing their answers to be wrong.1 Figure 1 illustrates two main error types of this problem — Rationale Drift and Answer Drift. Specifically, for the Rationale Drift case, given the question “What kind of status is the bald eagle given?”, the model can directly give the correct answer “protection”. However, in the rationale, the model explains “what is the bald eagle” as “a symbol of America”, which has a semantic drift from the question. Thus, the model chooses the wrong option “america” based on the drifting rationale. For the Answer Drift case, given the question
1Notably, our definition here differs from the toxicity in generated content (Shaikh et al., 2023), emphasizing the harm that CoT can bring to the model.
9206
“Metal is used to make what?”, the model can directly answer “instruments”. It can also generate a correct rationale “metal is to make tools and machines”, but when answering based on the rationale, the model drifts from it and selects the incorrect option “(4)”. We further conduct a statistical analysis over extensive commonsense reasoning datasets and find that, among all CoT errors, this problem accounts for 37% for the white-box model and 33% for the black-box model on average, indicating this problem has become a crucial bottleneck in CoT reasoning.2
So what is the mechanism behind this issue? In this paper, we attempt to answer this question by probing the inner workings of the LLM’s CoT reasoning. Specifically, we first make initial observations on examples of Rationale Drift and Answer Drift issues, which suggest that the model likely misses some important information from the question when generating corresponding rationales or answers. To further verify these findings, we use attribution tracing and causal tracing methods to probe the LLM in two stages (rationale generation stage and answer generation stage). By employing these methods under various experimental settings, we find that there is a significant loss of information flow from the question in the shallow attention layers when generating drifting rationales and answers. Therefore, we interpret the Toxic CoT problem as the model lacking information from the question in the two stages.
To validate our interpretation and mitigate this problem, we design an approach called RIDERS (Residual decodIng and sERial-position Swap) based on the interpretation. Concretely, for the Rationale Drift issue, we devise a decoding algorithm, promoting the model to generate tokens that pay more attention to question contexts. For the Answer Drift issue, we swap the positions of the output sequence, reducing the information loss from the question to the final prediction. We evaluate our method on five commonsense reasoning benchmarks and conduct extensive experiments. The results not only prove our interpretation, but also indicate that our method is effective in addressing the Toxic CoT problem and improving the model’s overall commonsense reasoning abilities.
We summarize the contribution of this paper as follows:
2Appendix A presents the details of settings and results in this statistical experiment.
(1) We identify a crucial bottleneck affecting LLM’s reasoning performance called the Toxic CoT problem, probe this issue through attribution tracing and causal tracing methods, and interpret the mechanism behind it as the model missing information from questions in shallow attention layers. The results contribute to a more in-depth understanding of the LLM’s reasoning mechanisms.
(2) To mitigate the Toxic CoT problem, we introduce RIDERS, which effectively compensates for the internal information loss during CoT reasoning from decoding and serial-position perspectives.
(3) We conduct extensive experiments on various benchmarks. The results not only verify the rationality of our interpretation, but also demonstrate the effectiveness of our method in addressing the Toxic CoT problem (the proportion of the problem decreased by 23.6%) and enhancing commonsense reasoning performance (overall accuracy increased by 5.5%). Our code is available at: https://github.com/BugMakerzzz/toxic_cot.

heading: 2 Problem Statement

heading: 2.1 Toxic Chain of Thought Reasoning
text: We start our work by formally defining the Toxic CoT problem as follows: 3
Definition 2.1 (Toxic CoT). Given a question q and the correct answer o∗, if the model’s output M meets the following conditions, it is considered a case of Toxic CoT:
o∗ = M(q, Pd) ∧ o∗ ̸= M(q, Pc)
where o∗ = M(q, Pd) indicates the model’s direct answering for q is correct, o∗ ̸= M(q, Pc) indicates the model’s cot-like answering for q is wrong, Pd, Pc are the corresponding prompts.

heading: 2.2 Two-stage Drift Issues
text: To investigate the reasons for the problem, we classify these Toxic CoT cases and identify a main error causing this problem (On average, it accounts for 67% on two datasets, see more details in Appendix B). Furthermore, if we divide the CoT process into two stages: rationale generation and answer generation, there exist two types of issues in this error:
Definition 2.2 (Rationale Drift). If the reasoning chain is factually correct but logically inconsistent
3In practice, CoT-type methods all have Toxic CoT problems, but to simplify the work, this paper mainly focuses on the basic CoT prompting.
with the question, this case is called “Rationale Drift” (see Figure 1a).
Definition 2.3 (Answer Drift). If the reasoning chain is both factually correct and logically consistent with the question, but the final answer is inconsistent with the rationale, this case is called
“Answer Drift” (see Figure 1b).

heading: 2.3 Hypothesis Formulation
text: To provide a direction for subsequent probing experiments, here we attempt to propose hypotheses for the mechanism of issues by analyzing some examples. For the Rationale Drift issue, the model tends to focus on part of the essential reasoning conditions in the question context. As an example, in Figure 1a, the CoT only focuses on the “bald eagle” in the question but misses another key information “status”. As for the Answer Drift issue, the model seems to be disrupted by CoT, losing attention to the question and resulting in an off-topic prediction. For instance, in Figure 1b, though the CoT gives correct information “to make tools and machines”, the model can only predict the wrong answer “(4) metal fabrication shop”. This is likely because the model loses the question’s target “to make what” and directly copies the entity “metal”, which frequently appears in CoT, as the answer. Therefore, we summarize our hypotheses as follows:
Hypothesis 1. The Rationale Drift issue arises from the model lacking information from the question context in the rationale generation stage.
Hypothesis 2. The Answer Drift issue arises from the model lacking information from the question in the answer generation stage.
To validate the above hypotheses, in the following two sections, we conduct probing experiments, exploring the LLM’s internal working mechanisms during the two stages of CoT reasoning.

heading: 3 Tracing Information Flow in Rationale
text: In this section, we aim to verify the Hypothesis 1 by tracing the information flow in the rationale generation stage. To this end, we start by describing our attribution tracing method (§3.1). Through this method, we conduct comparative experiments between the correct reasoning and the drifting one, figuring out the mechanism behind the issue (§3.2). At last, we use the attention score to validate our findings from another perspective (§3.3).

heading: 3.1 Tracing Method
text: To investigate the roles of different model components in the rationale generation stage, we use attribution scores (Hao et al., 2021; Dai et al., 2022; Wang et al., 2023b) to compute the contribution of a neuron ω:
Attr(ω) = ω ⊙ ∫ 1
α=0
∂F (αω) ∂ω dα ≈ ω m ⊙
m∑
k=1
∂F ( k m ω)
∂ω
(1)
where F (·) represents the model’s output. We compute the attribution score via Riemman approximation of the integration and m is the number of approximation steps. For neurons A(l) in the i-th attention layer, we sum the absolute values of scores on all attention heads to get the final attribution score. Since the attention module involves interactions between different tokens, we can compute the information flow between the question context q and the CoT c on it:
Q(l)qc = 1 |N | ∑
(i,j)∈Cqc
Attr(A (l) i,j)
Cqc = {(i, j)|qs ≤ i ≤ qe, cs ≤ j ≤ ce} (2)
Here, Attr(A(l)i,j) represents the intensity of information flow from the i-th token to the j-th token in the l-th attention layer and |N | denotes the number of CoT steps. More implementation details of this method are reported in Appendix C.

heading: 3.2 Attribution Tracing Experiment
text: Experimental Settings To validate the deficiency in Hypothesis 1, we need to figure out the context information flow difference between generating a drifting rationale and a correct one. Thus, we first use golden labels as hints to generate correct CoTs in the drifting cases. Then, we compute the average information flow under these two cases and compare their results. We choose Llama2-13B (Touvron et al., 2023) and Baichuan2-13B (Yang et al., 2023) as our probing models, since they are moderated-sized white-box models with decent
CoT performance. For datasets, we select Winogrande (Sakaguchi et al., 2020) and CSQA (Talmor et al., 2019).4 The detailed implementation of this experiment is shown in Appendix C.
Result and Analysis Figure 2 illustrates our experimental results on Winogrande (The results on CSQA are shown in Appendix C). We can find that: (Claim 1) When the Rationale Drift issue occurs, CoT receives less information from the question context compared to the correct case. On both datasets and different models, there is a significantly lower information flow between the question context and CoT when the LLM generates a drifting rationale (the blue line) compared to the correct one (the orange line). This aligns with Hypothesis 1. (Claim 2) The shallow attention layers are crucial for LLMs to extract contextual information. In all cases, both the information flow and the gap peak at around the 15th attention layer, indicating these layers are significant sites for the rationale generation.
Supplementary Experiment In the main experiment, we use golden labels to generate the correct CoT. To eliminate the influence of this additional factor on our results, we design a supplementary experiment. Concretely, we first use the label to generate CoTs from correct reasoning cases, which serves as a control group. Then, we compute the context information divergence between the newly generated CoT cn and the original one co, i.e.:
Attr(cn|co) = Q(l)qcn −Q(l)qco (3)
where q is the question context. We compare this divergence between the control and drifting group, whose results are reported in Figure 3 and Figure 10. As we can see, the gap between the correct CoT and the drifting one (the blue line) is larger
4Unless otherwise specified, we use the same models and datasets in the following probing experiments.
than the control group (the orange line) and the max divergence occurs in shallow layers (around the 15th layer). This indicates that a correct CoT indeed gets more information flow from the context in shallow attention layers, validating the effectiveness of Claim 1 and 2.

heading: 3.3 Attention Tracing Experiment
text: Experimental Settings We also design an experiment based on attention scores to validate Hypothesis 1 from another perspective. For a pair of rationales <c, c∗> targeting the same question context q (c denotes the correct CoT and c∗ is the drifting one), we compute their attention divergence:
Attn(c|c∗) = ∑
(i,j)∈Cqc
A (l,h) i,j |c| − ∑
(i,j)∈Cqc∗
A (l,h) i,j
|c∗| (4)
Here, we replace the Attr(A(l)i,j) in Equation 2 with the weights on the h-th attention matrix head A(l,h)i,j and repeat the calculation in Equation 3.
Result and Analysis The results on two datasets are shown in Figure 4 and Figure 11. We can get the following observations: (1) The attention divergence is greater than 0 in most heads, which indicates a lack of information from the question context in Rationale Drift cases (consistent with Claim 1). (2) The largest attention divergence appears around layer 15, which is consistent with the sites we find in Claim 2. This once again illustrates that attention heads of these layers are crucial for the LLM to obtain contextual information when generating CoT.

heading: 4 Tracing Information Flow in Answer
text: In this section, our goal is to verify the information loss based on Hypothesis 2. To achieve this, we first introduce the main tracing method in this section, which is called the causal tracing method
1 5 10 15 20 25 30 35 40 Layers
co nt
ex t
op tio n co
t la st
0.0
0.1
0.2
0.3
0.4
0.5
(a) Correct Case’s Attn
1 5 10 15 20 25 30 35 40 Layers
co nt
ex t
op tio n co
t la st
0.0
0.1
0.2
0.3
0.4
(b) Correct Case’s MLP
1 5 10 15 20 25 30 35 40 Layers
co nt
ex t
op tio n co
t la st
0.0
0.1
0.2
0.3
0.4
0.5
(c) Drifting Case’s Attn
1 5 10 15 20 25 30 35 40 Layers
co nt
ex t
op tio n co
t la st
0.0
0.1
0.2
0.3
0.4
(d) Drifting Case’s MLP
Figure 5: Intervention tracing results on Winogrande in correct and drifting answering cases.
(§4.1). Next, by employing it, we trace the information flow in the answer generation stage and identify the mechanism behind the Answer Drift issue through comparative experiments (§4.2). At last, we apply the attribution tracing method to verify our hypothesis from another perspective (§4.3).

heading: 4.1 Tracing Method
text: Since the task we study is in the form of multiplechoice questions, we set our focus on the feedforward pass that predicts the label. Inspired by the previous works (Meng et al., 2022; Stolfo et al., 2023; Geva et al., 2023), we take the causal tracing method to quantify the contribution of different intermediate variables during this pass. Specifically, for hidden states h(l)i in a clean run that predicts the answer, we have:
h (l) i = h (l−1) i + a (l) i +m (l) i a (l) i = attn (l)(h (l−1) 1 , ..., h (l−1) i ) m (l) i = mlp (l)(a (l) i + h (l−1) i )
(5)
where i, l is the i-th token in the l-th layer, a(l)i ,m (l) i represents the activations of attention and MLP modules in Transformer (Vaswani et al., 2017). Supposing that a certain input part is represented as z = [v(l)i , ..., v (l) j ] after passing through a model component, we set v(l)k = v (l) k + ϵ for k ∈ [i, j] to intervene this hidden vector, where ϵ is Gaussian noise.5 Thus, we can compute the direct effect (DE) of this component:
DE(z) = P (o)− P ∗z (o)
P (o) (6)
where P (o) is the probability of the model’s final prediction, P ∗z (o) is the probability after the intervention. Therefore, through this metric, we can quantify the contributions of different components in changing the final prediction, thereby tracing the information flow in this stage.
5We select ϵ to be 3 times larger than the empirical standard deviation of hidden embeddings in each dataset.

heading: 4.2 Intervention Tracing Experiment
text: Experimental Settings We sample correct and drifting answering cases from datasets, average over them and compute the average direct effect (ADE). Here we compute the impact of four components on the final prediction: context (question contexts), option (question options), CoT, and last (the last token before the label prediction).
Results and Analysis We report the result of Llama2-13B on Winogrande in Figure 5 and others in Appendix D, from which we can get two conclusions: (Claim 3) For attention modules, drifting cases loss information from the question. When the answer is correct, we can observe a high effect on the context and option in the first layer (see Figure 5a). But for the drifting case, the LLM extracts limited information at these positions (see Figure 5c). This aligns with Hypothesis 2. (Claim 4) For MLP modules, the information is not lost. We observe the same high-effect sites in the last layer and shallow layers of the last token, they do not show regular differences (see Figure 5b and 5d).

heading: 4.3 Attribution Tracing Experiment
text: Experimental Settings For further validation of our hypothesis, we also use the attribution score in §3 to trace the information flow in this stage. Referring to Equation 2, we compute the score between the question context and the last token
(since it’s used for generating the answer). We set the F (·) in Equation 1 as the loss for predicting the final answer, comparing the scores for correct and drifting cases after averaging across samples.
Results and Analysis The results of this experiment are reported in Figure 6 and 15. We can observe that, when the Answer Drift issue occurs, the information flow from the question significantly decreases. This verifies the information loss we mention in Hypothesis 2 and Claim 3.

heading: 5 Mitigating Toxic CoT Problem
text: In this section, we propose a novel method called RIDERS (Residual decodIng and sERial-position Swap) to address the Toxic CoT problem. We first introduce the two components in it, which are designed based on Hypothesis 1 and 2, respectively (§5.1). Then, we conduct experiments on commonsense reasoning benchmarks, demonstrating the effectiveness of our method (§5.2 and §5.3). At last, we conduct extra experiments to further emphasize the contribution of our approach (§5.4).

heading: 5.1 Mitigation Method
text: Residual Decoding We design a new decoding methodology to address the Rationale Drift issue, in which we construct a virtual residual structure during the CoT generation, “connecting” the question context with each CoT token. Our decoding algorithm is demonstrated in Algorithm 1. In each iteration of generating a new token, we first select the top n tokens with the highest probabilities and record their logits scores (line 3). Then we calculate the attention score between the context and current token like Equation 4, normalize it, and add it as an additional reward to promote more information flow (lines 6,7). Finally, we select the token with the highest score to update the input and repeat the process until the termination condition is met. We use the attention matrix in layer 15 to
Algorithm 1 Residual Decoding Algorithm Require: model M, input x, question context q, candi-
date_num n, weight ω. 1: for iteration i ∈ 0, 1, ... do 2: logits = M(x) 3: tokens, scores = top(logits, n) 4: for j ∈ 1, ...n do 5: t = tokens[i] 6: attn_score = Attn(q, t) / Attn(x, t) 7: scores[j] = scores[j] + ω ∗ attn_score 8: end for 9: idx = argmax(scores)
10: t = tokens[idx] 11: x = x+ t 12: if stop(t) then 13: break 14: end if 15: end for 16: return x
compute the attention score, since it is crucial for the exchange of contextual information according to Claim 2. More implementation details of this method are provided in Appendix E.
Serial-Position Swap In this method, we attempt to compensate for the information lack in the Answer Drift issue. According to previous research on the serial-position effect in context, models tend to utilize information better at the beginning and end of the input (Qin et al., 2023; Liu et al., 2023b). In our work, the beginning of the input are prompts, while the current question and the generated CoT are both located at the end. Therefore, when they are closer to the last token, their information is more easily utilized in the final prediction. As in Figure 7, we denote the lengths of the question and CoT as Lq and Lc, and assume that the key information is located at positions µLq and λLc (similar to the center of mass in physics). We can infer that, after the swapping operation in Figure 7, the distance from the question to the end is reduced (µLq + Lc → µLq). Besides, if we consider the total distance from the question and CoT to the end, we can perform the following calculation:
d1 = µLq + Lc + λLc d2 = λLc + Lq + µLq
∆d = d2 − d1 = Lq − Lc (7)
where d1 is the total distance in normal serial positions and d2 is the distance after swapping the two components. In most scenarios, we have Lq < Lc, thus, we can infer that ∆d < 0. That means, if we replace the original order of “[Question] + [CoT]” with the order of “[CoT] + [Question]”, we can not only increase the intensify of information flow from the question to the final prediction, but also reduce the total information loss due to the reduction in total distance. Although this method is straightforward in implementation, it proves to be effective in both theory and experiments.

heading: 5.2 Mitigation Experimental Settings
text: Datasets Following previous works, we use five representative commonsense reasoning benchmarks: WinoGrande (Sakaguchi et al., 2020), CSQA (Talmor et al., 2019), HellaSwag (Zellers et al., 2019), SIQA (Sap et al., 2019) and PIQA (Bisk et al., 2020). The specific information of each dataset is reported in Appendix F.
Metrics In addition to the commonly used Accuracy (ACC) metric, we also introduce a new metric — Toxic Rate (TR), to quantify the severity of Toxic CoT problems:
TR(f) = |Cd ∩Wf |/|Wf | (8)
where Cd denotes questions that models give correct answers directly and Wf denotes questions that models give wrong answers after applying method f . Thus, we can infer that the lower the TR, the fewer Toxic CoT problems the method introduces.
Baselines As our research focuses on enhancing CoT methods in commonsense reasoning, we
select some of the latest CoT-like methods applicable to this task for comparison: Few-shot Answer, Chain-of-Thought (Wei et al., 2022), SelfConsistency (Wang et al., 2023c), Self-Refine (Madaan et al., 2023), Least-to-Most (Zhou et al., 2023) and Contrasive CoT (Chia et al., 2023). For all methods, we employ a 5-shot prompt and use 4 NVIDIA GeForce RTX 3090 GPUs for inference. More implementation details can be found in the Appendix F.

heading: 5.3 Mitigation Results
text: The main result of our experiments on Llama2-13B is shown in Table 1. (The results on more models are presented in Appendix F.) We can get the following conclusions: (1) Our method effectively mitigates Toxic CoT problems. Compared to CoT prompting, our method reduces the Toxic Rate by an average of 23.6% across five datasets. Besides, compared to other advanced CoT-like methods, our method causes the fewest Toxic CoT problems (decreased by an average of 22.0% over SOTA methods). (2) Our method can also improve the model’s overall performance on commonsense reasoning. Our work improves the accuracy on all benchmarks (improved by 5.5% compared to CoT and 3.0% compared to SOTA methods on average). This proves that the Toxic CoT problem poses a bottleneck in LLM’s commonsense reasoning, highlighting the value of our work.

heading: 5.4 Discussion and Analysis
text: Performance on Two Drifting Issues To demonstrate the effectiveness of our method in addressing the Rationale Drift issue (Type1) and Answer Drift issue (Type2), we conduct experiments on these samples and report the results in Table 2. Both of our methods can mitigate the corresponding issues (RD solves 49.3% Rationale Drift issue on average, while SPS resolves 85.3% Answer Drift issue on average). This verifies the validity of our hypothesis 1, 2, as all of these methods are built upon them. Besides, combining the two methods leads to even greater improvements, demonstrating the necessity of optimizing from both of these perspectives.
Performance in the Model In §3 and §4, we probe information loss in two issues by tracing the information flow in models. Here, we repeat the attribution tracing experiments, comparing the differences before and after applying our method to further validate the effectiveness of our work. As we can see from Figure 8 and 16, our two methods (orange lines) increase the information flow from questions in two stages compared to CoT prompting (blue lines). This indicates our method indeed compensates for the information loss in the LLM.
Performance on Other Tasks Our work mainly focuses on commonsense reasoning tasks. To further evaluate the effectiveness of our approach on other types and forms of tasks, we conduct the main experiments on two logical reasoning tasks: ProofWriter (Tafjord et al., 2021) and FOLIO (Han et al., 2022)). As presented in Table 3, we
Method Wino CSQA Hella SIQA PIQA Avg
can find that the Toxic CoT problem also exists in other tasks, and our method can still mitigate it.
Cost Analysis For the applicability, we measure the computation and time cost of our approach. Here we compare the token cost between our method and the baseline. According to Table 4, our method requires fewer tokens compared to other SOTA methods (only 1.2× cost of the basic CoT method). We also compare the time cost of our decoding method in Appendix G and find that the speed of RD is comparable to existing decoding strategies. Therefore, we illustrate the costefficiency of our approach across different tasks.

heading: 6 Related Work

heading: 6.1 CoT Problems Analysis and Mitigation
text: Recently, many works have focused on analyzing and mitigating problems in CoT reasoning. For analytical work, most studies focus on blackbox LLMs. Through intervening or paraphrasing prompts and comparing outputs, researchers can interpret the reasons leading to errors in the model’s reasoning (Lanham et al., 2023; Liu et al., 2023c; Wang et al., 2023a). For optimization works, they design additional supervision signals or training processes for the model (Ramnath et al., 2023; Liu et al., 2023a) or leverage external resources for the model (Shinn et al., 2023; He et al., 2023; Lyu et al., 2023). However, these works lack the probing of inner mechanisms behind these problems, leading to insufficient analysis or less universally applicable optimization methods.

heading: 6.2 Mechanistic Interpretability
text: The work on mechanistic interpretability aims to understand the internal mechanisms of models when performing various tasks. Early work focused on how the model stores factual knowledge internally (Meng et al., 2022; Dai et al., 2022). In recent times, some research efforts have shifted towards examining how models retrieve and utilize knowledge. This includes internal knowledge
retrieval (Geva et al., 2023), knowledge retrieval from prompts (Wang et al., 2023b; Jin et al., 2024b), and the utilization of knowledge for reasoning purposes, such as math reasoning (Stolfo et al., 2023) and multi-step reasoning (Hou et al., 2023). However, there is limited existing work that explains commonsense reasoning and CoT reasoning, which are significant contributions of this work.

heading: 7 Conclusion
text: In this paper, we find a problem named Toxic CoT, which results in the model’s reasoning deviating from the original correct answer when utilizing CoT-like prompting. Through tracing the internal information flow of the LLM with attribution tracing and causal tracing methods, we prove that this problem is mainly caused by the model’s lack of information from the question in shallow attention layers when generating rationales or answers. Based on this result, we propose the RIDERS method to mitigate the Toxic CoT problem from both decoding and serial position perspectives. Through extensive experiments on multiple commonsense reasoning datasets, we verify the effectiveness of our approach in mitigating Toxic CoT problems and enhancing the model’s overall commonsense reasoning capabilities.
Limitations
Although our work conducts an in-depth interpretation and mitigation of the Toxic CoT problem, it has several limitations. Firstly, like former commonsense reasoning works (Liu et al., 2022, 2023a; Xie et al., 2023), our research focuses on the form of multi-choice questions. This stems from the absence of effective evaluation methods for openended commonsense reasoning, leading to the predominance of benchmarks in this format. This calls for advancements in benchmark-related research. Secondly, we refrain from analyzing Toxic CoT problems in more reasoning tasks such as math, primarily due to the poor performance of current moderately-sized white-box models on these tasks. For instance, Llama2-13B achieves a mere 7.2% accuracy on GSM8K (Cobbe et al., 2021) without utilizing the CoT technique. This calls for developments in model-related research. We leave these limitations as our future work to explore.

heading: Acknowledgement
text: This work is supported by the Strategic Priority Research Program of Chinese Academy of Sciences (No. XDA27020203), the National Natural Science Foundation of China (No. 62176257,62276095).

heading: A Early Statistical Experiments
text: In this section, we conduct early experiments on existing representative commonsense reasoning datasets to analyze the prevalence of Toxic CoT problems through statistical methods.
Datasets We utilize five representative commonsense reasoning datasets to analyze the distribution of Toxic CoT problems. The basic information of the dataset is outlined in Table 5. It is noteworthy that, owing to the extensive size of Hellaswag’s dev set (over 10,000), we extract 2,000 instances for the experiment.
Metric We design a new metric called Toxic Rate, which measures the proportion of Toxic CoT problems among all errors. Its calculation method is shown in Equation 8.
Results The result of our early statistical experiments is reported in Table 6. Here we use Llama213B-Chat-hf to present the white-box LLM and use GPT-3.5-turbo-1106 to present the black-boxmodel. The average Toxic Rates are as high as 37.0% and 32.8% across the five datasets, indicating that this issue cannot be ignored and warrants further investigation.

heading: B Toxic Reason Statistical Experiments
text: In this section, we manually categorize the error types of Toxic CoT problems through statistical classification. Specifically, we sample 1,000 examples from CSQA and 1,000 examples from Winogrande, classifying the Toxic CoT problems (In all of the probing experiments in the main text, we use these samples as our probing data). The results are presented in Table 7. In the inconsistent error, the model exhibits logical inconsistency with
the preceding context when generating CoT or the final answers. In the factual error, the CoT contains incorrect factual knowledge, which leads to erroneous answers. The presence of question errors reflects the subpar quality of the dataset. In such cases, questions may exhibit multiple viable answers or all options are incorrect. As for the other error, the questions trigger certain refusal-toanswer mechanisms in the model (e.g., inquiries about how to commit murder), leading to the identification of incorrect answers.
As the inconsistency error constitutes the predominant portion of all reasons, our work focuses on addressing this issue. We further categorize this error into Rationale Drift and Answer Drift based on the error occurrence (see §2.2 for their definitions).

heading: C More Details for Reasoning Tracing
text: Method Implementation We introduce the attribution score method in § 3.1. In Equation 1, we set m = 20 following the previous works. For F (·), we set it as the language modeling loss (for next-token prediction) during the CoT generation. Here, we obtain this value directly from the output of the LlamaForCausalLM module using the Transformers library. In Equation 2, we partition the step num-
bers |N | in CoT based on the occurrence of periods in the text. For models, we use Llama2-13B-Chat and Baichuan2-13B-Chat.
Attribution Tracing Experiment Figure 17 illustrates the prompt we use for generating correct CoTs from drifting cases. After the generation, we will manually filter out the wrong CoT and conduct the comparative experiment. We use a 5-shot to generate CoT and concatenate it to the question for our probing experiments. Figure 9 shows the remaining results of this experiment, from which we can get the same conclusions as Section 3.2. Additionally, we also conduct the supplementary experiments on CSQA and report the results in Figure 10.
Attention Tracing Experiment Figure 11 reports more results in this experiment.

heading: D More Details for Answering Tracing
text: Intervention Tracing Experiment Figure 12, 13, 14 and 15 report the remaining intervention tracing experiment results, which are consistent with our conclusion in the main text.

heading: E Mitigation Method Implementation
text: Residual Decoding Here, we provide a detailed explanation of Algorithm 1. At the beginning, we set the input to the entire question (contexts + options). In line 2, we get the logits from the output of the LlamaForCausalLM. In line 6, we calculate the attention score by summing the values on the attention matrix corresponding to the tokens. We use the output character “
” as the termination condition for Llama2-13B generation in line 12.
Serial-Position Swap In this method, we swap the positions of the question and the generated CoT, outputting the option with the highest logits score. This method can be implemented under both fewshot and zero-shot settings, demonstrating its costefficiency.

heading: F More Details for the Mitigation Experiment
text: Dataset In this experiment, the specific information of all datasets can be found in Table 5.
Baselines For the Self-Consistency method, we sample 5 CoTs and use a majority voting method to select the final predicted answer. For the Self-Refine method, we first conduct one round
of CoT reasoning and then follow it with one round of feedback to generate the final answer. For all the baselines, we release the prompts in our source code. We implement all of the methods on Llama2-13B-Chat-hf and Baichuan2-13B-Chat.
Our methods In our RD method, we set two hyperparameters — candidate_num n and weight ω, and here are their specific values in the experiments: for Winogrande, we set n to 4 and ω to 80, for CSQA, we set n to 10 and ω to 135, for HellaSwag, we set n to 3 and ω to 80, for SIQA, we set n to 10 and ω to 160, for PIQA, we set n to 4 and ω to 120. Additionally, in Figure 18,19,20,21,22, we list our method’s few-shot prompts on five datasets. Note that both CoT prompting and our two methods utilize the same prompt.
Results In our paper, we choose Llama2-13B for experiments because it is an open-source LLM of suitable size and has great influence. Here, we also repeat all experiments on Baichuan213B to verify the generality of our work (see Table 8). Additionally, in Table 9, we show our experimental results on two other representative models: Mistral-7B-Instruct-v0.2 and gpt-3.5-turbo-1106 (since it is a closed source model, here we only apply the SPS method).

heading: G More Details for Cost Analysis
text: Table 10 illustrates the time cost comparison of our residual decoding methods with other strategies. Here we set num_beams in the beam search strategy as 5, and candidate_num in the RD strategy as 3. We compute the average seconds cost per example over 50 samples for each dataset. On average, our decoding strategy takes 2.6 times longer than greedy search and 1.4 times longer than beam search. This reflects that our decoding method has significantly stronger performance while having a comparable overall time to these main decoding strategies.
System Instruction: You are a helpful, respectful and honest assistant. You
should use your reasoning abilities to answer the given questions in reasoning
tasks. You should reply the correct rationales and the answer. Your response
should be in this form: '{reason} So the answer is: ({option}) {answer}'. If
you don't know the answer to a question, please reply 'Answer: None’.
Question: The test was hard for Samuel but a breeze for Randy , since _ had
failed to study for it.\n(1) Samuel (2) Randy
Answer: To pass a test, a person need to study for it. If a person feel the test
like a breeze, it means the test is easy for him. A person feels the test easy,
because he studies hard for it. Since we know that Samuel feel the test very
hard, she may fail to study for it. So the answer is: (1) Samuel.
Question: Kyle slowly wormed their way into the life of Derrick, because _
was good and manipulating people.\n(1) Kyle (2) Derrick
Answer: A person wormes his way into other's life, because he is friendly
and approachable. A friendly person is considered good. A person is seen as
manipulating people, that means he like to interact with others and others like
him. Since Kyle slowly wormed their way into the life of Derric, Kyle will be
seen as good and manipulating people. So the answer is: (1) Kyle.
Question: Donald was very grounded but Michael often got lost in their
daydreams. _ was very capricious all the time.\n(1) Donald (2) Michael
Answer: A person is grounded means he works hard and does not like to
fantasize. A person often gets lost in his daydreams, he is seen as unrealistic
and egocentric. A person is capricious all the time means he does everything
only according to his own ideas. Since Michael often gets lost in their
daydreams but Donald does not, Michael is seen as very capricious all the
time. So the answer is: (2) Michael.
Question: After stopping when running, Betty was able to help Rachel treat
their asthma because _ has it too.\n(1) Betty (2) Rachel
Answer: A person who has a mild disease may knows how to treat it. Since
Since Betty was able to help Rachel treat their asthma, Betty may has it, too.

heading: So the answer is: (1) Betty.
text: Question: Jeffrey was sick with the stomach flu and not hungry, but
Christopher was starving. _ ordered food.\n(1) Jeffrey (2) Christopher
Answer: Stomach flu is a viral infection in the digestive system. A person is
starve so that he need to eat something next. A person is not hungry means he
does not want to eat something. Ordering food means to buy food online. A
person may eat food after ordering it. Since Christopher was starving and
Jeffery was sick and not hungry, Christopher tended to order food. So the
answer is: (2) Christopher.
Figure 18: 5-shot prompts for Winogrande.
System Instruction: You are a helpful, respectful and honest assistant. You
should use your reasoning abilities to answer the given questions in reasoning
tasks. You should reply the correct rationales and the answer. Your response
should be in this form: '{reason} So the answer is: ({option}) {answer}'. If
you don't know the answer to a question, please reply 'Answer: None’.
Question: Quinn wanted to help me clean my room up because it was so
messy. What will Quinn want to do next? \n (1) Eat messy snacks (2) help out
a friend (3) Pick up the dirty clothes
Answer: Quinn want to clean the room up. Picking up the dirty clothes is one
way to clean the room. Thus, quinn will want to pick up the dirty clothes next.

heading: So the answer is: (3) Pick up the dirty clothes
text: Question: Sasha's mom passed out in the middle of the party. Aubrey took
Sasha's mom to the hospital. What will Aubrey want to do next? \n (1) help
Aubrey go back home (2) keep on partying without the mom (3) going on
with the mom
Answer: Sasha's mom is sent to the hospital beacuse she passes out. After
that, she becomes a patient. A patient needs to be cared for. Aubrey may need
to take care of her. Thus, Aubrey will go on with the mom. So the answer is:
(3) going on with the mom
Question: Their cat kept trying to escape out of the window, so Jan placed an
obstacle in the way. How would Jan feel afterwards? \n (1) scared of losing
the cat (2) normal (3) relieved for fixing the problem
Answer: The cat try to escape so Jan need to stop it to avoid losing the cat.
Jan place an obstacle in the way so the cat can not escape. The problem is
been solved. Thus, Jan will fell relieved for fixing the problem. So the answer
is: (3) relieved for fixing the problem
Question: Sydney had so much pent up emotion, they burst into tears at work.
How would Sydney feel afterwards? \n (1) affected (2) like they released their
System Instruction: You are a helpful, respectful and honest assistant.
You should use your reasoning abilities to answer the given questions in
reasoning tasks. You should reply the correct rationales and the answer.
Your response should be in this form: '{reason} So the answer is:

heading: Future Work
text: We leave these limitations as our future work to explore