It’s Not Easy Being Wrong: Large Language Models Struggle with Process of Elimination Reasoning (2024)

Abstract: Chain-of-thought (COT) prompting can help large language models (LLMs) reason toward correct answers, but its efficacy in reasoning toward incorrect answers is unexplored. This process of elimination (PoE), when used with COT, can enhance self-consistency, interpretability, and tasks such as medical diagnoses of exclusion. Thus, we propose PoE with COT, where LLMs must reason toward incorrect options on multiple-choice questions. We evaluate the ability of GPT-3.5, LLaMA-2, and Falcon to perform PoE with COT on a total of four commonsense and scientific reasoning datasets. We find that the strategy of PoE always underperforms the strategy of choosing the correct answer. The agreement of these strategies is also lower than the self-consistency of each strategy. To study these issues further, we conduct error analyses and give suggestions for future work.1

heading: 1 Introduction
text: Recent research has aimed to unlock the reasoning capabilities of large language models (LLMs) (Nye et al., 2022). As part of this effort, researchers have proposed techniques such as chain-of-thought (COT) prompting to help LLMs verbally reason toward correct answers (Wei et al., 2022b; Kojima et al., 2022). Such reasoning can improve the accuracy and interpretability of LLM decision-making (Creswell et al., 2023; Huang and Chang, 2023).
While several works use COT to select correct answers (Figure 1 blue), they do not study if COT can identify incorrect answers (Figure 1 red). Thus, drawing from test-taking strategies for multiplechoice (MC) exams (Tversky, 1972), we propose a new task to probe LLM reasoning: process of elimination (PoE) with COT. Previous works have performed PoE by discarding low-confidence options when selecting correct answers (Ma and Du, 2023),
1Our code is available at: https://github.com/ nbalepur/PoE
but whether generative LLMs can directly reason toward incorrect options with COT is unexplored.
Analyzing PoE with COT poses several benefits. First, it can give insights into the consistency of LLM reasoning (Wang et al., 2023). For 2-choice MC questions, we can study if the decisions from PoE with COT and directly answering the question with COT agree, as both should arrive at the same answer (§4.2). Second, PoE with COT can enhance LLM interpretability. COT reasoning is often seen as a rationale as to why the model thinks an option is correct, but users may also want to know why the model thinks alternatives are incorrect. Third, many applications benefit from PoE, such as reaching medical diagnoses via exclusion (Fred, 2013; Kline, 2018), troubleshooting by eliminating root causes (Gutoff and Cohen, 2006; Gugerty, 2007), and ruling out scientific hypotheses using new evidence (Norton, 1995; Forber, 2011). Thus, to enable interpretable LLMs in domains like medicine, customer service, and research, we must know: Do LLMs have the ability to perform PoE with COT?
To study this question, we independently prompt the GPT-3.5 (Brown et al., 2020), LLaMA-2 (Touvron et al., 2023), and Falcon (Penedo et al., 2023) LLMs to directly select the correct option and eliminate the incorrect one, with and without COT, on
10143
2-choice commonsense (Sap et al., 2019; Talmor et al., 2019) and scientific (Clark et al., 2018; Mihaylov et al., 2018) reasoning datasets. In this 2- choice setup, we can juxtapose LLM reasoning abilities in direct answer selection against PoE, as ideal LLMs would always have both strategies agree.
We find that PoE always underperforms direct answer selection for our LLMs, but this gap narrows as model size scales, implying that PoE with COT may only be attainable for larger LLMs (§4.1). Further, these two strategies do not reliably arrive at the same answer, revealing an inherent inconsistency in LLM reasoning (§4.2). For more insights, we analyze the errors in the rationales of PoE with COT, finding that most stem from reasoning errors and task misalignment (§4.3), and show how these 2-choice errors propagate when using PoE on full MC questions (§4.4). Based on our results, we give suggestions for future work. Our contributions are: 1) We introduce PoE with COT, a new reasoning task for LLMs that can benefit interpretability, selfconsistency, and downstream applications. 2) We benchmark the abilities of GPT, LLaMA-2, and Falcon to perform PoE with and without COT on scientific and commonsense reasoning datasets. 3) We evaluate the consistency and errors of PoE with COT to suggest directions for future research.

heading: 2 Problem Definition
text: We study selecting correct and incorrect options in MC settings, where we are given a question q and n choices C, exactly one of which is correct. To simplify our discussion, we define two strategies: 1) Direct Answer (DA) Strategy: The LLM aims to select the correct answer choice ct ∈ C. 2) Process of Elimination (PoE) Strategy: The LLM aims to select an incorrect choice cf ∈ C.
We ideally want to perform PoE iteratively, generalizing the strategy for any number of choices. But as a prerequisite, we must know: Are LLMs inherently capable of reasoning toward incorrect choices? To answer this question, we analyze the logical consistency of LLMs via 2-choice questions (C = {ct, cf}), as a robust LLM would always have DA and PoE agree: predicting that one choice is correct entails that the other is incorrect. If PoE fails with two choices, it reveals that LLMs are inherently weaker at picking incorrect options and these errors will propagate when using PoE iteratively. We first show that PoE is unreliable in 2-choice settings (§4.1), and later show this leads
to error propagation in iterative settings (§4.4).

heading: 3 Experimental Setup

heading: 3.1 Strategy Implementation
text: To provide sufficient context for the LLMs, we use few-shot (10) prompts to implement the strategies, leaving the analysis of 0-shot PoE with COT for future work. We construct the following prompts: 1) DA Base and PoE Base: These prompts task the LLM with performing the DA and PoE strategies without reasoning. The in-context examples in these prompts follow a similar format to Figure 1, but only the correct/incorrect answer choice ((A) or (B)) follow the “Correct/Incorrect Answer:” labels. 2) DA COT and PoE COT: These prompts add COT to the DA and PoE Base prompts. The fewshot examples follow the same format as Figure 1, requiring the LLM to give a step-by-step rationale before answering. Notably, the DA and PoE COT examples are created to be distinct such that DA/PoE reason toward the correct/incorrect answer without discussing the other choice’s validity. This is meant to prevent unhelpful PoE rationales like “The answer is (A). So the incorrect answer is (B).”
To study if COT reasoning improves the accuracy of each strategy (§4.1), we can compare the accuracy of the Base prompts with their COT counterparts (e.g. PoE Base vs PoE COT). Further, comparing the outcomes of the DA and PoE prompts (e.g. DA COT vs PoE COT) allows us to measure the LLMs’ logical consistency in choosing correct answers and eliminating incorrect options (§4.2).
The COT rationales are written by one Ph.D. student and verified by a second (the authors). We also prepend a natural language instruction to the prompt explaining the strategy. More prompting details and examples can be found in Appendix A.2.

heading: 3.2 Datasets
text: We examine multiple-choice commonsense and scientific reasoning questions, given that the DA strategy has shown to perform better on these questions using COT reasoning (Wei et al., 2022b; Lu et al., 2022a; Zhang et al., 2023b; Lanham et al., 2023).2
Commonsense reasoning questions have models reason about knowledge understood by most humans. We use Social IQa (Sap et al., 2019, SIQA),
2We do not use binary Yes/No datasets, like BoolQA (Clark et al., 2019), as the reason for why a correct answer is correct and an incorrect answer is incorrect are the exact same. Thus, they do not align with our requirement of distinct rationales.
containing questions about social interactions, and CommonsenseQA (Talmor et al., 2019, CQA), concerning knowledge encoded in ConceptNet.
Scientific reasoning datasets have models make logical inferences based on scientific facts. We use ARC (Clark et al., 2018) and OpenBookQA (Mihaylov et al., 2018, OBQA), testing factual recall, multi-step reasoning, and some commonsense.
We sample 500 random questions from the test set of each dataset for evaluation, and sample 10 random training examples for the 10-shot prompts. To leave each question with only two choices, we randomly eliminate all but one incorrect choice. Dataset statistics can be found in Appendix A.1.

heading: 3.3 Models
text: We study three families of LLMs. First, we use GPT-3.5 (gpt-3.5-turbo-0613) (Ouyang et al., 2022) with the OpenAI API. We also use two open-source base LLMs: LLaMA-2 (7B, 13B, 70B) (Touvron et al., 2023) and Falcon (7B, 40B, 4-bit 180B) (Penedo et al., 2023), loaded via huggingface. Each LLM decodes with 0.3 temperature.

heading: 4 Results

heading: 4.1 Can LLMs Perform PoE?
text: In Figure 2, DA Base and DA COT surpass their PoE counterparts for every model and dataset. Notably, smaller LLMs (7B, 13B) using PoE often underperform a 50/50 guessing model, suggesting that the smaller models are fully misunderstanding the PoE task, despite being given 10 examples. In contrast, the accuracy discrepancy between the PoE and DA strategies is less pronounced for larger LLMs, implying that PoE may only be an attainable
ability for larger LLMs (Wei et al., 2022a). Our results show that LLMs, especially smaller models, may have a bias toward choosing correct answers, and thus struggle to perform PoE accurately. We speculate this occurs because LLM pre-training data likely contains many MCQA questions and explanations justifying the correct choice, but far fewer explanations justifying incorrect choices, leading to a bias towards choosing correct answers.
Further, while COT tends to slightly improve the accuracy of the DA strategy across LLMs, this is not true for PoE. Specifically, DA COT has equal or better accuracy than DA Base in 19/24 cases (8/12 for LLaMA, 7/12 for Falcon, 4/4 for GPT). However, while PoE COT surpasses PoE Base in 10/12 cases for LLaMA and 2/4 cases for GPT, this improvement is never seen for Falcon, indicating a weakness in reasoning. This discrepancy across models further motivates PoE as a promising task. While COT tends to improve or maintain accuracy when picking correct answers, its accuracy wavers by LLM when eliminating choices. Thus, we believe that the accuracy of PoE with COT could be used to evaluate the reasoning abilities of LLMs.

heading: 4.2 Are PoE and DA Consistent?
text: In Table 1, we study the logical consistency of DA and PoE, i.e., if the strategies converge to the same answer. For every LLM and dataset, we find that DA more often agrees with itself upon a repeated inference (i.e. self-consistency) than agrees with its PoE counterpart. This suggests that LLMs have an innate logical inconsistency when asked to execute these two strategies, which cannot be ascribed to sampling variation from our temperature selection.
We suggest three future directions based on our
results. First, the agreement of DA and PoE can assess logical robustness. Future LLMs can aim to achieve DA/PoE consistency closer to DA selfconsistency. Second, measuring when DA and PoE agree could be used for LLM confidence calibration (Cheng et al., 2023), as an LLM that arrives at the same solution with diverse strategies may be more confident. Lastly, as LLMs have shown to improve rationales by combining multiple reasoning chains (Yoran et al., 2023), future works could similarly try to synthesize DA and PoE reasoning chains.

heading: 4.3 Why does PoE with COT Fail?
text: To study the issues of PoE, we examine 50 rationales from CQA and ARC where PoE COT failed. We use our three largest LLMs, deemed the best at PoE. Our error types are: 1) Misaligned Rationale: Justifies a choice as correct instead of incorrect or fails to justify why its selection is incorrect; 2) Reasoning Error: Error in the reasoning chain, such as an inaccurate premise or conclusion; 3) No Breakdown: Restates choice without breakdown into reasoning steps; 4) Hallucination: References non-existent parts of the question; 5) PoE N/A: Question is unsuitable for PoE; and 6) Dataset: Dataset quality issue. Examples of rationales with each error type are in Appendix Table 14.
In Figure 3, most errors stem from reasoning or misaligned rationales. The prevalence of these errors suggests that our PoE COT setting reveals
the weaknesses of our tested LLMs to reason and follow in-context instructions, making our task a suitable testbed for these abilities. Notably, many misaligned rationales occur with negated questions (e.g. “What would Grace not do?”). LLMs struggle to reason under negation (Ravichander et al., 2022) and since PoE is a negated reasoning technique, we may expect LLMs to underperform with this “double negation.” One solution to these issues is to fine-tune LLMs on PoE rationales. This may enhance PoE, but it would be interesting to see if this could also bolster overall reasoning capabilities.

heading: 4.4 Is Iterative PoE Reliable?
text: We speculated that if PoE fails with two choices, errors will propagate with full MC questions. To confirm this, we run the setup in §3 on ARC/CQA with LLaMA, eliminating i ∈ [0, n−1] choices and then selecting the correct answer. The accuracy of this strategy decreases with each iteration both with PoE Base and COT (Figure 4). Thus, we suggest that future works improve the robustness of LLMs in 2-choice settings before trying iterative PoE, as iterative PoE currently leads to error propagation.

heading: 5 Related Work
text: LLM Reasoning: Several works explore if LLMs can reason with a chain-of-thought (COT) before giving a final answer (Wei et al., 2022b; Huang and Chang, 2023). Recent works on COT probe its faithfulness (Lyu et al., 2023; Lanham et al., 2023) and planning (Saparov and He, 2023). Similar to PoE with COT, prompt techniques like debate (Michael et al., 2023) and maieutic prompting (Jung et al., 2022) produce supporting rationales for multiple options or let the model make its own decision, but we are the first to pinpoint if LLMs can reason why options are incorrect, rather than cor-
rect. Appendix C discusses these works in-depth. Process of Elimination (PoE): PoE has been studied in various settings. Parikh et al. (2018) train a neural model to perform PoE for reading comprehension. Zhang et al. (2024) and Ma and Du (2023) use LMs to find the probability that candidates should be excluded for prompt ensembling and MCQA. Tong et al. (2023) design a pipeline where an LLM proposes answers and removes incorrect proposals. While these works discard options to pick correct answers, we assess if generative LLMs can reason toward incorrect options with COT. PoE with COT also has similarities with flipped classification label studies (Wei et al., 2023), but we do not override the semantics of labels.

heading: 6 Conclusion
text: We study COT with process of elimination (PoE) and direct answer (DA) strategies in 2-choice commonsense and scientific reasoning datasets. We find that PoE underperforms DA in accuracy across all tested LLMs, but this gap narrows as model size scales. We also explore the logical inconsistencies between DA and PoE and categories of errors present in PoE rationales. Based on our results, we suggest several research directions: using PoE with COT as LLM reasoning/consistency benchmarks, combining DA and PoE with COT for calibration and refined rationales, and fine-tuning LLMs for PoE to enhance overall reasoning capabilities.

heading: 7 Limitations
text: LLMs are sensitive to prompts (Lu et al., 2022b), so the accuracy of PoE COT could vary depending on the number of in-context examples and the human-written rationales in these prompts (Wei et al., 2022b). To mitigate this, we use an ample number (10) of in-context examples and write our prompts following the same format as Wei et al. (2022b). Each rationale is written by one Ph.D. student and verified by a second Ph.D. student, both of whom work on LLM research, to ensure the rationales are high-quality. Rationales were also edited after running inference and identifying problems on a small validation set. Future works could explore rationale refinement or self-correction techniques (Zhang et al., 2023a; Fu et al., 2023; Huang et al., 2024) to have LLMs create their own prompts.
Further, while PoE should ideally be able to give a rationale as to why one choice is correct without mentioning the other choice, this may be infeasible
in certain scenarios. For example, given a mathematics question like “What is 2+2*2?” with the choices “4” and “6”, the chain-of-thought leading the correct answer is clear (e.g. “2 * 2 is 4 and 2 + 4 is 6, so the correct answer is 6”). However, it is very difficult to justify why “4” is incorrect without knowing that “2+2*2=6”. To try to avoid these types of questions, we only look at scientific and commonsense reasoning questions, as we find it to be straightforward to explain why incorrect choices are incorrect. We also use these datasets to test PoE on questions with varying levels of objectivity (i.e. commonsense questions are more subjective than scientific questions). PoE is one of many test-taking strategies, so similar to students, future works could try teaching LLMs to choose the best strategies when answering multiple-choice questions, overcoming this limitation.
Lastly, due to resource constraints, we have not tested the ability of GPT-4 to perform PoE. In future studies related to PoE, it could be beneficial to evaluate GPT-4 to see if the LLM outperforms GPT-3.5 when executing PoE. If the gap between DA and PoE is smaller for GPT-4 than GPT-3.5, it would further support the idea that larger and more capable LLMs are more adept at performing PoE.

heading: 8 Ethical Considerations
text: Process of elimination is a strategy that aims to help LLMs eliminate incorrect options, with the goal of promoting interpretability and self-consistency, as well as enhancing downstream applications. However, our current findings suggest that PoE with COT may be an unreliable technique. Thus, we advise researchers and LLM practitioners to take caution before deploying PoE in any application.
Further, as with all reasoning techniques, researchers should ensure that PoE rationales are not based on stereotypes or biases. This is especially important in some of the downstream applications we mention in this work, like medical diagnoses of exclusion, where the backgrounds of certain individuals could introduce bias into the LLM’s reasoning. We hope that future research endeavors will jointly attempt to address these biases and improve the robustness of LLM reasoning, fostering more equitable decision-making frameworks.

heading: 9 Acknowledgements
text: We would like to thank members of the CLIP lab at the University of Maryland and external collabora-
tors for their feedback and discussions of this work, including Dayeon (Zoey) Ki, Yu (Hope) Hou, Yoo Yeon Sung, Shi Feng, and Jordan Boyd-Graber. We also thank the anonymous reviewers for their feedback. This material is based upon work supported by the National Science Foundation Graduate Research Fellowship Program under Grant No. DGE 2236417. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

heading: A Experimental Setup Details
text: A.1 Dataset Statistics In Table 2, we display summary statistics for the datasets used in our experiments. All datasets are publicly available and free to use on HuggingFace.
A.2 Prompt Details Examples of the prompts used for DA COT and PoE COT on each dataset can be found in Figures 6, 7, 8, 9, 10, 11, 12, and 13. The Base versions of these prompts follow the same format, except only the answer choice follows the “Correct Answer:” and “Incorrect Answer:” labels. For example, if the question was “Where is Chicago?” with choices “Illinois” and “the Moon,” the DA Base prompt would be:
Question: Where is Chicago? Choices: (A) Illinois (B) The Moon Correct Answer: (A)
and the PoE Base prompt would be:
Question: Where is Chicago? Choices: (A) Illinois (B) The Moon Incorrect Answer: (B)
Before the in-context learning examples, we prepend the instruction “Your goal is to identify the correct answer to the multiple choice question” to the DA prompts, and the instruction “Your goal is to identify the incorrect answer to the multiple choice question” to the PoE prompts. For the 10 total in-context examples, five of the examples select choice (A) and five of the examples select choice (B), which are shuffled. This ensures that our fewshot prompts are as strong as possible. We release the full prompts along with our code.
A.3 Model Implementation The 7B open-source LLMs were run on a single NVIDIA RTX A6000 GPU. The other open-source LLMs were run on 8 NVIDIA RTX A5000 GPUs. GPT-3.5 was run using CPU only. We allocated 24 hours for a single LLM to run all four strategies (DA Base, DA COT, PoE Base, PoE COT) on all four datasets. The experiments from Figure 2
are from a single run, while the experiments from Table 1 require up to two runs.

heading: B Detailed Results
text: B.1 Detailed Quantitative Evaluation
In Tables 3, 4, 5, and 6, we provide the numerical results from Figure 2. We also calculate the difference in accuracies of the DA and POE strategies, along with their associated p-values. We find that a majority of the differences in accuracy between DA and PoE are statistically significant.
B.2 Full Consistency Experiments
We report the consistency results from §4.2 for all LLMs and datasets. In addition to the selfconsistency of DA, we include the self-consistency of PoE to ensure that the lack of consistency between DA and PoE is not due to higher sampling variance for PoE. In Tables 7, 8, and 9, we report the results for LLaMA-2. In Tables 10, 11, and 12, we report the results for Falcon. In Table 13, we report the results for GPT-3.5.
We find that the agreement of DA and PoE is the lowest across all datasets and LLMs, except for LLaMA-2 70B on Commonsense QA. We also note that PoE self-consistency is typically lower than DA self-consistency, meaning that PoE has a higher variance which further points to its unreliability.
B.3 Qualitative Results
Examples of PoE rationales that fall into the categories of our error analysis can be found in Table 14. Examples of sound rationales generated by PoE COT can be found in Table 15. In Figure 5, we compare the distribution of errors of PoE COT and DA COT. For DA COT, we use the same error categories defined in §4.3, except “PoE N/A” is replaced with “DA N/A”. Further, we only evaluate 20 DA COT rationales, since there were fewer instances of errors from DA COT to choose from.
The majority of the DA COT errors were reasoning errors (and not misaligned rationales), suggesting that the LLMs better understand the DA task compared to PoE. This may further indicate that LLMs have a training bias toward identifying correct answers. Further, we find that many of the misaligned rationale errors that occur with DA COT are also due to negated questions (e.g. “What would Grace not do?”), reinforcing the difficulty of LLMs to properly address questions with negation.
B.4 Iterative PoE Details
In this section, we provide more details on our setup for the iterative PoE experiment in §4.4. If a question has n choices, we independently prompt the LLM using PoE Base and PoE COT to eliminate an answer n−1 times, so the model is unaware of its previous decisions. At each step when there are i ∈ [2, n] choices, we also run the DA Base and DA COT prompts, corresponding to every possible use case of PoE (i.e. eliminate i choices and then choose the correct answer). PoE Base only is used with DA Base, and PoE COT is only used with DA COT.
In Tables 16 and 17, we display the results for iterative PoE on Commonsense QA and ARC, respectively. Overall, we find that the raw accuracy decreases significantly as more choices get eliminated, confirming our intuition that error propagation is an issue in this setting. Although PoE has higher accuracy than DA when the number of choices remaining is higher, this finding is not that significant when you consider the probability of performing each strategy accurately by chance. For example, when there are 5 choices, there is a probability of 45 = 0.8 of guessing an incorrect answer correctly, while there is only a probability of 15 = 0.2 of guessing the correct answer correctly. If future research eventually leads to LLMs that can perform the DA and PoE strategies at similar abilities, it would be interesting to revisit the reliability and effectiveness of iterative PoE.
B.5 Why not compare with score-based PoE techniques?
As described in the related work, previous works have explored PoE as a means to discard lowconfidence options when selecting correct answers (Zhang et al., 2024; Ma and Du, 2023). These works execute PoE by first prompting an LM to choose the correct answer, and obtain a distribution of token probabilities over all of the options. The answers with token probabilities below a certain threshold are discarded. For example, Ma and Du (2023) discard answers with probabilities that are lower than the average probability of all answers.
This approach, however, is not very useful in the two-choice setting that we study, as the model will always be consistent. To illustrate, in a 2-choice setting, the LM will ascribe choice “A” probability a and choice “B” probability b (where a+ b = 1). Assuming a > b without loss of generality, a direct
answer strategy in this context would always select choice “A” (since a > b), and PoE would always eliminate choice “B” (since b < 12(a+ b)).
Thus, comparing with these score-based PoE techniques does not give us any useful signals into the reasoning and decision-making capabilities of LLMs, as the model will always be logically consistent. In the PoE and DA setup that we describe in this work, models have the potential to be logically inconsistent, providing deeper insights into the true decision-making capabilities of our tested LLMs. Further, the aspect of COT allows us to interpret the rationales from models (§4.3), which cannot be accomplished with score-based PoE techniques.

heading: C Related Work
text: There exist a set of techniques, such as Maieutic Prompting (MP) (Jung et al., 2022), that also employ LLMs to generate rationales for multiple options in multiple-choice question answering. Below, we describe the novelty of PoE with COT and how our task is different from the setup of MP.
In MP, the LLM is asked to generate unique rationales of why a True/False question (e.g. War cannot have a tie?) is true and why it is false. In other words, MP forces the model to generate supporting rationales for both a correct answer and an incorrect answer. This is subtly but critically distinct from our setup. In PoE with CoT, the model is given a (non-True/False) question with two answer choices, and is asked to select the incorrect choice and provide a rationale for why it is incorrect. In short, MP forces a model to explain why an incorrect (T/F) answer is correct, while we ask the model to select an incorrect (non-T/F) answer and explain why it is incorrect.
For example, given the T/F question “Is 1 greater than 2,” MP would ask a model to explain (impossibly) why it is true that 1 is greater than 2. Conversely, given the non-T/F question “Where do you put your grapes before checking out,” PoE would ask a model to reason why you don’t put grapes in your mouth before paying at a grocery store (per Figure 1). These setups are inherently different.
Other techniques, such as debate (Michael et al., 2023), also fall into this same category as MP. Further, in debate, is made even more explicit in debate that LLMs may be unreliable and inaccurately arguing for an incorrect option. In contrast, PoE with COT tasks LLMs with generating accurate rationales as to why an answer is incorrect.

heading: So the correct answer is "look at Alex" which is choice (B)

heading: So the correct answer is "know how to fix it" which is choice (B)

heading: Future Work
text: If future research eventually leads to LLMs that can perform the DA and PoE strategies at similar abilities, it would be interesting to revisit the reliability and effectiveness of iterative PoE. B. 5 Why not compare with score-based PoE techniques?
As described in the related work, previous works have explored PoE as a means to discard lowconfidence options when selecting correct answers (Zhang et al. , 2024; Ma and Du, 2023). These works execute PoE by first prompting an LM to choose the correct answer, and obtain a distribution of token probabilities over all of the options. The answers with token probabilities below a certain threshold are discarded. For example, Ma and Du (2023) discard answers with probabilities that are lower than the average probability of all answers. This approach, however, is not very useful in the two-choice setting that we study, as the model will always be consistent. To illustrate, in a 2-choice setting, the LM will ascribe choice “A” probability a and choice “B” probability b (where a+ b = 1). Assuming a > b without loss of generality, a direct
answer strategy in this context would always select choice “A” (since a > b), and PoE would always eliminate choice “B” (since b < 12(a+ b)). Thus, comparing with these score-based PoE techniques does not give us any useful signals into the reasoning and decision-making capabilities of LLMs, as the model will always be logically consistent. In the PoE and DA setup that we describe in this work, models have the potential to be logically inconsistent, providing deeper insights into the true decision-making capabilities of our tested LLMs. Further, the aspect of COT allows us to interpret the rationales from models (§4. 3), which cannot be accomplished with score-based PoE techniques