POE: Process of Elimination for Multiple Choice Reasoning
Authors: Chenkai Ma
Xinya Du
Venue: EMNLP 2023 Main (2023)
Keywords: language models
prompting
scoring
multiple choice reasoning
TL;DR: We present a two-step scoring method for multiple choice reasoning tasks, based on the process of elimination.

Abstract: Language models (LMs) are capable of conducting in-context learning for multiple choice reasoning tasks, but the options in these tasks are treated equally. As humans often first eliminate wrong options before picking the final correct answer, we argue a similar two-step strategy can make LMs better at these tasks. To this end, we present the Process of Elimination (POE), a two-step scoring method. In the first step, POE scores each option, and eliminates seemingly wrong options. In the second step, POE masks these wrong options, and makes the final prediction from the remaining options. Zero-shot experiments on 8 reasoning tasks illustrate the effectiveness of POE, and a following analysis finds our method to be especially performant on logical reasoning tasks. We further analyze the effect of masks, and show that POE applies to few-shot settings and large language models (LLMs) like ChatGPT.

PoE: Process of Elimination for Multiple Choice Reasoning
Chenkai Ma' and Xinya Du²
Sccool of Computer Science and Engineering, University of Electronic Science and Technology of China Deparment of Computer Science, University of Texas at Dallas xinya. wwwaaaaallldallscccmmmmm edu
kasmas3 1 6@gmail. com
Abstract
The wayk'
tringe
Select nulaaile opriin
ttat glowe Queetore Weech the taklwing
the question. Ignare DMASIQ options.
Language models (LMs)) are capable of con- ducting in in-context learning for multiple choice reasoning tasks, but the options in these tasks are treated equally. As humans often first elim- inate wrong options before picking the final correct answer, we argue a similar two-step strategy can make LMs better at these tasks. To this end, we present the Process of Eliiination (PoE). two-step scoring method. In the first step, POE scores each option, and eliminates seemingly wrong options. In the second step, POE masks these wrong options, and makes the final prediction from the remaining options. Zero-shot experiments on 8 reasoning tasks il- lustrate the effectiveness of POE, and a follow- ing analysis finds method be especially performant on logical reasoning tasks. We fur- ther analyze the effect of masks, and show that POE applies to few-shot settings and large lan- guage models (LLMs) like ChatGPT.
perkeroee basl chanadarioa mano Optione:
The word naiee' solecting things
I13t glow Qusetton Weich ofth tolowing
Wayey grapes are easy pick at
sertancea best chanadarioes wajyy gropes?
Wajpy gaares ore size solkutt
Rlasied Qycions
Wheay grapes have sutia tara oftiiccct
Whjey giases easy pick right.
Weer gropes wormo
[MARK]
grapes
DMASKG
X
Elivinated ( Wrong Options: B and D.
Real Producionn
First step: elmination
Second step: prediction
Figure I: An illustration of the Process of Elimination (PoTE) for a multiple choice question. In the first step, POE eliminates some wrong options. the second step, it enforces the masks, and makes the final prediction.
option. One common score is language model- ing likelihood (P(w)zz), Brown et al., 2020), and there are also many alternatives (Zhao et al., 2021; Holtzman et al., 2021; Fei et al., 2023; Min et al., 2022; Malkin et al., 2022; Robinson and Wingate, 2023; Lyu et al., 2023). Tailored for powerful LLMs (Chowdhery et al., 2022; Chen et al., 2021), prompting methods (Wei et al., 2022: Wang et al., 2023b; Kojima et al., 2022; Besta et al., 2023; Yao et al., 2023) append all options to the question, and prompt the model to generate a sequence, then extract the option from the sequence.
1 Introduction
How often have said to you that when you have eliminated the impossible what- ever remains, however improbable, must be the truth? (Doyle, 1890)
In natural language processing, many reasoning tasks are multiple choice-based, in which model chooses the best option from several options, given question. Current LMs exhibit remarkable per- formance on diverse reasoning tasks, like common- sense reasoning (Wang al., 2023a: Holtzman et al., 2021), logical reasoning (Ye et al., 2023), and arithmetic reasoning (Shum et al., 2023).
One limitation of these two types of approaches is that they both treat each option equally, i.e., they either consider each option independently, or con- sider all options jointly. On the contrary, when hu- mans solve a multiple choice reasoning task, they often eliminate some wrong options, then choose from the remaining ones. This so-called process of elimination is widely used in college exams and can be stronger than usual Gold style learn- ing (Freivalds et al., 2002). Therefore, assume that language models can similarly benefit from this elimination process, i.e., eliminating wrong op- tions and choosing the best option are two types of reasoning that can be disentangled into two steps.
There are two types of in-context learning meth- ods for multiple choice reasoning: scoring and prompting. Given the question, scoring methods score each option, and select the highest-scored
'Code is available at https: :/gi thwb. www.ttt......aaaatheeeee cow/KasMasVant PoE.

---

To this end, we present the Process of Elimina- tion (PoE), two-step scoring method for multiple choice reasoning, as shown in Figure 1. the first step, PoE scores each option, then eliminates some wrong options based on their scores. In the second step, it masks these wrong options, then chooses the best one from the remaining options. We con- duct experiments on 8 reasoning tasks that cover diverse domains. POE achieves the best zero-shot performance on most tasks. following analysis shows that it favors logical reasoning tasks. We also measure the effect of masks, and find our method applicable to few-shot settings and compatible with LLMs like ChatGPT (Ouyang et al., 2022).
We also implement prompting- variation of POE for LLM (Section 5).
The first step of PoE is elimination, in which it eliminates some wrong options. PoE starts by scoring each option. Then, unlike common scoring methods that choose the highest-scored option, it uses the scores to climinate some options with low scores. In particular, PoE computes the average score of all options, and eliminates options whose score below average, i.e., Ywrong
Y Wronng = {uils < avg(si....,..))
(5)
The intuition behind this elimination strategy ("Below Average") is that the scores of wrong op- tions deviate from others, and are thus easy to iden- üfy, which we verify in Appendix B.2. We compare other elimination strategies in Section 5.2.
Our contributions are twofold: (1) We present PoE, two-step scoring method for multiple choice reasoning; (2) Through comprehensive ex- periments and analysis, we demonstrate the effec- tiveness and generalizability of POE.
The second step of PoE is prediction, which chooses the best answer that is not in Ywrong- Specif- ically, POE computes binary masks m, for options:
2 Method
Problem Setting. A multiple choice reasoning task instance includes question x, several options Y = (....... )n and the correct option y. There are two kinds of in-context learning approaches to this problem: scoring and prompting.
0,
if ys C Ywrnng otherwise
mi
(6)
1.
Scoring uses an LM to compute a score for each option yi, and chooses the highest-scored option:
mask = [m],..., m,,.
(7)
For these options, POE enforces the masks.? 2 (1) particular (shown in Figure 1), it uses a template T to first wrap the question with all options and their symbols like "A", then append an instruction to the question which asks the model to neglect masked options, and finally replace eliminated options with special text sequence "LMASK]":
8/ score(x, Wt),
9 arg max 8,.
(2)
A commonly-used score is language modeling like- lihood (P(s (w) Brown et al., 2020), More recent scores include average log probability (Brown et al., 2020), calibrated log probability (Holtzman al.. 2021), and channel (Min et al., 2022). As shown Equation 1, most scoring methods consider each option in isolation, except multiple choice prompt- ing (Robinson and Wingate, 2023).
Tmaak T(x, Y, mask).
(8)
Then, POE scores each option, and chooses the highest-scored option, during which the scores of eliminated options are set to negative infinity:
In contrast, prompting methods provide all the options in the input (Wei et al., 2022: Wang et al.. 2023b; Kojima et al., 2022). The model then gen- erates output from the input. Finally, these methods extract the option from the raw output:
score( Vi), if yi & Y Ywrong inf.
8mask,
(9)
otherwise
raw output- = generate(x, Y),
(3)
ÿ = arg max maakk
(10)
(ra output).
(4)
We don't remove wrong options because that requires extra work like relaboling remaining options (c.g., "ADE" to "ABC"), but brings negligihle difference to performance.
Our method a two-step scoring method that considers all options but treats them differently.
POE.

---

Method
Task
LM
AVG
Calibration
Channel
MCP
PoE
ANLI
38.62.9
38.03.4
37.23.5
36.02.7
57.83.0
55.02.4
CQA
64.45.9
54.21.
69.64.8
44.65.6
87.22.6
89.22.2
SIQA
56.04.5
62.61.8
57.66.9
39.85,9
79.05.6
82.06.7
LD
48.82.7
45.84.5
39.24.0
20.83.6
39.83.7
53.67.0
DQA
45.86.6
51.820
48.02.9
39.66.6
67.81.9
67.42.5
CC
44.80.8
51.80.
54.40.5
44.0p.7
60.21.5
72.20.8
SS 39.02.5
40.61.8
43.41.7
26.61.7
74.00.7
76.60.9
SIT
24.81.9
20.65.3
21.03.9
17.04.6
25.42.9
25.24.0
Table 1: Accuracy scores (with standard deviation) on 8 tasks. The best scores are boldfaced, and the second -best scores underlined. LM refers language modeling, AVG refers to average language modeling, MCP refers multiple choice prompting. Our method (PoE) achieves the best or comparable performance on all tasks.
3 Experiment Setup
Settings. We evaluate POE in the zero-shot set- ting. We use accuracy as the metric, and average results over random seeds. We consider other settings in Section 5.3 and 5.4.
Data. We consider 8 multiple choice reasoning tasks to cover diverse domains. We include three traditional reasoning tasks: ANLI (Nie et al., 2020), CommonsenseQA (CQA, Talmor et al., 2019), and Social IQa (SIQA, Sap al., 2019). We also in- clude five BIG-bench tasks (Srivastava et al., 2023), with the first two from BIG-Bench Hard (Suzgun et al., 2023): Logical Deduction (LD), Disambigua- tion tion_QQ (DQA), Conceptual Combinations (CC), Strange Stories (SS), and Symbol Interpretation Task (SIT). For all tasks, we use their test sets if available, otherwise their development sets. To reduce cost, sample 100 instances from each task. We present more task information and pre- processing in Appendix A.
Results
In Table 1, we compare POE with other baselines on reasoning tasks, and present the accuracy and standard deviation. Our method achieves the best or second-bess performance on all 8 tasks. We notice that multiple choice prompting (MCP) is strong baseline that consistently outperforms other base- lines. Nevertheless, PoE beats MCP on tasks, and comparable on the remaining three. Since POE uses MCP in both steps, the result demon- strates the effectiveness of the former, especially climination. In addition, PoE is exceptionally per- formant on CC, beating the second-best method by 12% absolute. This corroborates our hypothesis two types of reasoning for multiple choice tasks, because while MCP sometimes fails to simultane- ously eliminate wrong options and choose the right option, PoE disentangles the two jobs in two steps, and thus achieves better performance. We present case study in Appendix B.2.
Model. We use FLAN-TS-XL (3B) (Chung et al., 2022) in both steps of POE, because it is an economical and performant instruction-tuned model. In Section 5, we also apply PoE to LLMs like ChatGPT (Ouyang et al., 2022).²
Methods. We consider scoring baselines: lan- guage modeling (LM, baseline in Zhao et al.. 2021), average language modeling (AVG, Brown al., 2020), calibration (Holtzman et al., 2021), channel (Min et al., 2022), and multiple choice prompting (MCP, Robinson and Wingate, 2023). For POE, we use MCP in both steps. We discuss other possi- ble scoring methods and elimination strategies for PoE in Section 5.2, and present input and output samples for all methods in Appendix C.4
Analysis
5.1 When is POE Better than MCP?
In Table 1, we find MCP to be the best baseline, and sometimes even beats POE. This finding leads us to wonder: when (on which tasks) can we expect PoE to dramatically outperform MCP?
"W not use recent larger instruction mnstluction-follooing models based on LLaMA (Touvron et al., 2023), because 1) they are not optimized for traditional NLP tasks, and 2) they evolve quickly, which makes it hard fair comparison.
Thought Prompting (Wei et al., 2022). because they are tai- lored for very large (c.g.. 175B) models.
WWo do not include prompting baselines like Chain-of-

---

[eeoring Mettod, Cimmation Serseggii
Method
Task
MCP
PoE
PoE MCP
LA
50.0
68.8
+18.8
IMT
34.0
47.2
+ +13.2
CLD
67.2
75.9
+8.7
iCaltrationn Bele Acerpee)
(C (Caliration Lowest)
(Chennet Aetore Avecppet
iChannll Lowesst
(ts, entre Averopni
ILAL, Lewest
oNce Betow Averspe
(NCS Lowentt
RACO
53.8
60.6
+6.8
CAT
84.1
81.8
-2.3
EIE
25.0
19.1
-5.9
Figure 2: Effect of different scoring methods and elim- ination strategies on masking accuracy mAce ) and final accuracy (Acc). The best configuration is (MCP, Lowest), with a large gap between AcCmusk and Acc.
RS
55.1
49.0
-6.1
10M
56.2
50.0
-6.2
Table Comparison of MCP and PoE accuracy scores on 8 new tasks. The top 4 tasks are logical reasoning tasks. POE largely outperforms MCP on logical rea- soning tasks, and underperforms MCP other 4 tasks.
tion of POE. We also find it underperforms MCP on control tasks (bottom 4 tasks), which means our method is not consistent on commonsense or social reasoning tasks. These findings support our hypothesis that compared to MCP, PoE is most performant on logical reasoning tasks.
We start by calculating their performance gaps (PoE MCP) on all 8 tasks. We find the gaps on LD (13.8%) and CC (12%) are much larger than those of other tasks (less than 3%). Furthermore, we find both LD and CC are logical reasoning tasks, while other tasks rely more on commonsense or social reasoning." We thus hypothesize that PoE is bet- ter than MCP on tasks that mainly require logical reasoning, while not being consistently performant on social and commonsense reasoning. One expla- nation is that logical reasoning general skill, whereas social or commonsense reasoning relies on specific knowledge, some of which may be absent in certain language models.
5.2 What Makes Good Masks?
As shown in Section 2, scoring methods and elim- ination strategies jointly determine which options to eliminate (Y WYwong) and create masks accordingly. In this section, we analyze the effect of using dif- ferent configurations of these two factors." We define masking accuracy (AcCmask) as the ratio of instances that have their correct option kept after elimination. AcCmask represents the upper bound of POE, which we aim to maximize to reduce er- ror propagation (Du and Cardie, 2020). We mea- sure two quantities: Accmask and final accuracy (Acc). We consider four scoring methods (Cali- bration, Channel, LM, MCP) and two elimination strategies ("Below Average" and "Lowest", the lat- ter of which means eliminating the option with the lowest score). We average zero-shot results over all 8 tasks in Table 1.
To verify this hypothesis. we compare PoE and MCP in the zero-shot setting on new tasks from BIG-Bench, including logical reasoning tasks: Logical Arguments (LA), Identify Math Theorems (IMT), Code Description (CLD), Reasoning about Colored Objects (RACO); and 4 control tasks that are based on commonsense or social reasoning: Counterfactual Conditionals (CAD, The Essential, the Excessive, and the Extraneous (EIE), Riddle- Sense (RS), Identify Odd Metaphor (IOM). We present task information in Appendix A.
As shown in Figure 2, the best scoring method is MCP, and the two elimination strategies are com- parable each other. We find that higher Accmask leads higher Acc. The best configuration of scor- ing method and elimination strategy (MCP, Lowest) leads to 91.9% Accmask." This means a moderate- sized LM (FLAN-TS-XL) can climinate wrong op-
As shown in Table 2, we find PoE consistently and dramatically outperforms MCP on all 4 logical reasoning tasks (top 4 tasks). We find the largest performance gap LA (18.8%), task of GRE- style logical questions. Human test-takers com- monly use elimination- methods to solve such GRE questions, which further supports the motiva-
"We also tried mask tokens other than "[MASK]". but did not find one consistently better than others. So we don't tune this token to avoid unnecessary costs.
We use (MCP, Below Average) in the main experiment, because this configuration makes PoE more separable on tasks that it performs well, i.c., the gap between PoE and the sccond-hest baseline is larger.
SSIT is another logical reasoning task, but it also requires visual reasoning, which is not idcal for language models.

---

ITaae It Shott
Method
Task
MCP
POE
POE MCP
SIT
2.0
19.4
+17,4
ANLI
40.2
44.6
+4.4
SS
79.4
82.8
+3.4
CQA
75.0
77.0
+2.0
LD
37.6
38.0
+0.4
LMRLL
SIQA
72.6
71.2
-1.4
DQA
48.8
42.6
-6.2
Figure 3: Three-shot and zero-shot accuracy scores on ANLI, CC, and LD. Although PoE shows mixed performance, it is applicable to few-shot settings.
CC
79.6
66.8
12.8
RACO
4.8
50.4
+45.6
IMT
34.0
49.8
+15,8
LA 50.0
55.0
+5.0
the scope of this work.
CLD
92.4
91.4
-1.0
5.4 Does POE Work in Few-Shot Settings?
Table 3: PoE's zero-shot performance on ChatGPT 3opt-3.5-turbo 3). The bottom 4 tasks are logical reasoning tasks from Section 5.1. POE consistently outperforms MCP, especially on logical reasoning tasks.
To measure POE in the few-shot settings, we com- pare zero-shot and three-short performance with MCP on ANLI, CC, and LD. We build three-shot demonstrations by randomly sampling from the training sets of ANLI and test sets of CC and LD, 8
tions while keeping the correct ones most of the time. The corresponding 68.7% Acc means large powerful
We present the result in Figure 3. For ANLI, POE underperforms MCP in both settings, but their performance gap drops from 5.3% (zero-shot) to 1.5% (three-shot). For LD and CC, PoE outper- forms MCP in both settings, but their performance gap similarly diminishes in the three-shot setting. Comparing PoE with MCP, We find three-shot PoE to be less performant than its zero-shot coun- terpart, and possible reasons include: 1) we did not optimize prompts or demonstrations for three- shot POE; 2) the instructions we use in zero-shot PoE are already powerful, and three-shot demon- strations may introduce some noise. Still, our find- ings suggest PoE is applicable few-shot settings, which we will continue explore in future work.
performance gap, which suggests LM like FLAN-T5-XXL for prediction. We also find Y Wrong and masks to make POE more inter- pretable, because they provide intermediate reason- ing outputs. Enforcing masks in prediction also makes our method faithful and factual.
5.3 Does POE Work with LLMs?
To measure whether POE is compatible with LLMs. we implement prompting-based variation of PoE and apply it ChatGPT (g-t-3.5-turbo S-turbo-06111 We then measure its zero-shot performance on original tasks in Table and 4 logical reasoning tasks in Table 2. For POE, we only use ChatGPT in the second step (prediction), as the base LM suffices for masking (Section 5.2). Concretely, we prompt ChatGPT to complete Timssk and extract the last occurrence of any option symbol as the prediction. We compare POE MCP, which is similarly modified work with ChatGPT.
Conclusion
We present POE, a two-step scoring method for multiple choice reasoning. PoE eliminates some wrong options in the first step, and chooses from the remaining options in the second step. PoE per- forms well on reasoning (especially logical reason- ing) tasks in the zero-shot setting, and also works with LLMs or in the few-shot setting. In the future, we will improve the generalizability our method, and use fine-tuning to better enforce the masks.
We present the result in Table 3. The result from ChatGPT is consistent with those from FLAN-T5- XL (Table 1): PoE beats MCP on 5 out of the 8 original tasks, and performs well on 4 logical rea- soning tasks. These findings suggest our method also works with LLMs. Nevertheless, we find Chat- GPT underperforms FLAN-T5 XL on some tasks, which requires further experiments and is beyond
"For CC and IDD, we run experiments on the remaining data after sampling.

---

Limitations
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint wrXiv:2204.02::1
The first limitation of our work is the enforcement of the masks. Although we replace eliminated op- tions with "[MASK]". and also replace their scores afterward, the model still considers these options in prediction. It would be better if the model does not consider these options at all. A second limitation is that we do not fully explore the components of PoE: We fix the prompts, which may not be op- timal. A third limitation of our work is the scope of the LLM experiment: We only test in the zero- shot setting. It would be better if we also run few- shot LLM experiments, and incorporate complex prompting methods like Chain-of-Thoughh Prompt- ing (Wei et al., 2022). A fourth limitation is that this work considers only one modality, i.e., lan- guage. It would be interesting to extend PoE to tasks involving other modalities, like vision (Seo et al., 2022).
Hyung Won Chung, Le Hou, Shayne Longpre, Bar- Zoph, Yi Tay, William Fedus, Fric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022. Scaling instruction finetuuned language models. arXiv preprint arXiv:2210.1 1416.
Arthur Conan Doyle. 1890. The Sign of Four. Spencer Blackett.
Xinya Du and Claire Cardie. 2020. Event extraction by answering (almost) natural questions. In Proceedings of the 2020 Conference Empirical Methods in Nat- ural Language Processing (EMNLP), pages 67 1-683, Online. Association for Computational Linguistics.
Yu Fei, Yifan Hou, Zeming Chen, and Antoine Bosselut. 2023. Mitigating label biases for in-context learning, In Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 14014-14031, Toronto, Canada. Association for Computational Linguisties.
Ethics Statement
The tasks and models in this work are publicly available. They could contain bias, and should be used with discretion.
Rusins Freivalds, Marek Karpinski, Carl H. Smith, and Rolf Wiehagen, 2002. Learning by the process of elimination. Inf. Comput., 176:37-50.
Ari Holtzman, Peter West, Vered Shwartz, Yejin Choi, and Luke Zettlemoyer. 2021. Surface form com- petition: Why the highest probability answer isn't always right. In Proceedings of the 2021 Conference on Empirical Methods Natural Language Process- ing, pages 1038-7051, Online and Punta Cana, Do- minican Republic. Association for Computational Linguistics.
References
Maciej Besta, Nils Blach, Ales Kubicek, Robert Ger- stenberger, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Michal Podstawski, Hubert Niewiadomski, Piotr Nyczyk, et al. 2023. Graph of thoughts: Solv- ing elaborate problems with large language models. arXiv preprint arXiv:2308.088. 09687.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2022. Large lan- guage models are zero-shot reasoners. In Advances in Neural Information Processing Systems.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah. Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz. Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
Xinxi Lyu, Sewon Min, Iz Beltagy. Luke Zettlemoyer, and Hannanch Hajishizzi. 2023. Z-ICL: Zero-shot in on-context learning with pseudo demonstrations. In ICLR 2023 Workshop Mathematical and Empiri- cal Understanding of Foundation Models.
Nikolay Malkin, Zhen Wang, and Nebojsa Jojic. 2022. Coherence boosting: When your pretrained language model is not paying enough attention. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 8214-82344 Dublin, Ireland. Association for Computational Linguistics.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Ka- plan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.0337..
Sewon Min, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022. Noisy channel language model prompting for few-shot text classification. In Proceedings of the 60th Annual Meeting of the As- sociation for Computational Linguistics (Volume 1:

---

Long Papers), pages 5316-5330, Dublin, Ireland. As- sociation for Computational Linguistics.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. CommonsenseQA: A ques- tion answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisticx: Humun Lunguage Tech- nologies, Volume (Long and Short Papers), pages 4149-4155 4158, Minneapolis, Minnesota. Association for Computational Linguistics.
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020, Adversarial NLI: A new benchmark for natural language under- standing. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 0885549001 Online. Association for Computa- tional Linguistics.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023, Llama: Open and effi- cient foundation language models. arXiv preprint 2xXiv:3302.13971
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems.
PeiFeng Wang, Aaron Chan, Filip llivskii Muhao Chen, and Xiang Ren. 2023a. PINTO: Faithful language reasoning using prompt-generater rationales. In The Eleventh International Conference on Learning Rep- resentations.
Joshua Robinson and David Wingate. 2023. Leveraging large language models for multiple choice question answering. In The Eleventh International Conference on Learning Representations.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery. and Denny Zhou. 2023b. consisteeny improves chain of thought reasoning in language models. In The Eleventh International Conference Learning Representations.
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. 2019. Social IQa: Com- monsense reasoning about social interactions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP-IJCNLL) pages 4463- 4473, Hong Kong, China. Association for Computa- tional Linguistics.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoe V Le, and Denny Zhou 2022. Chain of thought prompt- ing elicits reasoning in large language models. In Advances in Neural Information Processing Systems.
Minji Seo, YeonJoon Jung, Seungtack Choi, Seung- won Hwang, and Bei Liu. 2022. Debiasing event understanding for visual commonsense tasks. In Findings of the Association for Computational Lin- guistics: ACI. 2022, pages 782-787, Dublin, Ireland. Association for Computational Linguistics.
Shunyu Yao. Dian Yu, Jeffrey Zhao, Izhak Shafran. Thomas L. Griffiths, Yuan Cao. and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models. arXiv preprint arx.i:2305.1066
Xi Yc, Qiaochu Chen, Isil Dillig, and Greg Durrett, 2023, Satisfiability-adde language models using declara- tive prompting. arXiv preprint wrXiv:2305.006555
KaShun Shum, Shizhe Diao, and Tong Zhang. 2023. Automatic prompt augmentation and selection with of- from labeled data. arXiv preprint arXiv:2302. 12822.
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improv- ing few-shot performance of language models. In In- ternational Conference on Machine Learning, pages 12697 -1706. PMLR
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. 2023. Beyond the imitation game: Quantifying and extrapolating the capabili- ties of language models. Transactions on Machine Learning Research.
A Task Information and Pre-Processing
All tasks we use are expressed in English. ANLI is originally a classification task, and we convert it to multiple choice by mapping labels to options, i.e., "0" "entailment". "1" to "neutral", and "2" to "contradiction". For BIG-bench tasks, we remove instances whose number of options deviate from others. For ANLI and CC, we aggregate instances from all subtasks; For LD, we use the subtask with
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Se- bastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery. Quoc Lc. Ed Chi, Denny Zhou, and Jason Wei. 2023. Challenging BIG-bench tasks and whether chain-of-thought can solve them. In Findings of the Association for Computational Lin- guistics: ACL 2023, pages 13003-13051, Toronto, Canada, Association for Computational Linguistics.

---

5 options: For SS, we use the multiple choice sub- task, and treat options with scores of 0.5 wrong. We provide other details in Table The first three tasks can be accessed on Hugging Face." 9 The last thirteen tasks can be accessed on BIG-bench." 10
Task
Method
LD LD
LD (5)
LD (7)
LM
57.03.9
48.82.7
47.22.7
AVG
55.82.6
45.84.5
47.64.6
Calibration
45.03.1
39.24.0
38.85.1
Channel
40.03.8
20.83.6
18.22.
Task
Options
Train
Dev
Test
MCP
53.45.3
39.83.7
45.27.3
ANLI
162865
3200
3200
POE
70.21.2
56.03.8
53.05.5
CQA
9741
1221
N/A
SIQA
33410
1954
N/A
Table 5: Accuracy scores three subtasks of logical de- duction, which have different numbers of options. Best scores are bold, and second-bes scores are underlined. POE applies tasks of varying difficulties.
CAT
N/A
N/A
44
CC
N/A
N/A
103
CLD
N/A
N/A
60
DQA
N/A
N/A
255
EIE
N/A
N/A
68
B.2 Case Study
IMT
N/A
N/A
53
In Figure 4, present case study to show why POE works. We compare PoE with MCP one instance from CC. The question invents the word "wajey" with a definition, and forms a surprising and uncommon conceptual combination between "wajey" and "grape". The correct option is A.
IOM
N/A
N/A
47
LA
N/A
N/A
32
LD
N/A
N/A
500
RACO
18
N/A
N/A
2000
RS
N/A
N/A
49
SIT
N/A
N/A
990
SS
N/A
N/A
121
Question:
The word 'wajey' is an adjective for things that glow.
Table 4: Task Information. # Options means the number of options for the task.
Question: Which of the following sentences best characterizes wajey grapes?
Options:
A. W Wajey grapes are easy to pick at night.
B. W ajey grapes are the size of walnuts.
B Additional Analysis
C. W Wajey grapes have subtle hints of M Muscat.
D. Wajey grapes are full of worms.
B.1 Different Number of Options
Options
MCP Scores
PoE Scores
Since our method eliminates wrong options in the first step, we assume the number of options affects performance. Consequently, three subtasks of logical deduction (LD (3), LD (5), LD (7)), which have 3, 5, and options respec- tively. The other experiment settings are consistent with the main experiment (Section 3).
A
1.1426
0.6382 a
> [MASK]
2.9434
Inf
run experiments
C
0.8018 B
1.2979
D [MASK]
4.6562
Inf
Figure 4: A case study comparing PoE and MCP. The scores are negative log-likelihod (lower means better). MCP eliminates options B and D, but gets distracted by option C. In contrast, PoE eliminates options B and D in the first step, and chooses the correct option A in the second step when B and D are masked.
We present the result in Table 5. We find most methods (including PoE) perform worse the number of options increases. This conforms to intu- ition, because more options require more reasoning steps, and the questions get harder. Nevertheless, PoE is the best on all subtasks. This means al- though PoE does not counterintuitively perform better on harder questions, it works well on ques- tions of different difficulties, and thus applies to variety of multiple choice reasoning tasks.
We find MCP unable to solve this task, and chooses C. Taking a closer look at MCP scores (lower is better) for each option, find that MCP assigns high scores options B and D. two seem- ingly wrong options. This verifies our hypothe- sis that some wrong options' scores deviate from others, and can thus be climinated. Nevertheless, MCP distracted by option C, and we assume this
"ht hhtugs://hugginngfac.. phtttps://gitaee com/ con/google/Biin Baroogle/BBGGaae bench

---

is caused by the correlation between "grape" and "Muscat", the latter of which kind of grape. This instance shows although MCP can eliminate some wrong options, struggles to simultaneously choose the right option.
For POE, since it uses MCP in the first step, it masks options B and D. We examine the final PoE scores for each option, and find that it correctly chooses option A. In addition, lower than its MCP score, and
find the PoE
score for option conversely for option C. This means our method is not distracted by option This case study verifies our assumption that eliminating wrong options and choosing the best option are two types of reasoning that can be disentangled into two steps, and that such disentanglement is beneficial.
C Input and Output Samples for All Methods
We do not optimize prompts, because our goal is to evaluate PoE with a fixed prompt format. There- fore, we follow Holtzman et al. (2021) and Robin- and Wingate (2023) to construct task-agnostic prompts, and use them to wrap questions. We use one instance from SIQA. The question x is "Kendall was searching for ring with their eyes closed. They hit something. Why did Kendall do this?". The options Y are "kendall who has search- ing his ring" (y1). "kendall who has wanted to close their eyes" (y2), and "find the rings" (ys).
We present input and output samples for all methods for this instance in Table 6. Cali- bration computes two scores (log P(u[=r) and (log P(y text)), so we present samples for each them. PoE uses different prompts for each step, so we present them separately.

---

Method
Score
Input the answer is:
Output
LM AVG
log ( (x)
y1
1/len(ys) * log (()())
æ the answer is:
//1
log ( (r)
x the answer is:
51
Calibration
log (3) (txt)
the answer is:
Channel
log P(x y1)
/1
the answer is:
Question: x
A. V1
MCP
log P(yl,,
B. y2
C. yB
Answer:
Question:
A. y1
PoÉ: Elimination
log P( Y)
B. 92
C. y3
Answer:
Select the most suitable option
to answer the question. Ignore
[MASK] options.
POÉ: Prediction
log P(yil=mast Y Y w
B
Question:
A. [MASK]
B. 92
C. y3
Answer:
Table 6: Input and output samples for all methods, using one instance from SIQA.


Reviews:
[{"review_id": "kMN9C8oEML", "reviewer": "Reviewer_utpz", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for constructive feedback. We are encouraged to find that the reviewer finds our work “well motivated and reasonable”, and appreciates our experiments and reproducibility (4).\n\n# Reasons to Reject:\n\n> 1. ChatGPT (which model, gpt-3.5-turbo?) experiment is too simple, just two datasets, and on one of them [CQA] it doesn't really work well. I feel ChatGPT should be (obviously) more important than FLAN-T5-XL?\n\nThe model we used is gpt-3.5-turbo. \n\n**”ChatGPT experiment is simple”:** We agree, but we think it is justifiable:\n- We don’t intend to thoroughly compare ChatGPT and FLAN-T5 on all 8 tasks, but to highlight that our method (PoE) works on both open-sourced models like FLAN-T5 and LLMs like ChatGPT.\n- Implementing PoE for ChatGPT is non-trivial, because whereas PoE uses logits to mask options, ChatGPT does not provide logits.  \n- Our budget is limited.\n\n\n**Why ChatGPT does not work well on CQA:** Possibly the prompt, as we copy the ones designed for FLAN-T5. In the following experiment, we modify the prompt, and find PoE outperforms MCP (the best baseline) on CQA (2%). \n\n**Additional Experiments:** Using gpt-3.5-turbo, we compared PoE and MCP on all 8 tasks used in our paper, as well as 4 new logical reasoning tasks (our method performs well on logical reasoning tasks, which is discussed in the response to the second reason to reject). \n\n**Results:** For the 8 original tasks (shown below, numbers are accuracy scores), PoE outperforms MCP on 5 tasks, which is consistent with the performance of FLAN-T5. This means PoE applies to LLMs like ChatGPT.\n\n| Task   |   MCP |   PoE |   PoE - MCP |\n|:-------|------:|------:|------------:|\n| SIT    |   2   |  19.4 |        17.4 |\n| ANLI   |  40.2 |  44.6 |         4.4 |\n| SS     |  79.4 |  82.8 |         3.4 |\n| CQA    |  75   |  77   |         2   |\n| LD     |  37.6 |  38   |         0.4 |\n| SIQA   |  72.6 |  71.2 |        -1.4 |\n| DQA    |  48.8 |  42.6 |        -6.2 |\n| CC     |  79.6 |  66.8 |       -12.8 |\n\nFurthermore, PoE outperforms MCP by large margins on 4 logical reasoning tasks (shown below), which further corroborates our assumption that PoE performs well on logical reasoning tasks.\n\n| Task   |   MCP |   PoE |   PoE - MCP |\n|:-------|------:|------:|------------:|\n| RACO   |   4.8 |  50.4 |        45.6 |\n| IMT    |  34   |  49.8 |        15.8 |\n| LA     |  50   |  55   |         5   |\n| CLD    |  92.4 |  91.4 |        -1   |\n\n\nWe will continue to work on PoE for LLMs in the next version. \n\n**”ChatGPT is more important than FLAN-T5-XL”:** We agree that ChatGPT is one of the most important and powerful LLMs. Nevertheless, it is close-sourced and updated frequently, which is not ideal for interpretability and reproducibility. Also, our main focus is open-sourced LMs, which are prevalent and widely used by both academia and industry. We think we have provided sufficient evidence to support our claim, i.e., PoE works on open-sourced models like FLAN-T5.\n\n> 2. FLAN-T5-XL across many datasets, seems some show bigger improvements, while others show very little improvement, any intuition?\n\n**Intuition:** PoE works well on tasks that mainly require logical reasoning, which is  verified on 8 new tasks (This is a common concern shared by reviewer dmiM. Please refer to that response (first reason to reject) for more details).\n\n**Results:** As shown below, PoE largely outperforms MCP on the four logical reasoning tasks (first four rows); PoE underperforms MCP on the other 4 tasks, all of which require social or commonsense reasoning. One explanation is that logical reasoning is a general skill, whereas social or commonsense reasoning relies on specific knowledge, some of which may not be present in language models.\n\n\n| Task   |   MCP |   PoE |   PoE - MCP |\n|:-------|------:|------:|------------:|\n| LA     |  50   |  68.8 |        18.8 |\n| IMT    |  34   |  47.2 |        13.2 |\n| CLD    |  67.2 |  75.9 |         8.7 |\n| RACO   |  53.8 |  60.6 |         6.8 |\n| CAI    |  84.1 |  81.8 |        -2.3 |\n| EIE    |  25   |  19.1 |        -5.9 |\n| RS     |  55.1 |  49   |        -6.1 |\n| IOM    |  56.2 |  50   |        -6.2 |\n\n\n> 3. Few-shot performance gain seems small overall, not sure if added computation step is worthy.\n\nWe agree that few-shot performance gain is small. Still, we think the few-shot experiment is worthy and necessary, because it shows our method generalizes beyond zero-shot. We will continue to improve few-shot performance in the next version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "the technique is well motivated and reasonable, and multiple benchmarks and design choices are considered for FLAN-T5-XL.", "reasons_to_reject": "- ChatGPT (which model, gpt-3.5-turbo?) experiment is too simple, just two datasets, and on one of them it doesn't really work well. I feel ChatGPT should be (obviously) more important than FLAN-T5-XL?\n\n- FLAN-T5-XL across many datasets, seems some show bigger improvements, while others show very little improvement, any intuition?\n\n- few-shot performance gain seems small overall, not sure if added computation step is worthy.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper proposes a simple trick for multi-choice QA: eliminate some choices first, then re-answer the question with eliminated choices masked. It is shown with both ChatGPT and FLAN-T5-XL on multiple datasets to work (somehow).", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "57QMIMpP1t", "reviewer": "Reviewer_Q4aG", "summary": "", "questions": "1. For a question with N choices, instead of simply decompose the solution process into two steps, what if you decompose it into N-1 steps (first remove most unlikely solution, then remove the 2nd unlikely solution etc.)?\n2. How does the option elimination process compare with directly picking the top k scores as candidates (i.e. si >= top_k(s1, ..., si) in Eq. 5)?\n3. It remains unclear why the approach needs to replace eliminated option with the token \"[MASK]\". Does it matter if we use a different token to indicate invalid options?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for giving feedback and suggesting further experiments. We are encouraged that the reviewer finds our motivation sensible and our work highly reproducible (5). \n\n# Reasons to Reject:\n\n> 1. The scope of the proposed approach [PoE] is very limited (only fits multiple-choice questions).\n\nWe agree that PoE is tailored for multiple-choice tasks, but we think doing so is justifiable:\n- These tasks span many NLP tasks like reading comprehension, natural language inference, text classification, and more. We think the scope is appropriate for a [short paper](https://2023.emnlp.org/calls/main_conference_papers/#short-papers:~:text=taken%20into%20account.-,Short%20Papers,-Short%20paper%20submissions), not to mention other accepted long papers dealing with similar or narrower scopes ([Surface Form Competition](https://aclanthology.org/2021.emnlp-main.564.pdf), [Calibrate Before Use](http://proceedings.mlr.press/v139/zhao21c/zhao21c.pdf)).\n- Our method could extend to generation tasks (similar to beam search, i.e., keeping likely generation and removing unlikely ones), which we would like to explore in the future.\n- We have provided evidence (in the original paper and additional experiments) for our main claim, i.e., PoE works on multiple-choice questions.\n\n>2. ... unclear how effective this approach [PoE] is, compared with the straightforward multiple choice prompting [MCP].\n\nWe agree that clarity is essential. In fact, the comparison is a major focus of our paper, and we have presented it in Table 1 (last 2 columns), section 4 (line 184-199), and appendix B.2 (line 536-565). We find PoE outperforms MCP on 5 out of 8 tasks, and achieves comparable performance on the other 3. We would revise the paper to make the comparison more explicit.\n\nP.S. In response to Review dmiM (first reason to reject), we compare PoE to MCP on 8 new tasks.\n\n# Questions For The Authors:\n\n> 1. For a question with N choices … what if you decompose it into N-1 steps (first remove the most unlikely solution, then remove the 2nd unlikely solution etc.)?\n\n**Conclusion**: The proposed method (Iter PoE) achieves comparable performance to our method (PoE), but costs more time and compute. We appreciate this suggestion, but it seems PoE is a better choice.\n\n**Experiments**: We implement Iter PoE to compare with PoE on 8 tasks. The results are shown below, where numbers are accuracy scores (%).\n\n| Task   |   PoE |   Iter PoE |   PoE - Iter PoE |\n|:-------|------:|-----------:|-----------------:|\n| ANLI   |  55.6 |       55.6 |              0   |\n| CC     |  76   |       74.2 |              1.8 |\n| CQA    |  89.2 |       88.4 |              0.8 |\n| DQA    |  67.8 |       67.8 |              0   |\n| LD     |  56   |       57.2 |             -1.2 |\n| SIQA   |  82   |       82   |              0   |\n| SS     |  75.6 |       76.6 |             -1   |\n| SIT    |  23.4 |       23.8 |             -0.4 |\n\n\n> 2. How does the option elimination process compare with directly picking the top k scores as candidates (i.e. si >= top_k(s1, ..., si) in Eq. 5)?\n\n**Conclusion**: The proposed masking strategy (“Min K”) slightly underperforms the ones we used (“Below Average” and “Lowest”). “Min K” introduces a hyperparameter (k) that is task-dependent and requires tuning, whereas ours don’t. We appreciate this suggestion, but it seems our methods are better.\n\n**Experiments**:  We implement “Min K” and compare it to “Lowest”, and show the best one on each task in the following table (Min K (2) means the best value of k is 2). We find “Min K” is the best on 3 out of 8 tasks. In addition, k is task-dependent: the best value of k varies, and the effect of increasing k is uncertain (accuracy could increase or decrease).\n\n| Task | Best Mask | Accuracy |\n|:-------|:------------|-----------:|\n| ANLI | Min K (2) | 57.8 |\n| CC | Lowest | 76 |\n| CQA | Lowest | 89.5 |\n| DQA | Lowest | 67.8 |\n| LD | Lowest | 56 |\n| SIQA | Lowest | 81.7 |\n| SS | Min K (2) | 76.6 |\n| SIT | Min K (3) | 27.4 |\n\n\n> 3. Does it matter if we use a different token [other than “[MASK]”] to indicate invalid options?\n\n**Conclusion**: Yes, but there are many good replacements, and their performances are similar. Tuning this hyperparameter is time-consuming, and may not be critical to our method. Still, we appreciate this suggestion.\n\n**Experiments**: Following [Calibrate Before Use](http://proceedings.mlr.press/v139/zhao21c/zhao21c.pdf), we compare 5 mask tokens (shown below, where “empty” is an empty string, and “d…s” is a random string). Their performances are similar, and the best mask token is task-dependent. \n\n| Task | Best Mask Token | Accuracy |\n|:-------|:---------------------|-----------:|\n| ANLI | \"empty\" | 57.2 |\n| CC | \"[N/A]\" | 79.2 |\n| CQA | \"dasjhasjkdhjskdhds\" | 89.6 |\n| DQA | \"[N/A]\" | 68.2 |\n| LD | \"[N/A]\" | 56.8 |\n| SIQA | \"[mask]\" | 82.6 |\n| SS | \"empty\" | 78.6 |\n| SIT | \"[MASK]\" | 23.4 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The motivation of the paper, namely dividing the multiple-choice solution process into two steps, makes a lot of sense.", "reasons_to_reject": "The scope of the proposed approach is very limited (only fits multiple-choice questions). What is more, it remains unclear how effective this approach is, compared with the straightforward multiple choice prompting.", "questions_to_authors": "1. For a question with N choices, instead of simply decompose the solution process into two steps, what if you decompose it into N-1 steps (first remove most unlikely solution, then remove the 2nd unlikely solution etc.)?\n2. How does the option elimination process compare with directly picking the top k scores as candidates (i.e. si >= top_k(s1, ..., si) in Eq. 5)?\n3. It remains unclear why the approach needs to replace eliminated option with the token \"[MASK]\". Does it matter if we use a different token to indicate invalid options?", "paper_topic_and_main_contributions": "This paper proposes an approach to improve LLM's accuracy for multiple choice questions. The approach divides the LLM reasoning process into two steps, first eliminates unlikely choices, then choose the best option among the remaining items.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "VUu0JTUExK", "reviewer": "Reviewer_dmiM", "summary": "", "questions": "1. Why do you mask the incorrect answers rather than simply removing them from the prompt?\n2. Do you have any idea why the top-choice ordering changes in some cases after elimination?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are truly grateful for the constructive feedback from this review. We are excited to find that the reviewer appreciates our method, experiments and analysis, and the clarity and reproducibility (5) of the paper. We are delighted to find our paper “fairly scoped for a short paper”.\n\n# Reasons to Reject:\n\n> 1. *When* PoE [our method] can be expected to dramatically outperform MCP [best baseline] (i.e., what makes LD and CC tasks unique [our method is really good on these two tasks])\n\n**When:** PoE dramatically outperforms MCP on tasks that mainly require logical reasoning, but not on tasks about social or commonsense reasoning.\n\n**Observation**: Both tasks (LD and CC) are based on logical reasoning. Among the 8 tasks we used in our experiment, there is one more logical reasoning task: SIT, on which MCP is comparable to PoE. Nevertheless, SIT relies on visual reasoning, which may weaken PoE.\n\n**Hypothesis**: PoE is better than MCP on tasks that mainly require logical reasoning. In addition, PoE is not consistently good at social or commonsense reasoning. One explanation is that logical reasoning is a general skill, whereas social or commonsense reasoning relies on specific knowledge, some of which may not be present in models.\n\n**Experiment**: We compare PoE and MCP on 8 new tasks from BIG-Bench: 4 of them are mainly based on logical reasoning (logical_args (LA), identify_math_theorems (IMT), code_line_description (CLD), reasoning_about_coloered_objects (RACO)); the other 4 are control tasks that rely more on social or commonsense reasoning (crass_ai (CAI), evaluating_information_essentiality (EIE), riddle_sense (RS), identify_odd_metaphor (IOM)).\n\n\n**Results**: As shown below (numbers are accuracy scores), PoE outperforms MCP by large margins on all 4 logical reasoning tasks (LA, IMT, CLD, RACO), especially LA (18.8%), which is similar to improvements on LD (13.8%) and CC (12.0%). The performance on LA is convincing, because LA is made up of GRE-style logical questions, and a common strategy used by human test takers for such questions is PoE.\nOn all 4 control tasks (CAI, EIE, RS, IOM), PoE underperforms MCP.\n\n| Task   |   MCP |   PoE |   PoE - MCP |\n|:-------|------:|------:|------------:|\n| LA     |  50   |  68.8 |        18.8 |\n| IMT    |  34   |  47.2 |        13.2 |\n| CLD    |  67.2 |  75.9 |         8.7 |\n| RACO   |  53.8 |  60.6 |         6.8 |\n| CAI    |  84.1 |  81.8 |        -2.3 |\n| EIE    |  25   |  19.1 |        -5.9 |\n| RS     |  55.1 |  49   |        -6.1 |\n| IOM    |  56.2 |  50   |        -6.2 |\n\n\nIn addition, we report results on the same 4 logical reasoning tasks using gpt-3.5-turbo (shown below). We find PoE consistently outperforms MCP on logical reasoning tasks. This result also suggests the benefits of PoE generalize to different models.\n| Task   |   MCP |   PoE |   PoE - MCP |\n|:-------|------:|------:|----------:|\n| RACO   |   4.8 |  50.4 |      45.6 |\n| IMT    |  34   |  49.8 |      15.8 |\n| LA     |  50   |  55   |       5   |\n| CLD    |  92.4 |  91.4 |      -1   |\n\n\n\n**Conclusion**: Although PoE works on different reasoning tasks (Table 1), its performance is most significant and consistent on logical reasoning tasks.  \n\n> 2. [Compared to MCP,] Why does the scoring order [of PoE] change after masking [line 536-565]?\n\nBecause MCP is distracted by some wrong options. That may prevent it from selecting the right option, a problem addressed in PoE.\n\nMCP does two things (eliminate wrong answers, and choose from the remaining) in one step, and PoE does them separately in two steps. Inspired by human behavior, we assume disentangling these two steps is better than not. This assumption is supported by recent works on multi-step prompting, like [CoT](https://arxiv.org/abs/2205.11916), [ToT](https://arxiv.org/pdf/2305.10601.pdf), [GoT](https://arxiv.org/pdf/2308.09687.pdf). The change of scoring order verifies this assumption.\n\n\n> 3. … [few-shot] results [are missing] for LD or CC [to show if PoE still largely outperforms MCP].\n\n**Conclusion**: PoE seems not well-adapted to few-shot settings, but we do find PoE works well on logical reasoning tasks, compared to MCP.\n\n**Results**: As shown below, PoE is still better than MCP in the three-shot setting, but their performance gap is smaller (CC: 15.6% to 0.2%; LD: 16.2% to 1.6%). PoE does not perform well in 3-shot, and possible reasons include: 1) We do not optimize prompt or demonstrations for few-shot PoE; 2) The instructions we use in zero-shot PoE are already powerful, and few-shot demonstrations may introduce some noise.\n\nWe also notice both MCP and PoE perform worse on LD in 3-shot, which we will continue to investigate.\n\n\n| Task   |   N Shot |   MCP |   PoE |   PoE - MCP |\n|:-------|---------:|------:|------:|------------:|\n| CC     |        0 |  60.4 |  76   |        15.6 |\n| CC     |        3 |  76.8 |  77   |         0.2 |\n| LD     |        0 |  39.8 |  56   |        16.2 |\n| LD     |        3 |  17.6 |  19.2 |         1.6 |\n\n\n# Questions For The Authors:\n\n> 1. Why do you mask the incorrect answers rather than simply removing them from the prompt?\n\nWe do that for two reasons: \n- Our method is simpler to implement. Removing incorrect options would change the number of options, and the remaining options need to be relabeled (e.g., “ADE” to “ABC”). \n- We implement this method and compare it to our original method. We find their performance very close (<1%).\n\n> 2. Do you have any idea why the top-choice ordering changes in some cases after elimination?\n\nYes, we do. Please check our response to the second reason to reject above.\n\n# Typos Grammar Style And Presentation Improvements:\n\n> 1. 133: What does \" $T$ to 1\" mean?\n\nThat means “we use template $T$ for three things, and the first thing is …” \n\nWe will clarify this in our next version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper is clearly written and shows promising results for their method. Their contribution seems fairly scoped for a short paper. I appreciate their thoroughness in considering the space of design decisions for their method.", "reasons_to_reject": "What would make this paper interesting to me would be to have some idea of *when* PoE can be expected to dramatically outperform MCP (i.e., what makes LD and CC tasks unique). Why does the scoring order change after masking? Without answering, analyzing, or addressing this question the contribution seems uninformative. For instance, in Holtzman et al. (2021) the authors provide a hypothesis for a mechanism for exactly how their method works (surface form competition).\n\nA more easily resolved criticism I have is that, since MCP is the closest method to the paper's method, the authors should show a clear advantage to using PoE over MCP in some cases. Therefore, in Table 2 I would be more interested in seeing the results for LD or CC to see if the large gains from using PoE for these tasks persists in the few-shot setting. This should be easily addressable with an added experiment.", "questions_to_authors": "1. Why do you mask the incorrect answers rather than simply removing them from the prompt?\n2. Do you have any idea why the top-choice ordering changes in some cases after elimination?", "paper_topic_and_main_contributions": "This paper contributes a strategy to boost multiple-choice QA accuracy for language models by first eliminating low-score answers then rescoring the top-scoring answers. They also provide an analysis of different design choices for this procedure.\n\nThe authors hypothesize that the MCQA requires two skills: eliminating choices and choosing the correct choice. Their results give evidence for the claim by showing PoE usually performs comparably with MCP for most tasks, but for some tasks beats MCP by a large margin.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "133: What does \"$T$ to 1\" mean?"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors proposed a two-step scoring method for multiple choice reasoning tasks, based on the process of elimination.
The 2 out of 3 reviewers are selected strong soundness, and reproducibility is enough.
Decision: Accept-Main