Answering Questions by Meta-Reasoning over Multiple Chains of Thought
Authors: Ori Yoran
Tomer Wolfson
Ben Bogin
Uri Katz
Daniel Deutch
Jonathan Berant
Venue: EMNLP 2023 Main (2023)
Keywords: Question Answering
Multi-hop Question Answering
Reasoning
Few-shot
Chain of Thought
TL;DR: This paper introduces Multi-Chain Reasoning (MCR), an approach which prompts large language models to meta-reason over multiple chains of thought rather than aggregate their answers; MCR outperforms strong baselines on 7 multi-hop QA datasets.

Abstract: Modern systems for multi-hop question answering (QA) typically break questions into a sequence of reasoning steps, termed chain-of-thought (CoT), before arriving at a final answer. Often, multiple chains are sampled and aggregated through a voting mechanism over the final answers, but the intermediate steps themselves are discarded. While such approaches improve performance, they do not consider the relations between intermediate steps across chains and do not provide a unified explanation for the predicted answer. We introduce Multi-Chain Reasoning (MCR), an approach which prompts large language models to meta-reason over multiple chains of thought, rather than aggregate their answers. MCR examines different reasoning chains, mixes information between them and selects the most relevant facts in generating an explanation and predicting the answer. MCR outperforms strong baselines on 7 multi-hop QA datasets. Moreover, our analysis reveals that MCR explanations exhibit high quality, enabling humans to verify its answers.

Answering Questions by Meta-Reasoning over Multiple Chains of Thought
Ori Yoran*l
Tomer Wolfson +1,2
Ben Bogin'
Uri Katz²
Daniel Deutch Jonathan Berant'
Tel Aviv University or i. yor. yoranecs. tau.ac........ ac. i1
Allen Institute for AI
BBar Ilan University
tomerwolemall. tau. ac
Abstract
Did Brad Peyton need to know about seimmloggy?
Modern systems for multi-hop question answer- ing (QA) typically break questions into se- quence of reasoning steps, termed chain-of- thought (CoT), before arriving at final answer. Often, multiple chains are sampled and aggre- gated through a voting mechanism over the fi- nal answers, but the intermediate steps them- selves are discarded. While such approaches improve performance, they do not consider the relations between intermediate steps across chains and do not provide a unified explanation for the predicted answer. We introduce Multi- Chain Reasoning (MCR), approach which prompts large language models to meta-reason over multiple chains of thought, rather than ag- gregate their answers. MCR examines different reasoning chains, mixes information between them and selects the most relevant facts in gen- erating an explanation and predicting the an- swer. MCR outperforms strong baselines on multi-hop QA datasets. Moreover, our analy- sis reveals that MCR explanations exhibit high quality, enabling humans verify its answers.
Sampling Multiple Reasoning Chains
Reasoning chain #1
O: Whath Brad Paytarss eecopatan? Ac Bred Paskor is destor
O: Whalk sebrrskoyy?
Q Do fim diecbis need to know www.res/ttuiiieeee. torees
& No. fim dirccurs do not reed to knos diquakss
Ressoning chain #2
Q: Paylon
fndd Canation dinectrr
Answer
Q: Andraas about
Brad
massive qualo coused San Andreas Fasl..
Reanoning
Q: Whetis Brudd ssterss occupaior?
producer
@ What film cirecters
know?
Anzeer # No.
& Fllm diector have know of ings.
G: is sesmo bgy ane the thnng that Brnc Feyton has
A No.
Multi-Chain Reasoning (MCR)
Self-Consistency TBR
Ameovrt ayoo in tim diverio: and Arcnsas (Iin dirccadd Brod Paylar, abzuta massins aartiquaae cassed by the San Antrass Faut. Se final ansinee Yeu.
Stinsr No. Actarr #t: Yes.
Majerity answer: No.
An example from STRATEGYQA, show- ing the output of Multi-Chain Reasoning versus Self- Consistency. MCR uses reasoning chains its context for QA. SC solely relies on the chains' answers.
Figure I:
1 Introduction
In chain-of-thought (CoT) prompting, large lan-
guage model (Brown et al., 2020; Chowdhery et al., 2022; Kadavath et al., 2022: Touvron et al., 2023) is prompted to generate its answer following a step- by-step explanation (Wei et al., 2022; Nye et al., 2022). CoT prompting has been shown to dramat- ically improve performance on reasoning-hean heavy tasks (Kojima al., 2022; Zhou et al., 2022). Fur- thermore, Wang et al. (2023) showed that sampling multiple chains of thought and returning their ma- jority output further improves accuracy, a method which they term self-consistency (SC).
which case no significant majority will be formed. Second, focusing exclusively on the final output discards relevant information that is present in the intermediate reasoning steps. Consider answering the question "Did Brad Peyton need to know about seismology? " (Fig. 1). Reasoning chain #1 leads to an incorrect answer ("No") "), but its steps provide useful information. For example, the intermedi- ate question, and following answer, on "What is seismology?" ?" constitute an important fact that is absent from the other two chains. Last, using SC jointly with chain-of-thought prompting reduces in- terpretability, as there is no single reasoning chain that can be considered as an explanation.
While SC leads to performance gains, it also has several shortcomings. First, when the space of pos- sible outputs is large (Kalyan et al., 2021), each reasoning chain may lead to different output, in
In this work, we propose Multi-Chain Reasoning (MCR), where we prompt a large language model
"Both authors contributed cqually to this work.

---

How many ants would fit into The Shard?
Reasoning c chain
Reasoning chain #2
Reasoning chain #3
Decomposition
What height The Shard?
Q1: shard?
How many the world?
The Shard comprizes es floor office complex, accupied by...
E1: 1.016 feet: S Standing 309.6 metres
Al:Ant This study alse puts
Iterativly generate the reasoning chain interleaving retrieved evidence
(1.016 feet) high, The Shard
conservative estimate the ants at...
E1: The Shard comprises 26 26.f.u....
sanding
(1,016
This study also puts
E2:
feeti high.
The Shard comprises
D1:
height Shard?
Ants range in from
91: How many ants are the world?
Al height
26-
Q1: How high ss the shard?
41 There are around 20 quadrilion
Retrieval
Roors, sanding 309.6 metres tall
A1: The shard 1016 feet in height
ants the world.
Q2 What the volume of an and?
Q2: How long is the avarage ant?
92: What the volume The Shard?
A2: The yolume of an ant 20-20
A2: The longth of average ant
A2 The volume The Shard is 9000
Ciblc meters.
0. 75mm.
cubic
So the answer by +5/22 23)-2223)=2+222
the ansmeris
the
1048 20016.75--1.3555 354c
55900000 22222e+32.
What is the height The Sharo? The height The Shard is 26-noorr, standing 309.6 metres tall. What is the volume siue of an and The volume of an ant Ze 23 cubic meters. How high the shars? The shard 1016 foct height.
3. Meta-Reason
How long the average and The length of an average ant 0.75mm.
Generate the final answer based on the content of sampled reasoning chains
How volume
word?? volume The quadrillon ants the worls. 90000
Answer: The volume Shard 90000 meters. volume ant 2e-23 cublic meters. So the answer 4.5e+22.
An overview of MCR, given question from the FERMI dataset. Steps 1-2 generate multiple reasoning chains by conditioning the generation of intermediate questions and answers retrieved evidence sentences. In step 3, the meta-reasoner generates the final answer, given multiple reasoning chains from the previous steps.
Figure 2:
(LLM) to meta-reason across multiple reasoning chains and produce a final answer, alongside an explanation. Unlike prior work, sampled reason- ing chains are used not for their predictions (as in SC) but means to collect pieces of evidence from multiple chains. Fig. illustrates MCR com- pared SC. While both methods rely on sampling multiple reasoning chains, SC returns the major- ity answer, "No" (grey box, bottom right). By contrast, MCR concatenates the intermediate steps from each chain (blue boxes, top left) into a unified context, which is passed, along with the original question, to meta-reasoner model. The meta- reasoner is separate LLM, prompted to meta- reason on multiple reasoning chains and produce final answer along with an explanation (pink box, bottom left). By reasoning on multiple reason- ing chains, MCR is able to mitigate the aforemen- tioned drawbacks - it combines facts from multiple chains to produce the correct final answer, with an explanation of the answer's validity.
meta-rcasoner. Fig. highlights the ability of the meta-reasoner to combine facts from different rea- soning chains (intermediate answers in pink). The output explanation combines facts from each of the three chains: (1) "Seismology is the study of earthquakes"; ; (2) "San Andreas is film.."; (3) "Brad Peyton is a film director, writer..". SC (in grey) errs due to only using the answers, while the meta-reasoner reads entire reasoning chains, and is able to correctly answer the question.
We evaluate MCR on a wide range of challeng- ing multi-hop question answering (QA) datasets, in an open-domain setting. The datasets can be categorized into two types of tasks: implicit rea- soning tasks, where reasoning steps are implicit given the question text and need to be inferred us- ing a strategy (Tafjord et al., 2019; Geva al., 2021; Kalyan et al., 2021): explicit reasoning tasks. where a single reasoning strategy exists and can be directly inferred given the language of the ques- tion (Yang et al., 2018; Welbl et al., 2018; Press et al., 2022; Aly et al., 2021). As our baselines, we compare MCR to SC, well as to variants of Self- Ask (Press et al., 2022) and CoT augmented with retrieval, following Trivedi et al. (2022a). Our re- sults show MCR consistently outperforms all other baselines, in particular, beating SC by up to 5.7%, while using the same reasoning chains ($4).
MCR has three main components ($3). To gen- crate reasoning chains we use two components, decomposition model and retriever which jointly generate the chain (Fig. 2), similar to prior work (Press al., 2022; Trivedi et al., 2022a). These chains are then concatenated into a unified multi- chain context which is fed to the aforementioned

---

We analyze the qualities of MCR in $5, by man- ually scoring its generated explanations and esti- mating their accuracy. Our analysis shows that MCR generates high quality explanations for over 82% of examples, while fewer than 3% are unhelp- ful. To conclude, our main contributions are:
Is it true that Colonel Walter Phelps served the United Army for more years?
Decomposition step M1
Decomposition step 12
Q1: Who Colonal Walter Pholps?
Febeuary 20, 1878) was an offeer...
El: Waker P 20 1832-
01: Who Cokirel Waher helps?
Al: Colonal Pholos
officer
Retrieval step
Union Amy throughotthe
We introduce the MCR method for meta- reasoning on multiple chains-of-thought.
Colonel Watter Phelps the United States Army?
helpss (Get 29, February 1878) offier Aimy trroughout American servin as commansing the Eastam gadad.
We show that MCR outperforms all baselines, including self-consistency. on all 7 multi-hop open-domain QA benchmarks.
Photos IWwater [Aleglance) Unted Stetes of America Unieni [Se vis/trrannnnnn United States Amy Union Armyl (Tears of service.. 1186 116611 "Rank, Calonel Bull Brigacier Gineralt!
Decomposition
We analyze MCR for its explanation quality and its multi-chain reasoning capabilities.
Photos: 1832- 20, 1878) Who Cakkee Waller Phelos?
EL: Wolter Photos
Fabnaary 1978)
E2: Waller Pholps WlWller Peepps
Our data and codebase are publicly available.
All Colonel Walter Phelps was an officer the Union Army throughout the Amarican Civil War.
01. Who & Colonel Walter Pholos??
01- CaloCll Wa
02: How long did Colonel Walter Pholp
serve the United States Army?
2 Background
A.B: Celanel Walter Phelpe served
United States Army yeurs.
Recently, there has been a surge of interest in answering multi-hop questions through few-shot prompting of LLMs (Wei et al., 2022: Nye et al., 2022; Yao et al., 2022). The majority of these works follow a common standard: First, given question, plan a step-by-step reasoning chain to derive the answer and solve all intermediate steps, aided by a retriever to minimize model hallucina- tion (Khot et al., 2023; Press et al., 2022; Yao et al., 2022; Lazaridou et al., 2023; Trivedi et al., 2022a; Khattab et al., 2022). Then, incorporate multiple reasoning chains with answers to derive the final answer (Wang et al., 2023; Li et al., 2022). In our work, we follow this template and focus on the lat- ter part. However, our meta-reasoning approach differs from prior work by reasoning on multiple reasoning chains. Namely, we use multiple chains to collect relevant evidence for question answering.
Figure 3: Interleaving decomposition and retrieval steps.
we use a prompted LLM to generate multiple rea- soning chains, ,(1), r(.),..., ((k) (steps 1-2). Each yr( is generated by interleaving generated intermedi- ate questions with retrieved contexts ($3.1). Our main contribution is step 3: We introduce second LLM that is prompted to meta-reason multiple reasoning chains, collecting evidence facts its explanation and generating the final answer ($33.)).
3.1 Generating Reasoning Chains
Given a question q. we generate its reasoning chain using: (1) a decomposition model, and (2) a re- triever component. Our reasoning chain generation process is largely based on prior work (Press et al.. 2022; Trivedi et al., 2022a), discussed in $2. Fig. 3 describes the interleaving of decomposition and retrieval. At each step, the decomposition model generates an intermediate question Zi based on the original question q and the previous reasoning steps. Then, the retriever uses qi to retrieve relevant evidence e, E C. We feed e, and qi to the decom- position model (along with the previous steps) to generate intermediate answer During answer generation, we prepend intermediate evidence sen- tences to the beginning of the chain rather than interleaving them, as it improves the accuracy for all baselines. For decomposition prompts, see $D.
3 Method
We present a method for answering questions by meta-reasoning on multiple reasoning chains. Our focus is on open-domain QA, where the input is a question q. and the evidence to answer it is found in one more sentences in corpus C. When an- swering q requires multiple reasoning steps, it can be expressed by reasoning chain, denoted by r. The reasoning chain is list of one or more interme- diate mnestion-evidencc-ane answer triples (gi, C; oz). Evidence e E C' is a sentence that is relevant to answering the intermediate question fi-
3.2 Reasoning over Reasoning Chains
The meta-reasoner module is the core contribution of MCR. Instead of sampling multiple chains for their predicted answers (Wang et al., 2023), we utilize them for context generation. This context
Fig. 2 describes our approach when answering "How many ants would fit into The Shard?". First,

---

is fed to a prompted LLM read the generated chains and reason over them to return the answer.
Given question and context, answer the question step- by-step. If you are unsure, answer Unknown.
In $3.1, we defined reasoning chain list of (qs, ei, ai) triples. We first sample multiple chains and use all of their intermediate question- answer pairs (qi, a;) as our multi-chain context (a variant using question-evidens
Context:
Who is Robert Broderip? Robert Broderip was an English organist and composer.
Where did Robert Broderip live all his life? Robert Broderip lived Bristol all his life. When did Robert Broderip live? Robert Broderip lived during the 19th century.
pairs (q:. 6j) is de- scribed in $B.4). Fig. 2 presents the multi-chain context of the three sampled chains (lower pink box). Next, the multi-chain context and the origi- nal question are input to the meta-reasoner. This model is an LLM, few-shot prompted for QA over a multi-chain context. Fig. 4 presents one exemplar from the meta- meta-reasoner prompt for the FEVEROUS dataset (full prompts in $D). We instruct the LLM to "answer the question step-by-ster multi-chain context, where each line describes a (qi, a:) pair from one of the sampled chains. Next, we append the question and step-by-step reason- ing chain followed by the final answer. This last chain serves as the explanation for solving the ques- tion. The meta-reasoner prompted with 6-10 exemplars, based on the dataset ($4.1).
Where did Robert Broderip live? Broderip lived Bristol, During what part of the nineteenth century did Robert Broderip write music? Robert Broderip wrote music during the latter part of the eighteenth century.
Question: Is true that Robert Broderip lived London all his life and wrote considcrable quantity of music during the earlier part of the nineteenth century?
Answer: Robert Braderip lived in Bristol all his life, not in London. So the answer is: No.
given its
Figure 4: An exemplar from the meta- meta-reasoner prompt.
dataset. We also evaluate on the official test sets of STRATEGY QA and FERMI, as they target im- plicit reasoning, have multiple valid strategies, and their test set evaluation cost is reasonable. For all datasets, we make sure that no evaluation questions appear in any of our prompts. Tab. has exam- ple questions from each dataset. Our multi-hop QA benchmarks can be categorized based on their required reasoning skills:
Providing the meta-reasoner with multiple chains allows it to combine and aggregate facts across chains. Moreover, the model needs to ex- tract the most relevant facts in the chains to serve as its explanation. This enables MCR to be both more accurate and more interpretable than past multi-chain approaches (as we analyze in 85).
Implicit Reasoning: Questions that entail im- plicit reasoning steps (Geva et al., 2021). The reasoning steps for solving it cannot be explic- itly derived from the language of the question and require commonsense or arithmetic reason- ing. Such questions may have multiple valid rea- soning chains. We evaluate on: STRATEGYQA (Geva et al., 2021), FERMI (Kalyan et al., 2021) and QUARTZ (Tafjord et al., 2019).
4 Experiments
We compare MCR to existing methods on 7 multi- hop QA benchmarks. These cover a wide range of reasoning skills, including commonsense, com- position, comparison and fact verification. MCR consistently outperforms existing approaches on all benchmarks, when experimenting with two differ- ent LLMs and retrievers. Our setting is described in $4.1 and we discuss our main results in $4.2.
Explicit Reasoning: Multi-hop questions where the reasoning steps are explicitly ex- pressed in the language of the question (compo- sition, comparison). These include HOTPOTQA (Yang et al., 2018), 2WIKIMQA (Welbl et al., 2018) and BAMBOOGLE (Press et al., 2022). We also evaluate FEVEROUS (Aly et al., 2021), a fact verification dataset where claims require verifying multiple facts, and evidence may be either in sentences, tables or both.
4.1 Experimental Setting
4.1.1 Datasets
As our focus is on multi-hop questions (in an open- domain setting), all datasets require multiple rea- soning steps. Following prior work (Khattab et al.. 2022; Trivedi et al., 2022a) and to limit the cost of model API calls, we evaluate on 500-1000 ran- dom examples from the development set of each
For evaluation, we use F1 to compare predicted and gold answers for all explicit reasoning datasets
We use the entire development set for QUARTZ and BAM- BOOGLE, since they include less than 500 examples. For FERMI we use all 286 "Real Fermi Problems" in its train and development sets. Exact numbers are listed in Tab. 2.

---

Dataset
Example
reasoning on more than single chain.
StRAtEGYQA
Can Amold Schwarzenegger deadlift an adult Black rlinoceros?
MCR: The meta-reasoner is given five rea- soning chains as its multi-chain context ($3.2). We decode one chain with greedy decoding, and sample another four reasoning chains with temperature t 0.7.3 This enables the meta- reasoner to review different pieces of evidence when answering the full question ($5).
(implicit)
FERMI
How many high fives has LeBron James given/received?
(implicit)
QUARTZ (implicit)
Jeff drained his rice field in the winter- time. The field likely will produce crops when he uses it. A. more B. less
HOTPOTQA (explicit)
What city did the musician whose debut alhum shares its title with 1959 Al- fred Hitchcock film hail from?
SCR: Single-Chain Reasoning (SCR) serves as an ablation for the effect of the multi-chain context. In SCR, the meta-reasoner is given the same prompt as MCR aside from having only the greedy-decoded chain in its context. This disentangles the effect of using multiple chains from the effect of having an LLM that is separate from the decomposition model generate the final answer.
2WIKIMQA
Where was place of death Isabella Bourbon's 's father?
(explicit)
BAMBOOGLE
What the maximum airspeed (in km/h) of the third fastest bird?
(explicit)
FEVEROUS (explicit)
Is it true that Robert Broderip lived in London all his life and wrote consider- able quantity of music during the earlier part of the nineteenth century?
Baselines
We evaluate the following baselines: SA: Self-Ask (Press et al., 2022) returns the answer of a single reasoning chain, that was generated with greedy decoding.
Table 1: The multi-hop QA datasets in our experiments.
and exact-match for the binary-choice datasets. In FERMI, we use the official order-of-magnituue eval- uation by Kalyan et al. (2021). We provide addi- tional technical details evaluation in $A.
SC: Seff-Consistenyy serves as baseline which incorporates multiple reasoning chains (Wang et al., 2023). It returns the majority an- swer based on multiple chains sampled from the decomposition model. We experiment with variants of 3, 5 and 15 sampled chains (SC@3, SC@5 and SC@ 15), in line with prior work (Wang et al., 2023; Khattab et al.. 2022: Sun et al., 2023). As in MCR, we use the chain generated with greedy decoding along with ad- ditional chains sampled with 0.7.
4.1.2 Models
Our main models and baselines are all retrieval- augmented instances of code-davinci-O0 midincii-22, prompted with in-context learning exemplars (Brown et al., 2020). In $4.3, we include additional experiments with the open-source Vicuna-13B (Chiang et al., 2023) LLM. Prompt exemplars are formatted as described in $3.2. The number of exemplars varies from 6-12 between datasets. Decomposition prompt exemplars are based on random examples from the train and development sets, coupled with their gold reasoning chain. For the meta-reasoner exemplars, we use reasoning chains sampled from the decomposition model as the multi-chain context. We ensure that the answer can be inferred using the sampled chains and add an explanation before the final answer, as shown in Fig. 4. For the binary-choice datasets, STRATEGYQA, QUARTZ, and FEVEROUS, the prompt contains an equal number of exemplars from each label. For additional details regarding the full prompts, length statistics and robustness to a different choice of prompts. please refer SD.
Retrieval Similar Press al. (2022); Lazari- dou al. (2023); Paranjape et al. (2023), our mod- els and baselines use retriever based on Google Search, via the SerpAPI service.4 4 However, we also include results using an open-source retriever (Khattab and Zaharia, 2020) in $4.3. As most of our datasets contain evidence from Wikipedia (54. 1.1,, we consider it as our retrieval corpus. Therefore, format search queries "en. wikipedia org q;", with the Wikipedia domain preceding the in- termediate question. We return the top-1 evidence retrieved by Google. Retrieved evidence may be either sentences or parsed lists. Following Trivedi et al. (2022a), we also retrieve evidence for the original question q. Last, all retrieved evidence sen- tences are prepended to the decomposition ($3.1).
'Like Wang et al. (2023), we observe that greedy-decoded chains have higher accuracy compared the other chains. "https: http:///eepppp.//p// com/
Meta- Reta--Resooeer We experiment with two vari- ants of the meta-reasoner to measure the effect of

---

Dataset
Reasoning
Examples
Oracle
SA
SC@3
SC@5
SCR
MCR
STRATEGYQA
1,000
94.410011
69.3.10.3
71.5100.
72.2.10.8
70.0110.6
73.6.10.7
FERMI
implicit
286
63.1100.8
38.3.10.7
38.410.77
38.3.10.8
38.1
38.9.10.
QUARTZ
374
94.1100.5
78.3:0.4
78.240.77
77.6:00.5
80.740.1
81.6111.
НOTPOTQA
500
68.0100.
50.210..3
50.510.8
$1.310.2
56,4.0.44
57.0
2WIKIMQA
explicit
500
77.5+00.8
63.8+0.11
64.5+0088
65.4+0.6
67.24-0.2
67.9+1.44
BAMBOOGLE
120
77.3+00.5
64.6+0.6
64.6+00.4
65.0+1.5
64.7+00.4
66.5+1.77
FEVEROUS
500
88.0=0.4
66.0+1.0
67.8+0.2
67.9+0.6
65.1+0.4
69.4+1.0
Table 2: Experiments using code-davines mevinii--000 seven multi-hop open-domain QA datasets. Results are averaged over 3 runs. BAMBOOGIEE results are averaged over 5 runs due to its smaller size.
Additional implementation details about our re- trieval and MCR are described in $B.1 and $B.2.
Dataset
SCA15
MCR
MCR+SC03
STRATEGYQA
74.6
73.6
76.4
FERMI
38.6
38.9
39.2
4.2
Main Results
QUARTZ
78.3
81.6
82.6
HOTPOTQA
54.1
57.0
59.2
Next, we report our evaluation results. Overall, MCR outperforms our baselines on all 7 datasets.
2WIKIMQA
65.8
67.9
68.6
BAMBOOGLE
65.6
66.5
66.3
FEVEROUS
68.6
69.4
71.5
MCR Performance Tab. 2 presents the results for all 7 multi-hop datasets (evaluation described in $4.1.1). We evaluate both SC@5 and MCR using five reasoning chains. In addition, we list an oracle score which uses the best answer of all five chains. MCR outperforms all baselines all of the benchmarks, beating SC@5 on STRATEGYQA QA (+1.4%), FERMI (+0.6%), QUARTZ (+4.0%), HOT- POTQA (+5.7%), 2WIKIMQA (+2.5%), BAM- BOOGLE (+ (+155% and FEVEROUS (+1.5%).
Table 3: Running SC
MCR on reasoning chains.
performance
Strategy
Fermi
Quartz
HotpotCA
ZWKIMBA
Bambbogge
50
Adding Reasoning Chains We measure the gains of MCR and SC when adding reasoning chains. As extending MCR is bounded by context length, we follow straightforward approach and perform self-consistency compare this model, MCR+SC@3, which used 15 reasoning chains (5 for each MCR run), to SC@15. Tab. shows that MCR+SC@3 consistently out- performs SC@15. 5. Furthermore, though MCR uses only 5 reasoning chains, it beats SC@15 on all datasets, save STRATEGYQA. Fig. 5 plots, for each dataset, the effect that adding more reason- ing chains has meta-reasoning performance. It presents the results with chain (SCR), 5 chains (MCR) and 15 reasoning chains (MCR+SCC))
40
5
15
Number of reasoning chalrss
three MCR runs. We
Figure Per-datasel performance as function of the number of reasoning chains used by MCR (1, 15).
number of reasoning chains. In FERMI, both meth- ods perform similarly.
Recent Approaches Previously, we established the advantages of meta-reasoning over multiple reasoning chains. While an apples-to-apples com- parison with other recent approaches is impossible due to fundamental differences in the experimental setup (see $B.3), it serves as a rough measuring stick for the robustness of MCR across different tasks. In $B.3, Tab. 8 compare MCR five recent CoT-based approaches for multi-hop QA. MCR performance is comparable with the best re- sults on all datasets (shared between these works), showcasing its robustness.
Test Set Results We evaluate our models on the official test sets STRATEGYQA and FFRMI, which include 490 and 558 examples respectively. The results in Tab, 4 show that on STRATEGYQA MCR consistently beats SC, when using the same
"code-davinci-602 context is capped at 8.001 tokens. 'https: //laderbbarddddbdd allenai. wwwai.or//ste..eeeee org/strategyaa

---

Model
chains
STRATEGYQA
FERMI
Does Mercury help coronaiinr?
SC&5
5
71.4
39.8
MCR
S
72.5
39.7
Ressoning chain (Greedyy
Reasoning chein
SC@15
15
74.1
39.7
01: What the nume of coronev rus?
Q1: What me mercury used to denett!
MCR + SC03
15
75.3
40.1
Al: The coronadiuss co
&2/ Mercury tempereevre thinge.
COVID-18
02: Mieuruyy telp
symptoms
Table 4: Test set results for STRATEGYQA and FERMI.
COVID-137
conoraxitus A2: Common symptans of conrasitus are faser... Q3: Ooes the tamperature of the body ieroase....
A2: Merruyy does help delect
CDVID 18.
4.3 Open-source Models
Single-Chain Reasoning
Culti-Chain Reasoning
To further examine MCR's performance ($4.2) and for better reproducibility, we experiment with an additional open-source retriever and LLM. As our retriever, we use ColBERTv2 (San- thanam et al., 2022) over the 2018 Wikipedia dump from Karpukhin et al. (2020). In addi- tion to code- Lode-davinci-00. 0002, we experiment with Vicuna-1 3B (Chiang et al., 2023), 13-billion pa- rameters model shown outperform LLMs like LLaMA and Alpaca (Touvron al., 2023; Taori et al., 2023). We use the same prompts as in code-davinci-002. trimmed to fit a 2,048 tokens context length.
Mercury used tempereeure remperature leven Common coronaviuus fover..
The corenavrus COVID- does COVID-19 the znzweris: No.
So the arsiner by Yes.
Figure 6: An example from STRATEGYQA where the greedy chain is insufficient to answer the question. MCR beats SCR by having access multiple chains.
(+3.0%) and FEVEROUS (+1.3%). When eval- uating with 15 reasoning chains, Tab. 6, MCR+SC03 continually beats SC@15.
5 Analysis
We report the full results of the open-souree Col- BERTv2 retriever with code-davinci-e 0002 and Vicuna-138 in Tab. 5. In addition, we pro- vide results of open-source models when reason- ing over 15 reasoning chains in Tab. 6. For code-davinci-002, substituting Google Search with ColBERTv2 exhibits the same trend as in Tab. 2, albeit slight decrease in perfor- mance. MCR outperforms all other basclines, beating SC@5 STRATEGYQA (+2.3%), FERMI (43.4%), QUARTZ (43.9%), HOTPOTQA (+3.5%), 2WIKIMQA (+1.2%) ++.. BAMBOOGLE (+3.6%)
Next, we measure the importance of incorporat- ing multiple reasoning chains in MCR and qualita- tively assess its output.
When are Multiple Chains Helpful? $4.2 we observed that MCR consistently outperforms single-chain reasoning (SCR). We wish to prove that this advantage lies in cases where the meta- reasoner uses additional chains. To this end, we sort examples based the similarity of their greedy- decoded chain to the MCR explanation (details in SC.1). Lower similarity indicates less reliance of MCR on the greedy chain. Fig. 6 presents an example where the MCR explanation (pink box) includes relevant facts from a chain other than the greedy one (additional examples in $C.2). Results in Fig. 7 empirically demonstrate that on STRAT- BGYQA. MCR gains over SCR are highest when MCR explanations are less similar to the greedy chain. We observe this trend in all datasets (SC.1), serving as further evidence for MCR's 's strengths.
and FEVEROUS (+1.4%). Unsurprisingly, re- sults sharply decrease when evaluating the smaller Vicuna-138 with ColBERTv2. The comparison between MCR and SCR suggests that reasoning over multiple chains is a challenge for the weaker Vicuna-138 113B model. For example, it generates open-ended answers such as "Unknown" or "It depends" " for over 24% of the questions in STRAT- EGYQA. This suggests that meta-reasonigg over multiple chains has greater gains (compared to SCR) when both the decomposition model and meta-reasoner are larger LLMs.
Combining Reasoning Chains In addition to choosing between reasoning chains, interest- ing property of the meta-reasoner is that it combine facts from different chains. We estimate the prevalence of this phenomenon on the implicit datasets, STRATEGYQA and FERMI, which are more challenging. Given an example, we automati-
However, even on Vicuna-13B. MCR still out- performs all baselines on datasets and beats SC@5 on all 7 of them: STRATRGYQA (+0.5%), FERMI (+4.6%), QUARTZ (+3.6%), HOTPOTQA (+6.5%). 2WIKIMQA (+0.3%), BAMBOOGLE

---

Dataset
Model
Oracle
SA
SC@3
SC@5
SCR
MCR
STRATEGYQA
code-davinci-802
94.5100.7
67.110.6
69.910.11
70.810.6
67.810.5
73.1-12.1
FERMI
code-davinci-80)
64.3.10.7
33.210.3
33.210.4
33.110.4
33,910.6
36.5_2.11
QUARTZ
code-davinci-002 93.910.66
77.140.6
75.6+0.77
76.0
79.3+0.3
79.9-1.2
HOTPOTQA
code-davinci-802
67.7+0.7
50.7+0.33
51.5+0.6
52.5+0.1
55.3+0.2
56.0+1.1
2WIKIMQA
code-davinci-802 68.34:0.4
52.400.1
51.1
52.7=0.4
53.7=0.3
03.9-003
RAMBOOGLE
code-davinci-B02
56,4+1.2
45.9+1.11
47.2+1.4
47.0++.77
47.1+1.0
10.6+1.3
FEVEROUS
code-davinci--00
84.111..7
61.210.4
62.910.6
63.1=1.0
60.911.3
6..5
STRATEGYQA
Vicuna-138
82,4+0.2
59.7+0.1
61.4+0.5
62.2+0.8
63.7+00.
62.7=0.1
FERMI
Vicuna-13B
45.71:1.0
19.110.2
19.11033
18.810.3
21.5:000
2..4
QUARTZ
Vicuna-138
89.6
61.1+0.1
59.8+2.3
61.4+16
63.9+0.0
65.0=0.3
HOTPOTQA
Vicuna-13B
52.710.55
34.810.00
358110.2
37.110.4
43,4100
43.6
2WIKIMQA
Vicuna-138
52.2.0.33
32.240.4
3388106
34.01110
35.1-100
34.3-0.4
BAMBOOGLE
Vicuna-13B
42.3.116
30.710.00
30,41066
31,4106
31.3100
34
FEVEROUS
Vicuna-138
88.7+0..
61.540.6
61.040.6
61.0+08
60.6+10.0
62.3-1.2
Table 5: Experiments using ColBERTv2 retriever with code-davinci-002 and Vicuna-138, evaluated on the development sets of datasets. Results averaged over runs. The number of examples per dataset is the same as in Tab. 2. Vicuna-1 38 generation deterministic so, using greedy decoding, SCR has standard deviation of O.
Dataset
Model
SC@15
MCR+SC03
Model
SC&15
MCR+SCOO
YTRATEGYQA
code-davinci-002
72.6
75.6
Vicuna-138 13B
62.3
63.7
FERMI
code-davinci-002
34.0
36.3
Vicuna-133 13B
18.8
23,2
QUARTZ
code-davinci-002
76.5
80.7
Vicuna- 13B
60.1
64.3
HOTPOTQA
code-davinci-002
54.3
56.8
Vicuna-133 13B
37.8
44.8
2WIKIMQA
code-davinci-od
52.5
54.0
Vicuna- 13B
35.5
35.6
BAMBOOGLE
code-davinci-eet
48.9
51.8
Vicuna-138 13B
31.8
35.1
FEVEROUS
code-davinci -002
62.7
66.2
Vicuna-138
61.1
64.0
Table 6: Running SC and MCR 15 reasoning chains using ColBERTv2 retriever with code-davinci-002 (left columns) and Vicuna- 13B (right columns).
(see examples in SC.2, Fig. 10). For the remain- ing 90%, the reasoning expressed in the resulting combination is paraphrase of an individual chain.
0.8
Explanation Quality The meta-reasoner is prompted to generate an explanation alongside the final answer ($3.2). Inspired by past work (Pruthi et al., 2022), we test the quality of the MCR expla- nations. Four of the authors manually reviewed 600 random examples, 100 per dataset (sans FEVER- OUS $B.2) and scored their meta-reasoner explana- tions. Each explanation is scored either (irrele- vant), 2 (partially relevant) or 3 (highly relevant), based on its relevance to answering the question. We find the explanation is highly relevant in 82% of the cases (87% excluding FERMI, which is the most challenging). and is irrelevant in less than 3%.
Low (5 (3.5%)
Mud km 134.0%)
Migh 131.8%)
Edenticcal (24.5%)
Figure 7: MCR and SCR accuracy on STRATEGY QA, categorized by the similarity of the greedy chain to the MCR explanation. When MCR uses a chain other than the greedy one (lower similarity), it outperforms SCR.
cally check if its meta-reasoner explanation is the result of combining chains. We examine if of the output sentences appears exactly one chain, while another sentence absent from that chain and is part of a different chain. We consider sen- tences as similar if their ROUGE-1 precision is above 0.8, and distinct if it is below 0.2. Overall, in 20% of STRATEGYQA QA examples and 25% of FERMI, the MCR explanation results from bining reasoning chains. From a manual analysis of 50 such examples for each dataset, we observe that these multi-chain explanations are better than any individual reasoning chain in 10% of cases
Next, we evaluate the faithfulness of explana- tions (Jacovi and Goldberg, 2020), namely, whether person provided only with the question and MCR explanation would answer the same as the model. Our focus was on examples with quality explana- tions (score 3), since they are answerable given the explanation. We answered each question based on

---

Dotaset
Va.
De.
Re.
Co.
Ex.
An.
6
Related Work
STRATEGYQA
20%
8%
20%
FERMI
6%
39%
20%
4%
17%
23%
For a thorough survey on LLM reasoning see Lu et al. (2022); Huang and Chang (2022); Qiao et al. (2022). A slew of recent works have focused on eliciting multi-step reasoning in LLMs, including scratchpads (Nye et al., 2022), chain-of-thought prompting (Wei et al., 2022; Zhou et al., 2022), learned verifiers (Cobbe al., 2021), selection- inference (Creswell et al., 2022) and bootstrapping (Zelikman et al., 2022).
QUARTZ
14%
6%
13%
19%
11%
40%
HOTPOTQA
33%
24%
24%
11%
2WIKIMQA
39%
4%
35%
12%
8%
6%
BAMBOOGLE
266
8%
32%
24%
13%
0%
FEVEROUS
7%
14%
34%
23%
2015
6%
Table 7: Error classes per dataset: Valid (Va.), Decom- position (De.), Retrieval (Rc.), Contradicting facts (Co.),
Explamation (Ex.) and Answer (An.). We allow multiple error categories per example.
Self-consistency (Wang et al., 2023; Fu et al., 2022) selects the majority answer across multi- ple chains, outperforming learned verifiers and "sample-and-rank" approaches (Adiwardana et al., 2020; Freitas et al., 2020). Li et al. (2022) further improve SC by increasing chains' diversity and in- troducing a trained verifier. Tafjord al. (2022) over-samples chains and verifies them using a natu- ral language inference model on intermediate steps, while He et al. (2022) re-rank chains based on in- termediate retrieved evidence. In addition, meta- reasoning is closely tied to self-reflection in LLMs, which is becoming increasingly important in using the L.M to review multiple strategies (Yao et al., 2023; Shinn et al., 2023; Madaan al., 2023).
the model's explanation. In 90% of cases (95% ex- cluding FFRMT), the MCR predictions matched our own, highlighting the faithfulness of its explana- tions. We attribute part of the gap between human and MCR predictions to implicit reasoning tasks, where humans lead by five points, on average. For the full results, see $C.3.
Error Analysis We manually analyzed 700 er- rors by MCR (100 per dataset). We consider the following categories: Valid predictions where the generated answer is accurate or the original ques- tion is ambiguous; Decomposition errors where no chain has the necessary reasoning steps to answer the question; Retrieval errors where the retrieved contexts were irrelevant leading the model to hal- lucinate; Explanation errors where MCR generates a wrong explanation while a correct one is present in the multi-chain context; Answer errors are when the MCR explanation is correct, but the answer is not; Contradicting facts cases where MCR errs due to contrasting statements appearing in the multi-chain context.
Recent works proposed revising LL.M-generated texts by using retrieved sentences (Gao et al., 2022) or model-generated feedback (Madaan et al., 2023; Chen et al., 2023; Paul et al., 2023). MCR similarly reviews L.LM- generated reasoning chains however, its focus is meta-reasoning multiple chains.
Significant QA research has been dedicated to reasoning over multiple facts retrieved from an underlying corpus. Such tasks include multi-step questions that require explicit reasoning (Talmor and Berant, 2018; Welbl et al., 2018; Wolfson et al., 2020; Trivedi et al., 2022b), implicit reasoning (Geva et al., 2021) and multi-modal capabilities (Talmor et al., 2021).
Tab. 7 lists the prevalence of the error categories per dataset. In four datasets, over 20% of errors appear to be valid predictions, labeled incorrect due to ambiguous questions, outdated answers or dataset errors. Decomposition a challenge in the implicit datasets, STRATEGYQA and FERMI, with more than 24% of errors. Comparing errors on different reasoning datasets (excluding valid examples): Explanation and Answer errors are 50% implicit reasoning datascts compared to 23% on explicit reasoning ones; Retrieval errors are more prevalent in explicit reasoning tasks with 66% of errors being due to Retrieval or Contradicting facts, compared to 30% in implicit datasets. Additional technical details on our analysis are $C.4.
Recent works also target Srrieval-auugenteed LLMs, prompted solve open-domain questions (Lazaridou al., 2023; Khattab et al., 2022; Trivedi et al., 2022a; Ram et al., 2023; Yoran et al., 2023).
Conclusion
This work introduces MCR for meta-rcasoning over multiple reasoning chains. We evaluate MCR on 7 datasets for multi-hop QA that require both implicit and explicit reasoning in an open-domain setting and show that it outperforms previous ap- proaches on all evaluation benchmarks.

---

8 Limitations
Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, 1lya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems 33: Annual Conference on Neural Information Process- ing Systems 2020, NewrIPS 2020, December 6-12, 2020, virtual.
In this work we introduce a meta-reasoner model to reason over multiple reasoning chains. While we opt for prompted LLM as our meta-reasoner, we do not experiment with a fine-tuned meta-reasoning model. For the meta-reasoner context, experi- ment with variants which include either generated QA pairs or retrieved evidence sentences. We leave further improvements to the meta-reasoner con- text as future work. Due to the inference costs of current state-of-the-art LLMs we evaluate on the code-davinci-000 model, similar to prior work (Trivedi et al., 2022a; Wang et al., 2023). However, to improve the reproducibility of our work we also provide results with an open-source LLM (Chiang al.. 2023) and retriever (Khattab and Zaharia, 2020).
Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. 2023. Teaching large language models to self-debug.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. 2023. Vicuna: An open- source chatbot impressing gpt-4 with 90%* chatgpt quality.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebas- tian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam M. Shazeer, Vinod- kumar Prabhakaran, Emily Reif, Nan Du Benton C. Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengeheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev. Henryk Michalewski, Xavier García, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hycontack Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, An- drew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Re- won Child, Oleksandr Polozov, Katherine Lee, Zong- wei Zhou, Xuezhi Wang, Brennan Sacta. Mark Díaz. Orhan Firat, Michele Catasta, Jason Wei, Kathleen S. Meier-Hellstern, Douglas Tick, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022. Palm: Scaling language mod- eling with pathways. ArXiv, abs/2204.02311..
Acknowledgements
We would like to thank Harsh Trivedi, Ofir Press, Mor Geva, Peter Clark and Ashish Sabharwal for their feedback and insightful comments. We thank SerpAPI for their support by granting us an aca- demic discount. This research was partially sup- ported by the Yandex Initiative for Machine Learn- ing and the European Research Council (ERC) un- der the European Union Horizons 2020 research and innovation programme (grant ERC DELPHI 802800). This work was completed in partial ful- fillment of the Ph.D. of Ori Yoran and the Ph.D. of Tomer Wolfson.
References
Daniel Adiwardana, Minh-Thang Luong, David R. So.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. Training verifiers to solve math word problems. ArXiv preprint, abs/21 14168.
Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, and Quoe V. Le. 2020. Towards human-like open- domain chatbot.
Rami Aly, Zhijiang Guo, M. Schlichtkrull, James Thorne, Andreas Vlachos, Christos Christodoulopou- los, Oana Cocarascu, and Arpit Mittal. 2021. Feverous: Fact extraction and verification over unstructured and structured information. ArXiv, abs/2106.05700
Antonia Creswell, Murray Shanahan, and Irina Higgins. 2022. Selection-inference: Exploiting large language models for interpretable logical reasoning. ArXiv, Abs/2205.09712.
Daniel De Freitas, Minh-Thang Luong, David R. So. Jamie Hall, Noah Fiedel, Romal Thoppilan. Zi Yang. Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu. and Quoe V. Le. 2020. Towards a human- open- domain chatbot. ArXiv, haa/2000.099977
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess,
Yao Fu, Hao-Chun Peng, Ashish Sabharwal, Peter Clark, and Tushar Khot. 2022. Complexity-bane based prompt- ing for multi-step reasoning. ArXiv, abs/2210.000721

---

Luyu Gao, Zhuyun Dai, Panupong Pasupat, Anthony Chen, Arun Tejasvi Chaganty, Yicheng Fan, Vin- cent Y. Zhao, Ni Lao, Hongrae Lee, Da-Cheng Juan, and Kelvin Guu. 2022. Rarr: Rescarching and re- vising what language models say, using language models.
Omar Khattab and Matei Zaharia. 2020. Colbert: Ef- ficient and effective passage search via contextual- ized late interaction over BERT. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, SIGIR 2020, Virtual Event, China, July 25-30, 2020, pages 39-48. ACM.
Mor Geva, Daniel Khashabi, Elad Segal. Tushar Khot, Dan Roth, and Jonathan Berant, 2021. Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics, 9:346- 361.
Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. 2023. Decomposed prompting: A modular approach for solving complex tasks.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2022. Large lan- guage models are zero shoo-shot reasoners. In ICML 2022 Workshop on Knowledge Retrieval and Language Models.
Hangfeng He, Hongming Zhang, and Dan Roth. 2022. Rethinking with retrieval: Faithful large language model inference.
Jie Huang and Kevin Chen-Chuan Chang. 2022. To- wards reasoning in large language models: A survey.
Angeliki Lazaridou, Elena Gribovskaya, Wojciech Jan Stokowiec, and Nikolai Grigorev. 2023, Internet- augmented language models through few-shot prompting for open-domain question answering.
Alon Jacovi and Yoav Goldberg. 2020. Towards faith- fully interpretable NLP systems: How should we define and evaluate faithfulness? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4198-4205, On- line. Association for Computational Linguistics.
Yifei Li, Zeqi Lin, Shizhuo Zhang. Qiang Fu, Bei Chen, Hlan-Guang Lou, and Weizhu Chen. 2022. On the advance of making language models better reasoners. ArXiv, abs/2206.0255 022336..
Saurav Kadavath, Tom Conerly, Amanda Askell, T. J. Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zachary Dodds, Nova DasSarma, Eli Tran- Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage. Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson, John Kernion, Shauna Kravec, Liane Lovitt, Kamal Ndousse, Catherine Olsson, Sam Ringer, Dario Amodei, Tom B. Brown, Jack Clark, Nicholas Joseph, Benjamin Mann, Sam McCandlish, Christopher Olah, and Jared Kaplan. 2022. Language models (mostly) know what they know. ArXiv, 0is/2207.002211
Chin- Yew Lin. 2004. ROUGE: A package for auto- matic evaluation of summaries. In Text Summariza- tion Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Pan Lu, Liang Qiu, Wenhao Yu, Sean Welleck, and Kai-We Chang. 2022. A survey of deep learning for matherattial reasoning.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdan- bakhsh, and Peter Clark. 2023. Self-refine: Iterative refinement with
Ashwin Kalyan, Abhinav Kumar, Arjun Chan- drasekaran, Ashish Sabharwal, and Peter Clark. 2021. How much coffee was consumed during EMNLP 2019? fermi problems: A new reasoning challenge for AI. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7318-7328, Online and Punta Cana, Domini- can Republic. Association for Computational Lin- guistics.
Maxwell Nye. Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowyez. Maarten Bosma, David Luan, Charles Sutton, and Augustus Odena. 2022. Show your work: Scratchpads for intermediate computation with language models.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open- domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769-677 6781, Online. Association for Computational Linguistics.
Bhargavi Paranjape, Scott M. Lundberg, Sameer Singh. Hanna Hajishirzi. Luke Zettlemoyer, and Marco Tulio Ribeiro. 2023, Art: Automatic multi- step reasoning and tool-use for large language mod- els. ArXiv, abs/2303.09014.
O. Khattab, Keshav Santhanam, Xiang Lisa Li, David Leo Wright Hall, Percy Liang, Christopher Potts, and Matei A. Zaharia. 2022. Demonstrate-searchh predict: Composing retrieval and language models for knowledge-intensive nip. ArXiv, abs/2212. 14024.
Debjit Paul, Mete Ismayilzada, Maxime Peyrard, Beat- riz Borges, Antoine Bosselut, Robert West, and Boi Faltings. 2023. Retiner: Reasoning feedback on in- termediate representations.

---

Ofrr Press, Muru Zhang. Sewon Min, Ludwig Schmidt, Noah A. Smith, and Mike Lewis. 2022. Measuring and narrowing the compositionality gap in language models. ArXiv, abs/2210.03350.
Alon Talmor, Ori Yoran, Amnon Catav, Dan Lahav, Yizhong Wang, Akari Asai, Gabriel Ilharco, Han- naneh Hajishirzi, and Jonathan Berant. 2021. Mul- timodalqa: complex question answering over text, tables and images. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
Danish Pruthi, Rachit Bansal, Bhuwan Dhingra, Livio Baldini Soares, Michael Collins, Zachary C. Lipton, Graham Neubig, and William W. Cohen. 2022. Evaluating explanations: How much do ex- planations from the teacher aid students? Transac- tions of the Association for Computational Linguis- tics, 10:359-375.
Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin. Percy Liang. and Tatsunori B. Hashimoto. 2023. Stanford alpaca: llama model. https:/ An instruction-followiin stanford_alpsss _lpcca. gi thub. wtt.//oo//tttoo http:/ttts-labb/aa wawtssslaastttarrrrrrrroao
Shuofei Qiao, Yixin Ou, Ningyu Zhang, Xiang Chen, Yunzhi Yao. Shumin Deng, Chuanqi Tan, Fei Huang. and Huajun Chen. 2022. Reasoning with language model prompting: A survey.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Fric Hambro, Faisal Azhar, Aur' elien Rodriguez. Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. ArXiv, abs/2302. 13971.
Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. 2023. In-context relrieval-augmented lan- guage models. Transactions of the Association for Computational Linguisties.
Harsh Trivedi, Niramjan Balasubramaniaan Tushar Khot, and Ashish Sabharwal. 2022a. Interleaving retrieval with chain-of-thought reasoning for knowledge- intensive multi-step questions.
Keshav Santhanam, Omar Khattab, Jon Saad-Falcon,
Christopher Potts, and Matei Zaharia. 2022. Col-
BERTv2: Effective and efficient retrieval via lightweight late interaction. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, pages te. United States. Association for Computational Linguistics.
Harsh Trivedi. Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022b. MuSiQue: Multi- hop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 10:539-554.
Seat-
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Fdd H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations.
Noah Shinn, Federico Cassano, Edward Berman, Ash- win Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: Language agents with verbal rein- forcement learning.
Zhiqing Sun, Xuezhi Wang, Yi Tay, Yiming Yang, and Denny Zhou. 2023. Recitation-augmennte language models. ICLR.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Ed Huai hsin Chi, Xia, Quoe Le, and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models. NeurIPS.
Oyvind Tafjord, Matt Gardner, Kevin Lin, and Peter Clark. 2019. QuaRTz: An open-domain dataset of qualitative relationship questions. In Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNILP), pages 5941-5946, Hong Kong, China. Association for Computational I. Linguistics.
Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel. 2018. Constructing datasets for multi-hop reading comprehension across documents. Transactions of the Association for Computational Linguistics, 6:287- 302.
Tomer Wolfson, Mor Geva, Ankit Gupta, Matt Gard- ner, Yoav Goldberg, Daniel Deutch, and Jonathan Berant. 2020. Break it down: A question understand- ing benchmark. Transactions of the Association for Computational Linguistics, 8:183-198.
Oyvind Tafjord, Bhavana Dalvi Mishra, and Peter Clark. 2022. Entailer: Answering questions with faithful and truthful chains of reasoning.
Alon Talmor and Jonathan Berant. 2018 The web as a knowledge-base for answering complex questions. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume I (Long Papers), pages 641-651, New Or- leans, Louisiana. Association for Computational Lin- guistics.
Zhilin Yang. Peng Qi. Saizheng Zhang, Yoshua Bengio. William Cohen, Ruslan Salakhutdinov, and Christo- pher D. Manning. 2018 HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empiri- cal Methods in Natural Language Processing, pages 2369-2388 Brussels, Belgium. Association for Com- putational Linguistics.

---

Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griftiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models.
Q: What is San Andreas about?
Google
rn-uilipada
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2022. React: Synergizing reasoning and acting in language models. ArXiv preprint, abs/2210.03629.
Retieval Result
About 300,00 weits ib secondd
plot
massive earthqualee
Andreas Fault, devestating the West Coast United States. Principal photography of the film started on Aprli 22, 2014, in Queensland, Australio, and wrapped on 28
Francisco.
Ori Yoran, Tomer Wolfson, Ori Ram, and Jonathan Berant. 2023. Making strrival-augmenter language models robust to irrelevant context.
Wilpet'a
San Andreas (fim) Wilcpedis
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Good- man. 2022. STar: Bootstrapping reasoning with soning. In Advances in Neural Information Process- ing Systems.
San Andreas (fiim]: its plot centers on massive earthquake caused by
San Andreas Fault, devastating the West Coast the United
States. Principal photography the started on April 2014, Queensland, Australia, and wrapped up on July 28 in San Francisco.
Figure Example for retrieved evidence snippet for one of the intermediate questions from Fig. 1.
Denny Zhou, Nathanael Scharli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Olivier Bousquet, Quoe Le. and Ed Huai hsin Chi. 2022. Least-to-most prompting enables com- plex reasoning in large language models. ArXiv, 2oss2205510625..
input helps the model, we note that we provide to all our baselines for fair comparison with MCR. Nevertheless, even when given the gold units, predicting the final answers to FERMI prob- lems remains highly challenging.
A Evaluation
A.1 Generating Unknown as the Answer
B Models
As we prompt LLMs to generate answers, po- tential outcome for the model abstain from answering the question, by generating Unknown as its answer. Additional cases are when the model generates an end-of-sequence token without any final answer. In the binary-choice datasets, STRAT- EGYQA, QUARTZ and FEVEROUS, we assign score of 0.5 to such examples, thereby simulating random guess. When submitting predictions the STRATEGYQA test set, we identify cases of model abstains or null predictions beforehand. For these examples, we assign a label of either Yes or No at random. In datasets with open-ended answers, we assign a score of 0 when the predicted answer is either Unknown or null. To make Self-Ask stronger baseline, when the greedy decoded chain has null answer, we randomly choose prediction from one of the other chains. For SC, we do not consider predictions from chains where answers are Unknown or null.
B.1 Retrieval
For our retrieval, we use the Google Search Engine, via SerpAPI, and return the top-1 retrieved result as an evidence snippet. Snippets can include answer- boxes and tables.' We prepend the page title to the beginning of the snippet, as shown in Fig. 8.
B.2 Implementation Details
We describe the design choices made in our MCR model, such as preforming retrieval on the original question and a variant of the meta-reasoner prompt for FEVEROUS. Due to cost limitations, we evalu- atc our design choices at smaller scale and avoid running an exhaustive grid search.
Retrieving the Original Question We follow past work (Trivedi et al., 2022a) by incorporat- ing retrieved evidence for the original question in addition to evidence retrieved for the interme- diate steps ($3.1). This has a positive or neg- ligible effect on most datasets however, it dra- matically decreases the results of all models on the FERMI task. Results drop for SA (38.3+0.7 to 34.7+0.5), SC (38.340.8 to 34.4=0.3), SCR (38. 11.88 to 34.4+0.8) and MCR (38.920.8 to
A.2 FERMI
The FERMI dataset requires approximating numeric answers for open-ended questions. Example ques- tions are shown in Tab. 1 and Fig. 2. When provid- ing a FERMI question to our models and baselines we also add the gold answers measure units (c.g. meters, cubes, litres, etc.). While this additional

---

Model
Ret.
LLM
# Chains
STRGYQA
HOTPOTQA
2WIKIMQA
CoT (Wang al., 2023)
code-davinci-002
1
73.4
39.8
CoT+SC (Wang al., 2023)
code-davinci-002
40
79.8
44,6
Self-Ask (Press al., 2022)
yes
text-davinci-002
1
52.6
DSP (Khattab al., 2022)
yes
text-davinci-002
20
62.9
IR-CoT (Trivedi al., 2022a)
yes code-davinci-002
1
61.2
65.2
Self-Ask (ours)
yes
code-davinci-06%
1
69.3
50.2
63.8
MCR
yes
code-davinci-802
5
73.6 (+4.3)
57.0
(+6.8)
67.9 (+4.1)
MCR-SC@3
yes
code-davinci-0e%
15
76.4 (+7.1)
59.2 (+9.0)
68.6 (+4.8)
Table Recent ODQA results using CoT prompting on LLMs. We list whether retrieval component is used (Ret.),
the LLM, and the number of reasoning chains used to generate the answer. Different systems evaluate on different evaluation sets for each dataset. Retrieval-augmented systems vary in terms of their retriever and corpus.
0.70. Therefore, our models are run with- out original question retrieval when evaluated FERMI. Interestingly, while all models perform roughly the same without original question re- trieval, MCR appears better by 2 points when evi- dence for the original question is used. We hypoth- esize that it might be due to MCR being somewhat more robust to the addition of irrelevant evidence.
similar solutions.
Evaluation differences include the retrieval cor- pora, as both IR-CoT and DSP use the official Wikipedia dump provided with the HOTPOTQA dataset (Yang et al., 2018). Our retrieved evidence are from an updated version of Wikipedia, via Google Search. Since certain facts may change time, this could potentially explain the high percentage of MCR predictions labeled as valid in our error analysis ($5).
FEVEROUS Meta-Reasoner Prompt As de- scribed in $3.2, the metaareesonnr generates an ex- planation which precedes the final answer. FEVER- OUS is distinct from all other datasets as require verification of multiple facts in order to verify or disprove a complex statement. When a statement is false, we list one or more of its false intermediate facts along with its correction. For example, in Fig. 4 we list that Robert Broderip lived in Bristol, not London. When prompting the meta-reasoner list both true and false intermediate facts, observed decrease in performance for both MCR 1.0 to 66.4+0.7) and SCR (65.1 to 62.9 We hypothesize that repeating multiple true facts excessively prompts the model to predict the label "Yes" in cases where most (but not all) of the intermediate facts are correct.
We emphasize that our focus is on highlight- ing the potential of reasoning on reasoning chains. MCR is method aimed at improving models which generate reasoning chains. Compared to SC, observe that MCR further boosts the underly- ing SA model. While task-specific improvements are possible, they are orthogonal to our work.
B.4 Reasoning on Retrieved Evidence
The meta-reasoner answers questions given a multi- chain context of question-unswer (Yt, a;) pairs, ex- tracted from multiple reasoning chains ($3.2). We experiment with an alternative multi-chain con- text, comprised of questions and retrieved evi- dence (gi. e;) ($3.1). This setting resembles past work (Trivedi et al., 2022a) however, our sentences are intermediate evidence from multiple reason- ing chains, not just the greedy-decoded chain. We compare these variants, MCR- and SCR-Ev, to MCR and SCR that reason QA pairs. Tab. 9 shows that meta-reasoning on retrieved evidence is less effective. The gap more evident in implicit reasoning tasks, perhaps due retrieved evidence being less relevant on average. Example prompts for MCR-Ev and SCR-Ev are listed $D.
B.3 Empirical Comparison to Recent Approaches
Tab. 8, we compare MCR to recent CoT-based approaches for multi-hop reasoning. An apples-to- apples comparison is not possible, as these methods do not evaluate on all 7 of our datasets and use vary- ing samples of 500-1,000 dev examples for their evaluation. Moreover, different methods use dif- ferent retrieval corpora, hyperparameters, prompts and L.Mss. Nevertheless, we argue that a direct comparison serves as a measuring stick for MCR's robustness across multiple datasets, compared to

---

Dataset
SCR-Ev
SCR
MCR-Ev
MCR
STRATEGYQA
69.11104
70.0.10.6
73.2100.6
73.6-10.7
FERMI
34.110.6
38.1
33.910.3
38.910.8
QUARTZ
76.1+0.2
80.7-10.1
7..2+
81.6
HOTPOTQA
53.510.1
56,410.44
58.2.11.0
57.010.8
2WIKIMQA
66.24-0.2
67.2-0.2
67.1
67.9+0.4
BAMBOOGLE
64.1+000
64.7+0.4
27.4+2.3
66.5++177
FEVEROUS
64.0++0.
65.1=004
62.5+0.5
69.4+1.0
Table 9: Effect of using question-answer pairs versus question-evidence pairs input the meta-reasoner.
Analysis
question:
"What does the goddess associated with the goddess Frigg consists of what tales?"
C.1
When are Multiple Chains Helpful?
The input to our meta-reasoner model is a con- text comprised of (q, a.) pairs, generated by the de- composition model. As the decomposition model an LLM that conditioned on retrieved evidence (and prior decomposition steps) it may hallucinate false intermediate answers. In cases such hallu- cinations we distinguish between two error types, based on the relevant component. First, Retrieval errors are cases where no relevant information was retrieved, leading to the decomposition model hal- lucinating incorrect a,, passed reasoner's context. Second, we treat cases where relevant evidence was retrieved, but the decomposi- tion model ignored it and hallucinated an incorrect O2 as Decomposition errors.
In $5, we have shown that the advantage of MCR over SCR lies in examples where the meta- reasoner uses chains other than the one generated through greedy decoding. In Fig. 9 we provide the results for all other datasets, in addition to the STRATEGYQA results in Fig. 7. The similar trend among all datasets is that in examples with lower similarity the greedy chain, MCR gains over SCR are higher.
The similarity between the meta-reasoner expla- nation and the greedy decoded reasoning chain is defined as follows: We calculate the ROUGE-1- 1- precision (Lin, 2004) between the explanation and the chain. Low, Medium, and High are based on thresholds of 3 3 and 1 respectively, with the Identical category indicating exact match.
the meta-
Errors stemming from Contradicting Facts, are cases where the meta-reasoner context contains two contradicting facts, one accurate while the other was hallucinated by the decomposition model. For example, Fig. 12 displays an example where the context has contradicting facts on who was the father of Eliezer Yehuda. When the meta- reasoner has contradicting facts, it is expected to select the correct fact, based on the knowledge encoded in its parameters. Addressing such errors in future work could rely on refining generated text with methods such RARR (Gao et al.. 2022). As eтror classes mainly match the MCR components, this error breakdown could potentially help guide future improvements.
C.2 Combining Reasoning Chains
Fig. 10 provides additional examples for combining facts between multiple reasoning chains.
C.3 Explanation Quality Analysis
We provide additional details the annotation for the scoring meta-reasoner explanations. The annotation was performed by 4 graduate students that are authors of this paper. The annotators were presented with a question and an explanation, and asked to perform two tasks: (a) score the expla- nation for its quality and (b) answer the question based on the meta-reasoner explanation. We pro- vide the full instructions shown to the annotators in Fig. 11 and the full results in Tab. 10.
D Prompts
D.1 Prompt Details
C.4 Error Analysis
We provide example prompts for our models for one explicit dataset (2WIKIMQA, decomposition: Fig. 13, MCR/SCR: Fig. 15, MCR-Ev/SCR-EV Fig. 17) and one implicit dataset (STRATEGYQA, decomposition: Fig. 14, MCR/SCR: Fig. 16. ECR-EV Ev:Fig.. 18). All of our prompts
We provide additional details regarding our error analysis ($5). In less than 5%, we encountered grammatically bad questions which we were un- able to comprehend and were therefor discarded from our analysis. For example the HOTPOTQA

---

2014
Son
0.4
0.2
0.0
Low 494
Necomide PR)
Hew 10751
Medumi
Low 2.00)
Meriin Ja-h
H1 19.4%
Inder INS 251
Bamzoogie
HARDODA
Feverus
1.0
0.8
0.6
0.4
0.
0.0
H6h and iertcal (ALAn))
L0. Machum (4L,50 Hih AAAA Serezal LAVONS
Figure 9: MCR and SCR accuracy FERMI, QUARTZ, 2WIKIMQA, BAMBOOGLE, HOTPOTQA, and FEVEROUS, on examples categorized by their MCR explanation's 's similarity to the greedy chain. MCR performs similarly to SCR when similarity high, and outperfoms SCR when similarity is lower. Error bars indicate standard deviation, which tends to be high when the number of examples in the bin is small. For FEVEROUS we display the variant where MCR has to repeat relevant facts ($B.2), to make sure the MCR explanation not empty.
will be released along with our codebase. We use random examples and spend minimal effort on prompt engineering. The number of exemplars varies slightly between dataset and model, with the exact numbers listed in Tab. 11.
GYQA and HOTPOTQA. When using different ran- dom exemplars, we observe that MCR still outper- forms all baselines. Even though decomposition performance (SA) is more affected by the set of ex- emplars, the performance trend remains the same, with MCR being top. Tab. 13 lists the experi- ment results, evaluated 500 examples from each dataset. We also provide the original prompt results in parenthesis (averaged over 3 runs).
D.2 Prompt Statisties
In Tab. 12 provide statistics of the sequence lengths for all our models, which include all the decomposition prompts, output decomposition se- quences, retrieved evidence and the meta-rcasoning prompts. The statistics are for our decomposition model (used by all of our baselines), as well as for the meta-reasoning prompts (used by SCR and MCR). Note that generating single reasoning chain requires multiple LLM calls, for each decomposition step. Therefore, single decompo- sition generation is generally longer than applying one additional meta-reasoning step.
Results are averaged over multiple runs, corre- sponding to the results in Tab. 2. Sequence lengths in Tab. 12 correspond to the number of tokens pro- vided by the code-davinci-ee -002 tokenizer.
D.3 Robustness to Choice of Prompt
We empirically measure our method's sensitivity the prompt of choice. this end, we randomly sampled new exemplars for both our decompo- sition and meta-reasoning prompts for STRATE-

---

Q: Peyton about seismology?
many people our
Greedy roasoning (F1)
Reasoning
Greody roasoning (F1)
Roasoning
AU fined Paynon film directo.
rocuparion?
G2: Blad Paykon wark
A.1: A lin dineoor has kry rite..
QUI How maryy pezple do reet
a
Al: avorage erpedtancy Whe the oi people you can oet in de your? in
oxpectancy of porson?
carhauukos
Sananskogy
movie San Aecroas,
2i Brui Plytan cinedad the
day,
CUZ: many days
number
&S. No. lim drodurs do not reed axraaaanss.
eathouskee?
Bl San Andrras fin diraded whoue plut conters around Ragalae
AZ There are 305 days year.
Single-Chain Reasoning
Reasoring
Single-Chain
Ressoning
Onad Paykon a diredos eorthauckes San Andreas a 2015 American diseder iim by Flaetar, whese anrthguaion carood by Ran Andwas Fash
is
Bis dinctor.
le ammzlogy a the staty
We ment around 10 peopleir day, There days your the
There 365 days year. The zeera pagg expeclancy of person
nathousken. rester
the enswer No.
So aUT 11733500002222440 200223
Figure 10: Examples for combining facts from multiple reasoning chains.
dataset
Reasoning
%3
%2
%1
Sim predictions
Human
MCR
STRATEGYQA
79
18
3
89.9
77.2
72.2
FERMI
implicit
60
32
8
76.0
47.8
40.7
QUARTZ
91
7
2
98.9
87.9
86.8
HOTPOTQA
77
21
2
95.4
49.2
49.2
2W1KIMQA
explicit
97
2
1
94.9
69.6
69.2
BAMBOOGLE
90
9
98.2
71.5
71.4
Average
implicit
76.7
19.0
4.3
88.3
71.0
66.6
Average
explicit
88.0
10.7
1.3
63.3
Average
82.3
14.8
2.8
92.2
67.2
64.9
Table 10: Full results for the explanation quality analysis. Sim jin_prrcictions indicates the similarity between the human MCR prediction, calculated using dataset-specific metrics described in $4.1.1. Human _acc and MCR ace represent the accuracy of humans and MCR predictions. respectively. Since only explanations with score of 3 are guaranteed to contain the necessary information to arrive at an answer, we filter other examples when calculating sim predictions, Human acc, and MCR acc.
Dataset
SA
SCR
MCR
SCR-Ev
MCR-EV
STRATEGYQA
10
6
FERMI
6
6
QUARTZ
6
8
8
HOTPOTQA
12
10
10
10
10
2WIKIMQA
6
6
6
6
BAMBOOGLE
6
6
FEVEROUS
10
10
10
10
10
Table II: The numher of exemplars for each model and dataset. Since MCR and SCR, and MCR-Ev and SCR-Ev use the same prompt they have the same number of exemplars.

---

Datasct
Dec.
Dec, steps
Dec. out.
Ret. len.
Meta-reason
SCR
MCR
STRATEGYQA
2,242
2.920.6
103.34 48.0
190.6266.7
1,652
1,749.0= 52.0
2.032.9+110.7
FERMI
1,442
2.3
91.4+31.9
165.8478.9
1,681
1.765.5522.5
1.984.1479.6
QUARTZ
839
1.240.5
55.2+20.6
92.7+28.7
2,129
2.202.4#19.8
2,343.62:8..3
HOTPOTQA
2,508
1.7+0.7
86.1+91.9
153.0=84.1
2,380
2,460.3+28.1
2666.2290.0
2WIKIMQA
1,920
2.4+0.8
92.7+30.5
201.3+59.3
2,029
2,116.4425.6
2,363.0+104,6
BAMBOOGLE
1,342
74.3+37.5
204.5=72.1
10035.112.9
122.2.150.2
FEVEROUS
3,741
2.940.9
118.2436.4
197.1
2,826
2,956.8438.1
3,276.84123.1
Average
2,004.9
2.240.6
88.7+18.7
172.1=37.0
1,95199
2,040.8+563.1
2.270.0+587.7
Table 12: Prompt lengths (number tokens) used each dataset: decomposition prompts (Dec.); number output decomposition steps (Dec. steps); output decomposition length (Dec. out.); retrieved evidence length (Ret. len.); meta-reasoning prompt; SCR prompt length; MCR prompt length.
Dataset
Examples
SA
SC85
SCR
MCR
STRATEGYQA HOTPOTQA
500
66.6 06.33++.3)
71.0 (72.2+0)))))
68.2 (70.0+0.6)
72.0 (73.6+-0.7)
500
54.3 (50.2.0.3)
56.2 (51.3002)
57.1 (56.410.4)
59.3 (57.0:0.8)
Table 13: Experiments with ceeeedavvniii--ne -022 when using prompts with different random exemplars for both the decomposition and meta-reasoning prompts. Original prompt results are parenthesis, for comparison.

---

Task Instructions:
are each assigned random examples of questions along
explanations & final answers generated by MCR.
For each question ask you
(A) Score MCR explanation quality (1-3)
(B) Answer the question based the MCR explanation
Feel free leave you comments "comments" column!
Task (A): Explanation quality
Go over the original question and the MCR explanation Score the MCR explanation on scale 1-3:
1: The explanation is irrelevant or incorrect with respect to answering the question
The explanation is somewhat correct as contains partially information answering question
The explanation is correct and contains all the relevant the question
information
Task Answer question
Read the original question MCR explanation
Answer the original question based on the information provided by MCR explanation
the explanation does not have enough information answer with 'Unknown'
For Fermi questions please provide the equation instead of the final computation result
Figure 11: The annotation instructions for the MCR
explanation quality analysis.
Given a question and context, answer question step- by-step. If you are unsure, answer Unknown.
Context:
Who the father of modern Hebrew? The father of modern
Hehrew is Eliezer Ben-Yehuda
Who is the father of Eliezer Ben- Yehuda? The father of Eliezer Ben-Ychuda Abraham.
Who the father modern Hehrew? The father modem Hebrew is Eliezer Ben- Yehuda. Who the father Eliezer Ben- Yehuda? Bliezer Ben- Yehuda's 's father Yehuda Leib.
Question:
Who is the father of the father of modern He-
brew?
The father of modern Hebrew is Eliezer Ben- Yehuda. The father of Eliezer Ben-Yeh Ben-Yehuda is Abraham. So the answer is: Abraham.
Answer:
Gold answer is: Yehuda Leih
Figure 12: Example Contradicting Facts error. When generating the explanation, the meta-reasoner has to rely on knowledge encoded in its parameters decide between multiple contradicting facts in its context on who was the father of Eliezer Ben- Yehuda.

---

Given the following question, answer by providing follow up questions and intermediate answers. If no follow up questions are necessary, answer the question directly. You are also provided with the most relevant google snippet for each intermediate question.
Contextl: Xawery Zulawski: Polish-Russian Russian War (Wojna polsko-ruska) 2009 Polish film directed by Xawery Zulawski based on the novel Polish-Russian War under the white-red flag by Dorota Maslowska. So the answer is Xawery Zulawski. Context2: Xawery Zulawski: Xawery Zulawski National Film School in Łód2 399s-eeseent- Maria Strzelecka 2. Question: Who is the mother of the director of film Polish-Rae Russian War (Film)?
Are follow up questions needed here: Yes.
Follow up: Who is the director of the film Polish-Russian War (Film)?
Intermediate answer: The director of the film Folish-Russian War is Xawery Zulawski.
Follow up: Who the mother Xawery Zulawski? Intermediate answer: The mother of Xawery Zutawski is Matgorzata Braunck. So the final answer is: Rick Scott Malgorzata Braunek.
Contextl: John, Count Palatine of Neumarkt: John (Johann von Neumarre 1383 14 March 1443) was the Count Palatine of Neumarkt from 1410 to his death. The son of Rupert III of the Palatinate, he married Catherine of Pomerania
1407.
Context2: John, Count Palatine of Neumarkt: John (Johann von Pfalz- ;elmmrkeut 1383 - 14 March 1443) was the Count Palatine of Neumarkt from 1410 to his death. The son of Rupert III of the Palatinate, he married Catherine of Pomerania in
1407.
Question: Who is Catherine Of Pomerania, Countess Palatine Of Neumarkt's father-in-law.? Are follow up questions needed here: Yes.
Follow up: Who is the husband of Catherine of Pomerania, Countess Palatine of Neumarkt? Intermediate answer: The husband of Catherine of Pomerania, Countess Palatine of Neumarki is John, Count Palatine of Neumarkt.
Follow up: Who is the father of John, Count Palatine of Neumarkt? Intermediate answer: The father of John, Count Palatine of Neumarkt is Rupert III of the Palatinate. So the final answer is: Rupert III of the Palatinate.
Contextl: Crimen las tres: Crimen las tres a 1935 Argentine crime film directed and written by Luis Saslavsky. Crimen las tres. Directed by. Luis Saslavsky.
Context2: Elio Petri: The Working Class Goes to Heaven (Italian: La classe operaia in paradiso), released in the US Lulu the Tool, is 1971 political drama film directed Elio Petri. So the answer is Elio Petri.
Context3: March 20, 1995: Luis Saslavsky (April 21, 1903 March 20, 1995) was an Argentine film director, screenwriter and film producer, and one of the influential directors in the Cinema of Argentina of the classic So the answer is March 20, 1995.
Context4: Elio Petri: Final years. In 1981, Petri visited Geneva direct Arthur Millers new play The American Clock, with Marcello Mastroianni playing the lead role. Petri died of cancer 10 November 1982. He was 53 years old. Question: Which film has the director died first, Crimen Las Tres The Working Class Goes To Hcaven?
Are follow up questions needed here: Yes.
Follow up: Who the director Crimen las tres?
Intermediate answer: The director of Crimen a las tres Luis Saslavsky. Follow up: Who the director The Working Class Goes to Heaven? Intermediate answer: The director of The Working Class Goes to Heaven Elio Petri Follow up: When did Luis Saslavsky die?
Intermediate answer: Luis Saslavsky died on March 20, 1995. Follow up: When Elio Petri die?
Intermediate answer: Elio Petri died 10 Novemher 1982. So the final answer is: The Working Class Goes to Heaven.
Figure 13: Instruction and exemplars for the 2WIKIMQA decomposition prompt.

---

Given the following question, answer it by providing follow up questions and intermediate answers. For each follow up question, you are given context which is the top returned google snippet for the question from Wikipedia. If no follow up questions necessary, answer the question directly.
Contextl: Frost:
Frost is thin layer of ice on a solid surface, which forms from water vapor in an above- atmosphere coming in contact with a solid surface whose
Context2: Graduation: Graduation is the awarding of a diploma to a student by an educational institution. It may also refer to the ceremony that is associated with it.
Context3: Winter: Winter; Astronomical season, 22 December 21 March; ; Meteorological season, December 28/29
February: Solar (Celtic) season. November - 31 January.
Question: Is it common see frost during some college commencements?
Are follow up questions needed here: Yes.
Follow up: What scasons can you expect to see frost?
Intermediate answer: Frost is common during the winter. Follow up: When is college commencement?
Intermediate answer: College commencement ceremonies often happen during the months of December, May, June. Follow up: Do any of the months December, May, June occur during the Winter? Intermediate answer: December in the winter. So the final answer is: Yes.
Contextl: Last rites: The last rites, also known as the Commendation of the Dying, the last prayers and ministrations given individual Christian faith, when possible, shortly before death. They may be administered to those awaiting execution, mortally injured, or terminally ill.
Context2: Richard Dawkins: Dawkins is an outspoken atheist and a supporter of various atheist, secular, humanist organisations, including Humanists UK and the Brights movement. Dawkins suggests that atheists should be proud, not apologetic, stressing that atheism evidence of a healthy, independent mind. Context3: Prayer in Catholic Church: In Catholie Church, prayer is "the raising of ones mind and heart to God or the requesting of good things from God." is an act the moral virtue Question: Would Richard Dawkins hy hypothetically refuse an offering of the Last rites?
Are follow up questions needed here: Yes. Follow up: What are the last Rites?
Intermediate answer: The Last rites, in Catholicism, are the last prayers and ministrations given to an individual the faith, when possible, shortly before death. Follow up: What are Richard Dawkins religious beliefs?
Intermediate answer: Richard Dawkins is known as an outspoken atheist, well known for his criticism of creationism and intelligent design.
Follow up: Would an atheist participate in Catholics prayers?
Intermediate answer: is unlikely that an atheist would participate in Catholics prayers. So the final answer is: Yes.
Contextl: number I: Hydrogen is the chemical element with the symbol H and atomic number 1, Hydrogen is the lightest element. So the answer is number 1.
Context2: Spice Girls Simple English Wikipedia, the free encyclopedia: The group has five members. Each member uses nickname initillly given to them: Melanie Chisholm ("Sporty Spice"). Emma Bunton ("Baby Spice"). Melanie Brown ('Scary Spice"), Victoria Beckham (née Adams) ("Posh Spice"), and Geri Halliwell ("Ginger Spice"). Context3: Square number: In mathematics. square number perfect square is an integer that is the square of an integer: in other words, it is the product of some integer with..
Question: Hydrogen's atomic number squared exceeds number of Spice Girls? Are follow up questions needed here: Yes.
Follow up: What is the atomic number of hydrogen? Intermediate answer: Hydrogen has an atomic numher of 1. Follow up: How many people are in the Spice Girls band? Intermediate answer: The Spice Girls has members. Follow up: the square of greater than 5? Intermediate answer: The square of is which is less than 5. So the final answer is: No.
Figure 14: Instruction and exemplars for the STRATEGYQA QA decomposition prompt.

---

Given a question and a context, answer the question and explain why. you are unsure, answer Unknown.
#
Context:
Who is the wife of Douglas Leiterman? The wife of Douglas Leiterman is Beryl Fox.
Where was Beryl Fox born? Beryl Fox was born in Winnipeg, Manitoba.
When and where was Mary born? Mary was born in c. 18 BC or September 8 (21), 16 BC Herodian Kingdom of Judca.
Where was Beryl Fox born? Beryl Fox was born in Winnipeg, Manitoba,
Who is the wife of Douglas Leiterman? The wife of Douglas Leiterman is Mary.
Who is the wife of Douglas Leiterman? The wife of Douglas Leiterman is Beryl Fox.
Question:
Where was the wife of Douglas Leiterman born?
Answer:
The wife of Douglas Leiterman is Beryl Fox. Beryl Fox was born in Winnipeg, Manitoba. So the answer is: Winnipeg, Manitoba.
Context:
Who is Beatrice of Aragon's 's father? The father of Beatrice of Aragon King Ferdinand of Naples.
Who is the father of Rupert III, Elector Palatine? The father of Rupert III, Elector Palatine is Rupert II, Elector Palatine. Who is the husband of Catherine of Pomerania? The husband of Catherine of Pomerania John II, Count Palatine of Neumarkt.
Who is Catherine Of Pomerania, Countess Palatine Of Neumarkt's husband? The husband of Catherine Of Pomerania, Countess Palatine Of Neumarkt is John I. Count Palatine of Neumarkt. Who the father John II, Count of Holstein- RenhsbbrrR The father of John II, Count of Malstein-Rendsheeee Henry II. Count Goolstein-Rendhheee
Who is Catherine Of Pomerania, Countess Palatine Of Neumarkt's husband? The husband of Catherine Of Pomerania, Countess Palatine Neumarkt is John Count of Holstein-Rendsbeen Who is the father of John Count Palatine Neumarkt? The father of John I, Count Palatine of Neumarkt Rupert III, Elector Palatine. Who are the parents of Rupert III, Elector Palatine? The parents Rupert III, Elector Palatine are Rupert II, Elector Palatine and Beatrice Aragon. Who is the father of John II, Count Palatine Neumarkt? The father John Count Palatine Neumarkt is Rupert III, Elector Palatine.
Question:
Who is Catherine Of Pomerania, Countess Palatine Of Neumarkt's father-in-lawe
Answer:
The husband of Catherine Of Pomerania, Countess Palatine Of Neumarkt is John I, Count Palatine of Neumarkt. The father of John I, Count Palatine of Neumarkt is Rupert III, Elector Palatine. So the answer is: Rupert III, Elector Palatine.
Context:
When did Elio Petri die? Elio Petri died on 10 November 1982,
Who is the director of The Working Class Goes to Hcaven? The director of The Working Class Goes to Hcaven is Elio Petri. Who is the director of Crimen A Las Tres? The director of Crimen A Las Tres is Luis Saslavsky.
Who is the director of Crimen A Las Tres? The director of Crimen A Las Tres is Luis Saslavsky.
When did Luis Saslavsky die? Luis Saslavsky died on March 20, 1995.
Who is the director of Crimen A Las Tres? The director of Crimen A Las Tres is Luis Saslavsky.
When did Elio Petri die? Elio Petri died on 10 November 1982.
When did Luis Suslavsky die? Luis Saslavsky died on March 20, 1995.
When did Luis Saslavsky die? Luis Saslavsky died on March 20, 1995.
When did Elio Petri die? Elio Petri died on 10 November 1982.
Who is the director of The Working Class Goes Heaven? The director of The Working Class Goes to Heaven is Elio Petri.
Who is the director of The Working Class Goes Heaven? The director of The Working Class Goes to Heaven is Elio Petri.
Question:
Which film has the director died first, Crimen A Las Tres or The Working Class Goes To Heaven?
Answer:
The director of Crimen Las Tres is Luis Saslavsky. The director The Working Class Goes to Heaven Elio Petri. Luis Saslavsky died on March 20, 1995. Elio Petri died on November 1982.
So the answer is: The Working Class Goes To Hcaven.
Figure 15: Instruction and exemplars for the 2 2WIKIMQA meta-reasoner prompt.

---

Given a question and a context, provide Yes No answer and explain why. you are unsure, answer Unknown.
Context:
What type of animal is jellyfish? A jellyfish is type of invertebrate.
Do jellyfish have arteries? No. jellyfish do have arteries. They have type of nervous system called nerve net. disease where plaque builds up in the arteries.
What is atherosclerosis? Atherosclerosis
Do jellyfish have arteries? Jellyfish do not have circulatory system and therefore do not have arteries. So jellyfish don't 't have atheroselerosis because they don't ton't have arteries? Yes, that correct.
What atherosclerosis? Atherosclerosis a type of heart disease.
Is an invertebrate susceptible atheroselcosis?? No, invertehrates are not susceptible atherosclerosis.
What atherosclerosis? Atherosclerosis a disease in which plaque builds up on the walls the arteries.
Question:
Is a jellyfish safe from atherosclerosis?
Answer:
Jellyfish do not have a circulatory system and therefore do not have arteries, Atherosctcrosis is discase in which plaque builds up on the walls the arteries. So the answer is: Yes.
Context:
What is tonsure? Tonsure the practice of cutting or shaving the hair from the scalp. as sign of religious devotion or
humility.
What is Christopher Hitchens' religious affiliation? Christopher Hitchens was an atheist.
What is tonsure? Tonsure is practice of cutting or shaving the hair from the scalp.
Would Christopher Hitchens be likely to do something as sign of religious devotion? No.
Would Christopher Hitchens be religious? Christopher Hitchens was an atheist.
Is an atheist likely to engage in religious practices? No, an atheist is not likely to engage in religious practices.
What is tonsure? Tonsure the practice of cutting or shaving the hair from the scalp, as sign of religious devotion humility.
What is Christopher Hitchens' religious aftiliation? Christopher Hitchens was atheist.
Do atheists usually engage in tonsure? No, atheists do not usually engage in tonsure as it is religious practice.
Question:
Would Christopher Hitchens be very unlikely to engage in tonsure?
Answer:
Tonsure is the practice of cutting shaving the hair from the scalp, sign of religious devotion humility. Christopher Hitchens atheist. An atheist is not likely engage religious practices.
So the answer is: Yes.
Context:
What the evidence that Neanderthals used arithmetic? There is no direct evidence that Neanderthals used arithmetic, but indirect evidence suggests that they may have had some understanding basic mathematical concepts. What the carliest evidence of arithmetic? The carliest evidence of arithmetic dates back the ancient Sumerians, who lived in what now modern-day Iraq.
Did Neanderthals live before the ancient Sumerians?
What arithmetic? Arithmetic the branch mathematics that deals with the properties and manipulation numbers, Is there any reason to believe that Neanderthals could not have used arithmetic? There is no reason to believe that Neanderthals could not have used arithmetic.
we have any evidence that Neanderthals could manipulate numbers? We have no evidence that Neanderthals could manipulate numbers.
What arithmetic? Arithmetic the hranch of mathematics that deals with the properties and manipulation of numbers. What arithmetic? Arithmetic is the branch of mathematics that deals with the addition, subtraction, multiplication, and division whole numbers and with the manipulation of fractions and decimals.
Question:
Did Neanderthals use arithmetic?
Answer:
There is no evidence that Neanderthals used arithmetic. The earliest evidence of arithmetic dates back to the ancient Sumerians who lived after the Ncanderthals. So the answer is: No.
Figure 16; Instruction and exemplars for the STRATEGYQA meta-reasoner prompt.

---

Given question and a context, answer the question and explain why. you are unsure, answer Unknown.
#
Context:
Who is the wife of Douglas Leiterman? Douglas Leiterman (1927 - 19 December 2012) was Canadian television producer:
[[Douglas Leitermanj, ISpouse; Reryl Foxi]
Where was Beryl Fox born? I Winnipeg, Manitoba: Fox was born 1931 in Winnipeg, Manitoba. the answer Winnipeg, Manitoba.
Who is Douglas Leiterman's wife? Douglas Leiterman (1927- 19 December 2012) was a Canadian television producer:
[[Douglas Leitermanj, [Known for; This Hour Has Seven Days [Spouse; Beryl Fox
Where was Beryl Fox born? I Winnipeg, Manitoba: Fox was born in 1931 in Winnipeg, Manitoba. So the answer is Winnipeg, Manitoha.
Who is the wife of Douglas Leiterman? Douglas Leiterman (1927 19 December 2012) was Canadian television producer:
[[Douglas Leitermanj. [Spouse; Beryl Foxf] When and where was Mary horn? Mary was horn in c. 18 BC Septemher 8 (21), 16 BC Herodian Kingdom of Judca.
Question:
Where was wife Douglas Leiterman born?
Answer:
The wife Douglas Leiterman is Beryl Fox. Beryl Fox was born in Winnipeg, Manitoba. So the answer is: Winnipeg. Manitoba.
Context:
Who is the director Crimen A Las Tres? Crimen las tres: Crimen a las tres is a 1935 Argentine crime film directed and written by Luis Saslavsky. Who is the director of The Working Class Goes to Heaven? The Working Class Goes to Heaven: The Working Class Goes to Heaven (Italian: La classe operaia va in paradiso), released in the US as Lulu the Tool, is a 1971 political drama film directed by Elio Petri.
When did Luis Saslavsky die? Luis Saslavsky: Luis Saslavsky (April 21, 1903 - March 20, 1995) was an Argentine film director, screenwriter and film producer, and one of the influential directors in the Cinema of Argentina of the classic era. When did Elio Petri die? I Elio Petri: Petri died of cancer on 10 Novemher 1982. He was 53 years old, Who is the director of Crimen A Las Tres? Crimen las tres: Crimen a las tres is 1935 Argentine crime film directed and written by Luis Saslavsky.
Who is the director of The Working Class Goes to Heaven? The Working Class Goes to Heaven: The Working Class Goes to Heaven (Italian: La classe operaia va in paradiso), released in the US as Lulu the Tool, is a 1971 political drama film directed by Elio Petri.
When did Luis Saslavsky die? Luis Saslavsky: Luis Saslavsky (April 21, 1903 March 20, 1995) was Argentine film director, screenwriter and film producer, and one of the influential directors in the Cinema of Argentina of the classic era. When did Elio Petri die? Elio Petri: Petri died of cancer on 10 November 1982. He was 53 years old. Who is the director of Crimen A Las Tres? Crimen las tres: Crimen a las tres is 1935 Argentine crime film directed and written by Luis Saslavsky. When did Luis Saslavsky dic? I Luis Saslavsky: Luis Saslavsky (April 21, 1903 March 20, 1995) director, screenwriter and film producer, and one of the influential directors in the Cinema of Argentina of the classic era. Who is the director of The Working Class Goes to Heaven?? The Working Class Goes to Heaven: The Working Class Goes to Heaven (Italian: La classe operaia va in paradiso), released in the US as Lulu the Tool, is 1971 political drama film directed by Elio Petri.
Argentine film
When did Elio Petri die? Elio Petri: Petri died of cancer on 10 November 1982. He was 53 years old.
Question:
Which film has the director died first, Crimen A Las Tres or The Working Class Goes To Heaven?
Answer:
The director of Crimen Las Tres is Luis Saslavsky. The director The Working Class Goes to Heaven Elio Petri. Luis Saslavsky died on March 20, 1995. Elio Petri died on November 1982. So the answer is: The Working Class Goes Heaven.
Instruction and exemplars for the 2WIKIMQA meta-reasoner prompt for MCR-EV and SCR-EV reasoning over retrieved evidence.
Figure 17:

---

Given a question and a context, answer the question step-by-step. If you are unsure, answer Unknown.
Context:
What is atherosclerosis Atherosclerosis: Atherosclerosis a pattern of the disease arteriosclerosis in which the wall the artery develops abnormalities. called lesions. These lesions may lead to narrowing due to the buildup atheromatous plaque. At onset there are usually no symptoms, but they develop. symptoms generally begin around middle age. What type of animal is a jellyfish? Jellyfish- - Simple English Wikipedia, the free encyclopedia: Jellyfish are animals the phylum Cnidaria. They are monophyletic clade, the Medusozoa, Most of them live in the occans, in salt water, where they eat small sea animals like plankton and little fish, and float the sea.
Is an invertehrate susceptible to atherosclerosis? I Atherosclerosis: Atheroselerosis a pattern of the discase arteriosclerosis in which the wall of the artery develops abnormalities, called lesions.
What is atherosclerosis? Atherosclerosis, Atherosclcrosis a pattern of the discase arteriosclerosis in which the wall of the artery develops abnormalities, called lesions. These lesions may lead to narrowing due to the buildup of atheromatous plaque. At onset there are usually no symptoms, but if they develop. symptoms generally begin around middle age. Do jellyfish have arteries? Jellyfish:: Jellyfish are mainly free-swimming marine animals with lmbrella-shaped bells and trailing tentacles, although a few are anchored to the seabed by stalks rather
What is atherosclerosis? Atherosclerosis: Atherosclerosis is a pattern of the discase arteriosclerosis in which the wall of the artery develops abnormalities, called lesions. These lesions may lcad to narrowing due to the huildup of atheromatous plaque. At onset there are usually no symptoms, hut if they develop. symptoms generally begin around middle age. Do jellyfish fish have arteries? I Jellyfish: Jellyfish are mainly free-swimming marine animals with umbrella-shape shaped bells and trailing tentacles, although a few are anchored to the seabed by stalks rather So jellyfish don't 't have atherosclerosis because they don't 't have arteries? I Jellyfish: A free- rec-swwmming marine coclenterate that is the sexually reproducing form of hydrozoan or scyphozoan and has nearly transparent saucer-shaped body and
Question:
Is a jellyfish safe from atherosclerosis? Answer:
Jellytish do not have a circulatory system and therefore do not have arteries. Atheroseterosis is a discase in which plaque builds up on the walls of the arteries. So the answer is: Yes.
...
Context:
What is arithmetic? Arithmetic: Arithmetic is an elementary part of mathematics that consists of the study of the properties of the traditional operations on numhers-addition subtraction, multiplication, division, exponentiation, and extraction of roots.
What the evidence that Neanderthals used arithmetic? Neanderthal: In 2012, British-Americaa geneticist Graham Coop hypothesised that they instead found evidence of different archaic human species interbreeding with modern Is there any reason believe that Neanderthals could not have used arithmetic? Neanderthal: A large part the controversy stems from the vagueness of term "species", as it generally used distinguish two genetically isolated populations, but What is arithmetic? Arithmetic: Arithmetic is an elementary part of mathematics that consists of the study of the properties of the traditional operations on numbers addition, subtraction, multiplication, division, exponentiation, and extraction of roots.
Do we have any evidence that Neanderthals could manipulate numbers? Neanderthal: Neanderthals also written as Neandertals, are an extinct species or subspecies archaic humans who lived Eurasia until about 40,000 years ago. What is arithmetic? Neanderthal: Ncanderthals also written as Neandertals, are an extinct species or subspecies of archaic humans who lived in Eurasia until about 40,000 years ago.
What is the carliest evidence of arithmetic? Mathematics: is in Bahylonian mathematics that elementary arithmetic (addition, subtraction, multiplication, and division) first appear in the archaeological record. The Babylonians also possessed place-value system and used a sexagesimal numeral system which is still in use today for measuring angles and time. Did Neanderthals live before the ancient Babylonians? Neanderthal: Neanderthals also written as Neandertals, are an extinct species or subspecies of archaic humans who lived in Eurasia until about 40,000 years ago. Pre- and early Neanderthals, living before the Ecmian interglacial
Question:
Did Neanderthals use arithmetic?
Answer:
There is evidence that Neanderthals used arithmetic. The earliest evidence of arithmetic dates back to the ancient Babylonians who lived after the Neanderthals. So the answer is: No.
Figure 18: Instruction and exemplars for the STRATEGYQA prompt for MCR-Ev and SCR-Ev reasoning over retrieved evidence.


Reviews:
[{"review_id": "0YprUa7v1f", "reviewer": "Reviewer_aP2G", "summary": "", "questions": "1. How sensitive is MCR to the choices of prompts and few-shot examples? How long does it take for the authors to design the prompts in the appendix? And in general, will it be easy to design these prompts given a new task with little domain knowledge?\n2. Both CoT prompting and self-consistency methods can be used with a variety of reasoning chain formats. In the presented work, all the experiments are conducted with a specific style of reasoning chain (i.e., a sequence of questions, evidence, and answers). I wonder if the authors have tried other styles of reasoning chains (even if the absolute performance is worse than the reported numbers), and what are the performance trends there?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the positive comments regarding our evaluation and analysis. Please see our response below:\n\n*Q1: How sensitive is MCR to the choices of prompts and few-shot examples? How long does it take for the authors to design the prompts in the appendix? And in general, will it be easy to design these prompts given a new task with little domain knowledge?*\n\nA1. Designing the prompts did not take long, we tried only a handful of prompts and did not engage in any significant prompt engineering. For our decomposition prompts, we selected random examples from the datasets and annotated them with gold reasoning chains (intermediate questions, answers and evidence). To construct the meta-reasoning prompts, we used random examples, their decomposition-generated reasoning chains as evidence, and manually annotated them with a chain-of-thought explanation that filters the relevant evidence from the generated chains. Therefore, designing these prompts for a new task should be straightforward, given a handful of random examples and their gold decompositions.\n\nRegarding MCR’s sensitivity to the choice of prompt, we added new experiments where we randomly sampled new exemplars for both our decomposition and meta-reasoning prompts for StrategyQA and HotpotQA. When using different random exemplars, we observe that MCR **still outperforms** all baselines. Even though decomposition performance (self-ask) is more affected by the set of exemplars, the performance trend remains the same, with MCR being on top. Below are the experiment results, evaluated on 500 examples from each dataset (paper results are in parenthesis and are averaged over 3 runs with the original prompts).\n\n|   Dataset  | Number of examples | Number of seeds |     Self-Ask    |        SC@5       |       SCR       |       MCR       |\n|:----------:|:------------------:|:---------------:|:---------------:|:-----------------:|:---------------:|:---------------:|\n| StrategyQA |      500 (1000)     |      1 (3)      | 66.6 (69.3±0.3) |  71.0 (72.2±0.8)  | 68.2 (70.0±0.6) | **72.0 (73.6±0.7)** |\n| HotpotQA   |     500 (500)     |      1 (3)      | 54.3 (50.2±0.3) | 56.2 (51.3±0.2) | 57.1 (56.4±0.4) | **59.3 (57.0±0.8)** |\n\n\n*Q2. Both CoT prompting and self-consistency methods can be used with a variety of reasoning chain formats. In the presented work, all the experiments are conducted with a specific style of reasoning chain (i.e., a sequence of questions, evidence, and answers). I wonder if the authors have tried other styles of reasoning chains (even if the absolute performance is worse than the reported numbers), and what are the performance trends there?*\n\nA2. We had preliminary results using an alternative reasoning chain format: a sequence of intermediate questions and answers without any evidence. MCR **still outperformed** SC@5 on the alternative format on StrategyQA (75.5 vs 72.2); Quartz (79.8 vs 77.1); 2WikiMQA (44.2 vs 40.5). However, as our focus was on an open-domain setting, we ultimately opted to incorporate retrieved evidence as well. We view extending MCR to new formats and settings as an interesting topic to explore in future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This is a straightforward idea leading to nice improvement over self-consistency. The evaluation is comprehensive on seven different datasets, and the authors also show that the combination can achieve further improvements.\n2. This paper is well-written. It is very clear what the contribution of this paper is, and what is the similarities and differences compared to previous methods.\n3. This work contains a detailed analysis, providing useful insights behind the performance improvements. The error analysis can also be useful for future work.", "reasons_to_reject": "I have several questions to ask the author about the paper's details, but I don't see a strong reason to reject this paper.", "questions_to_authors": "1. How sensitive is MCR to the choices of prompts and few-shot examples? How long does it take for the authors to design the prompts in the appendix? And in general, will it be easy to design these prompts given a new task with little domain knowledge?\n2. Both CoT prompting and self-consistency methods can be used with a variety of reasoning chain formats. In the presented work, all the experiments are conducted with a specific style of reasoning chain (i.e., a sequence of questions, evidence, and answers). I wonder if the authors have tried other styles of reasoning chains (even if the absolute performance is worse than the reported numbers), and what are the performance trends there?", "paper_topic_and_main_contributions": "This work presents a new method for LLMs to perform reasoning. Unlike the previous self-consistency method that just takes the majority vote of multiple reasoning chains as the final answer, the proposed MCR method feeds all the questions and answers from every reasoning chain to another LLM and then uses it to generate the final answer. Following previous works, in the described system, the reasoning chain generation also involves a decomposition model and a retrieval model. The experiments are conducted on 7 different datasets. MCR consistently outperforms self-consistency or at least achieves similar results. When a large number of reasoning chains are available, a combination of MCR and SC can also lead to further improvements. Finally, this work provides a detailed analysis to demonstrate some reasons behind the success of this method, as well as an error analysis. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "s8ghfXc5O3", "reviewer": "Reviewer_reqd", "summary": "", "questions": "- How does the performance of MCR vary according to different number of reasoning chains? It would make the results more comprehensive if the authors could draw a figure showing performance vs. number of chains used by MCR. \n- How does the temperature affect the final performance?\n- The experiments in this paper are conducted only using multihop reasoning datasets. Do you think similar conclusion can be generalized to other domains such as mathematical reasoning?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer and are grateful for their appreciation of the reproducibility efforts put into this work. Regarding the reviewer questions:\n\n*Q1.How does the performance of MCR vary according to different number of reasoning chains? It would make the results more comprehensive if the authors could draw a figure showing performance vs. number of chains used by MCR.*\n\nA1. In the paper (Tables 2 and 3), we show that meta-reasoning with 5 reasoning chains (MCR) is better than with 1 chain (SCR) and that MCR can be extended to 15 reasoning chains by applying self-consistency over 3 MCR runs. We will be sure to include the suggested plot in the camera-ready version of our paper. Namely, we will plot the averaged accuracy of MCR as a function of the number reasoning chains used at different data points (n=1,3,5,...,15).\n\n\n*Q2. How does the temperature affect the final performance?*\n\nA2. We empirically observe that greedy decoded reasoning chains (temp=0) display a higher quality than those sampled with temp=0.7. For example, 64.6 vs 58.9 on Bamboogle; 78.3 vs 71.0 on Quartz; 63.8 vs 58.3 on 2WikiMQA. In our setting, we follow past work (e.g self-consistency) and sample multiple reasoning chains using temperature. For MCR we combine the greedy decoded chain with 4 chains sampled with temp=0.7 (lines 298-300). For SC, we use the same chains as MCR (lines 324-326). It could be interesting to explore other sampling methods in future work.\n\n\n*Q3.The experiments in this paper are conducted only using multihop reasoning datasets. Do you think similar conclusion can be generalized to other domains such as mathematical reasoning?*\n\nA3. Our focus in the paper was on open-domain QA, as it is a challenging setting, where there are different strategies to solve a question and different ways to leverage the retrieved evidence. We therefore opted not to experiment with mathematical reasoning datasets in this work. However, we have no reason to believe that MCR should fail on mathematical tasks, in particular those where multiple reasoning chains can be useful.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- To my knowledge, the proposed method to reason over multiple reasoning chains is novel and through comprehensive experiments, the authors demonstrate the effectiveness of the proposed method over strong baselines.\n- Although the idea is straightforward, the amount of efforts to make the whole pipeline work is substantial. Personally, I really appreciate the efforts the authors put into to ensure reproducibility of this work. I think this paper could foster future work that focuses on better utilizing multiple reasoning chains sampled from LLMs.\n- This paper is well written and presented.", "reasons_to_reject": "I don't have any specific reasons for rejecting this paper. However, I do have several questions, see below.", "questions_to_authors": "- How does the performance of MCR vary according to different number of reasoning chains? It would make the results more comprehensive if the authors could draw a figure showing performance vs. number of chains used by MCR. \n- How does the temperature affect the final performance?\n- The experiments in this paper are conducted only using multihop reasoning datasets. Do you think similar conclusion can be generalized to other domains such as mathematical reasoning?", "paper_topic_and_main_contributions": "This paper proposed Multi Chain Reasoning (MCR), an approach which prompts large language models to meta-reason over multiple chains of thought, rather than aggregate their answers. MCR examines different reasoning chains, mixes information between them and selects the most relevant facts in generating an explanation and predicting the answer. MCR generally contains two components, generating reasoning chains and reasoning over reasoning chains. The reasoning chains is generated by a question decomposition model and a retriever in a iterative way. With multiple reasoning chains generated in the last step, the authors then deploy meta-reasoner module to derive the final answer from multiple chains. All components except the retriever are based on LLMs. They experimented MCR on 7 multihop reasoning datasets and the results show that MCR outperforms all baselines, including self-consistency. They also examined the explanation quality generated by the meta reasoner and found that most of the explanations are faithful.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "CF2cdd3WG5", "reviewer": "Reviewer_hqxw", "summary": "", "questions": "1. Could you make statistics of your method and all baseline models about the entire length of the input sequence, including all the intermediate prompts, evidences, and reasoning chains? \n2. What is the performance of the retriever in retrieving evidence facts? If the retriever fails to retrieve relevant evidences, will it heavily hurt the final QA performance?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive comment regarding the motivation of the paper. We will now address all relevant points and questions:\n\n\n*P1. The method is not novel. Based on my understanding, you just prompt the LLM with a second-round CoT, additionally including the multiple reasoning chains sampled from the first-round CoT. It is more like an engineering practice. At least, I did not find this work to be exciting.*\n\n\nP1.A. Leveraging multiple CoTs is an active area of research [1,2], and to our knowledge meta-reasoning is **the first method to use a LLM to review multiple reasoning strategies**. Aside from aggregating multiple CoTs, meta-reasoning is closely tied to self-reflection in LLMs, which is becoming increasingly important [3,4] (contemporaneous work). While it is discussed in our related work (541-546), we will be sure to further emphasize this in the camera-ready version. Second, while the idea of meta-reasoning is simple, there is merit in its simplicity, as it is a general solution that is applicable to multiple combinations of LLMs, retrievers and QA tasks. \n\n[1] Self-Consistency Improves Chain of Thought Reasoning in Language Model;\n[2] Tree of Thoughts: Deliberate Problem Solving with Large Language Models;\n[3] Reflexion: an autonomous agent with dynamic memory and self-reflection;\n[4] Self-refine: Iterative refinement with self-feedback\n\n\n*P2. The improvements compared to baselines are marginal.*\n\n\nP2.A. Our approach demonstrates **consistent improvements on 7 different QA tasks**, when compared to strong baselines (self-ask, self-consistency). These gains are maintained across different retrievers (ColBERTv2, Google) and LLMs (code-davinci-002, Vicuna-13B), as shown in Tables 2-4, 7-8. Furthermore, MCR generates quality explanations, which serve as an additional advantage over self-consistency. We believe that these gains demonstrate the advantage of MCR, compared to existing approaches.\n\n\n*P3. Based on my understanding, the method is computationally intensive and not efficient. The input text is very long and it needs to trigger LLMs twice, one for CoT and one for meta-reasoning.*\n\n\nP3.A. In the paper, we compare MCR to self-consistency (SC) and show, in lines 366-8, that it is in fact **more efficient** than SC, achieving higher performance (on 6/7 tasks) while using much fewer LLM calls (sampling 5 chains + 1 call to the meta-reasoner for MCR vs sampling 15 chains for SC). We consider SC to be a natural baseline as it is a standard approach for CoT which involves multiple calls to the LLM and has been shown to boost QA performance [1,2,3]. For a comparison of LLM input lengths between MCR and SC see A1 below.\n\n[1] Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP;\n[2] Faithful Chain-of-Thought Reasoning;\n[3] Large Language Models are Zero-Shot Reasoners\n\n\n*Q1. Could you make statistics of your method and all baseline models about the entire length of the input sequence, including all the intermediate prompts, evidences, and reasoning chains?*\n\n\nA1. We provide below all the relevant statistics that were requested: input sequence lengths including all the intermediate prompts, evidence, and reasoning chains. Stats are provided for our SA baseline (also used by SC and MCR), as well as for the meta-reasoning models. We would be happy to include these in our camera-ready version. We will also make our prompts, data, and code publicly available as mentioned in the paper. A few notes:\n- The table below describes the number of tokens for code-davinci-002, matching Table 2.\n- As requested, we provide prompt lengths for all of our baselines: SA, SCR and MCR.\n- Generating a single reasoning chain (i.e. CoT prompting) requires multiple LLM calls, one for each decomposition step. This means that a single decomposition generation is generally longer than applying our additional meta-reasoning step. For more details on our method’s efficiency see P2.A above.\n\n|   Dataset  | SA prompt length | Number of decomposition steps | SA Inference length (avg, std) | Retrieved evidence length (avg, std) | SCR/MCR prompt length (no decompositions) | SCR prompt length (with decompositions) (avg, std) | MCR prompt length (with decompositions) (avg, std) | SCR inference length (avg, std) | MCR inference length (avg, std) |\n|:----------:|:----------------:|:-----------------------------:|:------------------------------:|:------------------------------------:|:-----------------------------------------:|:--------------------------------------------------:|:--------------------------------------------------:|:-------------------------------:|:-------------------------------:|\n| StrategyQA |       2242       |            2.9±0.6            |           103.3± 48.0          |              190.6±66.7              |                    1652                   |                    1749.0± 52.0                    |                    2032.9±110.7                    |            41.9±19.7            |            47.4±20.8            |\n| Fermi      |       1442       |            2.3±0.9            |            91.4±31.9           |              165.8±78.9              |                    1681                   |                     1765.5±25.5                    |                     1984.1±79.6                    |            42.6±17.0            |            46.6±18.1            |\n| Quartz     |        839       |            1.2±0.5            |            55.2±20.6           |               92.7±28.7              |                    2129                   |                     2202.4±19.8                    |                     2343.6±48.3                    |             26.1±7.4            |            36.2±18.1            |\n| HotpotQA   |       2508       |            1.7±0.7            |            86.1±91.9           |              153.0±84.1              |                    2380                   |                     2460.3±28.1                    |                     2666.2±90.0                    |            36.9±12.9            |            39.4±13.2            |\n| 2WikiHop   |       1920       |            2.4±0.8            |            92.7±30.5           |              201.3±59.3              |                    2029                   |                     2116.4±25.6                    |                    2363.0±104.6                    |            46.7±14.8            |            46.9±14.1            |\n| Bamboogle  |       1342       |            2.0±0.3            |            74.3±37.5           |              204.5±72.1              |                    966                    |                     1035.1±12.9                    |                     1223.1±50.2                    |             36.3±7.4            |            37.5±10.6            |\n| Feverous   |       3741       |            2.9±0.9            |           118.2±36.4           |              197.1±69.2              |                    2826                   |                     2956.8±38.1                    |                    3276.8±123.1                    |             14.6±9.6            |             15.4±9.2            |\n| Average    |   2004.9±882.1   |            2.2±0.6            |            88.7±18.7           |              172.1±37.0              |                1951.9±550.3               |                    2040.8±563.1                    |                    2270.0±587.7                   |            35.0±10.3            |            38.5±10.4            |\n\n\n*Q2. What is the performance of the retriever in retrieving evidence facts? If the retriever fails to retrieve relevant evidences, will it heavily hurt the final QA performance?*\n\n\nA2. In the error analysis (Table 5) we list the percentage of errors due to the retriever, for each dataset. Furthermore, we compare the performance of code-davinci-002 with two separate retrievers in Tables 2 and Table 7 (in the appendix). For both of these retrievers, MCR outperforms all other baselines. While the choice of retriever affects decomposition performance, it does so for all of our baselines (SA, SC and MCR). Overall, while retriever performance is important, it is orthogonal to our paper’s contribution which is the meta-reasoning approach.\n\nTo explicitly address the reviewer’s questions we performed a manual analysis of 50 random examples from 3 of our datasets, which we would add to our camera-ready version. We reviewed the greedy reasoning chain generated by our retrieval augmented decomposition model. For each chain, we labeled whether or not it contained all of the gold evidence for answering the multi-hop question. This analysis also suggests that good retrieval is necessary to achieve high accuracy.\n\n| Dataset (50 random examples) | Google retriever accuracy  (retrieved all gold evidence, SA greedy) | SA acc. where retriever is correct / incorrect |\n|:----------------------------:|:-------------------------------------------------------------------:|:----------------------------------------------:|\n| Bamboogle                    |                                 68%                                 |                  80.6% / 29.1%                 |\n| 2WikiMQA                     |                                 78%                                 |                  70.4% / 28.8%                 |\n| StrategyQA                   |                                 58%                                 |                  72.4% / 42.9%                 |\n\n\nNote that for StrategyQA retrieval is more challenging, as many questions involve commonsense reasoning which cannot be directly inferred from the retrieved evidence (e.g. the answer to the question “Can jackfruit be used as a weapon?” is “Yes” because it is covered with spikes, however, the retriever fails to retrieve this evidence).\n\n\n---------------\n\nIn conclusion, we have addressed all of the issues raised by the reviewer and will discuss them in our camera-ready version. We are hopeful that, given these clarifications, the reviewer would be willing to raise their Soundness score (and potentially find the work to be more exciting), in line with the other reviewers.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The motivation of this paper is clear. That is, a second-round of reasoning to pick the most relevant evidences from multiple reasoning chains.", "reasons_to_reject": "1. The method is not novel. Based on my understanding, you just prompt the LLM with a second-round CoT, additionally including the multiple reasoning chains sampled from the first-round CoT. It is more like an engineering practice. At least, I did not find this work to be exciting.\n2. The improvements compared to baselines are marginal.\n3. Based on my understanding, the method is computationally intensive and not efficient. The input text is very long and it needs to trigger LLMs twice, one for CoT and one for meta-reasoning.", "questions_to_authors": "1. Could you make statistics of your method and all baseline models about the entire length of the input sequence, including all the intermediate prompts, evidences, and reasoning chains? \n2. What is the performance of the retriever in retrieving evidence facts? If the retriever fails to retrieve relevant evidences, will it heavily hurt the final QA performance?", "paper_topic_and_main_contributions": "This paper proposes a meta-reasoning framework to improve the reasoning capability of LLM on multi-hop QA tasks. The meta-reasoning reasons among multiple reasoning chains generated by CoT prompting. Since the reasoning chains generated by CoT method may provide some hallucinated evidences that cause an incorrect answer, the proposed meta-reasoning is capable to pick up the correct evidences from multiple reasoning chains to make a reliable answer generation. To realize meta-reasoning, the reasoning chains for a question generated from an LLM are fed into another LLM to conduct another round of CoT reasoning before generating the answer.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper improves upon prior work that aggregates over multiple chain-of-thought outputs to generate a final answer. The idea is to use all reasoning chains and have the LLM use all of them as context to predict the answer. Experiments on seven datasets confirm the effectiveness of this approach. The reviewers found the paper convincing and well-written.
Decision: Accept-Main