Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoning
Authors: Antonia Creswell
Murray Shanahan
Irina Higgins
Venue: ICLR 2023 notable top 5% (2023)
Keywords: System 2
Logical reasoning
Language Models
Large Language Models
Reasoning
Neuro-symbolic
Neural Symbolic
Interpretability

Abstract: Large language models (LLMs) have been shown to be capable of impressive few-shot generalisation to new tasks. However, they still tend to perform poorly on multi-step logical reasoning problems. Here we carry out a comprehensive evaluation of LLMs on 46 tasks that probe different aspects of logical reasoning. We show that language models tend to perform fairly well at single step inference or entailment tasks, but struggle to chain together multiple reasoning steps to solve more complex problems. In light of this, we propose a Selection-Inference (SI) framework that exploits pre-trained LLMs as general processing modules, and alternates between selection and inference to generate a series of interpretable, casual reasoning steps leading to the final answer. We show that a 7B parameter LLM used within the SI framework in a 5-shot generalisation setting, with no fine-tuning, yields a performance improvement of over 100% compared to an equivalent vanilla baseline on a suite of 10 logical reasoning tasks. The same model in the same setting even outperforms a significantly larger 280B parameter baseline on the same suite of tasks. Moreover, answers produced by the SI framework are accompanied by a causal natural-language-based reasoning trace, which has important implications for the safety and trustworthiness of the system.

Published as a conference paper at ICLR 2023
SELECTION-INFERENCE EXPLOITING LARGE LAN- GUAGE MODELS FOR INTERPRETABLE LOGICAL REA- SONING
Antonia Creswell, Murray Shannahan Irina Higgins DeepMind, London, UK
(tonicreswell, mshanahan, irinah) Jdeeeedddeedcoooooo
ABSTRACT
Large language models (LLMs) have been shown to be capable of impressive few- shot generalisation to new tasks. However, they still tend perform poorly multi-step logical reasoning problems. Here we carry out comprehensive evalua- tion of LLMs on 46 tasks that probe different aspects of logical reasoning. We show that language models tend to perform fairly well at single step inference or entail- ment tasks, but struggle to chain together multiple reasoning steps solve more complex problems. In light of this, we propose a that exploits pre-trained LLMS as general processing modules, and alternates be- tween selection and inference to generate series of interpretable, casual reasoning steps leading to the final answer. Focusing on sub-set of 10 reasoning tasks from ProofWriter and bAbl, we show that 7B parameter, decoder-only LLM used within the framework in a 5-shot generalisation setting, setting.with no fine-tuning, yields a performance improvement of over 100% compared to an equivalent Vanilla baseline. The same model in the same setting even outperforms a significantly larger 280B parameter baseline the same suite of tasks. Moreover, answers produced by the SI framework are accompanied by a causal natural-language-bar bassed reasoning trace, which has important implications for the safety and trustworthiness of the system.
Inttrion--Ieeeeee (SI) framework
INTRODUCTION
Large language models (LLMs) are powerful few-shot learners (Bommasani et al. 2021 Brown et al. 2020: Lu et al. 2022). However, one area where they tend to perform poorly is logical reasoning (Rae et al. 2021). Yet the ability to perform multi-step. logically valid reasoning is fundamental for the discovery of new knowledge and explainability. It underpins many advancements that have been made in science, medicine, maths and philosophy. It also one of the most valued strengths of classical, symbolic AI over contemporary deep learning methods (Marcus & Davis] 2019) Marcus (0200) Bengio al. [021), ), prompting the recent increase the use neurosymbolic approaches bridge this gap [Garnelo & Shanahan, [019 Garcez & Lamb, (020). Here we propose Dnnectton-Inference (SI) framework that takes inspiration from the neurosymbolic literature improve the ability of LLMs
to do logically valid reasoning.
There are many flavours of neurosymbolic models (Garcez & Lamb 2020). Those from which we draw inspiration tend to have modular structure. where each module specialised for one type of operation (Mao et al. 2019 Andreas et al. (016)). For example, such modules may be neural networks or hand-crafted functions designed to attend to single object, or to compare the location or size of two inputs (Andreas et al. (2066] Yi et al. 2018). Neurosymbolic models can produce an answer to complex query by chaining these operations together, passing inputs from one module to another. This has the benefit of producing an interpretable trace of intermediate computations, in contrast to the "black-box" computations common end-to-end deep learning approaches. Importantly, the modularity of neurosymbolic methods allows them to generalise to significantly harder problems that require long chains of reasoning (Hudson & Manning 2019). However, the hand-crafted and

---

Published as a conference paper at ICLR 2023
Thought
Prool Witer (lorative)
Ertalinent Bank
Delecloniinleence
4,1
8.1
Figure I: Schematic comparison between Selection-Inference and other representative approaches for reasoning in natural language: Chain-of Thought (Wei et al. 2022). ProofWriter (Tafford et al.) 202I), and EntailmentBank (Dalvi et al. (02T). C context, Q question, A answer, S selection, I- inference. Grey circles- givens, white circles- model output. Loops indicate multi-step reasoning. The order of letters in a single circle indicates the order in which the corresponding steps are output by the model.
specialised nature of the modules often makes the resulting systems brittle and more difficult to extend to new domains al. 2018). We hope to address this limitation by leveraging LLMs in our modules.
Building on other modular Tatjord et al. 2021 Andreas et al. 2016) and step-wise approaches (Wei al. 2022 Dalvi et al. 2021). we propose SI which decomposes logical reasoning into two modular stages: 1) selection, which involves choosing a subset of relevant information sufficient to make a single step inference, and 2) inference, which only sees the limited information provided by the selection module, and uses it to infer new intermediate piece of evidence on the way to the final answer (see We implement both stages using pre-trained LLMs which, thanks their powerful few-shot generalisation capabilities, serve as more general alternatives to the hand aund-crafted, specialised modules typically used in neurosymbolic approaches. inference are chained together to produce a sequence of reasoning steps. As well as underpinning better performance on reasoning problems, this yields an interpretable trace that justifies the final answer.
Fig. D.
the SI framework, multiple steps of selection and
Furthermore, the reasoning trace produced by our system is causal, in the sense that each step follows from, and depends on, the previous step. Each inference step is made in isolation, based solely on the limited information provided by the Selection module, without direct access to the question or to previous steps of reasoning. This contrasts with the more common approach of obtaining post-hoc rationalisation, where the answer produced by the model has no direct dependence since the explanation is produced either in parallel to the answer or after the fact (Saha et al.. 2020 Lampinen et al. 2022 Cobbe et al.) (021). A notable example that sits in the grey area between post-hoc rationalisation approaches and the more causal explanation approaches is Chain-Of-Thousee (COT) (Wei al. (002)) (see Fig. [). In this approach LLMs are encouraged produce reasoning trace before the answer. However the dependence of the answer on the reasoning is not explicitly encouraged to be causal (as defined above). Indeed, the authors show that while the COT explanations help boost the final answer accuracy, the reasoning traces produced by the model are often wrong even when the final answer is correct (see the appendices Wei et al. (2022) for examples) and the model is prone to making up facts (Figure (Figure[A6). Developing system that can demonstrate how it reaches its answers using a causal reasoning trace has important benefits in terms of safety, explainability, interpretability, debugging, and trust. In this paper we make the following contributions:
the explanation,
1. We provide a comprehensive evaluation of LLMs on a set of 46 tasks probing different aspects of logical reasoning, and show that LLMs are good at simpler single step logical inference in 5-shot generalisation settings, but struggle with harder problems (Sec.
2. We introduce the Selection Infrrennce (SI) framework, a modular, iterative approach to solving reasoning problems (Sec. 4..
3. We demonstrate the utility of the SI framework by evaluating a 7B parameter, decoder-only, LLM on 10 logical reasoning tasks, showing overall that almost triples the performance of the same model used naively and almost doubles the performance of the same model used in the COT framework. Moreover, it often outperforms 40x larger 280B, decoder-only, LLM baseline used both naively and in the COT framework.
4. We illustrate further benefits of the SI framework in terms of the causal and interpretable reasoning traces produced (Sec. 5). These traces can help humans understand how the model reached its final answer, which is useful for debugging and opens the system's 's decisions to human critique.
I Language model references throughout the paper are removed to preserve anonymity. These will he added back if the paperin is accepted for publication. Note that use decoder-only language models.
2

---

Published as a conference paper at ICLR 2023
Contect:
wo'ree are straid mize
shoop are afraid valves
Context:
Julus 1s owan
ailla is gray
Context:
joluss gray
Juliue ewan
Context:
bornhard
bomhaadd
pemmaadd la awan
wintra le
brian lonn
porn yallom
cats are afroid of shoopp
progie lon
brianie
jessica a carl
brien white
greg
yollow grog rhino
germuude is shoep
Bl as frog
Questions in what umlly atraid
gray
By
Question:
greg?
julus a thino
we straid mice Inforince: cmill ofrald mnice
Selections
and
knnw that brian white
Eons are often white
Inderence: vos inferned that
Selection: brian allon We
Selection: juflus is swan. We know
Quostion:
Inference: we inferred that ewans are oitten gray
thus julius gray
color
Seleotion: jullus no. Wi know that greg a mino and greg green Inference jullus green
Question
what color juliss?
Selection: greg is
Selection: britn to fionn We know
that
that brian white
siiviee
Inference inferned that llona oftenn
Inferences greg wiite
(a) Correct reasoning bAbl deduction (left) and in- duction (right) tasks.
(h) ST can recover from error (left) and justify ambiguous answer with reasoning trace (right).
Figure 2: Qualitative results from the Selection-Inference (SI) model on bAbl tasks.
2 RELATED WORK
Our Lentrion--nfeencee framework sits at the intersection of classical, symbolic AI and deep learning. typical symbolic AI system might consist of a knowledge base, which is typically hand-curates by experts, and an inference engine that allows the system to perform logic- reasoning over its knowledge base. One of the primary benefits of symbolic Al systems over deep learning models is their interpretability; we can look at the reasoning steps such a system has taken to see how the final conclusion was reached. However, unlike deep learning approaches, symbolic Al systems require knowledge to be hand-crafted and are in general hard to scale. Although some approaches have attempted to bridge the gap between deep learning and symbolic AI by converting problems into formal logic (Nye al. 2021) and using existing solvers to help produce answer, this process
be brittle
and tends not to scale well. Alternatively, Mao Maa et al. (2019): Yi al. (2018)[Gupta (Gupta et al. (2019): : [Hudson] & Manning (2019) propose models that bridge this gap by combining the best parts of deep learning learning knowledge from data - and symbolic Al - producing an interpretable reasoning trace.
Recent work has attempted adapt large pre-trained language models, LLMs, the task of logical reasoning. At a high level these can be split into three groups: 1) approaches that try to fine-tune LLMs to produce the final answer directly, keeping reasoning implicit (Betz et al. 2021 Clark et al. 2021); 2) approaches that encourage LLMs to produce reasoning explicitly but all reasoning steps are produced
in one generative step (Nye al., 2022 Zelikman et al., 2022 Jhamtani & Clark 2020 Dalvi et al.
2021 Wei et al. 2022 Cobbe et al. (021). ) with notable examples including
and Chain
of Thought; and 3) approaches that use LLMs to produce each reasoning step one at a time (Tafjord et al. 2021; Bostrom et al. 2022) (e.g. PriefWriter). The latter is where our selection-Infernne framework sits (see Fig. D.. In general it was found that the approaches that incorporate explicit reasoning work better than those that only try predict the final answer. However, although explicit reasoning helps improve the accuracy of the models, encouraging the models to produce multiple steps of reasoning in single generative pass is not enough make the models manner. The authors found that the generated reasoning traces often contain unrelated or incorrect steps while still resulting in the correct answer (see examples in the appendices of Zelikman et al. (2022) and Wei et al. 2022) and Sec. J. CoT is also prone to making up facts while reasoning (Figure A6). Encouraging LLMs to generate each reasoning step one at time, for example ProofWriter (Tafford et al., 2021 Bostrom et al.) (002), is currently the most promising direction for achieving causal reasoning, and it the approach we take our paper. While ProofWrites is very impressive, unlike our approach it only works for "Prove this statement to be True/False" style questions, since it relies on enumerating all possible implications and checking whether the question statement or its negation are present in the inferred facts, which is also computationally expensive.
reasoning in causal
How WELL Do LARGE LANGUAGE MODELS REASON?
Past work has shown that LLMs are poor at logical reasoning (Rae et al. 2021 however the evaluation was done on relatively small set of tasks, and was not systematic. In particular, here interested in
3

---

Published as a conference paper at ICLR 2023
Models 10
22.3
27.5
85.0
22.9
liglc
2008
23
23
Driar ment
Single step Reasoning Type
Hultgre
Log Parambber
(b)
Figure 3: Vanilla language models perform poorly on multi-step logical reasoning tasks. (a) Scaling laws for natural language tasks (bigbench, dark purple line, squares, 56 tasks) and tasks involving logical reasoning (logic, light purple line, circles, 38 tasks). All accuracy results are calculated relative to the random baseline (0% accuracy means chance level). Only multi-choice tasks are used. (b) I Language models perform well for simple entailment tasks (AAC tasks, Entailed Polarity), their performance starts to get worse on single step inference problems (bAbl task 1, ProofWriter tasks L),
and they struggle with more complex multi-step reasoning problems (2WikiMultiHop tasks, bAbl tasks 2-3, PritfWriter tasks 2-5, StrategyQAN A detailed per-task breakdown is shown in Figure A5
1) how LLMs perform on simple entailment tasks compared to multi-step reasoning problems and 2) how scaling laws apply to logical reasoning. To this end, we evaluated decoder-only L.LMs of various sizes in 5-
following the same protocol use for the BigBench evaluation Rae et al. (2021). ), on larger set of 46 tasks. These tasks touch on different aspects of logical reasoning and vary in terms of the number of reasoning steps required. presence or absence of negation, whether the relevant context in- formation was provided, and whether the model is required to evaluate the accuracy of multiple choices or generate the answer among others. The additional tasks were collected from six sources: bAbl (Weston et al. 2016), BigBench (hazzal al. 2017), AAC (Betz [2021) Jeopardy [Tnguz 2019).
ProofWriter TTafoord et al. 2021 and 2WikiMultiH (Ho et al. 2020) (see Fig. A5a for raw results).
aspects of logical reasoning (Fig. 3b). In particular, they appear to be good at simple entailment and implication tasks (e.g. see AAC tasks and Entailed Polarity in Fig. ASa)) This appears to hold when negation is present (AAC Split Extended tasks), and both in generative (AAC Split) and multiple-choice scoring settings (AAC Split Extended tasks, Entailed Polarity). However, the performance of vanilla language models tends to decrease when they are presented with irrelevant facts alongside the ones relevant for reasoning (e.g. see 2WikiMultiHop With Context tasks, bAbl tasks 2-3 ProofWriter tasks), when they have infer the relevant facts from memory (e.g. 2WikiMultiHop or StrategyQA QA tasks), and as the questions start to require more steps of reasoning (e.g. see the performance drop between bAbl tasks 1-3 or ProofWriter Tasks).
Our analysis found that L.MSs are good
Our results confirmed that LLMs of larger sizes do perform better than the smaller models. However, we found that even the largest 280B model performed only 13.6% above chance level on average across the 38 available multi-choice logic tasks (see Figs. A5a A5bland Sec. for more details). Furthermore, we found that logical reasoning tasks were qualitatively different from other natural language tasks. The scaling law for logic- based tasks significantly worse than for other language tasks measured here as the average performance on the subset of BigBench tasks described in Sec. H This subset included no logic tasks (see Fig.
THE
(SI) FRAMEWORK
We are interested in solving logical reasoning problems expressed in natural language. In this work we assume that each question is accompanied by context information (see Fig. 2), which contains
2 We chose 5-shot setting because Min et al. 2022 have demonstrated that additional shots beyond 5 result in limited increase in multi-choice accuracy
4

---

Published as a conference paper at ICLR 2023
all the information necessary solve the problem, as well potentially irrelevant distractors. In the future this assumption can be relaxed, for example by extracting the necessary information through
search (Lazaridou et al. 2022 Menick et al. 2022). We also assume that all questions are well posed and definitively answerable given the context.
Logical reasoning problems require using existing information infer new relevant knowledge necessary to answer the question. This can be done through deduction, induction or abduction. although the datasets we use here contain mostly deductive and small number of inductive problems Some problems require multiple steps of inference, where later steps use the knowledge inferred in the earlier steps. Hence, we use an iterative framework where at each step the SI uses information in the existing context, C₁ to infer a new fact, ft. which is appended back to the context to create new context, C++1 = C. U ft- This process can then iterate until the solution to the question is found. In the current implementation of the SI framework, we repeat the process for fixed number of steps and take the final inference be the answer. Addressing the issue of halting is left for future work.
Inspired by neurosymbolic methods, we additionally split each step of reasoning into further two compo- nents: 1) Selection, which selects a subset of the information present the context, 8t, given the context and the question, C* Jg. This selection, s² is fed to the next step, 2) inference, which produces the new fact. ft, based on the information passed to it by the selection step (Fig. T. Examples of selection and inference are shown in Fig. 2 This splitting of each step of reasoning into selection and inference is the main contribution of our paper, and is important for several reasons. First, and most importantly, it makes the resulting reasoning causal, since both steps have limited capabilities by design, and inter- dependent. The selection step is constrained to only use the information available in the context (similar Gopttt et al. (2022) in the tabular setting), and the inference step only sees the subset of facts provided by the selection without access to the question. Hence, the model is unlikely to make up information to answer the question. and it cannot ignore reasoning when producing the final answer. The other benefit of this approach is that each step of reasoning is broken down into even smaller sub-tasks, which are easier for L.Mss to adapt to, and which helps make the reasoning more generalisable to harder problems.
In this paper we use pre-trained, frozen language models in a 5-shot generalisation setting using prompt engineering to implement the Selection and Inference modules. We settled on prompt engineering to evaluate the base utility of the SI framework, however it can also be used in the fine-tuning setting which We next describe the Selection and Inference modules in more detail.
we explore briefly in Sec. 6
4.1 SELECTION MODULE
We use prompt engineering encourage the model to output the correct selection, s.. The n-shot prompt is a string of the following form:
n-shot prompt
First example.
<context 1> \n kquestion 1>
# Example selection
<tact>. We know that <fact>1 and <fact>]*, Theretore,
Problem to solve. <context> \n <question>
where <fact>s are copied directly from the context, and [ and <fact>] means that the module is allowed to select more than one fact for each step of inference, where the total number of facts is cyyper-parameter
The simplest option to implement the selection is to feed this prompt directly to a pre-trainee LLM and take the output generated by the language model. However, this unconstrained approach may allow the model to make up facts, thus removing the causal aspect of the reasoning trace. Indeed during experimentation this is what we often found. So instead use the pre-trained I.LM to score each of the facts in the context, and select the one with the highest liceliool We can then repeat this process by appending each new fact the end of the previous prompt until the full selection is constructed. Note that for now we halt after fixed number of steps. See Algorithm 2 for more details.
'See Fig. for an example of deduction and induction problems used in this paper.
5

---

Published as a conference paper at ICLR 2023
Algorithm 1 "election-lnferenec
Require: An n-shot selection prompt, Pselect-
Require: An n-shot inference prompt, Pinfer.
Require:
Initial Context, C", made up of statements e.g. facts and rules.
Require:
The question, q.
Require: Language model, I.M.M
Require: The number of reascning steps, H. t=0 0
Start step O.
while <<H H do
s < Selection Module(Padedd C .q.LIM)
selection.
Inference Module (Pinfers))))))
D Do inference.
-CU
Add the newly inferred fact to the context. P Move onto the next step of reasoning
t+1+1
end while return s¹
4.2 INFERENCE MODULE
The n-shot prompt for the Inference module has the following form (shown below):
#n-shot inference prompt
# First example.
<fact>. We know that <fact>| and <fact>]*, Therefore, <new fact>.
# Problem selve. <output of the Selection module>, Therefore,
The n-shot prompt and the output of the Selection module, are fed to the pre-trained LLM serving as the Inference module. The first generated sentence is taken to be the newly inferred fact. This fact added to the context, which concludes one reasoning step of the framework. For now, we halt after fixed number of steps. See Algorithm T more details.
5 EXPERIMENTS AND RESULTS
We evaluate our SI framework on subset of 10 /46 logical reasoning tasks introduced in Sec. 3 These tasks were chosen based on whether they include context information necessary to answer question, whether the questions have definitive answer, and to ensure that they cover different kinds of reasoning abilities. The tasks include bAbl (Weston et al. 2016) Tasks 1-3, which require the model to use 1-3 supporting time-ordered facts respectively to answer question, and Tasks 1 15-16, which test deductive and inductive reasoning respectively. We also evaluate our model on a subset of the ProofWriter (Taforrd al.) 2 OWA datasets that matches closed world assumptions, for depths 0, 2, and 5 (there is no depth 4 task, also see Sec. D..4fr an explanation for we did not use CWA dataset directly). These language-based logical reasoning problems, where the depth the number of reasoning steps required to answer the question.
To baseline the performance of the SI framework, consider a 7B (same size as the LLM used in the SI framework) and a 40x larger 280B parameter LLM evaluated in a 5-shot setting. There are two types of evaluation for these vanilla baselines that we consider: multi-choice and generative evaluation. In generative evaluation, we measure the exact string match (first sentence in lower case and ignoring any non-alphabetic characters) between the output generated by the LLM and the ground truth answer. This is appropriate, since most of the tasks that we consider require either single word answer, or the dataset such that the answers are highly structured. multi-choice evaluation the I.LM used to score each of the answer choices, in et al. (2022). In general LLMs perform significantly better in multi-choice vs generative evaluation setting, since the chance level the multi-choice setting is significantly higher.
We also consider chain-of-thought (COT) Wei et al. 2022 inspired baseline, where the k-shot prompts to the 7B and 280B models include reasoning traces for the same examples that we use to prompt the SI framework (although with selection and inference combined, see Appendix ( for example prompts). This tests whether providing the reasoning examples alone sufficient to improve performance, whether the further breakdown into Selection and Inference sub-steps improves

---

Published as a conference paper at ICLR 2023
Oncrall
Nudels
make Tock 01
1B generation
St (instuned) punerst
Sale Toak 95
vumill geneuative
COT gemenativn
Sale Tedk
2008 generatio
Pract Wintee Depth 0
2000 van masteceaae
varilla chonce
Fact Winee Beptt
Prect Winter Depth?
Prodt Whiter Dupth 3
20
Accuracy (%6)
(h)
Figure 4: Quantitative results for the Delection-Inferenne (SI) framework. (a) Average accuracy over 11 datasets comparing like-for-like generative performance of the 7B and 280B parameter language models used in a 5-shot generalisation setting to predict the answer directly (vanilla), in the Chain-Of- Thought framework, COT, and in the SI framework. (b) Per task breakdown the performance of LLMs used naively, and within the COT and SI frameworks.
accuracy. Note that among all of the approaches outlined only the SI framework is explicitly set up to generate causal reasoning traces.
Fig. 4aleemonsttates that overall when evaluated generatively, the 7B parameter LLM within the SI framework performs better (5) than the same model evaluated naively (2.94%) or in the COT framework (41 (42%) (all p< 0.01, see Appendix[]for details of the significance test calculations). Not only that, the 7B parameter LLM within the SI framework also outperforms average the 40x larger 280B parameter LLM in both vanilla (33.84%) and COT framework (44.03%) ) (all p < 0.01). When evaluated in the easier multi-choice setting,
find that surprisingly the vanilla 7B parameter LLM outperforms the 280B parameter LLM (59.2) 159.21% % vs 52.94%). ) while still performing sigmificantly worse than the 7B SI model (p=00 0. 0.0.2). Note that the latter is evaluated in the harder generative setting. Per task breakdown shown in Fig. 4b demonstrates that the SI framework solves the bAbl 15 deduction task, the only model achieve 100% accuracy (significant difference from the other models, p < 0.01). Furthermore, it does so having seen only five examples in the prompt. The 7B SI model also significantly outperforms all other models on ProofWriter Depth 0 (p < 0.01), ProofWriter Depth 0.034....
As well as improving upon most baselines quantitatively, the SI framework also has additional qual- itative benefits: 1) it produces a causal, human interpretable reasoning trace that shows how the model reached its answer and 2) is able to recover from errors. We will now discuss each of these in turn.
Since the Selection module is only allowed to pick facts from the context and is separate from the Inference module, and since the latter does not have access to the question, the model has to use what is selected and cannot bypass the selection to compute its answer, thus creating a causal reasoning trace. Human evaluation of the quality of the produced reasoning traces bAbl tasks found that on average SI produced more valid selections (60% vs 47%), and more correct inferences (56% vs 38%) than CoT (see Tbl. AT for more details). Since the reasoning trace is in natural language and is causal, it can be audited, inspected and corrected by humans, which has significant implications for safety and interpretability (see demo video in Supplementary Materials).
Example reasoning traces produced by the SI model are shown in Fig. 2 In the bAbl 16 example shown on the right the model solving an inference problem, which requires the model infer the colour of an animal given facts about the colours of other animals. In this example, the model is asked "What co colour
greg" ", and told that "greg is lion". " This means first the model needs to use induction to infer rule about the colour of lions. On the first step, we see that the model induces a rule, "1 i ons a re wh te ", based on the fact that "brian is a 1 lion" and "brian is whi te " we can exactly what data underlies the model's 's decision to form a new rule. On the second step, we see that the model applies this newly inferred rule to the fact that "greg is a lion to reach the final conclusion that "greg is white" using deduction. Note that the ability of the SI framework
A "This could suggest that the 280B LLM has stronger priors, than the 7B LLM, which it favours over logical reasoning, For example, favouring sheep are afraid of wol ves despite context to the contrary MMin 2022. However this requires further investigation.
7

---

Published as a conference paper at ICLR 2023
Context:
Context:
a toaster is made of heating element for toasting heating elemant converts electrical energy/alectricity into heat energy for heating
adaptations are used for survival an animal can survive
environment little food by storing fat
camela humpa are used storing fat
energy change takes place toasted toaster? chemical energy light energy OR electrical energy
piece of
Question: Camels
humps on backs store fat, alloving them survive for meny desert life. charasteristic
days without
makes camells exampie adaptation OR Instinet migration libemaation
energy OR energy chemical energy light energy electrical energy.
Selection: adaptations are used for survival We know that animal can survive in envirerni nnvirrnmmnt with little food by storing fat
Selectiore a toaster is made of heating elemant for toasting, We know that heating element converts electrical pneryleleetriiity into heat energy for heating
Inference: an example of adaptation is storing fat for times when food is scarce
inference: the toaster convorts electrica energy into heat
Selection: an example of an adaptation storing fat for times when food is scarce. We
energy for toesting
know that a camel's humps are used for storing fat
Correct Answer: a toaster converts electrical energy to heat energy for toasting
Inference: camels humps are kind of adeptation for storing when food is scarce
Correct Answer: an example an aduptatiin camel humps storing
Figure 5: Handpicked example solutions obtained from the Inference framework on the EntailmeniBank dataset. Left: one-step reasoning; right: two-step reasoning. See Sec. for more examples which have not been handpicked.
to produce inductions relies on its ability to deal with uncertainty and understand what is "reasonable" something that L.MS are naturally capable of, while also being a known struggle point for symbolic AI.
Since the reasoning traces are output in natural language, they are easy for humans to interpret and potentially intervene. Consider scenario where there may be both a white lion and green lion mentioned in the context, in which case we could see which information the model used make its final decision and decide whether we want to trust it (example in Fig. 2bp. We could also imagine examples where the model puts together two unrelated facts to come up with an incorrect inference. and this could also be easily be spotted by human and rectified by replacing the wrongly inferred fact with a correct one, and re-running the consequent reasoning steps.
Aside from inspecting reasoning traces and using them to debug when something goes wrong, the additive nature of our model it accumulates new knowledge with each reasoning step, means that it also has the ability to recover from errors. Fig. 2b demonnttrss such an example. In the first step the model inferred that "swans are often gray", using the facts that " "juliur is a swan." and " "julius is gray". While this is correct, this new information is not useful for answering the question, which asks about lions. However, it is still possible for the model to make the more useful inference that " "lions are often white" " in later step and recover from its original misstep.
6 FINE-TUNING LANGUAGE MODELS FOR SELECTION AND INFERENCE
In Sec. 5 we have demonstrated significant improvements logical reasoning accuracy when using prompt-engineering to specialise L.MMs for Selection and Inference in the SI framework. Prompt- engineering has the additional benefit of not requiring large amounts of step-by-step reasoning data, which may be hard to obtain. In this section we investigate whether the accuracy of the SI framework can be further improved by fine-tune tuning the LLMs for Selection and Inference. We present extensive results the PreefWriter dataset. Finally, to show that our approach also works more complex natural language tasks we show qualitative results on science problems from EntailmentBBakk Dalvi al. 202T Both ProofWriter riter and EntailmentBank datasets provide ground truth reasoning traces.
The Selection LLM is fine-tuned to select a subset of sentences (including facts and rules) from the context by generating a string of the form "sent 2. We know that sent 4 [and sent 7] "given the context and the question. We ask the Selection LLM to generate sentence labels (e.g. "sent 2" or "sent 4") instead of the sentences themselves, because this prevents the Selection LLM from cheating by making up facts to answer the question quicker. This preserves the dependency of the selection and therefore subsequent reasoning steps on the context. The Inference LLM is fine-tuned to compute an entailment given the selection. Both models are fine-tuned on single steps of reasoning only. Example training data are shown in Fig. A2 A2
On the Priterriter dataset, the Inference LLM converged very quickly to >99% test accuracy after only 300 fine-tuning steps with a batch size of 16, which is to be expected as we found that pre-trained LLMs are good at single step entailment out of the box as shown in Fig. 3b Examples of inferences
made by the model are shown in Fig. A3 The Selection LLM was trained for x 10 steps (with batch TPU) with the exact string match accuracy reported in Fig. 6a Although we notice that the model is much better at predicting selections for problems that require fewer steps of inference than those that require more, ultimately the model still achieves high (> 80%) accuracy
size 16 for 50 hours
8

---

Published as a conference paper at ICLR 2023
1.0
Basifin
Exoos Strig Match
Jacoent Strebelty
-6
006
0.4
Depth
0.2
Depth 3
Dopth 6
3.0
Fine-tring Slep
Depth
Depth
(a)
(b)
Figure 6: Fine-tuning the SI framework on the ProofWriter dataset. (a) Average test fine-tuning accuracy for the Selection module trained on single-step selection across all ProofWriter datasets (depth 1, 2, 3 and 5). (b) Intersection over union between reasoning traces produced by model and the ground truth reasoning steps. Baseline, 7B parameter LLM fine-tuned predict all reasoning steps in one go; SI, using same 7B LLM fine-tuned for single step Selection and Inference.
across most of the reasoning depths. Predicting early selections for deeper reasoning problems is hard, because it requires planning. is important problem address in future work.
Fig. shows that fine-tuning LLMs on single steps of reasoning within the SI framework leads to significant improvements in final reasoning accuracy (78.95%) over the prompt-engineerel version of the SI framework (57.93 (57993%) and other prompt-enggnn enginnernd baselines (vanilla/COT generative 7B: 0.34/15.73% 280B: 31.58/21.12%) We also found that the fine-tuned 7B LLM used within the SI framework produces significantly more accurate reasoning traces compared to the same LLM fine-tuned to predict all reasoning steps in one go (Fig. 6bb. We quantified this using the Jaccard Similarity, Jaccard Simi aarrttrrtaar aaaaattyynnnniaa (MPICGT)MMMUCC
between the proof steps predicted by each model, M, and the ground- tround-truth truth reasoning steps, GT, as shown in, calculated using exact string match over alphanumeric characters. Qualitatively we observed that while the baseline model is good at predicting most of the reasoning steps, they often appear in the wrong order, there are additional the minimal reasoning path, and some steps get repeated number
reasoning steps that are of times. Fig. and Sec. MM) demonstrate that our approach able to generalise beyond procedurally generated datasets and can reason about the natural science questions too.
7 CONCLUSION
We have presented the Selection-Inference framework for improving the ability of pre- pre--rraince language models to solve logical reasoning problems expressed in natural language. Our approach borrows from the best practices of neurosymbolic approaches to break down logical reasoning into a modular recursive pipeline that not only significantly improves the reasoning accuracy, but also produces causal interpretable reasoning trace. We have demonstrated that prompt-engineered LLMs used in the SI framework significantly outperform both the vanilla and COT baselines evaluated in equivalent settings, and even 40x larger baselines. The performance of the SI framework can be further improved through fine-tuning if step-by-steg ste reasoning data is available.
A model capable of casual, interpretable and logically valid multi-step reasoning has potential appli- cations law, medicine, science, maths, economics, and other areas where trustworthy and verifiable logical inference is important. At the same time we recognise that special care will need to be taken to evaluate such a model before deployment in any these settings. Further work is also needed, for exam- ple, to improve the Selection module (e.g. by allowing the model search over and evaluate different rea- soning traces); to address the halting issue (both in terms of when to stop the selection and when to stop the overall reasoning); to incorporate verifiers that would help avoid false inferences being added to the context; to enable the system to source its own relevant context rather than relying on it being provided in the dataset; and extend the ability of the system to deal with ambiguous unanswerable questions.

---

Published as a conference paper at ICLR 2023
REFERENCES
Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Neural module networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 39-48, 2016.
Yoshua Bengio, Yann Lecun, and Geoffrey Hinton. Deep learning for AL. Communications the ACM, 64(7):58-65, 2021.
Gregor Betz, Christian Voigt. and Kyle Richardson. Critical thinking for language models. In Proceedings of the 14th International Conference Computational Semantics (/WCS), pp. 63-75, Groningen, The Netherlands (online), June 2021. Association for Computational Linguistics. URL
ort/ wtt.://0002...--11111111 iwcs-1.7
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258 2021.
Kaj Bostrom, Zayne Sprague, Swarat Chaudhuri, and Greg Durrett. Natural language deduc- tion through search over statement compositions. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 4900-4912, Abu Dhabi, United Arab Emirates, dec 2022. Association for Computational Linguistics. URL httpss//ac.lanlloooooo.orrr////p wtw.://ggndnggggnnn.....
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901,, 2020.
Peter Clark, Oyvind Tafjord, and Kyle Richardson. Transformers soft reasoners over language. In Proceedings of the Twenty-Ninh International Conference on International Joint Conferences Artificial Intelligence, pp. 3882-3890, 2021.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2/ 10. 14168, 2021.
Bhavana Dalvi, Peter Jansen, Oyvind Tafjord, Zhengnan Xie, Hannah Smith, Leighanna Pipatanangkura, and Peter Clark. Explaining answers with entailment trees. In Pro- ceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 7358-7370, Online and Punta Cana, Dominican Republic, November 2021. Asso- ciation for Computational Linguistics. doi: 10.18653/w1/2211.emmm main. URL pttpss:/aclanttin
wwt.orgg///// emnlp-main.55 585
Artur d' Avila Garcez and Luis C Lamb, Neurosymbolic Al: the 3rd wave. arXiv preprint arXiv:2012.05876. 2020.
Marta Garnelo and Murray Shanahan. Reconciling deep learning with symbolic artificial intelligence: representing objects and relations. Current Opinion in Behavioral Sciences, 29:17-23, 2019.
Ahmad Ghazal, Todor Ivanov, Pekka Kostamaa, Alain Crolotte, Ryan Voong, Mohammed Al-Kateb, Waleed Ghazal, and Roberto V. Zicari. Bigbench v2: The new and improved bigbench. In 2017 IEEE 33rd International Conference on Data Engineering (ICDE), pp. 1225-1236, 2017. doi: 10.1 10.1109//CDE2201...16
Nitish Gupta, Kevin Lin, Dan Roth, Sameer Singh, and Matt Gardner. Neural module networks for reasoning over text. In International Conference on Learning Representations, 2019.
Vivek Gupta, Shuo Zhang, Alakananda Vempala, Yujie He, Temma Choji, and Vivek Srikumar. Right for the right reason: Evidence extraction for trustworthy tabular reasoning. Proceedings the 60th Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. 1/22 U / antho wathhology.org//0022
.acl2-...---- long. 231
10

---

Published as a conference paper at ICLR 2023
Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing multi hop qa dataset for comprehensive evaluation of reasoning steps. In Proceedings of the 28th International Conference Computational Linguistics, pp. 6609-6625,200 2020.
Drew Hudson and Christopher D Manning. Learning by abstraction: The neural state machine. Advances in Neural Information Processing Systems, 32, 2019.
Harsh Jhamtani and Peter Clark. Learning to explain: Datasets and models for identifying valid reasoning chains in multihop apestion-answering In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 137-150, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v http3///1122202mmm/11. wwt.::00.mmmmmmmm....... main. URL
org/ 2020. emnlp- ma main. 10
Andrew K Lampinen, Nicholas A Roy, Ishita Dasgupta, Stephanie CY Chan, Allison C Tam, James McClelland, Chen Yan, Adam Santoro, Neil C Rabinowitz, Jane X Wang, al. Tell me whyl-explanations support learning of relational and causal structure. In Proceedings of the 39th International Conference on Machine Learning. 2022.
Angeliki Lazaridou, Elena Gribovskaya, Wojciech Stokowiec, and Nikolai Grigorev. Internet- augmented language models through few-shot prompting for open-domain question answering. preprint w20.v::203.05/1111 15, 2022.
Xiang Lorraine Li, Adhi Kuncoro, Cyprien de Masson d' Autume, Phil Blunsom, and Aida Nematzadeh. A systematic investigation of commonsense understanding in large language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp- 1188--1185,, Abu Dhabi, United Arab Emirates, dec 2022. Association for Computational Linguistics. URL
https: www.clnnhhhoooooo wwt.org///// 2022. emnlp ma iiainn. 812
Bill Yuchen Lin, Seyeon Lee, Rahul Khanna, and Xiang Ren. Birds have four legs?! NumerSense: Prob-
ing numerical commonsense knowledge of pre-trained language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6862-6868, 2020.
Kevin Lu, Aditya Grover, Pieter Abbeel, and Igor Mordatch. Pretrained transformers universal computation engines. In The Thirty-Sixth AAA/ Conference on Artificial Intelligence (AAAI-22), pp. 7628-7636, 2022.
Jiayuan Mao, Chuang Gan, Pushmeet Kohli, Joshua B Tenenbaum, and Jiajun Wu. The nneuro-symbolie concept learner: Interpreting scenes, words, and sentences from natural supervision. In 7th International Conference on Learning Representations, ICLR, 2019.
Gary Marcus, The next decade in ai: four steps towards robust artificial intelligence. arXiv preprint arXiv:2002.061177,22 2020.
Gary Marcus and Ernest Davis. Rebooting Al: Building Artificial Intelligence We Can Trust. Ballantine Books Inc., 2019.
Jacob Menick, Maja Trebacz, Vladimir Mikulik, John Aslanides, Francis Song, Martin Chadwick, Mia Glaese, Susannah Young, Lucy Campbell Camppell-Gillnngham. Geoffrey Irving, et al. Teaching language models to support answers with verified quotes. urXiv preprint arXiv:2203. 11 147, 2022.
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannanch Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 11048- 1044--1066 1064, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. URL htt.:ttss//acclemcoooogggo
emn main. 759
Reicchiro Nakano, Jacob Hilton, Suchir Balaji. Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse. Shantanu Jain, Vineet Kosaraju. William Saunders, et al. WebGPT: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2 12.09332, 2021.
Maxwell Nye, Michael Henry Tessler, Joshua B. Tenenbaum, and Brenden M. Lake. Improving coherence and consistency in neural sequence models with dual-system, neuro-symbolic reasoning. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wort- man Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL https::/ wwtp:eeeeeeetfeeetttueoe _v1-p44

---

Published as a conference paper at ICLR 2023
Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, et al. Show your work: Scratchpads for intermediate computation with language models. In Deep Learning for Code Workshop, 2022.
20222.
Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training Gopher. arXiv preprint arXiv: 2021.
Swarnadeep Saha, Sayan Ghosh, Shashank Srivastava, and Mohit Bansal. PRover: Proof generation for interpretable reasoning over rules. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 122-136, Online, nov 2020. Association for Computational Linguistics. doi: 10.18653/v 118853////2200.cm//// h/w.///emmlpmmmmmmeeeee main... URL https://ac: anthoolllllnooooooooooooo org/ 202 mtt.::rr/g02.m.mmmmmmmmmm.mm emnlp-mailie.....
Oyvind Tafjord, Bhavana Dalvi, and Peter Clark. Proofwriter: Generating implications, proofs, and abductive statements over natural language. In Findings of the Association for Computational Linguisticss: ACL-IJCNLP 2021, pp.
2021.
Bojan Tunguz. 200,000+ jeopardyy questions, Nov 2019. URL questions
com/
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL htttps://openreiiee net/forum?id= ViiieeBBiiMMBBB- J
Jason Weston, Antoine Bordes, Sumit Chopra, Alexander Rush, Bart Van Merriënboer, Armand Joulin, and Tomas Mikolov. Towards ai-complete question answering: A set of prerequisite toy tasks. In 4th International Conference on Leurning Representations, ICLR, 2016.
Kexin Yi, Jiajun Wu, Chuang Gan, Antonio Torralba, Pushmeet Kohli, and Josh Tenenbaum. Neural-symbolic vqa: Disentangling reasoning from vision and language understanding. Advances in neural information processing systems, 31, 2018.
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman.
STar: Bootstrapping rea- soning with reasoning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https://openreview net ner:/oormmmid___BRRRRsssss
12

---

Published as a conference paper at ICLR 2023
Appendix
Table of Contents
Example prompts for vanilla baselines
14
A.1 ProtfWriter
14
A.2 bAb
14
A.3 2WikiMuluHlop
15
B Example prompts for COT baselines
B.1 ProofWriter
15
2
b
16
C Example prompts for Selection- Inferenceeeeeee
17
bAbl2
17
ProofWriter
18
D Selection- Inference evaluation details
D.I Selection module
19
D.2 Inference module
19
D3. bAhl
19
D4 ProofWriter
19
E
Reasoning Traces Output by SI
20
E.1
bAbI
................................
20
F.2
bAbl
................................
22
F
Pine-tuning InffeciionInnfernnce Details and Extra Results
23
F11
Selection-Inference Reasoning Traces
23
G
Limitation Details
27
Baseline Datasets
28
Tests of statistical significance
30
CoT evaluation
30
Comparing CoT and SI computation
32
Human evaluation
M SI EntallmentBank
33
M. Depth
33
M.2 Depth
34
13

---

Published as a conference paper at ICLR 2023
A EXAMPLE PROMPTS FOR VANILLA BASELINES
In this section show example prompts used to obtain the Vanilla language model baselines. For each sample, 5-shot prompt is composed by sampling five (question, answers) pairs from the batch and composing them according simple template. See examples below.
A.1 PROOFWRITER
Here are some statements that describe a situation:
Bob is cold.
Charlie is quiet.
Gary is cold.
Harry is quiet.
Big things are cold.
All blue things are not cold. something is quiet and blue then it is not cold.
All quiet things are cold.
If something is big and rough then it is round.
If something is cold and not rough then it is blue.
If something is quiet and not furry then is not blue. Round things are big.
Based on the above, the statement "Charlie is cold" is true.
Here are some statements that describe situation:
Erin is not cold.
Erin is kind.
Erin is red.
Erin is smart.
Erin is not white.
Erin is young.
Gary is cold.
Gary is not furry.
Gary is kind.
Gary is red.
Gary is not smart.
Gary is young.
All cold, smart things are not furry. Young, cold things are not furry. If something is white and smart then it is furry. Gary is white then Gary is not furry.
If Erin is young then Erin is furry. If Gary is not young then Gary is smart. If Erin is cold then Erin is young. Red things are kind. Based the above, the statement "Erin is not furry" is
A.2 B
Context: daniel went to the bedroom daniel journeyed to the office daniel travelled to the bathroom mary went to the office john journeyed to the bedroom
14

---

Published as a conference paper at ICLR 2023
daniel went back to the kitchen john went to the garden
daniel travelled to the office Question: where is john?
Choice: garden
Choice: bathroom
Choice: office
Choice: kitchen
Choice: bedroom
Choice: hallway
Answer: garden
Context: sandra went to the kitchen sandra went to the office sandra travelled to the hallway sandra went back to the kitchen mary travelled to the hallway sandra went to the bedroom john went to the garden sandra travelled to the office Question: where is sandra?
Choice: garden
Choice: bedroom
Choice: kitchen
Choice: bathroom
Choice: hallway
Choice: office
Answer:
A.3 2WIKIMULTIHOP
New lines are added between facts to fit on the page.
Q: when did Michael Haden-Powell's father die?
Here are some relationshipt to help answer this question. Michael Baden-Powell: father: :Peter Baden-Powell, 2nd Baron Baden-Powell, Peter
Baden-Powell, 2nd Baron Baden-Powoll::daa of death::9 December 1962
A: December 1962
Q: Where dees Ekaterina Rybolovleva's father work at? Here are some relationships to help answer this question. Ekaterina Emitrievna Rysolovieva:: father: :Dmitry Ryboloviev, Dmitry sypelovvee::eepoorrr....... Utaa A:
B EXAMPLE PROMPTS FOR COT BASELINES
In this section show example prompts used to obtain the Chain-of-Thoughh Thought baselines.
B. PROOF WRITER 3
15

---

Published as a conference paper at ICLR 2023
Given set of rules
and facts, you have to reason whether statement is true or false. Here are some facts and rules: If someone is red then they are nice.
If someone is kind and red then they white.
If someone is nice then they are kind. Fiona is red.
Does it imply that the statement "Fiona is not white" is True? Reasoning: If someone is red then they are nice. If someone is nice then
know that Piona red. Therefore, Fiona is nice.
they are kind. We know that Fiona is nice. Therefore, Piona is kind.
If someone is kind and red then they are white. We know that Fiona is kind and Fiona is red. Therefore, Fiona is white.
Here
facts and rules:
If someone chases the then they the cow.
If someone is big then they chase the cow.
If someone needs the bald eagle then the bald eagle is big.
If the
bear is nice and the bear needs the then the bear eats the lion.
If someone needs the lion and they eat the bald eagle then they are blue. If someone
eats the bear and they do not chase the cow then the cow is young. the bald eagle eats the lion.
the bear is round.
the lion eats the bald eagle. the bald eagle needs the cow.
the bear is young.
the cow is not nice.
the cow aces not chase the bald eagle. the bear does not eat the bald eagle. the bear needs the bald eagle.
the bald eagle chases the bear.
Does it imply
that the statement "The bald eagle does not eat the cow" True?
Reasoning:
If someone needs the bald eagle then the bald eagle is big. We know that the bear needs the bald eagle. Therefore, the bald eagle is big. If someone big then they chase the cow. We know that the bald eagle is big. Therefore, the bald eagle chases the cow. If someone chases the cow then they eat the cow. We know that the bald eagle chases the cow. Therefore, the bald eagle eats the cow.
n*m
B.2 BABI2
Below are some
stories about people noving objects between rooms. After each story you have to answer question about where particular object is.
Story:
at t-0 mary grabbed the football there
at t-1 daniel got the apple there
at t=2 mary went to the kitchen
at t=3 daniel journeyed to the office
at t=4 daniel went to the bedroom
at t=5 mary moved
garden Question: where is the apple?
Reason: at t=1 daniel got the apple there. We know that at t -4 daniel went to the bedroom. Therefore, the apple is in the bedroom
16

---

Published as a conference paper at ICLR 2023
story;
at t=0 sandra went to the office
at t=1 john teck the milk there
at t=2 sandra got the milk there
at t-3 john dropped the milk
Question: where is the milk?
Reason: at t=2 sandra got the milk there. We know that at t=0 sandra went to the office. Therefore, the milk is in the office
nnK
C EXAMPLE PROMPTS FOR SELECTION-INFERENCE
In this section we show example prompts used for both the Selection and Inference models in the framework.
C.I B
The selection prompt:
.*.
Here are a collection
of stories about peopie carrying objects from one room to another You will be asked where any object is. To answer this question you need to figure out who last had the object and which room they examples:
have the object in by the end the story. Here
Story:
at t=0 mary grabbed the football there
at t=1 caniel got the apple there
at t-2 mary went to the kitchen
at t-3 daniel journeyed to the office
at t=4 daniel went to the bedroom
at t=5 mary moved to the garden Question: where is the apple?
Reason: at t=1 daniel got the apple there. We know that at t=4 daniel went to the bedroom
at t-0 john moved to the bathroem
at t=1 john travelled to the office
at t=2 john picked up the football there
at t=3 john journeyed to the bathreen.
Question: where is the football? Reason:
The inference prompt:
at t=1 daniel got the apple there. We know that t=4 daniel went to the bedroom. Therefore, the apple in the bedroom.
at t-2 john picked up the football there. know at t=0 john moved to the bathroom. Therefore,
17

---

Published as a conference paper at ICLR 2023
C.2 PROOFWRITER
Below is example selection prompt. Note that is for depth-2 problem and show examples of the first reasoning step where the conclusion would not directly prove or disprove the statement and the last reasoning step, where the conclusion would directly prove or disprove the statement.
Given set of rules and facts, you have to reason whether statement is true false.
Here are some facts and rules: Nice people are quiet.
If Dave is smart then Dave is nice. All white people Dave is smart.
smart.
Harry is cold.
Does it imply that the statement "Dave is not quict" is true? Reasoning: If Dave
smart then Dave nice. We know that Dave smart. Therefore,
facts and rules: Blue things are green.
Here
All blue things are white.
If Anne is not big then Anne is blue. Big things are white.
All kind things are round.
If something is white and big then it not kind. If something is big and not rough then it is green. If something is white and blue then it is not green. Erin is not white.
Anne is big.
Bob is rough.
Anne is white Does it imply that the statement "Anne is kind" is True?
Reasoning: If something is white and big then it not kind. We know that Anne is white and Anne big. Therefore,
Here
facts and rules:
If something
likes the squirrel and it is not young then it chases the lion. If something likes the squirrel then it is rough.
If something chases
the rabbit and the rabbit If something eats the lin then it is young. If something likes the rapsit then it chases the rabbit. All rough things are nice.
young then it chases the lion.
the rabbit is young.
the squirrel likes the rabbit.
the lion likes the squirrel.
Does it imply that the statement "The lion is not nice" is True? Reasoning: n*.
Example inference prompt:
Nice people
are quiet. We know that Dave is nice. Therefore, Dave is quiet.
18

---

Published as a conference paper at ICLR 2023
If the cow chases the bald eagle then the cow eats bald eagle. know that the cow chases the bald eagle. Therefore
D SELECTION-NFERENCE EVALUATION DETAILS
D.1 SELECTION MODULE
The algorithm for the Scoring Selection module is shown in Algorithm 2
Algorithm 2 Scoring Se ect on_Module
Require: An n-shot prompt, p. Initial Context, CO, made of statements e.g. facts and rules.
Require:
Require:
The question,
Require: Language model, LLM.
Require:
The number of statements to select from the context, K. s empty string
while << K do
Lemp orggnaxxule_or_eccceee oeennrule ract LLM(token|p.Ct (q,ss) Choose the rule or fact with the maximum log-likelihood under the LM model.
+ join(s Scemp)
Join the selected fact or rule to the selection string.
end whilereturn s
D.2 INFERENCE MODULE
To extract the new fact be added to the context we filter out the first sentence of the text generated by the LLM using the following regular expression: x "["?? n]+
D.3 BABI
For bAbl tasks, the answer is a single word. For example, in bAbl 1-3 the answer is one of [ ("hallway", "bathroom", "bedroom", "garden", "kitchen", "office"]; for bAbl the answer is one of ("sheep", "cat", "mouse", "wolf"] and for bAbl the answer is of ["yellow",
"gray", "green", "white"]. However, model outputs a complete sentence, for example "emily is afraid of mice". Therefore, we take the answer be the final word output by the inference model its last step.
To obtain the results for bAbl tasks 1-3, 15 and 16 shown in Fig. 4b prompted the language model to solve the problem in single step of reasoning. An example of such prompt shown in Sec. C. We run the model for only a single step of reasoning too. Additional steps may increase the chance of the model reaching the correct answer, however, we do not yet have a mechanism for halting reasoning when the answer is reached.
BAbl 16 is an inductive reasoning task that could be solved in two steps (rather than one). The first step requires a rule to be inferred and the second step requires the inferred rule to be applied to another fact. For this reason, we also apply SI for two steps solve the bAbl 16 problems, first inferring rule from number of facts and then applying the rule to the correct fact. An example of this is shown in Fig. 2 Using this two step approach, we can see exactly which facts contributed the formation a new rule.
D.4 PROOFWRITER
We subset of the ProofWriter Open World Assumption, OWA, dataset. In the Close World Assumption dataset, CWA, everything that can be proven is True otherwise it is False. This means things are either True False. This also means that reasoning traces only provided when statement
19

---

Published as a conference paper at ICLR 2023
is True, but not when a itatement False. To "show" something False one has to enumerate all possible facts (possibly up to a certain depth) and then if a statement is not shown to be True it is assumed to be False. is is therefore not simple to generate meaningful reasoning traces for these types of problems.
On the other hand, in the OWA data if it is not possible to prove something is True or False, then it is Unknown. This means that for True and False examples, where one may want to show p (x), reasoning traces are available that terminate in (x) (for True) or not p (x) (for False). If one cannot show P (x) or not P (x) then the answer Unknown, and again there not clear reasoning trace for this;
it is necessary to enumerate all possible facts (possibly up to certain depth) and then if one has not shown p (x) or not p (x) it's considered Unknown. Note that here p is predicate and x a variable.
It is for this reason that we used the ProofWriter OWA dataset and removed the Unknowns; this gives us a dataset with reasoning traces concluding in either True or False. If we used CWA we would only have traces that could conclude True.
We evaluate the on 5 tasks from the ProofWriter dataset, each requiring varying numbers of reasoning steps (1, 2, 3 and 5). This requires the model to learn to compute intermediate conclusions that may not directly lead to the final answer, but may be needed reach the final answer. While, this can be very hard to achieve using prompt engineering alone, we endeavour to do so, by demonstrating examples of inter- mediate and final steps of reasoning (for problems of depth >)) This means that the language model sees (I) examples that both encourage the model select rules and facts that may not answer the problem in one-step but may help the model to obtain an intermediate output that can be used in a future step and (2) examples of the final step, which takes the model to the final answer. See Sec. C.2 for an example prompt.
The ProofWriter tasks involve predicting if a given statement, for example "Bob is nice. ", is True or F False given the context of facts and rules. Our SI model attempts to derive the statement "Bob is nice. or the negation of the statement "Bob is not ni ce. from the context. To assign a label True Fa 1 se we follow the procedure proposed in the original ProofWriter paper (Tffoord et al. 2021) and test if any of the implications matches the given statement. If there is a match, the statement is considered be True, otherwise False.
ProofWriter results in Fig. 4b show that the -nferronnInfeeenee model outperforms the baselines for problems of depth zero and one, however, with increasing depth, the gap between SI and the baselines diminishes. This is in part because prompt engineering is not sufficient to obtain an optimal Selection module.
Another challenge with the ProofWriter dataset is deciding how many arguments should be selected for each nule. In the Protfrriter dataset, some rules take one argument, others take two. We experimented with various different ways to encourage the model to stop selecting arguments. For example, we append
Therefore, " as a choice to the context that the model can select. If the language model selects ". Therefore, facts to be selected. Note that when using fine-tuned models, estimating the number of statements via heuristics is no longer necessary since the model learns to generate string with the required number of sentences referenced by sentence label. For example, "sent 1. We know that sent 7 and sent 5."
then the selection step ends. We allowed maximum of two
To obtain results in Fig. 4b
run SI model for the minimum number of steps needed to solve the problem; a Depth d problem is run for d steps, with the exception of the depth-0 problem which run for i step. However, models may perform better when allowed to run for additional steps, in the case where the model makes a mistake on one step, but later recovers. Fig. Al shows how the number of SI steps can lead improved performance. There greater improvement to performance for depth-1 reasoning. For 2epth-2 reasoning, there was not enough variation in the selections at each step, so additional steps did not help as much as for 1epth-1. If the model is run for additional steps, once the answer has been reached, then the model often repeats its final reasoning step for the remainder of the iterations.
E REASONING TRACES OUTPUT BY SI
E.
BABI 15
Below show examples of reasoning traces output by our SI framework for the bAbl 15, deductive reasoning task. These examples are ones that the model got correct but are otherwise not cherry picked for their reasoning quality.
20

---

Published as a conference paper at ICLR 2023
Depth 1
Depth 2
75
Accuracy %%
65
55
step_0
step_1
step_2
step_3
step_4
step_5
step_6
step_7
step_8
step_9
Step
Figure A1: ProofWriter: effect of additional reasoning steps. With additional iterations of selection and inference the probability of the model producing the correct answer increases.
bAbl 15 Example 1
wolves are afraid of mice sheep are afraid of mice winona is a sheep mice are afraid of cats
cats are afraid of wolves jessica is mouse emily is a cat
gertrude is a wolf Question: what is emily afraid of?
Selection: emily is a cat. We know that cats are afraid of wolves Inference: emily is afraid of wolves
bAbl 15 Example2
wolves are afraid of mice sheep are afraid of mice winona is a sheep mice are afraid of cats
cats are afraid of wolves jessica is a mouse emily is a cat gertrude is a wolf Question: what is winona afraid of?
Selection: winona is sheep. We know that sheep are afraid of mice Inference: winona is afraid of mice
bAbl 15 3xample
wolves are afraid of mice sheep are afraid of mice winona is sheep
21

---

Published as a conference paper at ICLR 2023
mice afraid of cats cats are afraid of wolves
jessica is mouse emily is a cat gertrude is wolf
Question: what is gertrude afraid of?
Selection: gertrude is wolf. We know that wolves are afraid of mice Inference: gertrude is afraid of mice
bAbl 15 Example
welves are afraid of mice sheep are afraid of mice winona is a sheep mice are afraid of cats cats are afraid of wolves jessica is a mouse emily a cat gertrude is wolf Question: what is jessica afraid of?
Selection: jessica is a mouse. We know that mice are afraid of cats Inference: jessica is afraid of cats
E.2 BABI
In the example below there are two possible conclusions that may be drawn. Since brian, bernard and greg are all rhions but brian and bernard are white, while greg is green the answer could be green or white. black-box model may select one or the other answer without explaining why. Our model on the other hand shows that it use the facts that greg is green to infer that julius is green.
bAbl 16 Example 1: Multiple "correct" answers
bernhard is rhino brian is a rhino bernhard is white
brian is white lily is a lion lily is yellow greg is a rhino greg is green
julius is a rhino
Question: what color is julius?
Selection: julius is rhino. We know that greg is rhino and greg is green
Inference: julius is green
In the example below the model must answer the question what color is greg?. greg is a lion, which means the model needs to infer rule about lions. However, on the first step of reasoning the model infers that we inferred that swans are ofter gray. While this is factually correct given the context, it does not help answer the question. However, the model is able to recover and correctly infer that we inferred that lions are often white meaning that the model is still able solve the problem despite making mistake.
bAbl 16 Example 2: Recovering from an error
julius is swan julius is gray
bernhard is swan
22

---

Published as a conference paper at ICLR 2023
Selection model training example:
Inference model training example:
Promat:
Prompt:
sent D: If something is rough then visits the lion.
If something visits the cow and the cow visits the lion then the
sent 1: the cow visits
is nice. We know that the lion visits the cow and cow visit the lion.
sent 2: the lion visits the cow.
sent 3: the cow rough.
4: if something visits the and the visits the lion then
Tarnet
the lion is nice.
Therefore, the lion is nice.
something
the lion.
Does imply that the statement 'The lion visits the lion' is True?
Target:
sent 4. We know that sent and sent 1. Therefore, the lion nice.
Figure A2: Examples of inputs and targets used to fine-tune LLM for the Selection and Inference modules. On the is an example of supervision for step-1 in depth-2 problem. We create training data for all steps in the dataset by appending the ground-truth inferences for the intermediate steps to the context. On the right, example of data sample used train the inference model. Both the Selection and Inference LLM's are fine-tuned on only single steps of reasoning.
bernhard is yellow brian is a lien greg is a lion brian is white
lily is a frog
lily is gray
Question: what color is greg?
Selection: julius is a swan. We know that julius is gray Inference: we inferred that swans are often gray
Selection: brian is a lion. We know that brian is white Inference: we inferred that lions are often white
F FINE-TUNING DELECTION-INFEREIE DETAILS AND EXTRA RESULTS
Fig. A2 shows examples the format of the data used to fine-tune the Selection and Inference LLMs. The Selection module is trained to predict sentence labels rather than the sentence strings. This prevents the model from making up facts and forces the model use information in the context.
During selection, rather than scoring each sentence we can append sentence labels to each element in the context (as shown in Fig. [2]: use the Selection LLM generate selection strings and substitute in elements from the context using a dictionary. This process is much faster than scoring each element of the context, but still ensures that the selection consists only of samples from the context; the Selection module cannot make up facts to answer the question. Fig. A3 shows examples of entailment computed by the Inference LLM after fine-tuning.
Fig. 4b compares SI models incorporating fine-tuned vs. prompt-engineered LLMs. We that model using LLMs fine-tuned on single steps reasoning significantly outperform both the prompt-engineered LLMs and Vanilla LLM prompt-engineered to predict the final answer directly. Fig. A4 shows a reasoning trace output by the SI model on a challenging depth-5 problem.
F.1 SELECTION- CELECTION-INFERENCE REASONING TRACES
F.1.1 DEPTH-2 REASONING TRACES FOR DEPTH-2 PROBLEMS
Below we show examples of 22pth-2 reasoning traces produced via Selection-Infereence using modules fine-tuned the ProofWriter dataset.
Example 1:
If someone is cold then they eat the lion. someone is blue and they chase the dog then the dog chases the ofon.
23

---

Published as a conference paper at ICLR 2023
candidate
prompt
Therefore, Harry big
something is furry big.
Harry furry
Therefore, Anne not
Anne young and Anne smart then Anne is not cold. We know that Anne is young and Anne smart,
Therefore, Harry nice
If someone is white then they are nice. We know that Harry is white.
Therefore, the bald eagle eats the squirrei
someone choses the cat then they eat the squirrel. We know that the bald eagle cheses the
Therefore, the
All round, rough people We know the
and rough.
Therefore,
All green, young things are blue. know
green and Dave young.
Therefore, Erin is green
someone is then they are green. We know that Erin is red.
Therefore, Anne is rough
All big things are rough. We know that Anne is big.
Therofore, Harry smart
Green, cold people smart. We know that Harry and Harry cold.
Therefore, Bob young
people young. We know
big.
Therefore, dog dog
something is blue then it visits dog. know that dog is
Therefore, Dave is round
someone is kind and furry then they are round. We know that Dave is kind and Dave furry.
Therefore, the mouse is green
If the mouse visits the tiger then the mouse is green. We know that the mouse visits the tiger.
Figure A3: Inference Module fine-tuning test examples
Problem:
Reasoning trace output by the model: Selection: All smart, blue things kind. know that Dave is smart Dave blue.
something is quiet All young, cold things are kind.
then smart.
something kind then nice.
Inference: Dave kind.
All smart, blue things
kind.
something is cold then it is quiet.
Selection: If something is kind then is nice. We know that Dave is kind.
Anne is smart and Anne is kind then Anne is cold.
something is nice and blue then it is young.
Inference: Dave is nice.
something is nice and young then it is cold.
Anne is young.
Selection: something is nice and blue then is young know that Dave is nice and Dave blue.
Gary nice.
Charlie is nice.
Inference: Dave young.
Charlie is quiet.
Anne is blue.
Selection: something is nice and young then it is cold We know that Dave is nice and Dave is young.
(removed some facts for brevity)...
Dave is smart.
Inference: Dave is cold.
Anne is quiet.
Dave is blue.
Selection: something is cold then is quiet We know that Dave is cold.
Gary cold.
Anno is nice.
Inference: Dave quiet.
imply that the statement "Dvv not quiet" True?
Figure A4: ProofWriter depth-5 reasoning trace output by our model. The model produces an interpretable reasoning trace that allows inspect how the model reached its answer.
If someone eats the dog then the deg is young.
If someone is young and they eat the lion then they red.
someone nice then they eat the dog.
If someone
eats the lion and the lion eats the dog then the dog eats the lion. If someone
sees the lion and the lion chases the dog then the lion is nice. If the lien sees the dog the deg sees the lion then the dog nice. the lion sees the dog.
the dog sees the lion.
Does it imply that the statement "The dog eats the dog" is True?
step
24

---

Published as a conference paper at ICLR 2023
Selection
If the lion sees the dog and the dog sees the lion then the dog is nice. we know that the lion sees the dog and the dog sees the lion. Inference: The dog is nice.
step 1:
Selection: If
someone is nice then they eat the dog. We know that The dog nice.
Inference: The dog eats the dog.
Example 2:
If something is cold and red then it likes the mouse. If something needs the cat then the cat sees the dog. the mouse.
If something needs the cow then the
something sees the dog then the dog likes the cat. If the cat is not green then the cat does not see the mouse. If something sees the mouse then it is cold.
If something
likes the If something sees the cow then it needs the cow. the mouse needs the dog. the mouse needs the cat. the dog is nice. the cat is green. not nice.
and the cat needs the then the is nice.
the mouse the dog needs the cat. the dog sees the cow. the cow is not red. the cat likes the dog. the mouse sees the cow. the mouse needs the cow. the cow sees the dog. the
green.
the cow needs the dog.
the mouse is blue. Does it imply that the statement "The cow is cold" is True? step 0:
selection: If something needs the caw
then the cow sees the mouse. we know that the mouse needs the cow.
Inference: The cow sees the mouse.
step 1:
Selection: IE something
sees the rouse then it cold. We know that The cow sees the mouse.
Inference: The cow is cold.
Example 3:
Rough things are white. If Erin is smart and Erin is rough then Erin is white. If something is round then it is reugh. If Erin is round and Erin is not smart then Erin is white. All quiet things are not white.
If something is blue and white then it is not quiet. Erin is green. Erin is rough. Erin is blue.
Does it imply that the statement "Erin is not quiet" is True?
step 0:
Selection: If Erin is smart and Erin is rough
then Brin is white. We know that Erin is green and Erin is rough.
Inference: Erin is white.
step 1:
Selection: If something is blue and white
then it is not quiet. we know that Brin is blue and Brin is white. Inference: Erin is not quiet.
25

---

Published as a conference paper at ICLR 2023
Example 4:
If something chases the squirrel then the squirrel is big. If something is big then is not kind, If something chases the bald cagle and it sees the bald cagle then the bald cagle seos the lion. the cow docs not like the squirrel.
the cow the bald eagle likes the lion.
the lion.
the cow chases the squirrel. the 1ion chases the cow.
the bala eagle is not round. the squirred likes the cow. the cow likes the lion. the cow chases the bald eagle.
the squirrel likes the bald eagle. the cow is kind.
the lion chases the squirrel. the cow does not see the squirrel.
the lion chases the bald cagle. the squirrel likes the lion.
Does it imply that the statement "The squirrel is kind" True? step 0:
Selection: If something chases the squirrel
then the squirrel is big. we know that the cow chases the squirrel. Inference: The squirrel is big.
step 1:
Selection: IF something
is big then it is not kind. we know that The squirrel is big. Inference: The squirrel is not kind.
F.1.2 DEPTH-3 REASONING TRACES FOR DEPTH-3 PROBLEMS
Below we show examples of depth-3 reasoning traces produced via Selection Inffrennce using modules fine-tuned on the ProofWriter dataset.
Example 1:
If something eats the bald eagle then it eats the squirrel. If something
eats the squirrel then the squirrel does not the bald eagle. If the bald eagle not red then the bald eagle likes the squirrel. If the squirrel
1ikes the bald eagle then the squirrel visits the bald eagle. If something likes the bald eagle then the bald eagle is young. something is young then it eats the squirrel. the squirrel visits the bald eagle. the bald eagle visits the squirrel.
the squirrel likes the bald eagle.
Does it imply that
the statement "The squirrel does not eat the bald eagle" True? stop 0:
Selection: If something likes the bald eagle then the bald eagle is young. We know that the squirrel likes the bald eagle. Inference: The bald eagle young.
step 1:
Selection: If something is young
then it eats the squirrel. We know that The bald eagle is young. Inference: The bald eagle eats the squirrel.
step
Selection: If something eats the squirrel then the squirrel does not eat the bald eagle. We know that The bald eagle eats the squirrel. Inference: The squirrel does not eat the bald eagle.
Example 2:
26

---

Published as a conference paper at ICLR 2023
If scmecne is quiet then they are white. someone young and red then they are white. Young people are nice.
If somcone is nice then they are reund. All quiet people are young.
Red, big peeple Round, red people are white. If someone is round then they are quiet. Harry is young.
nice.
Bob is red.
Bob is big.
Does it imply that the statement "Harry is not quiet" is True?
step 0:
Selection: Young people are nice. Re know that Harry is young.
Inference: Harry is nice.
step 1:
Selection:
If semeene is nice then they are round. We know that Harry is nice.
Inference: Harry is round.
step
Selection:
If someone is round then they are quiet. We know that Harry is round.
Inference: Harry is quiet.
Example 3:
Here the model make the same selection on step and stee and therefore does not correctly show that the "The lion visits the mouse". Although the model has failed here it is (a) possible that the model can still recover (if ran for additional steps) and compute the correct answer and (b) we can see how the model reached its answer and interpret its failings because the reasoning trace is causal.
If something is green then it visits the mouse. then it is green.
If something chases the something chases the mouse then the mouse sees the bear. If something sees the bear and the bear visits the mouse then the mouse chases the lion. If something chases the mouse then it is green. If something visits the bear then it chases the mouse. the mouse visits the lion. the lion visits the bear. the bear chases the lion. Does it
imply that the statement "The 1ion does not visit the mouse" is True?
step 0:
Selection: If something visits the
bear then it chases the mouse. We know that the lion visits the bear. Inference: The lien chases the mouse.
step 1:
Selection: If something chases
the mouse then it is green. We know that The lion chases the mouse.
Inference: The lion is green.
step
Selection: If something chases
the mouse then it is green. We know that The lion chases the mouse. Inference: The lion is green.
G LIMITATION DETAILS
We have seen that our approach to solving reasoning problems, using SI, has many desirable properties and indeed this model is intended to be proof of concept to demonstrate that it is possible to build model with these properties. However, as proof of concept, this model has several limitations, which discuss in detail.
27

---

Published as a conference paper at ICLR 2023
When observing the outputs of our model, the main point of failure tends to be the Selection module. This is hard to quantify since we do not have labelled data for the intermediate reasoning steps. One reason for this is that we rrompt-engineeriin outputs, rather than fine-tuning While our current results are good, and do not require (large amounts of) task specific data, they can be significantly improved by fine-tuning our models for specific tasks, as demonstrated in Section
encourage language models to produce the correct
Prompt engineering works best for single modality cases (Nakano al., 2021 It is more difficult to get the model to do multi-step reasoning since the distribution or patterns for the intermediate steps differ the final step. is also difficult to get the model figure out how many arguments to select or how many arguments a rule takes, using only prompt engineering, again because there are multiple different patterns that the model needs to learn how and when to apply.
Other limitations of our work include the assumption that a database of facts or rules are given. In many practical settings we would need to be able to retrieve relevant knowledge from an existing knowledge base. There is exciting progress being made in this area (Lazaridou et al. 2022) and we hope in the future to combine these approaches with the SI model.
Finally, while our model has the benefit that performance scales with compute time; the longer we run our model the more likely it is to reach a correct answer, we do not have a good way of deciding when to halt the reasoning process to filter the reasoning traces. In our current approach we have fixed budget, and tend to report results based on the final inference. Results in Fig. Al suggests that accuracy could be improved if we had a mechanism for filtering the reasoning steps and selecting the best answer.
H BASELINE DATASETS
In this paper used tasks from six sources; bAbi (Weston et al. 2016). BigBench (Ghazal et al. 2017). AAC (Betz et al. 202T), Jeopardy (Tunguz 2019), ProofWriter TTajorr et al.) (021) and 2WikiMultiHop (Ho et al. 2020 All of these dataset are publicly available and our use of the data was in accordance to their respective license permissions. As far as we are aware, none of the datasets contain personally identifiable information or offensive content.
Task decomposition From bAbl dataset (Weston et al. 2016) used tasks 1-3 to measure the ability LLMS to cope with progressively larger numbers of reasoning steps on the same type of problem; task 6 to compare to task whether yes/no questions are easier for the models to answer compared to more free-form answers: task to compare to task I test how well the models can deal with negation; task 10 to test whether LLMs know that the facts they are given are not sufficient to answer question; tasks 15 and to test basic deduction and induction abilities respectively; task 18 compare task 2 for two step reasoning on a different kind of task (based on size).
Jeopardy (Tunguz. 2019) and 2WikiMultiHop (Ho et al., 2020) tasks measure the ability of LLMs to do reasoning in less structured settings, where the answer has to be generated in free form, and the level of available context varies between none (2WikiMultiHop. Jeopardy), to relevant and irrelevant unstructured context (2WikiMultiHop With Context and 2WikiMultiHop With Context & Facts), to relevant structured context only (2WikiMultiHop With Evidence).
AAC (Betz et al. 1 measures the ability of LLMs to do relatively shallow formal reasoning (1-2 steps) over a relatively large set of syllogistic argument schemes, both with (AAC Split Extended) and without (AAC Split) dealing with negation.
ProofWriter (Tafjord et al. 2021 tasks measure the ability of models to do formal reasoning over progressively more difficult tasks that require more steps of reasoning.
From BigBench (Ghazal et al. 2017) we imported the following tasks: Analytic Entailment, Epistemic Reasoning and measure the ability of LLMs to decide implicit entailment relationship given facts.
Entailed Polarity, Presuppositions as NLI and Logical Arguments measure the ability of LLMs to understand implied information from vague language.
Evaluating Information Essentiality and Sufficient Information measure how well LLMs can evaluate which context information is relevant and sufficient answer a question.
28

---

Published as a conference paper at ICLR 2023
Formal Fallacies Syllogisms Negation, Logic Grid Puzzle, Logical Fallacy Detection, and Logical Deduction test the ability of LLMs to do formal deductive reasoning.
Sequence Problems Tasks and Tracking Shufiled Objects are similar to bAbl tasks 1-3 and evaluates the ability of LLMs to do multi-hop reasoning based on sequenced facts.
Physics Questions and Unit Interpretation measure the ability of LLMs to reason about grade school science problems.
StrategyQA QA measures the ability of LLMs to do multi- reasoning based on general knowledge that is not explicitly provided as context.
Multiple choice normalisation We evaluated whether normalising log probability of the choices un- der the model by the token length resulted in better accuracy to avoid potential bias as reported in Lin et al. (2020). We found significant (p= 0.0002, two-tailed t-test with equal variance) but minimal difference between the average accuracy across all evaluated tasks, when evaluated with (67.92 + 09.--46.68 46.6668)) and without normalisation (68.3 46.53%). For this reason we use the unnormalised measures in the paper.
Evaluating dataset bias To check whether the datasets
are biased, compared how much the baseline multiple choice accuracy of LLMs when presented with options all by themselves, without any context question, deviates from the expected random performance calculated as 1 1/N, where N is the number of choices. We found that the two differed by very small amount 0.08 8.74% on average across all models and all multiple-choic datasets (p = 5e 16, two-tailed t-test with equal variance). Since the effect size was so small, we concluded that the datasets were not biased and present the expected random baseline where appropriate.
Appending choices to bAbl tasks We evaluated whether adding choices to the prompt improved the multiple choice accuracy of L.MMs on bAbl tasks. We found that this was not the case, with the average accuracy across all bAbl tasks being
when choices were appended, and 49. 44.%% when choices were not appended (p 2 61, two-tailed t-test with equal variance). For this reason, we report the latter results the paper.
2WikiMultiHop results evallatee the performance of LLMSs on 2WikiMultiHop (Ho 2020) dev subset using exact string match between the generated and the ground truth answers. In particular, the generated answer was truncated at the first sentence up to ".". "?" "!". "." or newline characters following the BigBench generative evaluation protocol (Ghazal et al. 2017). The two answers were then both stripped of all punctuation, white space and special characters before comparison is made. Dataset examples receive a score of 1 the post-processed answers match exactly, and otherwise.
We found that the models scored 13.62 + 34.3% average the original dataset, consisting of questions only. When the context of Wikipedia paragraphs with relevant and irrelevant facts to answer the question was added the context, the performance dropped to 1.47+ 1.47+12.005% on average. Adding information about the relevant facts within these context paragraphs did not help much, resulting in 1.97 13.9% accuracy. On the other hand, adding only the relevant facts extracted from the underlying knowledge graph triples has more than doubled the models' performance, resulting in 35.55 .87% average accuracy.
General insights LMs get progressively worse as more steps are needed for reasoning (see bAbl tasks 1-3 and 18, ProofWriter tasks; although not the case for Logical Deduction, Sequence Problems Tasks and Tracking Shuffled Objects is chance).
LMs find yes/no questions harder to answer than freeform questions (see bAbl task 6 vs 1).
LMs struggle to deal with negation (bAbi task 9 1) unless it is in very well formalised limited setup (AAC Split Extended).
LMs struggle with deciding when they do not have sufficient information (bAbl task 10 vs 1; Evaluating Information Essentiality and Sufficient Information are both at chance).
LMs average at formal deduction and induction tasks, although their performance very much depends on the difficulty of the task and the evaluation protocol (see bAbI task 15 and Logical Deduction, although Proof Writer, Formal Fallacies Syllogisms Negation, Logic Grid Puzzle, and
29

---

Published as a conference paper at ICLR 2023
Spit Fotended Denah
2090luutiio Contoct
AKC Spll Seenndes Out Beman
zwartutripp With Eeitence"
70tcc insnlligg
Are Enteliment
AAC Sel Extendes Donsan
Extanded
Bab 00
Frtaimnntt
Babi 66
bolt look
test Tuxt
Tysk
Dabi
lank
Babi Tass
tabl Tysk
look
Tp sontei Reeest no ISON
bult Taak
Evalusting irformaiin Exserbuuity
Polntty
Pormal talaais Syllegamm Negalton
Sekttrrss Ressoring
Mode's in
Mides
Logioal Aegumants
18
Icpardy
Loghal Decsutio: 5 Clecck
Logic Gnd Fizzle Tol
Lopnal Dhetcctin: Utrect
Logical Aeguments
Logial Deduction 0.gget
Lopkal Fallacy Detection
Restapaccttas
Detect
Frost Witter Taot own Depth
Phyaics Qambors
Frrst Whttr Taok onn Depti
Prroigpecterr.. be
hoou Wate tok UMA Dupth1
Poot
Treof Wrtter Taok own Degii
Depth
Sequance Problemo
Proof Witte Texk Cac Depth
secuarce/oossn lott
Taciing
Suffiient intomtionn
Taciing 5 Shufted Obeect
Tocking Coaeda
Faciie Snutted Otsects
wecintepetatinn Lxi
incintepraaaton Ltd
Ineepsaation Lo2 Unit ineeparation
aecinterpntatan Lx?
Ovenal
Cecral
Accuracy
Accurary
(a) Absolute multi-choice and generative (marked with (b) Relative accuracy compared to chance level for multi- asterix #) accuracy.
choice tasks. Red line chance performance.
Figure A5: Average accuracy of IB, 7B and 280B parameter Vanilla LLMs evaluated on logical reasoning tasks in 5-shot generalisation setting.
Logical Fallacy Detection results are close to chance, while AAC results, where the models are evaluated in very structured setting are very good).
In less formal mutli-hop question answering scenarios, LLMS are close to chance when no context provided (2WikiMultHHop. StrategyQA; although Jeopardy is an outlier) when the provided information unstructured (e.g. whole Wikipedia paragraphs as 2WikiMultiHop With Facts and 2WikiMultiHop With Facts & Rules), but do better when minimal structured context information is provided (2WikiMultiHop With Evidence).
LMs also perform poorly in solving grade school science problems (Physics Questions and Unit Inter- pretation) although this ability is better in multiple choice compared to generative evaluation settings.
LMs are, however, reasonable at doing simple implication, entailment and induction tasks (see bAbl task 16, Analytic Entailment, Entailed Polarity, and Logical Arguments; although on Epistemic Reasoning and Presuppositions NLI the models perform around chance level).
I TESTS OF STATISTICAL SIGNIFICANCE
To calculate statistical significance of differences between different models in Fig. 4 we used proportion hypothesis test for binary data. In particular, we used two-sided proportions -test from stat 1 s. stats. wtt.sssttttttooo.ttttooooooooooe proportion.
J COT EVALUATION
preliminary human evaluation test was run on 50 randomly sampled problems from each of the five bAbl tasks used in this paper, evaluating CoT and SI approaches based three metrics:
30

---

Published as a conference paper at ICLR 2023
Proportion of problems with made up facts in reasoning
7b
280b
100%
75%
50%
25%
babi1
babi2
babi3
babi15
bab16
pw0
pw1
pw2
pw3
pw5
Task
Figure A6: Proportion of problems in each dataset where COT generated a reasoning trace with made up facts.
1. Are the facts selected within the reasoning trace faithful to the facts presented in the context?
2. Even if the facts are made up. is the selection logically valid?
3. Even if the selection irrelevant to the question, is the resulting inference logically consistent?
Full results are presented in ATble Some representative qualitative examples of the CoT reasoning included below:
INPUT:
brian a swan greg is a lion greg is white brian is gray lily is rhine lily is gray julius is a frog julius is white bernhard is
Question: what color pernhard? OUTFUT: bernhard is a swan. We knew that
greg is lion and that greg is white. Therefore, bernhard is white
INPUT:
greg is frog greg is white
bernhare is a swan lily is a lion brian is a rhine bernhard is yellow brian is green lily is yellow
julius is a frog
Question: what color is julius?
OUTFUT: julius is frog. We know that brian is a frog and that brian is green. Therefore, julius is green
The above example shows CoT making up a fact to answer the question. The question above asks "what color is julius?". The CoT method correctly identifies that Julius is frog. Then, rather than correctly saying "We know that greg is frog" the model makes up a fact "brian is frog" and combines this with the (existing) fact that "brian is green" to incorrectly conclude that "julius is green". Our
31

---

Published as a conference paper at ICLR 2023
model, Selection-Inference. could not make this mistake, since the selection step is restricted to only selecting statements from the context (see Section 4.1
INPUT:
1ily is a freg brian is a 1ion greg is rhino
julius julius is white brian is green
rhino
lily is green
greg is white
bernhard is frog
Question: what color is bernhard? OUTFUT: bernhard is a frog. We knew that
brian is frog and that brian is green. Therefore, bernhard is green
INPUT:
cats are afraid of wolves mice are afraid of cats sheep afraid of mice gertruae is a cat wolves are afraid of sheep jessica is mouse emily is wolf winona is cat Question: what is emily afraid of? OUTPUT: emily is cat. We know that cats are afraid of wolves. Therefore, emily is afraid of wolves.
INPUT:
at t=0 john travelled to the office at t=1 mary journeyed to the kitchen at t=2 mary moved to the garden at t=3 daniel went to the office at t=4 john went to the hallway
at t-5 mary moved to the kitchen
at t-6 mary travelled to the bedroom
at t-7 daniel journeyed to the garden Question: where is mary?
OUTFUT: at t=7 daniel journeyed to the garden. Therefore, mary is in the garden
K COMPARING COT AND SI COMPUTATION
CoT involves a single forwards pass sampling max hat_aca_taaaeeeeeaeaaaaaaaeeeaee For a problem with H reasoning steps this would equate to roughly cot tokens t per_step max max_cot_toke
/ H.
Eelection-Inferenee on the other hand does involve a single forward pass, but rather (scores and) samples sequentially. For simplicity, let us initially assume that in the selection step we sample to- kens rather than score them. This means that we sample So in the case where cot_tckens Bettcknssper_steee max_in www_iinffeenne_tttee
(max nhax_selection_tokee
+ max innereenceeteeeenee
the number of computations is similar. This is reasonable assumption since models were prompted to solve these problems in a similar way and we do not expect one model to solve problems with significantly more or less tokens than another model.
_ee ect on on_tokens
Now consider that in the selection step, we do not sample tokens auto nuto-reeressive (as in CoT) but rather tokens. Tokens can be scored in parallel and therefore can be computed more efficiently than they can be sampled. Therefore, if implemented correctly, Selection Inference can be at least computationally efficient CoT, if not more efficient.
32

---

Published as a conference paper at ICLR 2023
L HUMAN EVALUATION
The reasoning traces produced by CoT and SI approaches on the bAbl datasets were evaluated for correctness by two of the authors of this paper. For each dataset 50 random problems were evaluated. For each problem the raters were shown the context, question, correct answer and reasoning trace produced by the model (the reasoning trace included the final answer produced by the model). The raters were asked to answer the following questions:
1. Is the selection in the context: y/n
2. Is the selection logically valid: y/n
3. Does the inference follow from the selection: y/n
Note that for SI all selections are always in the context by construction. See Table A1 for the full results breakdown.
Selection context
Valid selection
Inference correct
Task
CoT
SI
CoT
SI
CoT
SI
bAbI
82%
100%
100%
100%
52%
96%
bAbl
68%
100%
12%
24%
16%
24%
bAbI
6%
100%
4%
12%
2%
2%
bAbI 15
62%
100%
68%
100%
68%
100%
bAbl 16
42%
100%
52%
62%
52%
58%
Table A1: Human evaluation of the quality of the reasoning traces produced by CoT and SI approaches subset of bAbl problems.
M SI ENTAILMENTBANK
This section contains randomly sampled set of problems and their SI solutions from the EntailmentBBnk dataset.
M.1 DEPTH
the mantle is layer the earth the mantle located just below the crust of the earth
What layer of Earth is located just below the crust? Inner core OR Mantle OR Continental shelf OR Outer corc. step 0:
Selection: the mantle
layer the earth.
We know that the mantle is located just below the crust of the earth Inference: the
mantle is layer of the earth that is located just below the crust Correct Answer: mantle is the layer located belox the crust the earth
toaster is made heating element for toasting
heating element
converts electrical energy electricity into heat energy for heating What energy change
takes place when piece of bread is toasted in toaster? chemical energy light energy OR electrical energy to heat energy OR heat energy to chemical energy OR light energy to electrical energy.
step 0:
Selection: toaster is made
of heating element for toasting. We know that heating element converts electrical energy electricity into heat energy for heating Inference:
the toaster converts electrical energy into heat energy for toasting
Correct Answer
toaster converts electrical energy heat energy for toasting
33

---

Published as a conference paper at ICLR 2023
seat belt
applies force the opposite direction of the passenger motion seat belts are used for preventing injuries to passengers in cars In car accident, seat belt helps prevent injurics by applying force less than the force of the moving passenger OR greater than the force the noving car's motion OR in the opposite direction of the passenger's motion. step 0:
OR in the direction as the
Selection: seat belt applies force
in the opposite direction the passenger 's motion. Ne know that seat belts are used for preventing injuries to passengers in cars Inference
seat belts are used for preventing injuries to passengers in cars by applying force in the opposite direction of the passengers motion Correct Answer: belts used for preventing injuries by applying opposite force to the passenger's motion
metal flexible
a wire is usually made of metal Flexibility is physical property of some matter. Which
of these materials best demonstrates the property of flexibility
? a mirror CR a peneil a metal wire telephone pole. stop 0:
Solection
metal is flexible. We know that wire is usually made of metal is flexible and wire is made of metal Correct Answer: a metal wire is flexible
Inference:
trunk is part of a tree for supporting the tree providing support is a kind of function
The main function
of tree's trunk is to provide air OR fruit OR sunlight support. step 0:
Selection: a trunk is part of a tree for supporting the tree. we know that providing support is a kind of function inference: the function of trunk is to support tree provide support
Correct Answer: function of tree's trunk
gravity causes orbits
planets in the solar system orbit the sun The force necessary to keep planets in orbit
around the Sun is gravity. OR friction. OR magnetism. OR nuclear..
step
Selection: gravity
causes orbits. We know that planets in the solar system orbit the sun Inference
: gravity causes the planets in the solar system to orbit the Correct
Answer: gravity causes planets in the solar system to orbit the sun
iron is always magnetic iron nails are made of iron
Which object most likely has magnetic properties iron nail plastic clip step 0:
rubber eraser OR wooden spoon.
Selection
iron is always magnetic. Ne know that iron nails are made of iron Inference: iron nails magnetic Correct Answer: iron nails are magnetic
M.2 DEPTH
the sun is kind of star the earth revelves around the sun
34

---

Published as a conference paper at ICLR 2023
complete revolution
orbit of planet around its star takes one planstary year earth is a kind of planet
About how long does it take Earth te make
one revolution around the Sun? day OR week OR month OR year. step
Selection: the sun is kind of star. We know that a complete revolution orbit of a planet around its star takes one planetary year
Inference:
complete revolution of the sun around the earth takes solar year step
selection: complete revolution of the sun around the earth
takes one solar year. We know that the earth revolves around the sun Inference:
complete revolution of the earth around the takes solar year Correct Answer: a complete
revolution of the earth around the sun will take one earth year
distance from
an object decreases the pull of gravity on that object increases the moon is the celestial object that is closer to the earth than the sun the gravitational pull of the sun en earth 's oceans causes the tides the gravitational pull of the moen earth 's oceans causes the tides tides on Earth
The influence the Moon on
greater than that of the Sun. Which best explains this? The Moon is more similar in conposition to Earth than the Sun. OR The Moon has smaller mass than Barth and the Sun. OR The Moon has a higher density than the Sun. OR The Moon is closer to Earth than the Sun..
step
Selection: the moon is the celestial object that
is cleser to the earth than the sun. We know that the gravitational pull of the moon earth 'a oceans causes the tides and the gravitational pull of the aun earth 's oceans causes the tides Inference: the gravitational pull of the moon on earths oceans is closer to the earth than the gravitational pull of the sun
step 1:
selection: the gravitational pull
the moon on earth 's oceans causes the tides. know that the moon is the colestial object that is closer to the earth than the sun Inference
the moon is the celestial object that causes the tides earth Correct Answer: the gravitational pull of the earth affects the tides more than the gravitational pull the sun
moons orbit planets gravity
orbits
the moon is a kind of moon earth is a kind of planet
What keeps the Moon orbiting Earth? the Sun's gravity the Moon's rotation OR Earth's rotation OR Earth's gravity.
step
Selection: moons orbit planets. We know that earth is a kind of planet
Inference: earths moons orbit earth
step 1:
Selection: gravity causes orbits. We know that earths orbit earth
Inference: gravity causes the moon to orbit around earth Correct Answer: gravity causes the moon to orbit the earth
tool is kind of invention compass a kind of tool
natural magnetism is used for pointing north by a compass Which invention uses natural magnetism? satellite OR telescope OR compass OR binoculars. step 0:
35

---

Published as a conference paper at ICLR 2023
Selection: compass is kind of tcol. We
know that natural magnetism is used for pointing north by compass Inference: compass
is a kind of tool used for pointing north by natural magnetism step 1:
Selection: compass is kind of teol used for pointing north
by natural magnetism. We know that a tool is kind of invention Inference: a compass
is kind of invention used for pointing north by natural magnetism
Correct
Answer: compass a kind of invention that uses natural magnetism
adaptations are used for survival
an animal can survive in an envircnment with little food by storing fat camel's humps are used for stering fat Camels have humps their backs
that store fat, allowing them to survive for many days without food. This makes camels well suited to desert life. This characteristic is an example of adaptation. OR instinct. OR migration. OR hibernation.. step 0:
Selection: adaptations are used for survival. We know that
animal can survive an cavironment with little food by storing fat Inference: an
example of an adaptation is storing fat for times when food is scarce step 1:
adaptation is storing fat for times when food is scarce. We know that a camel's humps are used for storing fat Inference: a canels
Selection: an example
hamps are kind of adaptation for storing fat when food is scarce
Correct Answer: example of adaptation is camel humps storing fat
sunlight is kind of solar radiation uitraviolet light causes sunburn sunlight contains uitraviolet light Which form of solar radiation causes
sunburn? visible OR Uitraviolet OR Infrared OR x-rays OR Radio waves.
step
selection: sunlight is a kind of solar radiation. We know that ultraviolet light causes sunburn Inference
ultraviclet light is kind of solar radiation that causes sunburn step
Selection: ultraviolet light is a kind of solar radiation that causes sunburn. We know that sunlight is a kind of solar radiation Inference: ultraviolet light is kind of sunlight that causes sunburn Correct Answer: the ultraviolet in solar radiation causes sunburn
an automobilc usually a heavy object a kilogram is a kind of unit for measuring mass
kilogram is used to measure heavy objects Which is the most appropriate SI unit to use for expressing the mass automobiles le? millimeter step U:
milligram kilometer OR ki kilogram.
Selection: an automobile is usually a
heavy object. We know that kilogram is used to measure heavy objects Inference: kilogram can be used to measure the mass of an automobile
step
selection: kilogram can be used to measure the mass of an automobile We know that a kilogram is a kind of unit for measuring mass
Inference: kilogram can be used to measure the mass of an automobile used measure the of automobile
Correct Answer: kilogram
36


Reviews:
[{"review_id": "fh2Qm9pfrw_", "reviewer": "Reviewer_BSuH", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work presents a new way to perform multi-step logical reasoning tasks with frozen pre-trained large language models. In particular they first select a few (most often two?) pieces of information (Selection) by scoring each fact in the context with the model’s log-likelihood. Then the retrieved facts are combined and the model generates a conclusion based only on those facts (Inference).\n\nResults on ProofWriter and Baby tasks show that the Selection-Inference process is better than other traditional prompting methods (CoT). The work also shows improvement over models with more parameters.\n", "summary_of_the_review": "Overall, this is a good paper with impactful research, however it would benefit some clarifications and a discussion on how to resolve the weakness mentioned above.", "strength_and_weaknesses": "**Strengths** : This is a good paper with good motivation. The work can have a positive impact on the field of multi-step reasoning. However some aspects are not entirely clear (more details in the next section). Some improvement suggestions are listed below:\n\n**Weakness** : The Selection algorithm relies on the manual selection of K: number of facts to include in the prompt. This makes the framework hard to use in practice as manually selecting the number of required facts for each Selection step can be very tedious. Did the authors explore setting this parameter to something large (say 4 for all steps, for all tasks)? \n\nSimilarly and most importantly for the entire process of Selection & Inference the steps are repeated until a predefined number of steps. Hence the model is not responsible for identifying when it has answered the question and when it needs to stop this process.\nDo the authors have some thoughts about how to resolve this issue in future work? It would be nice to discuss this in the final section of the paper.\n\n**Question** : Do the authors have some intuition why in Figure 6b the baseline model performs better as the depth increases? The opposite would be a more natural behavior (like in the case of the SI model).\n\n**Recommendation** : Although to a smaller scale, and with models trained from scratch, the idea of using language models to generate step by step reasoning chains was first explored in “Measuring systematic generalization in neural proof generation with transformers” (Gontier et. al, 2020). The paper should compare their approach to this one in the Related Work section.\nSimilarly, the authors could consider testing their Selection & Inference framework on the CLUTRR benchmark (Sinha et. al, 2019) in addition to Baby & ProofWriter.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "fW5Iots2udF", "reviewer": "Reviewer_TQ9M", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a novel framework called Selection-Inference(SI), which exploits pre-trained large language models as general processing modules to solve logical reasoning problems. Using this framework, LLMs will continuously alternates between selection step and inference step to generate a sequence of casual reasoning steps. So not only the answers can be concluded from these reasoning steps, but also these causal natural-language-based reasoning trace helps us to learn how the model reached the answer and opens the system’s decisions to human critique. The results are very impressive. The paper shows that a 7B SI model outperforms the 280B model. ", "summary_of_the_review": "This paper proposes SI framework, which improves the ability of LLMs to solve reasoning problems and outperforms other framework including COT. Even though this framework still has many limitations, it does develop a new approach and achieves state-of-the-art results. And this work can be extended to many further works, which are inspirational for other works in the field.", "strength_and_weaknesses": "trengths\n\n- This framework of SI is different from all previous frameworks, as illustrated in Figure 1. SI use LLMs to produce each reasoning step one at a time, which is similar with ProofWriter. While ProofWriter can only answer “Prove this statement to be True/False” style question, SI is able to solve reasoning problems.\n\n- The designs of splitting of each step of reasoning into selection and interference makes the model more unlikely to make up information to answer and it cannot ignore previous reasoning when getting the answer.\n\n- The authors conduct extensive experiment and provide a comprehensive evaluation of LLMs on a set of 46 tasks. They show that LLMs are good at simpler single step logical inference in 5-shot generalisation settings, but struggle with harder problems.\n\n- Using SI framework, the 7B model shows impressive results and even outperform 280B LLM baseline using COT framework. Also, the SI framework has ability to produce a causal reasoning trace and is able to recover from errors.\n\nWeaknesses\n\n- Some important details are not presented. The algorithm now just simply halts after some times of steps. So will the result differs from different numbers of steps, and how. \n\n- The results are not convincing enough. In the paper, the authors just show the results of 7B SI. I wonder know that if the 280B version using SI will outperform all previous models.\n\n- There are many case studies, more quantitative metrics to evaluate the effect is better.\n\n- There are some typos in this paper, which causes additional burden for understanding. For example:\n\n1) typo in page 2, “Figure 1” paragraph, “indicate” -> “indicates”\n\n2) typo in page 5, “We use prompt...” paragraph, “the the following form” -> “the following form”\n\n3) typo in page 26, “We have seen...” paragraph, “which we we now” -> “which we now”\n\n4) typo in page 26, “When observing...” paragraph, “we us” -> “we use”", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "dxI8dbF4jn", "reviewer": "Reviewer_g5nq", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper explores the reasoning capabilities of large language models (LLMs). They carry out a comprehensive evaluation of 46 reasoning tasks to show that language models perform fairly well on single step reasoning problems, but suffer at multi-step reasoning problems. To that end, the paper proposes a new algorithm called selection-inference where LLMs solve multi-step reasoning problems by iterating over selection and inference modules to generate a series of interpretable causal reasoning steps. Through this method, they show that a smaller (in terms of parameters) LLM can outperform a baseline larger LLM. The paper also presents several interesting ablations with scale, fine-tuning, and performance on various levels of multi-step reasoning problems.  ", "summary_of_the_review": "The paper provides some very interesting understanding of large language model capabilities for single vs. multi-step reasoning. The proposed SI algorithm is novel and very useful. All the ablations are very useful. Overall, I recommend this paper. ", "strength_and_weaknesses": "\nStrengths: \n1.  The paper presents a comprehensive study of 46 reasoning tasks and gives an understanding of how good are the current LLMs in single step and multi-step reasoning problems. \n2. The proposed selection-inference algorithm is a very interesting and novel idea. It makes the reasoning trace more interpretable.\n3. The paper demonstrates that this approach will enable small language models to outperform baseline large language models that doesn’t use SI algorithm\n\nWeaknesses: \n1. It would have been a more interesting comparison to see how this method performs in comparison to other recent works such as self-consistency, verifiers, etc. The proposed method has its own merits, but comparing with more methods would have been more interesting.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "UDH0sOEL9Y", "reviewer": "Reviewer_QvM7", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces a new framework for multi-step reasoning by splitting reasoning into two main stages: selection and inference. Selection involves choosing relevant information from the question to the inference module. The inference module uses the information from the selection module to reason and produce an answer. The authors test the framework on 10 logical reasoning tasks and show that they can get a 7B parameter lm to outperform 280B parameter lm.", "summary_of_the_review": "The approach is empirically quite strong! The paper would be better with some additional baselines and justifications for design choices.", "strength_and_weaknesses": "## Strengths\n- ****************Clarity:**************** The paper is well-written and easy to follow. The problem, and motivation are clear and the method directly follows from them.\n- **************Novelty:************** Splitting reasoning into modules has been explored but to my knowledge, using something like the proposed SI method has not been seen.\n- The SI framework significantly improves performance on the reasoning tasks over vanilla CoT/ scratchpad approaches.\n- Selecting a subset of information ensures that the language model is restricted by the information it uses for reasoning.\n## Weaknesses\n- One missing baseline for the framework is where the CoT prompts are framed as SI prompts without splitting SI into two separate modules.\n- SI using larger language models: I expected to see how models a magnitude larger than 7B would perform with the SI framework. This would help us understand how important the specific components of the framework are.\n- For example, a larger model might not make facts up in the selection step when using greedy decoding (with temperature = 0).\n- Or, one might not need two separate calls to the LM for S and I, simply using a CoT script with steps for selection and inference is all that one might need.\n\n### Other\n\n- **********************************Reproducibility:********************************** The method is straightforward to implement and test. The authors provide the prompts but access to the larger 280B language model is restricted and the baseline results would be impossible to replicate.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "9_vPxDL4W5", "reviewer": "Reviewer_wTCG", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this work, the authors focus on leveraging pre-trained LLMs to solve language-based reasoning tasks, via in-context learning. \n\nFirst, on a wide range of tasks, the authors investigate how well existing LLMs at various model sizes tackle logical reasoning tasks in a standard 5-shot prompting setting. Results suggest that they perform better on simple single step logical inference, but suffer from more complex tasks such as multi-step reasoning. \n\nMotivated by the above observation, the authors propose a novel Selection-Inference (SI) framework, which is an iterative prompting method designed for solving logical reasoning tasks. At every iteration step, SI first ask the LLM to select a subset of most relevant supporting evidence from a larger set of input facts, conditioned on the question; then in an inference step, LLM is asked to infer a new fact from the selected evidence. The inferred new fact is either as the final answer, or added into the input fact set to the next iteration. \n\nThe authors show that equipped with this SI method, a LLM with 7B parameters can outperform a set of baselines, including a much larger 280B LLM, equipped with Chain of Thought (COT), the current state-of-the-art prompting method. \n\nThe authors also show that with some minor fine-tuning, a 7B LLM with SI can further a significant improvement. \n\nThe author also discuss and demonstrate how the SI framework can facilitate model interpretability, transparency, and humans-in-the-loop study. ", "summary_of_the_review": "Overall, I like this paper, it's clearly written, the proposed method makes sense to me and results suggest that the SI framework indeed improves the LLM's performance on a range of reasoning tasks. Unless I had some major misunderstandings, I recommend to accept this paper. ", "strength_and_weaknesses": "### Strength\n\n* **Neat idea**: The idea of the two step selection-inference framework can be seen as a way to leveraging human prior knowledge in prompt engineering. As the authors discussed in the motivation section, this to some extent bridges neurosymbolic AI with recent large-scale deep learning approaches. There are a few neat ideas I like in this paper:\n\n  1. Ranking negative log-likelihood for selection instead of directly prompting LLM for a list of outputs. This could effectively avoid the potential trouble of LLM generating something uncontrollable or hard to parse. Although I have to say this requires the LLM output probabilities to be available, which is not always the case (for practitioners who has less ML knowledge, or using online LLM APIs).\n  2. \"Occlude\" question from the inference module. This is in some sense adding an information bottleneck. \n\n* **Empirical value**: I'd love to see research along this line, which can potentially have great empirical value. Better prompting strategies can be useful in general and this is not limited to the ML community. \n* **Significance test**: I appreciate the authors include significance test in their experiment and result section. \n* **Interpretability, transparency, and humans-in-the-loop**: Compared to vanilla in-context learning and prior methods such as COT, the proposed framework can have a better interpretability and help researchers to better understand how an LLM makes certain decision. I also like the opportunity of humans-in-the-loop that the SI framework brings (e.g., users help to verify/filter selected evidence).\n* **Explicit knowledge representation / memory**: In some sense, the iteratively growing set of evidence can be seen as a working memory, or a representation of past experience that one can easily retrieve from. Thinking broadly, from a sequential decision making or robotics perspective, instead of multi-step reasoning, I can imagine something like the SI framework being used to facilitate multi-step planning. \n\n\n### Weaknesses\n\n1. Please see my question 3 below regarding the $H$ and $K$ values.\n2. The paper suffers from a few accessibility issues. Please see the minor issue and typos section below. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-5%