Making Large Language Models Better Reasoners with Step-Aware Verifier
Year: 2023

Abstract:
Few-shot learning is a challenging task that requires language models to generalize from limited examples. Large language models like GPT-3 and PaLM have made impressive progress in this area, but they still face difficulties in reasoning tasks such as GSM8K, a benchmark for arithmetic problems. To improve their reasoning skills, previous work has proposed to guide the language model with prompts that elicit a series of reasoning steps before giving the final answer, achieving a significant improvement on GSM8K from 17.9% to 58.1% in problem-solving rate. In this paper, we present DIVERSE (Diverse Verifier on Reasoning Step), a novel approach that further enhances the reasoning capability of language models. DIVERSE has three main components: first, it generates diverse prompts to explore different reasoning paths for the same question; second, it uses a verifier to filter out incorrect answers based on a weighted voting scheme; and third, it verifies each reasoning step individually instead of the whole chain. We evaluate DIVERSE on the latest language model code-davinci-002 and show that it achieves new state-of-the-art results on six of eight reasoning benchmarks (e.g., GSM8K 74.4% → 83.2%).

heading: 1 Introduction
text: Large pretrained language models (PLMs) have shown remarkable performance on various natural language processing tasks, either by few-shot learning with prompts (Radford et al., 2019; Le Scao and Rush, 2021; Jin et al., 2022) or by fine-tuning (Houlsby et al., 2019; Hu et al., 2021; He et al., 2022). However, despite the increasing size and capacity of PLMs such as GPT-3 with 175B parameters (Brown et al., 2020) and PaLM with 540B parameters (Chowdhery et al., 2022), their reasoning abilities are still limited and often require mul-
∗Work was done during an internship at Microsoft Research Asia.
tiple steps to produce correct answers, especially for tasks involving arithmetic, commonsense, or inductive reasoning (Cobbe et al., 2021).
Recent works (Wei et al., 2022; Zhou et al., 2022; Kojima et al., 2022; Lampinen et al., 2022) have demonstrated that PLMs possess some latent reasoning capabilities, but they need carefully designed prompts to activate them. For instance, Wei et al. (2022) proposed chain-of-thought reasoning, which inserts multi-step reasoning paths before generating the final answers, and achieved significant improvement on the GSM8K arithmetic benchmark (Cobbe et al., 2021). Wang et al. (2022c) further introduced a voting mechanism to select the most consistent answer among different reasoning paths, and achieved state-of-the-art results on several reasoning benchmarks using the PaLM model (Chowdhery et al., 2022). Building on these successes, this paper continues this line of research and advances the reasoning capabilities of PLMs in three aspects, as illustrated in Figure 1.
First, we propose to increase the diversity of reasoning paths by not only sampling from a single prompt, but also varying the prompt itself. We hypothesize that different prompts can elicit different ways of thinking, while the correct answer should be robust to these variations. Second, we propose to use a verifier to score the quality of each reasoning path and guide the voting mechanism. We argue that not all reasoning paths are equally good
5315
or reliable, and some may contain errors or inconsistencies that can be detected by the verifier. Third, we propose to assign a fine-grained label to each step of the reasoning path and use a step-aware verifier to attribute the correctness or wrongness of the final answer to each step. We conjecture that some steps may be correct but followed by wrong steps or vice versa, and identifying these cases can help diagnose and improve the reasoning process.
We name our method as DIVERSE (diverse verifier on reasoning step) and evaluate it on eight reasoning benchmarks that require different types of reasoning skills. We use three OpenAI PLMs (davinci, text-davinci-002, and code-davinci-002) and compare our results with recent state-of-the-art methods. We find that DIVERSE can consistently and significantly improve the performance of PLMs on these tasks, and achieve new state-of-the-art results on six of them1: GSM8K (74.4% → 83.2%), AsDiv (81.9% → 88.7%), MultiArith (99.3% → 99.8%), SVAMP(86.6% → 87.0%), SingleEq (79.5% → 94.9%), and CLUTRR (67.0% → 95.9%).
Our data is publicly available at https://github. com/microsoft/DiVeRSe.

heading: 2 Diverse Verifier on Reasoning Step
text: Figure 1 shows the overview of DIVERSE. The key insights are three-fold: (1) leveraging diverse prompts to induce more diverse reasoning paths from the language models (Section 2.1); (2) train-
1Most of the previous SOTA results were achieved by selfconsistency on PaLM-540B(Chowdhery et al., 2022).
ing a voting verifier to better derive the final answers from multiple reasoning paths (Section 2.2); (3) leveraging step correctness to further boost the voting verifier (Section 2.3).

heading: 2.1 Diverse Prompts
text: To reason effectively, it is beneficial to explore diverse reasoning paths, following the idea that “All Roads lead to Rome”. Wang et al. (2022c) proposed to generate various reasoning paths from language models by sampling decoding. However, their method relies on a fixed set of exemplars for all prompts, which may introduce bias and limit the diversity of the generated reasoning paths. To address this issue, we randomly select M1 different prompts for each question, and then sample M2 reasoning paths for each prompt using sampling decoding. This way, we obtain M = M1 × M2 diverse reasoning paths for each question.2

heading: 2.2 Voting Verifier
text: Verifier. The verifier takes a question and a candidate reasoning path as input, and outputs the probability that the reasoning path leads to the correct answer. We use deberta-v3-large (He et al., 2021) as the backbone model, with a small scalar head that outputs predictions on the [CLS] token.
Training the verifier. For each training question, we generate multiple candidate reasoning paths using chain-of-thought reasoning. We regard the reasoning paths that match the ground truth final answer as positive, and the others as negative.
Voting Verifier. Wang et al. (2022c) use majority voting to aggregate the predictions of different reasoning paths. This method may fail when the majority of the reasoning paths are misled, while the minority of the reasoning paths are reasonable. We propose voting verifier, which leverages both voting and verifier:
ŷ = argmax y
M∑
i=1
1yi=y · f(xi, zi,yi), (1)
where 1yi=y is an indicator function that returns 1 (or 0) if yi = y (or not), and f(·) is the probability produced by the verifier.

heading: 2.3 Step-aware Voting Verifier
text: Each reasoning path consists of several steps. We hypothesize that not all the steps in an incorrect
2Our main experiments use M1 = 5 and M2 = 20.
reasoning path are equally wrong, and some steps may still be useful for reasoning. To exploit this, we extend the voting verifier to a step-aware voting verifier by introducing an extended loss function:
L = L0 + α · L1,
L1 = |D̂|∑
i=1
|Si|∑
j=1
BCE(labeli,j , f ′(inputi, j)). (2)
α is a hyperparameter to balance the original loss L0 and the step-level auxiliary loss L1; Si,1, Si,2, ..., Si,|Si| are the steps in zi; labeli,j indicates whether Si,j is correct or not; f ′(inputi, j) represents the probability of the positive label for Si,j .3
To obtain the step-level labels (i.e., labeli,j) for negative training data with wrong answers, we design an algorithm that compares intermediate results among steps in positive/negative reasoning paths. Figure 3 illustrates this algorithm. This algorithm can not only work on math word problems, but also generalize to other reasoning tasks: we use an off-the-shelf natural language inference model, roberta-large-mnli (Liu et al., 2019), to check whether two reasoning steps are semantically equivalent or not. Given a reasoning step, if we cannot find any semantically equivalent step in
3Specifically, f ′(inputi, j) is predicted from the hidden state of the last token of Si,j in DEBERTA-V3-LARGE, similar to token classification tasks.
the positive reasoning paths, we label it and all the subsequent steps as negative steps.

heading: 3 Experimental Setup

heading: 3.1 Reasoning Tasks
text: Arithmetic Reasoning. Following Wang et al. (2022c), we use AsDiv (Miao et al., 2020), SingleEq (Koncel-Kedziorski et al., 2015), MultiArith (Roy and Roth, 2015), SVAMP (Patel et al., 2021), and GSM8K (Cobbe et al., 2021).
Commonsense Reasoning. Following Wang et al. (2022c), we use CommonsenseQA (Talmor et al., 2019) and StrategyQA (Geva et al., 2021).
Inductive Reasoning. We use CLUTRR (Sinha et al., 2019), a diagnostic benchmark for inductive reasoning, requiring inferring kinship relations between characters in short stories.

heading: 3.2 Details
text: Language Models. We use three OpenAI language models: davinci, text-davinci-002 and codedavinci-002. We use the default parameters except a temperature of 0.5 in sampling.
Exemplars. For arithmetic/commonsense/inductive reasoning, each prompt contains 5/7/7 exemplars. For DIVERSE, each question has 5 different prompts, and 20 reasoning paths are sampled from the language model for each prompt. For arithmetic reasoning, the exemplars are randomly sampled from the training dataset of GSM8K; for CLUTRR, the exemplars are sampled from its training dataset, with reasoning paths synthesized by handcraft rules (detailed settings for CLUTRR are listed in Appendix D); for StrategyQA and CommonsenseQA, their original datasets do not contain enough exemplars with well-annotated reasoning paths, so we construct 1, 000 pseudo exemplars by “self-teaching” (the approach and the noise issue are discussed in Appendix B) from “seed” exemplars provided by Wei et al. (2022).
Training Datasets. For each task, we sample 1, 000 ⟨question, answer⟩ pairs from the training dataset to train the verifier.
Verifier. We fine-tune deberta-v3-large (He et al., 2021) with learning rate 1 × 10−5 and batch size 128. For the step-aware verifier, we select the best α among 0.0/0.1/0.2/0.3.

heading: 4 Main Results
text: Table 1 shows the overall experimental results. We mainly compare DIVERSE with two baselines: (1) greedily decoding a single reasoning path (Wei et al., 2022), referred to as Greedy Decode; (2) sampling 100 reasoning paths, then select the final answer via majority voting (Wang et al., 2022c), referred to as Self-Consistency.

heading: 4.1 Effectiveness
text: Experimental results clearly demonstrate that DIVERSE can bring significant and consistent improvements over recent strong baselines. The improvements are across different models (davinci, text-davinci-002 and code-davinci-002) as well as different reasoning skills (eight tasks in three reasoning skills). Taking GSM8K as an example, compared to Greedy Decoding and Self-Consistency, DIVERSE brings improvements of 22.2%/12.0% on davinci, 33.1%/12.0% on text-davinci-002, and 27.0%/5.6% on code-davinci-002. Compared to
Self-Consistency, DIVERSE achieves average improvements of 5.6%/5.1%/54.3% on the three reasoning skills, respectively.

heading: 4.2 Comparing to Previous SOTAs
text: In Table 1, we also compare DIVERSE with: (1) previous SOTA results based on fine-tuning; (2) recent SOTA results (Wei et al., 2022) based on PaLM (Chowdhery et al., 2022), a gigantic language model with 540 billion parameters.4
On all the five arithmetic reasoning tasks, DIVERSE (with code-davinci-002) achieves new SOTA results, with an average improvement of 6.2%. On the two commonsense reasoning tasks, the performance of DIVERSE is slightly lower (−1.9%) than that of PaLM-based self-consistency. We speculate that the reason might be: these two commonsense reasoning tasks are multiple-choice tasks rather than open-ended generation tasks, re-
4DIVERSE can also be applied to PaLM, but PaLM is not publicly available.
sulting in more false-positive exemplars in the pseudo exemplar base (Details will be discussed in Section B.2). Regarding inductive reasoning, DIVERSE achieves a surprisingly good performance of 95.9% on the CLUTRR task, outperforming (+28.9%) previous SOTA result with fine-tuning (Sinha et al., 2019).5

heading: 5 Case Study
text: Table 2 shows an example of step-level scores given by the step-aware verifier. Steps in the correct reasoning path have relatively high scores, while the scores in the wrong reasoning path show where the path starts to be wrong. This indicates that besides improving the performance, the step-aware verifier can also bring interpretability to show the step-level correctness. We also show some extra examples of majority-voting in Table 10.

heading: 6 Analysis
text: We also conduct ablation experiments and analysis to investigate the keys to the success of DIVERSE.
5Sinha et al. (2019) also introduced a method with 100% accuracy. We do not take it into the comparison, as this method requires a domain-specific system with complicated rules to extract a knowledge graph for each input text.

heading: 6.1 The Effectiveness of Diverse Prompts
text: By diversifying both prompts and reasoning paths (⟨M1 = 5,M2 = 20⟩), we consistently improve performance over the sampling decoding approach (⟨M1 = 1,M2 = 100⟩) of Wang et al. (2022c), as shown in Table 3. Both methods use majority voting. Table 4 further reveals that neither only using diverse prompts nor only using sampling is optimal. In other words, the best performance is achieved by combining diverse prompts and sampling. Moreover, Figure 4 demonstrates that diverse prompts lead to more diverse reasoning paths. We hypothesize that this diversity contributes to the performance improvement by: (1) making correct results more distinguishable from varied errors during inference; and (2) providing more diverse negative samples for enhancing the verifier’s generalizability during training.

heading: 6.2 The Effectiveness of Voting Verifier
text: We compare three algorithms to conclude the agreement from diverse reasoning paths: majority voting, verifier, and voting verifier. Table 5 shows the results. Compared to majority voting, our voting verifier can significantly and consistently boost reasoning performance across different tasks and different language models. Verifier without voting often outperforms majority voting, but extending it to voting verifier can further boost the performance.

heading: 6.3 The Effectiveness of Step-aware Verifier
text: We evaluate the impact of incorporating step-level information into the voting verifier of DIVERSE. Table 6 shows the performance of DIVERSE with and without the step-aware mechanism on both the GSM8K and the CommonsenseQA datasets. We find that using the step-aware verifier improves the performance in most of the experiments. The only exception is code-davinci-002 on GSM8K, where the step-aware verifier slightly lowers the performance. We hypothesize that code-davinci-002 is more capable of generating high-quality reasoning paths, and thus does not benefit much from the step-level information.
Detailed Human Evaluation of Reasoning Steps. We further analyze the quality of generated reasoning steps, by asking human annotators to judge
whether the GSM8K reasoning steps produced by DIVERSE (with/without step-aware mechanism) are good or not. Here “good” means not only correct formulas and calculation results but also textual fluency and logical coherence.
We further examine the quality of the reasoning steps generated by DIVERSE (with/without stepaware mechanism) for GSM8K, by asking human annotators to rate them based on correctness, fluency, and coherence. For each test question, we compare three reasoning paths produced by codedavinci-002: the one with the highest verifier score, the one with the highest step-aware verifier score, and a randomly chosen one. The annotators (master students) label any incorrect or unsatisfactory reasoning steps in each path (single-blind) and explain why. We collect annotations for 200 test questions, half of which have correct final answers from all three paths, and half of which have incorrect final answers from all three paths.
We find that all the reasoning paths with correct final answers are also correct in every intermediate step, which shows that code-davinci-002 can reliably generate accurate reasoning steps, not just lucky guesses. However, we also find that many of the correct reasoning paths have unnecessary steps. Figure 5(a) shows that 40% of the random paths have redundant steps, and the verifier can lower this percentage to 31%. We also find that the step-aware verifier can further eliminate redundant reasoning steps from 31% to 20%.
Furthermore, for the incorrect reasoning paths, we find that the step-aware mechanism helps produce more correct steps before making mistakes. For each failed test question, we compare the number of correct steps in the path with the highest verifier score and the path with the highest step-aware verifier score (by human evaluation). Figure 5(b)
shows that for 33%/17% of the failed test cases, the step-aware verifier generates more/fewer correct steps than the verifier without the step-aware mechanism.
Step Error Types. Figure 6 shows the distribution of error types in the incorrect reasoning steps. We see that 95% of the errors are caused by incorrect formulations (i.e., using wrong intermediate results or operators and generating invalid formulas, which lead to incorrect answers). We also see that, although code-davinci-002 often makes division calculation errors (e.g., 10/3 = 3), both the verifier and the step-aware verifier can effectively assign low scores to such paths, thus improving the performance.
6.4 How Many Diverse Outputs Do We Need?
Figure 7 shows the accuracy at different M values, where M is the number of reasoning paths sampled from the 100 generated paths for each question. We observe that: (1) the accuracy increases with more reasoning paths, but the improvement becomes marginal at M ≥ 50; (2) DIVERSE outperforms self-consistency significantly and consistently at different M values.
6.5 How Many Training Data Do We Need?
DIVERSE requires a dataset with reasoning paths for training the verifier. Figure 8 shows how the size of this dataset affects the performance. We observe that: the performance is only reduced by about 2%, even if the size of training data is cut by 75% (from 1, 000 to 250). With the same reasoning paths, voting verifier performs better than majority voting, while verifier without voting causes significant performance drops.
GSM8K

heading: 6.6 The Impact of the Number of Exemplars
text: We conduct experiments for k = 3/5/8 (k is the number of exemplars used in each prompt) on GSM8K. Figure 9 shows the results. We observe that: using 8 exemplars in each prompt can further boost the accuracy of GSM8K to 83.2%.

heading: 7 Related Work
text: Reasoning Skills. Researchers in the literature have proposed many benchmarks requiring various reasoning skills, including commonsense reasoning (Zellers et al., 2018; Talmor et al., 2019; Bhagavatula et al., 2019; Geva et al., 2021) numerical reasoning (Dua et al., 2019), multi-hop reasoning (Yang et al., 2018), arithmetic reasoning (KoncelKedziorski et al., 2015; Roy and Roth, 2015; Miao et al., 2020; Patel et al., 2021; Cobbe et al., 2021), logical reasoning (Liu et al., 2020; Yu et al., 2020), inductive reasoning (Sinha et al., 2019) and tabular reasoning (Chen et al., 2020; Zhu et al., 2021).
Reasoning with Symbolic Systems. Much research in the literature enhances the reasoning capabilities of machine learning systems by exploiting symbolic systems, including knowledge graphs (Mihaylov and Frank, 2018; Bauer et al., 2018; Kundu et al., 2019; Wang et al., 2019; Lin et al., 2019; Ding et al., 2019; Feng et al., 2020; Wang et al., 2022b), or question taxonomies (Dua et al., 2019; Andor et al., 2019; Hu et al., 2019b; Wang et al., 2022a). Although these methods work well on specific benchmarks, they usually require domain-specific designs and human efforts, thus limiting the generalizability.
Reasoning via Language Models. This line of work aims to address reasoning tasks in a general sequence-to-sequence manner, empowered by reasoning-aware pre-training or fine-tuning of language models. For example, Deng et al. (2021)
proposed to train the language model with crawled data from the internet; Asai and Hajishirzi (2020) proposed a logic-guided data augmentation method to pre-train the language model; Shen et al. (2021); Cobbe et al. (2021) proposed to train a verifier to rank solutions sampled from fine-tuned language models; Geva et al. (2020); Yoran et al. (2022); Campagna et al. (2020); Wang et al. (2022a) proposed to equip language models with reasoning abilities by generating training examples with human-designed templates; Pi et al. (2022) proposed to inject reasoning capabilities into language models by continual pre-training on program execution data.
Reasoning via Prompting Gigantic Language Models. Gigantic language models like GPT-3 (Brown et al., 2020) have demonstrated impressive few-shot learning capabilities in many tasks and have attracted many research interests on making gigantic language models better few-shot learners (Zhao et al., 2021; Holtzman et al., 2021; Min et al., 2021; Liu et al., 2022; Lu et al., 2021; Rubin et al., 2021; Min et al., 2022). However, these methods struggle to address tasks requiring reasoning skills. To mitigate this, recently there is a line of research that focuses on unleashing the reasoning capabilities of gigantic language models via better prompting strategies. Wei et al. (2022) proposed chainof-thought reasoning, of which the key insight is the insertion of multi-step reasoning paths before generating the final answers; Wang et al. (2022c) proposed to improve chain-of-thought reasoning via self-consistency, of which the key insight is to conclude the most consistent answer from different reasoning paths sampled from the language model; Zhou et al. (2022); Creswell et al. (2022)
proposed to leverage gigantic language models to decompose questions into sub-questions, thereby addressing them in an iterative manner; Kojima et al. (2022) proposed that gigantic language models can even be good zero-shot reasoners, by designing prompts that can induce language models to do reasoning step-by-step; Lampinen et al. (2022) proposed building a prompt by selecting examples and explanations together, thus substantially improving performance over selecting examples alone. Despite their great successes, these works come with their limitations. This paper is a continuation of this line of research, focusing on diverse verifier on reasoning steps.

heading: 8 Conclusion and Future Work
text: In this paper, we present DIVERSE, a novel and general method to enhance the reasoning abilities of large language models. Our method builds on the idea of prompting language models with multistep reasoning paths, but introduces three key innovations: diverse prompts, voting verifier, and stepwise verifier. The latter is especially novel and effective, as it verifies each reasoning step separately and we provides a detailed analysis of the model’s behavior in each step. We demonstrate the superiority of DIVERSE through extensive experiments. For instance, using code-davinci-002, our method achieves state-of-the-art performance on most reasoning tasks, surpassing the 540B PaLM model with previous prompting techniques.
There are many directions for our future work. (1) As discussed in Appendix B.2, we will continue to investigate how to reduce or recognize false positive pseudo exemplars. (2) We plan to investigate mechanisms to produce better diverse prompts than
simple sampling. (3) We will extend DIVERSE to other tasks and continue to design better prompting techniques to elicit the power of gigantic language models.

heading: 9 Limitations
text: Computing Resources. Despite the surprising performance it achieves, our framework needs to be applied to large language models like GPT3 or PaLM. Inference with these models costs more time and budgets than fine-tuning models like RoBERTa (Liu et al., 2019).
Faithfulness. Although DIVERSE can significantly improve the accuracy of final answers, we still cannot guarantee that the reasoning paths produced by the language models are 100 percent faithful. This is the key challenge and future direction for this line of research (chain-of-thought reasoning).
More Training Data. DIVERSE needs more labeled data with well-annotated reasoning paths to construct diverse prompts, and it also needs a training dataset for supervising the verifier. However, from another point of view, this limitation can also be regarded as a contribution that studies how chain-of-thought reasoning can be further improved if we have more training data than just a few exemplars.
Human Evaluation of Reasoning Steps. We use human evaluation to measure the quality of the intermediate steps in reasoning paths since few current works provide reliable frameworks to evaluate the quality of reasoning steps.

heading: A Preliminaries
text: Prompting. Prompting means prepending a few exemplars to the task input x and generating the output y from the pretrained language model:
p(y|C,x) = |y|∏
t=1
pLM(yt|C,x, y<t), (3)
where C is the concatenation of K exemplars:
C = (x1,y1); (x2,y2); ...; (xK ,yK). (4)
We denote prompt as the concatenation of the exemplars C and the input x.
Reasoning Paths. For reasoning tasks that aim to generate an answer y for a question x, Wei et al. (2022) proposed the insertion of a reasoning path z before generating the answer y:
C ′ = (x1, z1,y1); ...; (xK , zK ,yK), (5)
where zi is a text reasoning path of how the answer yi is reasoned step-by-step for question xi.
Then, during inference, a reasoning path z will be generated before the answer y:
p(y|C ′,x) = p(z|C ′,x) · p(y|C ′,x, z). (6)
Figure 10 demonstrates this idea in arithmetic reasoning (GSM8K), and Table 7 demonstrates this idea in commonsense reasoning (StrategyQA) and inductive reasoning (CLUTRR).

heading: B Boosting Reasoning Paths via Self-Teaching
text: In this section, we first introduce self-teaching, the method we use to construct a larger exemplar base when the original dataset does not contain enough data with well-annotated reasoning paths (Appendix B.1). We then discuss the noise issue when facing multiple-choice tasks (Appendix B.2).
B.1 Self Teaching A critical issue of DIVERSE is how to provide diverse prompts.6 Supposing that there is an exemplar base E, we can sample K exemplars from it to construct a prompt, and repeat this M1 times independently to construct M1 prompts with diverse exemplars.
For scenarios that do not have sufficient exemplars (i.e., |E| < K ∗ M1), we propose to bootstrap the diversity of prompts by “self-teaching”, i.e., generating pseudo reasoning paths from a few exemplars and some ⟨question, answer⟩ pairs without reasoning paths.7 Suppose that D is a dataset without reasoning paths, consisting of
6Wang et al. (2022c) tried an ensemble-based approach, i.e., to permutate exemplars in the original prompt. However, this strategy does not increase diversity in terms of exemplars.
7This is motivated by Zelikman et al. (2022).
(x,y∗) pairs. Given the small exemplar base E, for each (x,y∗) ∈ D, we can use prompting to generate a reasoning path z and the predicted answer y. We define the pseudo exemplar base E′ as:
E′ = {(x, z,y)|(x,y∗) ∈ D,y = y∗}, (7)
then E ∪E′ can be regarded as the new exemplar base for generating diverse prompts.
B.2 Noises in Multiple Choice Tasks In our experimental setup, StrategyQA and CommonsenseQA are more challenging than other tasks, as they use pseudo exemplars generated through “self-teaching” (Appendix B.1).
“Self-teaching” may lead to bad exemplars, whose reasoning paths are invalid but happen to yield answers coinciding with the ground truth. Questions in StrategyQA/CommonsenseQA are twochoice/four-choice questions, respectively. Therefore, such noise would be more serious in StrategyQA than in CommonsenseQA. This somehow
explains why DIVERSE can achieve comparable performance (−0.8%) as the PaLM-based SOTA on CommonsenseQA, while it sees a 3.0% performance decline to PaLM on StrategyQA, which has only two choices. In other words, it is easier for StrategyQA to yield a right answer but a misleading reasoning path.

heading: C Data Statistics
text: Table 8 shows the reasoning benchmarks we use in this paper with examples. We use the same test sets as Wei et al. (2022) for GSM8K, AsDiv, MultiArith, SVAMP, SingleEq, and CommonsenseQA.
For StrategyQA, there are 2, 290 test cases (i.e., questions paired with TRUE/FALSE labels), but there is no other case that can be leveraged by DIVERSE to construct diverse exemplars (as introduced in Section 2.1). To address this problem, we randomly divide these 2, 290 test cases into two equal parts (denoted as T1 and T2). For each DI-
VERSE experiment of SQA, we conduct two runs: using T1 to construct diverse exemplars and T2 as the test set, and vice versa. The final reported solve rate is the average solve rate of these two runs.
For CLUTRR, Sinha et al. (2019) provided several versions: clean, supporting, irrelevant, and disconnected. The clean version is the basic dataset, while the others are the perturbed variations of it. Our experiments are conducted on the clean version.

heading: D Our Changes to CLUTRR
text: In our experiments, two changes are applied to the CLUTRR benchmark: (1) appending candidate answers to each question; (2) constructing reasoning paths based on rules. Table 9 shows an example of CLUTRR data after our modification.
Candidate Answers. Besides the original questions (e.g., “Mary, a female, took her husband who is a male, Roy, out for lunch. Ernest bought to dress for his father Roy. What kind of relative is Ernest to Mary?”), we also provide all the candidate answers (i.e., “The possible relationships are: sister, son, aunt, granddaughter, father, grandfather, grandmother, mother-in-law, uncle, niece, mother, brother, daughter, nephew, grandson, son-in-law, father-in-law, daughter-in-law”) in the input sequence. Our preliminary experiments show that, the gigantic language models cannot reach more than 50% accuracy without the sequence of candidate answers.
Reasoning Paths. For each question, Sinha et al. (2019) also provided a knowledge graph that formulates the relations directly mentioned in the question. Each knowledge graph consists of several ⟨e1, r, e2⟩ triplets, which means there is a relation r from e1 to e2. Take the aforementioned question as an example, the knowledge graph consists of two triplets: ⟨Mary, husband,Roy⟩ and ⟨Ernest, father,Roy⟩. For each question, we construct the reasoning path based on its knowledge graph. We first topologically sort all triplets in the knowledge graph. For each triplet, we convert it to a reasoning step using the template “{e2} is the {r} of {e1}”. After that, we can get the reasoning path by concatenating these reasoning steps. Take the aforementioned question as an example, the reasoning path is: “Roy is the husband of Mary. Roy is the father of Ernest. Thus, Ernest is the son of Mary.”
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Section 9
7 A2. Did you discuss any potential risks of your work? Left blank.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Left blank.
B1. Did you cite the creators of artifacts you used? No response.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? No response.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? No response.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No response.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? No response.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. No response.
C 3 Did you run computational experiments? Section 3 introduces the experiment setups of our work, and Section 4 contains the experimental results.

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? As mentioned in Section 3, the three models we used in our work are GPT-3 davinci, text-davinci-002 and code-davinci-002. However, we don’t know the inner details of these models and thus cannot estimate the total computational budget or the number of parameters.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 3
7 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Due to the cost of running large PLMs like GPT-3, we only run once to get the result under each setting.

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 3
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Section 6.3

heading: 3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots,
text: disclaimers of any risks to participants or annotators, etc.? Section 6.3
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. The data to be annotated are model outputs during the experiment. They are not obtained from other people or institutions.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. The data to be annotated are model outputs during the experiment. They are not obtained from other people or institutions.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: This is the key challenge and future direction for this line of research (chain-of-thought reasoning). More Training Data. DIVERSE needs more labeled data with well-annotated reasoning paths to construct diverse prompts, and it also needs a training dataset for supervising the verifier. However, from another point of view, this limitation can also be regarded as a contribution that studies how chain-of-thought reasoning can be further improved if we have more training data than just a few exemplars. Human Evaluation of Reasoning Steps. We use human evaluation to measure the quality of the intermediate steps in reasoning paths since few current works provide reliable frameworks to evaluate the quality of reasoning steps