SlowGuess/NeurIPS_2023_OCR huggingface dataset:
predictions to determine its final validity  $V(s_{i})$ . We then only retain the verified chain candidates  $\{S:V(S) = 1\}$ . In the plurality voting stage, we conduct a majority-based voting among the verified chain candidates to determine the final answer. This voting process ensures that the final answer is selected based on a consensus among the trustworthy reasoning chains.
In this section, we perform evaluations to demonstrate the effectiveness of our Natural Program-based deductive reasoning verification approach over diverse reasoning datasets. Firstly, we show that our deductive verification process leads to substantial improvements in the rigor and reliability of reasoning chains. Subsequently, we will examine the impact of deductive verification on the accuracy of final answers. Our findings reveal that by adopting our Natural Program reasoning format without verification, we improve answer correctness on challenging benchmarks. Further applying deductive verification leads to slight reductions in final answer accuracy. One reason for this phenomenon is that the verification process effectively identifies and eliminates flawed reasoning chains that still produce correct answers.
Benchmarks. We evaluate the deductive verification accuracy and the answer correctness of reasoning chains over a diverse set of reasoning tasks: arithmetic reasoning, symbol manipulation, and date understanding. For arithmetic reasoning, we utilize the following benchmarks: 1) AddSub [19]; 2) GSM8K [10]; 3) MATH [17]; 4) AQuA [24]. Among these benchmarks, the AddSub and GSM8K datasets involve middle school-level multi-step calculations to arrive at a single number as the final answer. The MATH dataset presents more challenging problems that require expressing the answer as a mathematical expression in LaTeX format. These problems involve concepts from linear algebra, algebra, geometry, calculus, statistics, and number theory. AQuA also features similarly challenging problems, except that questions are in a multiple-choice format. For symbol manipulation, we use Last Letter Concatenation [50], where the model is tasked with concatenate the last letters of all the words provided in the question. For date understanding, we use the one from BIG-bench [45]
Deductive verification evaluation setup. For each of the above benchmarks, we select 100 reasoning chains, where 50 of them are deductively valid and 50 of them exhibit reasoning mistakes. The ground-truth deductive validity of each reasoning chain is determined by human annotators.
Answer extraction. To extract answers from reasoning solutions, we first perform text splitting based on answer prefix patterns such as "answer is" or "option is". Then, using problem type-specific regular expressions, we extract the final answer. To extract the validity results from deductive verification processes, we only keep the last sentence of model response. We then extract the validity answer with regular expressions to obtain attitude words, e.g., "yes" or "no", to determine the validity answer. Sometimes, language models may not provide a direct answer and instead output phrases like "not applicable" at the end of the response. In such cases, we consider the answer from the model as "yes". Please refer to Appendix C for more details.
Model and Hyperparameters. We conduct our main experiments with GPT-3.5-turbo (ChatGPT) [32]. We also present results for the LLama model-family [47]) in Appendix A, where we find the deductive verification accuracy to be worse than larger models even after finetuning. For ChatGPT, we use a generation temperature of  $T = 0.7$ . For Unanimity-Plurality Voting, we set  $k = 10$  and  $k' = 3$  by default. We use 1-shot prompting for both reasoning chain generation and deductive verification (except reasoning chain generation for the date understanding task where we use 2-shot). See Appendix D.2 and Appendix D.3 for more details.
5.2 Comparison of Deductive Verification Accuracy
We compare the verification accuracy of reasoning chains using two methods: (1) verifying the entire reasoning chain at once (as described in Section 4.1) without utilizing the Natural Program, and
<sup>1</sup> Most results for Faithful CoT are from their official repository https://github.com/veronica320/Faithful-COT, except MATH and AddSub due to the unavailability. For these two datasets, we use our implementation and the same prompt for the math word problems in their paper. The prompt for Last Letters is not available, so we leave it blank.
Verification Method	Reasoning Correctness	GSM8k	AQuA	MATH	AddSub	Date	Last Letters	Overall
CoT	Correct	98%	96%	100%	92%	100%	96%	97%
Incorrect	2%	4%	0%	6%	26%	6%	7%
Two-shot	(Average)	50%	50%	50%	49%	63%	51%	52%
Natural Program	Correct	84%	72%	70%	95%	90%	96%	85%
Incorrect	84%	62%	76%	40%	56%	6%	54%
One-shot	(Average)	84%	67%	73%	68%	73%	51%	69%
Table 3: Comparison of deductive verification accuracy of reasoning chains for GPT-3.5-turbo (ChatGPT). We compare two approaches: (1) verifying entire reasoning chains generated by Chain-of-Thought prompting; (2) verifying reasoning chains generated in the Natural Program format with step-by-step decomposition. In the latter case, when we verify each reasoning step  $s_i$ , we only keep the necessary subset of premises  $\bar{p}_i \subseteq p_i$ . To calculate verification accuracy, for each dataset, we randomly sample 50 reasoning chains that are deductively valid and 50 reasoning steps exhibiting incorrect reasonings.
Methods	Arithmetic	Commonsense
GSM8K	AQuA	MATH*	AddSub	Date	Last Letters
CoT + Voting	87.62%	70.18%	35.93%	92.36%	69.97%	81.60%
Faithful CoT + Voting	75.80%	61.80%	\( 31.78\%^{1} \)	\( 88.35\%^{1} \)	\( 73.50\% \)	-
Ours (Natural Program (NP), No Verification)	87.05%	70.34%	36.75%	93.67%	72.49%	92.98%
Ours (NP + Deductive Verification + UPV)	86.01%	69.49%	36.48%	93.54%	71.45%	92.60%
Table 4: Final answer accuracy comparison on GPT-3.5-turbo (ChatGPT). All approaches generate  $k = 10$  reasoning chains for each problem before performing majority voting or reasoning chain filtering with our deductive verification approach.
(2) our Natural Program-based verification approach with step-by-step decomposition. The results, presented in Table 3, indicate that our approach achieves significantly higher reasoning verification accuracy across most datasets. It effectively identifies erroneous reasoning in faulty chains while maintaining a low rate of false positives for valid chains. However, we observe that our approach's effectiveness is limited on the "Last Letters" task. We hypothesize that this is due to the task's nature, where each subsequent reasoning step is conditioned on all previous steps, presenting greater challenges for reasoning verification due to the increased dependency among premises.
5.3 Impact of Natural Program and Deductive Verification on Final Answer Correctness
We then investigate the impact of our Natural Program reasoning format and our deductive verification process on final answer correctness. We conduct two experiments: (1) for each problem, we instruct language models to generate  $k = 10$  reasoning chain candidates in the Natural Program (NP) format and perform simple majority voting on final answers, without using deductive verification to filter out reasoning chain candidates; (2) applying our deductive verification approach to filter out reasoning chain candidates, and apply Unanimity-Plurality Voting (UPV) along the process to determine the final answer. As a reference, we also report the performance of Chain-of-Thought (CoT) [50] and Faithful CoT [27]. For these baselines, we perform simple answer-based majority voting with  $k = 10$  for fair comparison.
Results are presented in Tab. 4. While our major goal is to improve the trustworthiness and reliability of deductive reasoning, we find that prompting language models to reason in our Natural Program format achieves on-par or better final answer accuracy than baselines over many reasoning tasks. Upon further applying our deductive verification approach to filter out invalid reasoning chains, we observe a slight decrease in final answer accuracy. One major contributing factor to this decrease is the filtering out of reasoning chain candidates that provide correct answers but exhibit incorrect reasoning. We illustrate an example in Table 5, where ChatGPT generates the correct final answer but assigns incorrect premise numbers to support the first reasoning step. We note that in many such cases, our approach effectively identifies these reasoning errors, thereby enhancing the rigor and
Question: In March it rained 0.81 inches. It rained 0.35 inches less in April than in March.
How much did it rain in April?
Ground Truth: It rained  $0.81 - 0.35 = 0.46$  inches.
GPT-3.5-turbo (ChatGPT):
Question-related premises:
#1. It rained 0.81 inches in March.
#2. It rained 0.35 inches less in April than in March.
#3. How much did it rain in April?
The first reasoning step:
#4. (by #2) Step 1: Calculate how much it rained in April by subtracting the difference of 0.35 inches from the amount in March.
Amount of rain in April:  $0.81 - 0.35 = 0.46$  inches
Table 5: For this reasoning chain candidate of this GSM8k problem, ChatGPT generates the correct final answer but provides incorrect premise numbers for grounding the first reasoning step. In "Step 1", the correct premise numbers should be #1 and #2. Our deductive reasoning verification approach effectively identifies these reasoning errors, enhancing the rigor and trustworthiness of the reasoning process. It is worth noting that removing a correct answer from the candidate reasoning chains has a slightly negative impact on the overall final answer correctness.
Premise Context	# Shots	Reasoning Correctness	GSM8K	AQuA	MATH	AddSub	Date	Last Letters	Average
Full Premises	1	Correct	64%	54%	58%	95%	26%	96%	66%
Wrong	56%	68%	56%	24%	76%	5%	48%
(Average)	60%	61%	57%	60%	51%	51%	57%
Minimal Premises	0	Correct	84%	78%	90%	96%	90%	12%	75%
Wrong	26%	12%	28%	20%	20%	80%	31%
(Average)	55%	45%	59%	58%	55%	46%	53%
Minimal Premises	1	Correct	84%	72%	70%	95%	90%	96%	85%
Wrong	84%	62%	76%	40%	56%	6%	54%
(Average)	84%	67%	73%	68%	73%	51%	69%
Table 6: Ablation study on the impact of (1) premise context and (2) zero-shot vs. few-shot scenarios on deductive verification accuracy using our Natural Program-based approach with step-by-step reasoning chain decomposition. To verify each reasoning step  $s_i$ , we either the full premises  $p_i = QC \cup S_{\leq j}$ , or use the minimal subset of premises  $\bar{p}_i \subseteq p_i$  necessary as outlined in Sec. 4.1. The one-shot prompt we used is shown in Appendix D.3. For each dataset, we randomly sample 50 reasoning chains that are deductively valid and 50 reasoning steps exhibiting incorrect reasonings.
reliability of the language models' reasoning processes, albeit with a slight negative impact on the overall final answer correctness. Further discussions are presented in Appendix B.
In addition, we perform several ablation studies to gain further insights into the designs of our deductive verification approach. In Tab. 6, we compare two different approaches to verify a single reasoning step  $s_i \in S$  following our Natural Program format. The first approach utilizes all premises  $p_i = QC \cup S_{\leq j}$  for verification regardless of their relevance to  $s_i$ , potentially introducing irrelevant contexts. The second approach follows our design in Sec. 4.1 and only includes the necessary context and premises  $\bar{p}_i \subseteq p_i$ . We observe that removing irrelevant premises significantly improves the reasoning chain verification accuracy on many datasets, highlighting the importance of this technique.
We also ablate on our Unanimity-Plurality Voting strategy by investigating the impact of different  $k'$ . Recall that  $k'$  determines the number of votes to produce validity predictions of single-step reasoning. Results are shown in Tab. 7. We observe that increasing  $k'$  generally enhances reasoning validation accuracy, though we note that this is at the expense of more compute.
While we have demonstrated the effectiveness of Natural Program-based deductive reasoning verification to enhance the trustworthiness and interpretability of reasoning steps and final answers, it is
Answer Correctness	k' = 1	k' = 3	k' = 5	k' = 10
Correct	86%	90%	90%	92%
Wrong	38%	38%	38%	40%
Table 7: Ablation of different values of  $k^{\prime}$  on the verification accuracy of reasoning chains using our Unanimity-Plurality Voting strategy. Experiments are performed on AddSub using GPT-3.5-turbo (ChatGPT).
Question: Melanie had 10 quarters and 17 pennies in her bank. Her dad gave her 27 pennies and her mother gave her 19 pennies. How many pennies does Melanie have now?
Ground Truth: Melanie have 17 + 27 + 19 = 63 pennies.
ChatGPT's reasoning step:
#5. (by #1) Step 1: Calculate the number of pennies Melanie had initially.
Number of pennies in 10 quarters: 10 * 25 = 250
Number of pennies initially: 250 + 17 = 267
Table 8: An example question with ambiguous wordings. The term "pennies" in this question can be interpreted as either a type of coin or a unit of currency. In this particular question, "pennies" is treated as a type of coin. However, the initial reasoning step by ChatGPT mistakenly treats "pennies" as a unit of currency, resulting in the conversion of all Melanie's money into "pennies" (highlighted in red). Consequently, all subsequent reasoning steps follow this flawed logic, leading to an incorrect reasoning trace. Our deductive verification is not yet able to detect such errors.
important to acknowledge that our approach has limitations. In this section, we analyze a common source of failure cases to gain deeper insights into the behaviors of our approach. The failure case, as shown in Tab. 8, involves the ambiguous interpretation of the term "pennies," which can be understood as either a type of coin or a unit of currency depending on the context. The ground truth answer interprets "pennies" as coins, while ChatGPT interprets it as a unit of currency. In this case, our deductive verification process is incapable of finding such misinterpretations. Contextual ambiguities like this are common in real-world scenarios, highlighting the current limitation of our approach.
In this paper, we aim to enable Large Language Models (LLMs) to perform explicit and rigorous deductive reasoning while ensuring the trustworthiness of their reasoning processes through self-verification. To this end, we have proposed a novel framework based on "Natural Program", a natural language-based deductive reasoning format that facilitates reasoning verification and can be easily generated through in-context learning. Within this framework, we decompose the verification process of complex reasoning chains into step-by-step subprocesses that focus solely on necessary context and premises, allowing us to significantly enhance the accuracy of verification. Additionally, we introduce a Unanimity-Plurality Voting strategy to further improve verification accuracy. Experimentally, we demonstrate the superiority of our framework in improving the rigor, trustworthiness, and interpretability of reasoning steps and answers.
Broader Impact. While our deductive verification approach can mitigate hallucinations and reasoning errors of Large Language Models (LLMs), it does not completely eliminate these phenomena. LLMs can still produce harmful and biased content, make incorrect claims, and produce wrongful advice. This issue becomes particularly significant when LLMs engage in complex reasoning chains, increasing the risk of misleading users. Consequently, it is still crucial for users to exercise great caution when interacting with, deploying, or developing LLM-based applications.
We would like to express our sincere gratitude to Tongzhou Mu and Caiwei Xiao from UC San Diego, Kairong Luo from Tsinghua University, and Pulkit Madan, Reza Pourreza, Sunny Panchal, and Apratim Bhattacharyya from Qualcomm for their valuable discussions and feedback.
[57] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022.
[58] Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. Automatic chain of thought prompting in large language models. arXiv preprint arXiv:2210.03493, 2022.
[59] Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Olivier Bousquet, Quoc Le, and Ed Chi. Least-to-most prompting enables complex reasoning in large language models. arXiv preprint arXiv:2205.10625, 2022.
[60] Hattie Zhou, Azade Nova, Hugo Larochelle, Aaron Courville, Behnam Neyshabur, and Hanie Sedghi. Teaching algorithmic reasoning via in-context learning. arXiv preprint arXiv:2211.09066, 2022.
A Deductive Verification with Vicuna Models
We further explore the efficacy of deductive verification for open-source models. We select two popular models: Vicuna-7B and Vicuna-13B [7]. These models are fine-tuned versions of LLaMA-7B and LLaMA-13B [47] using the ShareGPT data<sup>3</sup>. We use the same Natural Program-based one-shot verification method we used in the main paper. Results are shown in the first and the third rows of Table 9. We observe for the original Vicuna models without finetuning, Vicuna-7B exhibits poor performance in deductive verification and fails to find out reasoning mistakes, while the larger Vicuna-13B exhibits better verification accuracy.
Models	Reasoning Correctness	GSM8K	AQuA	MATH	AddSub	Date	Last Letters	Overall
Vicuna-7B	Correct	80%	86%	96%	98%	96%	80%	89%
Wrong	14%	22%	16%	6%	20%	34%	19%
(Average)	47%	54%	56%	52%	58%	57%	54%
Vicuna-7B (fine-tuned)	Correct	68%	48%	46%	76%	46%	32%	53%
Wrong	72%	86%	54%	60%	72%	68%	69%
(Average)	70%	67%	50%	68%	61%	50%	61%
Vicuna-13B	Correct	86%	82%	92%	96%	72%	74%	84%
Wrong	32%	36%	20%	20%	34%	30%	29%
(Average)	59%	59%	56%	58%	53%	52%	57%
Vicuna-13B (fine-tuned)	Correct	74%	50%	56%	86%	72%	12%	58%
Wrong	72%	76%	72%	68%	62%	96%	74%
(Average)	73%	63%	64%	77%	67%	54%	66%
ChatGPT (GPT-3.5-Turbo)	Correct	84%	72%	70%	95%	90%	96%	85%
Wrong	84%	62%	76%	40%	56%	6%	54%
(Average)	84%	67%	73%	68%	73%	51%	69%
Table 9: One-shot Deductive Verification Accuracy of Vicuna-7B and Vicuna-13B. The models are evaluated with or without finetuning on our deductive verification dataset. For each dataset, we randomly sample 50 reasoning chains that are deductively valid and 50 reasoning steps exhibiting incorrect reasonings.
We therefore conduct an additional experiment to investigate if the verification accuracy of Vicuna models can be improved by fine-tuning. To this end, we generate a deductive verification dataset, which consists of 2000 reasoning steps evenly distributed between correct and incorrect categories. We automatically generate this dataset using GPT-3.5-turbo since it exhibits a very high accuracy of single-step verification. We first use GPT-3.5-turbo to generate solutions for problems in GSM8K's training set. We then execute step-by-step deductive verification on these solutions using GPT-3.5-turbo. For solutions that result in correct final answers, we retain the reasoning steps that pass deductive verification. For solutions that yield incorrect final answers, we retain the reasoning steps that cannot pass deductive verification. After constructing our dataset, we then fine-tune the Vicuna models using the verifications of the 2000 reasoning steps. Models were fine-tuned with 4 A100-80GB over 3 epochs. Training parameters are shown in Table 10.
As shown in Tab. 9, we observe that fine-tuning with our dataset can enhance the deductive verification accuracy of Vicuna models not only on the dataset where the training dataset is constructed (GSM8K), but also on many other datasets. However, the accuracy is still worse than non-finetuned GPT-3.5, which suggests that model capacity has a significant impact on deductive verification capabilities.
B More Discussion on Improvements of Deductive Verification Accuracy Versus Improvements on Final Answer Correctness
In the main paper, we demonstrated that our verification approach significantly improves the verification accuracy of reasoning chains (Tab. 3, 6, but barely improves the final answer accuracy (Tab. 4). We further analyze this phenomenon below:
<sup>3</sup>https://github.com/domeccleston/sharegpt
Hyperparameters	Value
Optimizer	AdamW
Learning rate	1 × 10-5
Weight decay	0.00
Num epochs	3
Batch size	64
Learning rate schedule	Linear
Table 10: Hyperparameters for finetuning Vicuna models with our deductive verification dataset.
Consider the GSM8K dataset as an example (recall that the final answer for a problem is obtained through majority voting). Among all problems,  $91.6\%$  of problems have  $|(\text{number of votes received by the correct answer}) - (\text{largest number of votes received by a single wrong answer})| > 2$ , and their final answers are unlikely to be changed through our deductive verification approach. For the rest of the cases  $(8.4\%)$ , where deductive verification is more likely to impact their final answers, we found that:
- Among all reasoning chains that arrive at correct answers (these correct-answer chains account for  $49.4\%$  of all reasoning chain candidates),  $46.2\%$  of reasoning chains are filtered out by our verification process.
- Among the reasoning chains that arrive at correct answer but are filtered out by our verification process,  $76.3\%$  indeed exhibit incorrect reasoning.
- Among the reasoning chains that arrive at correct answer and are not filtered out by our verification process,  $78.0\%$  indeed have correct reasonings.
- Among the reasoning chains that do not arrive at correct answer and exhibit incorrect reasonings (these account for  $50.6\%$  of all reasoning chain candidates),  $40.6\%$  are filtered out by our verification process.
The above statistics show that a significant portion of reasoning chains that arrive at correct answers but exhibit incorrect reasoning are successfully eliminated. Therefore, the reliability and trustfulness of reasoning chains that arrive at the correct answers are significantly improved. Combined with the fact that a significant proportion of reasoning chains that exhibit incorrect answers are eliminated, and that our approach's verification accuracy significantly improves over naive verification approaches, our primary goal to improve LLM reasoning reliability is accomplished.
Nevertheless, the removals of many reasoning chains yielding correct answers (specifically, a significant  $46.2\% \times 49.4\%$  of all chains) has a notable impact. This even exceeds the removals of reasoning chains with incorrect reasonings and answers ( $40.6\% \times 50.6\%$  of all chains). As a result, there are fewer votes for the correct answer when generating final answers through majority voting, which limits the final answer accuracy. In the future, we believe that when a greater proportion of incorrect reasoning chains with incorrect answers are filtered out, we can improve the final answer accuracy.
C More Details on Answer Extraction
In this section, we describe our process to extract the final answer from language models' responses. The process begins by selecting the last three non-empty lines. Then, these lines are processed through the following pipeline:
1. Firstly, we use a list of regular expressions to identify "No-Answer" patterns within the text, such as "we cannot answer (thisthe) question". This process helps us ascertain whether the model can provide a conclusive answer. If any such patterns appear in the text, we mark "No answer!" as the final answer. However, if we don't detect these patterns, we proceed to the next steps for extracting the final answer.
2. Secondly, if any "Answer-Split" patterns are found in the text, we divide the text into several blocks using the identified pattern. The last block of text is then utilized for extracting the answer.