SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zhan Ling $^{1*}$  Yunhao Fang $^{1*}$  Xuanlin Li $^{1}$  Zhiao Huang $^{1}$  Mingu Lee $^{2}$  Roland Memisevic $^{2}$  Hao Su $^{1}$
$^{1}$ UC San Diego,  $^{2}$ Qualcomm AI Research†
Large Language Models (LLMs) significantly benefit from Chain-of-Thought (CoT) prompting in performing various reasoning tasks. While CoT allows models to produce more comprehensive reasoning processes, its emphasis on intermediate reasoning steps can inadvertently introduce hallucinations and accumulated errors, thereby limiting models' ability to solve complex reasoning tasks. Inspired by how humans engage in careful and meticulous deductive logical reasoning processes to solve tasks, we seek to enable language models to perform explicit and rigorous deductive reasoning, and also ensure the trustworthiness of their reasoning process through self-verification. However, directly verifying the validity of an entire deductive reasoning process is challenging, even with advanced models like ChatGPT. In light of this, we propose to decompose a reasoning verification process into a series of step-by-step subprocesses, each only receiving their necessary context and premises. To facilitate this procedure, we propose Natural Program, a natural language-based deductive reasoning format. Our approach enables models to generate precise reasoning steps where subsequent steps are more rigorously grounded on prior steps. It also empowers language models to carry out reasoning self-verification in a step-by-step manner. By integrating this verification process into each deductive reasoning stage, we significantly enhance the rigor and trustfulness of generated reasoning steps. Along this process, we also improve the answer correctness on complex reasoning tasks. Code will be released at https://github.com/lz1ocean/verify_cot.
The transformative power of large language models, enhanced by Chain-of-Thought (CoT) prompting [50, 21, 59, 42], has significantly reshaped the landscape of information processing [14, 26, 49, 56, 13, 55, 23, 29], fostering enhanced abilities across a myriad of disciplines and sectors. While CoT allows models to produce more comprehensive reasoning processes, its emphasis on intermediate reasoning steps can inadvertently introduce hallucinations [4, 30, 16, 20] and accumulated errors [4, 51, 1], thereby limiting models' ability to produce cogent reasoning processes.
In fact, the pursuit of reliable reasoning is not a contemporary novelty; indeed, it is an intellectual endeavor that traces its roots back to the time of Aristotle's ancient Greece. Motivated by the desire to establish a rigorous reasoning process, in his "Organon," Aristotle introduced principles of logic, in particular, syllogism, a form of logical argument that applies deductive reasoning to arrive at a conclusion based on two or more propositions assumed to be true. In disciplines that rigorous reasoning is critical, such as judicial reasoning and mathematical problem solving, documents must be written in a formal language with a logical structure to ensure the validity of the reasoning process.
\*Equal contribution. Qualcomm AI Research is an initiative of Qualcomm Technologies, Inc All datasets and models were solely downloaded and evaluated by the University of California San Diego.
Figure 1: An overview of our proposed deductive reasoning and verification process. In response to an input question, LLMs generate deductive reasoning chains using the Natural Program format (bottom 3 boxes), a natural language-based deductive reasoning approach. The Natural Program format allows individual reasoning steps (an example in purple) and their corresponding minimal set of premises (an example in orange) to be easily extracted. This streamlined extraction process facilitates the step-by-step decomposition and verification of deductive reasoning (top-right box).
Figure 2: Through our Natural Program-based deductive reasoning verification approach, we identify and eliminate reasoning chains that contain errors in reasoning and grounding (we define grounding error as utilizing information that is not present in cited premises). By alleviating such errors, we significantly enhance the rigor, trustworthiness, and interpretability of the generated reasoning outputs.
We yearn for this sequence of reliable knowledge when answering questions. Our goal is to develop language models that can propose potential solutions through reasoning in logical structures. Simultaneously, we aim to establish a verifier capable of accurately assessing the validity of these reasoning processes. Despite recent significant explorations in the field, such as [48]'s emphasis on self-consistency and [27, 5]'s innovative use of codes to represent the reasoning process, these approaches still exhibit considerable limitations. For example, consistency and reliability are not inherently correlated; as for program codes, they are not powerful enough to represent many kinds of reasoning process, e.g., in the presence of quantifiers ("for all", "if there exists") or nuances of natural language (moral reasoning, "likely", ...).
We propose leveraging the power of natural language to achieve the deductive reasoning emphasized in ancient Greek logic, introducing a "natural program". This involves retaining natural language for its inherent power and avoiding the need for extensive retraining with large data sets. A natural program
represents a rigorous reasoning sequence, akin to a computer program. We expect implementations of the idea to have two properties: 1) that natural programs are generated with minimal effort from an existing language model capable of CoT reasoning, preferably through in-context learning; 2) that the natural program can be easily verified for reliability in the reasoning process.
Through a step-by-step investigation, we discovered that large language models have the potential to meet our expectation. Naive CoT prompts like "Let us think step by step." has many flaws, and entrusting the entire verification process to a large model like ChatGPT can still lead to significant error rates. However, we found that, if the reasoning process is very short, and only based on necessary premises and contexts, the verification of existing large language models is already quite reliable. Therefore, our approach is to design prompts that induce CoT processes comprised of rigorous premises/conditions and conclusions with statement labels, and verification can be done by gradually isolating very few statements within the long thought chain. Experimentally, we found that most reasoning that passed the verification was rigorous, and many that did not pass had elements of imprecision in the reasoning process, even if they occasionally arrived at correct answers.
It is worth emphasizing that, we are not looking for a method to just maximize the correctness rate of final answers; instead, we aspire to generate a cogent reasoning process, which is more aligned with the spirit of judicial reasoning. When combined with sampling-based methods, our method can identify low-probability but rigorous reasoning processes. When repeated sampling fails to yield a rigorous reasoning process, we can output "unknown" to prevent hallucinations that mislead users.
We demonstrate the efficacy of our natural program-based verification approach across a range of arithmetic and common sense datasets on publicly-available models like OpenAI's GPT-3.5-turbo. Our key contributions are as follows:
1. We propose a novel framework for rigorous deductive reasoning by introducing a "Natural Program" format (Fig. 1), which is suitable for verification and can be generated by just in-context learning;
2. We show that reliable self-verification of long deductive reasoning processes written in our Natural Program format can be achieved through step-by-step subprocesses that only cover necessary context and premises;
3. Experimentally, we demonstrate the superiority of our framework in improving the rigor, trustworthiness, and interpretability of LLM-generated reasoning steps and answers (Fig. 2).
Reasoning with large language models. Recent large language models (LLMs) [3, 8, 57, 47, 38, 18, 9, 37] have shown incredible ability in solving complex reasoning tasks. Instead of letting LLMs directly generate final answers as output, prior work have shown that by encouraging step-by-step reasoning through proper prompting, such as Chain-of-Thought (CoT) prompting [50] and many others [21, 59, 58, 44, 48, 60, 25, 54], LLMs exhibit significantly better performance across diverse reasoning tasks. To further improve the step-by-step reasoning process, some recent studies have investigated leveraging external solvers such as program interpreters [39, 5, 27], training and calling external reasoning modules [11], or performing explicit search to generate deductive steps [2, 46]. Parallel to these works, we do not rely on external modules and algorithms, and we directly leverage the in-context learning ability of LLMs to generate more precise and rigorous deductive reasonings.
Large language models as verifiers. Using language models to evaluate model generations has been a long standing idea [22, 36, 40, 4]. As LLMs exhibit impressive capabilities across diverse tasks, it becomes a natural idea to use LLMs as evaluation and verification tools. For example, [10, 11, 33] finetune LLMs to verify solutions and intermediate steps. LLMs aligned with RLHF [32, 31, 48] have also been employed to compare different model generations. In addition, recent works like [43, 52, 28, 6] leverage prompt designs to allow LLMs to self-verify, self-refine, and self-debug without the need for finetuning. However, these works do not focus on the rigor and trustworthiness of the deductive reasoning processes at every reasoning step. In this work, we propose a natural language-based deductive reasoning format that allows LLMs to self-verify every intermediate step of a deductive reasoning process, thereby improving the rigor and trustfulness of reasoning.
Question: Jerry is rolling a six-sided die. How much more likely is it (expressed as a percentage) that he rolls a number greater than $3 than that he rolls two even numbers in a row?
Ground Truth: There are 3 numbers greater than 3 on the dice, so the chances of rolling one of them are  $\frac{3}{6} = 50\%$ . The chance of rolling one even number is  $50\%$ , so the chance of rolling two in a row is  $50\% * 50\% = 25\%$ . The difference between these two probabilities is  $50\% - 25\% = 25\%$ .
Output: There are two possible outcomes for rolling a number greater than 3 out of 6 total outcomes, so the probability is  $2 / 6$  or  $1 / 3$ . For rolling two even numbers in a row, there are three possible outcomes (2-4, 4-6, 6-2) out of 36 total outcomes, so the probability is  $3 / 36$  or  $1 / 12$ . To find the difference in probability, we subtract the probability of rolling two even numbers in a row from the probability of rolling a number greater than 3:  $1 / 3 - 1 / 12 = 1 / 4$ . This means that it is  $25\%$  more likely to roll a number greater than 3 than to roll two even numbers in a row.
Table 1: An example question from GSM8K with a generated CoT reasoning path with GPT-3.5 (turbo), where the output provides a wrong reasoning chain with the correct answer.
Additionally, while some recent works [12, 53, 15, 34] have proposed methods to verify individual steps in a reasoning process, our approach distinguishes from these works in the following perspectives: (1) Our approach leverages in-context learning to achieve reasoning verification, without the need for language model finetuning. (2) Our Natural Program-based LLM verification approach not only identifies invalid reasoning steps, but also provides explicit explanations for why they are invalid, detailing the specific reasoning errors involved. (3) Our Natural Program-based reasoning and verification approach is compatible with in-context abstract reasoning tasks where reasoning steps do not possess proof-like entailment structures. For example, our approach is compatible with the Last Letters task, where the LLM is instructed to output the concatenation of the last letters of all words in a sequence as the final answer. (4) Our Natural Program approach allows the use of commonsense knowledge not explicitly listed in premises. For example, consider this problem: "Marin eats 4 apples a day. How many apples does he eat in November?" Even though "November has 30 days" is not explicitly listed in the premises, Natural Program permits the use of such common knowledge within a reasoning step. Our in-context verification process is also capable of handling these implicit premises (e.g., if LLM outputs "November has 29 days" in a reasoning step, it will be marked as invalid).
3 Motivation and Problem Formulation
A reasoning-based question-answering (QA) task can be defined as a tuple  $(Q, C, O, A)$  [35], where  $Q$  is the target question;  $C$  is the context of a question, such as the necessary background for answering a question;  $O = (o_1, o_2, \dots, c_k)$  are optional answer choices if  $Q$  is a K-way multiple choice problem; and  $A$  is the ground-truth answer. Given  $Q$  and  $C$  as inputs, large language models (LLMs) [3, 8, 47] generate a sequence of tokens  $T = (t_1, t_2, \dots, t_n)$  to answer the question. Recent works like Chain-of-Thought (CoT) [50, 21] leverage prompt engineering in the context  $C$  to encourage models to generate the intermediate reasoning process in  $T$ , which benefits LLM performance across diverse reasoning tasks. In this case,  $T$  consists of a set of  $m$  intermediate reasoning steps, which we denote as  $S = (s_1, s_2, \dots, s_m)$ . Each step  $s_i$  can be represented by a subsequence of the generated tokens  $(t_{l_i}, t_{r_i}) \subseteq T$ . In much prior work, a generated solution is "correct" if and only if the predicted final answer in  $s_m$  matches the ground truth  $A$ , which we call answer correct(ness).
We observe that for all cases where LLMs produce erroneous final answers, there exists at least one mistake among the intermediate reasoning steps  $S$ . Moreover, even when the final answer is correct, there might still exist some mistakes among  $S$ . This phenomenon, as illustrated in Tab. 1, occurs for all LLMs we tested, including state-of-the-art models such as ChatGPT and GPT-4 [32]. Since later reasoning steps are conditioned on prior reasoning steps, these mistakes often initiate a snowball effect, causing subsequent mistakes to compound. This significantly diminishes the likelihood of correct problem-solving and impedes the progress towards achieving human-level complex reasoning.
Therefore, in this work, we place significant emphasis on ensuring the validity of every reasoning step, not just the correctness of the final answer. In particular, we focus on the validity of deductive reasoning, an essential component of a logical reasoning process. In deductive reasoning, we are
Prompting	Reasoning Correctness	GSM8K	AQuA	MATH	AddSub	Date	Last Letters
Zero-shot	Correct	98%	96%	100%	98%	98%	100%
Incorrect	4%	6%	4%	2%	4%	4%
(Average)	51%	51%	52%	50%	51%	52%
Two-shot	Correct	98%	96%	100%	92%	100%	96%
Incorrect	2%	4%	0%	6%	26%	6%
(Average)	50%	50%	50%	49%	63%	51%
Table 2: Zero-shot and two-shot reasoning chain verification accuracy for GPT-3.5-turbo (ChatGPT), where an entire reasoning chain is verified at once. The two shot prompt we used is presented in Appendix D.1. To generate verification inputs, for each dataset, we perform Chain-of-Thought (CoT) prompting and randomly sample 50 reasoning chains that are valid and 50 reasoning chains that exhibit mistakes. We observe that when given an entire reasoning process, where the deductive graphs for all reasoning steps are entangled together, it is challenging even for strong language models like ChatGPT to verify its validity.
given a (premise, conclusion) pair, and we are interested in determining whether the conclusion follows from the premises. In the context of reasoning-based QA tasks, for each reasoning step  $s_i$ , we define its deductive validity  $V(s_i)$  as a binary variable. A reasoning step is deductively valid  $V(s_i) = 1$  if and only if  $s_i$  can be logically deduced from its corresponding premises  $p_i$ , which consist of the context  $C$ , the question  $Q$ , and all the previous reasoning steps  $s_j (j < i)$ . Then, we can also define the deductive validity for the entire reasoning chain  $S$  as  $V(S) = \wedge_{i=1}^{M} V(s_i)$ . Compared to evaluating answer correctness, which can be accomplished by simple functions such as exact string match, evaluating deductive validity is a lot more challenging. Thanks to the recent progress on LLMs, which demonstrate impressive in-context learning capabilities across diverse scenarios, we propose to use LLMs to examine reasoning chains and predict the deductive reasoning validity.
4 Deductively Verifiable Chain-of-Thought Reasoning
In this section, we introduce our specific approaches to performing deductive verification of reasoning chains. Specifically, we first introduce our motivation and method for decomposing a deductive verification process into a series of step-by-step processes, each only receiving contexts and premises that are necessary. Then, we propose Natural Program, a natural language-based deductive reasoning format, to facilitate local step-by-step verification. Finally, we show that by integrating deductive verification with unanimity-plurality voting, we can improve the trustworthiness of reasoning processes along with final answers. An overview of our approach is illustrated in Fig. 1 and Fig. 2.
4.1 Decomposition of Deductive Verification Process
Given a reasoning chain  $S = (s_1, s_2, \dots, s_n)$ , a straightforward idea to verify its deductive validity is to ask LLMs to examine the entire reasoning chain at once. To assess the effectiveness of this approach, we conduct a preliminary experiment: for a dataset problem and its reasoning chain  $S$  generated by ChatGPT, we prompt ChatGPT with "Do you think the above reasoning process is correct? Let's think step by step" such that its outputs whether there exists any mistake among any reasoning step in  $S$ . However, as demonstrated in Tab. 2, the verification accuracy is  $50\%$  for most datasets, and ChatGPT struggles at finding out mistaken reasonings. Notably, it persistently outputs "Correct" for most reasoning chain queries, regardless of their actual validity.
We conjecture that such phenomenon is caused by the abundance of irrelevant premises for each reasoning step. Recall that the premises  $p_i$  for a reasoning step  $s_i$  consist of the question  $Q$ , the question context  $C$ , along with the prior reasoning steps  $s_{\leq j} = \{s_j : j < i\}$ . For  $Q$  and  $C$ , we can further extract and decompose  $Q \cup C$  into a set of "question-related premises"  $QC = \{qc_1, qc_2, \dots, qc_m\}$ , where  $qc_i$  is a premise or condition inferred from  $Q \cup C$ . Then, it is often the case that most elements of  $p_i = QC \cup s_{\leq j}$  are irrelevant to the validity of  $s_i$ , leading to erroneous verifications from language models. A very recent work [41] also observes a similar phenomenon where LLMs are easily distracted by irrelevant context.
Hence, we propose a decomposition of the reasoning chain verification process into a series of step-by-step processes, where each step only considers the premises that are necessary. The overall validity
of the reasoning chain, denoted as  $V(S) = \wedge_{i=1}^{M} V(s_i)$ , can be naturally decomposed into individual step validity  $V(s_i)$ . However, achieving such decomposition is highly challenging without imposing constraints on the format of reasoning chains. Additionally, for each  $s_i \in S$ , we aim to ensure that it explicitly lists the minimal subset of premises  $\bar{p}_i \subseteq p_i$  required for deductive reasoning to avoid potential ambiguities during verification. This motivates us to introduce a natural-language-based deductive reasoning format in Section 4.2.
4.2 Natural Program Deductive Reasoning Format
As previously mentioned in Sec. 4.1, we desire LLMs to output deductive reasoning processes that can be easily verified by themselves, specifically by listing out the minimal set of necessary premises  $p_i$  at each reasoning step  $s_i$ . To accomplish its goal, we propose to leverage the power of natural language, which is capable of rigorously representing a large variety of reasoning processes and can be generated with minimal effort. In particular, we introduce Natural Program, a novel deductive reasoning format for LLMs. More formally, Natural Program consists of the following components:
- An instruction for models to extract question-related premises  $QC$ . We use the following instruction: "First, let's write down all the statements and relationships in the question with labels".
- A numbered-list of question-related premises, each prefixed with "#{premise_number}".
- An instruction for models to generate the reasoning chain  $S$  based on the question-related premises  $QC$ . We use the following instruction: "Next, let's answer the question step by step with reference to the question and reasoning process".
- A list of prefixed reasoning steps  $S_{i}$ . The prefix has the following format:  $\# \{number\}$  (by {list_of_premises_used}). Here "number" equals  $|QC| + i$ , and "list_of_premises_used" consists of numbers from the smallest subset of premises among  $QC \cup s_{\leq j}$  that are used for the deductive reasoning of  $s_{i}$ . In addition, for the last reasoning step  $s_{m}$ , we ensure that it (1) includes a special tag Final Step; (2) refers to the premise number of the target question to be answered; (3) explicitly gives the final answer to a question.
To encourage language models to reason in the Natural Program format, we have designed one-shot prompts for different datasets, which are shown in Appendix D.2. Given that LLM's reasoning outputs follow the Natural Program format, we can then verify the deductive validity of a single reasoning step  $s_i$  through an instruction that consists of (1) the full descriptions of premises used for the reasoning of  $s_i$ ; (2) the full description of  $s_i$ ; (3) an instruction for validity verification, such as "Double-check the reasoning process, let's analyze its correctness, and end with "yes" or "no". Note that throughout this verification process, we only retain the minimal necessary premise and context for  $s_i$ , thereby avoiding irrelevant context distraction and significantly improving the effectiveness of validation. Additionally, we employ a one-shot prompt for this verification process, which we find very helpful for improving the verification accuracy. The prompt is shown in Appendix D.3.
Figure 1 provides an overview of the complete Natural Program-based deductive reasoning and verification process. By using the Natural Program approach, we demonstrate that LLMs are capable of performing explicit, rigorous, and coherent deductive reasoning. Furthermore, Natural Program enables LLMs to self-verify their reasoning processes more effectively, enhancing the reliability and trustworthiness of the generated responses.
4.3 Integrating Deductive Verification with Unanimity-Plurality Voting
Given that we can effectively verify a deductive reasoning process, we can naturally integrate verification with LLM's sequence generation strategies to enhance the trustworthiness of both the intermediate reasoning steps and the final answers. In this work, we propose Unanimity-Plurality Voting, a 2-phase sequence generation strategy described as follows. Firstly, similar to prior work like [48], we sample  $k$  reasoning chain candidates along with their final answers. In the unanimity phase, we perform deductive validation on each reasoning chain. Recall that a chain  $S$  is valid (i.e.,  $V(S) = 1$ ) if and only if all of its intermediate reasoning steps are valid (i.e.,  $\forall i, V(s_i) = 1$ ). For each intermediate reasoning step  $s_i$ , we perform majority voting over  $k'$  sampled single-step validity