SlowGuess/NeurIPS_2023_OCR huggingface dataset:
3. Lastly, we use regular expressions, as outlined in Tab. 11, to scan the remaining text for possible final answers. If multiple matches are found for the pattern, we select the first match as the final answer. If no pattern matches are found in the remaining text, we default the final response to "No answer!".
"No-Answer" Patterns: "we cannot provide an answer to this question with (this|the) given information", "we cannot answer (this|the) question", "we cannot determine", "we can't determine", "we do not have enough information to answer (this|the) question", "we do not have enough information to provide a definitive answer to (this|the) question", "the answer(.*?)is unknown", "answer is not listed among the answer choices".
"Answer-Split" Patterns: "answer is", "final answer:", "answer to the question is", "answer to this question is", "concatenated letters are", "concatenate the letters - ", "The answer of".
Answer Type	Regular Expression
Number	(-?\\d[\\d, .]*)
Fractional number	(-?\\(d+\\d+\\)\\d+|-?\\d+\\d+)
Date	\( (\backslash d\backslash d\backslash \backslash d\backslash \backslash d\backslash d\backslash d) \)
Yes or No	(?!:Yes|No|yes|no|NO|YES)
Table 11: Regular Expression for extracting the final answers of different kinds of questions.
D.1 Prompt for Direct Reasoning Chain Verification Without Natural Program Format
For the results in Tab. 2 of the main paper, We use "Do you think the above reasoning process is correct? Let's think step by step." as the zero-shot prompt to verify an entire reasoning chain at once. We also design a two-shot prompt for reasoning chain verification as shown in Tab. 12, which covers one correct reasoning chain and one incorrect reasoning chain.
D.2 Prompts for Reasoning Chain Generation in the Natural Program Format
To instruct models to generate reasoning chains in the Natural Program format that facilitates step-by-step deductive verification, we have designed four distinct prompts to address different types of problems. These include:
1. Math word problems, as illustrated in Tab. 13, covering GSM8K, MATH, and AddSub datasets.
2. Math word problems with multiple-choice options, illustrated in Tab. 14, covering the AQuA dataset.
3. Date-related problems, illustrated in Tab. 15, covering the Date dataset.
4. Last Letters problems, illustrated in Tab. 16, covering the Last Letters dataset.
D.3 Prompt for Deductive Verification Following Natural Program Format and Step-by-Step Decomposition
We have designed a general one-shot prompt for the deductive verification of a single reasoning step on different datasets, as shown in Tab. 17. This prompt serves to instruct language models to generate the deductive validity of each reasoning step as illustrated in Sec. 4.2 and the top-right box of Fig. 1 of the main paper.
E More Deductive Verification Examples
In this section, we present more deductive verification examples using our Natural Program-based approach on single reasoning steps.
In Tab. 18, we demonstrate that the language model (ChatGPT) not only successfully identifies ungrounded information, but also identifies logical errors within the given solutions.
In Tab. 19, we illustrate a case where the language model fails to detect ungrounded premise numbers, mistakenly assuming that these numbers can be derived from grounded ones.
Lastly, in Tab. 20, we illustrate a case where the language model is sometimes unable to correctly identify grounded numbers.
Here is a math question and its solution:
"Question: Jason had 20 lollipops. He gave Denny some lollipops. Now Jason has 12 lollipops. How many lollipops did Jason give to Denny?
Answer: Jason started with 20 lollipops. Then he had 12 after giving some to Denny. So he gave Denny  $20 + 12 = 34$ . The answer is 34."
You are a math teacher. Do you think the reasoning process is correct? Let's check the "Answer" step by step, and then end with "yes" or "no".
Answer: The problem statement is clear, but the reasoning process has a major error in it. Let's go through it step by step:
1. "Jason started with 20 lollipops." This is correct according to the problem statement.
2. "Then he had 12 after giving some to Denny." This is also correct as per the problem.
3. "So he gave Denny  $20 + 12 = 34$ . The answer is 34." This is incorrect. The correct reasoning process should be subtracting the number of lollipops he ended up with from the number he started with. That would be  $20 - 12 = 8$  lollipops.
So, Jason gave Denny 8 lollipops, not 34. No, the reasoning process is incorrect.
"Question: Olivia has  $23. She bought five bagels for$ 3 each. How much money does she have left?
Answer: Olivia had 23 dollars. 5 bagels for 3 dollars each will be  $5 \times 3 = 15$  dollars. So she has 23 - 15 dollars left. 23 - 15 is 8. The answer is 8."
Let's go through this reasoning process step by step.
1. "Olivia had 23 dollars." This is the given starting amount of money that Olivia has. This step is correct.
2. "5 bagels for 3 dollars each will be  $5 \times 3 = 15$  dollars." This is correct as well. Olivia spends 3 dollars for each bagel, and she buys 5 of them. So the total cost of the bagels is 3 dollars/bagel x 5 bagels = 15 dollars.
3. "So she has 23 - 15 dollars left." This statement is also correct. After purchasing the bagels, Olivia will have her initial amount of money minus the cost of the bagels.
4. "23 - 15 is 8. The answer is 8." This step is correct as well. 23 dollars (Olivia's initial amount) - 15 dollars (the cost of the bagels) = 8 dollars. This means that Olivia has 8 dollars left after buying the bagels.
So overall, Olivia has 8 dollars left after buying five bagels for 3 dollars each. Yes, the reasoning process is correct.
Table 12: Two-shot prompt for direct reasoning chain verification without Natural Program format.
Alice buys 5 apples. Each apple costs her $2. She gets them for  $50\%$  off. After she bought she decides she doesn't like 2 of them and sells them to her friends for 1. How much money was she out?
First, let's write down all the statements and relationships in the question with labels.
1. Alice gets 5 apples.
#2. Each apple costs $2, but Alice gets apples for 50% off.
3. Alice sells 2 of 5 apples for 1.
4. How much money was Alice out?
Next, let's answer the question step by step with reference to the question and reasoning process:
#5. (by #2) Step 1: Calculate the price per apple after a  $50\%$  discount.
Original price per apple: $2
Discount:  $50\%$
Price per apple after discount: $2 * (1 - 0.50) = $2 * 0.50 = $1
#6. (by #1 #5) Step 2: Calculate Alice's cost of the 5 apples after a  $50\%$  discount.
Price per apple after discount: $1
Total apples: 5
Alice's total cost of 5 apples: $1 * 5 = $5
#7. (by #3) Step 3: Calculate the money Alice gets back from selling 2 apples.
Money that Alice gets back from selling 2 apples: $1
#8. (by #6 #7) Step 4: Calculate the total amount of money Alice is out.
Alice's total cost of 5 apples: $5
Money that Alice is out: $5 - $1 = $4
#9. (by #4 #8) The original question is #4.
How much money was Alice out? We do not miss information on the rewritten labels. So the answer to this question is Alice is out $4.
Please answer the following question by following the above template.
Table 13: One-shot Natural Program prompt for reasoning chain generation on math word problems.
Alice buys 5 apples. Each apple costs her $2.  
She gets them for  $50\%$  off. After she bought She decides she doesn't like 2 of them and sells them to her friends for 1.  
How much money was she out?  
Answer Choices:  
A) 4  
B) 2  
C) None of the above
1. Alice gets 5 apples.
#2. Each apple costs $2, but Alice gets apples for 50% off.
3. Alice sells 2 of 5 apples for 1.
4. How much money was Alice out?
#9. (by #4 #8) The original question is #4. How much money was Alice out? We do not miss information on the rewritten labels. So the answer to this question is Alice is out $4. Among all the answer choices, the best option is A) 4.
Table 14: One-shot Natural Program prompt for reasoning chain generation on math word problems with multiple choice.
Let's work on date computation.
Yesterday is 02/22/2012. What is the date one week from today? Please answer in format MM/DD/YYYY.
1. Yesterday is 02/22/2012.
2. What is the date one week from today? Please answer in format MM/DD/YYYY.
4. (by #1) Step 1: Calculate the date today.
The date of Yesterday is 02/22/2012.
The date of today is 02/23/2012.
#5. (by #2 #4) Step 2: Calculate the date one week from today and write the answer in MM/DD/YYYY.
Because 2012 is a leap year and February in 2012 has 29 days, the date one week from today is 03/01/2012.
Therefore, date one week from today is 03/01/2012.
Today is 3/2 and Bob is watching NBA 2007. What is the date one week before today?
Please answer in format MM/DD/YYYY.
1. Today is 3/2.
2. Bob is watching NBA 2007 today.
#3. What is the date one week before today? Please answer in format MM/DD/YYYY.
#4. (by #1 #2) Step 1: Calculate the date today.
Today is 3/2 and Bob is watching NBA 2007 today. Thus, Today is 03/02/2007.
#5. (by #3 #4) Step 2: Calculate the date one week before today and write the answer in MM/DD/YYYY.
Because 2007 is not a leap year and February in 2007 has 28 days, the date one week from today is 02/23/2007.
Therefore, the date one week before today is 02/23/2007.
Table 15: Two-shot Natural Program prompt for reasoning chain generation on the Date dataset.
Take the last letters of each word in "Apple Is" and concatenate them.
1. We need to take the last letter of the word "Apple".
2. We need to take the last letter of the word "Is".
#3. We need to concatenate the last letters of the words sequentially
Next, let's answer the question step by step with the complete reference to the question and reasoning process:
#4. (by #1, #3) Step 1: Taking the last letter of the word "Apple" and concatenate the latter.
The last letter of the word "Apple" is "e".
The concatenated last letters are "e".
#5. (by #2, #3, #4) Step 2: Taking the last letter of the word "Is" and concatenate the latter.
The last letter of the word "Is" is "s".
The concatenated last letters are "es".
6. (by #3, #5) Step 3: Provide the final answer of the concatenated letter.
Therefore, the answer is "es".
Table 16: One-shot Natural Program prompt for reasoning chain generation on the Last Letters dataset.
Here is some information:
"1. Alice gets 2 apples.
2. Alice gets twice as many oranges as bananas.
3. Original price per apple: $2
Price per apple after discount: $2 * (1 - 0.50) = $2 * 0.50 = $1"
Based on the given information, here is a reasoning process:
"Calculate Alice's cost of the 5 apples after a  $50\%$  discount.
Apples be eaten: 3
Oranges be eaten: 6
Total oranges: x
Total bananas: 2x
Alice's total number of fruits:  $5 + x + 2x = 5 + 4x$
Alice's total cost of 5 apples: $1 * (5 - 3) = $2"
Double-check the reasoning process, let's analyze its correctness, and end with "yes" or "no".
Let's think step by step.
Purpose: Find references for numbers in the reasoning process sequentially
Rule: Any ungrounded number makes the check fails
"x" is a variable, we do not care about variables.
"y" is a variable, we do not care about variables.
"5" is a quantity from the given information that represents the number of apples.
"50%" is a quantity from the given information that represents the discount.
"\(1" is a quantity from the given information that represents the price per apple after the discount.
"3" is a quantity not from the given information, it is ungrounded.
"6" is a quantity not from the given information, it is ungrounded.
" $\$ 2$  " is a quantity from the given information that represents the original price per apple.
Result: Based on the check, there are ungrounded numbers, so the grounding check fails.
Reasoning check:
Purpose: Check whether the reasoning logic correctly answers the question
Rule: Any logic error makes the check fails
To calculate the total cost of apples after a  $50\%$  discount, we should multiply the number of apples by the discounted price. But to answer the question, we do not need to calculate the total number of fruit Alice gets.
Result: Based on the check, there are some logic mistakes, so the reasoning check fails.
Calculation check:
Purpose: Check whether the calculation process is consistent
Rule: Any inconsistent calculation makes the check fails
calculation1:
equation: $1 * (5 - 3), answer: $2
$(5 - 3) = 2$
$1 * 2 = \$ 2$ is consistent with the answer, so the calculation is correct.
calculation2:
equation:  $5 + x + 2x$  , answer:  $5 + 4x$
$\mathrm{x} + 2\mathrm{x} = 3\mathrm{x}$
$5 + 3x$  is inconsistent with the answer, so the calculation is incorrect.
Result: Based on the check, the calculation process is inconsistent, so the calculation check fails.
Check results: Ground check fails, Reasoning check fails, Calculation check fails.
Rule: Any failed check makes the reasoning incorrect.
So the answer is "no".
Table 17: One-shot prompt for deductive verification of a single reasoning step, following our Natural Program format and step-by-step reasoning chain decomposition.
"For every 25 telephone calls he makes, he gets one person to come into the dealership."
"Calculate the number of people that would come into the dealership for 15 car sales.
Number of people needed to sell 15 cars:  $15 * 2 = 30$
Let's think step by step without any assumptions.
Grounding check:
"15" is a quantity from the question that represents the number of car sales.
"2" is a quantity not from the given information, it is ungrounded.
"25" is a quantity from the given information that represents the number of telephone calls per one person to come into the dealership.
It is not clear how the number of people coming into the dealership relates to the number of car sales.
The given information only tells us the ratio of telephone calls to people coming into the dealership, but it does not provide any information about the number of telephone calls made for each car sale.
Therefore, the reasoning process is not correct.
Result: Based on the check, there is a logic mistake, so the reasoning check fails.
equation:  $15*2$  , answer: 30
$15 * 2 = 30$  is consistent with the answer, so the calculation is correct.
Result: Based on the check, the calculation process is consistent, but the reasoning is incorrect, so the calculation check fails.
Check results: Grounding check fails, Reasoning check fails, Calculation check fails.
Table 18: Successful case: our deductive verification approach successfully discovers ungrounded information and reasoning mistakes.
"How many hours would it take for 5 people to paint a whole house?"
"Calculate the total number of hours it would take for 5 people to paint a whole house.
Total hours for one person to paint a whole house: 240 hours
Total number of people: 5
Total hours for 5 people to paint a whole house: 240 hours / 5 = 48 hours"
"240" is a quantity not from the given information, it is ungrounded.
"5" is a quantity from the given information that represents the number of people.
"48" is a quantity calculated from the ungrounded number and grounded number in the reasoning process.
Result: Based on the check, there is one ungrounded number, but it is calculated from another grounded number, so the grounding check passes.
The reasoning process correctly calculates the total number of hours for 5 people to paint a whole house by dividing the total hours for one person to paint a whole house by 5. Therefore, the reasoning check passes.
equation: 240 / 5, answer: 48
$240 / 5 = 48$  is consistent with the answer, so the calculation is correct.
Result: Based on the check, the calculation process is consistent, so the calculation check passes.
Check results: Ground check passes, Reasoning check passes, Calculation check passes.
Rule: All checks pass make the reasoning correct.
Table 19: Failure case: our deductive verification process fails to find out ungrounded information in the reasoning step. The number 240 in the reasoning step is ungrounded, but the model states that it can be calculated from grounded numbers.
Input:
Here is some information:
"Calculate the number of doughnuts Derrick makes in a day.
Number of doughnuts in a dozen: 12
Number of doughnuts in 10 dozens: 10 * 12 = 120
Calculate the revenue from selling one doughnut.
Price per doughnut: $2"
Based on the given information, here is a reasoning process:
"Calculate the revenue from selling all the doughnuts in a day.
Number of doughnuts in a day: 120
Revenue from selling all the doughnuts in a day: $2 * 120 = $240"
Double-check the reasoning process, let's analyze its correctness, and end with "yes" or "no".
Answer:
Let's think step by step without any assumptions.
Output:
Grounding check
Purpose: Find references for numbers in the reasoning process sequentially
Rule: Any ungrounded number makes the check fails
"12" is a quantity from the given information that represents the number of doughnuts in a dozen.
"10" is a quantity not from the given information, it is ungrounded.
"\$2" is a quantity from the given information that represents the price per doughnut.
"120" is a quantity not from the given information, it is ungrounded.
Result: Based on the check, there are ungrounded numbers, so the grounding check fails.
Reasoning check:
Purpose: Check whether the reasoning logic correctly answers the question
Rule: Any logic error makes the check fails
The reasoning process is correct. We have the number of doughnuts Derrick makes in a day, and we have the price per doughnut.
Multiplying these two yields the revenue from selling all the doughnuts in a day.
Result: Based on the check, the reasoning logic is correct.
Calculation check:
Purpose: Check whether the calculation process is consistent
Rule: Any inconsistent calculation makes the check fails
calculation1:
equation: 10 * 12, answer: 120
10 * 12 = 120 is consistent with the answer, so the calculation is correct.
calculation2:
equation: $2 * 120, answer: $240
$2 * 120 = $240 is consistent with the answer, so the calculation is correct.
Result: Based on the check, the calculation process is consistent, so the calculation check passes.
Check results: Ground check fails, Reasoning check passes, Calculation check passes.
Rule: Any failed check makes the reasoning incorrect.
So the answer is "no".
Table 20: Failure case: our deductive verification process sometimes treats grounded information as if they were ungrounded. The number 120 is provided in the given information, but the model states that it is ungrounded.