SlowGuess/NeurIPS_2023_OCR huggingface dataset:
To examine the influence of incorporating faithfulness on LLM final predictions, we plot the distributions of changes in different scores, specifically the faithfulness score  $\mathcal{C}$ , the generation confidence  $\mathcal{P}$ , and the overall decoding score  $\varepsilon_{\lambda}$  on the baseline reasoning chains and the reasoning chains generated by our method. We categorize the data points into 4 sets based on whether our approach changes the final prediction. Since the majority of the data points belong to the "both correct" set (in blue), where both baselines and our method generate accurate predictions, we particularly highlight the last two sets (in green and red), where our method results in improvement and degradation, respectively.
As shown in Figure 11, faithfulness typically works by significantly increasing the evaluation confidence  $\mathcal{C}$  of model predictions, while the generation confidence  $\mathcal{P}$  remains similar to that of the baseline methods. Specifically, for the evaluation confidence  $\mathcal{C}$ , our approach corrects the original predictions by increasing the confidence scores. This indicates that evaluation confidence plays a crucial role in guiding the decoding toward a better reasoning choice in decomposed reasoning. The increase is more significant for PAL when compared with CoT. This demonstrates that LLMs are generally better at judging the logic in reasoning that is more structured, while the free-text intermediate steps (e.g., CoT reasoning) may be challenging to conduct information extraction and soundness checking.
A similar conclusion can be drawn from Figure 10, where the difference in instance-level accuracy distributions between high-scored and low-scored predictions is more significant on the GSM8K dataset. For StrategyQA, while the incorporation of faithfulness helps, the level of the score value does not align well with whether the prediction is correct. For example, most of the low-scored predictions can still obtain the correct answers, as shown by the plot on the right of Figure 10b.
(a) Distributions of score shifts on GSM8K using PAL prompting.
(b) Distributions of score shifts on GSM8K using CoT prompting.
(c) Distributions of score shifts on StrategyQA using CoT prompting.
Figure 11: Distributions of changes in scores from baselines to our method. Since the prediction correctness keeps unchanged most of the time (i.e., "both correct/incorrect" in blue/orange), we specifically plot how the scores shift on data points where the predictions get corrected or incorrect, as shown in green and red, respectively.
Table 5: Impact of LLM backends (Codex vs. ChatGPT vs. GPT-4) and prompting methods (PAL vs. CoT). The results of ChatGPT (gpt-3.5-turbo) were obtained on 20 March 2023.
Model	GSM8K	StrategyQA
CoT Codex	65.6	73.2
PAL Codex	72.0	-
CoT ChatGPT	80.8	65.9
PAL ChatGPT	78.7	-
CoT GPT-4	92.0	-
Ours (CoT) Codex	71.9	77.2
Ours (PAL) Codex	80.2	-
LLM Backbone Study. We are interested in how stronger LLMs (i.e., ChatGPT, GPT-4 (OpenAI, 2023)) work, but they are not directly compatible with our approach since the API does not return token logits.
Table 5 compares the results of various backend LLMs (i.e., Codex, ChatGPT, and GPT-4) on GSM8K. In arithmetic reasoning with PAL prompting, our Codex-based method achieves competitive results (80.2% vs. 78.7%) even when compared with ChatGPT. The results are consistent across other datasets, including AQuA (55.9% vs. 54.7%), SVAMP (89.6% vs. 84.1%), ASDiv (84.9% vs. 84.1%), and TabMWP (79.1% vs. 80.6%). In commonsense reasoning, our method using Codex significantly outperforms ChatGPT-based methods across different datasets, including StrategyQA (77.2% vs. 65.9%), CommonsenseQA (78.6% vs. 75.2%) and Sports Understanding (98.4% vs. 95.9%). One possible explanation is that ChatGPT lacks sufficient world knowledge for effective fact checking and commonsense reasoning. Given the significant performance improvement of GPT-4, we conduct further analysis about how to synergistically combine it with our method.
GPT-4 Experiments The recently launched GPT-4 has demonstrated notable improvements in reasoning capabilities across a variety of tasks. In this section, we examine and compare the reasoning skills of different large language models (LLMs), specifically Codex and GPT-4, in assessing and determining the accuracy of each step in a reasoning chain. We contrast the confidence scores and corresponding explanations for Codex  $(\mathcal{C})$  and GPT-4  $(S)$  in the context of both arithmetic and commonsense reasoning, as shown in Figure 13 and Figure 14, respectively. For ease of visualization, we employ the same colormap (shown in Figure 12) as in Figure 7 to represent the scale of scores. Since OpenAI has not provided access to the token-wise likelihood of generated text, we directly request GPT-4 to score the reasoning steps using binary values  $^{8}$ . Moreover, we report the average of three evaluation results to reduce the variance of sampling discrete values, i.e.,  $S = (S_{1} + S_{2} + S_{3}) / 3$ ,  $S_{i} \in [0,1]$ .
As illustrated in Figure 13, GPT-4 demonstrates greater effectiveness in pinpointing the central logical error in arithmetic reasoning. For instance, we can observe that  $S < \mathcal{C}$  for alex_total = alex_weight + weight-multiplier * grace_weight and  $S > \mathcal{C}$  for answer = grace_weight + alex_total, where the former leads to an incorrect final answer. Additionally, GPT-4 typically offers detailed explanations and alternative solutions. As seen in the step answer = grace_weight + alex_total, GPT-4 can correct minor errors even when it arrives at the correct final answer. However, GPT-4 may still encounter difficulties in detecting small errors within the text, which can have a significant impact on logical consistency. This challenge is illustrated by the substantial variance in  $S$  for the step alex_total = alex_weight + weight-multiplier * grace_weight.
The benefits of well-crafted explanations in GPT-4 become more significant when handling complex reasoning tasks, as demonstrated in Figure 14. For instance, in the  $R_{42}$  of  $Q_4$  shown in Figure 7b, Codex generally assigns high evaluation scores for all steps. Although this reasoning chain leads to the correct final answer, it makes some overly definitive assumptions without reasonable justification (e.g., "must have attributes that match both"). In such cases, GPT-4 can accurately identify these vague statements through meticulous analysis. Moreover, the comprehensive analysis helps address the growing uncertainty arising from the ambiguity in understanding commonsense questions. For example, in the final step, GPT-4 offers extensive explanations for assigning low  $S$  scores, considering the limited information available to comprehend the question.
In summary, GPT-4 demonstrates promising improvements in LLM evaluation by providing more rigorous and in-depth explanations. While there is significant variation in its generated explanations for some challenging cases, these explanations are typically beneficial as they elaborate on understanding and interpretation based on the given information. This suggests the potential for using LLM explanations to enhance self-evaluation in future research.
The results using GPT-4 were obtained on 22 April 2023.
Figure 12: Score from low to high values visualized using colormap from orange to green.
Figure 13: Comparison between Codex  $(\mathcal{C})$  and GPT-4  $(S)$  on evaluating the reasoning chains in arithmetic reasoning. The example  $(R_{12}$  for  $Q_{1})$  is the same one from Figure 7a. We underline the key statements in explanations that justify the judgment of GPT-4.
$[Q_1]$  Grace weighs 125 pounds. Alex weighs 2 pounds less than 4 times what Grace weighs. What are their combined weights in pounds?  
[Ground-Truth  $a_1^*$ ] 623.0
$\left\lbrack  {\text{Predicteda}}_{12}\right\rbrack  {627.0}$ $\left\lbrack  {R}_{12}\right\rbrack$  in Python
grace_weight = 125
 $\mathcal{E}$  C correct
 $\mathcal{S}$  S1 correct
 $\mathcal{S}_2$  The line of code correctly assigns the given weight of Grace (125 pounds) to the variable
grace_weight
 $\mathcal{S}_3$  The given line of code correctly assigns the value of 125 to the variable grace_weight, representing
Grace's weight in pounds.
alex_weight = 2
 $\mathcal{E}$  C alex_weight should be calculated based on grace_weight
 $\mathcal{S}$  S1 alex_weight should be calculated as 4 times grace_weight minus 2
 $\mathcal{S}_2$  The line of code defining alex_weight should be calculated as 4 times Grace's weight minus 2, which
is (4 * grace_weight) - 2.
 $\mathcal{S}_3$  The calculation for Alex's weight is not correct. It should be calculated as 4 * grace_weight - 2.
weight-multiplier = 4
 $\mathcal{E}$  C correct
 $\mathcal{S}$  S1 The line of code is correct as it defines the multiplier for Alex's weight based on Grace's weight.
 $\mathcal{S}_2$  Correct because it assigns the correct value (4) to the variable weight-multiplier, which represents
the factor by which Grace's weight is multiplied to calculate Alex's weight.
 $\mathcal{S}_3$  The code correctly assigns the value 4 to the variable weight-multiplier, which will be used in the
calculation of Alex's weight.
alex_total = alex_weight + weight-multiplier * grace_weight
 $\mathcal{E}$  C correct formula to calculate alex_total
 $\mathcal{S}$  S1 The formula used to calculate Alex's weight is incorrect.
 $\mathcal{S}_2$  The above line of code calculates Alex's weight correctly as 2 pounds less than 4 times Grace's weight
(4 * grace_weight - 2).
 $\mathcal{S}_3$  Alex weighs 2 pounds less than 4 times what Grace weighs, not 2 pounds more.
answer = grace_weight + alex_total
 $\mathcal{E}$  C should be the sum of grace_weight and alex_weight
 $\mathcal{S}$  S1 The code calculates Alex's weight correctly using the given relationship and adds it to Grace's weight to
calculate the combined weight. The only minor issue is the unnecessary use of the separate variable
alex_weight which could directly be replaced with the calculation (4 * grace_weight - 2).
However, this does not affect the correctness of the answer.
The result variable holds the correct combined weight of Grace and Alex, and this line of code correctly
calculates the value of answer.
 $\mathcal{S}$  S1 The code above correctly calculates the total weight of Grace and Alex based on the given problem
statement. The only minor issue is that the alex_weight variable is initialized as 2, which might be a
bit confusing. It would be better to directly calculate alex_total as (4 * grace_weight - 2), but the
current code does not produce an incorrect result.
$[Q_4]$  Is Freya a combination of Athena and Aphrodite?
[Ground-Truth  $a_4^*$ ] yes
[Explanation] Athena was the Greek goddess of war. Aphrodite was the Greek goddess of love. Freya was the Norse goddess of war, love, and fertility.
[Predicted  $a_{42}]$  yes
$[R_{42}]$  in Chain-of-Thought
Freya is the goddess of love, beauty and fertility.
E	C	correct
S	S1	Freya indeed is associated with love, beauty, and fertility in Norse mythology. However, it is important to note that while the comparison to Athena and Aphrodite is understandable, it is not a direct or complete equivalence.
S2	The statement correctly identifies Freya. However, the statement does not mention that Freya is a Norse goddess, which could slightly impact the understanding of the comparison to Greek goddesses Athena and Aphrodite.
S3	The statement is mostly correct as Freya is indeed a goddess associated with love, beauty, and fertility. However, it might be worth mentioning that she is a Norse goddess, while Athena and Aphrodite are Greek goddesses, to provide more context.
You can compare these to the Greek gods Athena and Aphrodite.
E	C	correct
S	S1	It is true that Freya can be compared to the Greek gods Athena and Aphrodite in terms of their respective domains. However, the comparison is not entirely accurate as Freya comes from Norse mythology, and the attributes of the gods may not be an exact match. The comparison is a simplification, which is why the probability is not higher.
S2	The statement establishes a comparison between Freya and the Greek goddesses Athena and Aphrodite. While this step of reasoning is generally correct, it could be improved by specifically mentioning that Freya is a Norse goddess and by providing more information about the roles and attributes of Athena and Aphrodite for a clearer comparison.
S3	While it is true that Freya shares some similarities with Athena and Aphrodite, the comparison is not entirely accurate. Freya is a Norse goddess, while Athena and Aphrodite are Greek goddesses. The comparison is correct in the sense that they share some domains, but it might be misleading to suggest that Freya is a direct "combination" of these two goddesses.
It is not clear how their powers exactly compare.
E	C	correct
S	S1	This step of reasoning acknowledges that it is not entirely clear how the powers of Freya, Athena, and Aphrodite compare. Although there are some similarities, the differences between Norse and Greek mythologies make it difficult to determine whether Freya is a direct combination of Athena and Aphrodite. The high probability reflects the acknowledgment of the uncertainty in comparing the powers of the gods across different mythologies.
S2	The statement acknowledges that it is not clear how the powers of Freya, Athena, and Aphrodite exactly compare. However, the question asks if Freya is a combination of Athena and Aphrodite, which implies a synthesis or blending of their roles or attributes. The reasoning could be improved by specifically mentioning the roles and attributes of Athena and Aphrodite, and whether or not they overlap with Freya's roles and attributes.
S3	The reasoning here is more accurate, acknowledging that while there are similarities between Freya and Athena and Aphrodite, it is not clear how their powers exactly compare. It is important to consider that Freya is a Norse goddess and Athena and Aphrodite are Greek goddesses, so their roles, domains, and characteristics may not be directly comparable.
However, it seems that Freya must have attributes that match both.
E	C	correct
S	S1	incorrect because it assumes that Freya must have attributes that match both Athena and Aphrodite. While there may be some similarities in their attributes, it is not appropriate to conclude that Freya is a combination of the two Greek goddesses. Norse and Greek mythologies are distinct, and the deities in these pantheons may have different origins and roles.
S2	The statement acknowledges that Freya has attributes that match both Athena and Aphrodite. However, it would be more accurate to discuss the specific attributes of Athena and Aphrodite to determine the extent of the similarities.
S3	It assumes that Freya must have attributes that match both Athena and Aphrodite, but this may not be the case. While there are similarities between these goddesses, they are from different mythologies (Freya – Norse, Athena & Aphrodite – Greek), and their roles, domains, and characteristics may not be directly comparable or combined. It is not accurate to assume that Freya is a combination of Athena and Aphrodite without a more in-depth analysis in their respective mythologies.
So the answer is yes.
E	C	correct
S	S1	This step of reasoning is incorrect because it is just based on some similarities in their attributes. While there may be some similarities, it is not accurate to say that Freya is a combination of the two Greek goddesses. Norse and Greek mythologies are distinct, and the deities in these pantheons have different origins and roles.
S2	Although Freya shares some attributes with both Athena and Aphrodite, it is an oversimplification to claim that she is a combination of the two. The question seems to imply that Freya might be a direct mix of the two, but this is not accurate.
S3	The conclusion that Freya is a combination of Athena and Aphrodite is not accurate based on the information provided. While there are similarities between these goddesses, they belong to different mythologies (Freya is Norse, Athena and Aphrodite are Greek), and their roles, domains, and characteristics may not be directly comparable or combined.
Figure 14: Comparison between Codex  $(\mathcal{C})$  and GPT-4  $(S)$  on evaluating the reasoning chains in commonsense reasoning. The example  $(R_{42}$  for  $Q_{4})$  is the same one from Figure 7b. We underline the key points in GPT-4 rationales that explain the detailed understanding and analysis on the steps.
Similar to beam search, we maintain  $k$  distinct candidates in the beam and sample  $n$  completions for each one. Thus, for each reasoning step  $s^t$ , the search space has a size of  $k \cdot n$ . After acquiring  $k \cdot n$  samples, we retain  $k$  candidates by sampling from  $P_{beam}$  as Eq. 4. We set  $k = 5, n = 16$  with Codex backbone to balance the quality and efficiency. The maximum number of steps to decode is capped at 16. To control the computational cost and time complexity, one can also reduce the number of rollouts per beam and the beam size to  $n = 2$  and  $k \in [3,4]$  respectively, as we illustrate with Llama-2 backbone.
We set generation temperatures differently for various tasks and baselines. Regarding the generation temperature  $\gamma$  on Codex, for arithmetic and symbolic reasoning with PAL using deterministic beam search  $(\tau = 0.0)$ , we find that  $\gamma \in [0.4, 0.8]$  generally works well. In contrast, for commonsense reasoning with CoT, a lower temperature  $(\gamma \in [0.1, 0.5])$  is more effective, likely due to the increased randomness from the free-text format. Differently, when using Llama-2 backbone, PAL generally works better with lower generation temperature  $\gamma \leq 0.5$ , while CoT can tolerate larger  $\gamma > 0.5$  with better or comparable performance. This difference between Codex and Llama-2 may come from the different training tasks and data adopted for the two models, where the PAL reasoning is especially enhanced in Codex.
In majority voting, higher  $\gamma$  is preferred to better explore the search space in reasoning, e.g.,  $\gamma \geq 0.5$  for arithmetic reasoning. To further introduce sampling randomness in stochastic beam search for majority voting on multiple reasoning chains, we use  $\alpha = 0.5$  for all datasets but different values of  $\tau$  for each task. Specifically, we choose  $\tau = 0.5$  for PAL and  $\tau = 0.2$  for CoT, as PAL typically decomposes the reasoning problem into more steps than CoT. Here we tune  $\tau$  instead of  $\alpha$  to be smaller in CoT as CoT naturally contains more randomness due to its free-text formulation as we observe in practice, where a smaller  $\tau$  is more efficient to balance this randomness.
In previous works, majority voting on multiple reasoning chains involves sampling  $N$  (usually  $\geq 20$ ) reasoning chains and conducting a vote to determine the final answer, which can be time-consuming. In our approach, we simply perform majority voting among the  $N$  candidates in the last step of beam search from only a few times ( $\leq 10$ ) of searching. As a result, our method does not introduce additional time complexity compared to the conventional majority voting method, although we sacrifice some diversity in the final outcomes due to the similarity of candidates within a beam.
**Prompts.** We show examples of both the generation and evaluation prompts we use on different tasks in the following tables, where we only show one instance for each case. Full prompts and detailed formulations can be found in our code.
Table 6: Examples of few-shot exemplars of generation and evaluation CoT prompts on GSM8K. We adopt 8-shot and 5-shot learning for generation and evaluation, respectively.
GENERATION
Q: There are 15 trees in the grove. Grove workers will plant trees in the grove today. After they are done, there will be 21 trees. How many trees did the grove workers plant today?
A:
There are 15 trees originally.
And there were 21 trees after some more were planted.
So 21 - 15 = 6 trees were planted.
So the answer is 6.
EVALUATION
Q: Allison brought some CDs online. Each CD cost $7. There was an additional charge of $4 per order for shipping costs. The total bill came to $60. How many CDs did Allison buy?
A:
Each CD cost 7 dollars.
# Is the above step of reasoning:
# (A) Correct
# (B) Incorrect
# The above step of reasoning is (A)
And there was an additional charge of 4 dollars.
# Is the above step of reasoning:
# (A) Correct
# (B) Incorrect
# The above step of reasoning is (A)
So the total cost of each CD is 7 + 4 = 11 dollars.
# Is the above step of reasoning:
# (A) Correct
# (B) Incorrect
# The above step of reasoning is (B), because each CD costs 7 dollars.
So 60 / 11 = 5.45.
# Is the above step of reasoning:
# (A) Correct
# (B) Incorrect
# The above step of reasoning is (B), because it cost 4 dollars for shipping costs. So the cost of CDs is 60 - 4 = 56 dollars. So Allison bought 56 / 7 = 8 CDs.
So the answer is 5.
# Is the above step of reasoning:
# (A) Correct
# (B) Incorrect
# The above step of reasoning is (A), but the value of the number of CDs is incorrect.
Q: Olivia has  $23. She bought five bagels for$ 3 each. How much money does she have left?
Table 7: Examples of few-shot exemplars of generation and evaluation PAL prompts on GSM8K. We adopt 9-shot and 5-shot learning for generation and evaluation, respectively.
def solution():
    money_initial = 23
    bagels = 5
    bagel_cost = 3
    money_spent = bagels * bagel_cost
    money_left = money_initial - money_spent
    result = money_left
    return result
EVALUATION
Q: A piece of square paper has a perimeter of 32 centimeters. Nicky's dog, Rocky, tore off 1/4 of the paper. What is the area of the remaining paper?
def solution():
    perimeter = 32
    # Is the above line of code:
        # (A) Correct
        # (B) Incorrect
        # The above line of code is: (A)
        fraction_torn = 1 / 4
        # Is the above line of code:
            # (A) Correct
            # (B) Incorrect
            # The above line of code is: (A)
        area_total = (perimeter / 4) ** 2
        # Is the above line of code:
            # (A) Correct
            # (B) Incorrect
            # The above line of code is: (A), because the total area of the square  $\rightarrow$  can be calculated by (perimeter / 4) ** 2
area_remaining = (1 - fraction_torn) * area_total
    # Is the above line of code:
        # (A) Correct
        # (B) Incorrect
        # The above line of code is: (A)
        result = area_total
    # Is the above line of code:
        # (A) Correct
        # (B) Incorrect
        # The above line of code is: (B), because the result should be  $\rightarrow$  area_remaining
return result
    # Is the above line of code:
        # (A) Correct
        # (B) Incorrect
        # The above line of code is: (A), but the value of result is incorrect
Question: In a flight of  $600\mathrm{km}$ , an aircraft was slowed down due to bad weather. Its average speed for the trip was reduced by  $200\mathrm{km/hr}$  and the time of flight increased by 30 minutes. The duration of the flight is:
Answer Choices: A)1 hour; B)2 hours; C)3 hours; D)4 hours; E)5 hours
Table 8: Examples of few-shot exemplars of generation and evaluation PAL prompts on AQuA. Here . . . represents the same evaluation script as those in the previous steps. We adopt 8-shot and 5-shot learning for generation and evaluation, respectively.
def solution():
    duration = Symbol('duration', positive=True)
    delay = 30 / 60
    total_disntace = 600
    original_speed = total_disntace / duration
    reduced_speed = total_disntace / (duration + delay)
    solution = solve_it(original_speed - reduced_speed - 200, duration)
    duration = solution[duration]
    result = duration
    return result
Question: Two trains of length  $150\mathrm{m}$  and  $200\mathrm{m}$  are  $100\mathrm{m}$  apart. They start moving towards each other on parallel tracks, at speeds  $54\mathrm{kmph}$  and  $72\mathrm{kmph}$ . In how much time will the trains cross each other?
Answer Choices: A)100/7 sec; B)80/7 sec; C)57/7 sec; D)110/7 sec; E)50/7 sec