SlowGuess/NeurIPS_2023_OCR huggingface dataset:
def solution():
    train_1_speed = 54 / 60
    # Is the above line of code:
        # (A) Correct
        # (B) Incorrect
        # The above line of code is: (A)
    train_2_speed = 72 / 60
    # Is the above line of code:
        # (A) Correct
        # (B) Incorrect
        # The above line of code is: (A)
    distance_between_trains = 100
    # Is the above line of code:
        # (A) Correct
        # (B) Incorrect
        # The above line of code is: (A)
    train_1_length = 150
    # Is the above line of code:
        # (A) Correct
        # (B) Incorrect
        # The above line of code is: (A)
    train_2_length = 200
    # ...
    # The above line of code is: (A)
    time_to-cross = distance_between_trains / (train_1_speed + train_2_speed)
    # ...
    # The above line of code is: (B), because to cross each other, the
    # total distance should also contain the train length
    result = time_to.Cross
    # ...
    # The above line of code is: (B), because the final result should be in
    # seconds, and the value of time_to-cross is incorrect
    return result
    # ...
    # The above line of code is: (A), but the value of result is incorrect
Passage: James bought 93 red and 10 blue stickers, he used 31 red sticker on his fridge and 7 blue stickers on his laptop.
Question: How many red stickers does James have?
Table 9: Examples of few-shot exemplars of generation and evaluation PAL prompts on SVAMP and ASDiv. Here we utilize the same prompts as they have the same task formulation. We adopt 7-shot and 5-shot learning for generation and evaluation, respectively.
def solution():
    original_red_stickers = 93
    used_red_stickers = 31
    red_stickers = original_red_stickers - used_red_stickers
    result = red_stickers
    return result
Passage: A piece of square paper has a perimeter of 32 centimeters. Nicky's dog, Rocky, tore off 1/4 of the paper.
Question: What is the area of the remaining paper?
def solution():
    perimeter = 32
    # Is the above line of code:
    # (A) Correct
    # (B) Incorrect
    # The above line of code is: (A)
    side_length = perimeter / 4
    # Is the above line of code:
    # (A) Correct
    # (B) Incorrect
    # The above line of code is: (A)
    area = side_length ** 2
    # Is the above line of code:
    # (A) Correct
    # (B) Incorrect
    # The above line of code is: (A)
    result = area
    # Is the above line of code:
    # (A) Correct
    # (B) Incorrect
    # The above line of code is: (B), because should calculate the  $\leftrightarrow$  remaining area after torn off as result
    return result
    # Is the above line of code:
    # (A) Correct
    # (B) Incorrect
    # The above line of code is: (A), but the value of result is incorrect
Table of "Coin collections":
Name | Number of coins
Braden | 76 \ Camilla | 94 \ Rick | 86
Mary | 84 \ Hector | 80 \ Devin | 83
Emily I 82 \ Avery I 87
Question: Some friends discussed the sizes of their coin collections. What is the mean of the numbers?
Table 10: Examples of few-shot exemplars of generation and evaluation PAL prompts on TabMWP. We adopt 4-shot and 5-shot learning for generation and evaluation, respectively.
def solution():
    number_of Coins_forDIFFERENT_person = [76, 94, 86, 84, 80, 83, 82, 87]
    mean_of_theNumbers = sum(number_of Coins_for_DIFFERENT_person) /
     $\leftrightarrow$  len(number_of Coins_for_DIFFERENT_person)
    result = mean_of_theNumbers
    return result
Table of "Roller coasters per amusement park":
Stem | Leaf
1|0,0,1,6,8,9\2|4,4,5,7,8,8
31,2,4,4,9,9\412,3,5,6,8,9,9
Question: Rodrigo found a list of the number of roller coasters at each amusement park in the state. How many amusement parks have fewer than 40 roller coasters?
def solution():
number_of Roller_coasters_per_amusement_park = [10, 14, 14, 15, 16, 18, 19, 20, 24, 25, 26, 28, 29, 29, 29, 30, 34, 35, 36, 39, 40, 40, 40, 41, 42, 43, 44, 45, 45, 46, 46, 47, 48, 48, 49, 49, 49, 50, 50, 51, 51, 52, 52, 53, 53, 54, 54, 55, 55, 56, 56, 57, 57, 58, 58, 59, 60, 60, 61, 62, 62, 63, 63, 64, 64, 65, 65, 66, 66, 67, 67, 68, 68, 69, 69, 70, 70, 71, 71, 72, 72, 73, 73, 74, 74, 75, 75, 76, 76, 77, 77, 78, 78, 79, 79, 80, 80, 81, 81, 82, 82, 83, 83, 84, 84, 85, 85, 86, 86, 87, 87, 88, 88, 89, 90,
\[
\begin{array}{r}
\text{number_of_roller_coasters_per_amusement_park} = [10, \text{14}, \text{14}, \text{15}, \text{16}, \text{18}, \text{20}, \text{24}, \text{25}, \text{26}, \text{28}, \text{29}, \text{29}, \text{29}, \text{29}, \text{29}, \text{29}, \text{29}, \text{29}, \text{29}, \text{29}, \text{29}, \text{29}, \text{29}, \text{29}, \text{29}, \text{29}, \text{29}, \text{\ldots} \\
\text{number_of_roller_coaster's per_amusement_park} = [10] = [10]
\end{array}
\]
Is the above line of code:
(A) Correct
(B) Incorrect
The above line of code is: (B), because values in the rows of Stem and Leaf represent the decimal and
$\hookrightarrow$  individual digits, respectively
number_of_amusement_parks_with_fewer Than_40 Roller_coasters = 0
The above line of code is: (A), because this is to initialize the
$\hookrightarrow$  number_of_amusement_parks_withFewer Than_40_roller_coasters
for number_of Roller_coasters in
$\hookrightarrow$  number_of_roller_coasters_per_amusement_park:
if number_of_roller_coasters < 40:  
    number_of_amusement_parks_with_fewer Than_40_roller_coasters += 1  
    # Is the above line of code:  
        # (A) Correct  
        # (B) Incorrect  
        # The above line of code is: (A), but the value of  
            # number_of_roller_coasters_per_amusement_park is incorrect
result = number_of_amusement_parks_with_fewer Than_40 Roller_coasters
The above line of code is: (A), but the value of
$\hookrightarrow$  number_of_amusement_parks_with_fewer Than_40 Roller_coasters is
$\hookrightarrow$  incorrect
The above line of code is: (A), but the value of result is incorrect.
Table 11: Examples of few-shot exemplars of generation and evaluation PAL prompts on Date Understanding from Big-Bench. We adopt 6-shot and 3-shot learning for generation and evaluation, respectively.
Q: 2015 is coming in 36 hours. What is the date one week from today in MM/DD/YYYY?
def solution():
    # If 2015 is coming in 36 hours, then today is 36 hours before. today = datetime(2015, 1, 1) - relativedelta(hours=36)
    # One week from today,
    one Week From today = today + relativedelta(weeks=1)
    # The answer formatted with %m/%d/%Y is
    result = one_Week_from今天的 strftime("%m/%d/%Y")
    return result
Q: Yesterday was April 30, 2021. What is the date yesterday in MM/DD/YYYY?
def solution(): # Yesterday was April 30, 2021. yesterday  $=$  datetime(2021,4,30) # Is the above line of code: # (A) Correct # (B) Incorrect # The above line of code is: (A) # Yesterday, yesterday  $=$  yesterday - relativedelta(days=1) # Is the above line of code: # (A) Correct # (B) Incorrect # The above line of code is: (B), because the date of yesterday was  $\rightarrow$  already got and this is step is redundant # The answer formatted with  $\% m / \% d / \% Y$  is result  $=$  yesterday.strftime('%m/%d/%Y') # Is the above line of code: # (A) Correct # (B) Incorrect # The above line of code is: (A), but the value of yesterday is  $\hookrightarrow$  incorrect return result # Is the above line of code: # (A) Correct # (B) Incorrect # The above line of code is: (A), but the value of result if incorrect
Q: I have a drum, a flute, a clarinet, a violin, four accordions, a piano, a trombone, and a trumpet. How many musical instruments do I have?
Table 12: Examples of few-shot exemplars of generation and evaluation PAL prompts on 0bject Counting from Big-Bench. We adopt 4-shot and 1-shot for generation and evaluation, respectively.
def solution():
    musical_instruments_to_count = {
        'drum': 1,
        'flute': 1,
        'clarinet': 1,
        'violin': 1,
        'accordion': 4,
        'piano': 1,
        'trombone': 1,
        'trumpet': 1
    }
    num_musical_instruments = sum(musical_instruments_to_count.values())
    result = num_instruments
    return result
Q: I have a peach, an onion, two garlics, a nectarine, a yam, a carrot, a potato, a head of broccoli, a cabbage, a banana, and a stalk of celery. How many vegetables do I have?
def solution():
    # note: I'm not counting the peach, nectarine, banana, or celery
     $\leftrightarrow$  vegetables
vegetables_to_count = {
    'onion': 1,
    # ...
    # The above line of code is: (A)
    'garlic': 2,
    # ...
    # The above line of code is: (A)
    'yam': 1,
    # ...
    # The above line of code is: (A)
    'carrot': 1,
    # ...
    # The above line of code is: (A)
    'potato': 1,
    # ...
    # The above line of code is: (A)
    'broccoli': 1,
    # ...
    # The above line of code is: (A)
    'cabbage': 1
    # ...
    # The above line of code is: (A)
}
# ...
# The above line of code is: (B), because celery is vegetable
num_vegetables = sum(vegetables_to_count.values())
# ...
# ...
# ...
# ...
return result
# ...
# ...
(A), but the value of num_vegetables is incorrect
Table 13: Examples of few-shot exemplars of generation and evaluation CoT prompts on CSQA. We adopt 7-shot and 3-shot learning for generation and evaluation, respectively.
GENERATION
Q: What do people use to absorb extra ink from a fountain pen?
Answer Choices: (a) shirt pocket; (b) calligrapher's hand; (c) inkwell; (d) desk drawer; (e) blotter
A:
The answer must be an item that can absorb ink.
Of the above choices, only blotters are used to absorb ink.
So the answer is (e).
EVALUATION
Q: John's took his appointment book with him to a checkup but couldn't find it anywhere.
He looked and he looked but he didn't know where it was. It wasn't until he bought a new one, went home, took a shower that he found it. Where was it?
Answer Choices: (a) office supply store; (b) wallet; (c) doctor's office; (d) handbag; (e) pocket
A:
The answer must be a place where John could have put his appointment book.
# Is the above step of reasoning:
# (A) Correct
# (B) Incorrect
# The above step of reasoning is: (B), because it fails to reason out that John will check the place when taking a shower (taking off clothes)
Of the above choices, only wallet makes sense.
# Is the above step of reasoning:
# (A) Correct
# (B) Incorrect
# The above step of reasoning is: (B), because wallet is too small to put his appointment book, and it cannot be the only one which makes sense (pocket is also plausible)
So the answer is (b).
# Is the above step of reasoning:
# (A) Correct
# (B) Incorrect
# The above step of reasoning is: (A), but the previous step is incorrect
Table 14: Examples of few-shot exemplars of generation and evaluation CoT prompts on StrategyQA. We adopt 6-shot and 4-shot learning for generation and evaluation, respectively.
GENERATION
Q: Do hamsters provide food for any animals?
A:
Hamsters are prey animals.
Prey animals are food for predators.
Thus, hamsters provide food for some animals.
So the answer is yes.
EVALUATION
Q: While viewing "Scary Movie" is the viewer likely to experience an increase in adrenaline?
A:
Adrenaline is a hormone that is released when the body is under stress.
# Is the above step of reasoning:
# (A) Correct
# (B) Incorrect
# The above step of reasoning is: (A)
Scary movies are designed to make the viewer feel stressed.
# Is the above step of reasoning:
# (A) Correct
# (B) Incorrect
# The above step of reasoning is: (B), because Scary Movie is a film that is a comedy take on horror, intended to make viewers laugh
Thus, the viewer is likely to experience an increase in adrenaline.
# Is the above step of reasoning:
# (A) Correct
# (B) Incorrect
# The above step of reasoning is: (A), but the previous step is incorrect
So the answer is yes.
# Is the above step of reasoning:
# (A) Correct
# (B) Incorrect
# The above step of reasoning is: (A), but the previous step is incorrect
Table 15: Examples of few-shot exemplars of generation and evaluation CoT prompts on Sports Understanding on Big-Bench. We adopt 8-shot and 2-shot learning for generation and evaluation, respectively.
GENERATION
Q: Is the following sentence plausible?
“KYLE PALMIERI WAS CALLED FOR SLASHING.”
A:
Kyle Palmieri is a hockey player.
Being called for slashing is part of hockey.
So the answer is yes.
EVALUATION
Q: Is the following sentence plausible?
“ROBIN LEHNER LAUNCHED THE DESPERATION HEAVE”
A:
Robin Lehner is a hockey player.
# Is the above step of reasoning:
# (A) Correct
# (B) Incorrect
# The above step of reasoning is: (B), because Robin Lehner is a goaltender which is different from a player
Launching the desperation heave is part of hockey.
# Is the above step of reasoning:
# (A) Correct
# (B) Incorrect
# The above step of reasoning is: (B), because launching the desperation heave is for player, not goaltender
So the answer is yes.
# Is the above step of reasoning:
# (A) Correct
# (B) Incorrect
# The above step of reasoning is: (A), but the previous step is incorrect