SlowGuess/NeurIPS_2023_OCR huggingface dataset:
A few players are playing a boardgame. The current state of the game is as follows. The gecko has 13 friends, and hates Chris Ronaldo. And the rules of the game are as follows. Rule1: If the gecko has more than 8 friends, then the gecko does not proceed to the spot that is right after the spot of the bat. Rule2: Regarding the gecko, if it is a fan of Chris Ronaldo, then we can conclude that it does not proceed to the spot that is right after the spot of the bat. Rule3: If something does not proceed to the spot right after the bat, then it does not give a magnifier to the swordfish.
Q: Based on the game state and the rules and preferences, does the gecko give a magnifier to the swordfish?
If the gecko has more than 8 friends, then the gecko does not proceed to the spot that is right after the spot of the bat.
Implies (has_more Than_8 Friends(gecko), Not(proceed_to-spot_right_after(gecko, bat))) # Rule2: Regarding the gecko, if it is a fan of Chris Ronaldo, then we can conclude that it does not proceed to the spot that is right after the spot of the bat.
Implies(is_fan_of_chris_ronaldo(gecko), Not(proceed_to-spot_right_after(gecko, bat))) # Rule3: If something does not proceed to the spot right after the bat, then it does not give a magnifier to the swordfish.
ForAll([x],plies(Not(proceed_to-spot_right_after(x,bat)),Not(give_magnifier(x,swordfish))))
The current state of the game is as follows. The gecko has 13 friends, and hates Chris Ronaldo.
The gecko has 13 friends.
has_more Than_8_friends(gecko) == True
The gecko hates Chris Ronaldo.
is_fan_of_chris_ronaldo(gecko) == False
question: does the gecko give a magnifier to the swordfish?
solve(give_magnifier(gecko, swordfish))
CoT from Kazemi et al. (2023)
A: We know the gecko has 13 friends, 13 is more than 8, and according to Rule1 "if the gecko has more than 8 friends, then the gecko does not proceed to the spot right after the bat", so we can conclude "the gecko does not proceed to the spot right after the bat". We know the gecko does not proceed to the spot right after the bat, and according to Rule3 "if something does not proceed to the spot right after the bat, then it doesn't give a magnifier to the swordfish", so we can conclude "the gecko does not give a magnifier to the swordfish". So the statement "the gecko gives a magnifier to the swordfish" is disproved. The answer is no.
Figure 15: Prompt (excerpt) used for BOARDGAMEQA.
Prompts for BOARDGAMEQA
Prompts for CLUTRR
Dorothy took her daughter Michelle and her mother Gabrielle car shopping.
Q: How is [Michelle] related to [Gabrielle]?
# [Dorothy] took her daughter [Michelle] and her mother [Gabrielle] car shopping.
relation(Dorothy, Michelle) = (mother, daughter)
relation(Dorothy, Gabrielle) = (daughter, mother)
How is [Michelle] related to [Gabrielle]?
solve(relation(Michelle, Gabrielle))
Teresa and her brother Ellis were having a wonderful time at Disneyland. Ellis asked his grandmother, Molly, to read him a bedtime story. Molly read him Hansel & Gretel, which the boy always loved. Sandra is married to Thomas, the couple welcomed Teresa into the world.
Q: How is [Molly] related to [Sandra]?
# [Teresa] and her brother [Ellis] were having a wonderful time at Disneyland.
relation(Teresa, Ellis) = (sister, brother)
[Ellis] asked his grandmother, [Molly], to read him a bedtime story.
relation(Ellis, Molly) = (grandson, grandmother)
[ Sandra] is married to Thomas, the couple welcomed [Teresa] into the world.
relation(Sandra, Teresa) = (mother, daughter)
How is [Molly] related to [Sandra]?
solve (relation(Molly, Sandra))
PROGLM from Lyu et al. (2023)
To answer this question, we write a program to answer the following subquestions:
1. How is [Michelle] related to [Dorothy]? (independent, support: "[Dorothy] took
her daughter [Michelle] and her mother [Gabrielle] car shopping."
relation(Michelle, Dorothy) = daughter
2. How is [Dorothy] related to [Gabrielle]? (independent, support: "[Dorothy] took her daughter [Michelle] and her mother [Gabrielle] car shopping.")
relation(Dorothy, Gabrielle) = daughter
# 3. Final answer: How is [Michelle] related to [Gabrielle]? (depends on 1, 2)
relation(Michelle, Gabrielle) = relation(Michelle, Dorothy) @ relation(Dorothy, Gabrielle)
Teresa and her brother Ellis were having a wonderful time at Disneyland.... (omitted, see above)
1. How is [Molly] related to [Ellis]? (independent, support: "[Ellis] asked his grandmother, [Molly], to read him a bedtime story.")
relation(Molly, Ellis) = grandmother
2. How is [Ellis] related to [Teresa]? (independent, support: "[Teresa] and her brother [Ellis] were having a wonderful time at Disneyland.")
relation(Ellis, Teresa) = brother
3. How is [Teresa] related to [Sandra]? (independent, support: "[Sandra] is married to Thomas, the couple welcomed [Teresa] into the world.")
relation(Teresa, Sandra) = daughter
4. Final answer: How is [Molly] related to [Sandra]? (depends on 1, 2, 3) relation(Molly, Sandra) = relation(Molly, Ellis) @ relation(Ellis, Teresa) @ relation(Teresa, Sandra)
Figure 16: Prompt (excerpt) used for CLUTRR.
Here are some facts and rules:
If someone visits the squirrel and the squirrel visits the rabbit then they are round. All round people are not kind. If someone is round then they chase the rabbit. If someone is red and they chase the rabbit then they visit the dog. If someone is red then they visit the squirrel. If someone visits the squirrel then the squirrel visits the rabbit. the rabbit visits the dog.
the squirrel chases the bald eagle. the squirrel chases the rabbit. the dog sees the bald eagle. the bald eagle does not chase the dog. the bald eagle is red. the squirrel is round. the rabbit does not see the dog. the rabbit sees the bald eagle. the rabbit sees the squirrel. the dog does not see the rabbit. the rabbit does not visit the bald eagle. the dog does not chase the bald eagle.
Q: The statement "The bald eagle visits the dog" is True or False?
ForAll([x], Implies(And.visit(x, squirrel), visit(squirrel, rabbit)), round(x)))  
ForAll([x], Implies(round(x), Not(kind(x))))  
ForAll([x], Implies(round(x), chase(x, rabbit)))  
ForAll([x], Implies(And(red(x), chase(x, rabbit)), visit(x, dog)))  
ForAll([x], Implies(red(x), visit(x, squirrel)))  
ForAll([x], Implies.visit(x, squirrel), visit(squirrel, rabbit)))  
chase(squirrel, rabbit)  
see(dog, bald_eagle)  
Not(chase(bald_eagle, dog))  
red(bald_eagle)  
round(squirrel)  
Not(see(rabbit, dog))  
see(rabbit, bald_eagle)  
see(rabbit, squirrel)  
Not(see(dog, rabbit))  
Not.visit(rabbit, bald_eagle))  
Not(chase(dog, bald_eagle))  
solve.visit(bald_eagle, dog))
PROGLM adapted from Creswell et al. (2023)
Here are some facts and rules:   
If someone visits the squirrel and the squirrel visits the rabbit then they are round..... (omitted, see above) Q: The statement "The bald eagle visits the dog" is True or False?   
# the bald eagle is red.   
bald_eagle_is_red = True   
# If someone is red then they visit the squirrel.   
bald_eagle.visit_squirrel  $=$  bald_eagle_is_red   
# If someone visits the squirrel then the squirrel visits the rabbit. squirrel.visit_rabbit  $=$  bald_eagle.visit_squirrel   
# If someone visits the squirrel and the squirrel visits the rabbit then they are round. bald_eagle_is_round  $=$  bald_eagle.visit_squirrel and squirrel.visit_rabbit   
# If someone is round then they chase the rabbit.   
bald_eagle_chases_rabbit  $=$  bald_eagle_is_round   
# If someone is red and they chase the rabbit then they visit the dog.   
bald_eagle.visit_dog  $=$  bald_eagle_is_red and bald_eagle_chases_rabbit   
# Question: The statement "The bald eagle visits the dog" is True or False?   
return bald_eagle.visit_dog
Figure 17: Prompt (excerpt) used for PROOFWRITER.
Prompts for PROOFWRITER
Q: On the table, you see a bunch of objects arranged in a row: a purple paperclip, a pink stress ball, a brown keychain, a green scrunchiephone charger, a mauve fidget spinner, and a burgundy pen. What is the color of the object directly to the right of the stress ball?
What is the color of the object directly to the right of the stress ball?  
stressBall = next(x:objects, name(x) == 'stress ball')  
direct_right = next(x:objects, index(x) - index(stress BALL) == 1)  
solve(color(right))
Find the index of the stress ball  
stress_bankidx = None  
for i, object in enumerate(objects):  
    if object[0] == 'stress ball':  
        stress_bankidx = i  
    break
Find the directly right object direct_right = objects[i+1]
Check the directly right object's color  
direct_right_color = direct_right[1]  
answer = direct_right_color  
return answer
Figure 18: Prompt (excerpt) used for COLORED OBJECT.
Prompts for COLOREDOBJECT
Find the regex for the described patterns. Each regex r can be composed using sub-regexes r1, r2, r3, ...
Pattern:
Three strings separated by semicolons. The first string can either be 579 or 719, the second and third are composed by three digits or three lower case letters that can be followed by a lower case letter, a digit or a capital letter.  
r = concat(r1, concat(<;>, concat(r2, concat(<;>, r2))))  
r1 = or(<579>, <719>)  
r2 = concat(or(r3, r4), optional(r5))  
r3 = repeat(<num>, 3)  
r4 = repeat(<low>, 3)  
r5 = or(<low>, or(<num>, <cap>)
Find the regex for the described patterns.
Three strings separated by semicolons. The first string can either be 579 or 719, the second and third are composed by three digits or three lower case letters that can be followed by a lower case letter, a digit or a capital letter.
Regex:
concat(or(<579>,<719>), concat(<;>,concat(concat(or(repeat(<num>,3),repeat(<low>,3)), optional(or(<low>,or(<num>,<cap>))),concat(<;>,concat(or(repeat(<num>,3),repeat(<low>,3)), optional(or(<low>,or(<num>,<cap>)))))))
Figure 19: Prompt (excerpt) used for STRUCTUREDREGEX.
Prompts for STRUCTUREDREGEX
Prompts for  $\mathbf{SAT}_{\mathrm{COTSOLVER}}$
SATCOTSOLVER for GSM
Q: Jason had 20 lollipops. He gave Denny some lollipops. Now Jason has 12 lollipops. How many lollipops did Jason give to Denny?
jason_lollipops_initial = 20lollipops GIVEN = Variable()jason_lollipops_after = 12jason_lollipops_after = jason_lollipops_initial - lollipops GIVENresult = lollipops GIVENsolve(result)# Solve the value of result# We know jason_lollipops_after = jason_lollipops_initial - lollipops GIVEN# So lollipops GIVEN = jason_lollipops_initial - jason_lollipops_after = 20 - 12 = 8# result = lollipops GIVEN = 8# The answer is 8
SATCOTSOLVER for CLUTRR
Dorothy took her daughter Michelle and her mother Gabrielle car shopping.Q: How is [Michelle] related to [Gabrielle]?# [Dorothy] took her daughter [Michelle] and her mother [Gabrielle] car shopping.relation(Dorothy, Michelle) = (mother, daughter)relation(Dorothy, Gabrielle) = (daughter, mother)# How is [Michelle] related to [Gabrielle]?solve(relation(Michelle, Gabrielle))# Solve the problem# Gabrielle's daughter is Dorothy, and Dorothy's daughter is Michelle, so Gabrielle's granddaughter is Michelle.# The answer is granddaughter.
Figure 20: Prompt (excerpt) used for  $\mathrm{SAT}_{\mathrm{COTSOLVER}}$