SlowGuess/NeurIPS_2023_OCR huggingface dataset:
For LSAT, we randomly sample 8 exemplars and write prompts for CoT and SATLM. We note that LSAT is a particularly challenging task: we tried 3 CoT prompts written by 3 different authors of our paper, which all led to around  $20\%$  accuracy. Similar results are reported in other work (Liang et al., 2022; Ribeiro et al., 2023). In addition, we only report CoT results, leaving out PROGLM. This decision is due to the fact that PROGLM uses Python as its program interpreter. While Python is a general-purpose programming language, it does not provide native support for formal logic reasoning, including essential components like logical inference rules and manipulation of logical formulas. Solving problems from LSAT requires strategies like proof by contradiction (see Appendix I for a detailed example), which we see no way to represent in the PROGLM framework and is not addressed in prior work.
BOARDGAMEQA contains problems requiring 1-3 steps of reasoning. We sample 5 exemplars from the training set of depth 1 and depth 2 to construct the prompts for evaluation on the test sets of depth 1 and depth 2, respectively. We used the 5 exemplars of depth 2 to construct the prompt for test set of depth 3, as using exemplars of depth 3 would lead to prompts that exceed the context window size of our LLMs. Similarly, we only report CoT results as the baselines, leaving out PROGLM for BOARDGAMEQA. We use the proofs provided by the authors to construct the CoT prompts and manually annotate the SAT specifications to construct the SATLM prompts.
For CLUTRR, we use the COt prompt and PROGLM prompt provided in FAITHFULCOT (Lyu et al., 2023). For PROOFWRITER, we use the COt prompt from SELECTION-INFERENCE (Creswell et al., 2023), and adapt it to form the PROGLM prompt. We use the COt prompt and PROGLM from PAL (Gao et al., 2023) for COLORED OBJECT.
The task of STRUCTUREDREGEX, a regex synthesis dataset, is to parse natural language descriptions to regexes. This is not a typical reasoning dataset, and there is no COt prompt for this dataset. We randomly sample 8 exemplars and annotate the prompt for PROGLM and SATLM. In this setting, PROGLM directly translates NL descriptions into regexes (which are essentially programs), whereas SATLM parses an NL description into a set of constraints over the surface form of the regex. Note that this dataset provides multimodal specifications of regexes, featuring both NL descriptions and examples. The I/O examples can be used to reject synthesized regexes if they do not accept or reject the correct examples. When we report results for self-consistency inference, we follow past work (Ye et al., 2021) and filter out incorrect outputs using the I/O examples provided in the dataset (Ye et al., 2020). This setting therefore checks consistency with something other than the model itself, but uses a similar computation budget as self-consistency, so we group it with those results.
C Performance of Original CoT and PROGLM Prompts on Arithmetic Reasoning Datasets
Table 9: Performance of different approaches using our adapted exemplar set and the original exemplar set used in COt and PAL.
ADAPTED (OURS)	ORIGINAL
GSM-SYS	GSM	GSM-SYS	GSM
COT	46.5	62.7	35.7	62.4
PROGLM	43.4	72.7	36.1	71.7
SATLM	69.4	71.8	66.7	70.9
Recall that we construct our arithmetic reasoning prompt used in Table 1 by replacing one random exemplar in the original prompt used in CoT and PROGLM with an random example from GSM-SYS. We show the performance of CoT, PROGLM, and our SATLM in Table 9 using our adapted exemplar set and original exemplar set in Table 9.
Our adaptation significantly improves the performance of COT and PROGLM on GSM-SYS, and slightly improves the performance on GSM. Furthermore, we still see that SATLM outperforms both CoT and PROGLM by a large margin on GSM, using either our adapted set or the original set.
D Extended Discussion on Concurrent Work
Table 10: Performance of different approaches on ALGEBRA.
ALGEBRA	GSM
COT	53.6	62.4
PROGLM	52.3	72.7
SATLM (Ours)	77.5	71.8
MATHSYM (He-Yueya et al., 2023)	76.3	69.4
Similar to our work, He-Yueya et al. (2023) proposes to solve arithmetic reasoning problems by parsing the problem into a set of variables and equations and using an external solver to derive the
final answer. While their formalization is restricted to arithmetic problems, we use SAT problems encoded with first-order logical formulas, which unify a wide range of reasoning tasks.
In addition, we also evaluate our approach on the ALGEBRA dataset in He-Yueya et al. (2023), which consists of 222 examples from Algebra textbooks. We note that the results between ours and MATHSYM are not directly comparable, as MATHSYM picks a different exemplar set. As shown in Table 10, ALGEBRA is more challenging than GSM, and SATLM outperforms PROGLM and CoT by more than  $20\%$ .
E Detailed Performance on the BOARDGAMEQA Dataset
Table 11: Detailed performance on the BOARDGAMEQA dataset.
DEPTH 1	DEPTH 2	DEPTH 3	AGGREGATED
code-davinci-002 (greedy decoding)
STANDARD	52.5	42.8	38.5	44.6
COT	64.7	60.8	56.5	60.1
SATLM	87.6	81.7	69.0	79.4
code-davinci-002 (self consistency decoding)
COT	65.9	63.4	59.0	62.8
SATLM	88.0	84.2	70.1	80.8
Table 11 shows the performance breakdown on depths 1-3 of the BOARDGAMEQA dataset. SATLM outperforms COT by a substantial margin across all depths. The performance of all approaches decreases as the depth increases.
F Details of the SAT Specification
To better utilize the parametric knowledge that LLMs have acquired from pretraining on vast amount of code data, our work uses a specification that largely follows and simplifies the syntax for specifying constraints used in z3py.
Example SAT Specification  
x = Variable() # declare a variable  
People = [Alice, Bob] # declare enum set  
Cities = [Austin, Boston] # declare enum set  
Food = [Apple, Banana] # declare enum set  
visit = Function(people, Cities) # declare function  
eats = Function(people, Food) # declare function  
visit(Alice) != visit(Bob) # logic  
forall(x: People, Implies.visit(x) == Austin, eats(x) == Banana)) # quantifier
Figure 5: Example of our SAT specification. The syntax is largely the same as that for specifying constraints in z3py.
We give an example specification in Figure 5 demonstrating the syntax for different types of statements. See Figure 1, Figure 2, and Appendix I for more examples. These formulas are close to the actual python code formulas used by z3py but are slightly modified to be more amenable to prompting. As a result, we use a postprocessing step to form the actual Z3 input. We implemented a simple parser that transforms these formulas into actual specifications used by z3py via string transformation (using regexes). For example, we transform [ForAll(x: People, Impplies.visit(x, Austin), eats(x, Banana)))] into [x = Variable(People) ForAll([x], Impplies.visit(x) == Austin, eats(x) ==
$^{5}$ https://z3prover.github.io/api/html/namespacez3py.html
[Banana])], which is the actual z3py code. The transformed code can be executed using z3py to obtain the final answer.
G Examples of Error Cases of SATLM
G.1 Examples of Unsatisfiable and Ambiguous Formulas
We show examples of formulas that are unsatisfiable or have ambiguous answers in Figure 6, Figure 7, and Figure 8.
UNSAT Formulas: GSM
Q: If a rectangle has a width of 42 inches and an area of 1638, how many rectangles of the same size would reach a length of 390 inches?
rectangle_width = 42  
rectangle_area = 1638  
rectangle_length = 390  
rectangle_length = rectangle_area / rectangle_width  
rectangles_needed = rectangle_length / rectangle_width  
result = rectangles_needed  
solve(result)
Q: Jason's dog has a tail that's half the length of its body, and a head that's 1/6 the length of its body. If the dog is 30 inches long overall, how long is its tail?
dog_length = 30  
tail_length = Variable()  
head_length = Variable()  
dog_length = tail_length + head_length  
tail_length = dog_length * 1 / 2  
head_length = dog_length * 1 / 6  
result = tail_length  
solver(result)
Figure 6: Examples of UNSAT formulas from GSM. Both examples are generated by code-davinci-002. UNSAT occurs when LLMs misunderstand the problem and output conflicting constraints (highlighted in red).
AMBIG Formulas: GSM
Q: The sum of the three numbers is 500. If the first number is 200, and the value of the second number is twice the value of the third number, find the value of the third number.
num1 = 200  
num2 = Variable()  
num3 = Variable()  
num2 = num3 * 2  
total = num1 + num2 + num3  
result = num3  
solve(result)
Q: The total for the sum and product of Elvis's age and Arielle's age are 131. If Elvis's age is 10, how old is Arielle?
elvie_age = 10  
arielle_age = Variable()  
sum_age = elvie_age + arielle_age  
product_age = elvie_age * arielle_age  
total_age = sum_age + product_age  
result = arielle_age  
solve(result)
Figure 7: Examples of AMBIG solutions from GSM. Both examples are generated by code-davinci-002. The ambiguous formulas are under-constrained due to failure in encoding certain constraints specified in the problem (highlighted in red), resulting in multiple possible answers.
UNSAT Formulas: CLUTRR
Arlene and her husband Jeff went on a cruise. They had a wonderful time. Stephanie's father Jason loves his little princess even though she gets into a lot of trouble at school. Gloria's mother Ruth and brother Jeff were working in the kitchen, preparing soup. Stephanie, Ruth's daughter, was working outside in the garden.
Q: How is [Jason] related to [Arlene]?
[Arlene] and her husband [Jeff] went on a cruise. They had a wonderful time.
relation(Arlene, Jeff) = (wife, husband)
[Stephanie]'s father [Jason] loves his little princess even though she gets into a lot of trouble at school. relation(Stephanie, Jason) = (daughter, father)
[Gloria]'s mother [Ruth] and brother [Jeff] were working in the kitchen, preparing soup. relation(Gloria, Ruth) = (daughter, mother)
relation(Gloria, Jeff) = (daughter, brother)
# [Stephanie], [Ruth]'s daughter, was working outside in the garden.
relation(Stephanie, Ruth) = (daughter, mother)
How is [Jason] related to [Arlene]?
solve(relation(Jason, Arlene))
AMBIG Formulas: CLUTRR
Kirk loves talking to his grandfather Stanley on the phone. Paul's brother, Kirk, recently made it into town. Paul and his son Michael went to look at cars. Paul ended up buying the Mustang.  
Q: How is [Stanley] related to [Michael]?
[Kirk] loves talking to his grandfather [Stanley] on the phone.
relation(Kirk, Stanley) = (grandson, grandfather)
[Paul]s brother, [Kirk], recently made it into town.
relation(Paul, Kirk) = (brother, brother)
[Paul] and his son [Michael] went to look at cars.
relation(Paul, Michael) = (father, son) note: Michael is Kirk's son, not Paul's son
How is [Stanley] related to [Michael]?
solve(relation(Stanley, Michael))
Figure 8: An example of UNSAT formulas (top) and an example of AMBIG formulas (bottom) from CLUTRR. Both happen when the LLM mispasses a constraint (highlighted in red). In particular, ambiguous formulas usually occur when the language description itself is highly ambiguous (see the note in the ambiguous formulas for an example).
G.2 Examples of Incorrect Answers
We show examples where the SAT solver successfully returns an answer, but the answer is incorrect in Figure 9.
Examples of Incorrect Answers: GSM
Q: Martin rings the small bell 4 times more than 1/3 as often as the big bell. If he rings both of them a combined total of 52 times, how many times does he ring the big bell?
small_bell_rings = Variable()
big_bell_rings = Variable()
total_bell_rings = 52
small_bell_rings = big_bell_rings * 1 / 3 * 4
total_bell_rings = small_bell_rings + big_bell_rings
result = big_bell_rings
solve(result)
Q: Jonathan was sad to learn he needed 2 more toys than he already had to have enough toys to make a sorted play set five times larger than James's play set, which had 80 toys. How many toys does Jonathan currently have?
james_TOYS = 80
jonathan_TOYS = Variable()
jonathan_TOYS_needed = james_TOYS * 5 + 2
result = jonathan_TOYS_needed - 2
solve(result)
Examples of Incorrect Answers: CLUTRR
Donald and Russell are brothers. Jason is father of their father.
Q: How is [Russell] related to [Jason]?
# [Donald] and [Russell] are brothers.
relation(Donald, Russell) = (brother, brother)
# [Jason] is father of their father
relation(Jason, Donald) = (father, son)
relation(Jason, Russell) = (father, son)
# How is [Russell] related to [Jason]?
solve(relation(Russell, Jason))
Figure 9: Two examples of incorrect answers from GSM (top) and an example of incorrect answer from CLUTRR (bottom). This occurs when LLMs output incorrect constraints (highlighted in red) but the incorrect formulas can still derive a non-ambiguous answer.
H Examples of Execution Errors and Planning Errors Made by COTSOLVER
We show examples of execution errors and planning errors produced by COTSOLVER in Figure 10, Figure 11, and Figure 12.
As shown in Figure 10, LLMs can make errors when executing arithmetic operators, resulting in execution errors. While introducing an interpreter (PROGLM) can help fix such execution errors, it is not effective for handling planning errors in Figure 11, which involves manipulating symbolic variables to determine a chain of arithmetic equations leading to the final answer.
We note that for CLUTRR, the atomic function is applying a set of transitivity rules that infer the relation between A and C based on the relation between A and B and the relation between B and C (e.g., A's son is B, B's daughter is C. So A's granddaughter is C). So an execution error in CLUTRR refers to incorrect outputs when applying transitivity rules (see the top section of Figure 12). The set of transitivity rules we use are provided in Lyu et al. (2023); Zhang et al. (2022a).
Execution Errors of COTSOLVER: GSM
Q: Carly is thinking about buying a wallet that costs 22 and a purse that costs 3 less than four times the cost of the wallet. What is the combined cost of both items?
wallet_cost = 22
purse_cost = Variable()
purse_cost = wallet_cost * 4 - 3
total_cost = wallet_cost + purse_cost
result = total_cost
# We know purse_cost = wallet_cost * 4 - 3
# So total_cost = wallet_cost + purse_cost = 22 + (22 * 4 - 3) = 22 + 87 = 109
# result = total_cost = 109
# The answer is 109
Q: A person borrowed an amount of money for a year at an interest rate of  $12\%$ . If the total interest is $1500, what was the original borrowed amount?
interest_rate = 12
interest_total = 1500
borrowed_amount = Variable()
interest_total = borrowed_amount * interest_rate / 100 result = borrowed_amount
# We know interest_total = borrowed_amount * interest_rate / 100
# So borrowed_amount = interest_total / interest_rate * 100 = 1500 / 12 * 100 = 12500 / 12 = 1041.67
# result = borrowed_amount = 1041.67
# The answer is 1041.67
Figure 10: Examples of execution errors made by COTSOLVER on arithmetic reasoning tasks. Both examples are generated by code-davinci-002, which makes mistakes when calculating the final value of equations.
Planning Errors of COTSOLVER: GSM
Q: Griffin had 24 french fries, but Kyle took 5 of them. Billy took twice as many as Kyle. Ginger gave Griffin a handful of her fries, and then Colby took from Griffin 3 less than the number of fries that Kyle had taken. If in the end Griffin had 27 fries, how many fries did Ginger give Griffin?
fries_initial = 24  
kyle_TOok = 5  
billy_TOok = kyle_TOok * 2  
ginger_gave = Variable()  
colby_TOok = kyle_TOok - 3  
fries_after = 27  
fries_after = fries_initial - kyle_TOok - billy_TOok + ginger_gave - colby_TOok  
result = ginger_gave  
# Solve the value of result  
# We know fries_after = fries_initial - kyle_TOok - billy_TOok + ginger_gave - colby_TOok  
# So ginger_gave = fries_after - fries初始 + kyle_TOok + billy_TOok - colby_TOok = 27 - 24 + 5 + 10 - 2 = 16  
# result = ginger_gave = 16  
# The answer is 16
Q: The combined age of Peter, Paul and Jean is 100 years old. Find the age of Peter knowing that Paul is 10 years older than John and that Peter's age is equal to the sum of Paul and John's age.
john_age = Variable()
total_age = 100
paul_age = john_age + 10
peter_age = paul_age + john_age
total_age = peter_age + paul_age + john_age
result = peter_age
# We know paul_age = john_age + 10, and peter_age = paul_age + john_age. So peter_age = john_age + 10 + john_age = 2 * john_age + 10
# We also know total_age = peter_age + paul_age + john_age = 2 * john_age + 10 + john_age + 10 = 3 * john_age + 20
# Therefore, john_age = (total_age - 20) / 3 = (100 - 20) / 3 = 80 / 3 = 26.67
# peter_age = 2 * john_age + 10 = 2 * 26.67 + 10 = 53.33 + 10 = 63.33
# result = peter_age = 63.33
# The answer is 63.33
Figure 11: Examples of planning errors made by COTSOLVER from GSM. Both examples are generated by code-davinci-002. In the first example, LLM fails to flip the sign of colby Took. In the second example, the model drops an occurrence of john_age when expanding out the formula. In both case, LLM fails to arrange the symbols to derive a chain of numeric equations that gives the correct answer.
Execution Errors of COTSOLVER: CLUTRR
# [James] was looking forward to dinner with his granddaughter, [Joanne].  
relation(James, Joanne) = (grandfather, granddaughter)  
# [Helen] went to her son [Leo]'s House  
relation(Helen, Leo) = (mother, son)  
# [Gene] bought gifts for his lovely sister, [Joanne]  
relation(Gene, Joanne) = (brother, sister)  
# [Kecia] wanted to visit an art museum, so she asked her father, [James] to take her.  
relation(Kecia, James) = (daughter, father)  
# [Sharon] and her sister, [Kecia], frequently bicker. However, they always make up quickly.  
relation(Sharon, Kecia) = (sister, sister)  
# [Leo] was so happy for his brother, [Gene], getting a promotion.  
relation(Leo, Gene) = (brother, brother)  
# How is [Sharon] related to [Helen]?  
answer = relation(Sharon, Helen)  
# Solve the problem  
# Helen's son is Leo, and Leo's brother is Gene, so Helen's son is Gene.  
# Helen's son is Gene, and Gene's sister is Joanne, so Helen's daughter is Joanne.  
# Helen's daughter is Joanne, and Joanne's father is James, so Helen's son is James.  
# Helen's son is James, and James's daughter is Kecia, so Helen's granddaughter is Kecia.  
# Helen's granddaughter is Kecia, and Kecia's sister is Sharon, so Helen's granddaughter is Sharon.  
# The answer is granddaughter.
Planning Errors of COTSOLVER: CLUTRR
# [Rosa] took her daughter [Melissa] to the park.  
relation(Rosa, Melissa) = (mother, daughter)  
# [Melissa] was sad her sister [Annie] could not join.  
relation(Melissa, Annie) = (sister, sister)  
# [Lisa] is having a great day at the playground with her mother [Rosa] and [Rosa]'s sister [Ruth].  
relation(Lisa, Rosa) = (daughter, mother)  
relation(Rosa, Ruth) = (sister, sister)  
# [Rosa] asked her daughter, [Lisa], if she would like to go to a movie with her on Saturday night.  
relation(Rosa, Lisa) = (mother, daughter)  
# How is [Ruth] related to [Annie]?  
answer = relation(Ruth, Annie)  
# Solve the problem  
# Rosa's daughter is Lisa, and Lisa's mother is Rosa, so Rosa's mother is Rosa.  
# Rosa's mother is Rosa, and Rosa's sister is Ruth, so Rosa's aunt is Ruth.  
# Rosa's aunt is Ruth, and Rosa's daughter is Melissa, so Rosa's niece is Melissa.  
# Rosa's niece is Melissa, and Melissa's sister is Annie, so Rosa's niece is Annie.  
# The answer is niece.
Figure 12: Examples of planning errors made by COTSOLVER on CLUTRR. We omit questions for brevity. Both examples are generated by code-davinci-002. In the first example, the model outputs an incorrect value when applying the transitivity rule marked in red (correct output should be husband). In the second example, the model comes up with an incorrect procedure.
I Prompt Examples
We show one or two exemplars in the prompt for each dataset. We list prompts for PROGLM for comparison.
Prompts for GSM and GSM-Sys
SATLM
Q: Jason had 20 lollipops. He gave Denny some lollipops. Now Jason has 12 lollipops. How many lollipops did Jason give to Denny?
jason_lollipops_initial = 20
lollipops GIVEN = Variable()
jason_lollipops_after = 12
jason_lollipops_after = jason_lollipops_initial - lollipops GIVEN
result = lollipops GIVEN
solve(result)
Q: Jeff bought 6 pairs of shoes and 4 jackets for $560. Jackets cost 1/4 price of one pair of shoes. Find the shoe's price total price.
shoes_num = 6  
jerseys_num = 4  
total_cost = 560  
shoes_cost_each = Variable()  
jerseys_cost_each = Variable()  
shoes_cost_each * shoes_num + jackets_cost_each * jackets_num = total_cost  
jerseys_costeach = shoes_cost(each * 1 / 4  
shoes_cost_total = shoes_cost_each * shoes_num  
result = shoes_cost_total  
solve(result)
PROGLM from Gao et al. (2023)
jason_lollipops_initial = 20
jason_lollipops_after = 12
denny_lollipops = jason_lollipops_initial - jason_lollipops_after
result = denny_lollipops
return result
shoes_num = 6  
jerseys_num = 4  
total_cost = 560  
jersey_shoes_cost_ratio = 1 / 4  
shoes_cost_each = total_cost / (shoes_num + jersey_num * jersey_shoes_cost_ratio)  
shoes_cost_total = shoes_cost_each * shoes_num  
result = shoes_cost_total  
return result
Figure 13: Prompt (excerpt) used for GSM and GSM-Sys.
Nine different treatments are available for a certain illness: three antibiotics—F, G, and H—three dietary regimens—M, N, and O—and three physical therapies—U, V, and W. For each case of the illness, a doctor will prescribe exactly five of the treatments, in accordance with the following conditions: If two of the antibiotics are prescribed, the remaining antibiotic cannot be prescribed. There must be exactly one dietary regimen prescribed. If O is not prescribed, F cannot be prescribed. If W is prescribed, F cannot be prescribed. G cannot be prescribed if both N and U are prescribed. V cannot be prescribed unless both H and M are prescribed.
Question: If O is prescribed for a given case, which one of the following is a pair of treatments both of which must also be prescribed for that case?
(A) F, M (B) G, V (C) N, U (D) U, V (E) U, W
treatments  $=$  [F,G,H,M,N,O,U,V,W] antibiotics  $=$  [F,G,H] dietary_regimens  $=$  [M,N,O] physical_therapies  $=$  [U,V,W] prescribed  $=$  Function(treatments, bool) Count([t:treatments],prescribed(t))  $= = 5$  Count([a:antibiotics],prescribed(a))  $<   = 2$  Count([d:dietary_regimens],prescribed(d))  $= = 1$  Implies(Not(prescribed(O)),Not(prescribed(F))) Implies(prescribed(W),Not(prescribed(F))) Implies(And(prescribed(N),prescribed(U)),Not(prescribed(G))) Implies(prescribed(V),And(prescribed(H),prescribed(M)))
solve(Implies(prescribed(O), And(prescribed(U), prescribed(V)))) # (A)  
solve(Implies(prescribed(O), And(prescribed(G), prescribed(V)))) # (B)  
solve(Implies(prescribed(O), And(prescribed(N), prescribed(U)))) # (C)  
solve(Implies(prescribed(O), And(prescribed(U), prescribed(V)))) # (D)  
solve(Implies(prescribed(O), And(prescribed(U), prescribed(W)))) # (E)
C0T (annotated by our authors)
Nine different treatments are available for a certain illness: three antibiotics—F, G, and H—three dietary regimens—M, N, and O—and three physical therapies—U, V, and W ....... (omitted, see above)
Let's first analyze the conditions. We know there must be exactly one dietary regimen prescribed. If O is a prescribed dietary regimen, then M and N must not be prescribed. We know V cannot be prescribed unless both H and M are prescribed, so V cannot be prescribed.
Let's consider option (A). M must not be prescribed. This option is False.
Let's consider option (B). V cannot be prescribed. This option is False.
Let's consider option (C). N must not be prescribed. This option is False.
Let's consider option (D). V cannot be prescribed. This option is False.
Let's consider option (E). Let's assume U is not prescribed. Because exactly 5 of the treatments will be prescribed, they must be F, G, H, O, and W. In this way both F and W are prescribed, which contradicts the condition that if W is prescribed, F cannot be prescribed. So U must be prescribed. We can prescribe G, H, O, U, W. This option is True.
So the answer is (E).
Figure 14: Prompt (excerpt) used for LSAT. Another example can be found in Figure 1. Several of our authors authored different CoT prompts, leading to similar performance (ranging from  $20\%$  to  $22\%$ ).
Prompts for LSAT