SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Baselines We compare SATLM against 3 baselines, including standard prompting (directly giving the answer), chain-of-thought prompting (CoT), and executor-augmented LLMs (PROGLM). We do not compare to zero-shot baselines such as zero-shot CoT, which generally underperform few-shot CoT by a large margin on the tasks we investigate (Kojima et al., 2022).
For COt and PROGLM, we leverage prompts of existing work (Gao et al., 2023; Lyu et al., 2023; Creswell et al., 2023) whenever possible. For SATLM, we manually write prompts for the same exemplar sets used in COt and PROGLM to ensure a fair comparison. We note that some settings, such as PROGLM for LSAT, are not applicable. Please refer to Appendix B for more discussion of the setup, including details on the prompts we use. We also include example prompts for all the datasets in Appendix I.
Language Models & Decoding We conduct our main experiments and analysis on code-davinci-002 (Chen et al., 2021), a state-of-art LLM for code and code-adjacent tasks. We evaluate the performance with both greedy decoding and self-consistency decoding (Wang et al., 2022b). Following past work (Gao et al., 2023), we use 40 samples on all datasets except for LSAT, BOARDGAMEQA, and PROOFWRITER; we use 5 samples on these datasets involving long prompts and high computation cost. For COT and PROGLM, we use a temperature of 0.7; for SATLM, we use a higher temperature of 0.9, which we find to work better.
Table 1 shows the performance of our approach compared to the baselines. In general, our SAT-aided approach outperforms both COT and PROGLM by a substantial margin except on GSM with greedy decoding. We perform significance tests via bootstrap resampling, and all improvements of SATLM over PROGLM are statistically significant  $(p < 0.05)$ .
Figure 3: A variant of our approach which replaces the SAT solver with a "CoT solver" that takes the SAT problem as input and solves it in natural language.
Table 2: The performance of variants of our approach that use CoT Solver or No Solver. Using declarative prompting with CoT solver is more effective than imperative CoT prompting.
GSM-SYS	GSM	CLUTRR
STANDARD	21.0	22.2	41.2
COT	46.5	62.7	40.8
PAL	43.4	72.8	58.9
SATSYMSOLVER	69.4	71.7	68.3
SATCOTSOLVER	54.5	63.2	48.9
SATNOSOLVER	26.6	23.7	40.7
The first two columns show the performance on the GSM dataset. COt and PROGLM achieve much worse performance on GSM-SYS than on GSM, indicating that GSM-SYS is a challenging subset. On this subset, SATLM achieves  $69.4\%$  and  $80.9\%$  with greedy decoding and self-consistency decoding, surpassing both PROGLM and COt more than by  $20\%$ . On the original GSM dataset, the SATLM model has a slightly lower accuracy than PROGLM with greedy decoding, but outperforms it with self-consistency decoding by  $2.4\%$ ; we provide detailed analysis accounting for the differences later in this section. This self-consistency accuracy of  $84.8\%$  even exceeds recent work that uses the full training set with code-davinci-002 (82.3% in DIVERSE (Li et al., 2022b); 84.5% in LEVER (Ni et al., 2023)). On ALGEBRA, a challenging dataset of math problems extracted from algebra textbooks, SATLM also outperforms COt and PROGLM by more than  $20\%$ .
On LSAT, CLUTRR, PROOFWRITER, and COLOR, SATLM consistently achieves the best performance with either greedy decoding or self-consistency decoding. SATLM also sets the new SoTA on both LSAT and BOARDGAMEQA, surpassing previous models that are trained on the full training set. Specifically, SATLM elevates the SoTA from  $30.9\%$  (Zhong et al., 2022) to  $37.4\%$  on LSAT and from  $73.9\%$  (Kazemi et al., 2023)) to  $80.7\%$  on BOARDGAMEQA. See Appendix E for detailed performance breakdown on depth 1-3.
In the regex synthesis domain, with greedy decoding, directly translating natural language descriptions to Regexes (PROGLM) achieves  $37.1\%$ , whereas using declarative prompting achieves  $44.0\%$ . With self-consistency, we surpass the previous SoTA performance of  $55.6\%$  (Ye et al., 2021).
4.3 Impact of SAT Solver & Declarative Prompting
We conduct analysis to isolate the effectiveness of the two key components, the SAT solver and declarative prompting. Specifically, we test a variant of our approach that still uses declarative prompting but then solves the equations in natural language with CoT rather than using the symbolic solver (see Figure 3). Essentially, the LLM itself carries out planning and execution. This experiment helps isolate the benefits of the solver, which will compute an answer without making any mistakes, from the benefits of the declarative formulation. We also compare to prompting LLMs to directly give the answer (NOSOLVER).
Impact of Symbolic Solver As shown in Table 2, completely abating the solver and directly predicting the answer  $(\mathrm{SAT}_{\mathrm{NOSOLVER}})$  only yields performance that is on par with STANDARD. Interestingly,  $\mathrm{SAT}_{\mathrm{COTSOLVER}}$  can solve more SAT problems than NOSOLVER. This partially reflects the effectiveness of CoT and partially reflects the fact that many dataset instances require relatively simple planning and execution, allowing pure forward reasoning to solve them. However, using a symbolic solver
Table 3: Fraction of planning errors (incorrect reasoning chains) and execution errors (numeric errors) made by COTSOLVER.
GSM-SYS	GSM	CLUTRR
PLAN ERR	72.5	42.5	47.5
EXEC ERR	27.5	57.5	52.5
$(\mathrm{SAT}_{\mathrm{SYM SOLVER}})$ , which guarantees correct planning and execution, leads to further improvements.
Table 5: Analysis of accuracy and execution status of SATLM and PROGLM. We present the fraction of tasks solved correctly or incorrectly in GSM-SYS, GSM, and CLUTRR, along with the breakdown of feedback from the solver. SATLM generally makes fewer predictions than PROGLM (ANSWERED), but more frequently makes correct predictions when it returns an answer (SELECTIVE ACC) and gives a higher absolute number of correct predictions on GSM-SYS and CLUTRR.
GSM-SYS	GSM	CLUTRR
PROGLM	SATLM	PROGLM	SATLM	PROGLM	SATLM
CORRECT	43.3	69.4	72.7	71.8	58.9	68.3
INCORRECT	52.5	20.6	25.7	21.2	21.0	7.7
ERROR	4.2	2.6	1.6	2.1	20.1	3.5
UNSAT	-	2.4	-	1.5	-	15.5
AMBIG	-	5.0	-	3.4	-	5.0
ANSWERED	95.8	90.0	98.4	93.0	79.9	76.0
SELECTIVE ACC	45.2	77.1	73.8	77.2	73.7	89.9
We manually analyzed 40 cases where the symbolic solver yields the correct answer but  $\mathrm{SAT}_{\mathrm{COTSOLVER}}$  fails to solve them. We categorized the errors as planning errors, where the reasoning chains are incorrect, and execution errors, where the reasoning chains are correct but computations are incorrect (see Appendix H for examples). Table 3 shows that most errors by  $\mathrm{SAT}_{\mathrm{COTSOLVER}}$  are planning errors, especially on GSM-SYS which requires solving complex system of equations.
Impact of Declarative Prompting Table 2 also shows that decoupling parsing and planning/solving is still useful, even when not using a symbolic solver:  $\mathrm{SAT}_{\mathrm{COTSOLVER}}$  outperforms COT by  $7.9\%$ , and  $8.1\%$  on GSM-SYS and CLUTRR, respectively. We note that  $\mathrm{SAT}_{\mathrm{COTSOLVER}}$  can be viewed as a two-stage CoT prompting strategy, with a prompt showing that the first step is to formulate declaratively, then the next step is to solve.
We hypothesize that parsing a question into declarative formulas is more straightforward than parsing it into an imperative solving procedure. To evaluate this hypothesis, we use log likelihood of the generated tokens to assess how straightforward the translation is, as higher
Table 4: Log likelihood (unnormized / normalized) of the generated sequences (with greedy decoding) of PROGLM and SATLM on three datasets. Better log likelihood indicates higher LLM confidence in the parsing stage.
GSM-SYS	GSM	CLUTRR
PAL	-9.5/-6.910-2	-9.2/-6.010-2	-3.1/-8.510-3
SAT	-8.5/-5.910-2	-9.7/-6.210-2	-2.0/-7.910-3
log-likelihood typically indicates the outputs are more fluent to LLMs, a connection demonstrated in recent literature (Gonen et al., 2022; Ye and Durrett, 2023). We show both unnormalized (total) and normalized log likelihood in Table 4. On GSM-SYS and CLUTRR where SATLM outperforms PROGLM, its generated outputs are also associated with higher likelihood.
4.4 Advantages of SAT in Selective Prediction
A SAT solver may not always return an answer, particularly if there are parsing errors from the question. We show that this is an advantage of SATLM: these errors allow us to abstain from making likely incorrect predictions. Example outputs leading to different errors can be found in Appendix G.
Table 5 shows the fraction of correct predictions and incorrect predictions when the program or SAT solver successfully returns an answer as well as the fraction of different types of feedback signals. We report the fraction of questions answered as well as selective accuracy, defined by the fraction of overall accuracy (\% of correct answers) normalized by coverage (\% of answered problems). SATLM makes fewer predictions on all three datasets compared to PROGLM, as it can trigger both UNSAT and AMBIG errors. However, SATLM's selective accuracy is consistently better than PROGLM's, especially on GSM-SYS (77% vs 45%). As a result, SATLM's overall performance is significantly better than PROGLM on GSM-SYS and CLUTRR, even when making fewer predictions.
We note that on GSM, SATLM has slightly lower coverage but higher selective accuracy compared to PROGLM. This explains why SATLM lags behind PROGLM with greedy decoding but outperforms PROGLM with self-consistency decoding (Table 1). By drawing multiple samples, SATLM can increase its coverage and achieve higher accuracy than PROGLM since its predictions are more accurate.
Q: Farmer Brown has 60 animals on his farm, all either chickens or cows. He has twice as many chickens as cows. How many legs do the animals have, all together?
The llama is named Peddi. The pelikan has a card that is red in color, and is named Beauty. Rule2: If the pelikan has a name whose first letter is the same as the first letter of the llama's name, then the pelikan creates a castle for the gadwall. Rule3: The pelikan will create a castle for the gadwall if it (the pelikan) has a card with a primary color.
SAT Solution
animals_total = 60  
animals_chickens = Variable()  
animals_cows = Variable()  
animals_chickens = animals_cows * 2  
animals_total = animals_chickens + animals_cows  
legs_chickens = animals_chickens * 2  
legs_cows = animals_cows * 4  
legs_total = legs_chickens + legs_cows
Implies(has_same_letter_name(pelikan, llama), createcastle(pelikan, gadwall)) # Rule2  
Implies(has_card_with_primary_color(pelikan), create_card(pelikan, gadwall)) # Rule3  
# The first letter of Peddi is P. The first letter of Beauty is B. So the pelikan does not have the same first letter name as the llama.  
has SAME_first_letter_name(pelikan, llama) == False  
# The pelikan has a card that is red in color. red is a primary color.  
has_card_with_primary_color(pelikan) == True
Figure 4: Examples outputs from GSM (left) and BOARDGAMEQA (right) show that LLMs can perform commonsense reasoning while parsing.
LLMs Can Perform Commonsense Reasoning While Parsing There are many problems that do not state premises or constraints in a completely explicit way. Figure 4) shows two examples where commonsense inferences are required during parsing. For example, on the left, the model must recognize that animals refers to the chickens and cows collectively. Similarly, knowing that red is a primary color is needed to successfully apply rules on BOARDGAMEQA (right). We observe from the outputs in both cases that LLMs are capable of implicitly performing commonsense reasoning and produce correct logical formulas in the parsing step. As shown in Table 1, SATLM exhibits strong performance on BOARDGAMEQA, a dataset which requires this implicit background knowledge.
Results Across Different Language Models
In addition to the main LLM used in our work, code-davinci-002, we further test whether SATLM can generalize to other LLMs. We choose gpt-3.5-turbo (0613 version), text-davinci-003, and code-davinci-001. gpt-3.5-turbo is optimized for chat. text-davinci-003 is an LLM pretrained on NL, and tuned to align with human feedback (Ouyang et al., 2022). code-davinci-001 is also an LLM pretrained on code, but less capable compared to 002. As shown in Table 6, SATLM is better than PROGLM on the arithmetic reasoning and logical reasoning datasets except for GSM across these three LLMs. The trend is congruent with the results on code-davinci-002 (Table 1), which suggests the approach's general applicability across different LLMs, regardless of their varying capabilities.
Table 6: Results on gpt-3.5-turbo, text-davinci-003, and code-davinci-001. The effectiveness of SATLM can generalize across LLMs.
GSM-SYS	GSM	LSAT	CLUTRR	PROOF
gpt-3.5-turbo (greedy decoding)
COT	44.8	74.4	23.9	41.2	82.3
PROLM	51.2	77.9	-	45.9	76.4
SATLM	63.4	76.4	30.0	50.6	96.4
text-davinci-003 (greedy decoding)
COT	42.8	62.5	21.7	34.5	83.5
PROLM	40.4	71.7	-	41.2	83.7
SATLM	63.6	70.3	30.4	58.2	99.7
code-davinci-001 (greedy decoding)
PROLM	15.5	35.6	-	22.2	63.8
SATLM	16.5	34.2	19.6	30.2	86.6
Sensitivity to Different Exemplar Sets We test whether the advantages of SATLM is sensitive to different sets of exemplars. We experiment with 3 sets of exemplars on code-davinci-002. As shown in Table 7, SATLM consistently outperforms PROGLM by a large margin on GSM-SYS and CLUTRR, and achieves comparable performance on GSM. The results suggest the effectiveness of our approach is insensitive to varying the choice of exemplars.
Our work is built on top of few-shot prompting (Brown et al., 2020), which has proven effective on a wide range of tasks (Wei et al., 2022b; Liu et al., 2023b; Gehrmann
et al., 2021; Reif et al., 2022; Wei et al., 2022a; Sanh et al., 2022). In particular, we focus on improving LLMs on reasoning tasks, which are challenging for language models even with recent
Table 7: The performance of PROGLM and SATLM with varying exemplar sets. SATLM consistently outperforms PROGLM on GSM-SYS and CLUTRR.
GSM-SYS	GSM	CLUTRR
Set1	PROG	43.4	72.7	58.9
SAT	69.4	71.8	68.3
Set2	PROG	41.4	72.5	59.0
SAT	71.8	71.3	67.9
Set3	PROG	37.1	70.3	57.2
SAT	66.7	70.0	68.0
developments (Marcus, 2020; Garcez and Lamb, 2023). Various techniques have been proposed for improving reasoning abilities (Nye et al., 2021; Zhou et al., 2022; Kojima et al., 2022; Khot et al., 2022; Fu et al., 2022; Wang et al., 2022a; Li et al., 2022a; Lyu et al., 2023). They largely follow a chain-of-thought (Wei et al., 2022c) or scratchpad (Nye et al., 2021) paradigm. Among them, our work is the most related to the line of work that generates imperative programs to be executed by a symbolic executor, such as a Python interpreter (Gao et al., 2023; Chen et al., 2022) or domain-specific executors (Lyu et al., 2023). In this work, we propose a different paradigm that parses NL problems into declarative SAT problems and offloads the solving procedure to a SAT solver.
Previous work has also explored equipping LLMs with other tools, including search engines (Yu et al., 2023; Schick et al., 2023), calculators (Cobbe et al., 2021; Chowdhery et al., 2022), or other domain-specific special modules (Schick et al., 2023; Demeter and Downey, 2020). A line of work focuses on using program-related tools such as program executors (Poesia et al., 2022), program analysis tools (Jain et al., 2022), and synthesis tools (Rahmani et al., 2021) to enhance the quality of the generated code. Our works further explores improving LLMs with SAT solvers.
Concurrent work explores the intersection of LLMs and planning, parsing planning problems into PDDL descriptions and leveraging a classical planner to produce the plan (Liu et al., 2023a). Our work differs in that we use the SAT formulation to solve general reasoning tasks, including arithmetic reasoning and logical reasoning, which cannot be specified in PDDL.
Also concurrently, He-Yueya et al. (2023) combine LLMs and symbolic solvers for solving math problems. However, this work only focuses on arithmetic reasoning tasks and employs a math-specific symbolic solver (PySym). Our work takes a more general approach by formulating the problem within the scope of first-order logic and therefore is domain-agnostic. We also provide results of SATLM on the ALGEBRA dataset collected by He-Yueya et al. (2023) in Appendix D.
6 Conclusion & Limitations
We have presented a framework for satisfiability-aided language models, casting a wide range of reasoning tasks into SAT problems under a unified formulation. We use an LLM to parse an NL query into a declarative specification and leverages a SAT solver to derive the final answer. Evaluation results on 8 datasets spanning 4 tasks across several LLMs demonstrate the effectiveness of our approach over program-aided language models.
**Limitations** Our framework parses an NL problems into a set of declarative formulas. The NL description of some problems may already be more compatible with an imperative solving procedure, and our approach is likely to be less effective in these cases (e.g., SATLM slightly lags PROGLM on GSM). Future research can explore an integration or ensemble of these two prompting styles for more flexible reasoning.
SATLM heavily relies on the SAT solver and inherits some limitations of the SAT solver itself, such as computational cost when dealing with complex formulas involving quantifiers or nonlinear arithmetic. Moreover, SAT solvers can be limited by the expressiveness of the underlying theory, as not all theories can be easily encoded in first-order logic. Nevertheless, the wide range of tasks that we can instantiate our SATLM framework on shows its general applicability.
Our current approach parses a problem into a SAT specification, runs the solver, and returns the answer in a one-round fashion. One can imagine that unsatisfiable formulas or ambiguous formulas could be improved by re-prompting the model to improve the specification based on the exception signals, as explored in concurrent work for other problems (Paul et al., 2023; Madaan et al., 2023; Chen et al., 2023). We believe this is an exciting direction for future work.
Thanks to anonymous reviewers for their helpful feedback. This work was partially supported by National Science Foundation under Grants No.2145280, No.1918889, No.1762299, No.2210831 and the NSF AI Institute for Foundations of Machine Learning (IFML). We would also like to thank
authors of PAL (Gao et al., 2023) and Faithful CoT (Lyu et al., 2023) for providing the prompts used in the baselines.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le. 2022a. Finetuned language models are zero-shot learners. In International Conference on Learning Representations.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022b. Emergent abilities of large language models. Transactions on Machine Learning Research. Survey Certification.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. 2022c. Chain of thought prompting elicits reasoning in large language models. In Proceedings of the Conference on Advances in Neural Information Processing Systems (NeurIPS).
Xi Ye, Qiaochu Chen, Isil Dillig, and Greg Durrett. 2020. Benchmarking multimodal regex synthesis with complex structures. In Proceedings of the Annual Conference of the Association for Computational Linguistics (ACL).
Xi Ye, Qiaochu Chen, Isil Dillig, and Greg Durrett. 2021. Optimal neural program synthesis from multimodal specifications. In Findings of the Association for Computational Linguistics: EMNLP (EMNLP Findings).
Xi Ye and Greg Durrett. 2023. Explanation selection using unlabeled data for chain-of-thought prompting. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP).
Wenhao Yu, Dan Iter, Shuohang Wang, Yichong Xu, Mingxuan Ju, Soumya Sanyal, Chenguang Zhu, Michael Zeng, and Meng Jiang. 2023. Generate rather than retrieve: Large language models are strong context generators. In International Conference for Learning Representation (ICLR).
Hanlin Zhang, Ziyang Li, Jiani Huang, Mayur Naik, and Eric Xing. 2022a. Improved logical reasoning of language models via differentiable symbolic programming. In First Workshop on Pre-training: Perspectives, Pitfalls, and Paths Forward at ICML 2022.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuhui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022b. OPT: Open Pre-trained Transformer Language Models. ArXiv, abs/2205.01068.
Wanjun Zhong, Siyuan Wang, Duyu Tang, Zenan Xu, Daya Guo, Yining Chen, Jiahai Wang, Jian Yin, Ming Zhou, and Nan Duan. 2022. Analytical reasoning of text. In *Findings of the Association for Computational Linguistics: NAACL (NAACL Findings)*.
Denny Zhou, Nathanael Scharli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Olivier Bousquet, Quoc Le, and Ed Chi. 2022. Least-to-most prompting enables complex reasoning in large language models. ArXiv, abs/2205.10625.
A Detailed Statistics of Datasets
We show the statistics of all the datasets used in our paper in Table 8.
For CLUTRR, we follow the setting in FAITHFULCOT (Lyu et al., 2023): we construct the prompt using exemplars requiring 2-3 reasoning steps and test whether the model can generalize to examples requiring up to 10 steps. We used the pre-processed test data consisting of 1,042 test examples from past work (Lyu et al., 2023).
For PROOFWRITER, we use the closed world assumption setting, following past work (Creswell et al., 2023). We construct our test set by randomly sampling a subset of 1,000 examples (out of 10,000) from the test split of depth-5 setting, the most challenging setting.
For STREGEX, we merge the test and test-E split (see Ye et al. (2020)) to form a test set consisting of 996 examples in total.
Table 8: Number of few-shot exemplars, number of test examples and license for the datasets used in our paper.
# Shot	# Test	License
GSM (Cobbe et al., 2021)	8	1,319	MIT license
GSM-SYS	8	547	MIT license
ALGEBRA (He-Yueya et al., 2023)	8	222	Creative Commons Attribution Share Alike 4.0
LSAT (Zhong et al., 2022)	8	230	MIT license
BOARDGAMEQA (Kazemi et al., 2023)	5	3,000	CC BY 4.0.
CLUTRR (Sinha et al., 2019)	8	1,042	Attribution-NonCommercial 4.0
PROOFWRITER (Tafjord et al., 2021)	4	1,000	CC BY 4.0.
COLOREDOBJECT (BIG-BENCH)	3	2,000	Apache 2.0
STRUCTUREDREGEX (Ye et al., 2020)	8	996	MIT license
GSM-Sys Dataset We construct GSM-Sys, a special subset consisting of 547 examples extracted from GSM. Specifically, we filter the entire GSM dataset (train split and test split) to find examples whose human-annotated explanations involve a system of equations, using patterns like "let [letter] be", "assume [letter] be" and "[number][letter]". We manually inspected  $10\%$  of the examples and found  $80\%$  of those samples did involve systems of equations in the explanation. We refer to this more challenging dataset as GSM-Sys.
B Details of the Prompts
In general, we leverage COT prompts and PROGLM prompts from existing work whenever available, and manually write SATLM prompts for the same exemplar sets. Prompt examples for all datasets can be found in Appendix I.
For GSM and GSM-Sys, we adapt the original COt prompt and PROGLM prompt used in program-aided language models (Gao et al., 2023). Specifically, we replace one random exemplar in the original prompt with another exemplar sampled from GSM-Sys. This is to improve the performance of COt and PROGLM on GSM-Sys, as the original exemplar set achieves suboptimal performance for GSM-Sys. Our adapted COt and PROGLM prompts achieve better performance compared to the original ones on both GSM and GSM-Sys (see Appendix C for details).