SlowGuess/NeurIPS_2023_OCR huggingface dataset:
SATLM: Satisfiability-Aided Language Models Using Declarative Prompting
Xi Ye Qiaochu Chen Isil Dillig Greg Durrett
{xiye,qchen,isil,gdurrett}@cs.utexas.edu
Prior work has combined chain-of-thought prompting in large language models (LLMs) with programmatic representations to perform effective and transparent reasoning. While such an approach works well for tasks that only require forward reasoning (e.g., straightforward arithmetic), it is less effective for constraint solving problems that require more sophisticated planning and search. In this paper, we propose a new satisfiability-aided language modeling (SATLM) approach for improving the reasoning capabilities of LLMs. We use an LLM to generate a declarative task specification rather than an imperative program and leverage an off-the-shelf automated theorem prover to derive the final answer. This approach has two key advantages. The declarative specification is closer to the problem description than the reasoning steps are, so the LLM can parse it out of the description more accurately. Furthermore, by offloading the actual reasoning task to an automated theorem prover, our approach can guarantee the correctness of the answer with respect to the parsed specification and avoid planning errors in the solving process. We evaluate SATLM on 8 different datasets and show that it consistently outperforms program-aided LMs in the imperative paradigm. In particular, SATLM outperforms program-aided LMs by  $23\%$  on a challenging subset of the GSM arithmetic reasoning dataset; SATLM also achieves a new SoTA on LSAT and BOARDGAMEQA, surpassing previous models that are trained on the respective training sets. $^{1}$
Using large language models (LLMs) to perform complex reasoning has been a central thrust of recent research (Brown et al., 2020; Chowdhery et al., 2022; Rae et al., 2021; Zhang et al., 2022b). Techniques like scratchpads (Nye et al., 2021) or chain-of-thought prompting (CoT) (Wei et al., 2022c) enable LLMs to follow a sequence of reasoning steps before making a prediction. This paradigm is effective on various multi-step reasoning tasks, especially those with fixed forward reasoning procedures (Wei et al., 2022c), e.g., concatenating the last letters of several words. However, CoT prompting can fall short when scaling to problems that involve intensive computation (Gao et al., 2023) or long sequences of reasoning steps (Creswell et al., 2023; Saparov and He, 2023; Ribeiro et al., 2023).
Solving a complex reasoning problem involves three conceptual components: parsing a natural language description into a representation of the problem, deriving a plan to solve the problem, and executing that plan to obtain an answer. Recent work on improving CoT prompting focuses on fixing execution errors by augmenting LLMs with symbolic executors such as a Python interpreter, which
$^{1}$ Code available at https://github.com/xiye17/SAT-LM.
Each of five students—Hubert, Lori, Paul, Regina, and Sharon—will visit exactly one of three cities—Montreal, Toronto, or Vancouver, according to the following conditions: Sharon visits a different city than Paul. Hubert visits the same city as Regina. Lori visits Montreal or else Toronto. If Paul visits Vancouver, Hubert visits Vancouver with him. Each student visits one of the cities with at least one of the other four students.
Question: Which one of the following must be true?
(A) If any of the students visits Montreal, Lori visits Montreal. (B) [...]
Chain-of-Thought Prompting (imperative specification)
Satisfiability-Aided LM (ours; declarative specification)
We know each student visits one of the cities with at least one of the other four students. We know there are five students and three cities. So there must be three students visiting the one city and two other students visiting another city.
Let's consider option (A).
Assume someone visits Montreal, but Lori does not visit Montreal.
We know Lori visits Montreal or else Toronto. So Lori
visits Toronto.
Assume Sharon visits Toronto with Lori.  
We know Sharon visits a different city than Paul. So, Paul has to visit Montreal.
Hubert and Regina can visit Montreal with Paul with no conflicts. So Lori does not necessarily visit Montreal. This statement is False.
The LLM parses the question, plans the reasoning, and executes it all in the CoT (shown by dashed arrows)
students  $=$  [Hubert, Lori, Paul, Regina, Sharon] cities  $=$  [Montreal, Toronto, Vancouver] visits  $=$  Function(students, cities) # Sharon visits a different city than Paul
visits(Sharon) != visits(Paul)
Lori visits Montreal or else Toronto
2Or(visits(Lori) == Montreal, visits(Lori) == Toronto)
Each student visits one of the cities with at least one other student  
 $\oplus$  ForAll  $([\{s1\} ,$  exists([s2], And(s2 != s1, visits(s1) == visits(s2))))
#（A）
0 solve(Implies(Exists([s], visits(s) == Montreal), visits(L) == Montreal))
The LLM only parses the question to a problem specification in this step
Figure 1: Illustration of our Satisfiability-aided Language Modeling approach (right). We first parse an NL input into a declarative task specification (a set of logical constraints) using prompting (Section 3.1), then use a SAT solver to solve the problem (Section 3.2). The chain-of-thought strategy in prior work (left) yields imperative reasoning processes.
A SAT solver generates and executes a proof plan using automated theorem proving
leads to improved performance on arithmetic and symbolic reasoning tasks (Gao et al., 2023; Chen et al., 2022; Lyu et al., 2023). However, CoT prompting (Wei et al., 2022c; Nye et al., 2021) and its executor-augmented successors (Gao et al., 2023; Chen et al., 2022; Lyu et al., 2023) are oriented towards imperative solving procedures: a CoT or a program specifies the reasoning procedure as chained steps (Wei et al., 2022c; Gao et al., 2023) in the order of execution. While this is effective for problems whose natural language already provides a suitably clear "plan" for the reasoning, it only leads to limited success for reasoning problems like in Figure 1 that do not outline such a plan (Ribeiro et al., 2023). These problems often state a set of premises and constraints and ask questions that require sophisticated planning to deductively reason over the inputs, which is still challenging even for modern LLMs (Valmeekam et al., 2022).
Our work tackles both execution errors and, more importantly, planning errors. We propose SATisfiability-aided Language Modeling (SATLM) using declarative prompting. The core idea is to cast a natural language (NL) reasoning problem as a satisfiability (SAT for short) problem. As shown in Figure 1 (right), given a problem in NL, we prompt an LLM to parse it into a SAT problem specification which consists of a set of logical formulas, then obtain the solution by invoking a SAT solver. The LLM is specialized towards understanding the preconditions stated in the problem, while the solver is leveraged to plan out the reasoning procedure. In addition, the solver guarantees the correctness of execution, similar to the interpreter used in program-aided LMs (PROGLM).
We evaluate our approach on 8 datasets spanning 4 tasks, including arithmetic reasoning, logical reasoning, symbolic reasoning, and a regex synthesis task. Our SATLM consistently outperforms COT and PROGLM across all datasets, usually by a large margin. On GSM-SYS, SATLM outperforms PROGLM by a  $23\%$ ; on GSM, SATLM achieves  $84.8\%$  with self-consistency decoding using few-shot prompting, equaling past work that uses the full training set and the same LLM (Li et al., 2022b; Ni et al., 2023). SATLM also sets a new SoTA on LSAT (Zhong et al., 2022), BOARDGAMEQA (Kazemi et al., 2023), and STRUCTUREDREGEX (Ye et al., 2020).
Our analysis illustrates why the combination of SAT solver and declarative prompting is so effective. We find (1) program-aided LMs often make planning errors (e.g., manipulating equations incorrectly), which can be remedied by the SAT solver. (2) Forcing LLMs to explicitly state a declarative
Here, we use SAT solver to refer to any automated reasoning tool for checking the satisfiability of formulas in formal logic. Hence, "SAT solver" in this paper also includes first-order theorem provers and SMT solvers.
specification can even improve vanilla CoT prompting. (3) Our SATLM approach can abstain from making uncertain predictions if it parses a problem into an unsatisfiable or ambiguous specification, giving it even higher accuracy in the selective prediction setting (El-Yaniv and Wiener, 2010).
2 Overview
This work addresses the challenge of using LLMs to solve NL reasoning tasks. At a high level, an NL reasoning task is a natural language description of a collection of facts  $\Phi$  (such as propositions or constraints) about some objects and a question  $Q$  related to these objects. The goal of the reasoning task is to find an answer to  $Q$  that can be deduced from the information provided in  $\Phi$ .
We conceptualize the general procedure for solving NL reasoning tasks in three steps: parsing, planning, and execution. We are given natural language input  $x_{\mathrm{test}} = (NL(\Phi), NL(Q))$  which describes both  $\Phi$  and  $Q$ . Our first step is to parse this natural language into a predicted task specification  $(\hat{\Phi}, \hat{Q})$ , which is a formal description of the facts and the query.
Given  $(\hat{\Phi},\hat{Q})$ , the planning step then involves determining a sequence of reasoning steps  $[r_1,\dots ,r_n]$  beginning with the task specification and ending with the answer to the question. Each step involves invoking a function (e.g., arithmetic operator or logical operator) that produces intermediate results which can be utilized in subsequent steps. A plan can be formulated by an LLM with CoT prompting or by a symbolic solver as in our work here. Finally, we execute the plan systematically with either a symbolic executor (our method) or an LLM, returning the output of the last step,  $r_n$ , as the answer.
Our solution approaches the problem using exactly these three steps.
Parsing into declarative specification We prompt an LLM to generate a specification  $s_{\mathrm{test}}$  for  $x_{\mathrm{test}}$ . Note that the translation from this description into the specification is not straightforward and cannot be done in a rule-based way for most tasks; Figure 4 shows some particularly complex examples involving commonsense reasoning. The specification  $s_{\mathrm{test}}$  is a sequence of interleaved NL statements and logical formulas (LF):  $s_{\mathrm{test}} = [z_1, \dots, z_n]$  and  $z_i \in \Sigma_{NL} \cup \Sigma_{LF}$ , where  $\Sigma_{NL}$  and  $\Sigma_{LF}$  denote the space of natural language and logical formulas, respectively. We derive the formal specification  $(\hat{\Phi}, \hat{Q})$  by taking all the  $z_i$  in  $\Sigma_{LF}$  from  $s_{\mathrm{test}}$ . An example of the specification is presented on the right of Figure 1. Our specification is declarative since we do not explicitly generate the  $r_i$  from the LLM at this stage.
Planning and execution with a SAT solver Given the predicted formal specification  $(\hat{\Phi},\hat{Q})$ , we wish to derive the final answer of the query  $\hat{Q}$  from it. We say that a solution  $a$  is correct if  $\hat{\Phi}$  logically entails  $\hat{Q} = a$ , denoted as  $\hat{\Phi} \models \hat{Q} = a$ . The key insight behind our work is to offload both the planning and execution steps to a SAT solver. Specifically, we use a SAT solver to find a satisfying assignment for  $a$  in the formula:
$$
\forall V. (\hat {\Phi} \Rightarrow \hat {Q} = a)
$$
where  $V$  denotes the set of all variables used in  $\hat{\Phi}$  and  $\hat{Q} \in V$  is a variable that corresponds to the solution. Note that the only free variable in this formula is  $a$ ; hence, the assignment to  $a$  returned by the solver is the final answer to the reasoning problem.
The approach outlined above has two important strengths. First, because the SAT solver is sound (i.e., any assignment it produces satisfies the formula), the solution is correct by construction. Thus, assuming that the parsing is correct and  $\hat{\Phi}$  and  $\hat{Q}$  match  $\Phi$  and  $Q$ , we have a proof that the solution is indeed correct. Second, the planning step is done internally to the solver, and the chain of reasoning steps  $[r_1,\ldots ,r_n]$  can be obtained by asking the solver to produce a proof of the validity of the formula  $\hat{\Phi}\Rightarrow \hat{Q} = a^{*}$  where  $a^*$  is the assignment produced by the SAT solver. All solvers we consider can produce such a proof of validity (e.g., in the form of a resolution refutation (Davis and Putnam, 1960)).
Comparison with prior work Prior approaches to NL-based reasoning with LLMs can also be framed in the parse-plan-execute framework proposed above. In particular, the chain-of-thought paradigm (Nye et al., 2021; Wei et al., 2022c) uses LLMs to perform each of the three steps. Program-aided language models (Gao et al., 2023; Chen et al., 2022; Lyu et al., 2023) combine the parsing and
Input Q: Alex, Stan, and Adelwolf are trying to catch them all, Pokemon that is. Together they have caught 339 Pokemon. Alex has caught 5 more than Stan, and Stan has caught 13 less than 4 times as many as Adelwolf has caught. How many Pokemon has Stan caught?
Figure 2: Exemplar specifications for arithmetic reasoning problems generated by different approaches. COT makes errors when parsing an equation; PROGLM produces an incorrect reasoning chain (both errors are highlighted in red). By only using the LLMs to generate declarative specifications and relying on a solver to handle the reasoning, SATLM generates the correct answer.
planning steps to use an LLM to derive a program that corresponds to the plan. $^3$  The final execution step is then performed by using the interpreter of the underlying programming language to derive the final answer. In contrast to these approaches, our work uses an LLM only to perform the parsing step, which is an easier problem for LLMs than planning.
We show a concrete example comparing COT and PROGLM with our approach in Figure 2. CoT performs all three steps with the LLM. For instance, "Alex has caught  $X + 5$ " in the output corresponds to "Alex has caught 5 more than Stan" in the NL input (Parsing). Later, COT decides how to solve for the variable  $X$  with "Combining like terms ..." (planning). At the same time, it also derives the equation "  $5X = 334$  " directly in its generation (execution). However, COT incorrectly uses the same  $X$  in the equation "  $X + 5$  " and "  $4X - 13$  ", when it is supposed to be different. (Note that  $4X - 13$  would be correct if Stan and Adelwolfe's roles in the corresponding NL clause were reversed.) By allowing the LLM to focus only on translation, we find a lower incidence of this kind of error, in addition to eliminating planning errors. Notably, planning errors are not addressed by PROGLM, which does not use programmatic manipulation at this stage. Different from PROGLM, SATLM only parses the information provided in the input question, passes the parsed formulas to a solver for both planning and execution, and obtains the correct result.
3 SAT-Aided Language Models using Declarative Prompting
3.1 Declarative Prompting
We use few-shot prompting to generate the specification  $s_{\mathrm{test}}$  for the test input  $x_{\mathrm{test}}$ . Specifically, we include few-shot demonstrations  $(x_i, s_i)_{i=1}^k$  in the prompt, append test input  $x_{\mathrm{test}}$  after the prompt, and let the LLM complete the specification for  $x_{\mathrm{test}}$ , i.e.,  $s_{\mathrm{test}} \sim p(x_{\mathrm{test}} \mid x_1, s_1, \ldots, x_k, s_k)$ .
We show an example specification for a logical reasoning task in Figure 1, and an example specification for an arithmetic reasoning task in Figure 2. Observe that in both examples, our SAT formulas (i.e., the logical formulas of  $[z_1,\dots ,z_n]$  in  $\Sigma_{LF}$ ) are written as code following Python syntax, while the natural language in  $\Sigma_{NL}$  is written using comment syntax. We found that including the language here as comments was useful to improve the fidelity of the translation. Our declarative prompts also use meaningful variable names and descriptive comments following the style of prompts in prior work (Gao et al., 2023; Lyu et al., 2023). Finally, we use Python rather than a specialized DSL to be more congruent with our models' pretraining data (Ouyang et al., 2022; Chen et al., 2021). See Appendix F for more details on the SAT specification.
This is true for "faithful chain-of-thought" as well (Lyu et al., 2023). This paper describes a breakdown of the process into "translation" and "solving" stages, where the translation step corresponds to both our parsing and planning stages. The solver used in that approach for tasks like CLUTRR does not do additional planning, but merely executes the steps outlined in CoT. In addition, their approach uses Python for execution, whereas ours uses SAT and Z3 as the unifying solving framework.
3.2 Solving with a SAT Solver
SAT problem A SAT problem is a triple  $\mathcal{P} = (\Phi, \mathcal{T}, Q)$  where  $\Phi$  is a set of first-order logic formulas in some theory  $\mathcal{T}^4$  and  $Q$  is the query of interest. We use Variable  $(\mathcal{P})$  to denote the free variables in  $\Phi$ .  $Q$  contains only variables in Variable  $(\mathcal{P})$ . An example SAT problem is  $\mathcal{P} = (\{x + y = 3, x - y = 1\}, \mathcal{T}_E \cup \mathcal{T}_{\mathbb{Z}}, x - 2)$ , where  $\mathcal{T}_E \cup \mathcal{T}_{\mathbb{Z}}$  indicates that only equality and linear arithmetic operations on integers are allowed in the formulas.
Many NL reasoning tasks in the literature can be formulated as SAT problems and solved using an off-the-shelf solver. For arithmetic reasoning, the SAT formulas  $\Phi$  are equations encoding the relationships between variables, and  $t$  specifies the target variable asked in the question (see Figure 1). For logical reasoning,  $\Phi$  encodes preconditions and  $t$  specifies the target statement posed by the question. We also show that symbolic reasoning, regex synthesis, and other problems involving reasoning over arrays or strings can be handled in this framework.
Unlike prior work such as Faithful CoT (Lyu et al., 2023) that uses task-specific formulations and task-specific solvers for different problem types, all the tasks in this paper are formulated as general SAT instances that can be solved by a single solver (as described later in this section).
Parsing NL to a SAT problem Recall that we obtain a specification  $s_{\mathrm{test}}$  from a test NL task  $x_{\mathrm{test}}$ . To derive the SAT problem  $\mathcal{P}_{\mathrm{test}} = (\hat{\Phi}_{\mathrm{test}}, \mathcal{T}_{\mathrm{test}}, \hat{Q}_{\mathrm{test}})$  from  $s_{\mathrm{test}}$ , we extract the constraints  $\hat{\Phi}_{\mathrm{test}}$  and the target expression  $\hat{Q}_{\mathrm{test}}$  (marked by solve in our prompt) by taking all the  $z_i$  in  $\Sigma_{LF}$  of  $s_{\mathrm{test}}$ . We identify the theory  $\mathcal{T}_{\mathrm{test}}$  by analyzing the formulas in  $\hat{\Phi}_{\mathrm{test}}$ .
Solving the SAT problem Given the SAT problem  $\mathcal{P}$ , we invoke an automated theorem prover (such as the Z3 SMT solver (De Moura and Bjørner, 2008) used in our implementation) to obtain a model  $M$  that maps each free variable  $v \in \mathrm{Variable}(\mathcal{P})$  to a concrete value under theory  $\mathcal{T}$ . The final answer is obtained by substituting each free variable  $v_i$  in  $\hat{Q}$  with  $M[v_i]$ . For example, given the problem  $(\{x + y = 3, x - y = 1\}, \mathcal{T}_E \cup \mathcal{T}_{\mathbb{Z}}, x - 2)$ , we ask the solver to find a solution to the constraint  $x + y = 3 \wedge x - y = 1$  in the theory  $\mathcal{T}_E \cup \mathcal{T}_{\mathbb{Z}}$ , which yields  $x = 2$  and  $y = 1$ . Then, to obtain the final answer, we substitute  $x$  by 2 in the target expression  $x - 2$  to obtain the result  $2 - 2 = 0$ .
Feedback signals from the solver Given a set of  $\hat{\Phi}$  specified in  $\mathcal{P}$ , the SAT solver will try to search for a satisfying assignment  $M$  which satisfies all constraint formulas in  $\hat{\Phi}$ . If the solver succeeds in finding such an assignment within a certain time limit, it will use  $M$  to evaluate the query  $\hat{Q}$  and return the final result, otherwise it is a timeout. However, the solver may fail to find a solution for problematic  $\mathcal{P}$  and provide feedback in one of the following types: (1) error in execution (ERROR) caused by invalid formulas (e.g., syntax errors) or time-out; (2) unsatisfiable formulas (UNSAT), caused by conflicting formulas in the  $\hat{\Phi}$  (e.g.  $\hat{\Phi} = \{x = y + 1, y = x + 1\}$ ) (no feasible solution); (3) ambiguous formulas (AMBIG), caused by the existence of multiple feasible solutions (e.g.  $\hat{\Phi} = \{x = y + 1, x > 0\}$ ). Examples of SAT formulas leading to UNSAT or AMBIG can be found in Appendix G.
Unlike the executor used in PROGLM that can only detect errors in code execution, SAT solver can spot UNSAT and AMBIG in addition to ERROR. We show this unique characteristic allows our SATLM to abstain from potentially incorrect predictions much more effectively compared to PROGLM in the selective prediction setting (El-Yaniv and Wiener, 2010) (Section 4.4).
Tasks Our work investigates 8 datasets covering 4 tasks, with a focus on arithmetic reasoning and logical reasoning tasks. We list all dataset statistics in Appendix A. For arithmetic reasoning, we use GSM (Cobbe et al., 2021), GSM-SYS, and ALGEBRA (He-Yueya et al., 2023). GSM-SYS
4The theory defines the meaning of some of the symbols used in the formula. For example, in the theory of linear arithmetic, axioms of the theory give meaning to operators like addition, less than, etc.
Table 1: Comparison of our approach (SATLM) against standard prompting (directly predicting the answer), COT and PROGLM. Certain settings are not applicable (marked as  $-$ ) as described in Appendix B. With greedy decoding, SATLM outperforms COT and PROGLM on all datasets by a substantial margin except for GSM, where it is on par with PROGLM. With self-consistency decoding, SATLM is consistently better than PROGLM, giving SoTA accuracy on LSAT and BOARDGAMEQA.
GSM-SYS	GSM	ALGE	LSAT	BOARD	CLUTRR	PROOF	COLOR	REGEX
code-davinci-002 (greedy decoding)
STANDARD	21.0	22.2	45.9	22.0	44.6	41.2	76.6	75.7	-
COT	46.5	62.7	53.6	23.5	60.7	40.8	80.1	86.3	-
PROGLM	43.4	72.7	52.3	-	-	58.9	83.7	95.1	39.1
SATLM	69.4	71.8	77.5	35.0	79.4	68.3	99.7	97.7	41.0
code-davinci-002 (self-consistency decoding)
COT	56.1	77.3	64.9	23.1	62.8	45.7	88.7	90.6	-
PROGLM	53.4	82.4	57.7	-	-	71.9	91.2	98.0	56.5
SATLM	80.9	84.8	90.9	37.4	80.7	80.1	99.7	99.4	59.7
is a special subset of GSM containing examples that are paired with human-annotated solutions involving systems of equations (see Appendix A for more details). For logical reasoning, we use LSAT (Zhong et al., 2022), BOARDGAMEQA (Kazemi et al., 2023), CLUTRR (Sinha et al., 2019), and PROOFWRITER (Tajord et al., 2021). For BOARDGAMEQA, we report the average performance on the three data splits (depth 1 to depth 3).
For CLUTRR, we use exemplars requiring up to 3 intermediate steps but evaluate on test examples requiring up to 10 intermediate steps (Sinha et al., 2019), following past work (Lyu et al., 2023). For PROOFWRITER, we evaluate on the most challenging examples requiring depth-5 proofs (Tafjord et al., 2021). For symbolic reasoning, we use Colored Object (COLOR) from BIG-bench (et al., 2022) as an exemplar task. This task can be abstracted as finding elements in a list under certain constraints. We also evaluate on a regex synthesis dataset, STREGEX (Ye et al., 2020), which requires synthesizing a regex give NL description. We cast this task into synthesizing the surface form (i.e., a string) of the target regex, and use SATLM to parse NL description into constraints over the string.