Natural Language Satisfiability: Exploring the Problem Distribution and Evaluating Transformer-based Language Models (2024)

Abstract: Efforts to apply transformer-based language models (TLMs) to the problem of reasoning in natural language have enjoyed ever-increasing success in recent years. The most fundamental task in this area to which nearly all others can be reduced is that of determining satisfiability. However, from a logical point of view, satisfiability problems vary along various dimensions, which may affect TLMs’ ability to learn how to solve them. The problem instances of satisfiability in natural language can belong to different computational complexity classes depending on the language fragment in which they are expressed. Although prior research has explored the problem of natural language satisfiability, the above-mentioned point has not been discussed adequately. Hence, we investigate how problem instances from varying computational complexity classes and having different grammatical constructs impact TLMs’ ability to learn rules of inference. Furthermore, to faithfully evaluate TLMs, we conduct an empirical study to explore the distribution of satisfiability problems.

heading: 1 Introduction
text: The impressive performance of transformer-based language models (TLMs) in natural language inference tasks (Devlin et al., 2018; Yang et al., 2019; Raffel et al., 2019; Liu et al., 2019) has created a surge of interest in the development of linguistic and deductive reasoning benchmarks to evaluate these models (Richardson et al., 2020; Geiger et al., 2018; Tafjord et al., 2021; Yanaka et al., 2020). One such area of interest is the ability to recognise valid entailments, understood in a strictly logical sense (Clark et al., 2021; Richardson and Sabharwal, 2021), where the inference does not depend on background knowledge and commonsense. The ability of TLMs to understand this type of entailment is indicative of their ability to learn rules of inference, understand the logical semantics of natural language and emulate complex algorithms.
Since the inferences we shall be concerned with do not depend on background knowledge or intuitive plausibility, they can be captured using the apparatus of formal logic. Logicians usually find it convenient to reconstruct entailment in terms of satisfiability: a set of formulae Φ is satisfiable if there is a structure (in the model-theoretic sense) A such that every formula of Φ is true in A. A set of formulae Φ entails a formula ψ just in case Φ∪{¬ψ} is not satisfiable. The same duality holds for natural language (which incorporates a concept of negation) just as it does for formal logic. Thus, we consider the following problem: given a collection of sentences expressed in a natural language such as English, determine whether it is—in a strictly logical sense—satisfiable. Figure 1 illustrates two instances of the satisfiability problem in English, one positive and one negative. Note that any method for determining satisfiability yields a method for recognising entailments and vice versa.
We approach this problem in a controlled way. The sentences in Figure 1 feature only rudimentary grammatical resources: the determiners some, no and all, transitive verbs with unqualified nounphrases as subjects and objects, and the copula is. It is well-known from formal logic that the computational complexity of the satisfiability problem for a logic depends on its expressive power. For example, the satisfiability problem for propositional logic (known as SAT) is NPTIME-complete, but the corresponding problem for the two-variable fragment of first-order logic is NEXPTIME-complete, while for the whole of first-order logic is r.e.-complete (i.e. undecidable); for a survey, see Pratt-Hartmann (2023). The same is true when it comes to fragments of natural languages (Pratt-Hartmann, 2003, 2004; Pratt-Hartmann and Third, 2006), as we explain presently. The question therefore arises as to how TLMs’ ability to learn rules of inference is affected by the fragment of language within which they operate.
15278
In recent years, researchers have analysed the limitations of various neural approaches when solving satisfiability problems (Selsam et al., 2019; Evans et al., 2018; Cameron et al., 2020b), including natural language satisfiability problems (Richardson and Sabharwal, 2021). However, in all cases, the problems are in propositional logic or its close relatives, and as such, the expressive power of the expressions or sentences is limited. Moreover, it restricts the problem space to be in a single computational complexity class, NP-complete. We overcome these limitations by generating sentences from various fragments of English that are related to more varied fragments of first-order logic. By utilising language fragments of varying expressive power, we also provide an analysis of how computational complexity affects TLMs’ ability to grasp the rules governing logical entailment.
When investigating TLMs’ ability to solve instances of natural language satisfiability problems, it is important to ensure that the training and test data sets include a sufficient number of challenging problem instances. The difficulty is that randomly constructed sets of formulae (or sentences) are, depending on the sampling parameters, in many cases easily seen either to be satisfiable or to be unsatisfiable, a situation dramatically illustrated in the case of propositional logic satisfiability (Selman et al., 1996; Cook and Mitchell, 1997; Mitchell and Levesque, 1996), where challenging problems occur only when the ratio of clauses to propositional variables is close to a critical threshold, i.e., the so-called phase-change region. In this region, where the probability of a randomly generated problem instance is close to 0.5, algorithms for determining satisfiability typically exhibit long running times. Therefore, to demonstrate that TLMs can learn rules of inference, we must ensure that they work on the hardest region of the target problem
space. Indeed, recent work has shown some pitfalls associated with synthetic data due to undersampling challenging problems (Shin et al., 2019; Wu et al., 2021). Consequently, we conduct an empirical study to determine, for each of the language fragments investigated, the relevant “phase-change” region where the challenging problem instances are to be found.
The contributions of this paper are as follows. (1) We identify a number of fragments of English featuring a range of logico-syntactic constructions involving determiners, the copula, transitive verbs, relative clauses and bound-variable anaphora. (2) We empirically determine the phase-change region for these fragments and construct data sets containing instances of the satisfiability problems sampled from these regions. (3) We investigate the ability of transformer-based language models to solve the satisfiability problem for the investigated fragments and carry out a systematic analysis of how the underlying computational complexity of the satisfiability problem correlates with TLMs’ ability to grasp the relevant inferential principles. (4) Furthermore, we explore the proficiency of TLMs in solving instances of the satisfiability problem within a zero-shot context. To the best of our knowledge, this investigation represents the first attempt to probe the behaviour of TLMs in zero-shot scenarios with respect to their efficacy in solving satisfiability problems.

heading: 2 Related Work
text: Our work closely follows the literature on identifying strengths and weaknesses of neural approaches, including transformer-based language models on deductive reasoning tasks (McCoy et al., 2019; Glockner et al., 2018; Lin et al., 2019). Neural networks, particularly graph neural networks (GNN), have been utilised to solve satisfiability problem in-
stances (Xu et al., 2020; Cameron et al., 2020a). As mentioned in the introduction, various studies have extended that work to natural language satisfiability, utilizing TLMs instead of GNNs (Richardson and Sabharwal, 2021). However, the sentences considered fail to exercise the full range of inference patterns licenced by commonly encountered grammatical constructions. Moreover, in each case, the underlying satisfiability problems were derived from that for propositional logic (i.e. SAT), thus, confining attention to a single problem type. Indeed in the case of natural language satisfiability, the sentences are sometimes hardly natural-sounding. For example, the Grounded Rule Language (GRL) fragment introduced in Richardson and Sabharwal (2021) includes sentences such as “If carrot and not steak then apples”.
Our work can also be viewed as an attempt to identify various limitations that affect TLMs’ ability to emulate algorithms. There have been numerous studies of TLMs’ ability to solve algorithmic tasks, including SAT-solving (Selsam et al., 2019; Narodytska et al., 2020), semantic parsing (He and Choi, 2020; Kamath and Das, 2019), modelchecking (Madusanka et al., 2023a,b), theorem proving (Weber et al., 2019; Minervini et al., 2020; Saha et al., 2020; Welleck et al., 2021) etc. We extend this work to a family of related problems spanning multiple computational complexity classes, thus, providing a breakdown of the impact of computational complexity on TLMs’ ability to solve algorithmically challenging problems.

heading: 3 Methodology

heading: 3.1 Language Fragments
text: By a natural language fragment, we understand a set of sentences in some natural language equipped with truth-conditional semantics in terms of which the dual notions of validity and satisfiability can be defined. Consider, for example, the set of sentences of the forms
Every p is a q Some p is a q No p is a q Some p is not a q,
where p and q range over common (count) nouns such as artist, beekeeper, carpenter . . . . Their semantics can be given by the familiar translation into first-order logic over a non-logical vocabulary (of unary predicates) corresponding to the schematic variables in question: ∀x(p(x)→ q(x)) ∃x(p(x) ∧ q(x)) ∀x(p(x)→ ¬q(x)) ∃x(p(x) ∧ ¬q(x)).
These semantics assume that universal sentences have no existential import (i.e. Every p is a q does not entail that there are any p’s); otherwise, however, they are uncontentious, yielding a faithful reconstruction of the notions of validity (of an argument) and satisfiability (of a set of sentences) via the usual model-theoretic definitions. In this way, we have defined a fragment of English—one corresponding, modulo translation, to the classical syllogistic of Aristotle’s Prior Analytics, Book A (Aristotle, 1938).
Generalising, say that a sentence template is a sentence of some natural language in which certain open-class words have been replaced by schematic variables; and say that a formula template of some logic is a formula (with no free variables) in which the same schematic variables are treated as elements of the non-logical signature, appropriately typed. Then a simple way to define a natural language fragment is by means of a finite list of sentence templates paired with corresponding formula templates giving their semantics in a way judged appropriate by competent speakers. We consider five English fragments in the sequel, all defined in this way: (i) the fragment S, a version of the syllogistic in which—for the sake of logical uniformity—we additionally allow negation at the subject (e.g. Some non-artist is not a beekeeper); (ii) the fragmentW , which adds relative clauses to the subjects of sentences in S (e.g. Every artist who is not a musician is a writer); (iii) the fragment V , which extends S with main clauses featuring transitive verbs and a quantifying determiner (e.g. Every carpenter admires some writer); (iv) the fragment Z , which extendsW with relative clauses featuring transitive transitive verbs (e.g. Every carpenter who admires some writer is an electrician); and (v) the fragment A, which extends Z with certain sentence templates involving bound-variable anaphora (e.g. Some artist hates no beekeeper who admires him). The precise sets of sentence templates defining these fragments are presented at the end of this paper. Since the relevant sentence templates correspond to sets of formula templates of firstorder logic, our five language fragments may, to all intents and purposes, be identified with the corresponding fragments of first-order logic; hence we alternate freely between sentences and their logical translations, as the context requires.
If L is a natural language fragment, we denote by Sat(L) the problem:
Given: a finite set Φ of sentences in L, Return: Y if Φ is satisfiable; N otherwise.
It can be shown that Sat(S) is NLOGSPACEcomplete and Sat(V) is EXPTIME-complete (PrattHartmann and Moss, 2009, Theorems 4.11 and 6.3). It is completely routine to show that Sat(W) is NPTIME-complete. With a little effort, one can show that Sat(Z) is also EXPTIME-complete, using essentially the same techniques as for Sat(V). Finally, Sat(A) is NEXPTIME-complete: membership in NEXPTIME follows from the fact that all the corresponding formula templates are in the two-variable fragment of first-order logic (Grädel et al., 1997); NEXPTIME-hardness follows by an almost identical argument to that given for the rather larger English fragment E2V in (Pratt-Hartmann, 2003). The proof, along with a detailed complexitytheoretic analysis of the language fragments discussed, are depicted in Appendix B. Given these results, it is natural to ask whether language models have greater success in learning to recognise valid entailments (or, dually: satisfiability of sets of sentences) for the computationally easier fragments, or whether training on easier fragments aids in recognising entailments in harder ones.

heading: 3.2 Identifying the phase change region
text: It is important to realise that the worst-case complexity bounds for satisfiability problems are not necessarily representative of randomly generated problem instances, a phenomenon that is well understood in the case of propositional logic. Suppose that k is a positive integer and that a k-clause is a disjunction of k or fewer proposition letters or negated proposition letters. The k-SAT problem is the following: given a finite set of clauses Γ, return 1 if Γ is satisfiable (i.e. if there is a truth-value assignment making all the clauses in Γ true), and 0 otherwise. This problem is NPTIME-complete for all k ≥ 3. However, under certain conditions, it
is trivial to decide random instances of these problems with high probability. Fixing k = 3, consider a randomly generated instance Γ of 3-SAT consisting of m clauses featuring n proposition letters p1, . . . , pn (and their negations). If the ratio mn is large, we have a highly constrained problem with few degrees of freedom, so the probability of satisfiability is close to zero; if, on the other hand, the ratio mn is small, the probability of satisfiability is close to unity. In either case, it is trivial for any algorithm to achieve high performance when predicting the correct answer. Only for values in a relatively narrow range of mn , commonly referred to as the phase-change region, is the problem challenging, with this range (centred on a value close to 4.17) narrowing as n increases (Selman et al., 1996; Mitchell and Levesque, 1996).
A similar phenomenon is observed for the satisfiability problems for natural language fragments studied here: the satisfiability of a given set of sentences may in many cases be easily determined with high probability by measuring the number of degrees of freedom in the given instance. Consider, for example, the problems Sat(S) and Sat(W). Any instance of these problems is characterised by the number n1 of common nouns featured, and the number m of sentences involved. We find that the probability of satisfiability for randomly generated instances is close to 0.5 only when the ratio α = mn1 is in a relatively narrow band. For the other fragments considered in this study, which feature transitive verbs as well as common nouns, we must consider not only the ratio α but also the ratio β = mn2 , where n2 is the number of transitive verbs in the given problem instance. Again, we find that the probability of satisfiability for a randomly generated problem is close to 0.5 only for pairs of values (α, β) lying in a relatively constrained region. We continue to refer to this region, informally, as the phase-change region: notice that this is a region in
2-dimensional space. It is important to remember that the shape and location of the phase-change region depend on the fragment considered; however, in all cases, the gradients involved are observed to become sharper with increasing n. In assessing the ability of TLMs to learn to solve the satisfiability problem for the fragments considered in this paper, we must make sure to construct data sets involving only challenging instances, namely, those selected from the phase-change region: prowess at solving cases chosen uniformly over parameter space is a poor test of a system’s grasp of logical principles. For definiteness, we take the phase-change region for a fragment L, denoted λL, to be the set of input parameters for which the probability of satisfiability is [0.35, 0.65]. This set can be determined empirically by random sampling, much as in the original studies of 3-SAT. The variation of probability of satisfiability with other factors is outlined in Appendix E.

heading: 3.3 Data Construction
text: For each of the language fragments L considered above, we constructed a data set with which to finetune TLMs. Each data set is a set of data points. Each data point is a pair consisting of a set of sentences Φ from L, and a label (Y or N) indicating whether Φ is satisfiable. To generate a single sentence of Φ, we select at random one of the sentence templates defining L and instantiate its schematic variables by uniform sampling from a collection of n1 common nouns and (for the applicable fragments) n2 transitive verbs; repeating this whole process m times then yields a set Φ of cardinality m. The label determining the satisfiability of Φ is determined by applying a theorem prover (in our case, Z3) to the corresponding set of formulae of first-order logic as given by the formula templates. We construct data points for various values ofm, n1 and (where appropriate) n2, taking care to employ only those combinations within the critical region, λL. As explained above, λL has been determined empirically.
For the fragments S andW , we sample values of n1 from the range [nmin1 , n max 1 ], where n min 1 = 6, and nmax1 = 16. For the fragments V , Z , andA, we sample values of n1 from the range [nmin1 , n max 1 ], where nmin1 = 3, n max 1 = 8, and we sample values of n2 from the range [nmin2 , n max 2 ], where n min 2 = 3 and nmax2 = 8. In each case, we generate a set of values of values m for which α = m/n1 and
(for the appropriate fragments) β = m/n2 lie in λL. The entire protocol for generating data sets is given in Algorithm 1. We use the theorem prover Z3 to determine the satisfiability of the generated set of formulae, Φ = {ϕ1, . . . , ϕm}. For each language fragment, we construct a training set with 120K data points, an evaluation set with 10K data points and a test set with 10K data points to finetune and evaluate TLMs. We note that the above experimental setup can be directly adapted to any language fragment.
Algorithm 1 Data Construction - Natural language satisfiability Input : Language Fragment L; phase-change region λL (for L); vocabulary of unary predicates U ; vocabulary of binary predicates V; range for number of unary predicates [nmin1 , n max 1 ]; range for number of binary predicates [nmin2 , n max 2 ]. Output : natural language satisfiability dataset D.
1: D ← {} 2: repeat 3: m,n1, n2 ← sample m,n1, n2 such that
nmin1 ≤ n1 ≤ nmax1 , nmin2 ≤ n2 ≤ nmax2 and (mn1 , m n2 ) ∈ λL
4: U∗,V∗ ← sample n1 unary predicates and n2 binary predicates from U and V respectively, |U∗| = n1 and |V∗| = n2 5: for i = 1 to m do 6: ti ← randomly sample template from language fragment L 7: si ← substitute predicates from U∗ and V∗ for schematic variables in ti 8: ϕi ← translate si to a first-order logic formula 9: end for
10: ℓ← SAT-solver({ϕ1, ..., ϕm}) 11: D ← D ∪ {⟨ℓ, {s1, ..., sm}⟩} 12: until stop condition is met
For evaluating TLMs in zero-shot settings we set nmin1 = 5, and n max 1 = 10 for fragments S andW and nmin1 = 3, n max 1 = 5, n min 2 = 2 and n max 2 = 5 for fragments V , Z , and A. We then construct an additional 1200 problem instances for each language fragment and formulate the prompt using the constructed problem instances employing a templatebased approach. The exact template is depicted in Appendix C. Further details regarding the data sets are given in Appendix D.

heading: 4 Experimental Setup

heading: 4.1 Transformer-based language models
text: To examine TLMs’ ability to solve hard instances of natural language satisfiability problems, and investigate their capability to learn rules of inference, we fine-tuned two well-known TLMs which have a proven track record of solving textual entailment problems1.
T5. Following the work done by Richardson and Sabharwal (2021) and Tafjord et al. (2021) on logical reasoning in natural language, we primarily centred our experiments around the text-to-text transformer or T5 (Raffel et al., 2019). The T5 model employs a unified text-to-text format where all inputs and outputs are textual strings, in contrast to BERT-styled models. We utilised the T5-large model, which has around 700M parameters.
DeBERTa-v3. Due to the recent success of the DeBERTa-v3 model (He et al., 2021) in solving natural language inference tasks, we decided to use it as a baseline model. The DeBERTa architecture improves upon the BERT and RoBERTa models using a disentangled attention mechanism and enhanced mask decoder, and version 3 further improves the architecture by utilising an ELECTRA-style pretraining with Gradient Disentangled Embedding Sharing. We employ the DeBERTa-v3-large model with around 304M parameters.
Each of the TLMs is fined-tuned by reducing the binary cross entropy loss over the target using Adam optimiser (Kingma and Ba, 2015) and we used the HuggingFace (Wolf et al., 2019) implementation when experimenting with the abovementioned TLMs. A detailed description of the fine-tuning process is described in Appendix D.
To investigate TLMs’ ability to solve satisfiability problems in zero-shot settings, we employed three well-known models.
GPT. Due to the recent success of ChatGPT and GPT-4 solving many natural language processing tasks in zero-shot settings, we employed them in a similar context (Bang et al., 2023; OpenAI, 2023). Both ChatGPT and GPT-4 models are trained to follow human instructions utilising Reinforcement Learning from Human Feedback (RLHF).
1Link to the Dataset and Code: https://github. com/iTharindu/nl-sat
LLaMa. To better compare the effect of model size and architecture, we also utilise the LLaMA-2-chat-70B model in zero-shot settings. LLaMa-2 achieves comparable performance with state-of-the-art language models such as ChatGPT and PALM (Touvron et al., 2023). Similar to ChatGPT and GPT-4, the LLaMa-2-chat model also employs RLHF.

heading: 4.2 Proposed Dataset and Evaluation
text: To evaluate the ability of TLMs to solve natural language satisfiability problem instances, we designed several experiments. Firstly, to answer the questions, “Can TLMs solve hard natural language satisfiability problems?” and “How does computational complexity affect TLMs’ ability to perform a logical reasoning task?”, we trained and evaluated the TLMs mentioned in Sec. 4.1 against each of the language fragments introduced in Sec. 3.1 (see Table 1). Secondly, to answer the question “Do the computationally simpler language fragments help TLMs learn rules inference in complex ones?”, we trained the same TLMs using a dataset that comprises problem instances from all language fragments (see Table 2). We employed two variants of this joint dataset, one with 600K data points with 120K data points from each fragment and the other with 120K data points with 24K data points from each fragment. Thirdly, to answer the question “Do TLMs show the ability to generalise and show scale-invariance properties?”, we evaluated the same TLMs against a dataset containing more predicates (variables) than that of the training set (see Table 3). This provided clarity into TLMs’ ability to learn rules of inference from natural language satisfiability problem instances. Finally, to answer the question, "What factors affect TLMs’ ability to solve Satisfiability problem instances?", we evaluated large-scale transformerbased language models in zero-shot settings. We employ a much smaller dataset in this evaluation and the dataset description is detailed in Appendix D. The datasets we constructed were balanced with an equivalent number of satisfiable and unsatisfiable instances, and, consequently, we used accuracy as the evaluation metric.

heading: 5 Results and Discussion
text: The ability of TLMs to solve natural language satisfiability problems is affected by the underlying computational complexity of the problem, as
determined by the language fragment in question. As shown in Table 1, the performance of TLMs considered declines as the computational complexity class increases. It also can be observed that descent is steeper from S to W and more gradual henceforward. We posit two reasons for this phenomenon. Firstly, the problems that are in NLOGSPACE can be solved by the detection of relatively simple configurations (forbidden configuration of relatively low length in case the sentences are unsatisfiable) that TLMs can easily learn to recognise, while those that are NPTIMEhard (or harder) are characterised by more intricate structures. Secondly, the problems in the simpler fragments have fewer tokens in their input sentences compared to those in the more complex language fragments; and TLMs (like other neural approaches) find it more challenging to process long dependencies than short ones (Chen et al., 2020; Beltagy et al., 2020). However, computational complexity is not the only factor that influences the behaviour of the TLMs when performing logical inference tasks, as evidenced by the difference in performance when predicting the satisfiability of problem instances in V and Z , both of which are EXPTIME-complete. The two fragments contain the same language properties, such as relative clauses and transitive verbs, but Z has sentences that retain both of those properties together while V does not. Thus, some sentences in Z are intrinsically more linguistically complicated than those in V . As the neural approaches we employed are pretrained language models that were trained on large language corpora, the linguistic complexity of the input has a noticeable effect on their performance.
Provided adequate data, learning to solve satisfiability problem instances of simpler fragments can help TLMs learn to solve that of complex ones. When the dataset contains problem instances from all fragments, as shown in Table
2, the accuracy value for complex fragments increases when trained with an adequate number of data points (accuracy values for 600K). We hypothesise two reasons for this; first, fragments whose computational complexity for solving satisfiability problems is high have sentences from the fragments such as S , which are much simpler; second, even in complex language fragments such as A, the inference is heavily governed by the sentences belonging to simpler fragments (such as involved in proofs when the set of sentences is unsatisfiable). Thus, having problem instances belonging only to simpler fragments helps TLMs decode reasoning patterns caused by sentences belonging to them. However, this raised a concern as to the true difficulty of the underlying dataset, even when sampled from the phase-change region. When the number of data points is low, say 120K data points, the performance of the TLMs decreases for all fragments. We consider 24K data points from a single language fragment inadequate to learn any meaningful representation required for determining satisfiability. This is somewhat dissimilar to the results yielded by the computationally simpler rule reasoning problem of model-checking with natural language (Madusanka et al., 2023a).
TLMs failed to generalise and learn rules of inference required to solve satisfiability problem instances. We investigated whether TLMs learn to apply rules of inference and comprehend the underlying algorithm associated with the satisfiabilitysolving task by evaluating them against a dataset containing more predicates (thus more clauses) than that of the training test. As depicted in Table 3, TLMs failed to generalise, and accuracy across all fragments decreased regardless of the computational complexity, suggesting TLMs do not systematically generalise. This hypothesis is reinforced by the evident failure of TLMs to generalise even for fragment S, for which they exhibited a good accuracy level. Prior literature on the systematic generalisation of neural approaches (Lake and Baroni, 2017; Goodwin et al., 2020) also corroborates our finding that neural networks generally do not generalise well outside the training distribution. Moreover, experiments conducted by Richardson and Sabharwal (2021) on scale invariance yielded equivalent results.
Pre-trained transformer-based language models do not achieve adequate performance when solving even the simplest satisfiability problems in zero-shot settings, and factors such as model architecture, number of variables and exact prompt can affect models’ ability to follow instructions. The GPT-4 model achieved the best performance among the models we consider and is the only model whose results are reasonably higher than random guessing. As depicted in Figure 3, notably, the GPT-4 model outperformed ChatGPT by a considerable margin for all language fragments, which is in line with prior evaluations on algorithmic tasks such as coding and answering Mathematics questions (OpenAI, 2023). Another notable observation is that the effect of the computational complexity of the language fragments when solving natural language satisfiability problem instances is considerably lower compared to fine-tuned models.
A deeper analysis of the answers generated also shows that GPT-4 is better at explicitly following the instructions. For example, if the prompt indicates only to generate “satisfiable” or “unsatisfiable”, and asks to generate nothing else, the GPT-4 model follows that instruction explicitly more often than the other models. Conversely, our experimentation with LLaMa-2 reveals a contrary trend; for most problem instances, the models do not follow
the instructions and often generate something other than “satisfiable” or “unsatisfiable”.
When we introduced a modification to the prompt by incorporating an illustrative example and requesting the generation of “True" or “False" instead of “satisfiable” or “unsatisfiable”, the model consistently produced responses of True or False. We hypothesise the model’s failure to adhere to the instructions of the original prompt can be attributed to two factors. Firstly, the terms “satisfiable” and “unsatisfiable” are relatively infrequent in the text data the language model has been trained on. Consequently, from a probabilistic standpoint, the likelihood of these specific terms being the most probable next tokens is low. Therefore, the model often generates something other than “satisfiable” or “unsatisfiable”. Secondly, the inclusion of an illustrative example within the modified prompt provides the model with an unambiguous context regarding the desired output. However, even in this scenario, the model’s performance is equivalent to random guessing.
Unsurprisingly, when the number of variables increases the accuracy of GPT models decreases. We posit two reasons for this observed phenomenon, Firstly, from a logical perspective, the increment in the number of variables increases the difficulty of solving the satisfiability problem. As the number of variables grows, the search space expands, rendering the task of finding a valid assignment more challenging. Secondly, the increment of the number of variables increases the total number of
tokens within a given problem instance. This increase in token count compels the model to grapple with longer-term dependencies in the input. This gradual decrement along with the recurring failure of fine-tuned TLMs to generalise, leads to the hypothesis that, notwithstanding their impressive performance, state-of-the-art transformer models are still far from learning the rules of inference underlying logical reasoning tasks, and the algorithms required to apply them.

heading: 6 Conclusion
text: We have investigated transformer-based language models’ ability to solve instances of natural language problems belonging to different language fragments with varying computational complexity. Our investigation demonstrated that the computational complexity of the fragment has a noticeable effect on TLMs’ ability to perform a logical inferential task. Even in a simpler language fragment for which the transformer models achieved a reasonable performance, the model failed to adjust to distribution shift and generalise beyond its training distribution. Thus, we posit that TLMs do not reliably disentangle the patterns that exist within the dataset and the rules of inference needed when determining satisfiability. Therefore, it is imperative to acknowledge that a considerable body of research remains to be conducted in order to enhance the capacity of these models to comprehend rules of inference. We also acknowledge that the generalisation aspect of TLMs may need a deeper analysis, which we leave for future work.
Limitations
The empirical study presented in this paper exhibits two principal limitations. We explored several TLMs with different architectures that are in line with prior work and studied how computational complexity and other linguistic factors affect those models when learning an inferential task. However, due to the empirical nature of the research, there could exist a language model architecture whose behaviour deviates from the claims made in this paper. Similarly, the language fragments employed in this literature are not the only fragments in existence and, as such, would affect the comprehensiveness of the discussion.
Moreover, when constructing the dataset, we sampled problem instances from the phase-change region, and it was observed that the theorem prover
took significantly more time to determine the satisfiability when data points were sampled from the phase-change region as opposed to random sampling. However, it does not provide a guarantee as to whether all instances are non-trivial, and it is probable that some problem instances are still trivial to solve.

heading: Acknowledgements
text: Ian Pratt-Hartmann was supported by the Polish NCN, grant 2018/31/B/ST6/03662. The authors would like to acknowledge the use of the Computational Shared Facility at The University of Manchester and the Department of Computer Science at the University for funding this research. We also thank the anonymous reviewers from the ARR December 2023 cycle for their valuable feedback.

heading: A Appendix: Definition of language fragments
text: The sentence templates defining S, together with the corresponding formula templates, are as follows: Every (non-) p is a q No (non-) p is a q Some (non-) p is (not) a q.
We can define the first-order formulae for giving semantics for the above sentence templates as follows:
∀x(±p(x)→ ±q(x)) ∃x(±p(x) ∧ ±q(x)),
where ±ψ is either ψ or ¬ψ. The sentence templates for W are those of S together with
Every (non-) o who is (not) a p is a q No (non-) o who is (not) a p is a q Some (non-) o who is (not) a p is (not) a q.
We can define the first-order formulae for giving semantics for the above sentence templates as follows:
∀x((±o(x) ∧ ±p(x))→ ±q(x)) ∃x(±o(x)± p(x) ∧ ±q(x)).
The sentence templates for V are those of W together with
Some/every (non-) p r’s some/every (non-) q No (non-) p r’s any/every (non-) q Some (non-) p does not r any/every (non-) q.
We can define the first-order formulae for giving semantics for the above sentence templates as follows,
∀x(±p(x)→ ∀y(±q(y)→ ±r(x, y))) ∀x(±p(x)→ ∃y(±q(y) ∧ ±r(x, y))) ∃x(±p(x) ∧ ∀y(±q(y)→ ±r(x, y))) ∃x(±p(x) ∧ ∃y(±q(y) ∧ ±r(x, y))).
The sentence templates for Z are those of V together with
Some (non-) o who r’s/does not r some/every (non-) p is (not) a q Every (non-) o who r’s/does not r some/every (non-) p is a q No (non-) o who r’s/does not r any/every (non-) p is a q.
We can define the first-order formulae for giving semantics for the above sentence templates as follows: ∀x(±o(x) ∧ ∀y(±p(y)→ ±r(x, y))→ ±q(x)) ∀x(±o(x) ∧ ∀y(±p(y) ∧ ±r(x, y))→ ±q(x)) ∃x(±o(x) ∧ ∀y(±p(y)→ ±r(x, y)) ∧ ±q(x)) ∃x(±o(x) ∧ ∀y(±p(y) ∧ ±r(x, y)) ∧ ±q(x)).
The sentence templates for A are those of Z together with
Some/every (non-) o r’s some/every (non-) p who s’s/does not s him No (non-) o r’s any/every (non-) p who s’s/does not s him Some (non-) o does not r any/every (non-) p who s’s/does not s him.
We can define the first-order formulae for giving semantics for the above sentence templates as follows:
∀x(±o(x)→ ∀y(±p(y) ∧ ±s(y, x)→ ±r(x, y))) ∀x(±o(x)→ ∃y(±p(y) ∧ ±s(y, x) ∧ ±r(x, y))) ∃x(±o(x) ∧ ∀y(±p(y) ∧ ±s(y, x)→ ±r(x, y))) ∃x(±o(x) ∧ ∀y(±p(y) ∧ ±s(y, x) ∧ ±r(x, y))).

heading: B Appendix: Computational Complexity
text: of Sat(S), Sat(W), Sat(V), Sat(Z) and Sat(A)
In this section, we justify the complexity-theoretic information provided in section 3.1. The case of S requires no work: this is the fragment denoted S† in (Pratt-Hartmann and Moss, 2009), where the complexity of satisfiability is established in Theorem 4.10, p. 668. For W , the upper bound of NPTIME follows easily from the fact that all the relevant formulae are in the 1-variable fragment of first-order logic; the lower bound follows by a straightforward reduction from 3-SAT, as given, for example, for the smaller fragment denoted E1 in (Pratt-Hartmann, 2004, Theorem 2, p. 213). The fragment V is slightly larger than the fragment denoted E2 in (Pratt-Hartmann, 2004), whose satisfiability problem is shown to be EXPTIME-complete in Theorem 3, p.215. This settles the lower bound for Sat(V). For the upper bound, we note that the proof depends essentially on the fact that all of the first-order formulae involved contain just one occurrence of any binary predicate; therefore, the same proof applies to the fragment V almost without change of wording. Furthermore, the same argumentation applies to Z . The only remaining case to consider is A, which is dealt with by the following theorem.
Theorem 1 The computational complexity of the problem of determining satisfiability in a set of sentences in A is NEXPTIME complete
Proof: A moment’s thought shows that every sentence of A is expressed by a formula of the two-variable fragment of first-order logic, FO2, whose satisfiability problem is NEXPTIMEcomplete (Grädel et al., 1997). Thus, Sat(A) is in NEXPTIME. To establish a matching lower bound we take an existing proof of the NEXPTIMEhardness of Sat(FO2), and show that it can be reproduced using only the resources of A. The proof in question proceeds by reduction from a known NEXPTIME-hard problem P (in this case, P is the problem of determining whether an exponentialsized grid can be tiled in a certain tiling system), and encoding any instance of P as a set Φ of FO2-
formulae such that Φ is satisfiable if and only if the given instance of P is positive. It can be shown that such encodings (see for example Pratt-Hartmann 2023, pp. 90 ff.) require only that we can write FO2-formulae of the following forms:
∃x(p1(x) ∧ · · · ∧ ±pn(x)) (1) ∀x(±p1(x) ∨ · · · ∨ ±pn(x)) (2)
∀x∀y( n∧
i=1
(pi(x)↔ qi(y))→ r(x, y)) (3)
∀x(±p(x)→ ∃y(±q(y) ∧ ±r(x, y))) (4) ∀x(±p(x)→ ∀y(±q(y)→ ±r(x, y))). (5)
Thus, if we can reproduce the effect of such formulae inA, then we will have established that Sat(A) is NEXPTIME-hard.
Form (1) can be simulated by a collection of sentences of the forms “Some p∗ is a p∗”, “Every p∗ is a p1”, . . . “Every p∗ is a pn”, where p∗ is a fresh unary predicate/common noun. Form (2) is handled similarly, remembering that noun-level negation is available. Any formula of Form (3) can be simulated by the formulae ∀x∀y(rn(x, y) → r(x, y)) and ∀x∀y r0(x, y) together with the collection of formulae ∀x(±1pi(x)→ ∀y(±1qi(y) ∧ ri−1(x, y) → ri(x, y)) (1 ≤ i ≤ n), where the two occurrences of ±1 are resolved in the same way, and the r0, . . . , rn are new binary predicates/transitive verbs. Of these, the formula ∀x∀y r0(x, y) is equivalent to the conjunction of the four formulae ∀x(±q(x) → ∀y(±q(x) → r0(x, y)), which can be written directly inA, while the remaining formulae can be written easily in A. Forms (4) and (5) can be written directly in A.
Thus, any instance of an exponential tiling problem can be transformed, in polynomial time, to an instance of Sat(A) with the same answer. This establishes the NEXPTIME-hardness of Sat(A).

heading: C Appendix: Zero-shot Prompting Templates
text: Given a set of sentences s1, s2, ..., sm and a label ℓ indicating whether the set of sentences is satisfiable or not, we formulate the prompt using the following template.
Q: Given the following set of sentences, tell me whether they are satisfiable or not. Generate satisfiable if they are and unsatisfiable if they are not. Set of sentences: s1, s2, ..., sm
A:
As we mentioned in the Results and Discussion section, the LLaMa model did not generate “satisfiable” or “unsatisfiable” for the above prompt for most problem instances. Therefore, we modified the prompt by following an example problem instance and asking it to generate the words “True” or “False” instead. The example problem instance consists of a set of sentences s1, s2, ..., sm and a satisfiability label be ℓ (where ℓ is “True” if s1, s2, ..., sm is satisfiable and “False” otherwise). The resultant prompt is as follows:
Q: Given the following set of sentences, tell me whether they are satisfiable or not. Generate True if they are and False if they are not.
Set of sentences: s1, s2, ..., sm A: ℓ Given the following set of sentences, tell me whether they are satisfiable or not. Generate True if they are and False if they are not.
Set of sentences: s1, s2, ..., sm A:

heading: D Appendix: Dataset and Training Details
text: D.1 Dataset details We utilised five different language fragments, namely, Syllogistic S, relative clauses W , relative clauses with relational syllogistic V , relative clauses with transitive verbs Z and anaphora A. For each language fragment, we constructed a training set with 120K, an eval set with 10K and a test set with 10K data points. When constructing the dataset, we set the number of unary predicates n1 to be between 3 and 8 and the number of binary predicates n2 to be between 3 and 8 for fragments V , Z and A while setting the n1 value to be between 6 and 16 for fragments S and W . It is apparent that the distribution of the number of ways specific numbers between 6 and 16 can be formed by randomly adding two numbers in the range 3 and 8 formed a bell curve. Hence, the distribution of the number of problem instances per number of variables of V , Z and A seems to be a discrete approximation of normal distribution. Therefore, we sample the n1 values from a normal distribution for S andW while sampling n1, n2 values uniformly for the other fragments.
When testing for TLMs’ ability to generalise, we construct a separate test set with more predicates than that of the training set. We construct 6K data points for each fragment and the dataset
is balanced having an equal number of satisfiable and unsatisfiable problem instances. In this setup, we set the range of the n1 value in the test set between 16 and 24 for fragments S, and W , while varying it between 8 and 12 for other fragments. The range of the number of binary predicates n2 for fragments V , Z and A was set to be between 8 and 24 due to the asymmetric nature of the variation of probability of satisfiability with mn1 compared to m n2
. It is important to emphasise that all data points belonging to each of the datasets (train, eval and test) are unique.
To evaluate TLMs in zero-shot settings, we construct a separate test set with 1200 data points for each fragment, with 600 data points being satisfiable while the other data points being unsatisfiable. In this setup, we set nmin1 = 5, and n max 1 = 10 for fragments S andW and nmin1 = 3, nmax1 = 5, nmin2 = 2 and nmax2 = 5 for fragments V , Z , and A. . Moreover, for each value n where n = n1 + n2, the dataset contains 200 data points, with 100 of them being satisfiable and the other 100 being unsatisfiable.
When defining the vocabulary, we expanded the vocabulary introduced by Richardson and Sabharwal (2021). The vocabulary of unary predicates U comprises 156 nouns (of people’s professions) and the vocabulary of binary predicates V comprises 70 transitive verbs.
The mean, maximum, and minimum number of clauses for each of the language fragments are depicted in Table 4, while mean, maximum, and minimum instance lengths are illustrated in Table 5.
D.2 Training Details We chose two TLMs with somewhat different architectures; T5 and DeBERTa-v3, as both models have
proven track records on textual entailment tasks. Since the research question of interest relies on the learnability of TLMs, not on identifying the bestperforming model, we do not perform any hyperparameter tuning. Since according to prior literature (Raffel et al., 2019), the performance of the TLMs stems from model size and pre-trained data more so than the architectural choice, as well as, since the accuracy values yielded for the TLMs are similar, we anticipated similar performance for other TLMs. Moreover, exploring different transformer models and hyper-parameter tuning would leave a higher carbon footprint (Strubell et al., 2019), which is deemed unnecessary considering the nature of the research questions.
A detailed description of the hyperparameters is as follows,
Maximum sequence length : When training the DeBERTa-v3 model, we used the maximum sequence length between 512 and 700 tokens, and when training the T5 model using the joint dataset, we used 700 tokens as the maximum sequence length. Since we used the T5 models trained on individual fragments for testing for generalisation, we used a maximum sequence length of 1024 tokens. We did not rely on any truncation, as truncating input could alter the satisfiability of the input sentences.
Training epochs : We used five training epochs and used the eval dataset to identify the bestperforming model.
Batch size : Relying on the gradient checkpointing, we used a batch size of 24 for the DeBERTav3-large model. Similarly, when trained with the joint dataset, we used a batch size of 24, again relying on gradient checkpointing. Since we used the
T5 model trained on the individual dataset for generalisation experiments, we downgraded the batch size to 12 to prevent memory errors.
Each of the TLM is fined-tuned to predict the label by reducing the binary cross entropy loss over the target using Adam optimiser (Kingma and Ba, 2015) and we used the HuggingFace (Wolf et al., 2019) implementation when experimenting with the above-mentioned TLM.

heading: E Appendix: Phase Change Region of Language Fragments
text: We explore the problem distribution of the language fragments we considered along several analytical viewpoints. The quantifier ratio is used to define the ratio of the number of ∃ quantifiers to the number of ∀ quantifiers, and the subject quantifier ratio is the quantifier ratio between the quantifiers associated with the subject while the object quantifier ratio is that associated with the object. Since sentences belonging to the fragments S andW contain only one quantifier, which is associated with the subject, the object quantifier ratio is omitted for those fragments. The visualisation of variation in the probability of satisfiability is depicted in figures 4, 5, 6, 7 and 8. Figure 4 illustrates how the probability of satisfiability varies with mn1 and m n2
, which we have used when defining the phase change region.

heading: Future Work
text: We also acknowledge that the generalisation aspect of TLMs may need a deeper analysis, which we leave for future work. Limitations
The empirical study presented in this paper exhibits two principal limitations. We explored several TLMs with different architectures that are in line with prior work and studied how computational complexity and other linguistic factors affect those models when learning an inferential task. However, due to the empirical nature of the research, there could exist a language model architecture whose behaviour deviates from the claims made in this paper