Logically Consistent Language Models via Neuro-Symbolic Integration

## Abstract

Abstract Large language models (LLMs) are a promising venue for natural language understanding and generation.
However, current LLMs are far from reliable: they are prone to generating non-factual information and,
more crucially, to contradicting themselves when prompted to reason about relations between entities of the world.
These problems are currently addressed with large scale fine-tuning or by delegating reasoning to external tools.
In this work, we strive for a middle ground and introduce a loss based on neuro-symbolic reasoning that teaches an LLM to be logically consistent with an external set of facts and rules and improves self-consistency even when the LLM is fine-tuned on a limited set of facts.
Our approach also allows to easily combine multiple logical constraints at once in a principled way, delivering LLMs that are more consistent w.r.t. all constraints and improve over several baselines w.r.t. a given constraint.
Moreover, our method allows LLMs to extrapolate to unseen but semantically similar factual knowledge, represented in unseen datasets, more systematically.

## 1 Introduction

Developing reliable large language models (LLMs) and safely deploying them is more and more crucial, particularly when they are used as external sources of knowledge [, , , ].
To do so, one would need LLMs to be factual [], i.e., agreeing on single facts that appear in a knowledge base (KB), and logically consistent [, ], i.e., being able not to contradict themselves or a KB when prompted to perform complex reasoning.
It has been abundantly shown that training on large datasets for question answering (QA) [] alone cannot meet these desiderata [, , , ].

Factuality and consistency are intimately related.
Enforcing factuality alone generally boils down to fine-tuning an LLM on a large KB of atomic facts [].
When predicting the truth values of these facts, a number of works try to enforce the simplest form of consistency: that the probability of a true fact shall be one minus the probability of its negation [].
More sophisticated heuristics are possible, e.g., fine-tuning on a large QA dataset by jointly optimizing for truthfulness of model answers and contrastively pulling apart true and false facts [].
All these approaches require large KBs and more crucially are tailored towards specific logical constraints.

When it comes to self-consistency w.r.t. more complex reasoning scenarios, e.g., ensuring that LLMs can perform modus ponens without contradicting themselves [, ], one line of research focuses on employing external reasoning tools such as MAX-SAT solvers [] at inference time [, , ].
However, these approaches depend on the constant availability of a reasoner (and sometimes also of a natural language inference model []) which can increase the cost of inference for every reasoning step.
At the same time, training the LLM to reason is not possible or hindered by the hardness of backpropagating through the solver [].

In this work, we show how to improve factuality and self-consistency of LLMs without external components by leveraging recent advancements in neuro-symbolic learning [].
This is done by turning
complex reasoning tasks into logical constraints that can be incorporated via neuro-symbolic (NeSy) reasoning [, ].
Specifically, we fine-tune an LLM by a principled objective: maximising the probability of a constraint to hold, which goes under the name of weighted model counting [] in probabilistic reasoning or semantic loss [] when used as a regularizer in deep learning [, ].
This in turn encourages the LLM to perform principled probabilistic reasoning at training time by maximising the probability of beliefs that comply with the provided set of constraints.

We empirically show how
given incomplete factual knowledge, e.g., by providing only a limited number of known facts, the LLM can learn truth beliefs for new facts while keeping logical consistency w.r.t. prior knowledge.
Moreover, our approach is agnostic to the logical constraints considered and can deliver a single training objective that can improve multiple consistency scores at once.
In our experiments, with a single offline training session, LLMs trained with our objective outperform models relying on external solvers, and are more factual and logically consistent in low-data regimes when compared to standard supervised fine-tuning over KBs of facts.

Contributions. Summarizing, we:
i) introduce
Logically-Consistent LLMs (LoCo-LMs), a novel and principled
fine-tuning strategy designed to improve factuality and (self-)consistency of LLMs based on probabilistic NeSy reasoning (Section 3), and
ii) we rigorously evaluate the ability of LoCo-LMs to improve self-consistency w.r.t. several reasoning scenarios – when fine-tuned for certain constraints and evaluated over others – without hurting fluency (Section 5).

Figure: Figure 1: Our Logically Consistent (LoCo) LLMs can be fine-tuned in a unified way to be more factual and consistent to several different forms of logical constraints such as direct (left), reverse (middle) implications, negation and combinations thereof (Section 3) when compared to a pre-trained LLaMa 2 70B or fine-tuned baseline such as LLaMa 2 7B. viva la semantic loss antifascista
Refer to caption: x1.png

## 2 Logical consistency through the lenses of probabilistic reasoning

We formalize the different reasoning scenarios we would like an LLM to be (self-)consistent with, and highlight the shortcomings of commonly used LLMs
when prompted to reason in this way.

##### Factuality.

We view a pre-trained LLM as a collection of truth beliefs about facts over which it can reason.
The simplest reasoning task is factual reasoning, i.e., determining the veridicity of a fact.
For example, consider the fact $f$ in textual form “an albatross is a bird”.
It can be commonly encoded in knowledge bases (KBs) such as BeliefBank [] as a (subject-relation, property) pair, for instance,
(albatross-is, bird).
To inspect whether an LLM believes a fact to be true, we can prompt it with a question like “Is an albatross a bird?”, the LLM can supply a binary prediction of the form “Yes”/“No” or “True”/“False”,(^1^11 We note that such an answer can be highly dependent on the format of the prompt. For this reason, in our experiments we use several prompts, whose format is detailed in Section 5.) encoding its belief that the fact $f$ holds or not.
Therefore, given an LLM modeling a parameterized distribution $p_{\theta}$, we can consider the probability of generating a token $x_{t}$ encoding a binary answer, according to $p_{\theta}$, after observing the token sequence $x_{1},\ldots,x_{t-1}$ encoding the question about the fact, to be the probability of the LLM believing that the truth value $z_{f}$ of fact $f$ is either true ($\top$) or false ($\bot$). That is, for true facts,

$$ $p_{\theta}(z_{f}=\top)=p_{\theta}(x_{t}=\ell_{\mathsf{true}}\mid x_{1},\ldots, x_{t-1}=\text{``Is an albatross a bird?"})$ (1) $$

where $\ell_{\mathsf{true}}$ is an affirmative token, e.g., one among “yes”, “true”, “Y”, “T”, etc.
Analogously, we can compute $p_{\theta}(z_{f}=\bot)$ by checking if the LLM answers a token $\ell_{\mathsf{false}}$ is “no”, “false”, “N”, “F”, etc.
To determine the model’s belief, we query (^2^22We keep a default temperature $t=1.0$. Dropout is disabled to generate outputs systematically.) the most likely next token $\hat{x}_{t}$ and check whether it falls in $\ell_{\mathsf{true}}$ or $\ell_{\mathsf{false}}$, and set it to “undetermined” if it falls into neither.

Given an external KB, we say an LLM is factually consistent, or simply factual, w.r.t. a fact $f$ in the KB with truth value $z_{f}^{*}$, if its answer (mapped to a truth assignment
as described above) matches $z_{f}^{*}$, and factually inconsistent otherwise.(^3^33Similarly, one could say that an LLM is factually self-consistent w.r.t. $f$ if it answers in the same logically consistent way (e.g., $z_{f}$ is always $\top$) when asked to answer the same prompt or different, but semantically equivalent, prompts several times. Since this is harder to measure – as it strongly depends on the sampling strategy – in this work we focus on factual consistency only.)
This perspective leads to interpreting factual reasoning as a binary question answering (QA) task [, , ].
From Equation 1, one can see that a simple strategy to make an LLM more factual
is that of minimizing the cross-entropy (XENT) of $p_{\theta}$ over an external KB containing training questions with ground truth answers.
We compare against it in our experiments (Section 5).

##### Negation consistency.

While effective for many QA scenarios [, ], increasing factual consistency by XENT minimization does not prevent the LLM from being logically inconsistent under other simple constraints, e.g., contradiction [, , ].
Given a textual representation for a fact $f$, e.g., “an albatross is a bird”, and another one $\widetilde{f}$ encoding its negation, e.g., “an albatross is not a bird”, we say negation self-consistency holds if

$$ $z_{f}\oplus z_{\widetilde{f}}\iff(z_{f}\wedge\neg z_{\widetilde{f}})\vee(\neg z _{f}\wedge z_{\widetilde{f}}),$ ( Neg ) $$

where $\oplus$ denotes the logical operator XOR.
In other words, we would like an LLM to consistently answer either affirmatively or negatively when asked about the truth of a statement and its negation.
Negation consistency is very challenging for LLMs [, , ].
For example, in our experiments LLaMa-2 70b [] answers “true” to both questions “Is an albatross an organism?” and “Is an albatross not an organism?”.
From a probabilistic perspective, a simple sufficient condition for negation consistency is that $p_{\theta}(z_{f}=\top)=1-p_{\theta}(z_{\widetilde{f}}=\top)$. This is hard to be systematically guaranteed and in practice has been addressed by applying ad-hoc heuristics during fine-tuning [], which however cannot be exploited to enforce consistency to other constraints, such as implication, discussed next.

##### Implication consistency.

Given two textual representations of facts $f_{1}$ (antecedent, e.g., “an albatross is a bird”) and $f_{2}$ (consequent, “an albatross is an animal”) we say that the first implies the second if it holds that

$$ $(z_{f_{1}}\rightarrow z_{f_{2}})\iff\;(\lnot z_{f_{1}}\lor z_{f_{2}}).$ ( Imp ) $$

As with factuality, consistency (resp. self-consistency) holds if the answers of the LLM to a prompt satisfy the truth values according with the above implication and an external KB (resp. the inner beliefs of the LLM).
Furthermore, letting $z_{f_{1}}^{*}$ be the truth value of ${f_{1}}$ recorded in the KB, we can define a factual variant of the implication that restricts the constraint to take $z_{f_{1}}^{*}$ into account, that is, when the LLM is prompted about $f_{2}$, it should derive its truth value $z_{f_{2}}$ according to

$$ $(z_{f_{1}}=z_{f_{1}}^{*})\wedge(z_{f_{1}}\rightarrow z_{f_{2}})$ ( F-Imp ) $$

This can be seen as a relaxation of classical modus ponens reasoning [].
While simpler to capture from text corpora, implication consistency can still be challenging for LLMs [, ].
For example, given the rule $f_{1}\rightarrow\lnot f_{2}$, where $f_{1}$: “an albatross is an animal” and $f_{2}$: “an albatross is a virus”, we wish the LLM to answer with “Yes” and “No” respectively, which maps to the truth assignment $z_{f_{1}}=\top$, $z_{f_{2}}=\bot$. LLaMa-2 70b violates the provided rule with the inconsistent belief, $z_{f_{2}}=\bot$, i.e. “an albatross is a virus” is labeled as “Yes”.

##### Reverse implication consistency.

Equation Imp is logically equivalent to $\neg z_{f_{2}}\rightarrow\neg z_{f_{1}}$, nevertheless an LLM that is logically consistent w.r.t. the implication of $f_{1}$ over $f_{2}$ might not necessarily be consistent w.r.t. the implication of $\widetilde{f_{2}}$ over $\widetilde{f_{1}}$, representing the negation of $f_{2}$ and $f_{1}$ respectively.
For example, while LLaMa-2 70b is logically consistent w.r.t. $z_{f_{1}}\rightarrow z_{f_{2}}$ with $f_{1}:$ “an albatross is an organism”, $f_{2}:$ “an albatross is a living thing”, it violates $z_{\widetilde{f_{2}}}\rightarrow z_{\widetilde{f_{1}}}$ as it classifies $z_{\widetilde{f_{2}}}$: “an albatross is not a living thing” as false but $z_{\widetilde{f_{1}}}$: “an albatross is not an organism” as true.
Furthermore, an LLM that is logically consistent w.r.t. reverse implication and factual w.r.t. a KB should be able to satisfy

$$ $(z_{\widetilde{f_{2}}}=\neg z_{f_{2}}^{*})\wedge(z_{\widetilde{f_{2}}} \rightarrow z_{\widetilde{f_{1}}})$ ( Rev-F-Imp ) $$

where $\neg z_{f_{2}}^{*}$ indicates the opposite of the truth value stored in the KB for $f_{2}$.
This factual reverse implication scenario can be thought as a relaxation of modus tollens [].

##### More complex constraints.

As just discussed, constraints such as negation, logical implication and reverse implication already pose challenges to state-of-the-art LLMs in terms of consistency.
While we will focus on the Llama 2 LLM family in this work, similar shortcomings have been highlighted for even larger models such as ChatGPT and GPT-4 [].
Nevertheless, they constitute only a small fraction of the possible real-world reasoning scenarios LLMs can be asked to deal with.
Consider for example the following textual representations of facts, as extracted from EntailmentBank []:
$f_{1}:$ “melting is a kind of phase change”,
$f_{2}:$ “the ice melts”,
$f_{3}:$ “the ice undergoes a phase change”,
$f_{4}:$ “phase changes do not change mass”,
$f_{5}:$ “the mass of the ice will not change”.
They
obey the following logical constraint

$$ $(z_{f_{1}}\land z_{f_{2}}\rightarrow z_{f_{3}})\land z_{f_{4}}\rightarrow z_{f _{5}}.$ (2) $$

In the next section, we will introduce our general framework that can improve logical consistency of fine-tunable LLMs w.r.t.
any logical constraint expressible in propositional logic.

## 3 Logically-consistent LLMs via NeSy integration

We assume we are given a KB comprising a limited set of textual statements and associated truth values $\mathcal{D}_{F}=\{(f_{1},z_{f_{1}}^{*})\ldots,(f_{n},z_{f_{n}}^{*})\}$, encoding simple facts such as “an albatross is a bird” (true) and “a computer is a bird” (false),
and a set of logical constraints $\mathcal{D}_{C}=\{\alpha_{1},\ldots,\alpha_{m}\}$ defined over facts in $\mathcal{D}_{F}$, comprising implications, negations or more complex constraints as defined in Section 2.

Given a pre-trained LLM encoding a distribution $p_{\theta}$ over tokens, our objective is to fine-tune it
to be more consistent w.r.t. $\mathcal{D}_{F}$, $\mathcal{D}_{C}$ and itself.
As an important side benefit, we expect the fine-tuned LLM to generalize to – and be consistent with – the truth values of unseen facts $f_{n+1},f_{n+2},\ldots,$ that can be either logically inferred by applying the constraints in $\mathcal{D}_{C}$ to $\mathcal{D}_{F}$ (e.g., by applying modus ponens) or
that are semantically similar to facts in $\mathcal{D}_{F}$.
For example, since albatross and cockerel are birds, and since this is reflected by their semantic similarity as encoded by the LLM, we expect an LLM consistent with the constraint (“an albatross is a bird” $\rightarrow$ “an albatross can fly”) to correctly infer that “a cockerel can fly” too.

A principled probabilistic approach to do so is to encourage the LLM $p_{\theta}$ to allocate all probability mass to configurations of truth values that are consistent with the constraints $\alpha_{i}\in\mathcal{D}_{C}$, for instance by penalizing it proportionally to the probability it allocates to inconsistent truth values for all facts in the KB.
For every $\alpha_{i}$, the total probability allocated to the consistent configurations is

$$ $\mathsf{Pr}(\alpha_{i}):=\mathbb{E}_{\mathbf{z}\sim p_{\theta}(\mathbf{z})}[ \mathds{1}\{{\bf z}\models\alpha_{i}\}]=\sum\nolimits_{{\bf z}\models\alpha_{i }}\;\;p_{\theta}(\mathbf{z})$ (3) $$

where $\mathbf{z}$ is a vector containing the truth assignments $z_{1},\ldots,z_{K}$ of all the $K$
facts appearing in the constraint $\alpha_{i}$, and ${\bf z}\models\alpha_{i}$ indicates
that the assignment ${\bf z}$ satisfies the constraint.
For example, consider two facts $f_{1}:$ “a daffodil is a flower” and $f_{2}:$ “a daffodil is mortal” and the constraint $\alpha^{\prime}:z_{f_{1}}\rightarrow z_{f_{2}}$ stating that being a flower entails that the daffodil is mortal.
Then, all the configurations of $\mathbf{z}=(z_{f_{1}},z_{f_{2}})$ would satisfy $\alpha^{\prime}$ with the exception of $(\top,\bot)$ which clearly violates it.
Equation 3 is a special instantiation of computing the weighted model count (WMC) [, ] of a logical formula $\alpha_{i}$, where the weights associated to each model (a satisfying assignment to the formula) are given by the probabilities encoded by the LLM.

Furthermore, we can rewrite such probabilities $p_{\theta}(\mathbf{z})$ as the product the probabilities of the truth values of each fact, noting that for many LLM architectures they are conditionally independent given the embeddings at the last layer.
By taking the logarithm and reversing it into a minimization problem, we obtain the semantic loss (SL) [] objective that our LoCo-LMs minimize:

$$ $\mathcal{L}(\alpha_{i},p_{\theta})=-\log\sum\nolimits_{{\bf z}\models\alpha_{i }}\;\;\prod\nolimits_{j:{\bf z}\models z_{f_{j}}}p_{\theta}(z_{f_{j}})\;\; \prod\nolimits_{j:{\bf z}\models\lnot z_{f_{j}}}(1-p_{\theta}(z_{f_{j}}))$ (SL) $$

where $j:{\bf z}\models z_{f_{j}}$ (resp. $j:{\bf z}\models\lnot z_{f_{j}}$) indicates that the $j$-th fact in $\alpha_{i}$ is associated $\top$ (resp. $\bot$).
Consider the implication constraint $\alpha^{\prime}$ as defined before for encoding that a daffodil is mortal for being a flower.
Its satisfying assignments are $\mathbf{z}\models\alpha^{\prime}\in\{(\top,\top),(\bot,\top),(\bot,\bot)\}$.
Then, the summation in Equation SL
amounts to computing:

$$ $p_{\theta}(z_{f_{1}}=\top)p_{\theta}(z_{f_{2}}=\top)+(1-p_{\theta}(z_{f_{1}}= \top))p_{\theta}(z_{f_{2}}=\top)+(1-p_{\theta}(z_{f_{1}}=\top))(1-p_{\theta}(z _{f_{2}}=\top)))$ $$

where we can obtain the individual probabilities of facts being true directly by reading off the likelihood of utterances produced by the LLM, that is:

$$ $\displaystyle p_{\theta}(z_{f_{1}}=\top)$ $\displaystyle=p_{\theta}(x_{t}=\ell_{\mathsf{true}}\mid x_{1},\ldots,x_{t-1}= \text{``Is a daffodil a flower?''})$ $\displaystyle p_{\theta}(z_{f_{2}}=\top)$ $\displaystyle=p_{\theta}(x_{t}=\ell_{\mathsf{true}}\mid x_{1},\ldots,x_{t-1}= \text{``Is a daffodil a mortal?"}).$ $$

In the case of a constraint such as Equation F-Imp, the inner summation of the SL would reduce to a single configuration $\mathbf{z}=(\top,\top)$ when $z_{f_{1}}^{*}=\top$, which
can be interpreted as a special kind of cross-entropy computed only on pairs of facts considered to be jointly true in the KB, and to the set $\{(\bot,\top),(\bot,\bot)\}$ when $z_{f_{1}}^{*}=\bot$.
Note that Equation SL is agnostic to the kind of logical constraint involved, and therefore makes our approach general enough to tackle several settings where consistency-preserving solutions have been devised for specific constraints [, , ].

Crucially, the procedure to compute the models of a logical constraint can be automated.
However, naively computing the sum in Equation SL would require exponential time w.r.t. the number of possible facts in $\mathbf{z}$.
In fact, computing the WMC of a logical formula is a #P-hard problem in general [].
However, thanks to recent advancements in neuro-symbolic reasoning, we can compute that probability and differentiate through it efficiently [, , ].
Specifically, we rely on modern compilers that translate a logical formula $\alpha_{i}$ into compact and differential computational graphs called circuits [, ], such as sentential decision diagrams [, , ].

To recap, during training we
loop over every constraint in $\alpha_{i}\in\mathcal{D}_{C}$, prompt the LLM to gather the probabilities of every fact participating in $\alpha_{i}$ to be true and plug them in our only loss, as described in Equation SL.
Then, we
backpropagate as to fine-tune (some of) the parameters $\theta$ of the LLM, by using LoRA [] and quantization [] if necessary.
This simple and principled recipe is able to scale well and
is extremely effective
at improving logical consistency on a number of well-known benchmarks, as discussed in Section 5.

## 4 Related Work

##### LLMs and factual reasoning.

LLMs are
increasingly being employed as implict KBs [, ],
however
ensuring they are factually consistent is still an open challenge [, ].
A number of works augment LLMs with external KBs, especially in the context of QA, and with the primary aim of improving answer factuality [, , ]. A popular approach to do so is retrieval augmented generation [, ], which however is not yet suited for more complex reasoning scenarios.
Alternatively, external KBs have been used to improve reasoning, e.g., via prompt learning [] or ex-post model editing []. However, current knowledge editing methods, including supervised fine-tuning, do not guarantee the propagation of factuality between units of knowledge related by logical relations [, ].
Mitigating hallucinations in LLMs [, ] is related to enforcing factuality, but as generated inconsistencies might not map to a single entry in a KB, they are harder to detect and prevent [].

##### More complex reasoning with LLMs.

Much less attention has been posed to other forms of reasoning, such as combining modus ponens, consistent negation and combination thereof.
Even when this is done, reasoning is generally cast as a QA task, where an LLM has to predict the satisfiability of logical formulas of different complexities.
To this end, benchmarks such as SimpleLogic []
or LogicBench []
have been proposed.
Implication or entailment [, ] are also usually cast as a QA prediction task [].
Datasets such as BeliefBank [] provide collections of simple implication constraints to test this, while more sophisticated benchmarks such as EntailmentBank [] collect more complex implications, e.g., trees of natural language statements.
Shortcomings in consistent reasoning have been recently highlighted for larger LLMs such as ChatGPT and GPT-4 variants [], which are however harder to fine-tune efficiently.
Other works [] highlighted how (even large) LLMs suffer from not being able to recognize the logical equivalence of “A is-a B” relationships and “B is-a A” ones. These relationships could be seen as a type of logical constraint, specifically concept membership to an ontology class, and hence could be modeled in our framework.

For complex reasoning scenarios, logical consistency can be improved in a number of ways, the most successful of which involves external tools, such as MaxSAT solvers, which flip the predictions of an LLM to be (approximately) consistent with a set of related questions, as done by ConCoRD [].
Analogously, self-consistency can be ameliorated by first constructing a belief graph – a factor graph relating the beliefs of an LLM fine-tuned on implications such as Entailer [] – over which a MaxSAT solver is applied
[].
Higher level constraints can also be checked and enforced with external verifiers [].
Differently from LoCo-LMs, backpropagating through these external tools is hard [],
furthermore, while they can guarantee self-consistency among facts within every call of a MaxSAT solver, this cannot be done for the same facts across different calls.

##### Semantic loss & other NeSy approaches

There is a vast literature on NeSy integration methods [, ], most of which are used for enforcing constraint on tabular data [], image data [, , ] and more recently video recognition [] with the purpose of building trustworthy predictors.
Several variants of the semantic loss [, , ] and neural weighted model counting [] have been proposed but, to the best of our knowledge, never employed to enforce logical consistency in LLMs.
In our experiments we found that our simple formulation (Equation SL) is good enough to greatly improve consistency over previous state-of-the-art methods in NLP (Section 5).
Closer to our work, [] applied a semantic loss to instill first-order rule constraints in the embedding space of entities in encoder-only models to reason on the CLUTTR benchmark [], comprising semi-synthetic stories involving hypothetical families.
Fuzzy logic approaches [] can be used to distill regularizers that can promote consistency [].
Differently from our probabilistic logic approach however, they are syntax-dependent, i.e., rewriting a constraint into a logically equivalent one would yield a different penalty term and can greatly influence optimization [, ].

## 5 Experiments

We aim to answer the following research questions:
RQ1: Can LoCo-LMs achieve comparable or superior consistency to methods using external reasoners using less training data?
RQ2: Can LoCo-LMs retain good consistency to unseen types of constraint at training time? How much does training on all the constraints jointly improve consistency overall?
RQ3: Can LoCo-LMs transfer consistent knowledge to domains out of the training distribution?

### 5.1 RQ1: How do LoCo-LMs perform compared to external solvers?

We reproduce the experimental setting of Mitchell et al. [] to compare against ConCoRD, a symbolic layer that uses a MaxSAT solver to impose self-consistency for implication ex-post.

Data.
We train LoCo-LMs on the BeliefBank [].
We use the three splits as in Mitchell et al. []:
a “calibration" set of $1,072$ annotated facts about $7$ entities of the form (subject, property, true/false)
used for training,
a “silver" set of $12,636$ facts about 85 entities used for evaluation,
and a set of $2224$ valid abstract logical implications.
We generate ground implication rules ($\mathcal{D}_{C}$) by looking up the subjects of all facts in the training set: if the antecedent or the consequent fact of the general constraint is known for that subject, we add the subject ground implication constraint to the dataset.
Section A.1.1 details the whole process.

To measure generalization across entities, we generate two controlled splits of the training calibration set:
T1 facts, appearing either as antecedents or consequents in the constraints; T2 facts, appearing exclusively as consequents.
The goal is to correctly guess the consequents by seeing only the antecedents and the constraints.
We subsequently test the effects of pure supervised fine-tuning on a portion of random facts from the whole calibration set (T1+T2).

Models.
As in Mitchell et al. [],
we use Macaw-Large [] ($770M$ parameters), a sequence-to-sequence language model capable of multi-angle QA with fixed prompt templates.
We keep the same prompts used for Macaw, reported in Appendix D.1.
At test time, we verify the validity of the answer format and consider any invalid or negative response as a belief with label "false".
We adopt a similar set of hyperparameters as for Macaw []: we fine-tune our models for $3$ epochs with a learning rate fixed to $\gamma=3\cdot 10^{-4}$, batch size $4$ with gradient accumulation ($64/16$ steps), on one nVidia A30 24GB GPU. We use AdamW [] as optimizer with a default weight decay $\lambda=10^{-2}$.

Competitors and Metrics. We compare
ConCoRD as applied to Macaw-Large, using RoBERTa-ANLI [] for relationship inference, versus
a pre-trained Macaw-Large model from [] as zero-shot baseline and our LoCo version of it (LoCo-Macaw).
We evaluate our models for factuality and implication self-consistency.
We measure the former with the $F_{1}$ score to account for the
unbalance between false and true facts [].
Factuality is measured on the two splits (antecedents and consequents) and the complete facts set (Tot) for both calibration and silver splits.
For implication self-consistency, sometimes named just “consistency” [], we query beliefs from LLMs about the complete facts set and count the fraction of violated constraints in $\mathcal{D}_{C}^{\text{test}}$ according to the implication rule (Imp), that is, when a true antecedent for the model implies a false consequent, to then compute:

$$ $1-|\{\alpha_{i}=(z_{j}\rightarrow z_{k}):z_{j}=\top,z_{k}=\bot\}|\ /\ |\{ \alpha_{i}=(z_{j}\rightarrow z_{k}):z_{j}=\top\}|.$ (4) $$

Results.
Table 1 reports all metrics for all models.
We firstly observe a net improvement in both factuality and logical consistency with our LoCo-LMs, compared to pre-trained Macaw-Large and the ConCoRD variant.
Standard supervised fine-tuning with the XENT loss on antecedent facts is insufficient: due to a class imbalance between true facts ($\sim{10\%}$) and false facts ($\sim{90\%}$), the model tends to label any statement as “false”.
This is accentuated in the training distribution (see Table 4).
Assuming the language model can access to a portion of consequent facts, LoCo-LMs still yields better logical consistency and factuality for unseen consequents in low-data regimes (e.g., 5-10% of the T1+T2 dataset) compared to canonical supervised fine-tuning.
When they are allowed to see more data (e.g., 75% of the T1+T2 dataset), traditionally fine-tuned models can “cheat” and directly learn about the consequents (somehow equivalent to memorizing a single row of the truth table).
In this scenario, LoCo-LMs achieve comparable logical self-consistency and factuality over consequents, but less on the antecedents.

**Table 1: LoCo-LMs achieve better logical self-consistency and factuality than ConCoRD [] as measured via Equation 4 and $F_{1}$ scores when fine-tuned only on T1 facts only and boost performance in the presence of a small fraction of T1+T2 facts (5-10%). A similar trend is visible on training data (Table 4).**
| Method | Train Subset | Ant $F_{1}$ | Con $F_{1}$ | Tot $F_{1}$ | Imp |
| --- | --- | --- | --- | --- | --- |
| ConCoRD |  |  |  | 0.91 | 0.91 |
| Macaw-Large |  | 0.52 | 0.90 | 0.81 | 0.83 |
| Macaw+XENT | T1 | 0.13 | 0.01 | 0.03 | 0.72 |
| LoCo-Macaw | T1 | 0.79 | 0.98 | 0.96 | 0.99 |
| Macaw+XENT | T1+T2 (5%) | 0.23 | 0.78 | 0.72 | 0.82 |
| LoCo-Macaw | T1+T2 (5%) | 0.67 | 0.83 | 0.81 | 0.92 |
| Macaw+XENT | T1+T2 (10%) | 0.55 | 0.97 | 0.91 | 0.90 |
| LoCo-Macaw | T1+T2 (10%) | 0.45 | 0.97 | 0.89 | 0.93 |
| Macaw+XENT | T1+T2 (75%) | 0.85 | 0.99 | 0.97 | 0.98 |
| LoCo-Macaw | T1+T2 (75%) | 0.79 | 0.99 | 0.95 | 0.98 |

In conclusion, we observe our fine-tuning method allows Macaw-large to be more logically self-consistent than with an external solver.
We conjecture that this is possible thanks to the high semantic similarity between facts in the train and test splits (Section C.1).
In terms of inference speed, our LoCo-LMs take less time that querying the same base model and an additional reasoner(^4^44On BeliefBank, LoCo-LMs take 2405.28s at test time, compared to ConCoRD [], 3669.33s.), at the cost of a one-time training step that can be amortized.(^5^55Training LoCo-LM takes 2124.48s on BeliefBank.)
Moreover, our semantic loss is more sample-efficient than XENT fine-tuning to achieve higher logical consistency especially with small portions of ground-truth data.

### 5.2 RQ2: How do LoCo-LMs deal with different logical constraints?

Setting. As in Section 5.1, we use BeliefBank to train and evaluate our LoCo-LMs on different types of logical rules. We use $90\%$ and $10\%$ of T1 facts for training and validation, respectively; T2 facts for testing. We employ two sets of labels to make our models less sensitive to the prompt format; at training time, one format is chosen with $50\%$ chance for each batch; details in Appendix D.2. At test time we do not apply any strict parsing on the outputs: unless the token encodes the truth label (e.g., “Is a computer an electronic device? yes”), the output is considered as a negative answer.

Models. To train larger language models, we choose the LLaMa-2 [] family of decoder-only models, widely adopted in literature for its performance across a variety of tasks and domains.
We consider three baselines: the available pre-trained $7$b and $70$b models, 4-bit NormalFloat quantized [], with greedy sampling strategy, temperature $t=1.0$ and dropout disabled; we also perform supervised fine-tuning of the $7$b model (4-bit, with LoRA []) on the ground truth T1+T2 facts set, namely “LLaMa-2-7b + XENT”.
We derive our LoCo-LMs fine-tuning with our proposed method LLaMa-2 7b, with 4-bit quantization and LoRA.
We limit the generation to $4$ tokens following the input.
We adopt a similar set of hyperparameters to LoRA: we fine-tune our models for $5$ epochs keeping the learning rate fixed to $\gamma=3\cdot 10^{-4}$, batch size $64$, on $1$ nVidia A100-40GB GPU. We use AdamW [] as optimizer with a default weight decay $\lambda=10^{-2}$.
We use the SL to finetune three LoCo-LM variants: for negation (Neg), factual implication consistency (F-Imp) and their conjunction, i.e., given an implication $f_{1}\rightarrow f_{2}$ we provide the SL with the constraint:

$$ $(z_{f_{1}}\oplus z_{\widetilde{f_{1}}})\wedge(z_{f_{1}}=z_{f_{1}}^{*})\wedge(z _{f_{1}}\rightarrow z_{f_{2}})\wedge(z_{f_{2}}\oplus z_{\widetilde{f_{2}}})$ ( Super ) $$

where $\widetilde{f_{1}}$ and $\widetilde{f_{2}}$ encode the textual negation of $f_{1}$ and $f_{2}$, generated via ConCoRD’s templates.

Metrics. We fine-tune on Neg, F-Imp or Super and evaluate on all constraints.
Specifically, we measure the implication self-consistency, defined in Equation 4, as well as the implication consistency:

$$ $1-|\{\alpha_{i}=(z_{j}\rightarrow z_{k}):z_{j}^{*}=\top,z_{k}=\bot\}|\ /\ |\{ \alpha_{i}=(z_{j}\rightarrow z_{k}):z_{j}^{*}=\top\}|$ (5) $$

where $z_{j}^{*}$ is the ground truth value of a fact.
We also measure reverse implication consistency

$$ $1-|\{\alpha_{i}=(z_{\widetilde{k}}\rightarrow z_{\widetilde{j}}):\neg z_{k}^{* }=\top,z_{\widetilde{j}}=\top\}|\ /\ |\{\alpha_{i}=(z_{\widetilde{k}} \rightarrow z_{\widetilde{j}}):\neg z_{k}^{*}=\top\}|$ (6) $$

and the reverse implication self-consistency variant:

$$ $1-|\{\alpha_{i}=(z_{\widetilde{k}}\rightarrow z_{\widetilde{j}}):z_{\widetilde {k}}=\bot,z_{\widetilde{j}}=\top\}|\ /\ |\{\alpha_{i}=(z_{\widetilde{k}} \rightarrow z_{\widetilde{j}}):z_{\widetilde{k}}=\bot\}|$ (7) $$

where $z_{\widetilde{k}}$ and $z_{\widetilde{j}}$ are the truth values of the textual negations of facts $k$ and $j$ according to the model.
For negation self-consistency we compute

$$ $1-{|\{\alpha_{i}=(z_{j}\oplus z_{\widetilde{j}}):z_{j}=z_{\widetilde{j}}}\}|\ /\ {|\alpha_{i}=(z_{j}\oplus{z_{\widetilde{j}}})|}.$ (8) $$

As in Section 5.1, we measure factuality (FAC) as the $F_{1}$ score on a set of ground truth facts.
Finally, we account for possible shifts in the language modeling distribution by computing its perplexity (PPL) on WikiText [], formatted as a single token sequence.

Results. In Table 2, we first observe an overall boost in factuality for all LoCo-LMs over the 7b baselines.
Compatibly with Table 1, supervised fine-tuning is not sufficient to improve logical consistency significantly.
Our LoCo-LM trained exclusively on Imp constraints performs best in factuality and implication consistency; however, as we haven’t trained it on negated facts, scores on negation consistency and reverse implication are notably low.
Finally, fine-tuning a LoCo-LM on the combination of both constraints (Super), yields on average the most consistent language model, which on average surpasses even Llama 2 70B, a much larger model.
Overall, fine-tuning with our method does not impact negatively fluency, as measured by perplexity.

**Table 2: LoCo-LMs achieve higher (self-)consistency than off-the-shelf baselines and models trained with supervised fine-tuning (+XENT) on the BeliefBank test split. Scores are averaged across two sets of prompts and truth labels, for which results are reported in Appendix 8 and 9.**
|  |  |  | consistency | self-consistency |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| model | train | PPL | FAC | IMP | REV | NEG | IMP | REV | AVG |
| LLaMa-2-7b Zero Shot |  | 62.41 | 0.39 | 0.52 | 0.13 | 0.42 | 0.30 | 0.15 | 0.32 |
| LLaMa-2-7b Few Shot |  | 52.30 | 0.53 | 0.71 | 0.34 | 0.38 | 0.48 | 0.47 | 0.48 |
| LLaMa-2-7b CoT |  | 52.30 | 0.52 | 0.64 | 0.67 | 0.40 | 0.64 | 0.67 | 0.59 |
| LLaMa-2-70b Zero Shot |  | 44.90 | 0.47 | 0.69 | 0.81 | 0.13 | 0.31 | 0.91 | 0.55 |
| LLaMa-2-7b + XENT | T1+T2 | 116.85 | 0.25 | 0.46 | 0.01 | 0.07 | 0.81 | 0.01 | 0.27 |
| LoCo-LLaMa-2-7b (NEG) | T1 | 62.21 | 0.44 | 0.65 | 0.43 | 0.96 | 0.28 | 0.36 | 0.52 |
| LoCo-LLaMa-2-7b (F-IMP) | T1 | 67.15 | 0.99 | 0.99 | 0.07 | 0.00 | 0.99 | 0.07 | 0.51 |
| LoCo-LLaMa-2-7b (Super) | T1 | 62.23 | 0.74 | 0.77 | 0.77 | 0.87 | 0.71 | 0.77 | 0.77 |

### 5.3 RQ3: Can finetuning LoCo-LMs help consistency on unseen KB?

Data. We evaluate LoCo-LMs on the EntailmentBank [] test split, as proposed by Kassner et al. [] to reason on graphs of logical entailments. It consists of $302$ implication trees spawning 805 constraints, with an average of $6.57$ statement nodes and $2.66$ constraints per tree; we consider each node of each tree as a statement with natural language with truth label set to $1$. We limit the tree depth to $5$. An illustrated example is provided in Appendix 2. As in 5.2, we test two prompt and label formats. We assume that a possible semantic overlap between the training and test distributions, BeliefBank and EntailmentBank respectively, could underlie higher consistency scores across entailment trees; we estimate such overlap in Appendix 13.

Competitors and Metrics.
We test our LoCo-LMs based on LLaMa-2 7b and previously trained in 5.2 on BeliefBank, without applying any changes. As baseline model, we consider LLaMa-2 7b without quantization.
This experimental setup is inspired by Kassner et al. [], from whom we derive the notion of self-consistency on trees of entailments: each entailment tree $t\in\mathcal{T}$ is a direct acyclic graph with a single root encoding the hypothesis to be proved; a subtree $t^{\prime}$ consists in each parents-child relationship in $t$, representing an entailment between the parent nodes (antecedents in logical conjunction) and the child (consequent).
See Figure 2 for an example.
For each tree $t$, we count the amount of violated subtrees $t^{\prime}$, that is when a true conjunction of antecedents does not imply a true consequent. Finally, we measure logical consistency as the fraction of the total violated subtrees over the total number of subtrees in $\mathcal{T}$.

Results. In Table 3 we report logical consistency across depths. Scores are averaged across two sets of prompts and labels, detailed results are reported in Appendix 10. We observe the consistency decreases across depths for the baseline model, until it flattens out, as more implications are evaluated. Conversely, LoCo-LM (F-IMP) and LoCo-LM (Super) achieve higher consistency across depths.
While promising, these results should be interpreted with caution, for two reasons. Firstly, we observed variability in model predictions with varying prompt formats and labels (Appendix A.3), suggesting further engineering for more consistent answers.
Second, while be measure a discrete semantic similarity between the two datasets (Table 13) which can justify transfer, we note that our measure are cosine similarities and their effectiveness might depend on the pre-training task [].
This encourages further research on employing neuro-symbolic methods to improve multi-hop consistency in LMs w.r.t. external KB [] or the model’s own implications [].

**Table 3: LoCo-LMs can be consistent across unseen trees of entailments from EntailmentBank when trained for implication consistency (F-IMP) on BeliefBank. Finetuning for negation alone (NEG) does not seem to improve over the baseline.**
|  | depth |  |  |  |  |
| --- | --- | --- | --- | --- | --- |
| model | 1 | 2 | 3 | 4 | 5 |
| LLaMa-2-7b | 0.87 | 0.76 | 0.59 | 0.61 | 0.63 |
| LoCo-LLaMa-2-7b (NEG) | 0.51 | 0.51 | 0.51 | 0.52 | 0.52 |
| LoCo-LLaMa-2-7b (F-IMP) | 0.98 | 0.98 | 0.98 | 0.98 | 0.98 |
| LoCo-LLaMa-2-7b (Super) | 0.69 | 0.68 | 0.68 | 0.68 | 0.69 |

## 6 Discussion and Further Work

##### Limitations.

One limitation of our approach is sensitivity to the choice of prompt format, a general phenomenon [] that in our case means (self-)consistency improvements do not always carry over across formats. This can be partially addressed by fine-tuning using a mixture of formats, as we do in Section 5.
While our SL is constraint-agnostic, in practice we
fine-tune LoCo-LMs only against a combination of implications and exclusive ORs.
While this setup is already richer than those studied in related works (Section 4)
and achieves positive transfer to tasks requiring multiple reasoning steps, it leaves more room for future work on more complex benchmarks.

LoCo-LMs fine-tuning relies on two assumptions: that the probabilities of facts are conditionally independent given the LLM inner state, and that the constraints in the KB are correct.
The former readily applies to many LLMs,
but assuming independence can bias the solutions learned by the SL [].
For the latter, most KBs are well-curated, but fine-tuning models against incorrect or inconsistent rules can compromise consistency and fluency.
Naturally, malicious users could intentionally train LoCo-LMs against invalid rules to steer the model towards logical conclusions of their choice or potential reasoning shortcuts [, , ].

Our results show that LoCo-LMs have improved (self-)consistency compared to recently introduced consistency layers which rely on external solvers, such as ConCoRD.
In future work, we plan to extend our analysis to more complex logical operators [] and to consider more advanced probabilistic reasoning techniques that sport improved consistency guarantees [].
Another promising direction we have not explored is that of
first materializing the beliefs of an LLM such as in REFLEX [] and variants [] and use the SL to improve consistency while potentially storing and re-using derived rules in a writable external KB [, ].

## Acknowledgments and Disclosure of Funding

Funded by the European Union. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Health and Digital Executive Agency (HaDEA). Neither the European Union nor the granting authority can be held responsible for them. Grant Agreement no. 101120763 - TANGO.
AV is supported by the "UNREAL: Unified Reasoning Layer for Trustworthy ML" project (EP/Y023838/1) selected by the ERC and funded by UKRI EPSRC.

## Appendix A Detailed setting and results

### A.1 RQ1

#### A.1.1 Data preprocessing

We train LoCo-LMs on the BeliefBank [], calibration split.
This dataset is derived from ConceptNet [], a large curated knowledge graph encoding factual knowledge and logical relations between entities at different levels of abstraction; we use the splits introduced by Mitchell et al. [] for direct comparison.
It consists of three pieces:
a “calibration" set of $1,072$ annotated facts about $7$ entities of the form (subject, property, true/false) used for training,
a “silver" set of $12,636$ facts about 85 entities used for evaluation,
and a set of $2224$ valid abstract logical implications.
To use our SL, we require defining a set of ground constraints.
We derive these as follows. For each general implication constraint, we lookup the subjects of all facts in the training set: if the antecedent or the consequent fact of the general constraint is known for that subject, we add the subject ground constraint to the dataset $\mathcal{D}_{C}$.

We generate two splits: T1 facts, appearing either as antecedents or consequents in the constraints; T2 facts, appearing exclusively as consequents. The goal is to correctly guess the consequents by seeing only the antecedents and the constraints. In the calibration set, we count $796$ antecedents and $276$ consequents, spawning $14,005$ grounded constraints. In the silver set, we count $9,504$ antecedents and $3,132$ consequents, spawning $169,913$ grounded constraints.
We subsequently test the effects of pure supervised fine-tuning: a portion of random facts from the calibration set (T1+T2) is taken with the goal to predict the excluded antecedent or consequent facts.
We train on T1 facts and evaluate on T2 facts for RQ2 as well: T1 facts (antecedents) constitute a valid subset for all the considered logical rules.

**Table 4: LoCo-LMs achieve better logical self-consistency and factuality as measured via Equation 4 and $F_{1}$ scores when compared to cross-entropy fine-tuning (XENT) and baselines using external reasoners such as ConCoRD [] measured on train (calibration set) facts. For RQ1 (Section 5), LoCo-LMs fine-tuned on T1 facts only outperform training-free baseline for all metrics. For RQ2, they boost performance in the presence of a small fraction of T1+T2 facts (5-10%). For larger dataset sizes, LoCo-LMs are competitive for consistency and factuality on consequents.**
|  | Method | Train size | Antecedents $F_{1}$ | Consequents $F_{1}$ | Total $F_{1}$ | Logical consistency |
| --- | --- | --- | --- | --- | --- | --- |
| RQ1 | ConCoRD |  |  |  | 0.91 | 0.91 |
| MACAW |  | 0.47 | 0.84 | 0.78 | 0.82 |  |
| MACAW+XENT | T1 | 0.46 | 0.08 | 0.14 | 0.79 |  |
| LoCo-LM | T1 | 0.98 | 0.99 | 0.99 | 1.00 |  |
| RQ2 | MACAW+XENT | T1+T2 (5%) | 0.31 | 0.73 | 0.69 | 0.90 |
| LoCo-LM | T1+T2 (5%) | 0.34 | 0.77 | 0.72 | 0.92 |  |
| MACAW+XENT | T1+T2 (10%) | 0.48 | 0.88 | 0.85 | 0.87 |  |
| LoCo-LM | T1+T2 (10%) | 0.52 | 0.95 | 0.89 | 0.91 |  |
| MACAW+XENT | T1+T2 (75%) | 0.69 | 1.00 | 0.97 | 0.97 |  |
| LoCo-LM | T1+T2 (75%) | 0.65 | 1.00 | 0.97 | 0.99 |  |

### A.2 RQ2

**Table 5: LoCo-LMs evaluated on BeliefBank, training (calibration) split. Scores are averaged across two sets of prompts and truth labels. We observe fine-tuning with our method allows for higher logical consistency to different rules.**
|  |  |  | consistency | self-consistency |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| model | train subset | PPL | FAC | IMP | REV | NEG | IMP | REV | AVG |
| LLaMa-2-7b Zero Shot |  | 62.41 | 0.41 | 0.57 | 0.21 | 0.42 | 0.28 | 0.24 | 0.36 |
| LLaMa-2-7b Few Shot |  | 52.30 | 0.52 | 0.70 | 0.45 | 0.38 | 0.48 | 0.46 | 0.50 |
| LLaMa-2-7b CoT |  | 52.30 | 0.52 | 0.64 | 0.67 | 0.40 | 0.64 | 0.67 | 0.59 |
| LLaMa-2-70b Zero Shot |  | 44.90 | 0.49 | 0.72 | 0.80 | 0.12 | 0.32 | 0.91 | 0.56 |
| LLaMa-2-7b + XENT | T1+T2 | 116.85 | 0.21 | 0.39 | 0.01 | 0.10 | 0.44 | 0.01 | 0.20 |
| LoCo-LLaMa-2-7b (NEG) | T1 | 62.21 | 0.28 | 0.52 | 0.43 | 0.82 | 0.55 | 0.36 | 0.49 |
| LoCo-LLaMa-2-7b (F-IMP) | T1 | 67.15 | 1.00 | 1.00 | 0.08 | 0.00 | 1.00 | 0.08 | 0.53 |
| LoCo-LLaMa-2-7b (Super) | T1 | 62.23 | 0.86 | 0.89 | 0.76 | 0.88 | 0.80 | 0.77 | 0.83 |

**Table 6: LoCo-LMs evaluated on BeliefBank, training (calibration) split. Prompt format 1 [true, false] is used. We observe fine-tuning with our method allows for higher logical consistency to different rules.**
|  |  |  | consistency | self-consistency |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| model | train subset | PPL | FAC | IMP | REV | NEG | IMP | REV | AVG |
| LLaMa-2-7b Zero Shot |  | 62.41 | 0.43 | 0.63 | 0.33 | 0.38 | 0.29 | 0.39 | 0.41 |
| LLaMa-2-7b Few Shot |  | 52.30 | 0.53 | 0.74 | 0.36 | 0.28 | 0.42 | 0.37 | 0.45 |
| LLaMa-2-7b CoT |  | 52.30 | 0.67 | 0.76 | 0.77 | 0.32 | 0.74 | 0.77 | 0.66 |
| LLaMa-2-70b Zero Shot |  | 44.90 | 0.52 | 0,76 | 0.79 | 0.18 | 0.35 | 0.90 | 0.58 |
| LLaMa-2-7b + XENT | T1+T2 | 116.85 | 0.37 | 0.47 | 0.02 | 0.16 | 0.89 | 0.02 | 0.32 |
| LoCo-LLaMa-2-7b (NEG) | T1 | 62.21 | 0.46 | 0.70 | 0.85 | 0.93 | 0.28 | 0.72 | 0.66 |
| LoCo-LLaMa-2-7b (F-IMP) | T1 | 67.15 | 1.00 | 1.00 | 0.08 | 0.00 | 1.00 | 0.08 | 0.53 |
| LoCo-LLaMa-2-7b (Super) | T1 | 62.23 | 0.88 | 0.91 | 0.72 | 0.94 | 0.86 | 0.73 | 0.84 |

**Table 7: LoCo-LMs evaluated on BeliefBank, training (calibration) split. Prompt format 2 [yes, no] is used. We observe fine-tuning with our method allows for higher logical consistency to different rules.**
|  |  |  | consistency | self-consistency |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| model | train subset | PPL | FAC | IMP | REV | NEG | IMP | REV | AVG |
| LLaMa-2-7b Zero Shot |  | 62.41 | 0.39 | 0.51 | 0.08 | 0.46 | 0.27 | 0.09 | 0.30 |
| LLaMa-2-7b Few Shot |  | 52.30 | 0.52 | 0.66 | 0.55 | 0.48 | 0.55 | 0.55 | 0.55 |
| LLaMa-2-7b CoT |  | 52.30 | 0.38 | 0.52 | 0.57 | 0.48 | 0.54 | 0.57 | 0.51 |
| LLaMa-2-70b Zero Shot |  | 44.90 | 0.46 | 0.68 | 0.81 | 0.05 | 0.28 | 0.93 | 0.54 |
| LLaMa-2-7b + XENT | T1+T2 | 116.85 | 0.05 | 0.32 | 0.00 | 0.04 | 0.00 | 0.00 | 0.07 |
| LoCo-LLaMa-2-7b (NEG) | T1 | 62.21 | 0.09 | 0.33 | 0.00 | 0.70 | 0.82 | 0.00 | 0.32 |
| LoCo-LLaMa-2-7b (F-IMP) | T1 | 67.15 | 1.00 | 1.00 | 0.08 | 0.00 | 1.00 | 0.08 | 0.53 |
| LoCo-LLaMa-2-7b (Super) | T1 | 62.23 | 0.84 | 0.87 | 0.79 | 0.82 | 0.74 | 0.80 | 0.81 |

**Table 8: LoCo-LMs evaluated on BeliefBank, test (silver) split. Prompt format 1 [true, false] is used. We observe fine-tuning with our method allows for higher logical consistency to different rules.**
|  |  |  | consistency | self-consistency |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| model | train subset | PPL | FAC | IMP | REV | NEG | IMP | REV | AVG |
| LLaMa-2-7b Zero Shot |  | 62.41 | 0.41 | 0.55 | 0.22 | 0.41 | 0.30 | 0.25 | 0.36 |
| LLaMa-2-7b Few Shot |  | 52.30 | 0.53 | 0.75 | 0.37 | 0.27 | 0.41 | 0.37 | 0.45 |
| LLaMa-2-7b CoT |  | 52.30 | 0.67 | 0.76 | 0.77 | 0.32 | 0.74 | 0.77 | 0.67 |
| LLaMa-2-70b Zero Shot |  | 44.90 | 0.50 | 0.72 | 0.80 | 0.20 | 0.34 | 0.89 | 0.58 |
| LLaMa-2-7b + XENT | T1+T2 | 116.85 | 0.40 | 0.52 | 0.02 | 0.11 | 0.82 | 0.02 | 0.31 |
| LoCo-LLaMa-2-7b (NEG) | T1 | 62.21 | 0.44 | 0.64 | 0.86 | 0.92 | 0.28 | 0.72 | 0.64 |
| LoCo-LLaMa-2-7b (F-IMP) | T1 | 67.15 | 0.98 | 0.98 | 0.07 | 0.00 | 0.98 | 0.07 | 0.51 |
| LoCo-LLaMa-2-7b (Super) | T1 | 62.23 | 0.75 | 0.78 | 0.72 | 0.91 | 0.74 | 0.72 | 0.77 |

**Table 9: LoCo-LMs evaluated on BeliefBank, test (silver) split. Prompt format 2 [yes, no] is used. We observe fine-tuning with our method allows for higher logical consistency to different rules.**
|  |  |  | consistency | self-consistency |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| model | train subset | PPL | FAC | IMP | REV | NEG | IMP | REV | AVG |
| LLaMa-2-7b Zero Shot |  | 62.41 | 0.37 | 0.48 | 0.04 | 0.43 | 0.29 | 0.04 | 0.28 |
| LLaMa-2-7b Few Shot |  | 52.30 | 0.53 | 0.67 | 0.57 | 0.49 | 0.58 | 0.53 | 0.56 |
| LLaMa-2-7b CoT |  | 52.30 | 0.38 | 0.52 | 0.57 | 0.48 | 0.54 | 0.57 | 0.51 |
| LLaMa-2-70b Zero Shot |  | 44.90 | 0.44 | 0.65 | 0.82 | 0.05 | 0.29 | 0.93 | 0.53 |
| LLaMa-2-7b + XENT | T1+T2 | 116.85 | 0.11 | 0.39 | 0.00 | 0.03 | 0.80 | 0.00 | 0.22 |
| LoCo-LLaMa-2-7b (NEG) | T1 | 62.21 | 0.44 | 0.65 | 0.00 | 1.00 | 0.28 | 0.00 | 0.40 |
| LoCo-LLaMa-2-7b (F-IMP) | T1 | 67.15 | 0.99 | 0.99 | 0.07 | 0.00 | 0.99 | 0.07 | 0.52 |
| LoCo-LLaMa-2-7b (Super) | T1 | 62.23 | 0.73 | 0.75 | 0.81 | 0.83 | 0.67 | 0.82 | 0.77 |

**Table 10: LoCo-LMs can achieve higher consistency across depth than the baseline. Scores are computed with Format 1 [true, false], reported in Appendix D.2. LoCo-LM fine-tuned with on the implication rule achieves best consistency.**
|  | depth |  |  |  |  |
| --- | --- | --- | --- | --- | --- |
| model | 1 | 2 | 3 | 4 | 5 |
| LLaMa-2-7b | 0.73 | 0.77 | 0.79 | 0.80 | 0.80 |
| LoCo-LLaMa-2-7b (NEG) | 0.03 | 0.03 | 0.03 | 0.04 | 0.05 |
| LoCo-LLaMa-2-7b (F-IMP) | 0.97 | 0.96 | 0.97 | 0.97 | 0.97 |
| LoCo-LLaMa-2-7b (Super) | 0.75 | 0.74 | 0.73 | 0.73 | 0.74 |

### A.3 RQ3

**Table 11: LoCo-LMs can achieve higher consistency across depth than the baseline. Scores are computed with Format 2 [yes, no], reported in Appendix D.2. LoCo-LM fine-tuned with on the implication rule and the negation rule achieve best consistency. High sensitivity to prompts should be considered.**
|  | depth |  |  |  |  |
| --- | --- | --- | --- | --- | --- |
| model | 1 | 2 | 3 | 4 | 5 |
| LLaMa-2-7b | 1.00 | 0.75 | 0.38 | 0.42 | 0.46 |
| LoCo-LLaMa-2-7b (NEG) | 0.99 | 0.99 | 0.99 | 0.99 | 0.99 |
| LoCo-LLaMa-2-7b (F-IMP) | 0.99 | 0.99 | 0.99 | 0.99 | 0.99 |
| LoCo-LLaMa-2-7b (Super) | 0.62 | 0.62 | 0.63 | 0.63 | 0.64 |

**Table 12: Distribution of answer labels from LoCo-LMs for different prompt formats on the EntailmentBank test split.**
|  | labels: [yes, no] | labels: [true, false] |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| model | yes | no | invalid | true | false | invalid |
| LLaMa-2-7b | 1188 | 6 | 1441 | 615 | 1742 | 278 |
| LoCo-LLaMa-2-7b (NEG) | 2538 | 0 | 97 | 940 | 0 | 1695 |
| LoCo-LLaMa-2-7b (F-IMP) | 2557 | 0 | 78 | 2441 | 194 | 0 |
| LoCo-LLaMa-2-7b (Super) | 2079 | 486 | 70 | 874 | 1756 | 5 |

## Appendix B EntailmentBank

Figure: Figure 2: An illustration of an entailment tree, namely a “prof", from EntailmentBank []. Blue nodes are premises in logical conjunction, orange nodes are implications and the green node denote the hypothesis to prove.
Refer to caption: x2.png

## Appendix C Semantic overlap

We base our measurement for semantic overlap on cosine similarity, widely adopted in literature. We report our results with a note for caution: it is unclear whether embeddings could be similar for the semantic features we are seeking [], suggesting further research on the topic.

### C.1 BeliefBank

We measure the semantic overlap between the training and test distribution by constructing a Representation Dissimilarity Matrix (RDM) of Macaw’s embeddings (token average) between training and test entities. The main assumption is that semantically similar subjects may have similar properties, as a proxy for domain knowledge transfer.

Figure: Figure 3: Pairwise cosine similarities between entities in the training distribution (calibration, rows) and the test distribution (silver, columns).
Refer to caption: x3.png

### C.2 BeliefBank-EntailmentBank

We consider the training split, namely “calibration” in ConCoRD [], from BeliefBank [], and the test split from EntailmentBank [] to estimate the knowledge that LoCo-LMs could transfer to entailment trees. We process BeliefBank as a set of $1,072$ facts, while EntailmentBank as a set of $2,635$ facts. Both sets contain statements in natural language that are converted into vector embeddings through encoding with LLaMa-2-7b []; the last layer logits are considered and a sentence representation is obtained by averaging across tokens. We consequently compute the pairwise cosine similarities between fact embeddings from both sets. For each fact in BeliefBank, we take the maximum similarity with any fact from EntailmentBank, which should represent the existance of a unit of a similar knowledge between the two datasets. Given the volume of pairwise comparisons, we aggregate the results.

**Table 13: Fraction $k$ of facts in BeliefBank with cosine similarity above $t$ with any fact in EntailmentBank, for $t=\{0.80,0.85,0.90\}$.**
| $t$ | $k$ |
| --- | --- |
| 0.80 | 0.41 |
| 0.85 | 0.22 |
| 0.90 | 0.02 |

## Appendix D Prompts

### D.1 Prompts for Macaw-Large

We query the language model for a belief label about a statement in natural language. We adopt the format:

We fix <pos_label> = "Yes." and <neg_label> = "No.". We converted the (<subject>, <property>) tuple in natural language with a formatting function provided by Mitchell et al. [].

### D.2 Prompts for LoCo-LMs

We adopt two label sets to make the model less prompt sensitive:

Format 1: [true, false]

Format 2: [yes, no]

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: Please see "Weaknesses" section.
soundness: 3
strengths: The paper offers a novel approach by integrating neuro-symbolic reasoning into the fine-tuning of large language models (LLMs) to improve factuality and logical consistency. While existing approaches for enhancing consistency in LLMs often rely on external reasoning tools or extensive fine-tuning, this paper proposes a middle-ground solution: a neuro-symbolic-based loss function that promotes logical consistency by maximizing the probability of constraint satisfaction. This approach (LoCo-LMs) is grounded in weighted model counting and semantic loss, offering a flexible framework that applies consistently across various logical constraints, such as negation and implication.

The paper conducts extensive experiments to showcase LoCo-LMs' effectiveness over traditional approaches, demonstrating improvements in logical consistency, factuality, and transferability across different logical constraints and datasets. The method also proves efficient, achieving good performance even with limited training data.

By enhancing logical consistency without requiring external reasoning frameworks, the approach has important implications for deploying LLMs in tasks that demand reliable, logic-based reasoning. Its ability to generalize to unseen (yet semantically similar) facts presents a promising pathway for real-world applications where models need to work reliably with sparse data.
summary: The paper explores improving LLMs' factuality and logical consistency through neuro-symbolic reasoning. It introduces a neuro-symbolic loss function that is used to fine-tune LLMs on a given set of external facts and rules. Experiments show that this approach achieves improved consistency and generalizes more effectively to unseen yet similar constraints compared to baseline methods, including those that rely on external reasoning tools.
weaknesses: Evaluation scope:

The experiments primarily focus on logical constraints such as negation, implication, and reverse implication. While these are fundamental, they fall short of capturing the more complex reasoning scenarios often required in real-world applications. For instance, the paper could improve by incorporating evaluations on multi-hop reasoning tasks or exploring more sophisticated logical constraints.


Shift in language modeling distribution:

The authors assess possible shifts in the language modeling distribution by measuring changes in perplexity, yet their evaluation could be expanded. Adding downstream tasks (e.g, question answering, reading comprehension, mathematical reasoning, etc.) would allow to assess whether the proposed fine-tuning approach not only improves logical consistency but also maintains the language capabilities of the original model.


Robustness of the results:

The experiments reveal that fine-tuning LoCo-LMs improves generalization only within the same type of constraints, and it even hurts performance when the constraints differ between fine-tuning and testing (see Table 4). This limitation could be especially pronounced in smaller models, so testing on larger models could provide further insights. It would also be valuable to explore whether these performance gains also transfer to more capable models, such as comparing performance between LlaMa 2 and LLaMa 3, with and without LoCo-LMs.


Sensitivity to prompting:

The effectiveness of the approach appears to be sensitive to the specific prompt formats used during fine-tuning and evaluation. This suggests that the gains in consistency might be partially due to prompt selection rather than the model’s inherent logical coherence. Broader testing across diverse prompt templates would enhance the robustness and reproducibility of the results. Moreover, there are alternative prompting methods to elicit logical consistency, such as prompting the model to respond sequentially to a series of related questions, conditioned on previous answers.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: - For the pre-trained baseline models in Tables 1 and 2, do the scores improve with greedy decoding?
soundness: 4
strengths: - The idea of using a neuro-symbolic loss function to improve logical consistency and factuality in LLM responses is novel and interesting. The proposed loss function is generalizable, can be extended to complex logical constraints, and may prove useful in enhancing LLMs' reasoning capabilities.
- The detailed experimental results demonstrate the advantages of the proposed method over baselines, even on relatively small (5-10%) datasets.
summary: This paper introduces LoCo-LLM, a fine-tuning method for LLMs that leverages a neuro-symbolic inspired semantic loss function to enhance its factuality and logical consistency. The proposed semantic loss function is based on weighted model counting, with weights derived from the LLM’s probability estimates. LoCo-LLM employs sentential decision diagrams to efficiently compute this loss.

Detailed experiments compare LoCo-LLM with baselines that use external reasoners and traditional cross-entropy-based fine-tuning. Experimental results on the BeliefBank and EntailmentBank datasets show that the proposed framework outperforms baselines on metrics such as factuality and consistency.

The code to reproduce these results is provided as supplementary material and will be released on GitHub under a permissible license.
weaknesses: - Although the loss function is explained thoroughly, other components, such as circuits and sentential decision diagrams, are not discussed in detail. Including these details would improve the paper's readability.

- The experiments are conducted on datasets with outputs of fewer than 4 tokens, leaving it unclear how well the proposed method supports generating longer, factually and logically consistent responses.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 2
questions: * Line 192: the authors claim that they expect transfer from albatross to cockerel since they are similar - but there is no definition of what is similarity, and how should the model know when things are similar enough to conclude new facts about entities and when not. I assume this refers to some vague simlarity measure in the space of hidden representations, but this is still confusing.

* Line 469 - where are the resutls? are they in Table 3? the paper doesn't say

* What is the few-shot baselines precisely? what are the examples given and how are they helpful?
soundness: 3
strengths: The problem of improving logical consistency in language models is important. The approach is simple and does not require a lot of inference time compute since it is based on finetuning. The empirical results that show transfer and generalization beyond the training distribution are informative and interesting.
summary: This work proposes a fine tuning method for improving logical consistency in language models. Given a set of facts and a set of constraints the idea is to finetune the model to make sure that certain logical constraints are respected, typically implication and negation. The authors show that indeed finetuning allows to improve self consistency and that this transfers beyond the facts and constraints used for finetuning to other entities and and settings.
weaknesses: * Clarity: the paper can do a better job at explaining the details of its method. The authors spend two pages (section 2) on explaining logical constraints in a way that is too elaborate (for example, defining the xor operator in line 124, and defining implication in terms of negation and or in line 137) and unnecessary. On the other hand details on the actual method is limited (see questions below), specifically the paragraph in 232 and the precise process of how logical constraints are transformed into differentiable graphs are explained in a manner that is insufficient. The description of the experiments also mixes unimportant implementation details with more important details on the experimental setup which makes it hard to understand the details of the experiments and what can be concluded from them.

* Related to the above - Figure 1 takes a lot of real-estate but is not helpful. The only thing we see is that there is baseline that makes a mistake on 3 examples and the proposed model does not make the mistake. This does not say a lot on the method, or the aggregate results only we can learn about the types of logical constraints that will be used. This might be ok if the important parts of the paper were clear, but they are not sufficiently clear at this point.

* Key point that was unclear to me:  line 242 paragraph: I don’t understand if the method handles facts that can be inferred from \alpha and the KB but require more than one hop? When training the SL loss, are those considered? Say we have in the KB, “albatross is a bird”, “birds are an animal”, “albatross can fly”, “if an animal can fly then the animal can move”. Will the SL loss contain a term about whether albatrosses and whether they can move or not? Is this done implicitly somehow? Where do we do the inference of all potential things that can be inferred from the KB and the constraints and take those into account in the SL loss?

* More on clarity: in section 3, you define \mathcal{D}_c = \{alpha_1, \dots, \alpha_m\}. But the structure of \alpha is not clearl defined. It would be gold ot make this much clearer, it becomes clearer later as you read more, but should be explained better at this point.

* Clarity: z ~p_\theta(z) is confusing. Supposedly p_theta is the language model and it look like sampling from the unconditional distribution of text, but the text says something else, that it is sampling truth assignments conditioned on what appears in \alpha_i but this is not clear from the notation.

Another key point are some problems with clarity and worries about the experimental setup. 

* IIUC the only baseline that is reported that is not from the authors is ConCord for which two numbers exactly are reported and that's it. There is some reference to maieutic prompting but it is unclear if this should be another baseline or is too similar to ConCord. It is not clear if there are not reasonable baselines to compare other than that. There is reference to few-shot baselines, but it is not expalined what are the examples in the few-shot examples and how they are supposed to help, in fact in many cases results are worse for few-shot compared to zero-shot. Overall, the authors should make clear if there is no past work beyond ConCord and just finetuning on the KB (XENT) without using the constraints

* Second, for ConCord, it seems that the authors use ROBERTA-ANLI as an inference model. But for their LOCO method it seems like they are using hard constraints that are guaranteed to be true - if that's the case this is unfair towards ConCORD. Can the authors provide more details about how and why they outperform ConCord? Do the two methods use the same models and same constraints? Form the fact that the authors say that Concord requires ROBERTA-ANLI it sounds like the answer is "no" but would be good to understand better what's going on. Since we only have two numbers in the paper that are not baselines implemented by the authors it is important to understand the details in this setup.

To conclude, I found the overall premise of the paper interesting but the paper needs to be clearer both in terms of method and in terms of experimental results and how they relate to past work.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: The method relies on collecting the probabilities for specific tokens to estimate the yes/no probabilties.
How much is this going to be affected by the label bias of the LLMs?
https://openreview.net/forum?id=shr9PXz7T0
https://arxiv.org/pdf/2402.09910
soundness: 3
strengths: The paper is making advancements in neurosymbolic modelling.
It is certainly a nice achievent to not have to rely on an external solver and being able to push the knowledge into the main neural model.
summary: The paper describes an approach for improving the logical consistency and factuality of language models, using neuro-symbolic integration.
The paper starts with a list of facts and logical constraints. All the valid combinations of truth values for these facts are then iterated and used as targets during optimization.
The experiments evaluating the correctness and consistency of the learned facts show that this method outperforms vanilla models and a baseline using an external solver.
weaknesses: The evaluation is the weakpoint of the paper at the moment.

Macaw-Large, which is used for the main experiments, is quite old already (pre-LLM).
Even Llama-2 used in later experiments is much less capable on most tasks compared to the current Llama-3.2.
This raises questions how applicable the proposed methods are to the current generation of language models.

The main baseline is CONCORD, which is from 2019 and uses RoBERTa. 
The fact that the proposed system is able to outperform this baseline without using an external solver is great.
But there really should be some additional baselines with newer methods that also use model updating.
For example, there is a whole library of papers focussing on updating specific facts in language models using targeted fine-tuning.

The whole evaluation is performed on very artificial tasks. It would be very useful to see how these changes impact the model performance in practical applications.


Asking the LLM “Is an albatross not an organism?” is a very unnatural phrasing, whereas LMs are trained to predict natural continuations. I suspect that may be negatively affecting the performance for LMs.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Can LOCO-LMS be adapted for more complex, multi-level, or nonlinear logical reasoning scenarios?

2. How well does LOCO-LMS integrate with existing knowledge editing methods when it comes to incorporating new facts or updating knowledge bases?
soundness: 3
strengths: 1.  LOCO-LMS effectively improves the model's logical consistency, accommodating complex logical relationships such as positive implication, reverse implication, and negation. This alignment with common sense enhances the quality of responses generated by LLMs.

2.  By incorporating semantic loss, the method minimizes reliance on external reasoning tools, thereby lowering reasoning costs and increasing inference speed.
summary: This paper introduces LOCO-LMS, a fine-tuning method grounded in neural-symbolic reasoning, which significantly enhances the logical consistency and factuality of LLMs by integrating logical constraints as loss functions during training. Unlike traditional methods that rely on external reasoning tools, LOCO-LMS internalizes logical rules, allowing the model to reason independently and improving overall efficiency.
weaknesses: 1. The model assumes that facts are conditionally independent under a given model state, but in actual applications, there may be dependencies between facts, and this assumption may affect the consistency effect.

2. While it addresses factual inconsistencies in the Llama-7B model, I also concern that its efficiency and scalability may lag behind approaches based on RAG and knowledge editing.

3. LOCO-LMS is designed for specific tasks and fine-tuning, which limits its applicability to more complex reasoning tasks. Additionally, it may be vulnerable to attacks, such as just-in-time injection.

Meta-review: The paper proposes LoCo-LMs, a neuro-symbolic fine-tuning method using semantic loss to enhance LLMs’ logical consistency and factuality. It reduces reliance on external tools and shows improved consistency and generalization over baselines. Strengths include novelty, efficiency, and empirical validation. Weaknesses are limited evaluation scope, reliance on older models, and sparse comparisons to modern baselines. Decision: marginal acceptance.

Decision: Accept (Poster)