Logically Consistent Language Models via Neuro-Symbolic Integration
Authors: Diego Calanzone
Stefano Teso
Antonio Vergari
Venue: ICLR 2025 Poster (2025)
Keywords: probabilistic reasoning
logical consistency
LLMs
neuro-symbolic
semantic loss
Primary area: neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
TL;DR: We show that principled probabilistic reasoning can teach an LLM to be logically consistent with a set of external facts and rules (and itself), allowing to extrapolate to unseen but semantically similar factual knowledge.

Abstract: Current large language models (LLMs) are far from reliable: they are prone to generate non-factual information and, more crucially, to contradict themselves when prompted to reason about relations between real entities of the world. These problems are currently addressed with large scale fine-tuning or by delegating consistent reasoning to external tools. In this work, we strive for a middle ground and leverage a training objective based on a principled neuro-symbolic loss that teaches a LLM to be consistent with external knowledge in the form of a set of facts and rules. Fine-tuning with such a loss on a limited set of facts enables our LLMs to be more logically consistent than previous baselines for a given constraint. Our approach also allows to easily combine multiple logical constraints at once in a principled way, delivering LLMs that are more consistent w.r.t. all the selected rules. Moreover, our method allows LLMs to extrapolate to unseen but semantically similar factual knowledge, represented in unseen datasets, more systematically.

Published as a conference paper at ICLR 2025
LOGICALLY CONSISTENT LANGUAGE MODELS VIA NEURO-SYMBOLIC INTEGRATION
Diego Calanzone*
Stefano Teso
DISI, University of Trento
CIMeC & DISI, University of Trento stefano, www.eunitnnnnnnnmoo
di ego ca caallaazzoeeeedeeeen .nniidettiiiiiitiii unitn.i.
Antonio Vergari
School of Informatics, University of Edinburgh avergari@ed.co
ABSTRACT
Current large language models (LLMs) are far from reliable: they are prone to generating non-factual information and, more crucially, contradicting them- selves when prompted to reason about relations between entities of the world. These problems are currently addressed with large scale fine-tuning or by dele- gating reasoning to external tools. In this work, we strive for middle ground and introduce a loss based on neuro-symbolic reasoning that teaches an L.LM to be logically consistent with an external set of facts and rules and improves self-consistency even when the LLM fine-tuned on limited set of facts. Our approach also allows to easily combine multiple logical constraints at once in a principled way, delivering LLMs that are more consistent w.r.t all constraints and improve over several baselines w.r.. given constraint. Moreover, method allows LLMs to extrapolate to unseen but semantically similar factual knowledge, represented in unseen datasets, at https:: //g thub. com ddi dacus/ occo- 1mm.
systematically. Code available
INTRODUCTION
Developing reliable large language models (LLMs) and safely deploying them is more and more erucial, particularly when they are used as sources of knowledge (Petroni et al., 2019; al., 2023; Bommasani et al., 2021; Andriopoulos & Pouwelse, 2023). To do so, one would need LLMs to be fuctual (Wadden et al., 2020), i.e., agreeing on single facts that appear and logically consistent (Li et al., 2019; Mitchell et al., 2022), i.c., being able not to contradict themselves or a KB when prompted to perform complex reasoning. that training on large datasets for question answering (QA) (Tafjord & Clark, 2021) alone cannot meet these desiderata (Evans et al., 2021; Lin et al., 2021; Liu et al., 2023; Elazar et al., 2021).
knowledge base (KB),
has been abundantly shown
Factuality and consistency are intimately related. Enforcing factuality alone generally boils down to fine-tuning an LLM on a large KB of atomic facts (Kassner et al., 2021). When predicting the truth values of these facts, several works try enforce the simplest form of consistency: that the probability of true fact shall be one minus the probability of its negation (Burns et al., 2022). More sophisticated heuristics are possible, c.g., contrastive fine-tuning on a large QA dataset by jointly optimizing for truthfulness of model answers (Liu et al., 2023). All these approaches require large KBs and more crucially are tailored towards specific logical constraints.
When it comes to self- celf-connistency w.r.t. more complex reasoning scenarios, e.g., ensuring that LLMs can perform modus ponens reasoning without contradicting themselves (Tafjord al., 2022; Mitchell et al., 2022), one line of research focuses on employing external reasoning tools such as MAX-SAT solvers (Battiti, 2009) inference time (Mitchell al., 2022; Jung et al., 2022; Kassner et al., 2023). However, these approaches depend on the constant availability of reasoner (and
correspondence to,
shared supervision

---

Published as a conference paper at ICLR 2025
P(Z))
Facts
Constraints
"abutron had'- 1o sabuttue T
LoCo-LM
Semantic Loss
Knowlodge
(2 + 2pd
Compilation
Ep
- -pp)
VVC
Figure I: Pipeline of our Logically Consistent (LoCo) LLMs. LoCo-LMs are made factual and logically Jeonsiscont by fine-tuning base LLM according to knowledge base of facts and rules (Section 3). The constraints a; which can be arbitrary propositional logic formulas are compiled into circuit (just once) and then used to encourage the model to allocate non-zero prob- ability only to factual and consistent facts via a semantic loss (Xu et al., 2018), see Equation (SL).
sometimes also of a natural language inference model (Mitchell et al., 2022)) which can increase the cost of inference for every reasoning step. At the same time, training the LLM to reason is not possible or hindered by the hardness of backpropagating through the solver (Pogancic et al., 2020).
In this work, show how to improve factuality and self-consistency of LLMs without external components by leveraging recent advancements in neuro-symbolic (NeSy) reasoning (De Raedt al., 2021). This is done by turning complex reasoning tasks into logical constraints that can be compiled into computational graphs (Vergari et al., 2021). Specifically, we fine-tune an LLM by a principled objective: maximising the exact probability of constraint hold, which goes under the name of weighted model counting (Chavira & Darwiche, 2008) in probabilistic reasoning or seman- tic loss (Xu et al., 2018) when used as a regularizer in deep learning (Zhang al., 2023; van Kricken et al., 2024). This in turn encourages the LLM to perform exact probabilistic reasoning at training time by maximising the probability of beliefs that comply with the provided set of constraints.
We empirically show how given incomplete factual knowledge, e.g., by providing only limited number of known facts, the LLM can learn truth beliefs for new facts while keeping logical consis- tency w.r.t. prior knowledge. Moreover, our approach is agnostic to the logical constraints consid- ered and can deliver single training objective that can improve multiple consistency scores at once. In our experiments, with a single offline training session, LLMs trained with our objective outper- form models relying on external solvers, and are more factual and logically consistent in low-data regimes when compared to standard supervised fine-tuning over KBs of facts.
and principled Summarizing, we: i) introduce Logically-Consisse strategy designed to improve factuality and (self- Jeonsistency of LLMs based on probabilistic NeSy reasoning (Section 3). and ii) we rigorously evaluate the ability of LoCo-LMS to improve self-consistency w.r.t. several reasoning scenarios when fine-tuned for certain constraints and evaluated over others - without hurting fluency (Section 5).
Contributions.
LLMs (LoCo-LMs), a novel
2 CONSISTENCY THROUGH THE LENSES OF PROBABILISTIC REASONING
We formalize the different reasoning scenarios we would like an LLM to be self-))coniistett with, and highlight the shortcomings of commonly used LLMS when prompted to reason in this way.
Factuality. We view a pre-trained LLM as collection of truth beliefs about facts over which can reason. The simplest reasoning task is factual reasoning, i.e., determining the veridicity of fact. For example, consider the fact f in textual form "an albatross is a bird". can be commonly encoded in knowledge bases (KBs) such BeliefBank (Kassner al., 202 2) as (subject-relation, property) pair, for instance, (albatross-is is, bird). To inspect whether an LLM believes a fact to be true. we can prompt it with question like "Is an albatross bird?", the LLM can supply binary prediction of the form "Yes"/"No" or "Frrue""False", encoding its belief that the fact f holds not. Therefore, given LLM modeling a parameterized distribution Po. we can consider the probability of generating a token I, encoding a binary answer, according to Pa, after observing the token sequence 21, ...... - encoding the question about the fact, to be the probability of the LLM believing that the truth value 2f of fact ff is either true (T) or false (L). That is, for true facts,
(z = T) = (a
Ltrue F1. Fz-1 "Is an albatross a bird?")
(1)
We note that such an answer can be highly dependent the format of the prompt. For this reason, in our experiments we several prompts, whose format is detailed in Scction 5.
2

---

Published as a conference paper at ICLR 2025
where frrue is affirmative token, e.g., among "yes", "true", "Y", "T", etc. Analogously, can compute po(zy = 1) by checking if the LLM answers a token Lasse is "no". "false", "N", "F", ctc. To determine the model's belief, we query 2 the most likely next token it and check whether falls in frue or fatse, and set it to "undetermined" if it falls into neither.
Given an external KB, we say an LLM is factually consistent, or simply factual, w.r.t. a fact f in the KB with truth value 2j. if its answer (mapped to truth assignment as described above) matches 27, and factually inconsistent otherwise." This perspective leads to interpreting factual reasoning as a binary question answering (QA) task (Burns et al., 2022; Kassner et al., 2021; Mitchell al,, 2022). From Equation (I), one can see that a simple strategy to make an LLM more factual is that of minimizing the cross-entropy (XENT) of po over an external KB containing training questions with ground truth answers. We compare against it in our experiments (Section 5).
Negation consistency. While effective for many QA scenarios (Liu et al., 2023; Tian et al., 2023), increasing factual consistency by XENT minimization does not prevent the LLM from being log- ically inconsistent under other simple constraints, e.g., contradiction (Kassner & Schütze, 2019; Cohen et al., 2023; Jang & Lukasiewicz, 2023). Given a textual representation for a fact f. e.g. "an albatross is bird", and another one f encoding its negation, e.g.. "an albatross is not bird", say negation self-connistency holds if
2f + Fj
(2z (2yA-zz) (-23
(NEG)
where denotes the logical operator XOR. In other words, we would like an LLM to consistently
answer either affirmatively or negatively when asked about the truth of a statement and its negation.
Negation consistency is very challenging for LLMs (Kassner & Schütze, 2019; Elazar et al., 2021; Jang & Lukasiewicz, 2023). For example, in our experiments LLaMa-2 70b (Touvron et al., 2023) answers "true" to both questions "Is an albatross an organism?" and "Is an albatross not an organ- ism?". From a probabilistic perspective, simple sufficient condition for negation consistency is that po(2) = T = - pp = T). This hard to systematically guaranteed and in practice has been addressed by applying ad-hoc heuristics during fine-tuning (Burns al., 2022), which however cannot be exploited to enforce consistency to other constraints, such as implication, discussed next.
Implication consistency. Given two textual representations of facts fi (antecedent, c.g., "an alba- tross is bird") and f2 (consequent, "an albatross is an animal") we say that the first implies the second if it holds that
(IMP) As with factuality, consistency (resp. celf-consistency) holds if the answers of the LLM to a prompt satisfy the truth values according with the above implication and an external KB (resp. the inner beliefs of the LLM). Furthermore, letting 27, be the truth value of Si recorded in the KB, we can define a factual variant of the implication that restricts the constraint to take 27, into account, that is, when the LLM is prompted about f2, it should derive its truth value žy₃ according to
(z - 2)2) - (-2,, V 2222).
(2 2 A (zy 2/2))
(F-IMP)
This can be seen relaxation of classical modus ponens reasoning (Robinson & Voronkov, 2001). While simpler to capture from text corpora, implication consistency can still be challenging for LLMs (Kassner
al., 2023; Yang al., 2024). For example, given the rule fi - -/2, where fi: "an albatross is an animal" and f2: "an albatross is a virus", we wish the LLM to answer with "Yes" and "No" respectively, which maps to the truth assignment 2f1 T, Sf, 1. LLaMa-2 70b violates the provided rule with the inconsistent belief, 2/2 = 1, i.c. "an albatross is a virus" is labeled as a true statement.
Reverse implication consistency. Equation (IMP) is logically equivalent to -2/2 >25 never- theless an LLM that logically consistent w.r.t. the implication of f1 over fz it might not necessarily be consistent w.r.t. the implication of fa over Fr. representing the negation of f2 and fi respectively.
We keep a default temperature L.0. Dropout is disabled to generate outputs systematically. Simillaryy an LLM is factually self consisenntt w.r.t f if it answers in the same logically consistent way (e.g., 24 is always 1) when asked to answer the same or semantically equivalent prompts different several times. Since this is harder to measure- as it strongly depends on the sampling strategy we focus on factual consistency only.
3

---

Published as a conference paper at ICLR 2025
For example, while 2 70b is logically consistent w.r.t. 2/1 2/2 with fi "an albatross is an organism", f2 "an albatross is a living thing". it violates 3/2 - // as it classifies 27," "an alba- tross is not a living thing" as false but 2 "an albatross is not an organism" as true. Furthermore, an LLM that is logically consistent w.r.t. reverse implication and factual w.r.t. a KB should satisfy
- A (2%, +27,)
-
(REV-F-IMP)
where -2 indicates the opposite of the truth value stored in the KB for /2. This factual reverse implication scenario can be thought as a relaxation of modus tollens (Robinson & Voronkov, 2001).
More complex constraints. As just discussed, constraints such as negation, logical implication and reverse implication already pose challenges to state-of-the-aan LLMs in terms of consistency. While will focus on the Llama 2 LLM family in this work, similar shortcomings have been highlighted for even larger models such as ChatGPT and GPT-4 (Jang & Lukasiewicz, 2023). Nevertheless, they constitute only small fraction of the possible real-world reasoning scenarios LLMs can be asked to deal with. Consider for example the following textual representations of facts, as extracted from Entailmen(Bank (Dalvi al., 2022): f1 "melting is kind of phase change". f2 : "the ice melts", fa : "the ice undergoes a phase change". f4 "phase changes do not change mass", fs "the mass of the ice will not change". They obey the following logical constraint
(2 22 22,)) A 2f, + 2/s-
(2)
In the next section, we will introduce our general framework that can improve logical consistency of fine-tunable LLMs w.r.t. any logical constraint expressible in propositional logic.
LOGICALLY-CONSISTENN LLMS VIA NESY INTEGRATION
We assume we are given a KB comprising set of textual statements and associated truth values .... (Jn: (U,,,,,,)...., computer is a bird" (false), and a set of logical constraints Dc = {ai....., 2m } - e.g., implications, negations or more complex constraints like those defined in Section 2 over the facts in Dy.
Dr = [(/1,
=},)}, }} encoding simple facts such as "an albatross is bird" (true) and "a
Given a pre-trained LLM encoding a distribution pe over tokens, our objective is to fine-tune it to be more consistent w.r.t. Dr, De and itself. As important side benefit, expect the fine-tuned I.LM to generalize to and be consistent with - the truth values of unseen facts Jn+1,Jn+2.. that can be either logically inferred by applying the constraints in De DD (e.g., by applying modus ponens) or that are semantically similar to facts in Dr. E.g., since albatross and cockerel are semantically similar for LLM, we expect an LLM consistent with the constraint "an albatross is a bird' + "an albatross can fly" to correctly infer that "a cockerel can fly" too.
principled probabilistic approach do so is to encourage the LLM pe to allocate all probability mass to configurations of truth values that are consistent with the constraints a, E De, for instance by penalizing proportionally to the probability allocates to inconsistent truth values for all facts in the KB. For every a; the total probability allocated to the consistent configurations is
Pr(as) := (x) -= as)) = - 2F0 Po(z)
(3)
where z is a vector containing the truth assignments 21,..., 2, of all the K facts appearing in the constraint ag, and z - a₁ indicates that the assignment satisfies the constraint, and the individual probabilities pe (z) are obtained using Equation (I), For example, consider two facts fi : "a daffodil flower" and f2 : "a daffodil mortal" and the constraint a' 2f1 - 2f2 stating that being a flower entails that the daffodil is mortal. Then, all the configurations of z = (2)) would satisfy a' with the exception of (T, 1) which clearly violates computing the weighted model count (WMC) (Chavira & Darwiche, 2008; van Krieken et al., 2024) of a logical formula 0g where the weights associated to each model (a satisfying assignment to the formula) are given by the probabilities encoded by the LLM.
Equation (3) is special instantiation of
Furthermore, we can rewrite such probabilities pe (z(z) as the product the probabilities of the truth values of each fact, noting that for many LLM architectures they are conditionally independent given the embeddings at the last layer. By taking the logarithm and reversing it into a minimization problem, obtain the semantic loss (SL) (Xu et al., 2018) objective that our LoCo- minimize:
L( po) - -log] Σ x
II
po(2))) II
(1 (--o(2z5))
(SL.
jut 253
4

---

Published as a conference paper at ICLR 2025
where j F 2 (resp. j ::20 z F= ) indicates that the j-th fact in a; is associated T (resp.
L.. Consider the implication constraint a' as defined before for encoding that a daffodil is mortal for being a flower. Its satisfying assignments are }= a' (T, (L, Then, the summation in Equation (SI.) amounts to computing:
po(2n, = (2 = +) (1111ppp22, = T)) = T) (1 polpp = ()) = T)))
where we can obtain the individual probabilities of facts being true directly by reading off the like- lihood of utterances produced by the LLM, that is:
T1 Ft-1 "Is daffodil a flower?") po(2f2 = T) = pa (xt = frue ,
po(2/1 T) po (2 Liue
It-1 "Is a daffodil a mortal?").
In the case of a constraint such as Equation (F-IMP), the inner summation of the SL. would re- duce to a single configuration = (T, T) when 27, T, which can be interpreted as special kind of tross-entropy computed only on pairs of facts considered to be jointly true in the KB, and to the set {(L. T). -1.-)} when 27, L. Note that Equation (SL) is agnostic the kind of logical constraint involved, and therefore makes our approach general enough to tackle several set- tings where consistency-preserving solutions have been devised for specific constraints (Burns et al., 2022; Kassner et al., 2023; Mitchell al., 2022).
Crucially, the procedure to compute the models of a logical constraint can be automated. Now, naively computing the sum in Equation (SL) would require exponential time w.r.t. the number of possible facts in z. In fact, computing the WMC of a logical formula is a #P-hard problem in general (Chavira Darwiche, 2008). However, thanks to recent advancements in neuro-symbolic reasoning, we can compute that probability and differentiate through it efficiently (Darwiche, 2011; Xu et al., 2018; Ahmed et al., 2022a). Specifically, we rely on modern compilers that translate a logical formula ag into compact and differential computational graphs called circuits (Darwiche, 2003; Vergari et al., 2019), such as sentential decision diagrams (Darwiche, 2011; Oztok & Darwiche, 2015; Choi & Darwiche, 2013), ef. Appendix A for details. In our scenario, compilation is extremely fast taking only 2.5 milliseconds to compile a single logical formula and compute the loss on BeliefBank (Section 5).
To recap (cf. Figure 1), during training we loop over every constraint in o, E DDc, prompt the LLM to gather the probabilities of every fact participating in a; to be true and plug them in our only loss, as described in Equation (SL). Then, we backpropagate as to fine-tune (some of) the parameters 0 of the LLM, by using LoRA (Hu et al., 2021) and quantization (Dettmers al., 2023) necessary. This simple and principled recipe is able to scale well and is extremely effective at improving logical consistency on a number of well-known benchmarks, as discussed in Section 5.
4 RELATED WORK
LLMs and factual reasoning. LLMs are increasingly being employed as implict KBs (Petroni al., 2019; AlKhamissi al., 2022), however ensuring they are factually consistent is still an open challenge (Wang et al., 2023; Augenstein et al., 2023). A number of works augment LLMs with external KBs, especially in the context of QA, and with the primary aim of improving answer factuality (Kassner et al., 2023; Mitchell et al., 2022: Li et al., 2024b). A popular approach to do so is retrieval augmented generation (Lewis et al., 2020; Li et al., 2024a), which however not yet suited for more complex reasoning scenarios. Alternatively, external KBs have been used to improve reasoning, e.g., via prompt learning (Palagin al., 2023) or ex xx-post model editing (Shi al., 2023). However, current knowledge editing methods, including supervised fine-tuning, do not guarantee the propagation of factuality between units of knowledge related by logical relations (Cohen al., 2023; Akyürek et al., 2024). Mitigating hallucinations in LLMs (Andriopoulos & Pouwelse, 2023; Rawte et al., 2023) is related to enforcing factuality, but as generated inconsistencies might not map to single entry in a KB, they are harder to detect and prevent (Hong al., 2024).
More complex reasoning with LLMs. Much less attention has been posed to composite forms of reasoning, e.g., combining modus ponens and consistent negation. Even when this is done, reasoning is generally cast as a QA task, where an LLM has to predict the satisfiability of logical formulas of different complexities, as in benchmarks such Simpleloggie (Zhang et al., 2022) LogicBench
5

---

Published as a conference paper at ICLR 2025
(Parmar et al., 2023). Implication or entailment (MacCartney, 2009; Evans al., 2018) are also usually cast as a QA prediction task (Raj et al., 2023). BeliefBank (Kassner et al., 2021) provides collections of implication constraints test this, while more sophisticated benchmarks such En- tailmentBank (Dalvi et al., 2022) include more complex implications, e.g. trees of natural language statements. Shortcomings in consistent reasoning have been recently highlighted for larger LLMS such as ChatGPT and GPT-4 variants (Jang & Lukasiewicz, 2023), which are however harder to fine-tune efficiently. Other works (Berglund et al., 2023) highlighted how (even large) LLMs suffer from not being able to recognize the logical equivalence of "A is-a B" and "B is-a A" relationships.
For complex reasoning scenarios, logical consistency can be improved in a number of ways, the most successful of which involves external tools, such as MaxSAT solvers, which flip the predictions of an L.M to be (approximately) consistent with a set of related questions, as done by ConCoRD (Mitchell et al., 2022) and maieutic prompting (Jung et al., 2022). Analogously, self-consistency can be ameliorated by first constructing belief graph a factor graph relating the beliefs of an LLM fine-tuned on implications such Entailer (Tafjord et al., 2022) over which a MaxSAT solver is applied (Kassner et al., 2023). Higher level constraints can also checked and enforced with external verifiers (Wang et al., 2024). Differently from LoCo-LMs, backpropagating through these tools is hard (Pogančić et al., 2019). Moreover, while they can guarantee self celf-consittency among facts within every call to solver, this cannot be done for the same facts across different calls.
Semantic loss & other NeSy approaches There a vast literature on NeSy integration meth- ods (De Raedt al., 2019; 2021), most of which are used for enforcing constraint tabular data (Giunchiglia & Lukasiewicz, 2020), image data (Xu et al., 2018; Shindo et al., 2021; Ahmed et al., 2022a) and more recently video recognition (Giunchiglia al., 2023) with the purpose of building trustworthy predictors. Several variants of the semantic loss (Xu et al., 2018; Ahmed et al., 2022b; 2024) and neural weighted model counting (van Krieken et al., 2024) have been proposed. Closer to our work, Zhang et al. (2023) applied a semantic loss to instill first-order rule constraints in the em- bedding space of entities in encoder-only models to reason on the CLUTTR benchmark (Sinha al., 2019), comprising semi-synthetic stories involving hypothetical families. Richardson & Wijnholds (2024) propose to combine LLMs and a semantic loss for consistency analogous to ours. Faghihi et al. (2023) approximate semantic loss via sampling to improve consistency of only implications for small BERT-like models. We do not need approximations as we rely on exact computations via compilation while scaling to larger constraints and combining different constraints together. Fuzzy logic (van Krieken et al., 2022) can be used to distill regularizers that can promote consistency (Li al., 2019). Differently from our probabilistic logic approach however, they are syntax-dependent, i.e, rewriting a constraint into a logically equivalent one would yield a different penalty term and can greatly influence optimization (Di Liello et al., 2020).
5 EXPERIMENTS
Can LoCo-LMS achieve comparable superior consistency to methods using extemna reasoners using less training data? RQ2: Can LoCo-LMs retain good consistency to unseen types of constraint at training time? How much does training on all the constraints jointly improve consistency overall? RQ3: Can LoCo-LMs transfer consistent knowledge to domains out of the training distribution?
We aim to answer the following research questions: RQI:
5.1 RQI: : How DO LoCo-LMS PERFORM COMPARED TO EXTERNAL SOLVERS?
We reproduce the experimental setting of Mitchell al. (2022) to compare against ConCoRD, symbolic layer that uses MaxSAT solver to impose self-consistency for implication ex-post. Maieutic prompting employs essentially the same strategy (Jung et al., 2022).
Data. We train LoCo-LMs on the BeliefBank (Kassner et al., 2021). We use the three splits as in Mitchell et al. (Mitchell et al., 2022): a "calibration" set of 1, 072 annotated facts about entities of the form (subject, property, true/false) used for training. a "silver" set of 12, 636 facts about 85 entities used for evaluation, and a set of 2224 valid abstract logical implications. We generate ground implication rules (Dc) by looking up the subjects of all facts in the training set: if the antecedent or the consequent fact of the general constraint is known for that subject, we add the subject ground implication constraint to the dataset. Appendix 1. details the whole process.

---

Published as a conference paper at ICLR 2025
Table 1: LoCo-LMs achieve better logical self-consistency and factuality than ConCoRD (Mitchell et al., 2022) as mea- sured via Equation (4) and Fi
METHOD
TRAIN SUBSET
ANT F1
CON F1
IMP
CONCORD
0.91
0.91
MACAW-LARCE
0.52
0.90
0.81
0.83
MACAW+XENT
TI
0.13
0.01
0.03
0.72
LoCO-MACAW
T1
0.79
0,98
0.96
0.99
scores when fine-tuned only on TI facts only and boost perfor- mance in the presence of small fraction of T1+T2 facts (5-10%).
MACAW+XENT
T1+T2 (5%)
0.23
0.78
0.72
0.82
LoCo MACAW
(5%)
0.67
0.83
0.81
0.92
MACAW+XENT
T1+T2 (10%)
0.55
0.97
0.91
0.90
LoCo-MACAW MACAW
T1+T2 (10%)
0.45
0.97
0.89
0.93
MACAW+XENT LoCO.MACAW
T1+T2 (75%)
0.85
0.99
0.97
0.98
A similar trend is visible train- ing data (Appendix B.1.1).
T1+T2 (75%)
0.79
0.99
0.95
0.98
To measure generalization across entities, we generate two controlled splits of the training cali- bration set: T/ facts, appearing either as antecedents or consequents in the constraints; 72 facts, appearing exclusively as consequents. The goal is to correctly guess the consequents by seeing only the antecedents and the constraints. We subsequently test the effects of pure supervised fine-tuning on a portion of random facts from the whole calibration set (T1+12).
Models. As in Mitchell et al. (2022), we use Macaw-Large (Tafjord & Clark, 2021) (770M pa- language model capable of multi-angle QA with fixed prompt templates. We keep the same prompts used for Macaw, reported in Appendix F.1. At test time, we verify the validity of the answer format and consider any invalid or negative response as a belief with label "false". We adopt a similar set of hyperparameters as for Macaw (Tafjord & Clark, 2021): we fine-tune our models for epochs with a learning rate fixed to y = 10 batch size 4 with gradient accumulation (64/I) 16 steps), one Vidia A30 24GB GPU. We use AdamW (Loshchilov & Hutter, 2016) 6) as optimizer with a default weight decay X = 10-2
rameters), a
Competitors and Metrics. We compare ConCoRD applied Macaw-Large, using RoBERTa- ANLI (Liu et al., 2019) for relationship inference, versus a pre-trained Macaw-Large model from Tafjord & Clark (2021) as zero-shot baseline and our LoCo version of it (LoCo-Macaw) We eval- uate our models for factuality and implication self-consistency. We measure the former with the Fi score to account for the unbalance between false and true facts (Kassner et al., 2021). Factuality is measured on the two splits (antecedents and consequents) and the complete facts set (Tot) for both calibration and silver splits. For implication self-consistency,, sometimes named just "consistency" (Li et al., 2019), we query beliefs from LLMs about the complete facts set and count the fraction of violated constraints in Diss according to the implication rule (IMP), that is, when a true antecedent for the model implies false consequent, to then compute:
(4) Results. Table 1 reports all metrics for all models. We firstly observe a net improvement in both factuality and logical consistency with our LoCo-LMs, compared to pre-trained Marge and the ConCoRD variant. Standard supervised fine-tuning with the XENT loss on antecedent facts insufficient: due to class imbalance between true facts (~ 10%) and false facts (~ 90%), the model tends to label any statement as "false". This is accentuated in the training distribution (see Appendix 1.1. Assuming the language model can access to a portion of consequent facts, LoCo-LMs still yields better logical consistency and factuality for unseen consequents in low-data regimes (e.g., 5-10% of the T1+12 dataset) compared to canonical supervised fine-tuning. When they are allowed to see more data (e.g., 75% of the T1+T2 dataset), traditionally fine- finettuned models can "cheat" and directly learn about the consequents (somehow equivalent to memorizing a single row of the truth table). In this scenario, LOCO-LMS achieve comparable logical self-consistency and factuality over consequents, but less on the antecedents.
- 1a;; = (z₁ - 2x): 2, =, 2 = 1H [{:; - (2j + 2x) :2 = TH.
In conclusion, we observe our fine-tuning method allows Macaw-large he more logically self- consistent than with an external solver. We conjecture that this is possible thanks to the high semantic similarity between facts in the train and test splits (Appendix E. In terms of inference speed, our LoCo-LMS take less time that querying the same base model and an additional rea- soner', at the cost of one-time training step that can be amortized. Moreover, our semantic loss
*On BeliefBank, LoCo-LMs take 2405.28s at test time, compared ConCoRD, 3669.33s.
7

---

Published as a conference paper at ICLR 2025
is more sample-efficient than XENT fine-tuning achieve higher logical consistency especially with small portions of ground-truth data. We point out that our LoCo-LMs can be combined with external solvers at inference time, improving even more JeeffJJconssteneyyy
5.2 RQ2: How DO LoCO-LMS DEAL WITH DIFFERENT LOGICAL CONSTRAINTS?
Setting. As in Section 5.1, we use BeliefBank to train and evaluate LoCo- LMs on different types of logical rules. We use 90% and 10% of TI facts for training and validation, respectively; 72 facts for testing. We employ two sets of labels to make our models less sensitive to the prompt format; at training time, one format chosen with 50% chance for each batch; details Appendix F.2.
Models. To train larger language models, we choose the LLaMa-2 (Touvron et al., 2023) family of decoder-only models, widely adopted in literature for its performance across a variety of tasks and domains. We consider three baselines: the available pre-trained 7b and 70b models, 4-bit Nor- malFloat quantized (Dettmers et al., 2023), with greedy sampling strategy, temperature t = 1.0 and dropout disabled; we also perform supervised fine-tuning of the 7b model (4-bit. with LoRA (Hu et al., 2021)) on the ground truth TI+T2 facts set, namely "LLaMa-2-7b XENT". We derive our LoCo-LMS fine-tuning with our proposed method LLLMa-2 7b, with 4-bit quantization and L. We limit the generation to 4 tokens following the input. We adopt similar set of hyperparameters LoRA: we fine-tune our models for 5 epochs keeping the learning rate fixed to y = 3- 10 batch size 64, on 1 Vidia A100-40GB GPU. We use AdamW (Loshchilov & Hutter, 2016) as opti- mizer with a default weight decay À 10-2 We use the SL to finetune three LoCo-LM variants: for negation (NEG), factual implication consistency (F-IMP) and their conjunction, i.., given an implication fi + f2 we provide the SL with the constraint:
(z ① 27, A (z 2 A (En 2/22) A (2 ① 27.))
(SUPER)
where Ji and f2 encode the textual negation of fi and f2, generated via ConCoRD's templates. We compare against orthogonal baselines such chain-of-thought (CoT) and zero- and few-show prompting, which we note can be combined to LoCo-LMs.
Metrics. We fine-tune NEG, F-IMP or SUPER and evaluate on all constraints. Specifically, measure the implication self-consistency, cf. Equation (4), as well as the implication consistency:
1 1- If, (z, + 2x) 2 T, 2 131 Ha; (z) - 2x) : 23 = =TH
(5)
where 2/ is the ground truth value of fact. We also measure reverse implication consistency
Ho: = (3z - -3) = ,, = TH
Max = (zz + 25) - =2
(6)
and the reverse implication self-consistency variant:
[a; = - 2 1, =,,
Ma; = (Σz + 2 =
where "े and 2y are the truth values of the textual negations of facts k and according to the model. For negation self-consistency we compute
1 - Ha, = (z₁ @ 27) :2 =2]]/ la; ; = (z, + 2j)1.
As in Section 5.1, we measure factuality (FAC) as the F1 score on a set of ground truth facts. Finally, account for possible shifts in the language modeling distribution by computing its perplexity (PPL) on WikiText (Merity et al., 2016), formatted as a single token sequence.
Results. In Table 2, we first observe an overall boost in factuality for all LoCo-LMs over the 7b basclines. Compatibly with Table 1, supervised fine-tuning is not sufficient to improve logi- cal consistency significantly and outperforms baselines such as CoT and few-shot prompting. Our LoCo-LM trained exclusively on IMP constraints performs best in factuality and implication con- sistency; at the same time, scores on negation consistency and reverse implication are lower. We remark this is expected and common when doing multi-objective optimization. Note that, however, the great majority are cases of positive transfer, i.e., optimizing for one constraint also benefits oth- ers. For example, optimizing for NEG improves all columns of Table 2 wrt the baseline (C-FAC: + +19%, C-IMP: +20%. C-REV: +42%, SC-REV: +35%) but cels-connist IMP, and optimizing F- IMP only degrades self-consistency REV and NEG (C-FAC: +74% C-REV: +8%), it rightly does
8

---

Published as a conference paper at ICLR 2025
Table LoCo-LMS achieve higher Jsens-)consisteney than off-the-shelf baselines and models trained with supervised fine-tuning (+XENT) on the BeliefBank test split. Scores are averaged across four sets of prompts and truth labels, for which results are reported Tables 13 and 18.
CONSISTENCY
SELP-CONSISTENCY
MODEL
FAC
REV
NEG
IMP
REV
AVG
LLAMA-2-70 ZERO SHOT
52.30
0.27
0.45
0.47
0.34
0.45
0.48
0.41
LLAMA-2-7B FEW SHOT
52.30
0.53
0.70
0.40
0.35
0.47
0.39
0.48
LLAMA-2-78 COT
52.30
0.52
0.64
0.67
0.40
0.64
0.67
0.59
LLAMX-2-70B ZERO
44.90
0.47
0.81
0.13
0.91
0.55
LLAMA-2-7B+ + XENT
T1+12
116.85
0.21
0.42
0.30
0.10
0.76
0.30
0.35
LoCo-LLAMM--2-1 (NEG)
62.21
LoCo-LLAMA-2-7a (F-IMP)
TI
67.15
0.98
0.98
0.52
0.01
0.99
0.52
0.66
LoCo-LLAMa-2-78 (SUPER)
62.23
0.75
0.79
0.84
0.82
0.76
0.82
0.80
LLAMA-3, 88 ZERO SHOT
78.22
0.45
0.58
0.54
0.42
0.54
0.54
0.52
LLAMx-3.1-80 FLW SHOT
78.22
0.41
0.51
0.36
0.51
0.47
LoCo-LLAMA--31-88 (SUPER)
T1
78.22
0.73
0.80
0.80
0.89
0.76
0.80
0.78
Table LoCo-LMS improve Logical consistency bass-knowledge transfer as measured on ConceptNet when trained on high-level class properties for 10 epochs.
SELI-CONSISTENCY
MODEL
FAC
IMP
REV
NEG
IMP
REV
AVG
ALAAMA--7B-ZEER SHOT
0.21
0.41
0.84
0.26
0.63
0.84
0.53
LLAMx-2277-FFNS SHOT
0.47
0.69
0.20
0.10
0.31
0.20
0.33
LoCo-LLAMA-2-71 (SUPER)
0.72
0.80
0.59
0.42
0.80
0.59
0.65
not consider negation, while delivering much better performance over all cases than using XENT. Finally, fine-tuning moderately-sized LoCo-LM on the combination of both constraints (Su- PER), yields average the most consistent language model, which average surpasses Llama 2 70B, a much larger model. Overall, fine-tuning with our method does not impact nega- tively fluency, measured by perplexity.
5.3 RQ2: CLASS KNOWLEDGE-TRANE
IN CONCEPTNET
Setting We further investigate how our fine-tuning method affects the internal knowledge of an LLM by querying specific properties across hierarchies of entities. For this purpose, the Con- ceptNet dataset (Speer et al., 2018b), is a rich source of knowledge about entity properties and relationships. We thus construct train split by selecting 6 high level entities ([human, dog, cat, manmal, car, boat] and properties of type [CapableOf, AtLoat i TSA], spawning 1.227 constraints with the format e.g. (dog, IsA, mammall ) (dog, CapableOf, mother of a puppy); similarly to experiments with Belic/Bank, we fine-tune LLaMa 7b with our objective on the conjunction of all the considered logical constraints (SUPER) with the same hyperparameters.
Metrics. We construct test set with 432 sub-entities deriving from the entities considered in the train set: we consider only ground truth facts that are shared with the parent class; the underling assumption is that the LM knows the relationship (sub-class, IsA, class) and thus some properties should be inherited. We thus look for gains in class-knowledge transfer by comparing LoCo-LMS with the pre-trained baseline logical consistency on sub-entity properties, which are sparse and scarce in the LM distribution. To tackle class imbalance, augment the training set with properties that entities don't have, e.g. (human, CapableOf, live underwater).
Table 3 indicates consistent gains from fine-tuning method in factuality, implication (self and objective) consistency and negation self consistency; on average, LoCo-LMS surpass base models with zero or 2 examples of factuality queries, e.g. "Fact: the earth is round. Label: true". Increased factuality in LoCo-LMs directly reflects implication consis- tency, suggesting antecedent facts learned about class are transferred to the subordinate.
Results.

---

Published as a conference paper at ICLR 2025
5.4 RQ3: CAN FINETUNING LoCo-LMS HELP CONSISTENCY ON UNSEEN KB?
Data. We evaluate LoCo-LMS on the EntailmentBank (Dalvi et al., 2022) test split, as proposed by Kassner et al. (2023) to reason on entailment trees. It consists of 302 implication trees spawning 805 constraints, with an average of 6.57 statement nodes and 2.66 constraints per tree; we consider each node of each tree as a statement with natural language with truth label set to We limit the tree depth to 5. An illustrated example is provided in Appendix 2. As in 5.2, we test two prompt and label formats. We assume that a possible semantic overlap between the training and test distributions, BeliefBank and EntailmentBank respectively, could underlie higher consistency scores across entailment trees; we estimate such overlap in Appendix E.2. Note that constraints in EntailmentBank involve more than one implication step and are akin to multi-hop reasoning.
Competitors and Metrics. We test our LoCo-LMs based on LLaMa-2 7b and previously trained in 5.2 on BeliefBank, without applying any changes. As baseline model, we consider LLaMa-2 7 without quantization. This experimental setup is inspired by Kassner et al. (2023), from whom we derive the notion of self-consistency on trees of entailments: each entailment tree E T is direct acyclic graph with a single root encoding the hypothesis to be proved; a subtree t' consists in each parents-child relationship in i, representing an entailment between the parent nodes (antecedents in logical conjunction) and the child (consequent). See Figure 2 in the Appendix for an example. For each tree t, we count the amount of violated subtrees t', that is when a true conjunction of antecedents does not imply a true consequent. Finally, we measure logical consistency as the fraction of the total violated subtrees over the total number of subtrees in T.
Results. In Table B.3 we report logical consistency across several depths. Scores are averaged across two sets of prompts and labels, detailed results are reported in Appendix B.2. We observe the consistency decreases across depths for the baseline model, until it flattens out, as more implications are evaluated. Conversely, LoCo-LM LM (F-IMP) and LoCo-LM (Super) achieve higher consistency across depths, validating the usefulness of our approach. Fine-tuning LoCo-LME on set of con- straints allows to generalize over unseen constraints of the same type. As expected, fine-tuning for negation does not bring any added benefit (and can worsen performance) as the in EntalimentBaan only implications are considered. Therefore, our recommendation for practitioners is to fine-tune for the constraints that considered in the downstream task, and when in doubt use a conjunc- tion of all constraints as in our LOCO-LMS SUPER which still improves w.r.t. a vanilla LLM when chaining more than two implications together.
DISCUSSION AND FURTHER WORK
Our results show that LOCo-LMS have improved selfs)jconistteny compared to recently intro- duced consistency layers which rely on external solvers, such as ConCoRD maicutic prompt- ing. This is especially important for small and medium-sized LLMs, that suffer from (self- jinconsistency and for which prompting techniques
not the final panacea (see our experiments in Section 5). In future work, we plan to extend our analysis to more complex logical operators (Vergari et al., 2021) and to consider more advanced probabilistic reasoning techniques that sport improved consistency guarantees (Ahmed et al., 2022a). Another promising direction we have not explored is that of first materializing the beliefs of an LLM such as in REFLEX (Kassner et al., 2023) and variants (Akyürek et al., 2024) and use the SL to improve consistency while potentially storing and re-using derived rules in writable external KB (Modarressi et al., 2023; 2024).
One limitation of our approach is relying on finetuning, and thus implying sensitivity to the choice of prompt format (White al., 2023). This can be partially addressed by fine-tuning using a mixture of formats, as we do in Section 5. While our SL. is constraint-ggnostic, in practice we fine-tune LoCo- LMS only against a combination of constraints known in advance. LoCo-LMS fine-tuning relies on two assumptions: that the probabilities of facts are conditionally independent given the LLM's inner state, and that the constraints in the KB are correct. The former readily applies to many LLMs, but assuming independence can bias the solutions learned by the SL (van Krieken et al., 2024). For the latter, most KBs are well-curated, but fine-tuning models against incorrect or inconsistent rules can compromise consistency and fluency.
10

---

Published as a conference paper at ICLR 2025
ETHICS STATEMENT
All authors have read and approved the ICLR Code of Ethics. Concerning societal consequences, the aim of this work is encourage LLMs to be more factual, (self-)consistent and, ultimately, reliable. However, it can also potentially enable malicious users to intentionally train LoCo-LMS against invalid rules to steer the model towards conclusions of their choice or potential reasoning shortcuts (Marconato et al., 2024b;a; Bortolotti et al., 2024). This issue is common to all strategies for aligning LLMs toward externally specified goals, like RLHF (Ouyang et al., 2022).
REPRODUCIBILITY STATEMENT
The data preprocessing pipeline described in detail in Appendix B. Code and data released at: https: / thub htm. ddi dww.s/dddacccccccoccccc 1mmoo-1...
ACKNOWLEDGMENTS
Funded by the European Union. Views and opinions expressed are however those of the author(s) only and do necessarily reflect those of the European Union the European Health and Digital Executive Agency (HaDEA). Neither the European Union nor the granting authority can be held responsible for them. Grant Agreement no. 101 101120763 TANGO. AV is supported by the "UN- REAL: Unified Reasoning Layer for Trustworthy ML" project (EP/Y023838/1) selected by the ERC and funded by UKRI EPSRC.
REFERENCES
Pysdd. In Recent Trends in Knowledge Compilation, Report from Dagstuhl Seminar 17381, sep 2017.
Kareem Ahmed, Stefano Teso, Kai-Wei Chang, Guy Van den Broeck, and Antonio Vergari. Seman- tic Probabilistic Layers for Neuro-Symbolic Learning. In NewrIPS, 2022a.
Kareem Ahmed, Eric Wang, Kai-Wei Chang, and Guy Van den Broeck. Neuro-symbolic entropy regularization. In Uncertainty in Artificial Intelligence, pp. 43-53. PMLR, 2022b.
Kareem Ahmed, Kai-Wei Chang, and Guy Van den Broeck. pseudo-semantic loss for autoregres- sive models with logical constraints. Advances in Neural Information Processing Systems, 36, 2024.
Afra Feyza Akyürek, Ekin Akyürek, Leshem Choshen, Derry Wijaya, and Jacob Andreas. Deductive closure training of language models for coherence, accuracy, and updatability, 2024.
Badr AlKhamissi, Millicent Li, Asli Celikyilmaz, Mona Diab, and Marjan Ghazvininejad. A review language models knowledge bases. arXiv preprint (rXiv:220400603) 2022.
Konstantinos Andriopoulos and Johan Pouwelse. Augmenting Ilms with knowledge: survey on hallucination prevention. arXiv preprint arXiv:2309.16455 2023.
Isabelle Augenstein, Timothy Baldwin, Meeyoung Cha, Tanmoy Chakraborty, Giovanni Luca Ciampaglia, David Corney, Renee DiResta, Emilio Ferrara, Scott Hale, Alon Halevy, et al. Fac- tuality challenges in the era of large language models, arXiv preprint 10.iv:2310.05/88 2023.
Roberto Battiti, Maximum satisfiability problem. pp. 2035-2041 Springer US, Boston, MA, 2009. ISBN 978-0-387-74755.00 doi: 1111111/97780-887777777777
364.
Lukas Berglund, Meg Tong, Max Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Kor- bak, and Owain Evans. The reversal curse: Llms trained on" is b" fail learn" is a". arXiv preprint arXiv:2309. 12288, 2023.
Rishi Bommasani, Drew A Hudson. Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportu- nities and risks of foundation models. arXiv preprint a08.v:2 2021.
11

---

Published as a conference paper at ICLR 2025
Samuele Bortolotti, Emanuele Marconato, Tommaso Carraro, Paolo Morettin, Emile van Krieken, Antonio Vergari, Stefano Teso, and Andrea Passerini. A benchmark suite for systematically eval- uating reasoning shortcuts, 2024. URL /ttttp:://ariiii orggoorggao hatg//ab//aae 2 246. 10368.
Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. Discovering latent knowledge in lan- guage models without supervision, 2022.
Mark Chavira and Adnan Darwiche, On probabilistic inference by weighted model counting, Arti- ficial Intelligence, 172(6-7).722-799, 2008.
Arthur Choi and Adnan Darwiche. Dynamic minimization of sentential decision diagrams. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 27, pp. 187-194, 2013.
YooJung Choi, Antonio Vergari, and Guy Van den Broeck. Probabilistic circuits: A unifying frame- work for tractable probabilistic modeling. 2020.
Roi Cohen, Eden Biran, Ori Yoran, Amir Globerson, and Mor Geva. Evaluating the ripple effects of knowledge editing language models, 2023.
Bhavana Dalvi, Peter Jansen, Oyvind Tafjord, Zhengnan Xie, Hannah Smith, Leighanna patanangkura, and Peter Clark. Explaining answers with entailment trees, 2022.
Adnan Darwiche. A differential approach to inference in bayesian networks. Journal of the ACM (JACM), 50(3):20-3305 2003.
Adnan Darwiche. Sdd: new canonical representation of propositional knowledge bases. In Swenty-Second International Joint Conference on Artificial Intelligence, 2011.
Luc De Raedt, Robin Manhaeve, Sebastijan Dumancic, Thomas Demeester, and Angelika Kimmig. Neuro-symbolic neural logical + probabilistic. In NeSy 19@ IJCAI, the 14th International Workshop on Neural-Symbort Learning and Reasoning, 2019.
Luc De Raedt, Sebastijan Dumančić, Robin Manhaeve, and Giuseppe Marra. From statistical rela- tional to neural-symbolic artificial intelligence. Proceedings of the Twenty-Ninth International Conference International Joint Conferences Artificial Intelligence, pp. 4943-4950, 2021.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized Ilms, 2023.
Luca Di Liello, Pierfrancesco Ardino, Jacopo Gobbi, Paolo Morettin, Stefano Teso, and Andrea Passerini. Efficient generation of structured objects with constrained adversarial networks. Ad- vances neural information processing systems, 33:14663-14674, 2020.
Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard Hovy, Hinrich Schütze, and Yoav Goldberg. Measuring and improving consistency in pretrained language mod- cls. Transactions of the Association for Computational Linguistics, 9:1012-1031, 2021.
Owain Evans, Owen Cotton-Barratt, Lukas Finnveden, Adam Bales, Avital Balwit, Peter Wills, Luca Righetti, and William Saunders. Truthful ai: Developing and governing ai that does not lic, 2021.
Richard Evans, David Saxton, David Amos, Pushmeet Kohli, and Edward Grefenstette. Can neural networks understand logical entailment? arXiv preprint arXiv:1802.0m 08535, 2018.
Hossein Rajaby Faghihi, Aliakbar Nafar, Chen Zheng, Roshanak Mirzace, Yue Zhang, Andrzej Uszok, Alexander Wan, Tanawan Premsri, Dan Roth, and Parisa Kordjamshidi. Gluecons: A generic benchmark for learning under constraints. In Proceedings of the AAAI Conference Artificial Intelligence, volume 37, pp.
2023.
Elconora Giunchiglia and Thomas Lukasiewicz. Coherent hierarchical multi-label classification networks. Advances in neural information processing systems, 33:9662- 66736, 2020.
Elconora Giunchiglia, Mihaela Cätälina Stoian, Salman Khan, Fabio Cuzzolin. and Thomas Lukasiewicz. Road-r: The autonomous driving dataset with logical requirements. Machine Lear- ing, 11(())3261-329 2023.
12

---

Published as a conference paper at ICLR 2025
Giwon Hong, Aryo Pradipta Gema, Rohit Saxena, Xiaotang Du, Ping Nie, Yu Zhao, Laura Perez-Beltrachini, Max Ryabinin, Xuanli He, and Pasquale Minervini. The hallucinations leaderboard an open effort to measure hallucinations in large language models. arXiv preprint arXiv:2404.05904, 2024.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation large language models, 2021.
Myeongjun Erik Jang and Thomas Lukasiewicz. Consistency analysis of chatgpt. arXiv preprint arXiv:2303.06223, 2023.
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12):1-38, 2023.
Jachun Jung, Lianhui Qin, Sean Welleck, Faeze Brahman, Chandra Bhagavatula, Ronan Le Bras, and Yejin Choi. Maieutic prompting: Logically consistent reasoning with recursive explanations, 2022.
Nora Kassner and Hinrich Schütze. Negated and misprimed probes for pretrained language models: Birds can talk, but cannot fly. arXiv preprint arXiv: 119911.030 1.03343, 2019.
Nora Kassner, Oyvind Tafjord, Hinrich Schütze, and Peter Clark. Beliefbank: Adding memory to pre-trained language model for a systematic notion of belief, 2021.
Nora Kassner, Oyvind Tafjord, Ashish Sabharwal, Kyle Richardson, Hinrich Schuetze, and Peter Clark. Language models with rationality, 2023.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et Retrieval-aie -uggenttd genera- tion for knowledge-intensinn nlp tasks. Advances in Neural Information Processing Systems, 33; 9459-9474, 2020.
Jiarui Li, Ye Yuan, and Zehua Zhang. Enhancing Ilm factual accuracy with rag to counter hallu- cinations: A case study on domain-specifte queries in private knowledge-bel arXiv preprint 2rXiv:2403.10444 2024a.
Tao Li, Vivek Gupta, Maitrey Mehta, and Vivek Srikumar. A logic-driven framework for consistency of neural models, 2019.
Zhenyu Li, Sunqi Fan, Yu Gu, Xiuxing Li, Zhichao Duan, Bowen Dong, Ning Liu, and Jiany- ong Wang. Flexkbqa: A flexible Ilm-powered framework for few-shot knowledge base question answering. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pp- 18608-18616, 2024b.
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods, 2021.
Jiacheng Liu, Wenya Wang. Dianzhuo Wang, Noah A. Smith, Yejin Choi, and Hannaneh Hajishirzi. Vera: A general-purpose plausibility estimation model for commonsense statements, 2023.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach, 2019.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. In Interna- tional Conference on Learning Representations, 2016.
Bill MacCartney. Natural language inference. Stanford University, 2009,
Emanuele Marconato, Samuele Bortolotti, Emile Krieken, Antonio Vergari, Andrea Passerini, and Stefamo Teso. Bears make neuro neuro-symbolic models aware of their reasoning shortcuts. In The 40th Conference Uncertainty in Artificial Intelligence, 2024a.
13

---

Published as a conference paper at ICLR 2025
Emanuele Marconato, Stefano Teso, Antonio Vergari, and Andrea Passerini. Not all heuro-symbolic concepts are created equal: Analysis and mitigation of reasoning shortcuts. Advances in Neural Information Processing Systems, 36, 2024b.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016.
Eric Mitchell, Joseph J. Noh, Siyan Li, William S. Armstrong, Ananth Agarwal, Patrick Liu, Chelsea Finn, and Christopher D. Manning. Enhancing self-consistency and performance of pre-trained language models through natural language inference, 2022.
Ali Modarressi, Ayyoob Imani, Mohsen Fayyaz, and Hinrich Schütze. Ret-Ilm: Towards general read-write memory for large language models. arXiv preprint arXiv:2305. (40,:2335..332,,, 2023.
Ali Modarressi, Abdullatif Köksal, Ayyoob Imani, Mohsen Fayyaz, and Hinrich Schütze. Memllm: Finetuning iims to use an explicit read- wrrte memory. arXiv preprint arXiv:2404. /672, 2024.
Long Ouyang. Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35: 17730-27744 2022.
Umut Oztok and Adnan Darwiche. A top-down compiler for sentential decision diagrams. In Twenty-Fourth International Joint Conference on Artificial Intelligence, 2015.
Oleksandr Palagin, Vladislav Kaverinskiy, Anna Litvin, and Kyrylo Malakhow. Ontochatgpt infor- mation system: Ontology-driven structured prompts for chatgpt meta-learning. arXiv preprint MxXiv:2307.05082 2023.
Mihir Parmar, Neeraj Varshney, Nisarg Patel, Santosh Mashett, Man Luo, Arindam Mitra, and Chitta Baral. Logicbench: A benchmark for evaluation of logical reasoning. 2023.
Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H. Miller, and Sebastian Riedel. Language models knowledge bases?, 2019.
Marin Vlastelica Pogančić, Anselm Paulus, Vit Musil, Georg Martius, and Michal Rolinck. Differ- entiation of blackbox combinatorial solvers. In International Conference Learning Represen- tations, 2019.
Marin Vlastelica Pogancic, Anselm Paulus, Vit Musil, Georg Martius, and Michal Rolínck. Dif- ferentiation of blackbox combinatorial solvers. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
Harsh Raj, Vipul Gupta, Domenic Rosati, and Subhabrata Majumdar. Semantic consistency for assuring reliability of large language models. arXiv preprint SIti:2308.09138,, 2023.
Vipula Rawte, Amit Sheth, and Amitava Das. A survey of hallucination in large foundation models. arXiv preprint 13Xi:2309900922.. 2023.
Kyle Richardson and Gijs Wijnholds. Lectures on language model programming, August 2024.
URL https: / com/ hatm//yaaaamiiresslslsl 2ateele_ssssll_________- 1e144__m__prroro programming.
Alan JA Robinson and Andrei Voronkov. Handbook of automated reasoning, volume 1. Elsevier, 2001.
Yucheng Shi, Shaochen Xu, Zhengliang Liu, Tianming Liu, Xiang Li, and Ninghao Liu. Mededit: Model editing for medical question answering with external knowledge bases. arXiv preprint 10X10:2309.16035, 2023.
Hikaru Shindo, Devendra Singh Dhami, and Kristian Kersting, Neuro-symbolic forward reasoning. arXiv preprint arXiv:2110.09383, 2021.
Koustuy Sinha, Shagun Sodhani, Jin Dong, Joelle Pineau, and William L Hamilton. Clutrr: A diagnostic benchmark for inductive reasoning from text. arXiv preprint 1:0ivv/90.006/111 2019.
14

---

Published as a conference paper at ICLR 2025
Robyn Speer, Joshua Chin, and Catherine Havasi. Conceptnet 5.5: An open multilingual graph of general knowledge, 2018a.
Robyn Speer, Joshua Chin, and Catherine Havasi. Conceptnet 5.5: An open multilingual graph of general knowledge, 2018b. URL https: httpa//.orrgbbbbabboeeeo
03975.
Harald Steck, Chaitanya Ekanadham, and Nathan Kallus. Is cosine-similarity of embeddings really about similarity? Companion Proceedings of the ACM on Web Conference 2024, WWW '24. ACM, May 2024, doi: 10.1 2..1445/58833333655522 URL http:/ www.xxddc..oooo wtw.ooo.....og.oo 1145/
3589335. 3651526.
Oyvind Tafjord and Peter Clark. General-purpose question-answering with macaw, 2021.
Oyvind Tafjord, Bhavana Dalvi Mishra, and Peter Clark. Entailer: Answering questions with faithful and truthful chains of reasoning, 2022.
Katherine Tian, Eric Mitchell, Huaxiu Yao, Christopher D. Manning, and Chelsea Finn. Fine-tuning language models for factuality, 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fermandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.
Emile van Krieken, Erman Acar, and Frank van Harmelen. Analyzing differentiable fuzzy logic operators. Artificial Intelligence, 302: 103602. 2022.
Emile van Kricken, Pasquale Minervini, Edoardo M Ponti, and Antonio Vergari. On the indepen- dence assumption in neurosymbolic learning. In Forty-first International Conference on Machine Learning. 2024.
Antonio Vergari,
Nicola Di Mauro, and Guy Van den Brock. Tractable probabilistic models: Repre- sentations, algorithms, learning, and applications. Tutorial at UAI, 2019.
Antonio Vergari, YooJung Choi, Anji Liu, Stefano Teso, and Guy Van den Broeck. A compositional atlas of tractable circuit operations for probabilistic inference. Advances in Neural Information Processing Systems, 34:13189-13200 1301,, 2021.
David Wadden, Shanchuan Lin, Kyle Lo. Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Hannanch Hajishirzi. Fact or fiction: Verifying scientific claims, 2020.
Cunxiang Wang, Xiaoze Liu, Yuanhao Yue, Xiangru Tang, Tianhang Zhang, Cheng Jiayang, Yunzhi Yao, Wenyang Gao, Xuming Hu, Zehan Qi, et al. Survey factuality in large language models: Knowledge, retrieval and Nomain--secificin arXiv preprint arXiv:2310.0752.. 2023.
Fei Wang, Chao Shang, Sarthak Jain, Shuai Wang. Qiang Ning, Bonan Min, Vittorio Castelli, Yas- sine Benajiba, and Dan Roth. From instructions to constraints: Language model alignment with automatic constraint verification. arXiv preprint arXiv:2403.06322 2024.
Jules White, Quchen Fu, Sam Hays, Michael Sandborn, Carlos Olea, Henry Gilbert, Ashraf El- nashar, Jesse Spencer-Smith, and Douglas C Schmidt. A prompt pattern catalog to enhance prompt engineering with chatgpt. arXiv preprint arXiv:2302.1 1182, 2023.
Jingyi Xu, Zilu Zhang, Tal Friedman, Yitao Liang, and Guy Van den Broeck. A semantic loss function for deep learning with symbolic knowledge, 2018.
15

---

Published as a conference paper at ICLR 2025
Sohee Yang. Elena Gribovskaya, Nora Kassner, Mor Geva, and Sebastian Riedel. Do large language models latently perform multi-hop reasoning?, 2024.
Hanlin Zhang, Jiani Huang, Ziyang Li, Mayur Naik, and Eric Xing. Improved logical reasoning of language models via differentiable symbolic programming, 2023.
Honghua Zhang, Liunian Harold Li, Tao Meng, Kai-Wei Chang, and Guy Van den Broeck. On the paradox of learning to reason from data, arXiv preprint arXiv:2205. 1502, 2022.
16

---

Published as a conference paper at ICLR 2025
A BACKGROUND ON CIRCUITS, COMPILATION AND WMC
In this section, we provide additional background and report classical results from the circuit litera- ture (Choi al., 2020; Vergari et al., 2021).
Circuits (Vergari et al., 2019; Choi et al., 2020) are constrained computational graphs that enable tractable computations. For our purposes, they enable the tractable computation of the Weighted Model Count (WMC) encoded in the semantic loss (Equation (SL)).
Definition A.1 (Cireuit). circuil is a parameterized directed acyclic computational graph over variables Z encoding a function e(Z), and comprising three kinds of computational units: inpur, product, and sum units. Each product or sum unit n receives the outputs of other units as inputs, denoted with the set in(n). Each unit encodes function on defined as: (i) f. (sc(n)) if n is an input unit, where Sn a function over variables sc(n) C Z, called its scope, (ii) II sein(w) (s((i)) if n is a product unit, and (iii) ΣjEin(n) Wyyc (scr(sc(j)) if is sum unit, with wj E R denoting the weighted sum parameters. The scope of a product or sum unit n. is the union of the scopes of its inputs, i.c., sc(m) Ujein(n; sc(j).
Tractable WMC can be achieved by ensuring that these computational graphs abide certain structural properties: smoothness, decomposability and determinism (Vergari et al., 2021).
Definition A.2 (Smoothness & Decomposability). A circuit is smooth if for every sum unit n, its inputs depend on the same variables: Vei, C2 E in(n), sc(ci) = sc(c2). It decomposable the inputs of every product unit depend disjoint sets of variables: in(n) (2,, n sc(c2) 0.
The next step to translate a logical constraint a; into a smooth and decomposable circuit c(z). To this end, we employ a special type of PCs, defined as follows.
Definition A.3 (Constraint circuits). A PC e over variables Z a constraint circuit encoding prior knowledge a; if it computes 1(z a;} for every configuration
As a practical way to realize such a circuit, we will consider constraint circuits that have all sum unit parameters equal I and input functionals that are indicator functions over their scope. Fur- thermore, we require each sum unit in it to be deterministic.
Definition A.4 (Determinism). sum unit n is deterministic if its inputs have disjoint supports, i.e., Vc,,02 E in(n), e # C2 supp(c1) O supp(c2) = 0.
Compilation. We use standard compilation tools from the knowledge compilation community to turn a logical constraint into a smooth, decomposable and deterministic circuit. Specifically, we use PySDD* (pys, 2017) python SDD compiler (Darwiche, 2011; Choi & Darwiche, 2013). Note that SDDs are just smooth, decomposable and deterministic circuits (Vergari et al., 2019).
Consider the following facts:
fi aan albatross is a bird"
fa a' albatross breathes"
fa a'an albatross is an animal"
and their corresponding truth values represented three binary variables 21, 22,22- We want to represent the following constraint
(z2 - 23) A (2)
- 23).
We will now sketch how a circuit compiler would proceed: the objective of compilation is to encode the above logical constraint into a compact form representing all possible assignments to 21, #2, A3- We refer the reader to Choi & Darwiche (2013) for details. Our compiler proceeds in bottom up fashion, where bottom-up compilation can be seen as composing Boolean sub-functions whose domain is determined by variable ordering (Darwiche, 2011; Choi & Darwiche, 2013). It would start by compiling a constraint circuit that a function of 2 and 22, and compose it with a constraint circuit that is function of 23 We first introduce input functionals representing indicators associated
17

---

Published as a conference paper at ICLR 2025
with each fact truth value. We will denote by 24 the indicator 1(2 1} and by -2 the indicator 1 = 0}.
[(21 - 03 O [ - lf
[ - 0f
( = 1f
c
( = 0f
O
( 1]
We start by disjoining the indicator 21 with 721. This corresponds to introducing deterministic and smooth sum units in our circuits.
1/:2 11
1 = -]
Deterministic sum units represent disjoint solutions to the logical formula, meaning there exists distinct assignments, characterized by the children, that satisfy the logical constraint.
The compilation process proceeds by conjoining the constraint circuits for 22 V mz2 with 21, 22 V -22 with -2 and -2 with -21-
1)
00
I(=--)
I(a
A decomposable product unit decomposes functions over disjoint sets of variables. The above prod-
uets represent the Boolean functions (z2 V -22) A 21, (22 -22) A -Z1, and 1 A -22
We disjoin (z2 V -z2) identity.
A =2222A
with (z2 V /222) A 1 and 1 A -22 with true, the logical multiplicative
=1==111 =1
1( 0)
1(s
1(2: =
So far, have compiled constraint circuits for the logical formulas
((22 V -22) A 21) v ((2) -22) -21)
and
38 A -22.
We are left to conjoin the first one with 23, and the second one with m23 and disjoin the resulting constraint circuits. What we get is mixture over the possible solutions: If the model says that f1, f2. or both, are true, then it better predict that fa is true as well.
1 1)
1( = 0)
O
1(2 1}
O
1(4 11
1/20
0
18

---

Published as a conference paper at ICLR 2025
For constraints compilation is extremely fast: taking only 2.5 milliseconds compile straint and compute the loss on BeliefBank. The loss computation requires computing the WMC (Equation (SL)) in closed form. This can be done easily after compiling the logical constraint into circuit as illustrated above. Its complexity is linear in the size of the circuit (Darwiche, 2003; Vergari al., 2021).
Impact on LOCO-LMS. Thank to circuits, we can evaluate loss (Equation (SL)) exactly without having explicitly enumerate truth assignments; this operation takes time linear in the size fo the circuit, yielding efficient fine-tuning.
The compilation step is also extremely fast, taking only 2.5 milliseconds compile a constraint and compute the loss on BeliefBank. Moreover, many data points will share the same constraint during training. enabling caching.
Given ours is pure fine-tuning approach, it has no inference-time overhead. For reference, ConCord takes 3669 seconds to perform inference on BeliefBank (silver calibration sets) for Macaw-large. whereas LoCo applied to the same model requires only 2405 seconds.
B DETAILED SETTING AND RESULTS
B.1
RQI
B.1.1 DATA PREPROCESSING
We train LoCo-LMS on the BeliefBank (Kassner et al., 2021), calibration split. This dataset is derived from ConceptNet (Speer et al., 2018a), large curated knowledge graph encoding factual knowledge and logical relations between entities at different levels of abstraction; we use the splits introduced by Mitchell al. (Mitchell al., 2022) for direct comparison. consists of three pieces: a "calibration" set of 1, 072 annotated facts about 7 entities of the form (subject, property, true/false) used for training, "silver" set of 12, 636 facts about 85 entities used for evaluation, and a set of 2224 valid abstract logical implications. To use our SL, we require defining set of ground constraints. We derive these as follows. For each general implication constraint, we lookup the subjects of all facts in the training set: if the antecedent or the consequent fact of the general constraint is known for that subject, we add the subject ground constraint to the dataset Dc.
We generate two splits: TI facts, appearing either as antecedents consequents in the constraints; T2 facts, appearing exclusively as consequents. The goal is to correctly guess the consequents by seeing only the antecedents and the constraints. In the calibration set, count 796 antecedents and 276 consequents, spawning 14, 005 grounded constraints. In the silver set, we count 9, 504 antecedents and 3, 132 consequents, spawning 169, 913 grounded constraints. We subsequently test the effects of pure supervised fine-tuning: a portion of random facts from the calibration set (T1+T2) taken with the goal predict the excluded antecedent or consequent facts. We train Tl facts and evaluate on T2 facts for RQ2 as well: 77 facts (antecedents) constitute a valid subset for all the considered logical rules.
19

---

Published as a conference paper at ICLR 2025
Table 4: LoCo-LMS achieve better logical self-consistency and factuality as measured via Equation (4) and Fi scores when compared to cross-entropy fine-tuning (XENT) and basclines using external reasoners such as ConCoRD (Mitchell et al., 2022) measured on train (calibration set) facts. For RQ1 (Section 5), LoCo-LMs fine-tuned on T1 facts only outperform training- free baseline for all metrics. For RQ2, they boost performance in the presence of small fraction of TI+T2 facts (5-10%). For larger dataset sizes, LOCO-LMS are competitive for consistency and factuality on consequents.
Method
Train size
Antecedentss Fj
Consequents F
Total Fi
Logical consistency
ConCoRD
0.91
0.91
MACAW
0.47
0.84
0.78
0.82
RQI
MACAW+XENT
II
0.46
0,08
0.14
0,79
LoCo-LM T1
0.98
0.99
0.99
1.00
MACAW+XENT
(5%)
0.31
0.73
0.69
0.90
LoCo-LM
(5%)
0.34
0.77
0.72
0.92
RQ2 MACAW-XENT
TI+T2
0.48
0.88
0.85
0.87
LoCo-IM
T1+T2 (10%)
0.52
0.95
0.89
0.91
MACAW+XENT
1.00
LoCo-LM
T1+T2 (75%)
1.00
B.2 RQ2
Table 5: LoCo-LMs evaluated BeliefBank, training (calibration) split. Scores averaged across four prompt formats and truth labels. We observe fine-tuning with our method allows for higher logical consistency to different rules.
CONSISTENCY
SELF-CONNISTENCY
MODEL
TRAIN SUBSET
PPL
FAC
IMP
REV
NEG
IMP
REV
AVG
LLAMA-2.7 ZERO SHOT
$2.30
0.29
0.47
0.48
0.33
0.44
0.50
0.42
LLAMA-2-78 Few SHOT
$2.30
0.55
0.72
0.42
0.36
0.47
0.42
0.49
LLAMA-2-7D XENT
T1+T2
116.85
0.14
0.35
0.47
0.11
0.57
0.47
0.31
LaCo-IIAAA-2-7h (NEG)
TI
62.21
0.14
0.41
0.71
0.41
0.28
0.68
0.44
LoCo-LLAMA-2-7h (F-IMP)
67.15
1.00
0.52
0.00
1.00
0.52
0.68
LoCo-LLAMA-2-78
LLAMA-3.1-88 ZERO SHOT
78.22
0.44
0.55
0.57
0.38
0.54
0.57
0.52
LLAMA-3.1 FEW SHOT
78.22
0.41
0.53
0.48
0.36
0.45
0.48
0.44
LoCo-LIAMA-3. 3.1--8 (SUPER)
T1
78.22
0.82
0.89
0.84
0.81
0.81
0.84
0.83
20

---

Published as a conference paper at ICLR 2025
Table 6: LOCo-LMS evaluated on BeliefBank, training (calibration) split. Prompt format [true, false] is used. We observe fine-tuning with our method allows for higher logical consistency different rules.
CONSISTENCY
SELF-CONSISTENCY
PPL
FAC
IMP
REV
NEG
IMP
REV
AVG
LLAMM-22 7- ZERO SHOT
52.30
0.43
0.63
0.33
0.38
0.29
0.39
0.41
LLAMA-2-7B FLW SHOT
52.30
0.53
0.74
0.36
0.28
0.42
0.37
0.45
LLAMA-2-7B CoT
52.30
0.67
0.76
0.77
0.32
0.74
0.77
0.66
1.I.AA-2-770 7ERO SHOT
44.90
0.52
0,76
0.79
0.18
0.35
0.90
0.58
LLAMA-2-7B- XENT
T1+T2
116.85
0.37
0.47
0.16
0.02
0.32
-L---L--M-----BB (NEG)
1110O-LLMM---777 IMP)
LoCo-LLAMA-2-76 (SUPER)
62.23
0.88
0.91
0.72
0.94
0.86
0.73
0.84
I.I.MA-3.1 1888 ZERO SHOT
78.22
0.47
0.58
0.63
0.48
0.61
0.63
0.57
LLAMA-3.1-88 FEW
0.52
I.L.MA-3. (SUPER)
Table 7: LOCo-LMS evaluated on BelielBank, training (calibration) split. Prompt format [yes, no] used. We observe fine-tuning with our method allows for higher logical consistency to different rules.
CONSISTENCY
SELF-CONSISTENCY
MODEL
TRAIN SUBSET
PPL
FAC
IMP
REV
NEG
IMP
REV
AVG
LLAMA-2-7B ZERO SHOT
52.30
0.39
0.51
0.08
0.46
0.27
0.09
0.30
L.LAMA-277B FEW SHOT
52.30
0.52
0.66
0.55
0.48
0.55
0.55
0.55
1.I.MA-2-77 CoT
52.30
0.38
0.52
0.57
0.48
0.54
0.57
0.51
LLAMA-2-70B ZERO SHOT
44.90
0.46
0.68
0.81
0.05
0.28
0.93
0.54
LLAMA-2-7B- XENT
T1+T2
116.85
0.05
0.32
0.00
0.04
0.00
0.00
0.07
LoCo-LLAMA-2-76 (NEG)
62.21
0.33
0.70
0.00
0.32
LoCo-LLAMX-2-76 (F-IMP)
67.15
1.00
0.08
0.53
LLLAMA---7 (SUPER)
I.L.MA-311 7ERO SHOT
78.22
0.43
0.49
0.75
0.44
0.57
0.75
0.57
I.I.MA-3.1 FEW SHOT
78.22
0.31
0.42
0.51
0.31
0.42
0.51
0.43
LLAMA-3.1-8
0.82
Table 8: LoCo-LMs evaluated on BeliefBank, training (calibration) split. Prompt format [correct, incorrect] is used. We observe fine-tuning with our method allows for higher logical consistency to different rules.
CONSISTENCY
SELF- CONS-SONSITTNCCY
MODEL
TRAIN SUBSET
PPL
FAC
IMP
REV
NEG
IMP
REV
LLAMA-2-7B
LLAMA-2-70 XENT
T1+T2
116.85
0.20
0.86
0.50
LoCo-LLAMA-2-7u (NEG)
62.21
0.30
0.02
1.00
0.39
LaCo-LLLAM42-7A. (F-IMP)
TI
67.15
0.99
1.00
0.96
0.01
1.00
0.96
0.82
LoCo-LLAMA-2-72 (SUPER)
TI
62.23
0.80
0.92
0.80
0.80
0.85
0.80
0.84
LLAMA-3.
0.13
0.25
0.13
LLAMA-3.1-88 FEW SHOT
78.22
0.40
0.56
0.15
0.19
0.31
0.15
0.29
LoCo-LLAMA 3.1-A (SUPER)
78.22
0.79
0.86
0.83
0.80
0.78
0.83
0.82
B.3 RQ3
21

---

Published as a conference paper at ICLR 2025
Table LoCo-LMs evaluated on BeliefBank, training (calibration) split. Prompt format (right, wrong] is used. We observe fine-tuning with our method allows for higher logical consistency different rules.
CONSISTENCY
SELF-CONSISTENCY
PPL
FAC
IMP
REV
NEG
IMP
REV
AVG
LLAMA-2.1 2- ZERO SHOT
52.30
0.05
0.31
0.96
0.05
0.67
0.96
0.50
LLAMA-2-7D FEW SHoT
52.30
0.69
0.78
0.76
0.66
0.64
0.76
0.71
LLAMA-2-7B XENT
T1+T2
116.85
0.02
0.29
0.98
0.04
0.75
0.98
0.34
LoCo-IILAMA-277h (NEG)
62.21
0.00
0.30
1.00
0.00
0.00
1.00
0.38
LoCo-LLAMA-2-7h (F-IMP)
67.15
0.99
1.00
0.96
0.00
0.96
0.82
LoCo-LLAMA-2-78
LLAMa-3, ZERO ShoT
78.22
0.43
0.50
0.75
0.34
0.63
0.75
0.62
LLAMA-3.1 38 FEW SHOT
78.22
0.48
0.57
0.67
0.46
0.56
0.67
0.52
LoCo-LLAAM-3.I (SUPER)
78.22
0.89
0.90
0.90
0.90
0.86
0.90
0.89
Table 10: LoCo-LMS evaluated on BeliefBank, test (silver) split. Prompt format 1 [true, fa sls]] is used. We observe fine-tuning with method allows for higher logical consistency to
different rules.
MODEL
TRAIN SUBSET
PPL
FAC
IMP
REV
NEG
1MP
REV
AVG
LLAMA-2-78 ZERO SHOT
$2.30
0.41
0.55
0.22
0.41
0.30
0.25
0.36
LLAMA-2-70 FEW SHOT
$2.30
0.53
0.75
0.37
0.27
0.41
0.37
0.45
LLAMA-2-70 CoT
$2,30
0.67
0.76
0.77
0.32
0.74
0.77
0.67
LLAMa-2-708 ZERO SHOT
44,90
0.50
0.72
0.80
0.20
0.34
0.89
0.58
ILAMA-2-77 XENT
TI+T2
116.85
0.40
0.52
0.02
0.11
0.82
0.02
0.31
LoCo-LLAMA-2-7B (NEG)
TI
62.21
0.44
0.64
0,86
0.92
0.28
0.72
0.64
LoCo-LLAMA-2-73 (F-IMP)
67.15
0.98
0.98
0.07
0,00
0.98
0.07
0.51
LoCo-LLAMA-2-7h (SUPE))
62.23
0.75
0.78
0.72
0.91
0.74
0.72
0.77
1.AMA-3.1 8B ZERO SHOT
78.22
0.46
0.60
0.65
0.50
0.60
0.65
0.59
LLAMa-3,1-80 FLW SHOT
78.22
0.48
0.60
0.65
0.49
0.55
0.65
0.57
LACo-LLAMA-3.J-85 (SUPER)
78.22
0.71
0.81
0.72
0.71
0.74
0.72
0.74
Table 11: LoCo-LMs evaluated on BeliefBank, test (silver) split. Prompt format [yes, no] is used. We observe fine-tuning with our method allows for higher logical consistency to different rules.
CONSISTENCY
SHLF-CONSSSSENCY
MODEL
TRAIN SUBSET
PPL
FAC
IMP
REV
NEG
IMP
REV
AVG
LLAMA-2-76 SHOT
52.30
LLAMA-2-78 FEW SHOT
$2.30
0.53
0.67
0.57
0.49
0.58
0.53
0.56
LLAMA 78 CoT
$2.30
0.38
0.52
0.57
0.48
0.54
0.57
0.51
LLAMa-2-70p ZURO SHOT
44,90
0.44
0.65
0.82
0.05
0.29
0.93
0.53
LLAMA-2-7B XENT
T1+T2
116.85
0.11
0.39
0.00
0.03
0.80
0.00
0.22
LACo-IIAMA-2-7H (NEG)
TI
62,21
0.44
0.65
0.00
1.00
0.28
0.00
0.40
LaCo-LLAMs-2-7h (F-IMP)
TI
67.15
0.99
0.99
0.07
0.00
0.99
0.07
0.52
LoCo-LLAMA-2-7B (SUPER)
62.23
0.73
0.75
0.81
0.83
0.67
0.82
0.77
LLAMA-3.1-48 1-8u ZERO SHoT
78.22
0.44
0.55
0.72
0.42
0.61
0.72
0.58
1.AMA-3.11. SB Few SHOT
78.22
0.33
0.46
0.54
0.32
0.42
0.54
0.43
LoCo-LLAMs-3.1 1-88 (SUPER)
T1
78.22
0.71
0.81
0.75
0.76
0.74
0.75
0.75
C CONTEPTNET
22

---

Published as a conference paper at ICLR 2025
Table 12:
LoCo-LMS evaluated on BeliefBank, test (silver) split. Prompt format [correct, incorrect] is used. We observe fine-tuning with our method allows for higher logical consistency to different rules.
CONSISTENCY
SELF-CONSISTENCY
PPL
FAC
IMP
REV
NEG
IMP
REV
AVG
LLAMA-2.1 2-78 ZERO SHOT
52.30
0.25
0.40
0.65
0.45
0.52
0.65
0.49
LLAMA-2-70 FEW SHoT
52.30
0.44
0.64
0.00
0.00
0.28
0,00
0.23
LLAMA-2-7B XENT
T1+T2
116.85
0.12
0.35
0.89
0.17
0.66
0.89
0.51
LoCo-IIAAA-2-77 (NEG)
62.21
0.00
0.35
1.00
0.01
0.00
1.00
0.39
LoCo-LLAMA-2-7h (F-IMP)
67.15
0.98
0.96
0.01
0.96
0.81
LoCo-LLAMA-2-78
LLAMa-3, ZERO SHoT
78.22
0.42
0.60
0.15
0.26
0.34
0.15
0.32
LLAMA-3.1 1- FEW SHOT
78.22
0.40
0.55
0.15
0.19
0.31
0.15
0.31
LoCo-IIAAM-3.I (SUPER)
78.22
0.72
0.79
0.83
0.81
0.77
0.83
0.79
Table 13: LoCo-LMs evaluated on BeliefBank, test (silver) split. Prompt format 4 [right, wrong] is used. We observe fine-tuning with method allows for higher logical consistency to different rules.
MODEL
TRAIN SUBSET
PPL
FAC
IMP
REV
NEG
IMP
REV
AVG
LLAMA-2-78 ZERO SHOT
$2.30
0.03
0.35
0.97
0.05
0.67
0.97
0.51
LLAMA-2-70 FEW SHOT
$2.30
0.63
0.75
0.67
0.64
0.62
0.67
0.66
LLAMa-2-70 XENT
TI+T2
116.85
0.21
0.42
0.30
0.10
0.76
0.30
0.35
LaCo-LLAMA-2-74 (NEG)
TI
62.21
0.00
0.35
1.00
0.00
0.00
1.00
0.39
LaCo-IIAAM-2-77 (F-IMP)
TI
67.15
0.98
0.98
0.96
0.01
0.99
0.96
0.81
LoCo-LLAMA-2-7B (SUPER)
62.23
0.80
0.80
0.91
0.78
0.81
0.91
0.83
LLAMA-3. 1-88 ZERO SHoT
78.22
0.47
0.58
0.63
0.48
0.61
0.63
0.57
LLAMA-3.1-88 FEW SHOT
78.22
0.43
0.53
0.68
0.44
0.52
0.68
0.55
3.1 (SUPER)
TI
78.22
0.77
0.80
0.89
0.90
0.80
0.89
0.84
Table 14: LoCo-LMs can achieve higher consistency across depth than the baseline. Scores are computed with Format [true, false], reported in Appendix F.2. LoCo-LM fine-tuned with the implication rule achieves best consistency.
DEPTH
LLAMA-2-7B
0.73
0.77
0.79
0.80
0.80
ROCO-LLAMA-21 2.78 (NEG)
0.03
0.03
0.03
0.04
0.05
LoCo-LLAMA-2-TB (F-IMP)
0.97
0.96
0.97
0.97
0.97
LOCo-LLAMA-2-7E (SUPER)
0.75
0.74
0.73
0.73
0.74
D ENTAILMENTBANK
23

---

Published as a conference paper at ICLR 2025
Table 15: LoCo-LMS can be consistent across unseen trees of entailments when trained for implication consistency (F-IMP) on BeliefBank and evaluated as is on EntailmentBank (Dalvi et al., 2022).
depth
model
2
4
5
LLaMa-2-7b
0.87
0.76
0.59
0.61
0.63
LaCo-IHaMa-227h (NEG)
LoCo-LLaMa-2-70 (F-IMP)
0.98
0.98
0.98
0.98
0.98
LaCo-IAaMa-2-7h (Super)
0.69
0.68
0.68
0.68
0.69
Table 16: LoCo-LMS can achieve higher consistency across depth than the baseline. Scores are computed with Format 2 [yes, no], reported in Appendix F.2. LoCo-LM fine-tuned with on the implication rule and the negation rule achieve best consistency.
DEPTH
LLAMA-2-78
1,00
0.75
0.38
0.42
0.46
LoCo LLAMA- 78 (NEG)
0.99
0.99
0.99
0.99
0.99
LoCo-LLAMa-2-77 (F-IMP)
0.99
0.99
0.99
0.99
0.99
LOCO-LLAMA-2-78 (SUPER)
0.62
0.62
0.63
0.63
0.64
Table 17: Distribution answer labels from LOCo-LMS for different prompt formats on the EntailmentBank test split.
[Yee, RG]
[CRCE, FALSS]
MODEL
YES
NO
INVALID
TRUE
FALSE
INVALID
LLAMA- 2-78
1188
1441
615
1742
278
LLCO-LLAMA-2- (NEG)
1695
LOCO-LL (M-IMP)
LoCo-LLAMA-2-TE (SUPER)
2079
486
874
1756
Table 18: LoCo-LMS evaluated BeliefBank, train (calibration) split, compared by decod- ing strategy. All few shot prompts in Appendix F.3 were used. The default configuration consists in 1.0, temperature
1.0: greedy decoding consists in top_k 1.0. We observe no significant difference as in the current setup, the truth label represented by single token and thus changes the sampling tech- nique could be better observed on an output sequence.
top_k 50, top-p
1, top-p = 1 0, t temperature
CONSISTENCY
SELF-CONSISYENCY
MODEL
DECODING
PPL
FAC
IMP
REV
NEG
IMP
REV
AVG
LoCO-LLAMA 28 (SUPER) LOCO-LLAMA-2-11 (SUPER)
0.79
0.57
GREEDY
62.41
0.79
0.82
0.82
0.57
0.76
0.82
0.76
E SEMANTIC OVERLAP
We base measurement for semantic overlap cosine similarity, widely adopted in literature. We report our results with a note for caution: is unclear whether embeddings could similar for the semantic features seeking Steck al. (2024), suggesting further research the topic.
24

---

Published as a conference paper at ICLR 2025
Table 19: Logical consistency in class-knowledge transfer measured in LoCo-IMS ConceptNet with Format 2 lyes, no]. On average, LoCo-LMs trained on joint logical constraints surpass baseline methods, with consistent gains in factuality and implication consistency. LoCo-LMs have been trained on high-level class properties for 10 epochs.
CONSISTENCY
SELF-CONSISTENCY
MODEL
FAC
IMP
REV
NEG
IMP
REV
AVG
2-78--278888
LLAMA-2-70-rUW SHOT
0.56
0.71
0.56
0.48
0.56
LoCo-LLAMA-2-70 (SUPER)
0.74
0.82
0.39
0.41
0.83
0.59
0.66
melting is kind of phase change
the ice melts
the ice undergoes phase change
phase changes change mass
the mass of the ice will not change
Figure 2: An illustration of entailment tree, namely "prof", from EntailmentBBnk Dalvi al. (2022). Blue nodes are premises in logical conjunction, orange nodes are implications and the green node denote the hypothesis prove.
E.1 BELIEFBANK
We measure the semantic overlap between the training and test distribution by constructing a Representation Dissimilarity Matrix (RDM) of Macaw's embeddings (token average) between training and test entities. The main assumption is that semantically similar subjects may have similar properties, as a proxy for domain knowledge transfer.
Gmmal
carp
cott
Schain
sesnnn an
Gent Seatie
fummir
ropprdd
liamia
magens
mill
Bicer
Whelee
paony
albatross
computer
edger
cypress
0.2
0.0
Figure 3: Pairwise cosine similarities between entities in the training distribution (calibration, rows) and the test distribution (silver, columns).
E.2 BELIEFBANK BELIFBANK-ENTALMMENNBBR. ENTAILMENTBANK
We consider the training split, namely "calibration" in ConCoRD Mitchell et al. (2022), from Be- liefBank Kassner al. (2021), and the split from EntailmentBank Dalvi al. (2022) estimate
25

---

Published as a conference paper at ICLR 2025
Table 20: Fraction of facts in BeliefBank with cosine similarity above with any fact in Entail- mentBank, for = {0.80, 0.85, 0.90}.
0.80
0.41
0.85
0.22
0,90
0.02
the knowledge that LoCo-LMs could transfer to entailment trees. We process BeliefBank as a set of 1, 072 facts, while EntailmentBank as set of 2, 635 facts. Both sets contain statements in natural language that are converted into vector embeddings through encoding with LLaMa-2-7b Touvron al. (2023); the last layer logits considered and sentence representation is obtained by averaging across tokens. We consequently compute the pairwise cosine similarities between fact embeddings from both sets. For each fact in BeliefBank,
take the maximum similarity with any fact from EntailmentBank, which should represent the existance of unit of similar knowledge between the two datasets. Given the volume of pairwise comparisons, we aggregate the results.
PROMPTS
F.1
PROMPTS FOR MACAW-LARGE
We query the language model for a belief label about statement in natural language. We adopt the format:
Prompt
SanswerS; Smcoptions$ = (A) pos label (B) neg label SquestionS = Is subject a property?
We fix <pos_label> "Yes." and <neg_label> "No We converted the (<subject>> <property>) tuple in natural language with formatting function provided by Mitchell et al. (Mitchell et al., 2022).
Expected answers
SanswerS pos label SanswerS neg label
F.2 ZERO-SHOV PROMPTS FOR LoCo-LMs
We adopt two label sets to make the model less prompt sensitive:
Format 1: [true, false]
Prompt
You can answer only with "true" or "false". Is the fact true? Fact: statement
Expected answers
Answer: true
Answer: false
Format 2: [yes, no]
26

---

Published as a conference paper at ICLR 2025
Prompt
You can answer only with "yes" or "no". Is the fact true? Fact: statement
Expected answers
Answer: yes
Answer: no
Format 3: [correct, incorrect]
Prompt
You answer only with "correct" or "incorrect". Is the fact true? Fact: statement
Expected answers
Answer: correct
Answer: incorrect
Format 4: [right, wrong]
Prompt
You can answer only with "right" or "wrong". Is the fact true? Fact: statement
Expected answers
Answer: right
Answer: wrong
F.3 FEW-SHOT PROMPTS FOR LoCo-LMS
We adopt two label to make the model less prompt sensitive:
Format 1: [true, false]
Prompt
Fact: the earth is round. Label: true. Fact: the sun is cold. Label: false. Fact: [fact]. Label:
Expected answers
Answer: true
Answer: false
Format 2: [yes, no]
27

---

Claim: the earth is round. Label: yes. Claim: the sun is cold. Label: no. Published as a conference paper at ICLR 2025
Prompt
Fact: the earth is round. Label: yes. Fact: the sun is cold. Label: no.
Fact: [fact]. Label:
Expected answers
Answer: yes
Answer:
Format 3:
[correct, incorrect]
Prompt
Statement: the earth is round. Label: yes. Statement: the sun is cold. Label: no. Statement: {fact). Label:
Expected answers
Answer: correct
Answer: correct
Format [right, wrong]
Prompt
Claim: {fact). Label:
Expected answers
Answer: right
Answer: wrong
28


Reviews:
[{"review_id": "YzFlT82Ydc", "reviewer": "Reviewer_jLBW", "summary": "The paper explores improving LLMs' factuality and logical consistency through neuro-symbolic reasoning. It introduces a neuro-symbolic loss function that is used to fine-tune LLMs on a given set of external facts and rules. Experiments show that this approach achieves improved consistency and generalizes more effectively to unseen yet similar constraints compared to baseline methods, including those that rely on external reasoning tools.", "questions": "Please see \"Weaknesses\" section.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper offers a novel approach by integrating neuro-symbolic reasoning into the fine-tuning of large language models (LLMs) to improve factuality and logical consistency. While existing approaches for enhancing consistency in LLMs often rely on external reasoning tools or extensive fine-tuning, this paper proposes a middle-ground solution: a neuro-symbolic-based loss function that promotes logical consistency by maximizing the probability of constraint satisfaction. This approach (LoCo-LMs) is grounded in weighted model counting and semantic loss, offering a flexible framework that applies consistently across various logical constraints, such as negation and implication.\n\nThe paper conducts extensive experiments to showcase LoCo-LMs' effectiveness over traditional approaches, demonstrating improvements in logical consistency, factuality, and transferability across different logical constraints and datasets. The method also proves efficient, achieving good performance even with limited training data.\n\nBy enhancing logical consistency without requiring external reasoning frameworks, the approach has important implications for deploying LLMs in tasks that demand reliable, logic-based reasoning. Its ability to generalize to unseen (yet semantically similar) facts presents a promising pathway for real-world applications where models need to work reliably with sparse data.", "weaknesses": "Evaluation scope:\n\nThe experiments primarily focus on logical constraints such as negation, implication, and reverse implication. While these are fundamental, they fall short of capturing the more complex reasoning scenarios often required in real-world applications. For instance, the paper could improve by incorporating evaluations on multi-hop reasoning tasks or exploring more sophisticated logical constraints.\n\n\nShift in language modeling distribution:\n\nThe authors assess possible shifts in the language modeling distribution by measuring changes in perplexity, yet their evaluation could be expanded. Adding downstream tasks (e.g, question answering, reading comprehension, mathematical reasoning, etc.) would allow to assess whether the proposed fine-tuning approach not only improves logical consistency but also maintains the language capabilities of the original model.\n\n\nRobustness of the results:\n\nThe experiments reveal that fine-tuning LoCo-LMs improves generalization only within the same type of constraints, and it even hurts performance when the constraints differ between fine-tuning and testing (see Table 4). This limitation could be especially pronounced in smaller models, so testing on larger models could provide further insights. It would also be valuable to explore whether these performance gains also transfer to more capable models, such as comparing performance between LlaMa 2 and LLaMa 3, with and without LoCo-LMs.\n\n\nSensitivity to prompting:\n\nThe effectiveness of the approach appears to be sensitive to the specific prompt formats used during fine-tuning and evaluation. This suggests that the gains in consistency might be partially due to prompt selection rather than the model’s inherent logical coherence. Broader testing across diverse prompt templates would enhance the robustness and reproducibility of the results. Moreover, there are alternative prompting methods to elicit logical consistency, such as prompting the model to respond sequentially to a series of related questions, conditioned on previous answers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5KBzkztJmp", "reviewer": "Reviewer_hYcP", "summary": "This paper introduces LoCo-LLM, a fine-tuning method for LLMs that leverages a neuro-symbolic inspired semantic loss function to enhance its factuality and logical consistency. The proposed semantic loss function is based on weighted model counting, with weights derived from the LLM’s probability estimates. LoCo-LLM employs sentential decision diagrams to efficiently compute this loss.\n\nDetailed experiments compare LoCo-LLM with baselines that use external reasoners and traditional cross-entropy-based fine-tuning. Experimental results on the BeliefBank and EntailmentBank datasets show that the proposed framework outperforms baselines on metrics such as factuality and consistency.\n\nThe code to reproduce these results is provided as supplementary material and will be released on GitHub under a permissible license.", "questions": "- For the pre-trained baseline models in Tables 1 and 2, do the scores improve with greedy decoding?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The idea of using a neuro-symbolic loss function to improve logical consistency and factuality in LLM responses is novel and interesting. The proposed loss function is generalizable, can be extended to complex logical constraints, and may prove useful in enhancing LLMs' reasoning capabilities.\n- The detailed experimental results demonstrate the advantages of the proposed method over baselines, even on relatively small (5-10%) datasets.", "weaknesses": "- Although the loss function is explained thoroughly, other components, such as circuits and sentential decision diagrams, are not discussed in detail. Including these details would improve the paper's readability.\n\n- The experiments are conducted on datasets with outputs of fewer than 4 tokens, leaving it unclear how well the proposed method supports generating longer, factually and logically consistent responses.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cYb4OD6Vxl", "reviewer": "Reviewer_g2CN", "summary": "This work proposes a fine tuning method for improving logical consistency in language models. Given a set of facts and a set of constraints the idea is to finetune the model to make sure that certain logical constraints are respected, typically implication and negation. The authors show that indeed finetuning allows to improve self consistency and that this transfers beyond the facts and constraints used for finetuning to other entities and and settings.", "questions": "* Line 192: the authors claim that they expect transfer from albatross to cockerel since they are similar - but there is no definition of what is similarity, and how should the model know when things are similar enough to conclude new facts about entities and when not. I assume this refers to some vague simlarity measure in the space of hidden representations, but this is still confusing.\n\n* Line 469 - where are the resutls? are they in Table 3? the paper doesn't say\n\n* What is the few-shot baselines precisely? what are the examples given and how are they helpful?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The problem of improving logical consistency in language models is important. The approach is simple and does not require a lot of inference time compute since it is based on finetuning. The empirical results that show transfer and generalization beyond the training distribution are informative and interesting.", "weaknesses": "* Clarity: the paper can do a better job at explaining the details of its method. The authors spend two pages (section 2) on explaining logical constraints in a way that is too elaborate (for example, defining the xor operator in line 124, and defining implication in terms of negation and or in line 137) and unnecessary. On the other hand details on the actual method is limited (see questions below), specifically the paragraph in 232 and the precise process of how logical constraints are transformed into differentiable graphs are explained in a manner that is insufficient. The description of the experiments also mixes unimportant implementation details with more important details on the experimental setup which makes it hard to understand the details of the experiments and what can be concluded from them.\n\n* Related to the above - Figure 1 takes a lot of real-estate but is not helpful. The only thing we see is that there is baseline that makes a mistake on 3 examples and the proposed model does not make the mistake. This does not say a lot on the method, or the aggregate results only we can learn about the types of logical constraints that will be used. This might be ok if the important parts of the paper were clear, but they are not sufficiently clear at this point.\n\n* Key point that was unclear to me:  line 242 paragraph: I don’t understand if the method handles facts that can be inferred from \\alpha and the KB but require more than one hop? When training the SL loss, are those considered? Say we have in the KB, “albatross is a bird”, “birds are an animal”, “albatross can fly”, “if an animal can fly then the animal can move”. Will the SL loss contain a term about whether albatrosses and whether they can move or not? Is this done implicitly somehow? Where do we do the inference of all potential things that can be inferred from the KB and the constraints and take those into account in the SL loss?\n\n* More on clarity: in section 3, you define \\mathcal{D}_c = \\{alpha_1, \\dots, \\alpha_m\\}. But the structure of \\alpha is not clearl defined. It would be gold ot make this much clearer, it becomes clearer later as you read more, but should be explained better at this point.\n\n* Clarity: z ~p_\\theta(z) is confusing. Supposedly p_theta is the language model and it look like sampling from the unconditional distribution of text, but the text says something else, that it is sampling truth assignments conditioned on what appears in \\alpha_i but this is not clear from the notation.\n\nAnother key point are some problems with clarity and worries about the experimental setup. \n\n* IIUC the only baseline that is reported that is not from the authors is ConCord for which two numbers exactly are reported and that's it. There is some reference to maieutic prompting but it is unclear if this should be another baseline or is too similar to ConCord. It is not clear if there are not reasonable baselines to compare other than that. There is reference to few-shot baselines, but it is not expalined what are the examples in the few-shot examples and how they are supposed to help, in fact in many cases results are worse for few-shot compared to zero-shot. Overall, the authors should make clear if there is no past work beyond ConCord and just finetuning on the KB (XENT) without using the constraints\n\n* Second, for ConCord, it seems that the authors use ROBERTA-ANLI as an inference model. But for their LOCO method it seems like they are using hard constraints that are guaranteed to be true - if that's the case this is unfair towards ConCORD. Can the authors provide more details about how and why they outperform ConCord? Do the two methods use the same models and same constraints? Form the fact that the authors say that Concord requires ROBERTA-ANLI it sounds like the answer is \"no\" but would be good to understand better what's going on. Since we only have two numbers in the paper that are not baselines implemented by the authors it is important to understand the details in this setup.\n\nTo conclude, I found the overall premise of the paper interesting but the paper needs to be clearer both in terms of method and in terms of experimental results and how they relate to past work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lPkvUVcIrR", "reviewer": "Reviewer_nHXh", "summary": "The paper describes an approach for improving the logical consistency and factuality of language models, using neuro-symbolic integration.\nThe paper starts with a list of facts and logical constraints. All the valid combinations of truth values for these facts are then iterated and used as targets during optimization.\nThe experiments evaluating the correctness and consistency of the learned facts show that this method outperforms vanilla models and a baseline using an external solver.", "questions": "The method relies on collecting the probabilities for specific tokens to estimate the yes/no probabilties.\nHow much is this going to be affected by the label bias of the LLMs?\nhttps://openreview.net/forum?id=shr9PXz7T0\nhttps://arxiv.org/pdf/2402.09910", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is making advancements in neurosymbolic modelling.\nIt is certainly a nice achievent to not have to rely on an external solver and being able to push the knowledge into the main neural model.", "weaknesses": "The evaluation is the weakpoint of the paper at the moment.\n\nMacaw-Large, which is used for the main experiments, is quite old already (pre-LLM).\nEven Llama-2 used in later experiments is much less capable on most tasks compared to the current Llama-3.2.\nThis raises questions how applicable the proposed methods are to the current generation of language models.\n\nThe main baseline is CONCORD, which is from 2019 and uses RoBERTa. \nThe fact that the proposed system is able to outperform this baseline without using an external solver is great.\nBut there really should be some additional baselines with newer methods that also use model updating.\nFor example, there is a whole library of papers focussing on updating specific facts in language models using targeted fine-tuning.\n\nThe whole evaluation is performed on very artificial tasks. It would be very useful to see how these changes impact the model performance in practical applications.\n\n\nAsking the LLM “Is an albatross not an organism?” is a very unnatural phrasing, whereas LMs are trained to predict natural continuations. I suspect that may be negatively affecting the performance for LMs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jIgfIdt9U5", "reviewer": "Reviewer_G8eP", "summary": "This paper introduces LOCO-LMS, a fine-tuning method grounded in neural-symbolic reasoning, which significantly enhances the logical consistency and factuality of LLMs by integrating logical constraints as loss functions during training. Unlike traditional methods that rely on external reasoning tools, LOCO-LMS internalizes logical rules, allowing the model to reason independently and improving overall efficiency.", "questions": "1. Can LOCO-LMS be adapted for more complex, multi-level, or nonlinear logical reasoning scenarios?\n\n2. How well does LOCO-LMS integrate with existing knowledge editing methods when it comes to incorporating new facts or updating knowledge bases?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.  LOCO-LMS effectively improves the model's logical consistency, accommodating complex logical relationships such as positive implication, reverse implication, and negation. This alignment with common sense enhances the quality of responses generated by LLMs.\n\n2.  By incorporating semantic loss, the method minimizes reliance on external reasoning tools, thereby lowering reasoning costs and increasing inference speed.", "weaknesses": "1. The model assumes that facts are conditionally independent under a given model state, but in actual applications, there may be dependencies between facts, and this assumption may affect the consistency effect.\n\n2. While it addresses factual inconsistencies in the Llama-7B model, I also concern that its efficiency and scalability may lag behind approaches based on RAG and knowledge editing.\n\n3. LOCO-LMS is designed for specific tasks and fine-tuning, which limits its applicability to more complex reasoning tasks. Additionally, it may be vulnerable to attacks, such as just-in-time injection.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)