Language Models with Rationality
Authors: Nora Kassner
Oyvind Tafjord
Ashish Sabharwal
Kyle Richardson
Hinrich Schuetze
Peter Clark
Venue: EMNLP 2023 Main (2023)
Keywords: Interpretability
question answering
belief
entailment
belief graphs
consistency
TL;DR: To improve interpretability, we show that by adding a rational layer to a LLM, we can materialize it's "belief graph", repair inconsistencies, and produce faithful reasoning chains drawn from a belief system with significantly improved consistency.

Abstract: While large language models (LLMs) are proficient at question-answering (QA), it is not always clear how (or even if) an answer follows from their latent "beliefs". This lack of interpretability is a growing impediment to widespread use of LLMs. To address this, our goals are to make model beliefs and their inferential relationships explicit, and to resolve inconsistencies that may exist, so that answers are supported by interpretable chains of reasoning drawn from a consistent network of beliefs. Our approach, which we call REFLEX, is to add a **rational, self-reflecting layer** on top of the LLM. First, given a question, we construct a **belief graph** using a backward-chaining process to materialize relevant model beliefs (including beliefs about answer candidates) and their inferential relationships. Second, we identify and minimize contradictions in that graph using a formal constraint reasoner. We find that REFLEX significantly improves consistency (by 8%-11% absolute) without harming overall answer accuracy, resulting in answers supported by faithful chains of reasoning drawn from a more consistent belief system. This suggests a new style of system architecture in which an LLM extended with a rational layer can provide an interpretable window into system beliefs, add a systematic reasoning capability, and repair latent inconsistencies present in the LLM.

I Language Models with Rationality
Nora Kassner
Oyvind Tafjord' Ashish Sabharwal' Peter Clark!
Kyle Richardson'
Hinrich Schütze?
Allen Institute for AI, Seattle, WA Center for Information and Language Processing, LMU Munich, Germany
kassnerêcis ddu foyvindt,ashishhs kyler, peterc)@@aleenii.org
Abstract
Question:
Which animal gives live birth? (A) giraffe (B) spider
While large language models (LLMs) are pro- ficient at question-answering (QA), it is not always clear how (or even if) an answer fol- lows from their latent "beliefs". This lack of interpretability is growing impediment to widespread use of L.MMs. To address this, our goals are make model beliefs and their in- ferential relationships explicit, and to resolve inconsistencies that may exist, so that answers are supported by interpretable chains of reason- ing drawn from a consistent network of beliefs. Our approach, which we call REFLEX, is to add a rational, self-reftecting layer on top of the
Direct model answers:
S1: qiraffes qive live birth? true
S2: spiders give live birth? true
"Rational" Layer:
R
R
LLM Layer: SP
LLM
LLM
LLM. First, given question. we construct be- lief graph using backward-chaining process materialize relevant model beliefs (includ- ing heliefs about answer candidates) and their inferential relationships. Second, we identify and minimize contradictions in that graph us- ing a formal constraint reasoner. We find that REBLEX significantly improves consistency (by 8 11% absolute) without harming overall an- swer accuracy, resulting in answers supported by faithful chains of reasoning drawn from more consistent belief system. This suggests a new style of system architecture in which an LLM extended with a rational layer provide an interpretable window into system beliefs, add systematic reasoning capability, and re- pair latent inconsistencies present in the LLM.
(a) Generate Belief Graph (b) Resolve Inconsistencies
S1: giraffes give live birth
S6: a spider rodent
$2: spiders give live
S7: rodents memmals
$3: mammals give live birth
R1: S3 $4 S1
S4: giraffe mammal
R2: S5 S6 S2
$5: rodents give live birth
R3: S3 S7 S5
Figure I: (Top) When queried about each answer option independently, the model incorrectly believes both are true, and is more confident in the wrong answer (S2). (Bottom) REFLEX adds "rational" layer above the LLM layer, in which belief graph is constructed (by iteratively querying the LLM, up/down arrows), contain- ing relevant model-believed facts (white/grey believed T/F) and their inferential relationships. Inconsistencies are then identified (red) and minimized by constraint reasoner that flips T/F labels on beliefs (green v v/X). here resulting in the correct answer (S1, green box) explanation (graph) by the overall system (blue).
1 Introduction
While large language models (LLMs)) are impres- sive question-answerinn
where properties of explainability, interpretability. and trust are paramount. Our goal is to help alle- viate such opacity by constructing an explicit rep- resentation of system beliefs and their inferential relationships (including to answer candidates), so that answers are supported by interpretable chains of reasoning. These constructed belief graphs. e.g., Figures and 2, form rational layer above the L.M explaining how answers follow from be- liefs, and provide a window into some of the latent contents of the model, potentially helping users
(QA), it is not always clear how (or even if) an answer follows from their latent "beliefs" about the world, or whether the LLM even has a coherent internal belief system. This general opacity is a growing impediment to widespread use of LLMs, e.g., in critical applica- tions such as medicine, law, and hiring decisions,
We adopt a simple definition of belief, namely that a model believes X if it answers "yes" to the question "Is X truc?". Other definitions could also he used; see Section 2.

---

understand and trust model answers.
value on S2 from T to F (Figure 1, right), repairing the contradiction. This builds on earlier techniques (K (Kassner et al., 2021; Mitchell et al., 2022; Jung et al., 2022), though in a notably richer setting with over 350 nodes and 80 constraints per question, joint inference across answer candidates, and va- riety of constraint types. The overall result is fully autonomous, self-reflective system that is able to deliberate (and if necessary change) its answers, thereby resolving latent inconsistencies that would otherwise go unnoticed, and provide faithful expla- nations drawn from a consistent belief system.
In addition, when we do this, we find such graphs expose latent inconsistencies in the model's beliefs. We show how such inconsistencies can be resolved using constraint satisfaction techniques. When we do this, the rational layer becomes not just win- dow onto the model, but an active reasoning com- ponent in its own right in larger, overall system. comprising the (frozen) LLM plus rational layer (blue box, Figure 1). We show this results in a more consistent set of belief's in the overall system, without harming overall answer accuracy (although some individual answers may change). The result is answers supported by faithful, system-believed chains of reasoning drawn from consistent belief' system.
We evaluate our implementation of REFLEX on three datasets: EntailmentBank (Dalvi et al.. 2021), OBQA (Mihaylov et al., 201 8), and QuaRTz (Tafjord et al., 2019). We find that REFLEX is able construct belief graphs with significantly improved consistency (by 8%-11% %% absolute) with- out harming overall answer accuracy. In addition, answers are now supported by more consistent, system-believed chain of reasoning, providing a window into the previously latent beliefs of the model. Our contributions are thus:
Our approach, called REFLEX, introduces a ra- tional layer consisting of two parts. First, to pro- duce a belief graph, we recursively ask the LLM to explain why each candidate answer might be true, expressed as a set of sentences that entail the answer. This builds on carlier work on general- ing entailment--ased and chain-of- explana- tions (Tafford et al., 2022; Weir and Durme, 2022; Wei et al., 2022). We then add self-verification step to check that the model itself believes those generations (i.e., that the model believes what it says), allowing us to identify sentenees reflecting the model's own internal knowledge. For example, in Figure 1, when asked to explain S1 ("giraffes give live birth"), the model generates S7 ([because] "mammals give live birth") and S4 ([and] "a giraffe is a mammal"). Self-querying then checks if the model actually believes its generations ("Do mam- mals give live birth?"). The answer ("yes"/"no") assigns a true/false (T/F) value to each generation, indicated in Figure 1 by white/grey nodes. This procedure is then applied recursively to the gener- ated, supporting sentences. The resulting network of model beliefs and their dependencies provides a window into the model.
1. A new style of system architecture in which an LLM is extended with a rational layer in which an explicit representation of system beliefs and relationships is constructed and which can be reasoned over. This layer pro- vides an interpretable window into system beliefs, adds a systematic reasoning capablity, and allows latent inconsistencies present in the LLM to be repaired.
2. An implementation of this architecture demon- strating that the consistency of the overall system's network of beliefs can be signif- icantly improved without harming answer accuracy. Answers are now supported by ex- plicit, interpretable chains of reasoning drawn from a more consistent network of beliefs.
2 Related Work
Materializing a Model's Internal Knowledge: It is now well recognized that L.MSs contain exten- sive world knowledge (Petroni et al., 2019, 2020; Davison et al., 2019; Peters et al., 2019; Jiang et al., 2020; Roberts et al., 2020) that somehow enables them to perform well. Recent work has attempted to expose that knowledge in various ways, both to justify answers and improve performance, and our work falls into this genre. Standard explana- tion generation methods (Wiegreffe and Marasović, 2021) can produce compelling explanations, but
Second, we apply a formal constraint reasoner to this graph to resolve inconsistencies, by find- ing the optimal (minimal cost, Section 3.3) way of flipping T/F values. For example, on the left in Figure 1, S2 and S3 ("spiders do/don't it give live birth") are in an XOR relationship (i.e., exactly one must be false), but both are believed as true (white) by the LLM a latent contradiction within the LLM. Constraint reasoning then seeks to re- move such inconsistencies, here flipping the belief'
2

---

Question:
Heating ice (A) changes it's chemical make-up (B) will leave a puddle [correct] (C) makes even colder
Growing the Belief Graph:
A
1.Heating changes chemical make up T
2. Heating
puddle F
3. Healing ice makes even colder
F
Heating
5. Ice
Hesting ice ice to mell. T
Melling will result
8. The apposite heating something makiing
9: Healling ios makes il colider
changes substance's chemical make up T
subslanoot
T
B
1. Healligg loe changes chericaal make-up
Heating ice will leave puddle F
3. Heelling ice makees colder
C
(Part of) The Final Belief Graph:
15. Heal cannot cause sold subslance to mell.
Adding subslance does not change that substance's chem- makee-up
Heating substance changes that substance s chemical make up
xar
10, puddle is formed iiuui water guthering logeiher T
13.
14. Heel can solid
16. heat energy is addedd to
a kind of solid T
T
T
11. Melting usil produce lqqid water
xor
Heating cause the ice to melt.
21. Heating subssance does not change that substance's chem- make-up
kind of subeianceT
not tauee the ice to melt
Melting will result in puddle
1. Heeling ice changes chemical make-upp
12. Motting produce liquid water
2. Heating ice will
18. Heating lce will meit the F
20. Melling leaving puddle behind
The opposile of healing something is making
F
18. Heating ice will puddlef
3. Heatling ice makes it iteven colder
mekes It colder F
Figure 2: Given a question, each answer choice is first converted to hypothesis statement (A). The belief graph is then constructed in stages, first generating rules that conclude the hypotheses (B), then back backward-chaining to generate rules concluding the premises of those first rules, etc., and adding negated versions of graph statements connected with the originals via XOR links (e.g., nodes 11 and 12), until the stopping criterion is met (C). Statements are then labeled with the model's belief in them (true/false), found via self-querying (white believed true, grey = used believed false). Finally, logical conflicts identified (colored red), and constraint satisfaction techniques to resolve them. In this case, as there is strong evidence that node 2 is actually true (7 & + 2, not(19) + 2), the solver finds that minimum cost repair flip node 2's label from FALSE TRUE, Here, node ends up being selected as the final answer, thus correctly answering the original question.
with no guarantee that the generated sequence of tokens expresses the model's s internal knowledge, nor entails the actual answer. Similarly, chain-of- thought (CoT) (Wei et al., 2022) and Least-to- to-t (Zhou et al., 2023) prompting generate (in different ways) step-by-step reasoning chain along with an answer, but again with no claim that the chain reflects the model's internal knowledge nor is valid reasoning (Subramanian al., 2020).
reflect model- facts (by self-querying "Is true?") (e.g., Kassner et al., 2021: Jung et al., 2022), or model-believed rules (by self-querying "Does p imply q?") (e.g., Tafjord et al., 2022). We build on these to construct a belief graph, namely network of model-believer facts and their infer- ential relationships, which can then be reflected on.
Beliefs: We refer the model's factual opin- ions as "beliefs" rather than "knowledge" because those opinions may wrong. general, an agent
To add semantics to generations, several systems have used queryingginnggggg to verify that generations
3

---

can be said to believe p if it acts as if p was true (Schwitzgebel, 2019). Following Kassner et al. (2021) and Richardson et al. (2022), we take sim- ple, syntactic operationalizatinn of this, namely the agent answers "yes" to the question "p?", but also note that more semantic versions could be used, e.g., the agent also answers "yes" to paraphrases and implications of p.
with facts. Our goal is extract a model's ini- tial beliefs about statements inferentially related to all top-level hypotheses of interest, and perform reasoning to update these beliefs so as to make them more consistent with respect to the rules, and ideally also factually more accurate.
A belief graph is type of factor graph com- monly used in the probabilistic inference litera- ture (Loeliger, 2004). Formally, is defined as an undirected graph and edges E. Nodes are of two types: A state- ment node (referred to as a "variable node" in a factor graph) is a triple (s,4, (g,) containing a natural language statement 8, associated value I E {T, F) initially denoting M's belief that is true or false, and confidence C3 E [0, 1] denoting confidence in that label. A rule node (referred to as a "factor node" in a factor graph) is a pair (r, c,) denoting a disjunctive rule or constraint over statements, with confidence G-- takes the form r = (-81 V... V -8 V 88+1 V... V 8k). For ease of interpretation, we view this constraint as r= p h where p 81 .As is conjunctive premise and h 82+1 V ... 5 is disjunctive hypothesis. The rule says that if p is true, so must be h; and the contrapositive of this.
Reducing Inconsistency: LLMs are known to be inconsistent in their answers (Ettinger, 2020;
(N, E) with nodes N
Kassner and Schüttze, 2020; Davison et al., 2019; Ravichander et al., 2020; Elazar et al., 2021; Sub- ramanian et al., 2020; Gu et al., 2023), and several recent works have used constraint reasoners to iden- tify and reduce inconsistency. BeliefBank used a MaxSAT solver to resolve inconsistencies between model beliefs, but required hand-provided set of constraint rules (Kassner al., 2021). ConCoRD (Mitchell et al., 2022) similarly used MaxSAT to ensure model answers were consistent with NLI- derived entailment constraints between them, but did not introduce additional model-believed facts and rules. Maicutic Prompting (Jung et al., 2022) also used MaxSAT to resolve inconsistencies be- tween facts in prompt-induced explanation chains. However, those chains were not validated re- flecting model-believed constraint rules?, and did not support conjunction. REFLEX extends these reasoning chains to provide a full semantic account of how answers are supported by the model's in- ternal knowledge. Additionally, it performs joint reasoning across answer candidates and operates at much larger scale (c.g., over 350 nodes on average for each question) and with variety of constraint types.
Edges F connect rule nodes to the statements they constrain, denoting their dependence. For leg- ibility, we draw edges directionally to depict the way the rule reads: the statements in p point to r, which in turn points to h. Mathematically, the influ- ence is bidirectional and the depicted directionality is irrelevant during reasoning (Section 3.3). just as in a standard factor graph.
We adopt the standard probabilistic semantics of factor graphs, thereby associating belief graph with well-defined probability distribution over any set of statement beliefs. For statement node (s, I, cs), the cost cost, for setting it to / is O, and that for setting it against / is c,; the corresponding weight of this node is w. = exp(- cocct)) Costs and weights for a rule node (r, C7) are defined similarly, based on whether the beliefs satisfy or not. Finally, the overall weight of a T/F assign- ment to all statements is II, I,, II. Wr, which, when normalized by the total weight across all pos- sible assignments, yields a probability distribution over such assignments. We will be interested in finding the most consistent set of beliefs, i.e., a T/F assignment to statements with the minimum overall weight, which equivalent to minimizing
3 REFLEX: Our Approach
3.1 Belief Graphs
Our belief graphs are defined over a set of natu- ral language true/false statements and represent a set of rules that constrain the truth values of these statements. We refer to statements that are fac- tually true in the world facts. The truth value assigned by model M to statement is referred as M 's belief in that statement (cf. Footnote 1). A model's internal beliefs may not always align
PREFLEX checks whether both the statements st, and the rules (8; - h), are believed by the model via self-querying, e.g.. by asking "Does s; - A??". and also scores the strength of those beliefs. In maicutic prompting, the generated rules are not checked against the model, resulting in rules that the model itself may not believe, if queried about them.
4

---

E. costs + E,, cost,. This is referred to as the MPE (most probable explanation) problem in the graphical models literature, which we later solve exactly using a MaxSAT constraint solver based on a standard translation of MPE into weighted MaxSAT (Park, 2002; Sang et al., 2007).
Algorithm 1 The recursive algorithm for construct- ing belief graph of max depth dmea for hypoth- esis set H. The subroutine EXTEND-GRAPH takes a partial graph G as an input and extends it in place with one statement and its subgraph.
1: procedure GENER SENERATE-GRAPH(Hyeee ARAhhyyeeete H. max depth dmax)::
3.2 Constructing Belief Graphs
2:
let G empty graph
3: foreach E 1
Given an initial node (statement) s, a belief graph G is produced by mockward-chaining process de- scribed below, in which recursively expanded to add statements that together may entail s.
4: call EXTEND-GRAPH(L, O, dmax.. G)
5:
add MC rule node (V KeN h, 20) to G
6: foreach pair (h4, of hypotheses in H
add MC rule node (-h, V mhy, Cmme) to
return G
3.2.1 Basic Operations
procedure
s, current depth
d, max depth deax. partial graph G):
Let h denote a hypothesis (language statement s) of interest and p a premise-a a set of statements
10:
call operator = (1. c.) to score statement
11:
add statement node (s, cx) G
12:
gen. negation sentence negs = neg(s)
(s. that together may entail h. Given these,
13:
add rule node (XOR(s, negs), Geor G call EXTEND-GR BHAND--GAAPHHnggg,,, d dmaxs G)
there are three basic operations required to gener- ate belief graphs:
14:
15:
if < dmas do:
16:
let h
1. h + p: Given h, generate a that may entail h.
17:
call operator >> y to generate p
2. > cl): Given a statement 8, output true/false value l and a confidence in the belief
18:
call operator (p. h) = c, to score rule p + h add rule node (p + h, cx) to G
19:
20:
foreach 8; E p
that has truth value (as assessed via yes/no question-answeringg)
21:
call EXTEND-GRAPH(EE dmmax G)
3. (p. h) + G: Given p and h, output a confidence that the candidate rule r = p >> holds. The most important of these the first operation, in which the model self-generates conjunctive rules concluding h (i.e., reason p for believing h), thus adding new nodes to the graph.
model's confidence (Kadavath et al., 2022).
3.2.2 Initial Hypothesis Generation
Given question, we first generate a set H of hy- pothesis sentences (c.g., "Is the sky (A) blue (B) yellow" + { h1 = "The sky is blue.", h2= h2 "The sky is yellow."). An N-way multiple choice question yields N hypotheses in H. A true/false question yields 2 hypotheses. To handle open-ended ques- tions, candidate answers can be generated, c.g., using nucleus sampling (Holtzman et al., 2019).
There are several ways of implementing these basic functions, and our algorithm is agnostic to the method used. In our work here, we use Entailer, an off-the-shelf T5-11B trained model with these functionalities (Tafjord et al., 2022). Further, since the raw score produced by the model tends to be skewed towards 0 or 1, when computing c, and o, in practice, we re-scale the raw model score using a set of hyperparameters (ef. Appendix B).
3.2.3 Belief Graph Generation
The belief graph generation process is shown Algorithm 1. An example of (part of) a generated belief graph is shown in Figure 2.
One may use alternative ways to implement these operators, such as chain-of-thought prompt- ing a model like GPT3 (Wei et al., 2022) or Chat- GPT (OpenAI, 2022). For example, to generate a rule concluding a hypothesis h such as "Plants require CO2 to make their food.", the model could be prompted with h followed by "Explain the last statement with 2-step reasoning chain.", the num- bered generations forming the premise p. Similarly, generated statements and rules can be validated as reflecting the model's 's beliefs by self-querying ("Is s truc?", "Does p imply h?"", and then using the generated yes/no answer token probabilities as the
Given set H of hypotheses, we generate a sin- gle belief graph G by using our basic operations (Section 3.2.1) to recursively generate rules that conclude each h4 E H. up to fixed maximum depth dmax. (Each original h; is at depth d = 0.) For each statement 8, we also generate nodes negs (and their recursive subgraphs) expressing its negation, c.g., "The sky is not blue." from "The
CConversion of a QA pair to a declarative hypothesis D uses a custom T5-1 18 model trained on the QA2D dataset (Demszky et al., 2018).
5

---

sky is blue.".* Each pair s and negs is connected with an XOR rule, indicating a (soft) preference for setting exactly one of them to true; this repre- sented as two disjunctive constraints (s V negs) and (-&V - -negs) whose weight Cxor is a fixed hyper- parameter. Lastly, we add a multiple-choice (MC) constraint which has two parts: hard constraint (with infinite cost) that at least one hypothesis must be chosen, and a soft constraint that no more than one should be chosen. The soft constraint is associ- ated with a fixed hyperparameter weight Cin-
how the internal reasoning of the LLM component operates.) Thus, REFLEX provides the user with an interpretable reasoning trace, allowing the user to understand how it derived the answer from more rudimentary facts (Subramanian et al., 2020).
We note that the original belief graph (before reasoning) may reveal that the model's original ex- planation is, in fact, not faithful to its own beliefs. For example, in Figure 2, the model believes state- ments 6, 7, and that 6 & 7 entail 2, but does not believe 2 (colored grey). Thus, the global reason- ing layer of REFLEX plays a critical role in arriving at faithful explanations.
3.3 Reasoning Over Belief Graphs
Belief graphs provide window into the model's beliefs about some of the relevant statements and their (believed) inferential relationships to candi- date answers question. As others have shown (Kassner et al., 2021; I; Mitchell al., 2022), such beliefs can be inconsistent, and materializing those inconsistencies provides one the opportunity to re- move or reduce them.
Experiments and Results
The goal of our experiments is to evaluate the ex- tent to which our overall system, namely an LLM plus a self-reflecting, rational layer, helps expose and resolve inconsistencies in the LLM's beliefs without harming accuracy. Importantly, REFLEX is evaluated in a zero-shot setting, without relying on training instances of the target datasets.
In similar vein, and as discussed in Section 1, REFLEX performs inference over belief graphs in order compute an updated set of beliefs that is consistent as possible with the rules. To this end, it converts belief graphs into an equivalent weighted MaxSAT problem and uses an off-the- shelf MaxSAT solver (RC2, (Ignatiev, 2019)) to compute the optimal flips of initial true/false belief's that minimize global inconsistency. It then discards all rules that are in conflict with the updated state- ment beliefs, obtaining smaller, updated belief graph. This smaller belief graph produced by REFLEX is self-consistent and provides inferential support for the tep-level hypotheses.
Datasets. We use the test partitions of three ex- isting multiple-choice datasets: EntailmentBank (Dalvi et al., 2021), OBQA (Mihaylov et al., 201 8), and QuaRTz (Tafjord et al., 2019). We chose our datasets as they contain inferentially rich questions (typically) requiring reasoning. The partitions con- tain 339, 500, and 784 examples, respectively.
The baseline LLM use is LLM that has been trained perform QA and also sup- ports the basic operations discussed in Sec. 3.2. 1, enabling us to assess how much it can be improved by adding a REFLEX layer. To this end, we use a publicly available, frozen, off-the-shelf T5-11B LLM called Entailer (Tafjord et al., 2022). To an- swer an MC question with this LLM, we score each answer hypothesis (c,, Scction 3.2.1) and select the one with the highest truth confidence. If Entailer assigns false values to all answer choices, we select the hypothesis with the lowest false confidence. REFLEX then adds a rational layer to this LLM, creating new system that is also able to self-reflect and modify its beliefs. To ensure the different be- lief graph scores in REFLEX appropriately cali- brated, we use nine hyperparameters, tuned once on the dev partition of EntailmentBank (Dalvi al., 2021) and then kept fixed for all experiments. De- tails are in Appendix B. Note the LLM itself re- mains frozen, with belief revision occurring in the
Models.
3.4 Generating Faithful Explanations
Notably, the smaller updated belief graph produced by REFLEX provides faithful explanation of the answer it predicts, in the sense that it accurately represents the reasoning process behind the overall system's prediction (Lyu et al., 2022). This is true as the MT reasoning process results precisely in a self-consistent set of beliefs from which RE- FLEX determines whether to believe candidate answer or not, and produces its final prediction based on this (rather than the LLM output alone; note that we do not make any claims about
*We use simple, custom-built utility for this, namely a ba-base model trained on 9k Turk-generated examples. Ssoft, to allow for cases with multiple valid answers, e.g., open-ended questions or those asking for the best answer.
6

---

rational (belief graph) layer above it.
Entail- mentBank
System
OBQA
Quartz
Metrics. For measuring self-consistency, we fol- low Li et al. (2019) and report the conditional con- straint violation (T) metric, defined as follows: the fraction of rules whose premises p are believed true,
LLM
79.4
74.0
80.2
LLM rational layer
79.9
75.0
80.0
(REFLEX)
but whose hypothesis h is not. In other words, over all rules of the form p h, is:
Table 2: QA accuracy: REFLEX's belief revision the rational layer preserves overall QA accuracy.
{p + h p - T, = FH
7
{p h
TH
To do this, given the belief graph for a question, we mask out (separately, rather than cumulatively) each type of rule in turn when providing the graph to the MaxSAT solver. We then run the constraint solver and measure the resulting self-consistency of beliefs on the original graph.
where T denotes the system believes statement 8 to true (similarly for 8 - F). The numerator r thus captures the number of constraints the system violates. The denominator captures the number of applicable constraints. We then report the following metric: consistency=11 1
System
EntailmentBank
For QA performance, multiple-choice accuracy: point for predicting the correct answer, 1/N points for predicting N answers including the correct one, 1/k points for no prediction (A: = answer options), 0 otherwise.
report standard
REFLEX (our system):
96.1
without p - / rules
93.8
without XOR rules
90.4
-
wwithout MC rule
95.8
Table 3: Consistency: Ablations on EntailmentBank (Dev) suggest that all three types of rules contribute improving self-consistency.
4.1 Results
Consistency. Table shows consistency results on the test partitions of our datasets. We observe significant consistency gains (by 8%-11% % abso- lute), showing REFLEX's effectiveness at creating a consistent belief network within the overall system.
The results are shown in Table (the MC rule is the constraint that exactly one multiple-choice option should be chosen, Section 3.2.3). The results indicate that all three types of rules contribute the system's consistency improvements.
Entail-
System
mentBank
OBQA
Quartz 85.7
LLM
87.0
88.2
4.2
Success Analysis
LLM + rational layer
96.1
95.9
96.6
(REFLEX)
We identify three classes of successful reasoning by the constraint reasoner: (a) latent model beliefs correct an initially wrong answer (Figure 3); (b) the system corrects an initially erroncous, latent model belief (Figure 4): and (c) strong model beliefs iden- tify and reject a bad rule (Figure 5). These types of system corrections help to improve accuracy and produce answers supported by valid chains of rea- soning, allowing users insight into why an answer follows from the model's knowledge.
Table 1: Consistency: By adding rational layer to the baseline LLM, REFLEX significantly improves consis- tency among beliefs by resolving uncovered conflicts.
Accuracy. Table 2 shows overall performance our three datasets (test partitions). As can be seen, we observe stable accuracy, as well as the answers now being faithful to the reasoning chains in the belief graph. This is significant, as it allows users to understand how answers follow from system beliefs (and in cases where an LLM belief was flipped, why that belief is untenable in the broader system).
4.3 Failure Analysis
Reasoning can also make mistakes. From man- ual analysis of 50 random questions from Entail- mentBank that REFLEX answered incorrectly, we identified five main causes of failure and their ap- proximate frequency (Note that multiple categories can apply, hence total is > 100%):
Ablations.
To study the impact of the three dif- ferent types of rules on consistency improvement, we using the EntilmentBank dataset (dev partition).
7

---

Heating ice (A) changes chemical make-up (B) leave puddle [correct] (C) makes even colder
Which of these objects will most likely float in water? (A) glass marble (B) steel ball (C) table tennis ball [correct]
Before reasoning
After reasoning
Before reasoning
After reasoning
Heallng
2. Melting will result
1. Heating ice will ice to melt.
2. Melling will result
object
2. Glass
If object
2. Glass marble is very light in weightf
ice will
light in weight that abjact will most likely float liquid
marble is very ligtt in weight
light weight that object will most likely liquid liquid water.
cause
Ice melt. T
puddle T
puddle
T
liquid water.
T
3. Heating ics will leave a puddle F
3. Heating ice will leave puddle T
marble will most likely float in water. F
marble will most likely
xor
xor
in water. F
4. Heating ice will not leave a pudkle
4. Heating ice will leave puddle
Figure 4: Example of good reasoning: Although the model correctly believes option (A) is false (grey, node
F
3), this answer conflicts with other beliefs (red). Rea- soning leads the system to realize that its weakest belief" (2) is actually false, correctly flipping its label from true (white) to false (grey, right side) restoring consistency.
Figure 3: Example of good reasoning: The model's beliefs in I and 2, and the rule I & 2 - 3, 3, as well as the xor constraint, causes to (desirably) flip its belief in 3 from false (grey, before) to true (white, after).
What build something over millions years? (A) river [correct] (B) person (C) society (D) dinosaurs
1. Missing Rules (=30%) %)): In some cases, the system generates irrelevant rules but misses an im- portant one needed to support the correct answer, resulting in incorrect conclusions. While somewhat subjective, this is a notable error category that we observe. For example for the question:
Before reasoning
After reasoning
person
Miliion of years is period time
1. person build some- thing over period of time T
Millions
build some-
years
thing a
period of
period of time T
time T
A human cannol survive the loss (A) The liver [correct] (B) A lung (C) A kidney the system incorrectly concludes (B) is true, ignor- ing the commonsense rule that with two lungs, person can survive without one of them.
3. A person can build something millions of years. F
A person
build something over millions of years.
Figure 5: Example of good reasoning: Here the rea- soner (desirably) chooses to reject the violated (bad) rule rather than flip belief, as the minimum cost way to restore consistency.
Incorrect Beliefs (=:0%): Sometimes the reasoner fails to correct incorrect model beliefs, either because the model's confidence is high or evidence against them is weak or missing. In the example shown in Figure 7, the model's strong, incorrect beliefs that "river deltas are reservoirs" and "reservoirs always provide freshwater" (untrue of oceans, say) causes it to incorrectly conclude that "deltas are freshwater reservoirs".
people don't mind not breathing for an hour."
4. Ambiguous Statements, Unexpected Reason- ing (x100)) A common cause of error the surprising ambiguity of belief statements, which can often be read in multiple ways. In several cases, the model adopts valid but unexpected interpre- tation, resulting in "errors" compared to the gold answer label. For example, in Figure 6, the model takes the word "always" in literal sense ("glaciers will not always be there"), resulting in an answer that differs from the gold label. Developing ways to attach context to these statements to help disam- biguate them would help alleviate such errors.
Incorrect Rules (=10%) Rule generation can produce bad rules, e.g., in Figure 5), and in some cases the constraint reasoner fails to reject them if they are strongly believed. In particular, confusion or ambiguity over quantifiers can result in bad rules, c.g., (emphasis added) "Some animals catch their prey with trickery." & "A spider is kind of animal." " + "Spiders catch their prey with trick- ery.". Similarly the model generates the fallacy: "Some people don't mind not moving for an hour" & "breathing is a kind of movement" + "Some
5. Multiple Valid Answers 10%0)): final cause of "error" at least with respect to the gold label is that multiple answers may be valid, and
8

---

Which type of water reservoir could always provide
Which type of water reservoir could always provide
freshwater? (A) river deltas (B) mountain glaciers
freshwater? (A) river deltas (B) mountain glacters
[correct] (C) tropical seas
[correct] (C) tropical seas
Before reasoning
After reasoning
Before reasoning
After reasoning
1.A river delta is kind water reservair.
river
water reservoir could always provide freshwater
Mountain
2. Glaciers
1. Mountain
Glaciers
reservoir could
delta is kind water reservair.
glaciers
provide a source of water used T meltingT
glaciers
provide
always provide
would not
would not always be there T for metting T
source of
freshwater T
always be there
water used
3. River deltas type water reservoir that could always provide freshwater
3. Rivar dallas are type water reservoir that could always provide freshwater T
Mountain glaciers could not allways provide freshwater
Mountain glaciers could not always provide freshwater
xor
xor
Failure due bad beliefs: The model strongly believes both and 2 (although both are f'actu- ally incorrect), here causing 3's label to undesirably flip from false (correct) to true (incorrect).
Figure 7:
4. Mountain glaciers could always provide freshwater T
4. Mountain glaciers always provide freshwater F
Figure 6: Unexpected reasoning: Here the model unex- pectedly pays particular attention the world "always". Because strongly believes that glaciers will not always be there (I, white), the system prefers to flip its beliefs in 3 and 4, rather than flipping I, thus rejecting answer option B (arguably correctly).
helping avoid similar inconsistencies in the future.
Conclusion
While LLMs perform well, the interdependencies between their answers and their other beliefs is opaque, and may even be in conflict. This lack of interpretability is a significant impediment to widespread use of LLMs. To reduce this opac- ity, and reduce these conflicts, we have proposed REFLEX, a new system architecture in which an explicit, interpretable representation of beliefs the belief graph is added as a rational layer above the LLM. This layer providing a window into sys- tem beliefs, and allows latent inconsistencies in the LLM alone to reasoned about and repaired. Our im- plementation shows that belief consistency of the overall system is significantly improved, without harming answer accuracy, resulting in answers sup- ported by interpretable chains of reasoning drawn from a more consistent belief system. This new architecture is an important step towards improv- ing confidence in system behavior, and towards trustable deployment of LLMs in practical applica- tions.
the question is asking for the best answer; eg. for "What could fill beach ball? (A) Oxygen (B) Wa- ter ...", A is labeled correct, while is also a valid answer. REFLEX (desirably) finds valid reasoning chains for both, but the notion of highest-scoring proof does not fully correlate with the notion of "best answer" intended by the question author.
5 Future Work
There are several impactful ways this work could be further extended. First, incorporating the ques- tion's 's context in the belief statements in our ratio- nal layer could make the semantics of the beliefs more precise, thus avoiding potential ambiguity in their truth value. Second, one could use the belief graph itself identify the key reasoning pieces that the LLM is most uncertain about. This could then guide a human-in--le -it--iii-----opppeee mechanism to correct or validate uncertain pieces via user interaction. Third, maintaining a persistent belief graph over multiple questions could help make the system more consis- tent across questions. This, in turn, would make a user's conversational experience with the system more coherent in longer dialog setting. Lastly, after resolving inconsistencies in the rational layer, we could consider propagating information back to the LLM layer in order to update it (via ane-ttning, model editing, memory-bart based architectures, etc.),
Limitations
We have shown how an LLM can be extended with a self-reflective component, allowing latent model knowledge to be made explicit in the form of a belief graph, providing window into the model's system of beliefs. While exciting, there are several limitations with the current work and opportunities for the future.
First, the reasoning component in the rational
9

---

layer can make mistakes, resulting in the overall system rejecting true statements or accepting false ones. A detailed analysis and classification of these failure modes was presented in Section 4.3.
Dorottya Demszky, Kelvin Guu, and Percy Liang. 2018. Transforming question answering datasets into natural language inference datasets. ArXiv,
Second, for our experiments, we used the T3- 1IB based Entailer system as the baseline LLM. While there is every reason to expect our pro- posed architecture to be effective in reducing in- consistency with newer and larger LLMs such as ChatGPT and LLaMA, this is still be evalu- ated. Doing so would require implementing the basic operations needed to construct belief graphs (Section 3.2.1) using instruction prompting and in- context learning. Other work has demonstrated such implementations (e.g., Wei et al., 2022; Jiang et al., 2020), making the outlook promising, but indeed their combination still needs be demon- strated at scale in an architecture like REFLEX. Lastly, we found consiseency-minimizee belief graphs to be highly valuable in understanding the system's i successes and failures. We expect these graphs to be a valuable starting point for provid- ing explanations and gaining a user's 's trust in the system. However, we have not conducted a formal user study to measure this.
Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, E. Hovy, Hinrich Schütze, and Yoav Goldberg. 2021. Mcasuring and improving consis- tency in pretrained language models. TACL, 9.
Allyson Ettinger. 2020. What BERT is not: Lessons from new suite of psycholinguistic diagnostics for language models. TACL, 8:34 48.
Yuling Gu, Bhavana Dalvi, and Peter Clark. 2023, Do language models have coherent mental models of everyday things? ACL.
Ari Holtzman, Jan Buys, Maxwell Forbes, and Yejin Choi. 2019. The curious case neural text degener- ation. In ICLR.
Alexey Ignatiev. 2019. RC2: an efficient MaxSAT solver. J. Satisf. Boolean Model. Comput., 11.
Zhenghao Jiang, Frank F Xu, Jun Araki, and Graham Neubig. 2020. How can we know what language models know? TACL,
Jaehun Jung, Lianhui Qin, Sean Welleck, Faeze Brah- man, Chandra Bhagavatula, Ronan Le Bras, and Yejin Choi. 2022. Maieutic prompting: Logically consistent reasoning with recursive explanations. In EMNLP.
Ethics Statement
Like any other project using LLMs, despite the best intentions there is a risk the model produc- ing biased or offensive statements as part of its explanations, and thus must be used with care and appropriate guards and warnings.
Saurav Kadavath, Tom Conerly, Amanda Askell, Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer. Zachary Dodds, Nova DasSarma, Eli Tran- Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yushi Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Heranndez, Josh Jacobson, John Kernion, Shauna Kravec, Liane Loviu, Kamal Ndousse, Catherine Olsson, Sam Ringer, Dario Amodei, Tom B. Brown, Jack Clark, Nicholas Joseph, Benjamin Mann, Sam McCandlish, Christopher Olah, and Jared Kaplan. 2022. Language models (mostly) know what they know. ArXiv, abs/2207 .05221.
Acknowledgements
This research was made possible, in part, by fund- ing from Open Philanthropy, the European Re- search Council (#740516) and by the German Fed- eral Ministry of Education and Research (BMBF) under Grant No. D1IS18036A, We also thank Google for providing the TPUs for conducting ex- periments. Finally, we are grateful for the valuable feedback from the anonymous reviewers.
Nora Kassner and H. Schütze. 2020. Negated and mis- primed probes for pretrained language models: Birds can talk, but cannot fly. ACL.
Nora Kassner, Oyvind Tafjord, Hinrich Schutze, and Peter Clark. 2021. BeliefBank: Adding memory to pre-trained language model for a systematic notion of belief. In EMNLP.
References
Bhavana Dalvi, Peter Alexander Jansen, Oyvind Tafjord, Zhengnan Xie, Hamnah Smith, Leighannaa Pipatanangkura, and Peter Clark. 2021. Explaining answers with entailment trees. In EMNLP.
Tao Li, Vivek Gupta, Maitrey Mehta, and Vivek Sriku- mar. 2019. logic-driven framework for consistency of neural models. In EMNLP.
Joe Davison, Joshua Feldman, and Alexander Rush. 2019. Commonsense knowledge mining from pre- trained models. In EMNLP.
AndreAndrea Loeliger. 2004. An introduction to factor graphs. IEEE Signal Processing Magazine, 21:28 https://people.
10

---

Qing Lyu, Marianna Apidianaki, and Chris Callison- Burch. 2022. Towards faithful model explanation in NLP: A survey. ArXiv, abs/2209.1 1326.
Oyvind Tafjord, Bhavana Dalvi, and Peter Clark. 2022. Entailer: Answering questions with faithful and truth- ful chains of reasoning. In EMNLP.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can suit of armor conduct elec- tricity? A new dataset for open book question an- swering. In EMNLP.
Oyvind Tafjord, Matt Gardner, Kevin Lin, and Peter Clark. 2019. QuaRTz: An open-domain dataset of qualitative relationship questions. In EMNLP.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoe Le, and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models. In NewrIPS.
Eric Mitchell, Joseph J. Noh, Siyan Li, William Armstrong, Ananth Agarwal, Patrick Liu, Chelsea Finn, and Christopher Manning. 2022. Enhancing self-consistency and performance of pre-trained lan- guage models through natural language inference. In EMNLP.
Nathaniel Weir and Benjamin Van Durme. 2022. Dynamic generation of interpretable inference rules neuro-symbolic expert system. ArXiv, abs/2209.07662.
OpenAL. 2022. ChatGPT: Optimizing language mod- els for dialog. Technical report, openai.com. https///opnaa.ccom/bogeeeee com/blog//hatggp/.
Sarah Wiegreffe and Ana Marasović. 2021. Teach me to explain: A review of datasets for explainable nat- ural language processing. NewrIPS Datasets and Benchmarks.
James D Park. 2002. Using weighted MAX-SAT en- gines solve MPE. In AAAI/IAAI.
Matthew E Peters, Mark Neumann, Robert Logan, Roy Schwartz, Vidur Joshi, Sameer Singh, and Noah A Smith. 2019. Knowledge enhanced contextual word representations. In EMNLP.
Denny Zhou, Nathanael Scharli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang. Dale Schuurmans, Olivier Bousquet, Quoe Le, and Huai hsin Chi. 2023. Least-to-most prompting enables complex rea- soning in large language models. ICLR.
Fabio Petroni, Patrick Lewis, Aleksandra Piktus, Tim Rocktäschel, Yuxiang Wu, Alexander H. Miller, and Sebastian Riedel. 2020. How context affects lan- guage models' factual predictions. In Automated Knowledge Base Construction.
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowl- edge bases? In EMNLP.
Abhilasha Ravichander, Eduard Hovy, Kaheer Suleman, Adam Trischler, and Jackie Chi Kit Cheung. 2020. On systematicity of probing contextualized word representations: The case of hypernymy in BERT. In Proceedings of the Ninth Joint Conference Lexical and Computational Semantics.
Kyle Richardson, Ronen Tamari, Oren Sultan, Reut Tsarfaty, Dafna Shahaf, and Ashish Sabharwal. 2022. Breakpoint Transformers for Modeling and Tracking Intermediate Beliefs. In EMNLP.
Adam Roberts, Colin Raffel, and Noam Shazeer. 2020. How much knowledge can you pack into the parame- ters of language model? In EMNLP.
Tian Sang, Paul Beame, and Henry A Kautz. 2007. dynamic approach for MPE and weighted MAX-SAT.
IJCAL.
Eric Schwitzgebel. 2019.
Belief. Stan- ford Encyclopedia of https://plato.stamit
Philosophy.
Sanjay Subramanian, Ben Bogin, Nitish Gupta, Tomer Wolfson, Sameer Singh, Jonathan Berant, and Matt Gardner. 2020. Obtaining faithful interpretations from compositional neural networks. ACL.

---

A Additional Results
Hyperparameter
Value
k
9
We report results on the dev set of the Entailment- Bank dataset in Table A1.
Kentailer
36
kxor
30
kmc
System
EntailmentBank (dev)
Consistency
Accuracy
fentailer
1.02
LLM
87.5
78.6
(xor
1.1
tme
0.98
LLM + rational layer
(REFLEX)
96.1
81.8
Mxxor
0.3
dmax
5
Table Al: Results on EntailmentBank (dev), used to tune the system's hyperparameters.
Table B1: Hyperparameters.
we remove (ignore) those where there is signif- icant uncertainty, namely where (score(s:) score(negs;) I Mror. where myor is tuned hyperparmeter.
B Hyperparameters and Runtime
MaxSAT finds the optimal assignment of true/false labels statement nodes that minimizes the total penalty of constraint violations. If the true/false la- bel on a statement node is flipped, then the penalty is the model confidence Cg in the original label. Similarly if a rule (constraint) violated by the true/false labels on its associated statements, then the penalty is the model confidence C, in that rule.
Additionally, we tune damping parameter that downscales rules on the boundary of the graph. Belief nodes involved in these rules are not supported by any premises and should therefore have less influence than rules with strong support.
6. Finally, we tune the maximum depth dmax of the belief graph.
We set number of hyperparameters to ensure that the various sources of confidence are appropri- ately balanced, and tune these on a development set (EntailmentBank (dev) which is separate from our test sets). We use the same set of hyperparameters for all test sets.
The performance on this dev partition is shown in Table A and the hyperparameter values are shown in Table B1.
The runtime for MaxSAT constraint solving is fast (<1 millisecond per question). However, con- structing the belief graph is computationally in- tensive: Each call to expand score node takes ~ seconds, and our graphs typically contain ~600 nodes, so if these calls were maximally parallelized. with each step growing the graph one level deeper, the runtime would be the maximum graph depth (5) x 2 seconds = ~0 seconds total (or several minutes if naive sequential implementation were used).
1. As raw model confidences
are highly skewed towards 0 and 1, we re-calibrate these with (el.-1)) where k is fixed hyperparam- eter. Note, that for the MC and XOR rule, the raw input score is 1.0.
2. We calibrate rule confidences in the same way as we calibrate belief confidences use sep- arate calibration parameters different types of rules namely:
Entailer rules p +
XOR rules
MC rules
i.e., the raw rule score e is re-calibrated to confidence (c-pe-cc-1) 1) where keype is the re-
spective hyperparameter per rule type.
3. We set three hyperparameters tuning the re- spective importance of the three different types of rules. Therefore, the final rule score is computed by type (eppe((e-1) where teype the respective hyperparameter constant per rule type.
4. For xor rules between statements 8; and negs;,
12


Reviews:
[{"review_id": "sC8ZZ3VwUC", "reviewer": "Reviewer_jJr9", "summary": "", "questions": "A: You mention that you use the model's confidence in answering \"yes\" to a particular statement to assess beliefs, and that a more \"semantic\" notion of belief would involve querying multiple paraphrases. Anecdotally, do you see evidence that there is substantial inconsistency in belief scores across paraphrases?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your thorough review and encouragement!\n\n*>How portable is the approach to other LMs which haven't been explicitly tuned to present the functionalities listed in section 3.2.1?\n\nIn principle if the required operations are implemented using few-shot capabilities, then the approach should port easily. Other work has demonstrated the individual components already exist: chain-of-thought systems demonstrate premise generation (h -> p) [Wei et al 2022]; and several methods exist for assigning confidences to LLM's answers about whether a fact/rule is true, e.g., [Jiang et al., 2021]. This makes the outlook promising, but indeed their combination still needs to be demonstrated at scale in an architecture like REFLEX. \n\n[Wei et al 2022] Chain of Thought prompting elicits reasoning in LLMs. NeurIPS.\n[Jiang et al., 2021] How can we know when LMs know? TACL.\n\n*>You mention that you use the model's confidence...to assess beliefs, and that a more \"semantic\" notion would involve querying multiple paraphrases. Anecdotally, do you see evidence that there is substantial inconsistency in belief scores across paraphrases?\n\nGood question: Anecdotally, for the model we used, the truth assignment seems very consistent among paraphrased statements, but the scores can still vary depending on phrasing. But we would expect high confidence scores to correlate with more consistency across paraphrases, and overall behavior is mainly affected by high-confidence scores.\n\n*>Figure 1 is a little cluttered/overwhelming\n\nThanks for this feedback and useful suggestions for improving the figure! We will be sure to improve it in future versions!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Attempting to construct systems that provide *faithful* explanations while performing on par with less-structured competitors is usually difficult, so achieving the reported consistency gains without hurting system performance is impressive.\n\n- While the question of whether models can be said to have \"beliefs\" at all is open to interpretation, the authors do a good job of making their working definitions explicit, and operationalizing them in such a way that the whole thing hangs together, i.e. the system's explanations are causally faithful, but it still makes sense to frame them as products of the model's \"knowledge\".", "reasons_to_reject": "- The three datasets used for evaluation are all roughly aligned with the style of text the base model (Entailer) is tailored to. This raises the question of whether the approach is actually portable to other domains/LMs which haven't been explicitly tuned to present the functionalities listed in section 3.2.1. While the authors sketch a means of implementing these operations using few-shot capabilities, the broader usefulness of the contribution would be less questionable if evaluated in other domains.", "questions_to_authors": "A: You mention that you use the model's confidence in answering \"yes\" to a particular statement to assess beliefs, and that a more \"semantic\" notion of belief would involve querying multiple paraphrases. Anecdotally, do you see evidence that there is substantial inconsistency in belief scores across paraphrases?", "paper_topic_and_main_contributions": "This paper proposes to extend a language model with a \"reflection\" system which queries the model for beliefs, assembles those beliefs into an implication graph, then harmonizes this graph via MaxSAT in order to improve the consistency and faithfulness of automated natural language reasoning.\n\nThe authors evaluate the effectiveness of their proposal using three multiple-choice question-answering datasets, assessing system performance in terms of both the consistency of explanations and multiple-choice accuracy. The results indicate that the proposed reflection harness drastically improves the overall system's consistency while maintaining the accuracy of the underlying model.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 463: ... how answers [follow/derive/?] from...\n\nVisually, figure 1 is a little cluttered/overwhelming. It might be more useful for readers if the LLM Layer was shown as a simple box, maybe with one or two arrows to belief nodes indicating that beliefs are extracted from the model (the current vertical yellow arrows are ambiguous)."}}, {"review_id": "0vBRumt8tP", "reviewer": "Reviewer_udQV", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your thorough review and encouragement! We really appreciate your crystal-clear summary and enthusiasm for the work!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Self-reflection has become common in code-generation models, for models (like ChatGPT) to debug their own code.  This paper essentially builds a system for doing something similar with graphs of an LLM's beliefs about the world, while applying a neurosymbolic approach (LLM + probabilistic graphs + constraint reasoner) to solve the task at scale (\"350 nodes and 80 constraints per question\").  This is a substantial and elegant contribution, making the knowledge an LLM uses to reason much less opaque.\n\n- The system achieves a large (~10%) performance increase in the task of generating accurate belief graphs compared to previous models, on three benchmark datasets (EntailmentBank, OpenBookQA, and Quartz).\n\n- Good literature review for the paper's size\n\n- Clear exposition of the model\n\n- Straightforward results, clear evaluation measurements, ablation testing. \n\n- Detailed error analysis\n", "reasons_to_reject": "- I don't see any reasons for rejecting this paper", "questions_to_authors": "", "paper_topic_and_main_contributions": "This is an NLP experiment paper.  This work proposes a system called \"REFLEX\" that builds large (hundreds of nodes) belief graphs from language models, to support inferring whether the answer to a given question is correct.  The belief graphs include both facts relevant to inferring whether the answer to a question is correct, as well as rules.  Using a combination of querying the LLM for self-consistency, as well as GOFAI techniques (probabilistic reasoning graphs, constraint satistfaction solvers), the system is shown to generate substantially better belief graphs than a baseline system without this modification. \n", "excitement": 5, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- \"eg\" -> \"e.g.\" (L164, L166)"}, "extra_text": {}}, {"review_id": "eHnHReDdde", "reviewer": "Reviewer_mJrZ", "summary": "", "questions": "1. Line 198-200: Why Maieutic's explanation chain are not validated as reﬂecting model-believed constraint rules (or not able to expand to such a belief graph that facilitates that)?\n2. According to my knowledge, the full test set size of OBQA is 500 (ref https://allenai.org/data/open-book-qa), of QuaRTz is 784 (ref https://allenai.org/data/quartz). Why does this paper has such a numerical discrepancy? \n3. In Section 4.3, among 50 random questions, can you give the exact number of mistakes (I assumes it is equivalent to incorrect conclusions or incorrect answers, please clarify if my assumption is wrong) that the model makes? Also the exact number of samples in each failure categories. In those failure categories, which one surely entails incorrect answer for the test questions (e.g questions from EntailmentBank)?\n4. I believe that when we represent the reasoning chain of LMs decently, the next step is to use it to improve the reasoning capability. Since the naive decision made by the optimized belief graph does not work, do you have any general idea to use the belief graph in a different way to improve model reasoning capability? e.g. propagating the updated belief graph back to the model during inference time, or design a pretraining task to improve model consistency? ", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your thorough review and encouragement!\n\n*>The belief graph to some extent are problematic. [1] suggests that language models' generated text do not always reflex what they internally \"think\", thus the belief graph may not faithfully represent the rationale for the models' output (yet it is a common argument against all explitcit interpretation methods).\n\nYes, care is needed to interpret the graph. Importantly, you are correct that chains in the graph do not necessarily represent how the model *actually* reasoned internally, and we make no such claims (Lines 385-7). Rather, the (initial) graph is a snapshot of the model's beliefs and their (model-believed) inferential relationships, hence an explanation from the (initial) graph denotes one model-believed chain of reasoning to an answer, even if it actually arrived at that answer via a different mechanism. We think of it as an orthogonal yet still model-believed “proof” of an answer.\n\n\n*>Although the paper does not claim that the belief graph improves the reasoning capability, I believe that the authors did attempt to do so by determining the final answer of LMs using the optimize belief graph. Given that situation, experimenting on data sets which are different from that in previous works ([2], [3]) hinders the comparison between different methods.\n\nWe chose our datasets as they contain inferentially rich questions (typically) requiring reasoning, the focus of our work. You are correct that we could add datasets from [2] and [3], although it's not clear how to then draw comparisons: Specifically, we can't simply compare accuracies with published results as the underlying models differ; and as [2] and [3] have restricted expressivities (e.g., they don't represent rules with multiple premises), it is non-trivial how one might generate a comparable belief graph with them to fairly compare consistencies. However, there are qualitative comparisons that can be drawn, which we attempted in the Related Work. \n\n*>1. Line 198-200: Why Maieutic's explanation chain are not validated as reﬂecting model-believed constraint rules (or not able to expand to such a belief graph that facilitates that)?\n\nYes: REFLEX checks whether both (a) the statements s_i, and (b) the rules (s_i -> h), are believed by the model via self-querying, e.g., by asking “Does s_i -> h?”, and also scores the strength of those belief. In maieutic prompting, only (a) is done - the generated rules are not checked against the model, hence maieutic prompting may generate a rule that it itself may not believe, if it queried itself about it. (This self-check could be easily added to the next generation of maieutic prompting work).\n\n*>2. Exact number of mistakes …\n\nThank you for catching this, it is a simple typo, we forgot to update the numbers. We worked with smaller subsets in the past but used the full test sets for this submission. The results were computed on the full test sets. We will correct this in future version.\n\n*>3. In Section 4.3, among 50 random questions, can you give the exact number of mistakes (I assumes it is equivalent to incorrect conclusions or incorrect answers, please clarify if my assumption is wrong) that the model makes? Also the exact number of samples in each failure categories. In those failure categories, which one surely entails incorrect answer for the test questions (e.g questions from EntailmentBank)?\n\nYes, just to clarify: all 50 were cases sampled from when the system produced the incorrect answer, and our goal was to identify the primary cause(s) of failure overall. We will clarify this in the introduction to 4.3, our apologies. \nIn the five failure categories, the counts respectively were: 16, 17, 4, 6, and 4, indicating that (1) failing to generate an important rule, and (2) having bad beliefs, were the primary causes of wrong answers.\n\n*>4.I believe that when we represent the reasoning chain of LMs decently, the next step is to use it to improve the reasoning capability. Since the naive decision made by the optimized belief graph does not work, do you have any general idea to use the belief graph in a different way to improve model reasoning capability? e.g. propagating the updated belief graph back to the model during inference time? or design a pretraining task to improve model consistency?\n\nYes, we are completely on the same page here, this is a natural follow-up direction to look into. We have an elaborate error analysis examining why accuracy improvement is minimal, and attacking these could improve systematic reasoning accuracy. Your suggestions are right on: somehow propagating updated beliefs back into the model would be highly desirable (though current model editing techniques are not quite up to this), and similarly pretraining for consistency may improve the model's initial self-consistency (e.g., [Ribeiro et al., 2019]). Additional approaches we have considered include: identifying areas of maximal conflict in the belief graph and doing additional localized inference to materialize more knowledge to help resolve conflicts the right way; using external resources to help identify and correct bad model beliefs; or even include a user in the loop, e.g., using the belief graph to identify areas of model confusion, and leveraging that to ask the user judicious questions to resolve the conflict. These would all be great directions for future work.\n\n[Ribeiro et al., 2019] Are red roses red? ACL.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Belief graph advances the model internal knowledge's materialization by facilitating richer logical relations (a.k.a rule node) and on-the-fly context-dependent belief generation. Like some other previous work, the graph serves as proxy to interpret model's output, as well as allows to externally fix the inconsistency among those belief. This will benefit a lot of reasoning tasks.\n2. The method is clearly described and illustrated, including the construction of the belief graph from the initial state and how to optimize it.\n3. The paper demonstrates thorough quantitative and qualitative analyses, especially the case of good reasoning and failure, and the relationship of model false belief and failure in the belief graph optimization stage.\n", "reasons_to_reject": "1. The belief graph to some extent are problematic. [1] suggests that language models' generated text do not always reflex what they internally \"think\", thus the belief graph may not faithfully represent the rationale for the models' output (yet it is a common argument against all explitcit interpretation methods).\n\n2. Although the paper does not claim that the belief graph improves the reasoning capablity, I believe that the authors did attempt to do so by determining the final answer of LMs using the optimize belief graph. Given that situation, experimenting on data sets which are different from that in previous works ([2], [3]) hinders the comparison between different methods.\n\n[1] Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting, Turpin et. al. (2023)\n\n[2] Enhancing Self-Consistency and Performance of Pre-Trained Language Models through Natural Language Inference, Mitchell. et al. (2022)\n\n[3] Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanations, Jung et. al. (2022)\n", "questions_to_authors": "1. Line 198-200: Why Maieutic's explanation chain are not validated as reﬂecting model-believed constraint rules (or not able to expand to such a belief graph that facilitates that)?\n2. According to my knowledge, the full test set size of OBQA is 500 (ref https://allenai.org/data/open-book-qa), of QuaRTz is 784 (ref https://allenai.org/data/quartz). Why does this paper has such a numerical discrepancy? \n3. In Section 4.3, among 50 random questions, can you give the exact number of mistakes (I assumes it is equivalent to incorrect conclusions or incorrect answers, please clarify if my assumption is wrong) that the model makes? Also the exact number of samples in each failure categories. In those failure categories, which one surely entails incorrect answer for the test questions (e.g questions from EntailmentBank)?\n4. I believe that when we represent the reasoning chain of LMs decently, the next step is to use it to improve the reasoning capability. Since the naive decision made by the optimized belief graph does not work, do you have any general idea to use the belief graph in a different way to improve model reasoning capability? e.g. propagating the updated belief graph back to the model during inference time, or design a pretraining task to improve model consistency? ", "paper_topic_and_main_contributions": "Research question: This paper works on materializing language models' internal knowledge to enhance the interpretability of models in QA tasks.\n\nContribution:\n1. It introduces a rational layer (which is in fact separated from the language model) which consists of belief graphs i.e. explicit representation of system beliefs and their logical and inferential relationship. This layer serves as a medium to interpret the model's output and allows reasoning capability of the model to be improved.\n2. Implementation of the layer\n3. Experiments (in the zero-shot manner) on three datasets show that consistency of the overall system's network of beliefs can be significantly improved without harming answer accuracy.\n\n", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes to add a component to LLMs that constructs a "belief graph", repair inconsistencies, and produce faithful reasoning chains drawn from a belief system with significantly improved consistency. All reviewers praised the work, highlighting the provided consistency gains, exposition, and analyses. Some concerns were inherent to whether the methodology proposed in this paper could transfer to other domains, about the choice of the datasets, and whether the belief graphs faithfully reflect how the model operates internally. The authors provided clarifications in their rebuttal. Overall, the reviewers found the paper a significant contribution to NLP, which could open interesting research avenues. Two reviewers out of three recommend it for a best paper award.
Decision: Accept-Main