Does Reasoning Emerge? Examining the Probabilities of Causation in Large Language Models
Authors: Javier Gonzalez
Aditya V. Nori
Venue: NeurIPS 2024 poster (2024)
Keywords: Reasoning
Large Language Models
probabilities of causation
Primary area: generative_models

Abstract: Recent advances in AI have been significantly driven by the capabilities of large language models (LLMs) to solve complex problems in ways that resemble human thinking. However, there is an ongoing debate about the extent to which LLMs are capable of
actual reasoning. Central to this debate are two key probabilistic concepts that are essential for connecting causes
to their effects: the probability of necessity (PN) and the probability of sufficiency (PS). This paper introduces a framework that is both theoretical and practical, aimed at assessing how effectively LLMs are able to replicate real-world reasoning mechanisms using these probabilistic measures. By viewing LLMs as abstract machines that process information through a natural language interface, we examine the conditions under which it is possible to compute suitable approximations of PN and PS. Our research marks an important step towards gaining a deeper understanding of when LLMs are capable of reasoning, as illustrated by a series of math examples.

Does Reasoning Emerge? Examining the Probabilities of Causation in Large Language Models
Javier González Gonzalez. aetierrmiiiosss com Microsoft Research, Cambridge
Aditya V. Nori
Adi Aditya tya. Nori @mi crosoft. com Microsoft Research, Cambridge
Abstract
Recent advances in AT have been significantly driven by the capabilities of large language models (LLMs) to solve complex problems in ways that resemble human thinking. However, there is an ongoing debate about the extent to which LLMs are capable of actual reasoning. Central to this debate are two key probabilistic concepts that are essential for connecting causes to their effects: the probability of necessity (PN) and the probability of sufficiency (PS). This paper introduces a framework that is both theoretical and practical, aimed at assessing how effectively LLMs are able replicate real-world reasoning mechanisms using these proba- bilistic measures. By viewing I.LMs abstract machines that process information through natural language interface, we examine the conditions under which it is possible to compute suitable approximations of PN and PS. Our research marks an important step towards gaining a deeper understanding of when LLMs are capable of reasoning, as illustrated by series of math examples.
Introduction
Large language models (LLMs) have revolutionized the way we interact with technology, enabling more natural and intuitive communication between humans and computers in applications like writing
assistants [8), sentiment analysis social media (29). healthcare [10, 331 and many others. Despite
the surge of interest and recent breakthroughs [5], the ability of LLMs to reason about real-world
problems as humans do continues to topic of intense research 14
Reasoning is cognitive process that involves drawing conclusions, making judgments, or forming inferences based on facts or premises. This process has been explored from various perspectives. Symbolic reasoning [17 involves the manipulation of symbols that represent ideas or objects and it is often used in mathematics and logic to represent numerical values or logical propositions. Causal reasoning (26) focuses on discerning the relationship between a cause and its effect, aiming to understand how certain events can impact other. Other forms of reasoning include inductive reason- ing 7 (making broad generalisations from specific observations), deductive reasoning (27) (applying general principles to specific cases), and abductive reasoning (forming the best hypothesis based on incomplete information).
In the realm of L.MMs, reasoning typically understood to be the ability of these models to demon- strate emergent capabilities that surpass mere statistical pattern recognition in the training set. entails systematically breaking down problems into logical sequence of smaller, manageable steps and then processing these steps internally to arrive at accurate conclusions that are grounded in reality, This concept the foundation for techniques such as chain of thoughts prompting (34] which aim
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Div. by 6 questions
Direct prompt
Queszion type
Dreet
Does six divide [number]? ? Use the factor method to this
0.30
question. Be concise as possible.
0.23
Counterfactual prompt
Imagine that [number) has three as a prime fuctor while retaining all its other prime factors. With this assumption does six divide [number)? ? Use the factor method answer this question. Be as concise as possible.
6.10
0.00
Model
Figure I: Illustration of the actual vs, perceived reasoning abilities of GPT-2, GPT-35-turbo and GPT- 4 for a simple arithmetic problem. We posed two distinct types of questions (direct and counterfactual) to the models, each repeated 10 times, for every (number) from I to 50. All three models showed an inflated sense of reasoning capability when answering the direct questions. The discrepancy is especially pronounced in GPT-35-turbo, which performed nearly flawlessly on direct questions, but experienced surge in error rate, exceeding 25%, when handling counterfactual questions.
teach LLMs how to reason by providing examples where problems are solved through a sequence of smaller steps.
Assessing the reasoning abilities of LLMs involves distinguishing between two aspects: the accuracy with which LLM solves problem, and its capacity to understand and process the fundamental elements that lead to that solution. Judea Pearl, his hierarchy of causality [25), asserts: "Only machines that can correctly perform correlations, interventions and counterfactuals will have reason- ing abilities comparable to human". ". As demonstrated in [16], while LLMs are remarkable in using learnt patterns from their training data generate correct answers (correlations), they falter when faced with prpotheticalimmainann This is depicted in Figure 55) which presents straightforward arithmetic problem (this is the direct prompt in the figure). Both GPT-35-turbe and GPT-4 can accurately determine the divisibility of numbers by 6, suggesting at first glance that they can reason about divisibility. However, when the questions are framed in counterfactual manner (this is the counterfactual prompt in the figure), only GPT-4 maintains low error rate, indicating its superior ability to handle such reasoning tasks.
scenarios that were not part of their training (counterfactuals)
In this paper, we introduce a systematic method to assess the reasoning capabilities of LLMs by examining the concepts necessity and sufficiency, which are key elements of logical reasoning and have been studied in multiple fields such as logic, probability, and causality 22
19 12]. In propositional logic. sufficient condition is defined as X = Y. indicating that the presence of ensures the occurrence of Y. On the other hand, a necessary condition is defined as Y = X. signifying that the occurrence Y necessitates the prior occurrence X. We focus on the probabilistic interpretations of necessity and sufficiency 24]. The probability of necessity (PN) between two boolean variables X and Y is defined as PN y) == P(y' xr. y). Here, V',, represents the counterfactual value of Y y', had X been set to different value x'. By conditioning on both X = and Y = this measure captures probability of observing different outcome the absence of the event X = x. The probability of sufficiency (PS), on the other hand, is defined as PS(ar, y) (P |a', v/) and measures the probability that X x results in Y y, for cases where both originally had different
values.
We show that when a problem can be solved via a reasoning graph of boolean conditions, denoted by G. the PN and PS can be computed using a causal model underlying G. As described in (24), the exact computation of PN and PS requires samples from the (causal) data generative model, counterfactual data (experiments) as well as other monotonicity assumptions. As reasoning test, we statistically compare the true PN and PS measures (computed by sampling from the original and the intervened graph) with those simulated via factual and counterfactual datasets generated by an LLM. Figure 2 presents an informal illustration of the reasoning test advocated this paper, focusing on the specific problem of determining whether number N is divisible by 6. The test relies on the reasoning principle that: "A natural number N that is divisible by both 2 and is also divisible by 6*. This logic is represented in reasoning graph G that links the conditions C2 (divisibility by 2) and C3

---

A
B
C
Factual (F) and counterfactual (CF) datasets from G
Assessment
Divisibility rule: natural number that divisible by both 2 (C2) and (Cg) also divisible by (C₂)-
N
PN
NOT reasoning
do(Css True] _ Cp
Dp
True
DCR
LLMM-based PN, PS
Drrt
densities
DCF
Reasoning graph G:
Factual prompe
Counterfactual prompt
PN
LLM 2:
L.M
LLM
Reasoning
Boolenn inpur (C3)
Boolean ippl (C3)
output (C.)
ouspui (C6)
Factual (F) and counterfactual (CF) datasets
PS
from LLM
Figure 2: Reasoning test for assessing LLM's reasoning abilities. A) Divisibility rule and the corresponding reasoning graph. B) Dataset generation for computing PN and PS. C) Analysis comparing actual values of PN and PS with PN and PS estimates for the 4LM--eneraaed data.
(divisibility by 3) to the conclusion Co (divisibility by 6). We test the reasoning abilities of an LLM using natural numbers N from to 400. This is shown in Figure ([A)
As indicated in Figure [B),, create two sets of data based G. The first is factual dataset (Dr) which captures whether each number N satisfies conditions C2 and C3- The second is a counterfactual dataset, (PcF), which assumes condition C33 is always true and then records whether each number X would satisfy Cg under this assumption/interventte the figure). For the LLM being evaluated, the LLM's response for Cg for each number N, when the prompt is based on the reasoning graph g. The second, D.LLM involves a hypothetical scenario where we assume C3 is true and then record the LLM's prediction for CG given this "counterfactual prompt". This process is repeated multiple times (several answers from the LLMs are collected from each prompt). We assess the LLM's reasoning capability by comparing the estimated (distribution of) PN and PS from the DI.L and DUL" datasets with the actual values derived from Dr and Der datasets. Figure ( displays these comparisons, plotting PN vs. PS. The closer the estimated PN/PS values the actual PN/PS values, the better it is at reasoning. In this case, LLM 2 demonstrates better reasoning abilities than LLM
(realised by do(C's = True) in also produce two datasets. The first, D documents
presents an overview paper that elucidates key reasoning concepts utilised by L.Ms.. [3] examines the similarity between reasoning with a language model and planning with a world model, proposing novel reasoning framework that redefines the LLM both a world model and a reasoning agent. Various studies (28 4 have focused on assessing the reasoning and problem-solvine abilities of LLMs, yet none have used the probabilities of causation as the primary objects of computation done in our research. 32 carries out a series of experiments to show that LLMs can indeed derive benefits from reasoning errors, offering potentially
Related work: Reasoning in LLMs has been studied from multiple perspectives.
15
effecctfe strategies by using mistakes bolster reasoning capabilities. Recent research indicates that LLMs like GPT-3.5 and GPT-4 are effective at causal reasoning tasks, including pairwise causal discovery [8]. These models have achieved state-of-the. art performance multiple causal benchmarks, outperforming existing algorithms. Nevertheless, L.MS also exhibit unpredictable failure modes, and currently, they are not capable of discovering new knowledge making high-stake decisions with high level of precision [20 36 21 3).

---

QLLM
60
81
[prompr]
[output]
Q
We define a problem which assigns values from a specified domain to a set of variables V (V..... Vn The query well-defined output state. To d (X 16, Co H+1
01 = (X 16, Cs + False)
Figure 3: The HEX diagram depicts two approaches for solving the problem (Q, outlined in Example I The dotted path corresponds to the actual process of solving the problem, while the solid path represents the one taken by the LLM.
Contributions: This paper presents two main contributions.
i) A novel, theoretical and practical, framework to evaluate the reasoning capabilities of language models using the probabilistic of causation, specifically the probability of necessity and the probability of sufficiency. Our approach is unique in the sense that it allows to differentiate generalization by reasoning from merely replicating statistical patterns in the training data.
ii) Empirical tests on various reasoning problems as well as several insights about the reasoning abilities of language models in the GPT family.
2 LLMs as abstract machines
As described by the HEX framework [9] an LLM functions as an abstract machine that uses natural language an interface. In this section, we introduce the core elements of this framework, which will subsequently enable to define an LLM's internal representation of PN and PS.
query-state pair (Q) The state is mapping defined by V:V- C.
Q 2 + 2V is mapping that transforms an input state solve problem is to calculate σj = (())) where do and σn represent the states before and after the query Q is applied. To clarify this, consider the following example:
"Given that natural number divisible by both 2 and is also divisible by 6, determine whether the number 10 is divisible by 6."
Example 1.
To solve Example 1 apply the query Q the state op (N 10, Co I)], where Q Aa (N) (mod 2) = 0) A (o(N) (mod 3) = оf This results final state a (N + 10, C6 + False), ), thereby resolving the problem with (Ca) = ( False.
We now turn to the question of how an LLM solves a problem defined by a query-state pair (Q. oo).
This process involves three essential steps illustrated by Figure 3
1. First, an abstraction mapping translates the initial state du into a latent state on via a prompt.
2. Next, the LLM processes (via the query QLLM) this latent state dg.
3. Finally, the output mapping transforms the L.MM output latent state â back into concrete state, producing the final output 01.
Formally, solving a problem (Q, with an LLM can be described as sequence of function applications resulting in the output 01 ( o QLLM a)())). To illustrate this, the problem statement is given as prompt input to GPT-4 23 The response from GPT-4 is "False". which matches the result obtained by applying the query Q directly to the input state dg- When both the direct application of Q and the LLM computation yield the same answer, we say that the diagram, as shown in Figure is commutative-meaning that following either the dotted line or the solid lines lead to the same result. For a more in-depth explanation of this framework, please refer to 9
'See http:://ww..ikiiipdiaaoiil////ld calcullss for quick introduction to Lambda calculus.

---

3 Probabilities of causation for an LLM
To assess the reasoning abilities of an LLM, we must link its generated responses to the actual reasoning processes that produced those responses. For a problem (Q, o). we postulate the existence of causal model My defined over variables in V, and by a set of structural equations and endogenous variables. For a detailed introduction to causal models, refer to Appendix A Additionally, the seminal work by Pearl 25 on causality provides foundational insights on this area. Here, we are particularly However, interested in causal models that represent the logical steps involved in problem-solvine is important to note that the concept of a causal model is broadly applicable beyond this specific application.
We assume that V (X, Y, Z), which includes X and Y as boolean variables, and Z variable (which may be multivariate) that encompasses all necessary factors that are required to understand how an intervention on X would affect Y. In the context of causality, this means that the distribution P(Y do = x') z')), where do denotes the intervention operator defined in [251, is identifiable. This means we can predict the outcome for Y, and that the counterfactual Yx=s', that can be read as "the value Y Y had X been z'", is well-defined, For further details, please refer to Appendix A For ease of exposition in the following text, we will simplify our notation by omitting the explicit reference to Z. Therefore, we will denote Yx=x(Z = 2) more succinctly as Yx=x.
As studied in 31), if Y is monotonic with respect X, then PN and PS can be computed follows:
PN y) = P(y)= P(yldo(zz)) and PS(x, = P(yldo(x)) P(y) P(z,
(1)
P(x',y')
To estimate PN and PS, we need two different types of datasets. The first is a Sactual dataset DF = (ris yb: 21 )= which is used to infer PP(y), P(x; y) and P(aa',2)), The second dataset
DcF (z,, x / is counterfactual one, and is necessary to determine P(y]do(x)) and
Fly (o(())
There are various methods to acquire the datasets Dr (factual) and Der (counterfactual), For physical process, the usual method would be through observation and experimentation. However, in this paper, we presume access to a comprehensive reasoning graph that is equivalent to a causal model My. This allows to simulate and generate the Dr and DeF datasets. Both My and the sub-model My My.do(X==)) define two distinct joint probability distributions PMM and PMyy X=( over X, and Z. We obtain the datasets Dr and Der by sampling from these respective probability distributions. These datasets are then used to calculate PS and PN using Eq.
3.1 LLM-based counterfactuals
Can an LLM reason in a manner that is consistent with PP P.My? ? In Example D we obtained consistent answers (that is. the corresponding HEX diagram commutes) for a direct divisibility question. However, to evaluate the reasoning abilities of the LLM, is crucial that this consistency is also observed when the queries are framed in a counterfactual manner. This is necessary to ensure that the LLM can apply its reasoning to imaginary situations that are unlikely be present in the training set, demonstrating its ability to generalise based on a correct internal representation of the reasoning logic of the problem. Practically, this means employing the LLM as a "counterfactual data simulator". where the data it generates under these hypothetical conditions are used to estimate PN and PS.
Definition (Counterfactual query). Consider a problem (Q, oo)) with og = {X + x. Y + y, Z + z} being initial state. Let My be a causal model over the variables V. We can then define counterfactual query Q' as follows: ('(oo) = (X > x', Y H Yx=r", Z + 2}.
In other words, a counterfactual query updates two variables of the state: it sets X to its new value x'. LLLM is computed as follows: and Y to the counterfactual Yx=r'. An LLse counterfactual Y X m=x
LLM = (7 o Q oa)(o))
where σo 2X 2). and Q'LLM is a counterfactual query. This entire process simulates counterfactual reasoning within the LLM, and is facilitated through textual prompts that are structured to elicit the desired counterfactual outcome.
Definition (Counterfactual prompt). A counterfactual prompt is a textual encoding of a counterfuc- tual query for some initial state do-
5

---

Figure shows an example of a counterfactual prompt. To create comprehensive dataset D LLMM CP Cg C==0 -0=0
Dr
C -1
C3
267
0
C =1
67
66
EvenSum
CandyParty
DCF
Ca 0
Ca
=
N
M
Cer
E
Cle
C3
133
0
C3
133
134
Y
Y
D
Cg = 0
Ca
R
C =
267
0
Cn
C M
C
C3
67
66
Cerl
Crel
CF
Co =
Cg
C3
132
C =1
132
135
Cumt
C+
Figure 4: Left: Contingency tables for Dp, Der and DCF DPP in Example | Right: Reasoning
graphs for the other math problems in this paper. C-type nodes in the graph represent boolean conditions. See Appendix for details.
of counterfactuals based on an L.M,, we start with the factual dataset DLLLM. From this dataset, we generate set of initial states oo, {X + 24, Y + yı, Z + 24), which serve as the basis for deriving counterfactuals using the LLM. To compute PN and PS, substitute Dp with DLLM and DeF with CF in Eq. [
Example revisited. We construct four distinct datasets using every integer in [1, 400): the factual dataset Dr, the counterfactual dataset Der, the LLM-based factual dataset DLLM and the L.L based counterfactual dataset D...M. These datasets, shown in Figure 4 (Left) are generated following the causal model shown in Figure its modified version with interventions, and the LLM prompting methods mentioned previously. We obtain PN 1 and PS = 0.50 for the datasets DD and DoF. the other hand, PNGPT-4 0.984 and PSGPT-4 0.505, when we use the factual DLLM and counterfactual DLLM datasets generated by GPT-4.
3.2 Counterfactual consistency in LLMs
Definition 3 (8-counterfactual consistency). Consider a structural causal model My with vari- ables V {X, Z}. Let Ax=e(2) be function that generates counterfactuals for Y. We say that A is 3-counterfactual consistent with My if the following condition is satisfied: Ep(x,Y,2) [Ax= 2) + Yx=x = 2)] I where 0.
8-counterfactual consistency defines the limit error rate for counterfactuals produced by Ax=r 2). This eиror rate should ideally be zero for an LLM that exhibits flawless reasoning abilities. The following lemma specifies the conditions necessary for this property to hold (the proof can be found in Appendix D)
Lemma Let My, with variables v {X,Y, Z}, be a structural causal model for a problem (Q. oo), and let M be an LLM that generates counterfactuals for Y. Then M is B-counterfactual
consistent with My if and only associated HEX diagram for the problem (Q', ao), where Q' the counterfactual version of Q, is commutative for all admissible values of X, Y and Z.
Lemma 1, provides theoretical criterion to test the consistency of an LLM with some reasoning graph described by My. As we show next in the experimental section, the commutability of the diagram is tested in expectation by sampling over the variables in V.
Empirical illustrations
We focus on three math problems, each with a progressively higher level of difficulty.
Divisibility by 6 (Div6): We compute the PN and PS to determine the impact that an integer Ns divisibility by (denoted as Ca) has on its divisibility by 6 (denoted Co). For our analysis, we consider N E [1,400 400).

---

Elements PIR (%)- Div6
Necessity
0MP2-
8:31
GP74
6.1
100
120
162
Sree
Elements CIR (%) Div6
6P72
7.5-
CPT35tecco
5.0
GP74
2.5
100
120
144
252
270
200
390
524
340
360
370
3000
0.0
-0.2
0.0
0.2
Probubitity
Figure 5: Left: Heatmaps comparing the consistency of data generated by GPT-2, GPT-3.5-turbe and GPT-4 for the Div6 problem. Each heatmap cell represents the error rate of the corresponding model for each element of the problem across 10 replicated tests. Right: Sensitivity the simulated PN relative to varying levels of random noise introduced in the true counterfactuals.
Even sum of integers (EvenSum): We examine scenarios where the sum of three integers M, .N and T' even. This can occur under two conditions: when all three integers are even, when one is even and the other two are odd. We evaluate PN and PS for impact that M being odd or even (C.,) has on the resulting sum being odd or even (Cmnt). For our analysis, we consider all possible values for M, N and T, with each integer ranging from 1 and 8.
Candy party (CandyParty): In this hypothetical scenario, Rafa is having his birthday party with two guests, Lara and Emma. They have 20 candies to distribute among themselves. The party will be considered 'happy' if the candy distribution satisfies least one of the following conditions: (i) Each person gets the same number of candies, or (ii) Rafa gets more candies than both Lara and Emma, but Lara and Emma each receive an equal number of candies, with both receiving at least one candy each. We compute the PN and PS for the impact that Lara and Emma receiving an equal number of candies (denoted as Cmn) has on the party being 'happy' (denoted as C²)).
fourth problem (ConPref) is included in Appendix B The reasoning graphs for the problems
EvenSum and CandyParty are shown in Figure 4 (Right). The structural equations corresponding
to each of these graphs can be found in the Appendix C We estimate the PN and PS for each of these problems using three difference language models: GPT-2, GPT-3.5-turbo and GPT-4 (23). Our objective is to investigate whether the ability to reason, as conceptualised in this paper, emerges as the complexity and size of the models grow. While similar evaluations could be conducted using other families of LLMs, such as Llama [33). Gemini (30]. Phi [TT] etc., we have chosen to limit our analysis to the GPT series here for the sake of a clearer and more straightforward exposition, but more results are available in Appendix
To assess the reasoning abilities of various models, we the following metrics:
1. Factual Inconsistency Rate (FIR): This measures the rate of inconsistencies when the models respond to factual queries.
2. Counterfactual Inconsistency Rate (CIR): Similar to FIR, but this metric measures inconsis- tencies in responses to counterfactual queries.
For detailed explanation of these metrics, please refer Appendix HH We estimate the standard errors of FIR and CIR by examining the variations in outputs across multiple model responses. We take this aspect into account by collecting multiple answers from the models and propagating the stochasticity of the answers to the computation of PN and PS. Additionally, we capitalise on this variability construct the densities over the inferred PN and PS. This process involves generating 500 bootstrap samples from the model's 's factual and counterfactual responses"
2 From these densities,
we calculate -veelllr,,,,, which measures the concentration of the probability distribution within radius 7 around the actual PN, and --PS-overlap does the same for P
Note that while generating larger number of model answers could potentially increase accuracy, the computational costs are prohibitive. Therefore, the bootstrap approach serves as reasonable compromise. 'The code to reproduce the analyses and figures can be provided upon request, and will be made open source if this work is accepted for publication.
7

---

Div6
EvenSum
CandyPara
GPT4
X
Suffilency
0.4
Sulticiency
-0.2
Sufficiency
GPT-4
0.3
GPT-3.5-4
-0.4
GPD2
0.2
GPF-3.5+
0.3
GPT3.5-1
0.0
GPT2
-1.0-
X
0.63
0.70 0.75
0.00 0.85 0.90 0.85 1.00 3.05
0.0
0.2
0.4
0.
0.0
1.0
-04-02 00 02 04 06
0.8 1.0 1.2 Necessity
Necessity
Necessity
Figure 6: True PN and PS vs. inferred PN and PS using GPT-2, GPT-35-turbo and GPT-4. The densities of the estimated probabilities capture the uncertainty associated with the responses by each model.
4.1 Factual vs. counterfactual predictions
Figure (Left) illustrates the alignment between the outputs of GPT-2, GPT-3.5-turbo, and GPT-4, and the factual predictions and counterfactuals for the Div6 problem. The shading within each cell of the heatmap indicates the degree of mismatch between model-generated outputs and the true information, with the colour intensity reflecting the level of disagreement based on the 10 answers from the models. As highlighted in Figure where the average disagreement across the first 100 columns of these heatmaps informs the results-mort more sophisticated models like GPT-4 demonstrate a closer match with the counterfactuals derived from the true reasoning graph. For similar comparisons involving other problems, please refer to Appendix
One might wonder if the evaluation of reasoning truly requires PN and PS, or if it could be sufficiently assessed by examining only the inconsistency rates in factual/counterfactual data. Figure (Right) underscores the importance of PN and PS. It presents the estimated distributions of PN for the Div6 problem, based on 500 replicates under five scenarios where true counterfactuals are randomly altered with probabilities 0.005, 0.001, 0.05, 0.1 and 0.2. As we might anticipate, the greater the deviation from dataset free of counterfactual errors, the more significant the discrepancy from the actual lead substantial shifts in the
PN = 1 for this example. Notably, even minor perturbations estimated PN. For example, with a 0.05 probability of counterfactual perturbation, the estimated PN varies between 0.5 and 0.9. This suggests that relying soiely on counterfactual errors could lead to an overestimation of the models' reasoning abilities, particularly their understanding of the necessary and sufficient conditions within problem. Furthermore, counterfactual error rate of 0.2 in this example results in entirely inconsistent (negative) probabilities due to the mismatch between the conditional and interventional distributions, defined in Eq.
4.2 Evaluation of LLMs reasoning
We computed the CIR, FIR, --PNoveerlap, and ----rrle for the problems Div6, EvenSum and CandyParty using GPT-2, GPT-3.5-turbo, and GPT-4.
Figure iillustaass the estimated PN and PS for each problem, obtained through bootstrap resampling.
Each density labeled with the model that was used used to generate the data for those results.
The true values of the PS and PN in each problem is marked with cross. A model is considered capable reasoning if the PN-PS density estimates overlap with the true probabilities of causation. Such overlap was only achieved by GPT-4 for Div6 problem. Other results varied, indicating generally weak reasoning abilities. Negative values of PN and PS in several instances, are due to inconsistencies in D.L. and CF detailed in Section 4.1
Figure 7 (Left, Centre) features the 7-PN-overlap and 7-PN-overlap curves for all models and problems, where ideal reasoning corresponds to the metrics equalling one for any value of GPT- shows this level of reasoning for the Div6 problem. However, GPT-2 had an accurate PN for Even-Sun but the PS estimates were notably less accurate.

---

y-PN-overlap
y-PS-overlap
Div6
1.0
1.0
1.0
0.8
0.8
Overlog
0.6
0.6
0..4
0.4
04
GPT2
GPC2
0.2
GPF2
0.2
GPT-35-1
0.2
GPT-3544
GPE-35-4
GPT4
GPT4
0.0
GPF4
0.0
0.0
0.2
0.4
0.6
1.0
0.0
0.4
0.8
0.8
1.0
0.0
0.1 FIR
0.6
0.8
1.0
y-PN-overlap
y-PS-overlap
SumEven
1.0
1.0
0.8
0.8
0.6
0.6
GPF-2
GPT22
GPE2
GPT-35-1
GPT-354
GPE33-4
GPT4
GPT-4
0.0
X
GPT4
0.0
0.2
0.1
0.6
0.8
1.0
0.0
0.2
0.1
0.8
0.8
1.0
0.0
0.2
0.1 PIR
0.6
0.0
1.0
y-PN-overlap
y-PS-overlap
CandyParty
0.8
0.8
Overapp
0.6
0.6
0.6
0.4
0.4
0.4
0.2
GPT-35-1
0.2
GPT-35-4
GPF-35+
0.0
0.0
0.0
0.6
1.0
0.0
0.4
1.0
0.4 0.6 0.8
1.0
Figure 7: Left, Centre: Reconstruction of the 7-PN-overlap and 7-PS-overlap curves for GPT-2, GPT-35-turbo and GPT-4. Ideal reasoning is achieved when overlap is one for all values of y. Right: Visualization of FIR and PIR. Ideal reasoning is attained when both metrics are zero (denoted by a x).
Figure 7 (Right) presents the values of CIR and PIR (with the standard deviations included brackets). An emerging trend towards reasoning observed in the GPT family of models, particularly seen with GPT-4 for the Div6 problem. An intriguing question is whether future versions of these models will similarly approach the PN and PS for other problems as well.
5
Discussion
The primary objective of this paper was to explore and understand the reasoning abilities of LLMs, which essential for their successful deployment in range of applications. Given the growing dependence on LLMs for complex reasoning tasks, such as mathematics, programming, strategic planning, understanding this is crucial. To evaluate these reasoning abilities, we introduced novel framework that employs probabilistic measures of necessity and sufficiency, and find that while various models (GPT-2, GPT-3.5-turbo and GPT-4) can replicate aspects of reasoning to some degree, they often falter when it comes to counterfactual reasoning. What makes our approach unique is that we test the models with scenarios where generalization by reasoning, rather than replicating statistical patterns in the training data, is required to provide correct answers. Notably, the ability to reason, as defined in this paper, does improve with more complex models, yet it is still far from flawless. This observation leads to the question of whether future versions of these models will achieve perfect reasoning. Our results are significant, as they reveal the limitations of LLMs, and emphasize the need for further research to enhance their reasoning capabilities.
In general, reasoning goes beyond the math examples that we have included in the paper. believe that the same theory and tools can be used in other domains. The Hex framework, that serves as a mathematical framework to formalize our ideas, requires the definition of query, state, and an abstract execution machine that is used to predict how the state is modified given the query. In this

---

framework, for instance, one could think about problems in vision, where the elements of the state are the objects in an image, and the query corresponds to a counterfactual query that describes an intervention in the environment. The concepts of necessity and sufficiency still apply in this scenario. In an image where an object is removed or altered, the framework can help determine the impact of this change on property of the overall scene. We believe that this approach and will be key in other fields such as robotics, and or social sciences, where understanding the necessity and sufficiency between different elements is crucial for accurate reasoning and decision-making.
Limitations: Our approach has several limitations that we acknowledge, but did not address within the scope of this research.
1. Dependence on reasoning graphs: our method requires access to reasoning graphs. This requirement may hinder our ability to fully understand the reasoning abilities of LLMs in situations where it is challenging to derive relationships, including causal ones.
2. Boolean variable restriction: our method is designed to work with boolean valued variables, which is restrictive, particularly for cases involving multiple states or conditions occurring at the same time. However, we believe that this issue can be addresss with further research.
results: The findings we report are based on an LLM's s reasoning abilities 3. determined by two specific types (factual/counterffacuaa) of prompts that we used. Of course, other techniques like chain-of-thought prompting could be used, but our focus remains on establishing consistent baseline. Future work could explore the impact of different prompting strategies on reasoning performance, potentially leading to more refined and effecttive methods for evaluating and enhancing reasoning capabilities in LLMs. Our experiments did not aim to fine-tune these prompts or to 'optimise reasoning -a separate area of ongoing research. Instead, our goal was offer valuable insights that can aid the community in developing new benchmarks and employing LLMs responsibly.
Evaluating the reasoning capabilities of LLMs is essential as it significantly influences their effectiveness in various domains. In education and research, it is important for the model to be able to provide accurate explanations and to formulate meaningful hypotheses. In the commercial sector, the effectiveness of automated processes/systemss relies heavily on how well the model can reason. When it comes to accessibility, the model must be able to understand and meet diverse user needs, which hinges on its reasoning ability. Moreover, identifying and mitigating biases in AI systems-2 key aspect of ethical and equitable AI Alrrqqurrss detailed examination of the models' reasoning processes. Therefore, while LLMs hold immense promise, ensuring their responsible and beneficial use is predicated on a thorough appraisal of their reasoning abilities. We believe that our research is an important step in this direction.
Broader impact:
Acknowledgments
We would like to thank Ted Meeds, Alicia Curth and Sushrut Karmalkar for their invaluable discus- sions and feedback throughout the process of writing this paper.
References
[1] J. Ahn, R. Verma, R. Lou, D. Liu, R. Zhang, and W. Yin. Large language models for mathemat- ical reasoning: Progresses and challenges, 2024.
[2] A. Aliseda. Abductive Reasoning: Logical Investigations into Discovery and Explanation. Spring Publications, 2006.
[3] S. Ashwani, K. Hegde, N. R. Mannuru, M. Jindal, D. S. Sengar, K. C. R. Kathala, D. Banga, Jain, and A. Chadha. Cause and effect: Can large language models truly understand causality?, 2024.
[4] C. Bowen, R. Sætre, and Y. Miyao. A comprehensive evaluation of inductive reasoning capabilities and problem solving in large language models. In Findings of the Association for Computational Linguistics: FACL. 2024, page 323-339, St. Julian's ss Malta, March 2024.
Association for Computational Linguistics.
10

---

[5] S. Bubeck, V. Chandrasekaran, R. Eldan, J. A. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y. T. Lee, Y.-F. Li, S. M. Lundberg. H. Nori, H. Palangi. M. T. Ribeiro, and Y. Zhang. Sparks of Artificial General Intelligence: Farly experiments with GPT-4. ArXiv, abs/2303. 0et/2233311212 12712, 2023.
[6] Z. Chen, Q Gao, A. Bosselut, A. Sabharwal, and K. Richardson. DISCO: Distilling counter- factuals with large language models. In A. Rogers, J. Boyd-Graher, and N. Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5514-5528, Toronto, Canada, July 2023. Association for Computational Linguistics.
[7] A. Feeney and Heit, editors. Inductive Reasoning: Experimental, Developmental, and Computational Approaches. Cambridge University Press, 2007.
[8] W. Gan, Z. Qi, J. Wu, and J. C.-W. Lin. Large language models education: Vision and opportunities. arXiv preprint arXiv:231 /3/60, 2023.
[9] J. González and A. V. Nori. Beyond words: A mathematical framework for interpreting large language models. arXiv preprint arXiv:231 1.03033, 2023.
[10] González, C. Wong, Z. Gero, J. Bagga, R. Ueno, I. Chien, E. Oravkin, E. Kiciman, A. Nori, R. Weerasinghe, R. S. Leidner, B. Piening, T. Naumann, C. Bifulco, and H. Poon. Trialscope: A unifying causal framework for scaling real-world evidence generation with biomedical language models, 2023.
[11] S. Gunasekar, Y. Zhang, J. Aneja, C. Cesar, T. Mendes, A. D. Giorno, S. Gopi, M. Javaheripi, P. Kauffimann, G. de Rosa, O. Saarikivi, A. Salim, S. Shah, H. Singh Behl, X. Wang. S. Bubeck,
R. Eldan, A. T. Kalai, T. Lee, and Y. Li. Textbooks are all you need, June 2023.
[12] Halpern and Pearl. Causes and explanations: structural-moder approach. part i: Causes. British Journal of Philosophy of Science, 56(4):843-887, 2005.
[13] S. Hao, Y. Gu, H. Ma, J. J. Hong, Z. Wang. D. Z. Wang, and Z. Hu. Reasoning with language model is planning with world model. In Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, pages 8154-8173, Singapore, December 2023. Association for Computational Linguistics.
[14] J. Huang and K. C.-C. Chang. Towards reasoning in large language models: A survey, 2023.
[15] J. Huang and K. C.-C. Chang. Towards reasoning in large language models: A survey. In Findings of the Association for Computational Linguistics: ACL 2023, pages 1049-1065. Toronto, Canada, July 2023. Association for Computational Linguistics.
[16] Z. Jin, J. Liu, Z. Lyu. S. Poff, M. Sachan, R. Mihalcea. M. Diab, and Schölkopf. Can large language models infer causation from correlation?, 2024.
[17] D. Kelley. The Art of Reasoning (with symbolic logic). Unknown Publisher, 1990.
[18] E. Kıeıman, R. Ness, A. Sharma. and C. Tan. Causal reasoning and large language models: Opening a new frontier for causality, 2023.
[19] D. Lewis. Causation. Journal of Philosophy, 70:556-567, 1973.
[20] J. Li, L. Yu, and A. Ettinger. Counterfactual reasoning:
Testing language models' understanding of hypothetical scenarios. In A. Rogers, J. Boyd-Graber, and N. Okazaki, editors, Proceed- ings of the 6l Annual Meeting of the Association for Computational Linguistics (Volume Short Papers), pages 804-815, Toronto, Canada, July 2023. Association for Computational Linguistics.
[21] Y. Li, M. Xu, X. Miao, S. Zhou, and T. Qian. Large language models counterfactual generator: Strengths and weaknesses, 2023.
[22] Mackie. Causes and conditions. American Philosophical Quarterly. 2(4):245 1965.
[23] OpenAl. Gpt-4 technical report. ArXiv, abs/2303.08774, 2023.
[24] J. Pearl. Probabilities of causation: three counterfactual interpretations and their identification.
Synthese, 121(1-2):93-149, 1999.
[25] J. Pearl. Causality: Models, Reasoning, and Inference. Cambridge University Press, 2000.
[26] J. Pearl. Causality. Cambridge University Press, 2 edition, 2009.
[27] I.. J. Rips. The Psychology of Proof: Deductive Reasoning Human Thinking. The MIT Press,
1994.
11

---

[28] S. M. Seals and V. L. Shalin. Evaluating the deductive competence of large language models. arXiv preprint arXiv:2309.05452, 2023.
[29] Simmering and Huoviala. Large language models for aspect-based sentiment analysis. arXiv preprint arXiv:2310. 11000.882223 18025, 2023.
[30] G. Team, R. Anil, S. Borgeaud. J.-B. Alayrac. J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, K. Millican, D. Silver, M. Johnson, I. Antonoglou, J. Schrittwieser, A. Glaese, Chen, E. Pitler, T. Lillicrap. A. Lazaridou, O. Firat, J. Molloy, M. Isard, P. R. Barham, T. Hennigan, B. Lee, F. Viola, M. Reynolds, Y. Xu, R. Doherty, E. Collins, C. Meyer, E. Rutherford, E. Mor- eira, K. Ayoub, M. Goel, J. Krawczyk, C. Du, E. Chi, H.-T. Cheng, E. Ni, P. Shah, P. Kane, B. Chan, M. Faruqui, A. Severyn, H. Lin, Li, Y. Cheng, A. Ittcheriah, M. Mahdieh, M. Chen, P. Sun, D. Tran, S. Bagri, B. Lakshminarayanan, J. Liu, A. Orban, F. Güra, H. Zhou, X. Song, A. Boffy, H. Ganapathy, S. Zheng, H. Choe, Ágoston Weisz, T. Zhu, Y. Lu, S. Gopal, J. Kahn, M. Kula, Pitman, R. Shah, E. Taropa, M. A. Merey, M. Baeuml, Z. Chen, L., E. Shafey, Y. Zhang, O. Sercinoglu, G. Tucker, E. Piqueras, M. Krikun, I. Barr, N. Savinov, I. Danihelka, B. Roelofs, A. White, A. Andreassen, T. von Glehn, L. Yagati, M. Kazemi, L. Gonzalez. M. Khalman, J. Sygnowski, A. Frechette, C. Smith, L. Culp, L. Proleev, Y. Luan, X. Chen, J. Loutes, N. Schucher, F. Lebron, A. Rrustemi, N. Clay, P. Crone, T. Kocisky, J. Zhao, B. Perz, D. Yu, H. Howard, A. Bloniarz, J. W. Rae, H. Lu, L. Sifre, M. Maggioni, F. Alcober. D. Garrette, M. Barnes, S. Thakoor, J. Austin, G. Barth-Maron, W. Wong, R. Joshi, R. Chaabouni, D. Fatiha, A. Ahuja, G. S. Tomar, E. Senter, M. Chadwick, I. Kornakov, N. Attaluri, I. Iturrate, R. Liu, Y. Li, S. Cogan, J. Chen, C. Jia, C. Gu, Q. Zhang. J. Grimstad, A. J. Hartman, X. Garcia, T. S. Pillai, J. Devlin, M. Laskin, D. de Las Casas, D. Valter, C. Tao, L.. Blanco, A. P. Badia, D. Reitter, M. Chen, J. Brennan, C. Rivera, S. Brin, S. lqbal, G. Surita, J. Labanowski, A. Rao, S. Winkler, E. Parisotto, Y. Gu, K. Olszewska, R. Addanki, A. Miech, A. Louis, D. Teplyashin, G. Brown, E. Catt, J. Balaguer, J. Xiang, P. Wang. Z. Ashwood, A. Briukhov, A. Webson, S. Ganapa- thy, S. Sanghavi, A. Kannan, M.-W. Chang, A. Stjerngren, J. Djolonga, Y. Sun, A. Bapna, M. Aitchison, P. Pejman, H. Michalewski, T. Yu, C. Wang. J. Love, J. Ahn, D. Bloxwich, K. Han, Humphreys, T. Sellam, J. Bradbury, V. Godbole, S. Samangooei, B. Damoc, A. Kaskasoli, S. M. R. Arnold. V. Vasudevan, S. Agrawal, J. Riesa, D. Lepikhin, R. Tanburn, S. Srinivasan, H. Lim, S. Hodkinson, P. Shyam, J. Ferret, S. Hand, A. Garg, T. L. Paine, J. Li. Y. Li, M. Gi- ang, A. Neitz, Z. Abbas. S. York, M. Reid, E. Cole, A. Chowdhery, D. Das, D. Rogozinska, V. Nikolaev, P. Sprechmann, Z. Nado, L. Zilka, F. Prost, L. He, M. Monteiro, G. Mishra, C. Welty. J. Newlan, D. Jia, M. Allamanis. C. H. Hu, R. de Liedekerke, J. Gilmer, C. Saroufim, S. Rijhwani, Hou, D. Shrivastava, A. Baddepudi, A. Goldin, A. Ozturel, A. Cassirer, Y. Xu, D. Sohn, D. Sachan, R. K. Amplayo, C. Swanson, D. Petrova, S. Narayan, A. Guez, S. Brahma, J. Landon, M. Patel, R. Zhao, K. Villela, L. Wang, W. Jia, M. Rahtz, M. Giménez, L. Ye- ung. J. Keeling. P. Georgiev, D. Mincu, B. Wu, S. Haykal, R. Saputro, K. Vodrahalli, J. Qin, Z. Cankara, A. Sharma, N. Fernando, W. Hawkins, B. Neyshabur, S. Kim, A. Hutter, P. Agrawal, A. Castro-Ros, G. van den Driessche. T. Wang, F. Yang. S. yii Chang, P. Komarek, R. Mcllroy, M. Lučic, G. Zhang, W. Farhan, M. Sharman, P. Natsev, P. Michel, Y. Bansal, S. Qiao, K. Cao,
S. Shakeri, C. Butterfield, J. Chung, P. K. Rubenstein, S. Agrawal, A. Mensch, K. Soparkar, K. Lenc, T. Chung, A. Pope, L. Maggiore, J. Kay, P. Jhakra, S. Wang. J. Maynez, M. Phuong, T. Tobin, A. Tacchetti, M. Trebacz, K. Robinson, Y. Katariya, S. Riedel, P. Bailey, K. Xiao, N. Ghelani, L. Aroyo, A. Slone, N. Houlsby, X. Xiong, Z. Yang, E. Gribovskaya, J. Adler, M. Wirth, L. Lee, M. Li, T. Kagohara, J. Pavagadhi, S. Bridgers, A. Bortsova, S. Ghemawat, Z. Ahmed, T. Liu, R. Powell, V. Bolina, M. linuma, P. Zablotskaia, J. Besley, D.-W. Chung,
T. Dozat, R. Comaneseu, X. Si, J. Greer, G. Su, M. Polacek, R. L. Kaufman, S. Tokumine, H. Hu, F. Buchatskaya, Y. Miao, M. Elhawaty, A. Siddhant, N. Tomasev, J. Xing, C. Greer, H. Miller, S. Ashraf, A. Roy, Z. Zhang, A. Ma, A. Filos, M. Besta, R. Blevins, T. Klimenko, C.-K. Ych, S. Changpinyo, J. Mu, O. Chang, M. Pajarskas, C. Muir, V. Cohen, C. L. Lan, K. Haridasan, A. Marathe, S. Hansen, S. Douglas,
Samuel, M. Wang, S. Austin, C. Lan, J. Jiang, J. Chiu, J. A. Lorenzo, L. L. Sjösund, S. Cevey, Z. Gleicher, T. Avrahami, A. Boral, H. Srinivasan, V. Selo, R. May, K. Aisopos, L. Hussenot, L. B. Soares, K. Baumli, M. B. Chang, A. Recasens, B. Caine, A. Pritzel, F. Pavetic, F. Pardo, A. Gergely, J. Frye, V. Ramasesh, D. Horgan, K. Badola, N. Kassner, S. Roy, E. Dyer, C. Campos, A. Tomala, Y. Tang, D. E. Badawy, F. White, B. Mustafa, O. Lang, A. Jindal, S. Vikram, Z. Gong, S. Caelles, R. Hemsley, G. Thornton, F. Feng, W. Stokowiec, C. Zheng, P. Thacker, Çağlar Unlü, Z. Zhang, M. Saleh, J. Svensson, M. Bileschi, P. Patil, A. Anand, R. Ring, K. Tsihlas, A. Vezer, M. Selvi, T. Shevlane,
12

---

M. Rodriguez, T. Kwiatkowski, S. Daruki, K. Rong, A. Dafoe. N. FitzGerald, K. Gu-Lemberg,
M. Khan, L. A. Hendricks, M. Pellat, V. Feinberg, J. Kobon-Kerr, T. Sainath, M. Rauh, S. H. Hashemi, R. Ives, Y. Hasson, F. Noland, Y. Cao, N. Byrd, I.. Hou, Q. Wang. T. Sottiaux, M. Pa- ganini, J.-B. Lespiau, A. Moufarek, S. Hassan, K. Shivakumar, J. van Amersfoort, A. Mandhane, P. Joshi, A. Goyal, M. Tung, A. Brock, H. Sheahan, V. Misra, C. Li, N. Rakićevic, M. De- hghani, F. Liu, S. Mittal, J. Oh, S. Noury, E. Sezener, F. Huot, M. Lamm, N. D. Cao, C. Chen, S. Mudgal, R. Stella, K. Brooks, G. Vasudevan, C. Liu, M. Chain, N. Melinkeri, A. Cohen, V. Wang, K. Seymore, S. Zubkov, R. Goel, S. Yue, S. Krishnakumaran, B, Albert, N. Hur- ley, M. Sano, A. Mohananey, J. Joughin, E. Filonov, T. Kępa, Y. Eldawy, J. Lim, R. Rishi, S. Badiezadegan, T. Bos, J. Chang, S. Jain, S. G. S. Padmanabhan, S. Puttagunta, K. Krishna, L. Baker, N. Kalb, V. Bedapudi, A. Kurzrok, Lei. A. Yu, O. Litvin, X. Zhou, Z. Wu, S. So- bell, A. Siciliano, A. Papir, R. Neale, J. Bragagnolo, T. Toor, T. Chen, V. Anklin, F. Wang, R. Feng, M. Gholami, K. Ling, I.. Liu, J. Walter, H. Moghaddam, A. Kishore, J. Adamek, T. Mercado, J. Mallinson, S. Wandekar, S. Cagle, F. Ofek, G. Garrido, C. Lombriser, M. Mukha, B. Sun, H. R. Mohammad, J. Matak, Y. Qian, V. Peswani, P. Janus, Q. Yuan, L. Schelin, O. David, A. Garg, Y. He, O. Duzhyi, A. Älgmyr. T. Lottaz, Q. Li, V. Yadav, L. Xu, A. Chinien, R. Shivanna, A. Chuklin, J. Li, C. Spadine, T. Wolfe, K. Mohamed, S. Das, Z. Dai, K. He, D. von Dincklage, S. Upadhyay, A. Maurya, L. Chi, S. Krause, K. Salama, P. G. Rabinovitch, K. R. M. A. Selvan, M. Dektiarev, G. Ghiasi, E. Guven, H. Gupta, B. Liu, D. Sharma, I. H. Shtacher, S. Paul, O. Akerlund, F.-X Aubet. T. Huang, C. Zhu, E. Zhu, E. Teixeira, M. Fritze, Bertolini, L.-E. Marinescu, M. Bölle, D. Paulus. K. Gupta. T. Latkar, M. Chang. J. Sanders, R. Wilson, X. Wu, Y.-X. Tan, I.. N. Thiet, T. Doshi, S. Lall, S. Mishra, W. Chen, T. Luong, S. Benjamin, Lee, E. Andrejezuk, D. Rabiej, V. Ranjan, K. Styre, P. Yin, J. Simon, M. R. Harriott, M. Bansal, A. Robsky, G. Bacon, D. Greene, D. Mirylenka, C. Zhou, O. Sarvana, A. Goyal, S. Andermatt, P. Siegler, B. Horn, A. Israel, F. Pongetti, C.-W. L. Chen, M. Selvatici, P. Silva, K. Wang. J. Tolins, K. Guu, R. Yogev, X. Cai, A. Agostini, M. Shah, H. Nguyen, N. O. Donnaile, S. Pereira, L. Friso, A. Stambler, A. Kurzrok, C. Kuang, Y. Romanikhin, M. Geller, Z. Yan, K. Jang, C.-C. Lee, W. Fica, E. Malmi, Q. Tan. D. Banica, D. Balle. R. Pham, Y. Huang, D. Avram, H. Shi, J. Singh, C. Hidey, N. Ahuja, P. Saxena, D. Dooley, S. P. Potharaju, E. O'Neill, A. Gokulchandran, R. Foley, K. Zhao, M. Dusenberry, Y. Liu, P. Mehta, R. Kotikala- A. Goodman, J. Kessinger, E. Globen. P. Kolhar, C. Gorgolewski, A. Ibrahim, Y. Song. A. Eichenbaum, T. Brovelli, S. Potluri, P. Lahoti, C. Baetu, A. Ghorbani, C. Chen, A. Crawford, S. Pal, M. Sridhar, P. Gurita, A. Mujika, I. Petrovski, P.-L. Cedoz, C. Li, Chen, N. D. Santo, S. Goyal, J. Punjabi, K. Kappaganthu, C. Kwak, P. LV, S. Velury, H. Choudhury, J. Hall, P. Shah, R. Figueira, M. Thomas, M. Lu, T. Zhou, C. Kumar, T. Jurdi, S. Chikkerur, Y. Ma, A. Yu, S. Kwak, V. Ähdel, S. Rajayogam, T. Choma, Liu, A. Barua, C. Ji, J. H. Park, V. Hellendoorn, A. Bailey, T. Bilal, H. Zhou, M. Khatir, C. Sutton, W. Rzad- kowski, F. Macintosh, K. Shagin, P. Medina, C. Liang, J. Zhou, P. Shah, Y. Bi, A. Dankovies, S. Banga, S. Lehmann, M. Bredesen, Z. Lin, J. E. Hoffmann, J. Lai, R. Chung, K. Yang, N. Bal- ani, A. Bražinskas, A. Sozanschi, M. Hayes. H. F. Alcalde, P. Makarov, W. Chen, A. Stella, L. Snijders. M. Mandl, A. Kärrman, P. Nowak, X. Wu, A. Dyck, K. Vaidyanathan, R. R, J. Mallet, M. Rudominer, E. Johnston, S. Mittal, A. Udathu, J. Christensen, V. Verma, Z. Irving, A. Santucci, G. Elsayed, E. Davoodi, M. Georgiev, I. Tenney, N. Hua, G. Cideron, E. Leurent, M. Alnahlawi, I. Georgeseu, N. Wei, I. Zheng. D. Scandinaro, H. Jiang, Snoek, M. Sundarara- jan, X. Wang. Z. Ontiveros, I. Karo, J. Cole, V. Rajashekhar, L. Tumeh, E. Ben-David, R. Jain, J. Uesato, R. Datta, O. Bunyan, S. Wu, J. Zhang, P. Stanczyk, Y. Zhang, D. Steiner, S. Naskar, M. Azzam, M. Johnson, A. Paszke, C.-C. Chiu, J. S. Elias, A. Mohiuddin, F. Muhammad, J. Miao, A. Lee, N. Vieillard, J. Park, J. Zhang, J. Stanway, D. Garmon, A. Karmarkar, Z. Dong, J. Lee, A. Kumar, L. Zhou, J. Evens, W. Isaac, G. Irving, E. Loper, M. Fink, I. Arkatkar, N. Chen, I. Shafran, I. Petrychenko, Z. Chen, J. Jia, A. Levskaya, Z. Zhu, P. Grabowski, Y. Mao, A. Magni, K. Yao, J. Snaider, N. Casagrande, E. Palmer, P. Suganthan, A. Castaño, I. Giannoumis, W. Kim, M. Rybiński, A. Sreevatsa, J. Prendki, D. Soergel, A. Goedeckemeyer, W. Gierke, M. Jafari, M. Gaba, J. Wiesner, D. G. Wright, Y. Wei, H. Vashisht, Y. Kulizhskaya, J. Hoover, M. Le, L. Li, C. Iwuanyanwu, L. Liu, K. Ramirez, A. Khorlin, A. Cui, T. LIN, M. Wu, R. Aguilar, K. Pallo, A. Chakladar, G. Perng, E. A. Abellan, M. Zhang, I. Dasgupta, N. Kushman, I. Penchev, A. Repina, X. Wu, T. van der Weide, P. Ponnapalli, C. Kaplan, J. Simsa, S. Li, O. Dousse, F. Yang. J. Piper, N. Ie, R. Pasumarthi, N. Lintz, A. Vijayakumar, D. Andor, P. Valenzuela, M. Lui, C. Paduraru, D. Peng, K. Lee, S. Zhang, S. Greene, D. D. Nguyen, Kurylowicz, C. Hardin, L. Dixon, L. Janzer, K. Choo, Z. Feng, B. Zhang, A. Singhal, D. Du.
pudi, C. Safranck-Shradee
13

---

D. McKinnon, N. Antropova, T. Bolukbasi, O. Keller, D. Reid, D. Finchelstein, M. A. Raad, R. Crocker, P. Hawkins, R. Dadashi. C. Gaffney, K. Franko, A. Bulanova, R. Leblond, S. Chung, H. Askham, L. C. Cobo, K. Xu, F. Fischer, J. Xu, C. Sorokin, C. Alberti, C.-C. Lin, C. Evans, A. Dimitriev, H. Forbes, D. Banarse, Z. Tung, M. Omernick, C. Bishop, R. Sterneck, R. Jain, J. Xia, E. Amid, F. Piccinno, X. Wang, P. Banzal, D. J. Mankowitz, A. Polozov, V. Krakovna, S. Brown, M. Bateni, D. Duan, V. Firoiu, M. Thotakuri, T. Natan, M. Geist, S. tan Girgin, H. Li, J. Ye, O. Roval, R. Tojo, M. Kwong, J. Tee-Th C. Yew, D. Sinopalnikov, S. Ramos, J. Mellor, A. Sharma, K. Wu, D. Miller, N. Sonnerat, D. Vnukov, R. Greig, J. Beattie, E. Caveness, L. Bai, J. Eisenschlos, A. Korchemniy, T. Tsai, M. Jasarevic, W. Kong, P. Dao, Z. Zheng, F. Liu, F. Yang, R. Zhu, T. H. Teh, J. Sanmiya, E. Gladchenko, N. Trdin, D. Toyama, E. Rosen, S. Tavakkol, L. Xue, C. Elkind, O. Woodman, J. Carpenter, G. Papamakarios, R. Kemp, S. Kafle, T. Grumina, R. Sinha, A. Talbert, D. WU, D. Owusu Afriui-, C. Du, C. Thornton, J. Pont-'Tuset, P. Narayana, J. Li, S. Fatehi, J. Wieting, O. Ajmeri, B. Uria, Y. Ko, L. Knight, A. Héliou, N. Niu, S. Gu, C. Pang. Y. Li, N. Levine, A. Stolovich, R. Santamaria-Fermander S. Goenka, W. Yustalim, R. Strudel, A. Elqursh, C. Deck, H. Lee, Z. Li, K. Levin, R. Hoffmann, D. Holtmann-Riin O. Bachem, S. Arora, C. Koh, S. H. Yeganch, S. Pöder, M. Tariq, Y. Sun, L. Ionita, M. Seyed- hosseini, P. Tafti, Z. Liu, A. Gulati, J. Liu, X. Ye, B. Chrzaszez, L. Wang, N. Sethi, T. Li, B. Brown, S. Singh, W. Fan, A. Parisi, J. Stanton, V. Koverkathu, C. A. Choquette-Choo, Bahargam, R. Willoughby, D. Gaddy, G. Desjardins. M. Cornero. B. Robenek, B. Mittal, B. Albrecht, A. Shenoy, F. Moiseev, H. Ja- cobsson. A. Ghaffarkhah, M. Rivière, A. Walton, C. Crepy. A. Parrish, Z. Zhou, C. Farabet, C. Radebaugh, P. Srinivasan, C. van der Salm, A. Fidjeland, S. Scellato, F. Latorre-Chimota D. Bridson, D. de Cesare, T. Hudson, Mendolicchio, L.. Walker, A. Morris, M. Mauger, A. Guseynov, A. Reid, S. Odoom, Loher, V. Cotruta, M. Yenugula, D. Grewe, A. Petrushkina, T. Duerig, A. Sanchez, S. Yadlowsky, A. Shen, A. Globerson, L. Webb, S. Dua, D. Li, S. Bhupatiraju, D. Hurt, H. Qureshi, A. Agarwal, T. Shani, M. Eyal, A. Khare, S. R. Belle, L. Wang, C. Tekur, M. S. Kale, J. Wei, R. Sang, B. Saeta, T. Liechty, Y. Sun, Y. Zhao, S. Lee, P. Nayak, D. Fritz, M. R. Vuyyuru, J. Aslanides, N. Vyas, M. Wicke, X. Ma, E. Eltyshev, N. Martin, H. Cate, J. Manyika, K. Amiri, Y. Kim, X. Xiong. K. Kang, F. Luisier, N. Tripuraneni, Madras, M. Guo, A. Waters, O. Wang. J. Ainslie, J. Baldridge, H. Zhang. G. Pruthi, J. Bauer, F. Yang, R. Mansour, J. Gelman, Y. Xu, G. Polovets, J. Liu, H. Cai, W. Chen, X. Sheng, E. Xue, S. Ozair, C. Angermueller, X. Li, A. Sinha, W. Wang, J. Wiesinger, E. Koukoumidis, Y. Tian, A. Iyer, M. Gurumurthy, M. Goldenson, P. Shah, M. Blake, H. Yu, A. Urbanowicz, J. Palomaki, C. Fernando, K. Durden, H. Mehta, N. Momchev, E. Rahimtoroghi, M. Georgaki, A. Raul, S. Ruder, M. Redshaw, J. Lee, D. Zhou, K. Jalan, D. Li, B. Hechtman, Schuh, M. Nasr, K. Milan, V. Mikulik, Franco, T. Green, N. Nguyen, J. Kelley, A. Mahen- dru, A. Hu, J. Howland, B. Vargas, J. Hui, K. Bansal, V. Rao, R. Ghiya, E. Wang, K. Ye, J. M. Sarr, M. M. Preston, M. Elish, S. Li, A. Kaku, J. Gupta, I. Pasupar, D.-C. Juan, M. Someswar, T. M., X. Chen, A. Amini, A. Fabrikant, E. Chu, X. Dong, A. Muthal, S. Buthpitiya, S. Jaubari, N. Hua, U. Khandelwal, A. Hitron, Ren, L. Rinaldi, S. Drath, A. Dabush. N.-J. Jiang, H. God- hia, U. Sachs, A. Chen, Y. Fan, H. Taitelbaum, H. Noga, Z. Dai, J. Wang. C. Liang, J. Hamer, C.-S. Ferng, C. Elkind, A. Atias, P. Lee, V. Listík, M. Carlen, van de Kerkhof, M. Pikus, K. Zaher, P. Müller, S. Zykova, R. Stefanec, V. Gatsko, C. Hirnschall, A. Sethi, X. F. Xu,
Y. Li, T. Lu, A. Ittycheriah, P. Shroff, M. Varadarajan,
H.
C. Ahuja, B. Tsai, A. Stefanoiu, B. Feng. K. Dhandhania, M. Katyal, A. Gupta, A. Parulekar, D. Pitta. J. Zhao, V. Bhatia, Y. Bhavnani, O. Alhadlaq, X. Li, P. Danenberg, D. Tu, A. Pine, V. Filippova, A. Ghosh, B. Limonchik, B. Urala, C. K. Lanka, D. Clive, Y. Sun, E. Li, H. Wu, K. Hongtongsak, I. Li, K. Thakkar, K. Omarov, K. Majmundar, M. Alverson, M. Kucharski, M. Patel, M. Jain, M. Zabelin, P. Pelagatti, R. Kohli, S. Kumar, J. Kim, S. Sankar, V. Shah, L. Ramachandrumi, X. Zeng, B. Bariach, L. Weidinger, A. Subramanya, S. Hsiao, D. Hassabis, K. Kavukcuoglu, A. Sadovsky, Q. Lc, T. Strohman, Y. Wu, S. Petrov, J. Dean, and O. Vinyals. Gemini: A family of highly capable multimodal models, 2024.
[31] J. Tian and Pearl. Probabilities of causation: Bounds and identification. Annals of Mathematics and Artificial Intelligence, 28(1 (111)144)-731111 2000.
[32] Y. Tong, D. Li, S. Wang, Y. Wang, F. Teng, and J. Shang. Can Ilms learn from previous mistakes? investigating 1mss errors to boost for reasoning. arXiv preprint arXiv:2403.2000 2024.
[33] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample. Ilama: Open and efficient foundation language models, 2023,
14

---

[34] J. Wei, X. Wang, D. Schuurmans, M. Bosma, b. ichter, F. Xia. E. Chi, Q. V. Le, and D. Zhou. Chain-of-thought prompting elicits reasoning large language models. In Koyejo. S. Mo- hamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 24824-24831 Curran Associates, Inc., 2022.
[35] C. Wong. S. Zhang. Y. Gu, C. Moung, J. Abel, N. Usuyama, R. Weerasinghe, B. Piening, T. Naumann, C. Bifulco, and H. Poon. Scaling clinical trial matching using large language models: A case study in oncology. 2023.
[36] C. Zhang, S. Bauer, P. Bennett, J. Gao, W. Gong, A. Hilmkil, J. Jennings, C. Ma. T. Minka,
N. Pawlowski, and J. Vaughan. Understanding causality with large language models: Feasibility and opportunities, 2023.
15

---

Appendix for: 'Does Reasoning Emerge? Examining the Probabilities of Causation in Large Language Models'
A Causal models, counterfactuals and probabilities of causation
This section provides technical background on structural causal models, counterfactuals and probabil- ities of causation 25
Definition 4 (Causal Model), A causal model M is triple (V,F, where:
1. V = (Vi.... 3 is set of endogenous variables.
2. 5 fei....,En! En is a set of exogenous variables. The exogenous variables & are assumed to be independent of each other and represent the unobserved factors that influence the values of V.
3. == = if....., f is a set of functions. Each function ft determines the value of as function of its parents PA; C V U E, where PA, are the variables that directly cause V,,
Any causal model can be represented by directed acyclic graph (DAG) G. where the nodes represent the variables V, and the edges are the direct causal relationships between these variables. Let X be subset of variables in V, and be specific realization of the values these variables take. We define a submodel Mx=z to be a causal model (V.Fe. e), where Fi = {f: V U =
Definition (Intervention, operator). Consider a causal model M = (V,F,e), with being a subset of variables in X, and a particular realization Tf The effect of the intervention do(X x) M given by the submodel Mx
Definition 6 (Potential outcome and counterfactual), Let Y be variable in V, and let X be subset of V. The potential outcome of Y resulting from the intervention do(X = x), denoted by Yx==(e) y, the solution for Y in the set of equations Fr, A counterfactual is defined as the potential outcome Xx=z(e) for the hypothetical scenario "what would the value that Y have been if x=".
X had been
A distribution P over the exogenous variables & establishes a corresponding a probability distribution over the endogenous variables X' well as the potential outcomes. In practical applications, PP(e) characterizes the target population of the study. The probability of counterfactual Yx=z² induced by the submodel Mx==
P(Yx=I = -))
Σ
P(e)
(elxx=r(e)=v)
In addition, probabilities of the type P(Yx=m' X = =x, =,Y = y) can be computed as
-Yx=>=>>|X*
v) Σ P(Yx=r y') X = x=x
By conditioning on X = x and Y = the counterfactual outcome y' under the intervention do(X z') is the expectation of the index function Yx=z' y' with respect to the updated probability distribution PP(c|X = x, Y y). Three special cases of distributions of this type are of special interest for us.
Definition 7 (Probability of necessity, 24 Let X and Y be two binary variables in a causal model
M F.E). The probability of necessity (PN) is defined as:
PN = P((X=r'
Definition (Probability of sufficiency, 24)). Let X and and Y be two binary variables causal model M = (xx F.E). The probability of sufficiency (PS) is defined as:
PS : P(Y= v -'
The PN is the probability observing a different outcome the absence of the event X æ. The PS is the probability of z to generate in cases where both had different values (x' and 3²).
16

---

Definition 9 (Probability of necessity and sufficiency, (24)). Let X and and Y be two binary variables in causal model M F,,FF,E, E). The probability of necessity and sufficiency defined
PNS P(ye
P(t.))))
The PNS computes the probability that X = z is the only way of obtaining Y y, other words, the probability that X = x is both necessary and sufficient to observe Y y. The probabilities PN, PS and PNS are not identifiable with observational experimental data unless Y is monotonic with respect to X, and both observational and experimental data are available [31] this condition is satisfied, then they identifiable and can be computed follows:
P(y) P(yldo(x))) and PS ((y]do(z))-Pi)) P(x. v)
PN
P(x',1"
Note that PN and PS require the knowledge of do x) and do z'). These quantities are generally unobserved for the whole population since observed individuals are only subject to one of the two conditions, unless experimental data available.
B ConPref problem
Congruent preferences (ConPref): Consider three real numbers M, N and T. If M N and N < T. then M T. We compute PN and PS for the condition M < N (Cmn) to having enough evidence know if M S T (Cmaa). If M N or N < T false then Cmnt is false. For our evaluation, we consider all combinations of values for M, N and T,, for numbers between I and 8.
Reasoning graph
ConPref
N
M
Cum C
Cumit
Figure Reasoning graph for the ConPref problem.
Reasoning results
ConPref
1.0
GPT-4
0.5
GPT3.5-L
Sufttleene
0.4
0.2
0.0
X
True PoC
GPT2
Necessity
Figure 9: True PN and PS inferred PN and PS using GPT-2, GPT-35-turbo and GPT-4 for the ConPref problem..
17

---

y-PN-overlap
y-PS-overlap
ConPref
1.0
1,0
1.0
0.8
0.8
Overle
0.6-
0.6-
00.4
0.4
04
GPT-2
GPC2
0.2
GPF2
0.2
GPT-35-1
0.2
GPT--354
GPE-35-4
GPT4
GPP4
0.0
GPF4
0.0
0.2
0.4
0.6
1.0
0.0
0.4
0.8
0.8
1.0
0.1
0.6
0.8
1.0
FIR
Figure 10: Left. Centre: Reconstruction of the PN-oerlp and y-PS-overlap curves for GPT-2, GPT-35-turbo and GPT-4 for the ConPref problem. Ideal reasoning is achieved when the overlap is one for all values of y- Right: Visualization FIR and PIR. Ideal reasoning attained when both metrics are zero (denoted by x).
C Problems: structural equations
C.1 Div6
N PN
C2 N (mod 3)
C3 = (mod 2) =
C. C2 C3
where C2. C3 and Co represent boolean values that indicate whether the number is divisible by 3 and 6 respectively. PN is the mechanism for generating the original numbers (1 to 400 in our examples).
C.2 EvenSum
N
PN
M
PM
T
Pr
Cn
N
(mod 2)
Cm
M
(mod 2)
C. T
(mod 2)
Came (C, + Om + C, 1) (Cn + On + C. =3) 3)
where C, Omn Ct and Camt represent boolean values and PN, PM, Py the mechanisms to generate the original numbers (1-8 in our examples).
C.3 ConPref
N
PN
M
PM
T
Pr
Cum N NSM M
Cmt
MST
Camt Cam A Cent
where Cam, Gmt and Crmtt represent boolean values and Pv. PM, Pr are the mechanisms to generate the original numbers (1-8 in our examples).
18

---

C.4 Candy Party
R
PR
L Pt
E PE
Cr = >
Cr L
Ce>o >
Cr22 R 2
C122
L 2 2
Cez2 E F
Crl R> L
Cre R> E
Cize L = E
C220,20,220 Cr20 A O1z0 A Co20
Cr>trze
Cre
Cr22,22,022
Cr222 A Coz2
C
(Cr22/22.e22 = 1) A (Cize C220.20.220)
where PR. PL, PE are the mechanisms generate the original numbers (all combinations in which 20 candies can be shared in our example).
D Proof of LLMs Zero-counterfactual consistency
Proof. Commutability of the Hex diagram implies that all paths from do or result in the same outcome. This holds for all counterfactuals, which implies that LLMM and Therefore:
X=x -x== for any value of X
Ex(xYy (YLL + +X==] =0
for any P(X, Y, Z).
E Direct and counterfactual prompts
E.1 Div6 problem
Direct Prompt:
"Does divide {'X}? Use the factor method to this question. Be as concise as possible."
Counterfactual Prompt:
"Imagine that {'X²} ('has'/'ha not'} prime factor while retaining all its other prime factors. With this assumption does {self divisor) divide {'P?? Use the factor method to answer this question. Be as concise possible."
E.2 EvenSum problem
Counterfactual Prompt:
"Let N, and T be three integers. Then N+M+T even if the three numbers are even or if only one is even and the remaining two are odd. Consider the numbers N=(N), M=(M) and T=(T). N+M+T even? as concise as possible." "
19

---

Direct Prompt:
Let M and T be three integers. Then N+M+T is if the three numbers are even or if only one is even and the remaining two are odd. Consider the numbers N=(N;, M=(M} and T=(T) and imagine that N (is/is not} even. With this assumption, is N+M+T even? Be as concise as possible."
E.3 ConPref problem
Direct Prompt:
"Let N, and T three integers. We know that if smaller or equal that M and M is smaller equal than T then N is smaller or equal than Consider numbers N=(N), M=(O and T=(T). By only looking at the relationships (N=(N) vs. (={M)) and (M=(M) vs. T=(T)), can we know if is smaller or equal that T? Be as concise as possible."
Counterfactual Prompt:
"Let N, M and T be three integers. We know that if N is smaller equal that M and M is smaller or equal than then N is smaller or equal than T. Consider the numbers N=(N), M=(O} and T=(T). Now imagine that the number N {'is smaller or equal '/ 'is not smaller or equal'} than M. Even if this contradict the values of the numbers X and relationships between and (=(M) and T=(T), to decide if can we tell if N is smaller or equal that T? Don't make any conclusion or comment based on the values, just based on the assumption and the relationships. Be as concise as possible."
this assumption and the
E,4 CandyParty problem
Direct Prompt:
"Rafa has invited Lara and Emma to his birthday party. He has (num candies) to distribute among them. They all will happy in party one the following cases: 1) Each of them gets at least 2 candies or 2) Lara and Emma get the same number of candies, but at least one candy each, and Rafa gets more than them. After distributing the candies, Lara gets {L}, Emma gets {E} and Raphael gets (R) candies. With this candies distribution, will they all be happy in the party? Be concise possible."
Counterfactual Prompt:
Rafa has invited Lara and Emma to his birthday party. He has {num candies) candies to distribute among them. They all will be happy in the party in one of the following cases: 1) Each of them gets at least 2 candies or 2) Lara and Emma get the same number of candies, but least one candy each, and Rafa gets more than them After distributing the candies. After distributing the candies, Lara gets {L}, Emma gets {E} and Rafa gets (R) candies. Consider the number of candies distributed to each of them and imagine that they think that {'Lara and Enma have the same number of candies' LLLara and Emma have different number of candies'}. With this assumption, will they all happy in the party? Be as concise as possible."
20

---

GPT-4 concretization
Prompt:
"You are an entity extractor expert. I am going give you guestion-ansser pair. want you to say if the meaning of answer is positive or negative. If the answer has words like "Yes' this will make it positive. If the answer contains words like 'No' this will make it negative. Always answer with only one word (Positive or Negative). For the question '(question)' and the answer '(answer)' the meaning is"
G HEX for counterfactual query
Qco
60
81
0 (N + 10,
d =
{N + 10,
(N + 10,
C2 + F,
Qc, True
C3 +T
Qco
C3 T,
C6 + F,
Cs F,
Co -T
C2 T}
C2 T}
C2 + T}
Figure 11: The HEX diagram for counterfactual query in the Div6 problem. We split the query in two sub-queries Qca True and Qc, that performs the two operations need to compute the counterfactual state. QCyeTr only sets the value of C3 to True. Qc, replaces the value of Cs by its counterfactual. This operation can be executed via the concrete path (using the structural causal model the problem) of by using an LLM.
H Evaluation metrics
Let be the number of instances of each problem. For example, 400 for the Div6 problem because we use the first 400 integers to test reasoning. For the intervention node X and the outcome note Y, distinguish between factual predictions Y|X (simulated from the original reasoning graph) and counterfactual predictions Yx=z (simulated from the intervened graph). We respectively denote the LLM versions of this quantities y factual and counterfactual prompts.
X and X== which are computed via
FIR
(YLLM x + (Y|X = x)]
(3)
CIR
Σ I YLLM n
1
# Yx=r)
(4)
Let m be the number of bootstrap samples used from the binary answers of the LLM. PN, and PS; are estimation of PN and PS for the ith bootstrap sample. Then
7 PNO
m
PP
LLM
PN
s
(5)
m j=1
PNS =
PS
LLM
-PS| 5
(6)
21

---

I Element-wise PIR and FIR
Elements PIR (%) SumEven
Average PIR
GPT2
GPT-35-turbo
005
GPT4
-00
68
880
110
132
154
176
196
308
330
352
374
396
418
440
462
464
506
0.5
Ervor
Number
Elements CIR (%)- SumEven
Average CIR
10
GPT-35-turbo
GРP4
22
A
66
80
110
13R
154
176
1960
242
264
300
330
352
374
418
440
404
508
0.5
Error
Number
Elements PIR (%) CandyParty
Average PIR
GPT2
GPT-35-turbo-
30
80
100
120
130
140
150
150
170
180
190
200
210
220
0.5 Error (%)
Elements CIR (%) CandyParty
Average CIR
GFP2
GPT-35-turbo
65
GPF4
30
50
60
70
80
100
120
130
150
170
100
190
200
210
220
0.5
Number
Elements PIR (%) ConPref
Average PIR
100
GPT2
GPE-35-turbo
GPT4
22
4
66
08
110
132
154
176
1990
220
242
2014
206
308
330
352
374
396
418
440
462
484
506
0.5 Error (%)
Nombes
Hamber-2
Elements CIR (%) ConPref
Average CIR
GPT2
GPT-35-turbo
GPF4
-00
66
80
110
132
154
176
198
220
242
2014
300
330
352
374
396
418
440
662
494
506
0.5
Number
Error (%)
Figure 12: Element and aggregated CIR and FiR for the SumEven and CandyParty problems.
22

---

Experiments with other model families
Div6 llama2
Div6 Phi
0.8
86
Liama
Phi-1
Liama
Phi-2
0.5
True
PoC
Sufficiency
Sufficiency
0.4
0.3
Uama 2-7b
Usma 2:130
MR-2
07
0.8
10
Necessity
Necessity
Figure 13: in other families of models. We re-run the Divó problem with the same setup used in the paper with two new families of models: Llama (7-7b and 13b) and Phi (1,2). The results consistent with the findings discussed in the main body of the paper
23

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: All claims in abstract and title match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes].
Justification: We have included list limitations in the conclusions of the work.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: All proofs to the claims in the paper are available in the supplementary materials.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
The methods and the experimental setup are detailed the main body of the paper and are enough to reproduce the evaluation metrics and the experimental results of this work. No new model or achitecture is proposed.
Justification:
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: Code is not provided, but the prompts used to extract the results of the language models are detailed in the supplementary materials and can be used to reproduce the results this work.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All details needed to reproduce the experimental section this work are included in the paper.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Error bars are included in the results the paper as well metrics that capture the statistical significance of the experiments.
8. Experiments Compute Resources

---

For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [NA]
Justification: Experiments do not require extensive compute resources (like large memory of GPUs) since they only require calls to GPT API models and some simple local computation. The experiments of this work can be reproduced in any personal laptop.
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: have followed the NeurIPS Code of Ethics.
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: An impact statement has been included after the conclusions.
Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: No models are trained or datasets released in this work. All the experiments are based simulated data that can be reproduced following the indications described in the work.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have properly reference all authors and assets used in this work.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: No assets were generated in the development these research. The metrics proposed in this work can be easily implemented are used by anyone interested.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Only simulated data used in the research of this work.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human
Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Only simulated data used in the research of this work.
25


Reviews:
[{"review_id": "i6M3xeaLdh", "reviewer": "Reviewer_1YAi", "summary": "This paper evaluates the reasoning capabilities of LLMs within the framework of Judea Pearl's hierarchy of causality, focusing particularly on the ability to perform counterfactual reasoning.\n\nLLMs are perceived as (non-deterministic) abstract machines within the HEX framework. The study assesses these models by comparing their performance on factual versus counterfactual problems, which are structured as reasoning graphs of boolean conditions. The necessity (PN) and sufficiency (PS) probabilities are estimated for 4 reasoning tasks: Divisibility by 6, Even Sum, Candy Party, and ConPref, ranging in complexity.\n\nThe paper investigates whether the actual PN and PS align with those estimated from both factual and counterfactual data, using models from the GPT family (GPT-2, GPT-3.5, and GPT-4).\n\nFindings show limited reasoning capabilities for the evaluated model family, with GPT-4 performing the best on the divisibility task.", "questions": "- I assume all findings are based on temperature > 0 to fulfill the variability criterion.\nDid you generate multiple completions for each prompt (factual and counterfactual), and if so, how were these incorporated into the final analysis? \"10 replicated tests\" is mentioned under Figure 5, How these 10 contribute to the final findings?\n\n- Was prompt optimization attempted for smaller models as well, or just for GPT-4? Given the sensitivity of LMs to prompts, could there be a bias towards GPT-4 because it handles more complex prompts better?\n\n-While the paper discusses PN and PS as measures of reasoning, there is less focus on how these measures help in interpreting the decisions making process. What are the main benefits in comparison to just using accuracy?", "limitations": "yes. See Weaknesses and Questions.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Applies probabilistic measures of necessity and sufficiency to asses specifically the reasoning capabilities of LLMs, offering a nuanced evaluation that extends beyond the accuracy metric. \n\n- Evaluates and compares LLMs performance on both factual and counterfactual problems.", "weaknesses": "- From the introduction \"...reasoning is typically understood to be the ability of these models to demonstrate emergent capabilities that surpass mere statistical pattern recognition in the training set.\". Without knowledge of or control on pre-training/post-training data for the GPT model family, it is difficult to infer that a measured reasoning capability via the introduced framework is due to \"emergence\" and not pattern recognition in the training data.\n\n- The framework's applicability is reduced to reasoning tasks that can be represented as graphs of boolean variables, which may limit its broader adoption.\n\n- Line 84-85: \"The closer the estimated PN/PS values to the actual PN/PS values, the better it is at reasoning. \" needs further elaboration. Specifically, an explanation of how these metrics correlate with enhanced reasoning capabilities would help in understanding the effectiveness of the framework.\n\n- The framework imposes that LMs are abstract machines, suggesting deterministic behavior. Yet estimating probabilities of necessity (PN) and sufficiency (PS) inherently relies on variability in model responses. \n\n- Statistical significance of observed results could further support the validity of the framework.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the feedback and appreciate the comment that our methods offers an nuanced evaluation that extends beyond the use of accuracy to evaluate reasoning in language models metric. \n\n**Comment of weaknesses:**\n\n*1. Emergence vs pattern recognition:* As illustrated in Figure 1, our method makes an explicit differentiation between answers given from probable pre-collected knowledge (factuals) and those from situations where the model likely hasn't been trained (counterfactuals). While it is likely that that an LLM was trained on factual examples (like the ones in the Div6 problem) it is very unlikely that counterfactuals containing imaginary scenarios were used in the training data set. To reinforce this point, we have included an extra experiment in the extra page allowed in this rebuttal. In the CandyParty problem, we use synthetically generated data to fine tune Phi-3-mini-128k-instruct\" using counterfactual data. We observe that when counterfactual examples are used to fine tune the model (for the node \"L=E\"), the approximations to the true PN and PS drastically improve. In addition, we observe that the approximation of the PN and PS for other nodes in the graph also improves (\"R>E,R>L\") which shows generalization abilities in counterfactual scenarios. We will include this result in the experimental section of the paper if this work is accepted.\n\n*2. Boolean nodes:*  as we have acknowledged in the discussion of our work, the use of binary variables may represent a limitation. However, we believe that solutions are possible.  For example, it is possible to add dummy nodes in the graph by defining new binary variables using a threshold, and therefore condition the computation of PN and PS to such threshold. This will lead to two PN and PS probability curves for different values of the threshold that can be used to evaluate reasoning with continuous nodes.\n\n*3. PN, PS approximation and emergence:*  we appreciate the comment, and we have clarified this point in an update version of the the paper. In our results for the GPT family of models, we observe that the more sophisticate the model is (GPT4 > GPT35 > GPT2) the lower are the errors in approximating factual, counterfactuals, PN and PS.  This indeed correlates with the common knowledge of other emergent properties in this family. The metrics that we propose in our work are therefore in alignment with the patterns observed in the literature. However, they highlight that if we define reasoning as the ability of a language to replicate necessity and sufficiency, there is still room from improvement even for basic arithmetic examples.\n\n*4. LMs as abstract machines, deterministic behaviour:* we would like to clarify that although we consider LLMs as abstract execution machines, we don’t imply that they are deterministic. Indeed, we believe that this is an important and relevant aspect of our work. Because LLMs are not deterministic (the same prompt may provide multiple answers) validations tools need to have a statistical nature, and this is indeed the approach that we take in our work. In the experimental section, we take this aspect into account by collecting multiple answers from the models and propagating the stochasticity of the answers to the computation of PN and PS. We have clarified this point in the main body of the paper.\n\n*Statistical significance:*  Although individual results for statistical tests can be included, the gamma-overlap in the experimental section captures the concentration of the probability distribution within a radius γ around the true PN, PS. We followed this approach, rather than providing a single statistical test for a given significance to better illustrate the behaviour of the metrics and their trade-offs. In the case of the FIR and CIR, we have included confidence intervals in all the results, which can also be used to test the statistical validity of the results.  See Figure 7 for details.\n\n**Answers to questions:**\n\n*1. Temperature and replicates:* In the all the experiments, we kept the default temperature (temp = 1) in all the models. For each factual and counterfactual questions we collected 10 answers that we later bootstrapped (500 times) to build the full distribution over PN and PS as described in Figure 2.  The densities of PN and PS in Figure 6 are the result of the propagation of the variation of those answers.\nPrompt optimization: We queried all the models using a zero-shot prompting approach where the factual and counterfactual questions are written in a way that is consistent with each other. We did not perform any optimization of the prompt. However, to guarantee fairness in the experiments we used the same approach for all models and experiments. Therefore, there is no bias towards GPT4 in the experiments. Instead, we observed that this model is better at providing factual and counterfactual answers that its predecessors.\n\n*2. Interpreting decision making:* We thank the reviewer for this comment. We believe that focusing on PN and PS helps to understand to what extent an LLMs is answering a question by composing basic elements of the solution or by simply memorizing answers. Guaranteeing that predictions are achieved by means of a correct reasoning process enhances robustness in the answers because they need to be grounded in a correct real-world reasoning model. Focusing only on predictions may lead to drawing the wrong conclusion that the model ‘understands’ the world when it is merely replicating patterns in the training data, which also leads to hallucinations.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TVKiiLQ6Rw", "reviewer": "Reviewer_65be", "summary": "The paper introduces a systematic method for assessing the reasoning capabilities of large language models (LLMs) by focusing on the concepts of necessity and sufficiency in logical reasoning. It leverages a probabilistic interpretation of these concepts and uses a reasoning graph based on boolean conditions to test the models.", "questions": "1. Is subsection 3.2 of the paper complete?\n\n2. The paper treats large language models (LLMs) as black boxes to measure the probabilities of causation. Does this approach have deeper significance and value? Additionally, is this evaluation method applicable to recurrent neural networks (RNNs) or large vision models (LVMs)?", "limitations": "The limitations are mentioned by the author in Section 5.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper introduces a novel method for evaluating the reasoning capabilities of LLMs, which  which is a critical issue in the field of LLMs.\n\n2.The paper includes numerous diagrams, effectively illustrating the background and contributions of the study.", "weaknesses": "1. The workload and the introduction of related work in the main body of the article appears somewhat insufficient. It is recommended to incorporate the more significant content from the appendix into the main text.\n\n2. Due to the lack of more rigorous and in-depth theoretical analysis, the results may not have sufficient theoretical persuasiveness. Thus, the paper would be more valuable if it provided some conclusive summaries or proposed improvement methods for the assessment results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the appreciation that the work proposes a novel method for a critical issue and the diagrams help to communicate our ideas. We appreciate the feedback, that we have now incorporated in the manuscript.\n\n**Comment of weaknesses:**\n\n*1. Incorporating more content in the main text:* Although we would have like to include more content in the main body of the paper the page limitation of the submission format forced us to leave some content in the appendix. If the paper is accepted, we will take this consideration into account since an extra page is permitted. \n\n*2. On conclusive summaries or proposed improvement:* We appreciate this comment, and we have incorporated this summary in the introduction of an updated version of our work that we will make available upon acceptance.\n\n**Answers to questions:**\n\n*1. Completeness of Section 3.2*: The section 3.2 is indeed competed, with some more relevant material in the appendix. However, to make the transition smoother to the next section we have added a last concluding sentence that explains the motivation of Lemma 1 in an updated version of the paper.\n\n*2. Applicability of this approach to other models:* Although in our work we focus on language, the same theory and tools can be used in other domains, and this will be the focus of our future research. The Hex framework, our core mathematical structure for conceptualizing these theories, requires defining a query, state, and an abstract computational engine that forecasts state alterations in response to the query. For example, within this paradigm, one might address challenges in computer vision: the 'state' comprises various objects in a visual setting, while a 'query' might represent a hypothetical modification to that environment. The principles of necessity and sufficiency are pertinent here as well. If an object in an image is edited or excluded, our framework could assess the resulting impact on a particular characteristic of the scene. This technique is equally applicable other disciplines where discerning the essential and causal relationships among components is critical for sound analysis and decision processes. It can also be applied to other models (like RNNs and LVMs) in which both factual and counterfactuals queries can be encoded.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VqSwCKeHol", "reviewer": "Reviewer_Hhfy", "summary": "To assess the reasoning abilities of large language models (LLMs) in complex tasks e.g., causation scenarios, this paper introduces a novel framework that utilizes probabilistic measures of necessity (PN) and sufficiency (PS). Through a series of mathematical examples, the study computes approximations of PN and PS using reasoning graphs and LLM that generate factual and counterfactual datasets. The generated results from the LLMs are then compared with true simulated PN and PS values from reasoning graphs. The experimental findings indicate an emerging trend towards improved reasoning capabilities within the GPT family of models, including GPT-2, GPT-3.5-turbo, and GPT-4.", "questions": "It would be clearer if the author(s) could explain:\n- What is the relationship between the experiment evaluation metrics, Factual Inconsistency Rate (FIR) and Counterfactual Inconsistency Rate (CIR), and the results of PN and PS?\n- Could you explain more about how the datasets (factual and counterfactual) were acquired from reasoning graphs?\n- What does the random noise introduced in the true counterfactuals mean, and how did you measure the noise level?", "limitations": "The limitations and broader impact of the study are clearly outlined in the paper.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The paper provides a clear example case at the beginning, effectively illustrating the differences in reasoning abilities among various GPT models.\n+ The examination of the concepts of necessity and sufficiency is well-founded, as these are crucial elements in causation and logical reasoning tasks, making them suitable for measuring LLMs' reasoning abilities.\n+ The experimental design and results are easy to understand, facilitating comprehension of the study's findings.", "weaknesses": "- Does four mathematic experiments representative in causal reasoning tasks?\n- Before presenting Sections 2 and 3 on the probabilities of causation in an LLM, the paper should first explain the method for computing true PS and PN values using reasoning graphs to provide better context for readers.\n- In Section 4.2 line 261: “Each density is labeled with the model that was used used to generate xxx”->typos", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful comments and positive feedback. We appreciate your acknowledgment that our exposition is clear and that the necessity and sufficiency are crucial elements in reasoning. We’re also pleased that you found our experimental design and results easy to understand. Your feedback is valuable and will help enhance our work.\n\n**Comment of weaknesses:**\n\n*1. Maths and reasoning tasks*: we would like to remark here that the goal of our paper is not to evaluate causal reasoning in LLMs. Our aim is to evaluate reasoning in a general sense, and we find that some core ideas from the causality literature (the probabilities of causation) are a natural way to evaluate that. Indeed, mathematical statements like the one used in the paper are deterministic, but testing if a probabilistic system (like an LLM) can reproduce them accurately is an inference problem. By no means are we implying that the arithmetic examples in the paper represent any sort of benchmark for causal reasoning for LLMs. Instead, the reason why we used arithmetic examples is because their logic is unambiguous and easily testable, so they provide a useful basis for evaluating reasoning in LLMs and developing the concepts presented in our work.\n\n*2. Order in exposition:* we appreciate the suggestion to change the order in which the computation of PN and PS is presented, which we have considered in an updated version of our work.\n\n*3. Typo:* Thank you for catching up the typo. We have now been corrected it in the paper.\n\n**Answers to questions**: \n\n*1.Connection between CIR, FIR, PN and PS*: We appreciate this comment. We have clarified this important connection in a new version of the paper, that will be made available upon acceptance.  To analyse it in more detail, let’s start with the eight frequencies that we need to compute such metrics. First, the 4 factual frequencies: (y, x), (y’, x’), (y’, x) and (y, x’). Second the 4 counterfactual ones (y, do(x)), (y’, do(x’)), (y’, do(x)) and (y, do(x’)). The FIR and CIR are the average factual and counterfactual rates. The computation of an exact FIR requires all factuals, while. an exact CIR requires all counterfactuals. In the computation of PN and PS however, each quantity requires some factuals and counterfactuals to be correct that we now clarify. To compute PN we need to compute P(y), P(y, x) and P(y|do(x’)). This means that only the frequencies of (y, x), (y, x’) and (y, do(x’)), (y’, do(x’)) are needed. A model with perfect error rate in these quantities will approximate the PN perfectly irrespectively of the accuracy of the rest of factual and counterfactuals. On the other hand, to estimate the PS we need to compute P(y), P(y, x’) and P(y|do(x)). This requires to correctly compute the set of frequencies (y, x), (y, x’) (y, x’) and (y, do(x)), (y’, do(x)), which are different to the case of the PN. This is the reason why in some of the experiments we see small models like GPT2 doing a good job approximating the PN (even better that other more powerful modes) but a terrible job approximation of the PS. Even with low FIR and CIR values, the results of what factual and counterfactuals are correct (all the positives or all the negatives) creates this imbalance. In summary, each one of the four metrics CIR, FIR, PS and PS, requires of a different set of factual and counterfactuals to be estimated correctly. However, the pairs of metrics CIR-FIR and PN-PS requires correctness in all factual and counterfactuals.\n\n*2. Details about data generation:* True factual data in all the examples are generated by following the structural equation models included in Appendix C. The set of numbers to use in each example is detailed in the experimental section. For example, in the Div6 example (appendix C.1), we start with all the numbers between 1 and 400. We compute their modulo 3 and 2 (variables C2 and C3) and we multiply these vectors to compute C6. Counterfactual data are generated by the same generative process but replacing the value of the variable we are intervening on by the intervention value. \n\n*3. Random noise in true counterfactuals?* In our work we do not add any noise to the counterfactual explicitly. The noise is the result of extracting multiple replicates from each query to the LLM.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zZ7VgTAfWT", "reviewer": "Reviewer_Hvct", "summary": "This paper investigates the reasoning capabilities of large language models (LLMs) by examining the concepts of necessity and sufficiency, which are key elements of logical reasoning. To assess the LLMs' reasoning abilities, the authors introduce a framework that computes the probability of necessity (PN) and the probability of sufficiency (PS) by comparing the actual values derived from factual and counterfactual datasets with those simulated by the LLMs. The paper presents a reasoning test for assessing LLMs' reasoning abilities using a divisibility problem as an example. The authors create factual and counterfactual datasets based on a reasoning graph and compare the actual PN and PS values with those generated by the LLMs. The results show that the closer the estimated PN/PS values are to the actual PN/PS values, the better the LLM is at reasoning.", "questions": "- How do the authors distinguish the results from the influence of the LLMs' prior knowledge? The LLMs' performance in the reasoning test may be affected by the patterns and relationships they have learned from their training data, especially commonsense reasoning questions related to Math. Are there any steps taken to control or account for this influence when evaluating the LLMs' reasoning abilities? In the interpretation of results, how do you account for the potential influence of the prior knowledge and the LLM's understanding of reasoning capabilities?\n- Have the authors tried any prompt tuning or do they have some primary resuls? Although the authors have made this point clear in the limitations, I think the prompt actually greatly influence the output of the LLMs and may have a influence on the conclusions.", "limitations": "N/A", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The main strength of the paper is the introduction of a systematic method for evaluating the reasoning capabilities of large language models (LLMs) by examining the concepts of necessity and sufficiency. This novel framework can be beneficial for researchers and developers working on improving LLMs. The method tries to evaluate from two different angles:  the accuracy with which an LLM solves a problem, and its capacity to understand and process the fundamental elements that lead to that solution. I think the authors tackles an important problem in LLM reasoning, which is an important and ongoing debate about the extent to which LLMs are capable of actual reasoning.", "weaknesses": "- The paper uses a single example, the divisibility problem, to demonstrate the reasoning test. While this example helps illustrate the concepts, it may not be representative of the wide range of problems LLMs are expected to solve. Including additional examples from various domains would strengthen the paper's claims and generalizability.\n- The paper does experiments on specific LLMs (gpt series) but doesn't provide results on other models or comparisons between multiple models. Including a variety of LLMs with different architectures and training datasets would provide a more comprehensive evaluation of the proposed framework. And, as GPT series are not open-sourced, it's unclear how OpenAI's hidden prompts or value-alignment finetuning will influence the results. \n- The paper focuses on the probabilistic interpretations of necessity and sufficiency in a creatain math problem but does not explore a wider range forms of reasoning (I'm not seeing this point as a signifigant weaknesses though, but I think the word \"reasoning\" actually covers more complex scenarios).   \n\nAlso see the questions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for highlighting the strengths of our paper. We appreciate your recognition of our systematic method for evaluating LLM reasoning through the probabilities of necessity and sufficiency. Your positive feedback on our dual-angle approach comparing the predictive vs the reasoning abilities of LLMs and its relevance to the ongoing debate about LLM reasoning is encouraging.\n\n**Comment of weaknesses:**\n\n*1. On single example:* While we agree that assessing reasoning skills through additional tasks would be beneficial, such exploration exceeds the intended scope of this work. The primary objective of this paper is to describe and demonstrate the significance of the probabilities of causation as essential metrics for evaluating reasoning in large language models. The development of comprehensive benchmarks for reasoning is indeed the focus of our current research.\n\n*2. Regarding experiments with open-source models:* Our paper concentrates on the GPT series to make our exposition clearer but the same principles are relevant to other open-source models as well. We have added further results on the Div6 issue in the extra page, utilizing two more model families—Phi (1, 2) and Llama (2-7b, 2-13b)—which show consistent behaviour with the findings discussed in the paper. These figures have also been included in the appendix of the paper.\n\n*3. Other reasoning tasks:* We agree that reasoning, in general, goes beyond the math examples that we have included in the paper. We believe that the same theory and tools can be used in other domains, and this will be the focus of our future research. The Hex framework, that serves as a mathematical framework to formalize our ideas, requires the definition of a query, state, and an abstract execution machine that is used to predict how the state is modified given the query. In this framework, for instance, one could think about problems in vision, where the elements of the state are the objects in an image, and the query corresponds to a counterfactual query that describes an intervention in the environment. The concepts of necessity and sufficiency still apply in this scenario. In an image where an object is removed or altered, the framework can help determine the impact of this change on a property of the overall scene. This approach can be extended to other fields such as robotics, and or social sciences, where understanding the necessity and sufficiency between different elements is crucial for accurate reasoning and decision-making. We will further clarify this point in the paper to make sure the readers are aware of the generality of our approach.\n\n**Answers to questions:**\n\n*1. Influences of LLM previous knowledge:* As illustrated in Figure 1, a key aspect of our approach is to make a clear distinction between answers that the model is giving based on likely previously collected knowledge (factuals) and based on scenarios in which it is very unlikely that the model has been trained on (counterfactuals). Using the Div6 example, one can expect that the concepts of divisibility and the factor method are present in the training data set of a language model. Indeed, in Figure 1, the three models achieve a low error rate on this question, with GPT-35-turbo achieving errors close to zero. However, it is less likely to expect that imaginary scenarios where the standard rules of arithmetic have been violated to be present in the training data. This is consistent with Figure 1 and all the results in the experimental section - the counterfactual errors are significantly larger than factual errors for all models and problems. What makes our approach unique is that we test the models with scenarios where generalization by reasoning, rather than replicating statistical patterns in the training data, is required to provide correct answers. We have clarified this point in the main body of the paper.\n \n*2. On prompt tuning.* We agree that prompt tuning may affect the results. However, as we detail in the limitations of the work, it is not our aim to optimize reasoning in LLMs, but rather to propose metrics that can characterize whether reasoning is happening. To guarantee fairness across all comparisons in the paper, we have used the same patterns in the way we write the prompts across all experiments (in a zero-shot manner). Of course, other techniques like chain-of-thought prompting could be used, but our focus remains on establishing a consistent baseline. Future work could explore the impact of different prompting strategies on reasoning performance, potentially leading to more refined and effective methods for evaluating and enhancing reasoning capabilities in LLMs. We have expanded the discussion of this topic in the discussion in a new version of the paper that we will make available upon acceptance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the four reviews for the positive feedback and the comments that have helped to improve our work. It is encouraging to see that the reviewers find our approach to offer "... a nuanced evaluation that extends beyond the accuracy metric." and that they agree with us that ''...The examination of the concepts of necessity and sufficiency is well-founded, as these are crucial elements in causation and logical reasoning tasks...". We also believe that this work is the right step toward evaluating reasoning in LLMs. 

We have added a rebuttal for each reviewer that cover all the points in the discussion. We hope our answers will clarify all the reviewers questions and concerns. To complement some of these answers we have added an extra page of material with another two experiments. The first one shows that our approach can be easily extended to other families of models beyond GPTs. The second experiment demonstrates that counterfactuals are an effective way of testing the models in scenarios not seen an training time. We show that by fine-tuning the models with counterfactual data (most likely unseen at training time but presented to the models in the fine-tunning process) the models performance increases in their approximations to PN and PS.  

We are confident that these additions will further substantiate the robustness and utility of our approach, and we look forward to the continued discussion and feedback from the reviewers. Thank you for your valuable feedback and consideration.

Meta-review: Reasoning capabilities of large language models is a topic of great interest to the larger NeurIPS community. This paper provides a novel framework that draws from concepts from causal reasoning to identify whether LLMs are actually able to perform reasoning. The introduction of such a systematic framework to this space is a meaningful contribution to this space, which is otherwise filled with papers that look at shallower metrics like accuracy. However, the primary concern multiple reviewers have raised is the authors’ choice to focus the evaluation on a specific class of reasoning problems. While from the point of view of a conference paper, the choice to focus on a specific class is reasonable, it would be helpful to update both the title and abstract to accurately reflect this limitation. This is also important because the current formulation also limits its focus to problems with boolean variables. Additionally, reviewers have pointed out a lack of a more detailed discussion about the implications of the current results. If accepted, the authors will have access to an additional page. It is recommended that at least part of that space is used to incorporate discussion along these lines.

Given the relevance of the paper and its contribution, I would recommend the paper be accepted.
Decision: Accept (poster)