Does Reasoning Emerge? Examining the Probabilities of Causation in Large Language Models

Authors: javier  gonzález
aditya v nori
g  team
r  anil
s  borgeaud
j.-b  alayrac
j  yu
r  soricut
j  schalkwyk
a m dai
a  hauth
k  millican
d  silver
m  johnson
i  antonoglou
j  schrittwieser
a  glaese
j  chen
e  pitler
t  lillicrap
a  lazaridou
o  firat
j  molloy
m  isard
p r barham
t  hennigan
b  lee
f  viola
m  reynolds
y  xu
r  doherty
e  collins
c  meyer
e  rutherford
e  mor- eira
k  ayoub
m  goel
j  krawczyk
c  du
e  chi
h.-t  cheng
e  ni
p  shah
p  kane
b  chan
m  faruqui
a  severyn
h  lin
y  li
y  cheng
a  ittycheriah
m  mahdieh
m  chen
p  sun
d  tran
s  bagri
b  lakshminarayanan
j  liu
a  orban
f  güra
h  zhou
x  song
a  boffy
h  ganapathy
s  zheng
h  choe
ágoston  weisz
t  zhu
y  lu
s  gopal
j  kahn
m  kula
j  pitman
r  shah
e  taropa
m a merey
m  baeuml
z  chen
l e shafey
y  zhang
o  sercinoglu
g  tucker
e  piqueras
m  krikun
i  barr
n  savinov
i  danihelka
b  roelofs
a  white
a  andreassen
t  von glehn
l  yagati
m  kazemi
l  gonzalez
m  khalman
j  sygnowski
a  frechette
c  smith
l  culp
l  proleev
y  luan
x  chen
j  lottes
n  schucher
f  lebron
a  rrustemi
n  clay
p  crone
t  kocisky
j  zhao
b  perz
d  yu
h  howard
a  bloniarz
j w rae
h  lu
l  sifre
m  maggioni
f  alcober
d  garrette
m  barnes
s  thakoor
j  austin
g  barth-maron
w  wong
r  joshi
r  chaabouni
d  fatiha
a  ahuja
g s tomar
e  senter
m  chadwick
i  kornakov
n  attaluri
i  iturrate
r  liu
s  cogan
c  jia
c  gu
q  zhang
j  grimstad
a j hartman
x  garcia
t s pillai
j  devlin
m  laskin
d  de
las  casas
d  valter
c  tao
l  blanco
a p badia
d  reitter
j  brennan
c  rivera
s  brin
s  iqbal
g  surita
j  labanowski
a  rao
s  winkler
e  parisotto
y  gu
k  olszewska
r  addanki
a  miech
a  louis
d  teplyashin
g  brown
e  catt
j  balaguer
j  xiang
p  wang
z  ashwood
a  briukhov
a  webson
s  ganapa- thy
s  sanghavi
a  kannan
m.-w b chang
a  stjerngren
j  djolonga
y  sun
a  bapna
m  aitchison
p  pejman
h  michalewski
t  yu
c  wang
j  love
j  ahn
d  bloxwich
k  han
p  humphreys
t  sellam
j  bradbury
v  godbole
s  samangooei
b  damoc
a  kaskasoli
s m r arnold
v  vasudevan
s  agrawal
j  riesa
d  lepikhin
r  tanburn
s  srinivasan
h  lim
s  hodkinson
p  shyam
j  ferret
s  hand
a  garg
t l paine
j  li
m  gi- ang
a  neitz
z  abbas
s  york
m  reid
e  cole
a  chowdhery
d  das
d  rogozińska
v  nikolaev
p  sprechmann
z  nado
l  zilka
f  prost
l  he
m  monteiro
g  mishra
c  welty
j  newlan
d  jia
m  allamanis
c h hu
r  de liedekerke
j  gilmer
c  saroufim
s  rijhwani
s  hou
d  shrivastava
a  baddepudi
a  goldin
a  ozturel
a  cassirer
d  sohn
d  sachan
r k amplayo
c  swanson
d  petrova
s  narayan
a  guez
s  brahma
j  landon
m  patel
r  zhao
k  villela
l  wang
w  jia
m  rahtz
m  giménez
l  ye- ung
j  keeling
p  georgiev
d  mincu
b  wu
s  haykal
r  saputro
k  vodrahalli
j  qin
z  cankara
a  sharma
n  fernando
w  hawkins
b  neyshabur
s  kim
a  hutter
p  agrawal
a  castro-ros
g  van den driessche
t  wang
f  yang
s  yiin chang
p  komarek
r  mcilroy
m  lučić
g  zhang
w  farhan
m  sharman
p  natsev
p  michel
y  bansal
s  qiao
k  cao
s  shakeri
c  butterfield
j  chung
p k rubenstein
a  mensch
k  soparkar
k  lenc
t  chung
a  pope
l  maggiore
j  kay
p  jhakra
s  wang
j  maynez
m  phuong
t  tobin
a  tacchetti
m  trebacz
k  robinson
y  katariya
s  riedel
p  bailey
k  xiao
n  ghelani
l  aroyo
a  slone
n  houlsby
x  xiong
z  yang
e  gribovskaya
j  adler
m  wirth
l  lee
m  li
t  kagohara
j  pavagadhi
s  bridgers
a  bortsova
s  ghemawat
z  ahmed
t  liu
r  powell
v  bolina
m  iinuma
p  zablotskaia
j  besley
d.-w  chung
t  dozat
r  comanescu
x  si
j  greer
g  su
m  polacek
r l kaufman
s  tokumine
h  hu
e  buchatskaya
y  miao
m  elhawaty
a  siddhant
n  tomasev
j  xing
c  greer
h  miller
s  ashraf
a  roy
z  zhang
a  ma
a  filos
m  besta
r  blevins
t  klimenko
c.-k  yeh
s  changpinyo
j  mu
o  chang
m  pajarskas
c  muir
v  cohen
c l lan
k  haridasan
a  marathe
s  hansen
s  douglas
r  samuel
m  wang
s  austin
j  jiang
j  chiu
j a lorenzo
l l sjösund
s  cevey
z  gleicher
t  avrahami
a  boral
h  srinivasan
v  selo
r  may
k  aisopos
l  hussenot
l b soares
k  baumli
a  recasens
b  caine
a  pritzel
f  pavetic
f  pardo
a  gergely
j  frye
v  ramasesh
d  horgan
k  badola
n  kassner
s  roy
e  dyer
v c campos
a  tomala
y  tang
d e badawy
e  white
b  mustafa
o  lang
a  jindal
s  vikram
z  gong
s  caelles
r  hemsley
g  thornton
f  feng
w  stokowiec
c  zheng
p  thacker
çaglar  ünlü
m  saleh
j  svensson
m  bileschi
p  patil
a  anand
r  ring
k  tsihlas
a  vezer
m  selvi
t  shevlane
m  rodriguez
t  kwiatkowski
s  daruki
k  rong
a  dafoe
n  fitzgerald
k  gu-lemberg
m  khan
l a hendricks
m  pellat
v  feinberg
j  cobon-kerr
t  sainath
m  rauh
s h hashemi
r  ives
y  hasson
e  noland
y  cao
n  byrd
l  hou
q  wang
t  sottiaux
m  pa- ganini
j.-b  lespiau
a  moufarek
s  hassan
k  shivakumar
j  van amersfoort
a  mandhane
p  joshi
a  goyal
m  tung
a  brock
h  sheahan
v  misra
c  li
n  rakićević
m  de- hghani
f  liu
s  mittal
j  oh
s  noury
e  sezener
f  huot
m  lamm
n d cao
c.-w l chen
s  mudgal
r  stella
k  brooks
g  vasudevan
c  liu
m  chain
n  melinkeri
a  cohen
v  wang
k  seymore
s  zubkov
r  goel
s  yue
s  krishnakumaran
b  albert
n  hur- ley
m  sano
a  mohananey
j  joughin
e  filonov
t  kępa
y  eldawy
j  lim
r  rishi
s  badiezadegan
t  bos
j  chang
s  jain
s g s padmanabhan
s  puttagunta
k  krishna
l  baker
n  kalb
v  bedapudi
a  kurzrok
s  lei
a  yu
o  litvin
x  zhou
z  wu
s  so- bell
a  siciliano
a  papir
r  neale
j  bragagnolo
t  toor
t  chen
v  anklin
f  wang
r  feng
m  gholami
k  ling
l  liu
j  walter
h  moghaddam
a  kishore
j  adamek
t  mercado
j  mallinson
s  wandekar
s  cagle
e  ofek
g  garrido
c  lombriser
m  mukha
b  sun
h r mohammad
j  matak
y  qian
v  peswani
p  janus
q  yuan
l  schelin
o  david
y  he
o  duzhyi
a  älgmyr
t  lottaz
q  li
v  yadav
l  xu
a  chinien
r  shivanna
a  chuklin
c  spadine
t  wolfe
k  mohamed
s  das
z  dai
k  he
d  von dincklage
s  upadhyay
a  maurya
l  chi
s  krause
k  salama
p g rabinovitch
a  selvan
m  dektiarev
g  ghiasi
e  guven
h  gupta
b  liu
d  sharma
i h shtacher
s  paul
o  akerlund
f.-x  aubet
t  huang
c  zhu
e  zhu
e  teixeira
m  fritze
f  bertolini
l.-e  marinescu
m  bölle
d  paulus
k  gupta
t  latkar
j  sanders
r  wilson
x  wu
y.-x  tan
l n thiet
t  doshi
s  lall
s  mishra
w  chen
t  luong
s  benjamin
j  lee
e  andrejczuk
d  rabiej
v  ranjan
k  styrc
p  yin
j  simon
m r harriott
m  bansal
a  robsky
g  bacon
d  greene
d  mirylenka
c  zhou
o  sarvana
s  andermatt
p  siegler
b  horn
a  israel
f  pongetti
m  selvatici
p  silva
k  wang
j  tolins
k  guu
r  yogev
x  cai
a  agostini
m  shah
h  nguyen
n o donnaile
s  pereira
l  friso
a  stambler
c  kuang
y  romanikhin
m  geller
z  yan
k  jang
c.-c  lee
w  fica
e  malmi
q  tan
d  banica
d  balle
r  pham
y  huang
d  avram
h  shi
j  singh
c  hidey
n  ahuja
p  saxena
d  dooley
s p potharaju
e  o'neill
a  gokulchandran
r  foley
k  zhao
m  dusenberry
y  liu
p  mehta
r  kotikala- pudi
c  safranek-shrader
a  goodman
j  kessinger
e  globen
p  kolhar
c  gorgolewski
a  ibrahim
y  song
a  eichenbaum
t  brovelli
s  potluri
p  lahoti
c  baetu
a  ghorbani
a  crawford
s  pal
m  sridhar
p  gurita
a  mujika
i  petrovski
p.-l  cedoz
s  chen
n d santo
s  goyal
j  punjabi
k  kappaganthu
c  kwak
p  lv
s  velury
h  choudhury
j  hall
r  figueira
m  thomas
m  lu
t  zhou
c  kumar
t  jurdi
s  chikkerur
y  ma
s  kwak
v  ähdel
s  rajayogam
t  choma
a  barua
c  ji
j h park
v  hellendoorn
a  bailey
t  bilal
m  khatir
c  sutton
w  rzad- kowski
f  macintosh
k  shagin
p  medina
c  liang
j  zhou
y  bi
a  dankovics
s  banga
s  lehmann
m  bredesen
z  lin
j e hoffmann
j  lai
r  chung
k  yang
n  bal- ani
a  bražinskas
a  sozanschi
m  hayes
h f alcalde
p  makarov
a  stella
l  snijders
m  mandl
a  kärrman
p  nowak
a  dyck
k  vaidyanathan
j  mallet
m  rudominer
e  johnston
a  udathu
j  christensen
v  verma
z  irving
a  santucci
g  elsayed
e  davoodi
m  georgiev
i  tenney
n  hua
g  cideron
e  leurent
m  alnahlawi
i  georgescu
n  wei
i  zheng
d  scandinaro
h  jiang
j  snoek
m  sundarara- jan
x  wang
z  ontiveros
i  karo
j  cole
v  rajashekhar
l  tumeh
e  ben-david
r  jain
j  uesato
r  datta
o  bunyan
s  wu
j  zhang
p  stanczyk
d  steiner
s  naskar
m  azzam
a  paszke
c.-c  chiu
j s elias
a  mohiuddin
f  muhammad
j  miao
a  lee
n  vieillard
j  stanway
d  garmon
a  karmarkar
z  dong
a  kumar
l  zhou
j  evens
w  isaac
g  irving
e  loper
m  fink
i  arkatkar
n  chen
i  shafran
i  petrychenko
j  jia
a  levskaya
z  zhu
p  grabowski
y  mao
a  magni
k  yao
j  snaider
n  casagrande
e  palmer
p  suganthan
a  castaño
i  giannoumis
w  kim
m  rybiński
a  sreevatsa
j  prendki
d  soergel
a  goedeckemeyer
w  gierke
m  jafari
m  gaba
j  wiesner
d g wright
y  wei
h  vashisht
y  kulizhskaya
j  hoover
m  le
l  li
c  iwuanyanwu
k  ramirez
a  khorlin
a  cui
t  lin
m  wu
r  aguilar
k  pallo
a  chakladar
g  perng
e a abellan
m  zhang
i  dasgupta
n  kushman
i  penchev
a  repina
t  van der weide
p  ponnapalli
c  kaplan
j  simsa
s  li
o  dousse
j  piper
n  ie
r  pasumarthi
n  lintz
a  vijayakumar
d  andor
p  valenzuela
m  lui
c  paduraru
d  peng
k  lee
s  zhang
s  greene
d d nguyen
p  kurylowicz
c  hardin
l  dixon
l  janzer
k  choo
z  feng
b  zhang
a  singhal
d  du
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 10:10:19

Abstract: Recent advances in AI have been significantly driven by the capabilities of large language models (LLMs) to solve complex problems in ways that resemble human thinking. However, there is an ongoing debate about the extent to which LLMs are capable of
actual reasoning. Central to this debate are two key probabilistic concepts that are essential for connecting causes
to their effects: the probability of necessity (PN) and the probability of sufficiency (PS). This paper introduces a framework that is both theoretical and practical, aimed at assessing how effectively LLMs are able to replicate real-world reasoning mechanisms using these probabilistic measures. By viewing LLMs as abstract machines that process information through a natural language interface, we examine the conditions under which it is possible to compute suitable approximations of PN and PS. Our research marks an important step towards gaining a deeper understanding of when LLMs are capable of reasoning, as illustrated by a series of math examples.

Main paper: 
 introduction: 
 Large language models (LLMs) have revolutionized the way we interact with technology, enabling more natural and intuitive communication between humans and computers in applications like writing assistants [8] , sentiment analysis in social media [29] , healthcare [10, 35] and many others. Despite the surge of interest and recent breakthroughs [5] , the ability of LLMs to reason about real-world problems as humans do continues to be a topic of intense research [1, 14] .Reasoning is a cognitive process that involves drawing conclusions, making judgments, or forming inferences based on facts or premises. This process has been explored from various perspectives. Symbolic reasoning [17] involves the manipulation of symbols that represent ideas or objects and it is often used in mathematics and logic to represent numerical values or logical propositions. Causal reasoning [26] focuses on discerning the relationship between a cause and its effect, aiming to understand how certain events can impact other. Other forms of reasoning include inductive reasoning [7] (making broad generalisations from specific observations), deductive reasoning [27] (applying general principles to specific cases), and abductive reasoning [2] (forming the best hypothesis based on incomplete information).In the realm of LLMs, reasoning is typically understood to be the ability of these models to demonstrate emergent capabilities that surpass mere statistical pattern recognition in the training set. It entails systematically breaking down problems into a logical sequence of smaller, manageable steps and then processing these steps internally to arrive at accurate conclusions that are grounded in reality. This concept is the foundation for techniques such as chain of thoughts prompting [34] , which aim to Figure 1 : Illustration of the actual vs. perceived reasoning abilities of GPT-2, GPT-35-turbo and GPT-4 for a simple arithmetic problem. We posed two distinct types of questions (direct and counterfactual) to the models, each repeated 10 times, for every {number} from 1 to 50. All three models showed an inflated sense of reasoning capability when answering the direct questions. The discrepancy is especially pronounced in GPT-35-turbo, which performed nearly flawlessly on direct questions, but experienced a surge in error rate, exceeding 25%, when handling counterfactual questions.teach LLMs how to reason by providing examples where problems are solved through a sequence of smaller steps.Assessing the reasoning abilities of LLMs involves distinguishing between two aspects: the accuracy with which an LLM solves a problem, and its capacity to understand and process the fundamental elements that lead to that solution. Judea Pearl, in his hierarchy of causality [25] , asserts: "Only machines that can correctly perform correlations, interventions and counterfactuals will have reasoning abilities comparable to human". As demonstrated in [16] , while LLMs are remarkable in using learnt patterns from their training data to generate correct answers (correlations), they falter when faced with hypothetical/imaginary scenarios that were not part of their training (counterfactuals). This is depicted in Figure 5 , which presents a straightforward arithmetic problem (this is the direct prompt in the figure). Both GPT-35-turbo and GPT-4 can accurately determine the divisibility of numbers by 6, suggesting at first glance that they can reason about divisibility. However, when the questions are framed in a counterfactual manner (this is the counterfactual prompt in the figure), only GPT-4 maintains a low error rate, indicating its superior ability to handle such reasoning tasks.In this paper, we introduce a systematic method to assess the reasoning capabilities of LLMs by examining the concepts necessity and sufficiency, which are key elements of logical reasoning and have been studied in multiple fields such as logic, probability, and causality [22, 19, 12] . In propositional logic, a sufficient condition is defined as X =⇒ Y , indicating that the presence of X ensures the occurrence of Y . On the other hand, a necessary condition is defined as Y =⇒ X, signifying that the occurrence of Y necessitates the prior occurrence of X. We focus on the probabilistic interpretations of necessity and sufficiency [24] . The probability of necessity (PN) between two boolean variables X and Y is defined as PN(x, y) := P(y ′x ′ |x, y). Here, y ′ x ′ represents the counterfactual value of Y = y ′ , had X been set to a different value x ′ . By conditioning on both X = x and Y = y, this measure captures probability of observing a different outcome in the absence of the event X = x. The probability of sufficiency (PS), on the other hand, is defined as PS(x, y) := P(y x |x ′ , y ′ ) and measures the probability that X = x results in Y = y, for cases where both originally had different values.We show that when a problem can be solved via a reasoning graph of boolean conditions, denoted by G, the PN and PS can be computed using a causal model underlying G. As described in [24] , the exact computation of PN and PS requires samples from the (causal) data generative model, counterfactual data (experiments) as well as other monotonicity assumptions. As a reasoning test, we statistically compare the true PN and PS measures (computed by sampling from the original and the intervened graph) with those simulated via factual and counterfactual datasets generated by an LLM. Figure 2 presents an informal illustration of the reasoning test advocated in this paper, focusing on the specific problem of determining whether a number N is divisible by 6. The test relies on the reasoning principle that: "A natural number N that is divisible by both 2 and 3 is also divisible by 6". This logic is represented in a reasoning graph G that links the conditions C 2 (divisibility by 2) and C 3 (divisibility by 3) to the conclusion C 6 (divisibility by 6). We test the reasoning abilities of an LLM using natural numbers N from 1 to 400. This is shown in Figure 2(A) .As indicated in Figure 2 (B), we create two sets of data based on G. The first is a factual dataset (D F ) which captures whether each number N satisfies conditions C 2 and C 3 . The second is a counterfactual dataset, (D CF ), which assumes condition C 3 is always true and then records whether each number X would satisfy C 6 under this assumption/intervention (realised by do(C 3 = True) in the figure). For the LLM being evaluated, we also produce two datasets. The first, D LLM F , documents the LLM's response for C 6 for each number N , when the prompt is based on the reasoning graph G. The second, D LLM CF , involves a hypothetical scenario where we assume C 3 is true and then record the LLM's prediction for C 6 given this "counterfactual prompt". This process is repeated multiple times (several answers from the LLMs are collected from each prompt). We assess the LLM's reasoning capability by comparing the estimated (distribution of) PN and PS from the D LLM F and D LLM CF datasets with the actual values derived from D F and D CF datasets. Figure 2(C ) displays these comparisons, plotting PN vs. PS. The closer the estimated PN/PS values to the actual PN/PS values, the better it is at reasoning. In this case, LLM 2 demonstrates better reasoning abilities than LLM 1.Reasoning in LLMs has been studied from multiple perspectives. [15] presents an overview paper that elucidates key reasoning concepts utilised by LLMs. [13] examines the similarity between reasoning with a language model and planning with a world model, proposing a novel reasoning framework that redefines the LLM as both a world model and a reasoning agent. Various studies [28, 4] have focused on assessing the reasoning and problem-solving abilities of LLMs, yet none have used the probabilities of causation as the primary objects of computation as done in our research. [32] carries out a series of experiments to show that LLMs can indeed derive benefits from reasoning errors, offering potentially cost-effective strategies by using mistakes to bolster reasoning capabilities. Recent research indicates that LLMs like GPT-3.5 and GPT-4 are effective at causal reasoning tasks, including pairwise causal discovery [18] . These models have achieved state-of-theart performance on multiple causal benchmarks, outperforming existing algorithms. Nevertheless, LLMs also exhibit unpredictable failure modes, and currently, they are not capable of discovering new knowledge or making high-stakes decisions with a high level of precision [20, 36, 21, 6, 3] .σ0 σ1 σ 0 = {X → 16, C 6 →⊥} σ 1 = {X → 16, C 6 → F alse} α [prompt] Q LLM γ [output] Q Figure 3:The HEX diagram depicts two approaches for solving the problem (Q, σ 0 ) outlined in Example 1. The dotted path corresponds to the actual process of solving the problem, while the solid path represents the one taken by the LLM.Contributions: This paper presents two main contributions. i) A novel, theoretical and practical, framework to evaluate the reasoning capabilities of language models using the probabilistic of causation, specifically the probability of necessity and the probability of sufficiency. Our approach is unique in the sense that it allows to differentiate generalization by reasoning from merely replicating statistical patterns in the training data. ii) Empirical tests on various reasoning problems as well as several insights about the reasoning abilities of language models in the GPT family. 
llms as abstract machines: 
 As described by the HEX framework [9] , an LLM functions as an abstract machine that uses natural language as an interface. In this section, we introduce the core elements of this framework, which will subsequently enable us to define an LLM's internal representation of PN and PS.We define a problem as a query-state pair (Q, σ). The state σ is a mapping defined by σ : V → C, which assigns values from a specified domain C to a set of variables V = {V 1 , . . . , V n }. The query Q : 2 V→C → 2 V→C is a mapping that transforms an input state σ to a well-defined output state. To solve a problem is to calculate σ 1 = Q(σ 0 ), where σ 0 and σ 1 represent the states before and after the query Q is applied. To clarify this, we consider the following example: Example 1. "Given that a natural number divisible by both 2 and 3 is also divisible by 6, determine whether the number 10 is divisible by 6."To solve Example 1, we apply the query Q to the stateσ 0 = {N → 10, C 6 →⊥}, where Q = λσ . (σ(N ) (mod 2) ≡ 0) ∧ (σ(N ) (mod 3) ≡ 0) 1 . This results in a final state σ 1 = {N → 10, C 6 → False}, thereby resolving the problem with σ 1 (C 6 ) = Q(σ 0 ) = False.We now turn to the question of how an LLM solves a problem defined by a query-state pair (Q, σ 0 ). This process involves three essential steps as illustrated by Figure 3: 1. First, an abstraction mapping translates the initial state σ 0 into a latent state σ0 via a prompt.2. Next, the LLM processes (via the query Q LLM ) this latent state σ0 . 3. Finally, the output mapping transforms the LLM output latent state σ1 back into a concrete state, producing the final output σ 1 .Formally, solving a problem (Q, σ 0 ) with an LLM can be described as a sequence of function applications resulting in the outputσ 1 = (γ • Q LLM • α)(σ 0 ).To illustrate this, the problem statement is given as a prompt input to GPT-4 [23] . The response from GPT-4 is "False", which matches the result obtained by applying the query Q directly to the input state σ 0 . When both the direct application of Q and the LLM computation yield the same answer, we say that the diagram, as shown in Figure 3 , is commutative-meaning that following either the dotted line or the solid lines lead to the same result. For a more in-depth explanation of this framework, please refer to [9] . 
probabilities of causation for an llm: 
 To assess the reasoning abilities of an LLM, we must link its generated responses to the actual reasoning processes that produced those responses. For a problem (Q, σ), we postulate the existence of a causal model M V defined over variables in V, and by a set of structural equations and endogenous variables. For a detailed introduction to causal models, refer to Appendix A. Additionally, the seminal work by Pearl [25] on causality provides foundational insights on this area. Here, we are particularly interested in causal models that represent the logical steps involved in problem-solving. However, it is important to note that the concept of a causal model is broadly applicable beyond this specific application.We assume that V = {X, Y, Z}, which includes X and Y as boolean variables, and Z as a variable (which may be multivariate) that encompasses all necessary factors that are required to understand how an intervention on X would affect Y . In the context of causality, this means that the distribution P(Y |do(X = x ′ )), where do denotes the intervention operator defined in [25] , is identifiable. This means we can predict the outcome for Y , and that the counterfactual Y X=x ′ , that can be read as "the value of Y had X been x ′ ", is well-defined. For further details, please refer to Appendix A. For ease of exposition in the following text, we will simplify our notation by omitting the explicit reference to Z. Therefore, we will denote Y X=x (Z = z) more succinctly as Y X=x .As studied in [31] , if Y is monotonic with respect to X, then PN and PS can be computed as follows:PN(x, y) = P(y) -P(y|do(x ′ )) P(x, y) and PS(x, y) = P(y|do(x)) -P(y) P(x ′ , y ′ ).To estimate PN and PS, we need two different types of datasets. The first is a factual datasetD F = {x i , y i , z i } n i=1, which is used to infer P(y), P(x, y) and P(x ′ , y ′ ). The second datasetD CF = {x i , Y X=xi , z i } ni=1 is a counterfactual one, and is necessary to determine P(y|do(x)) and P(y|do(x ′ )). There are various methods to acquire the datasets D F (factual) and D CF (counterfactual). For a physical process, the usual method would be through observation and experimentation. However, in this paper, we presume access to a comprehensive reasoning graph that is equivalent to a causal model M V . This allows us to simulate and generate the D F and D CF datasets. Both M V and the sub-model M V,do(X=x) define two distinct joint probability distributions P M V and P M V,do(X=x) over X, Y and Z. We obtain the datasets D F and D CF by sampling from these respective probability distributions. These datasets are then used to calculate PS and PN using Eq. (1).Can an LLM reason in a manner that is consistent with P M V ? In Example 1, we obtained consistent answers (that is, the corresponding HEX diagram commutes) for a direct divisibility question. However, to evaluate the reasoning abilities of the LLM, it is crucial that this consistency is also observed when the queries are framed in a counterfactual manner. This is necessary to ensure that the LLM can apply its reasoning to imaginary situations that are unlikely to be present in the training set, demonstrating its ability to generalise based on a correct internal representation of the reasoning logic of the problem. Practically, this means employing the LLM as a "counterfactual data simulator", where the data it generates under these hypothetical conditions are used to estimate PN and PS. Definition 1 (Counterfactual query). Consider a problem (Q, σ 0 ), with σ 0 = {X → x, Y → y, Z → z} being an initial state. Let M V be a causal model over the variables V. We can then define a counterfactual query Q ′ as follows:Q ′ (σ 0 ) = {X → x ′ , Y → Y X=x ′ , Z → z}.In other words, a counterfactual query updates two variables of the state: it sets X to its new value x ′ , and Y to the counterfactual Y X=x ′ . An LLM-based counterfactual Y LLM X=x ′ is computed as follows:Y LLM X=x ′ = (γ • Q ′LLM • α)(σ 0 )(Y )where σ 0 = {X → x, Y → y, Z → z}, and Q ′LLM is a counterfactual query. This entire process simulates counterfactual reasoning within the LLM, and is facilitated through textual prompts that are structured to elicit the desired counterfactual outcome. Definition 2 (Counterfactual prompt). A counterfactual prompt is a textual encoding of a counterfactual query for some initial state σ 0 . Definition 3 (β-counterfactual consistency). Consider a structural causal model M V with variables V = {X, Y, Z}. Let A X=x (Z) be a function that generates counterfactuals for Y . We say that A is β-counterfactual consistent with M V if the following condition is satisfied:E P(X,Y,Z) [A X=x (Z = z) ̸ = Y X=x (Z = z)] ≤ β, where β ≤ 0.β-counterfactual consistency defines the limit error rate for counterfactuals produced by A X=x (Z = z). This error rate should ideally be zero for an LLM that exhibits flawless reasoning abilities. The following lemma specifies the conditions necessary for this property to hold (the proof can be found in Appendix D).Lemma 1. Let M V , with variables V = {X, Y, Z}, be a structural causal model for a problem (Q, σ 0 ), and let M be an LLM that generates counterfactuals for Y . Then M is β-counterfactual consistent with M V if and only if its associated HEX diagram for the problem (Q ′ , σ 0 ), where Q ′ is the counterfactual version of Q, is commutative for all admissible values of X, Y and Z.Lemma 1, provides a theoretical criterion to test the consistency of an LLM with some reasoning graph described by M V . As we show next in the experimental section, the commutability of the diagram is tested in expectation by sampling over the variables in V.The findings we report are based on an LLM's reasoning abilities as determined by two specific types (factual/counterfacutal) of prompts that we used. Of course, other techniques like chain-of-thought prompting could be used, but our focus remains on establishing a consistent baseline. Future work could explore the impact of different prompting strategies on reasoning performance, potentially leading to more refined and effective methods for evaluating and enhancing reasoning capabilities in LLMs. Our experiments did not aim to fine-tune these prompts or to 'optimise reasoning'-a separate area of ongoing research. Instead, our goal was to offer valuable insights that can aid the community in developing new benchmarks and employing LLMs responsibly.Broader impact: Evaluating the reasoning capabilities of LLMs is essential as it significantly influences their effectiveness in various domains. In education and research, it is important for the model to be able to provide accurate explanations and to formulate meaningful hypotheses. In the commercial sector, the effectiveness of automated processes/systems relies heavily on how well the model can reason. When it comes to accessibility, the model must be able to understand and meet diverse user needs, which hinges on its reasoning ability. Moreover, identifying and mitigating biases in AI systems-a key aspect of ethical and equitable AI-requires a detailed examination of the models' reasoning processes. Therefore, while LLMs hold immense promise, ensuring their responsible and beneficial use is predicated on a thorough appraisal of their reasoning abilities. We believe that our research is an important step in this direction.Definition 9 (Probability of necessity and sufficiency, [24] ). Let X and and Y be two binary variables in a causal model M = ⟨X , F, ε⟩. The probability of necessity and sufficiency is defined as: P N S := P(y x , y ′ x ′ ) = P(x, y)P N + P (x ′ , y ′ ) P SThe PNS computes the probability that X = x is the only way of obtaining Y = y, in other words, the probability that X = x is both necessary and sufficient to observe Y = y. The probabilities PN, PS and PNS are not identifiable with observational or experimental data unless Y is monotonic with respect to X, and both observational and experimental data are available [31] . If this condition is satisfied, then they are identifiable and can be computed as follows:P N = P(y) -P(y|do(x ′ )) P(x, y)and P S = P(y|do(x)) -P(y) P(x ′ , y ′ ) .(EQUATIONNote that PN and PS require the knowledge of do(X = x) and do(X = x ′ ). These quantities are generally unobserved for the whole population since observed individuals are only subject to one of the two conditions, unless experimental data is available. C Problems: structural equationsC.1 Div6 N ∼ P N C 2 = N (mod 3) ≡ 0 C 3 = N (mod 2) ≡ 0 C 6 = C 2 ∧ C 3where C 2 , C 3 and C 6 represent boolean values that indicate whether the number is divisible by 2, 3 and 6 respectively. P N is the mechanism for generating the original numbers (1 to 400 in our examples).N ∼ P N M ∼ P M T ∼ P T C n = N (mod 2) C m = M (mod 2) C t = T (mod 2) C nmt = (C n + C m + C t = 1) ∧ (C n + C m + C t = 3)where C n , C m , C t and C nmt represent boolean values and P N , P M , P T are the mechanisms to generate the original numbers (1-8 in our examples). N ∼ P N M ∼ P M T ∼ P T C nm = N ≤ M C mt = M ≤ T C nmt = C nm ∧ C mtR ∼ P R L ∼ P L E ∼ P E C r>0 = R > 0 C l>0 = L > 0 C e>0 = E > 0 C r≥2 = R ≥ 2 C l≥2 = L ≥ 2 C e≥2 = E ≥ 2 C rl = R > L C re = R > E C l=e = L = E C r≥0,l≥0,e≥0 = C r≥0 ∧ C l>0 ∧ C e≥0 C r>l,r>e = C rl ∧ C re C r≥2,l≥2,e≥2 = C r≥2 ∧ C l≥2 ∧ C e≥2 C h = (C r≥2,l≥2,e≥2 = 1) ∧ (C l=e ∨ C r≥0,l≥0,e≥0 )where P R , P L , P E are the mechanisms to generate the original numbers (all combinations in which 20 candies can be shared in our example).Proof. Commutability of the Hex diagram implies that all paths from σ 0 to σ 1 result in the same outcome. This holds for all counterfactuals, which implies that Y LLM X=x = Y X=x for any value of X and Z. Therefore: E P(X,Y,Z) Y LLM X=x ̸ = Y X=x = 0 for any P(X, Y, Z). "Does 6 divide {'X'}? Use the factor method to answer this question. Be as concise as possible.""Imagine that {'X'} {'has'/'has not'} 3 as prime factor while retaining all its other prime factors. With this assumption does {self.divisor} divide {'X'}? Use the factor method to answer this question. Be as concise as possible."Counterfactual Prompt: "Let N, M and T be three integers. Then N+M+T is even if the three numbers are even or if only one is even and the remaining two are odd. Consider the numbers N={N}, M={M} and T={T}. Is N+M+T even? Be as concise as possible."Let N, M and T be three integers. Then N+M+T is even if the three numbers are even or if only one is even and the remaining two are odd. Consider the numbers N={N}, M={M} and T={T} and imagine that N {is/is not} even. With this assumption, is N+M+T even? Be as concise as possible."Direct Prompt: "Let N, M and T be three integers. We know that if N is smaller or equal that M and M is smaller or equal than T then N is smaller or equal than T.Consider the numbers N={N}, M={M} and T={T}. By only looking at the relationships (N={N} vs. M={M}) and (M={M} vs. T={T}), can we know if N is smaller or equal that T? Be as concise as possible."Counterfactual Prompt: "Let N, M and T be three integers. We know that if N is smaller or equal that M and M is smaller or equal than T then N is smaller or equal than T. Consider the numbers N={N}, M={M} and T={T}. Now imagine that the number N {'is smaller or equal '/ 'is not smaller or equal'} than M. Even if this contradict the values of the numbers X and Y, use this assumption and the relationships between and M={M} and T={T}, to decide if can we tell if N is smaller or equal that T? Don't make any conclusion or comment based on the values, just based on the assumption and the relationships. Be as concise as possible.""Rafa has invited Lara and Emma to his birthday party. He has {num candies} to distribute among them. They all will be happy in the party in one of the following cases: 1) Each of them gets at least 2 candies or 2) Lara and Emma get the same number of candies, but at least one candy each, and Rafa gets more than them. After distributing the candies, Lara gets {L}, Emma gets {E} and Raphael gets {R} candies. With this candies distribution, will they all be happy in the party? Be as concise as possible."Rafa has invited Lara and Emma to his birthday party. He has {num candies} candies to distribute among them. They all will be happy in the party in one of the following cases: 1) Each of them gets at least 2 candies or 2) Lara and Emma get the same number of candies, but at least one candy each, and Rafa gets more than them After distributing the candies. After distributing the candies, Lara gets {L}, Emma gets {E} and Rafa gets {R} candies. Consider the number of candies distributed to each of them and imagine that they think that {'Lara and Emma have the same number of candies'/'Lara and Emma have different number of candies'}. With this assumption, will they all be happy in the party? Be as concise as possible."F GPT-4 concretization Prompt:"You are an entity extractor expert. I am going to give you a question-answer pair. I want you to say if the meaning of answer is positive or negative. If the answer has words like 'Yes' this will make it positive. If the answer contains words like 'No' this will make it negative. Always answer with only one word (Positive or Negative). For the question '{question}' and the answer '{answer}' the meaning is" G HEX for counterfactual query σ0 σ1σ 0 = {N → 10, C 3 → F, C 6 → F, C 2 → T } σ ′ 0 = {N → 10, C 3 → T, C 6 → F, C 2 → T } σ 1 = {N → 10, C 3 → T, C 6 → T, C 2 → T } α c Q C6 γ Q C6 Q C3 = T rueFigure 11 : The HEX diagram for a counterfactual query in the Div6 problem. We split the query in two sub-queries Q C3=T rue and Q C6 that performs the two operations need to compute the counterfactual state. Q C3=T rue only sets the value of C 3 to True. Q C6 replaces the value of C 6 by its counterfactual. This operation can be executed via the concrete path (using the structural causal model of the problem) of by using an LLM.Let n be the number of instances of each problem. For example, n = 400 for the Div6 problem because we use the first 400 integers to test reasoning. For the intervention node X and the outcome note Y , we distinguish between factual predictions Y |X (simulated from the original reasoning graph) and counterfactual predictions Y X=x (simulated from the intervened graph). We respectively denote the LLM versions of this quantities as Y LLM |X = x and Y LLM X=x , which are computed via factual and counterfactual prompts. (3)EQUATIONLet m be the number of bootstrap samples used from the binary answers of the LLM. P N i and P S i are estimation of P N and P S for the ith bootstrap sample. Then 38th Conference on Neural Information Processing Systems (NeurIPS 2024).See https://en.wikipedia.org/wiki/Lambda_calculus for a quick introduction to Lambda calculus.Note that while generating a larger number of model answers could potentially increase accuracy, the computational costs are prohibitive. Therefore, the bootstrap approach serves as a reasonable compromise.The code to reproduce the analyses and figures can be provided upon request, and will be made open source if this work is accepted for publication. 
empirical illustrations: 
 We focus on three math problems, each with a progressively higher level of difficulty.Divisibility by 6 (Div6): We compute the PN and PS to determine the impact that an integer N 's divisibility by 3 (denoted as C 3 ) has on its divisibility by 6 (denoted as C 6 ). For our analysis, we consider N ∈ [1, 400] . Even sum of integers (EvenSum): We examine scenarios where the sum of three integers M , N and T is even. This can occur under two conditions: when all three integers are even, or when one is even and the other two are odd. We evaluate PN and PS for impact that M being odd or even (C m ) has on the resulting sum being odd or even (C mnt ). For our analysis, we consider all possible values for M , N and T , with each integer ranging from 1 and 8.Candy party (CandyParty): In this hypothetical scenario, Rafa is having his birthday party with two guests, Lara and Emma. They have 20 candies to distribute among themselves. The party will be considered 'happy' if the candy distribution satisfies at least one of the following conditions: (i) Each person gets the same number of candies, or (ii) Rafa gets more candies than both Lara and Emma, but Lara and Emma each receive an equal number of candies, with both receiving at least one candy each. We compute the PN and PS for the impact that Lara and Emma receiving an equal number of candies (denoted as C lm ) has on the party being 'happy' (denoted as C h ).A fourth problem (ConPref) is included in Appendix B. The reasoning graphs for the problems EvenSum and CandyParty are shown in Figure 4 (Right). The structural equations corresponding to each of these graphs can be found in the Appendix C. We estimate the PN and PS for each of these problems using three difference language models: GPT-2, GPT-3.5-turbo and GPT-4 [23] . Our objective is to investigate whether the ability to reason, as conceptualised in this paper, emerges as the complexity and size of the models grow. While similar evaluations could be conducted using other families of LLMs, such as Llama [33] , Gemini [30], Phi [11] , etc., we have chosen to limit our analysis to the GPT series here for the sake of a clearer and more straightforward exposition, but more results are available in Appendix J.To assess the reasoning abilities of various models, we use the following metrics:1. Factual Inconsistency Rate (FIR): This measures the rate of inconsistencies when the models respond to factual queries. 2. Counterfactual Inconsistency Rate (CIR): Similar to FIR, but this metric measures inconsistencies in responses to counterfactual queries.For a detailed explanation of these metrics, please refer to Appendix H. We estimate the standard errors of FIR and CIR by examining the variations in outputs across multiple model responses. We take this aspect into account by collecting multiple answers from the models and propagating the stochasticity of the answers to the computation of PN and PS. Additionally, we capitalise on this variability to construct the densities over the inferred PN and PS. This process involves generating 500 bootstrap samples from the model's factual and counterfactual responses 2 . From these densities, we calculate γ-PN-overlap, which measures the concentration of the probability distribution within a radius γ around the actual PN, and γ-PS-overlap does the same for PS 3 . One might wonder if the evaluation of reasoning truly requires PN and PS, or if it could be sufficiently assessed by examining only the inconsistency rates in factual/counterfactual data. Figure 5 (Right) underscores the importance of PN and PS. It presents the estimated distributions of PN for the Div6 problem, based on 500 replicates under five scenarios where true counterfactuals are randomly altered with probabilities 0.005, 0.001, 0.05, 0.1 and 0.2. As we might anticipate, the greater the deviation from a dataset free of counterfactual errors, the more significant the discrepancy from the actual PN = 1 for this example. Notably, even minor perturbations can lead to substantial shifts in the estimated PN. For example, with a 0.05 probability of counterfactual perturbation, the estimated PN varies between 0.5 and 0.9. This suggests that relying solely on counterfactual errors could lead to an overestimation of the models' reasoning abilities, particularly their understanding of the necessary and sufficient conditions within a problem. Furthermore, a counterfactual error rate of 0.2 in this example results in entirely inconsistent (negative) probabilities due to the mismatch between the conditional and interventional distributions, as defined in Eq. 1.We computed the CIR, FIR, γ-PN-overlap, and γ-PS-overlap for the problems Div6, EvenSum and CandyParty using GPT-2, GPT-3.5-turbo, and GPT-4. Figure 7 (Left, Centre) features the γ-PN-overlap and γ-PN-overlap curves for all models and problems, where ideal reasoning corresponds to the metrics equalling one for any value of γ. GPT-4 shows this level of reasoning for the Div6 problem. However, GPT-2 had an accurate PN for Even-Sum, but the PS estimates were notably less accurate. An emerging trend towards reasoning is observed in the GPT family of models, particularly seen with GPT-4 for the Div6 problem. An intriguing question is whether future versions of these models will similarly approach the true PN and PS for other problems as well. 
discussion: 
 The primary objective of this paper was to explore and understand the reasoning abilities of LLMs, which is essential for their successful deployment in a range of applications. Given the growing dependence on LLMs for complex reasoning tasks, such as mathematics, programming, or strategic planning, understanding this is crucial. To evaluate these reasoning abilities, we introduced a novel framework that employs probabilistic measures of necessity and sufficiency, and find that while various models (GPT-2, GPT-3.5-turbo, and GPT-4) can replicate aspects of reasoning to some degree, they often falter when it comes to counterfactual reasoning. What makes our approach unique is that we test the models with scenarios where generalization by reasoning, rather than replicating statistical patterns in the training data, is required to provide correct answers. Notably, the ability to reason, as defined in this paper, does improve with more complex models, yet it is still far from flawless. This observation leads to the question of whether future versions of these models will achieve perfect reasoning. Our results are significant, as they reveal the limitations of LLMs, and emphasize the need for further research to enhance their reasoning capabilities.In general, reasoning goes beyond the math examples that we have included in the paper. We believe that the same theory and tools can be used in other domains. The Hex framework, that serves as a mathematical framework to formalize our ideas, requires the definition of a query, state, and an abstract execution machine that is used to predict how the state is modified given the query. In this framework, for instance, one could think about problems in vision, where the elements of the state are the objects in an image, and the query corresponds to a counterfactual query that describes an intervention in the environment. The concepts of necessity and sufficiency still apply in this scenario.In an image where an object is removed or altered, the framework can help determine the impact of this change on a property of the overall scene. We believe that this approach and will be key in other fields such as robotics, and or social sciences, where understanding the necessity and sufficiency between different elements is crucial for accurate reasoning and decision-making.Limitations: Our approach has several limitations that we acknowledge, but did not address within the scope of this research.1. Dependence on reasoning graphs: our method requires access to reasoning graphs. This requirement may hinder our ability to fully understand the reasoning abilities of LLMs in situations where it is challenging to derive relationships, including causal ones.2. Boolean variable restriction: our method is designed to work with boolean valued variables, which is restrictive, particularly for cases involving multiple states or conditions occurring at the same time. However, we believe that this issue can be addresss with further research. 
 
 Appendix:  


Decision: Reasoning capabilities of large language models is a topic of great interest to the larger NeurIPS community. This paper provides a novel framework that draws from concepts from causal reasoning to identify whether LLMs are actually able to perform reasoning. The introduction of such a systematic framework to this space is a meaningful contribution to this space, which is otherwise filled with papers that look at shallower metrics like accuracy. However, the primary concern multiple reviewers have raised is the authors’ choice to focus the evaluation on a specific class of reasoning problems. While from the point of view of a conference paper, the choice to focus on a specific class is reasonable, it would be helpful to update both the title and abstract to accurately reflect this limitation. This is also important because the current formulation also limits its focus to problems with boolean variables. Additionally, reviewers have pointed out a lack of a more detailed discussion about the implications of the current results. If accepted, the authors will have access to an additional page. It is recommended that at least part of that space is used to incorporate discussion along these lines.

Given the relevance of the paper and its contribution, I would recommend the paper be accepted.

Reviews: review: paper_summary: paper_summary: This paper evaluates the reasoning capabilities of LLMs within the framework of Judea Pearl's hierarchy of causality, focusing particularly on the ability to perform counterfactual reasoning.

LLMs are perceived as (non-deterministic) abstract machines within the HEX framework. The study assesses these models by comparing their performance on factual versus counterfactual problems, which are structured as reasoning graphs of boolean conditions. The necessity (PN) and sufficiency (PS) probabilities are estimated for 4 reasoning tasks: Divisibility by 6, Even Sum, Candy Party, and ConPref, ranging in complexity.

The paper investigates whether the actual PN and PS align with those estimated from both factual and counterfactual data, using models from the GPT family (GPT-2, GPT-3.5, and GPT-4).

Findings show limited reasoning capabilities for the evaluated model family, with GPT-4 performing the best on the divisibility task.
strength_weakness: strength_weakness: - From the introduction "...reasoning is typically understood to be the ability of these models to demonstrate emergent capabilities that surpass mere statistical pattern recognition in the training set.". Without knowledge of or control on pre-training/post-training data for the GPT model family, it is difficult to infer that a measured reasoning capability via the introduced framework is due to "emergence" and not pattern recognition in the training data.

- The framework's applicability is reduced to reasoning tasks that can be represented as graphs of boolean variables, which may limit its broader adoption.

- Line 84-85: "The closer the estimated PN/PS values to the actual PN/PS values, the better it is at reasoning. " needs further elaboration. Specifically, an explanation of how these metrics correlate with enhanced reasoning capabilities would help in understanding the effectiveness of the framework.

- The framework imposes that LMs are abstract machines, suggesting deterministic behavior. Yet estimating probabilities of necessity (PN) and sufficiency (PS) inherently relies on variability in model responses. 

- Statistical significance of observed results could further support the validity of the framework.
questions: questions: - I assume all findings are based on temperature > 0 to fulfill the variability criterion.
Did you generate multiple completions for each prompt (factual and counterfactual), and if so, how were these incorporated into the final analysis? "10 replicated tests" is mentioned under Figure 5, How these 10 contribute to the final findings?

- Was prompt optimization attempted for smaller models as well, or just for GPT-4? Given the sensitivity of LMs to prompts, could there be a bias towards GPT-4 because it handles more complex prompts better?

-While the paper discusses PN and PS as measures of reasoning, there is less focus on how these measures help in interpreting the decisions making process. What are the main benefits in comparison to just using accuracy?
limitations: limitations: yes. See Weaknesses and Questions.
paper_summary: paper_summary: The paper introduces a systematic method for assessing the reasoning capabilities of large language models (LLMs) by focusing on the concepts of necessity and sufficiency in logical reasoning. It leverages a probabilistic interpretation of these concepts and uses a reasoning graph based on boolean conditions to test the models.
strength_weakness: strength_weakness: 1. The workload and the introduction of related work in the main body of the article appears somewhat insufficient. It is recommended to incorporate the more significant content from the appendix into the main text.

2. Due to the lack of more rigorous and in-depth theoretical analysis, the results may not have sufficient theoretical persuasiveness. Thus, the paper would be more valuable if it provided some conclusive summaries or proposed improvement methods for the assessment results.
questions: questions: 1. Is subsection 3.2 of the paper complete?

2. The paper treats large language models (LLMs) as black boxes to measure the probabilities of causation. Does this approach have deeper significance and value? Additionally, is this evaluation method applicable to recurrent neural networks (RNNs) or large vision models (LVMs)?
limitations: limitations: The limitations are mentioned by the author in Section 5.
paper_summary: paper_summary: To assess the reasoning abilities of large language models (LLMs) in complex tasks e.g., causation scenarios, this paper introduces a novel framework that utilizes probabilistic measures of necessity (PN) and sufficiency (PS). Through a series of mathematical examples, the study computes approximations of PN and PS using reasoning graphs and LLM that generate factual and counterfactual datasets. The generated results from the LLMs are then compared with true simulated PN and PS values from reasoning graphs. The experimental findings indicate an emerging trend towards improved reasoning capabilities within the GPT family of models, including GPT-2, GPT-3.5-turbo, and GPT-4.
strength_weakness: strength_weakness: - Does four mathematic experiments representative in causal reasoning tasks?
- Before presenting Sections 2 and 3 on the probabilities of causation in an LLM, the paper should first explain the method for computing true PS and PN values using reasoning graphs to provide better context for readers.
- In Section 4.2 line 261: “Each density is labeled with the model that was used used to generate xxx”->typos
questions: questions: It would be clearer if the author(s) could explain:
- What is the relationship between the experiment evaluation metrics, Factual Inconsistency Rate (FIR) and Counterfactual Inconsistency Rate (CIR), and the results of PN and PS?
- Could you explain more about how the datasets (factual and counterfactual) were acquired from reasoning graphs?
- What does the random noise introduced in the true counterfactuals mean, and how did you measure the noise level?
limitations: limitations: The limitations and broader impact of the study are clearly outlined in the paper.
paper_summary: paper_summary: This paper investigates the reasoning capabilities of large language models (LLMs) by examining the concepts of necessity and sufficiency, which are key elements of logical reasoning. To assess the LLMs' reasoning abilities, the authors introduce a framework that computes the probability of necessity (PN) and the probability of sufficiency (PS) by comparing the actual values derived from factual and counterfactual datasets with those simulated by the LLMs. The paper presents a reasoning test for assessing LLMs' reasoning abilities using a divisibility problem as an example. The authors create factual and counterfactual datasets based on a reasoning graph and compare the actual PN and PS values with those generated by the LLMs. The results show that the closer the estimated PN/PS values are to the actual PN/PS values, the better the LLM is at reasoning.
strength_weakness: strength_weakness: - The paper uses a single example, the divisibility problem, to demonstrate the reasoning test. While this example helps illustrate the concepts, it may not be representative of the wide range of problems LLMs are expected to solve. Including additional examples from various domains would strengthen the paper's claims and generalizability.
- The paper does experiments on specific LLMs (gpt series) but doesn't provide results on other models or comparisons between multiple models. Including a variety of LLMs with different architectures and training datasets would provide a more comprehensive evaluation of the proposed framework. And, as GPT series are not open-sourced, it's unclear how OpenAI's hidden prompts or value-alignment finetuning will influence the results. 
- The paper focuses on the probabilistic interpretations of necessity and sufficiency in a creatain math problem but does not explore a wider range forms of reasoning (I'm not seeing this point as a signifigant weaknesses though, but I think the word "reasoning" actually covers more complex scenarios).   

Also see the questions below.
questions: questions: - How do the authors distinguish the results from the influence of the LLMs' prior knowledge? The LLMs' performance in the reasoning test may be affected by the patterns and relationships they have learned from their training data, especially commonsense reasoning questions related to Math. Are there any steps taken to control or account for this influence when evaluating the LLMs' reasoning abilities? In the interpretation of results, how do you account for the potential influence of the prior knowledge and the LLM's understanding of reasoning capabilities?
- Have the authors tried any prompt tuning or do they have some primary resuls? Although the authors have made this point clear in the limitations, I think the prompt actually greatly influence the output of the LLMs and may have a influence on the conclusions.
limitations: limitations: N/A

Comments: comment: Thank you for the detailed rebuttal and the additional experiments. Although the framework's reliance on binary variables may limit its broader application, the proposed approach could inspire similar research, given how unexplored evaluating reasoning in LLMs is (besides accuracy on reasoning tasks). Regarding prompt optimization, my concern was that GPT-4 might perform better because it can handle various prompt formats, suggesting that "better" results might be achievable for other (smaller) models with different prompts. I have increased my score to 6.
Thanks to the authors for their responses. I appreciate the effort they’ve put into addressing the feedback. The clarifications help enhance my understanding of the work.