Reasoning Elicitation in Language Models via Counterfactual Feedback
Authors: Alihan Hüyük
Xinnuo Xu
Jacqueline R. M. A. Maasch
Aditya V. Nori
Javier Gonzalez
Venue: ICLR 2025 Oral (2025)
Keywords: language models
reasoning
fine-tuning
counterfactuals
Primary area: causal reasoning
TL;DR: New approach to improve reasoning in language models via fine tuning with counterfactual synthetic data

Abstract: Despite the increasing effectiveness of language models, their reasoning capabilities remain underdeveloped. In particular, causal reasoning through counterfactual question answering is lacking. This work aims to bridge this gap. We first derive novel metrics that balance accuracy in factual and counterfactual questions, capturing a more complete view of the reasoning abilities of language models than traditional factual-only based metrics. Second, we propose several fine-tuning approaches that aim to elicit better reasoning mechanisms, in the sense of the proposed metrics. Finally, we evaluate the performance of the fine-tuned language models in a variety of realistic scenarios. In particular, we investigate to what extent our fine-tuning approaches systemically achieve better generalization with respect to the base models in several problems that require, among others, inductive and deductive reasoning capabilities.

Published as a conference paper at ICLR 2025
REASONING ELICITATION IN LANGUAGE MODELS VIA COUNTERFACTUAL FEEDBACK
Alihan Hüyük," *1 Xinnuo Xu, Jacqueline Maasch, Aditya V. Nori,² Javier González
Harvard University, *Microsoft Research Cambridge, *Cornel Tech
ABSTRACT
Despite the increasing effectiveness language models, their reasoning capabili- ties remain underdeveloped. In particular, causal reasoning through counterfactual question answering is lacking. This work aims to bridge this gap. We first derive novel metrics that balance accuracy in factual and counterfactual questions, cap- turing a more complete view of the reasoning abilities of language models than traditional factual-only based metries. Second, propose several fine-tuning approaches that aim to elicit better reasoning mechanisms, in the sense of the proposed metries. Finally, we evaluate the performance of the fine-tuned language models in a variety of realistic scenarios. In particular, we investigate to what extent our fine-tuning approaches systemically achieve better generalization with respect to the base models in several problems that require, among others, inductive and deductive reasoning capabilities.
INTRODUCTION
Large language models (LLMs) are shown to be ca- pable of delivering astounding performance in nu-
Factual Question
A number is divisible by six it has both and three prime factors. Is (N) divisible by six?
0.3
merous tasks across various domains. Examples stretch from writing assistants (Gan et al., 2023), sentiment analysis in social media (Simmering and Huoviala, 2023), and even applications in healthcare (González et al., 2023; Wong et al., 2023). While the ine-iinceg accuracy of these systems now undeniable, it is still rather unclear to what extent this accuracy due to effective recall of their train- ing data vs. genuine ability to reason by extract- ing, understanding, and adapting the fundamental concepts underlying that training data (Huang and Chang, 2023; Li et al., 2023). Previous work sug- gests that LLMS might exhibit some emergent soning capabilities (Bubeck et al., 2023; Kıcıman al., 2023). However, many have observed sig- nificant reasoning-reco substantially better on recall-based tasks that do explicitly require reasoning (Zhang et al., 2023a; Ahn et al., 2024; Seals and Shalin, 2024).
Eiror of Answers
Counterfactual Question
0.2
A number is divisible by six if it has both and factors. Suppose that {X) had three one of prime factors (retaining all its other prime fac- tors). Then, would have been divisible by six?
as prime
0.1
Figure Error rate of Phi3-Mini in answering factual vs. counterfactual questions sampling 10 answers for each N E {1...., 100). It performs disproportionately better for the factual question (cf. recall) as opposed the counterfactual ques- tion (cf. reasoning).
gap: LLMs still perform
Motivated by this discrepancy between how well LLMs can recall vs. reason, our goal in this paper is to see whether they can be fine-tuned explicitly to improve their reasoning. While reasoning can take different forms, we will focus on causal reasoning as provides us with a clear distinction between recall and reasoning': the former is limited to inferring statistical correlations, whereas the latter involves working with interventions and counterfactuals (Pearl, 2000). It has been previously shown that LLMs struggle with counterfactual questions compared to purely factual questions (Jin et al., 2024; González and Nori, 2024). This difficulty highlights the recall-reasoning discrepancy within the causal domain (Figure 1).
Work done as an intern at Microsoft Research Cambridge. Correspondence: ahayuk waa.ssas..aaaaraeeee harvard.cm evard.edd As an example, a different kind of reasoning would be symbolic reasoning, which involves manipulating symhols that represent mathematical statements (MacColl, 1897; Kelley, 1992).

---

Published as a conference paper at ICLR 2025
Adopting a causal framework also allows us consider LLM's ability to identify higher concepts that are essential for connecting causes to their effects in causal reasoning, such as necessity and suffi- ciency (Mackie, 1965; Lewis, 1973; Halpern and Pearl, 2005). For instance, cause X is said to be necessary for an effect Y if (i) without intervention, X and Y occur together and (ii) intervening to re- move X results in no Y (Pearl, 1999). Therefore, for LLM to be able to identify that X is necessary for Y. it needs to not only determine the factual in (i) is indeed the case but also simultancously recog- nize the counterfactual would have been different as in (ii). This makes identification of necessity, or similar relationships like sufficiency, a particularly good test of reasoning because it requires the LLM to understand when to recall (cf. factual thinking) vs. when to reason (cf. counterfactual thinking).
We improve the causal reasoning of LLMs by adapting established methods of fine-tuning, In particular, we consider supervised fine-tuning (SFT, e.g. Dai and Le (2015); Peters et al. (2018); Radford et al. (2018); Khandelwal et al. (2019); Howard and Ruder (2018), used in Ziegler et al. (2019); Ouyang et al. (2022)) and direct preference optimization (DPO, Rafailov et al. (2024), used in Tian et al. (2023); Lin et al. (2024a)). For both of these approaches, we propose procedures to generate supervised and preference- questions. We argue that generating demonstrations on a question-by wew---y---uuutto basis only improves the correctness of individual answers. As we discussed, identifying higher concepts such as necessity and sufficiency requires coordination between how factual and counterfactual questions are answered together. To target these higher concepts directly, we propose generating preference-base datasets over dialogues involving both factual and counterfactual questions.
datasets using factual questions as well as counterfactual
When the goal of fine-tuning is specifically to improve reasoning, a unique problem arises in evaluating the fine-tuned LLMs: we cannot just measure performance for held-out set of test samples within the same reasoning task. If we do, it would be impossible to tell whether the LLM actually learned to reason or whether is still recalling the demonstrations we have made during fine-tuning." 2 Hence, measuring the generalization performance with respect to new reasoning tasks becomes crucial. 3 We cannot expect fine-tuning on one problem instance to arbitrarily generalize to all problem instances either. So, building systematic understanding regarding to what extent fine-tuning for reasoning should be expected to generalize becomes important as well.
To build that understanding, we identify different modes in which reasoning in one problem is trans- ferred to other problems. Notably, we define inductive generalization and deductive generalization. Given causal system where X + Y + Z, inductive generalization is the ability to reason about the transitive relationship X + Z when demonstrated how to reason about X - Y and Y ->> Z. Conversely, deductive generalization is the ability to reason about the relationships X - Y and Y + Z when demonstrated how to reason about X + Z. We show that fine-tuning for reasoning generalizes much more effectively in an inductive mode rather than deductive mode (among many other insights in Section 5).
Contributions. We have four major contributions, corresponding to each of the following sections:
$2 We describe a framework for fine-tuning based on causal reasoning and formally categorize the ways in which reasoning generalizes from one problem to another. These categories common-effect, common-caus, inductive, and deductive.
$3 We introduce novel metrics to measure the reasoning performance of an LLM, defining necessity and sufficiency inconsistency rates (N-IR & S-IR) based on probabilities of necessity and suffi- ciency from the causality literature. We also introduce the concepts of absent necessity and absent sufficiency to supplement cause-effect relationships covered neither by necessity nor sufficiency. $4 We propose procedures to generate datasets to be used with SET and DPO to fine-tune for reasoning by incorporating counterfactual feedbuck. In particular, we argue for generating dialogues that involve paired factual and counterfactual questions to directly target the reasoning metrics introduce in Section 3. We call this causal consistency feedback.
$5 Finally, we evaluate the performance of the procedures proposed in Section 4 using the metrics introduced in Section 3. Moreover, we investigate to what extent that performance generalizes in relation to our categorization in Section 2.
For instance, chain thhtuughtt prompting aims to improve reasoning by providing examples of how prob- lem can he solved in smaller steps. While such prompting effective, unless tested on cases that require novel re- arrangement those smaller steps, its effectiveness can be attributed to successful imitation the provided exam- plcs and is not necessarily the result of true reasoning (Wei et al., 2022, see Appendix E. for further discussion). We interested in this particular notion of generalization although there are other notions, see Appendix F.
2

---

Published as a conference paper at ICLR 2025
trained
trained
trained
X
trained
trained
X
X
tested
trained
fested
tested
tested
(a) Common-Caues
(b) Common-Effed
(c) Inductive
(d) Deductive (Effect-Based)
Figure 2: Different modes of generalization, terms of the cause-effect relationships demonstrated during fine-tuning (i.e. D, blue) vs. the relationship that the fine-tuned model is evaluated on (i.e. Px-Y, orange).
FINE-TUNING FOR REASONING
World Model. We consider a causal world model, in which X (cause) and Y (effect) are two binary variables, indicating the absence presence of some conditions. We will denote with æ, y the values taken by X and Y respectively when the conditions they represent are present, and with x', y the complements of these values (i.e. the values taken by X and Y when the conditions they represent absent). The context, denoted by U, consists of all exogenous variables. Without any loss of generality, assume that all randomness in the model captured through these exogenous variables, and all endogenous variables, including X and Y, are deterministic functions of the exogenous variables (i.e. the context U). We denote these deterministic functions X = fx( and Y = fr U). Additionally, we denote the the potential effects under the potential interventions for each unit in the population as Y, Yldo(X x) = fy (x: U) and Y = Y|do X = x²) = fy (x',U)
Language Model. We can estimate different effects using language model. Formally, let (u) be a factual question template that describes the world model in natural language and asks what the factual effect would be for specific context Denoting the language model by E, let = (()()) be the model's answer to this question, which will be in natural language form. To transform the answer into binary form, we use a mapping h such that Y h(a) = h(E(g(u))) E {v, y J'* Similar to the factual case, suppose also have interventional question templates dz (u) and dz (u) that describe the world model. However, these templates ask for the potential effects under interventions do(X = z) do(X = z²). This leaves with the following estimates for the two potential effects. For given context u, we rely on the factual question template when the effect is factual, and on the interventional question template when the effect is counterfactual:
Y. [holoq()) = (U)
x Y. ( h q(U)
X = if X =
(1)
Problem. Let P describe the context distribution such that U ~ Moreover, let Px--y denote the corresponding distribution of cause X = fU) and potential effects Y, (U) fy (z, U), = (z- fy U) such that X,Y,, Y Px-rr. Suppose
interested in optimizing some
metric Vxc; E IR that measures the reasoning performance of the language model for the cause-effect relationship Px-r (we discuss the design of V in the subsequent section). Then, the problem of fine-tuning for reasoning can be expressed as
maximize
V(e:Px--t
given P.D = (Px.
where &o is the target language model, and D is the set of different cause-effect relationships Px,--; that are available as demonstrations, These relationships may involve causes {X₂} and effects [Y;) other than the cause X the effect Y of interest. We refer to the case where only the cause-effect relationship of interest is demonstrated such that D = {Px-yy) } as the "in-domain" problem.
Modes of Generalization. As we have discussed in the introduction, an in-domain evaluation is not sufficient alone to assess the success of fine-tuning for reasoning. Therefore, we categorize different ways in which reasoning can generalize-the -tht is, how D might relate Px-y when PX-Y & D. We identify four main structures, summarized in Figure 2:
(i) Common-Cause When the relationship X + Y demonstrated, common-cause generalization refers the ability reason about other relationships X -> Y that involve the same cause X.
'n practice, this mapping would also he language model prompted to reduce given answers to binary "negative" "positive" (see the appendix for the exact prompt used in this paper).
3

---

Published as a conference paper at ICLR 2025
(ii) Common-Efffet: When the relationship X + Y is demonstrated, common-effect effect general- ization refers to the ability to reason about other relationships X - Y that involve the same effect Y. Unlike common-cause generalisation, the task of determining the factual effect without intervention remains the same, regardless of whether or X is the cause of interest.
(ii) Inductive:
When the relationship A + B and B + C are demonstrated, inductive general- ization refers to the ability to reason about the transitive relationship A + C. This ability may be hindered if A has a direct effect on C' that is not mediated by B. We will investigate this empirically in Section 5.
(iv) Deductive: Similar inductive generalization, consider the causal relationship A - B C. When the relationships A - C and B + C are demonstrated, effect- based deductive generalization is the ability reason about the relationship A B. Similarly, when the relationships A + C' and A + B are demonstrated, cusse-based deductive generalization refers to the ability to reason about the relationship - C. We will investigate the potential differences between the two scenarios empirically in Section
METRICS OF REASONING
Having defined the problem of fine-tuning for reasoning, we now discuss what would be good measure of reasoning ability (i.e. a good choice for V). Section 3.1, we define error rates based on the correctess of answers given by the language model to individual questions. In Section 3.2, we go beyond these simple error rates and propose various inconsistency rates that capture the causal consistency between the factual and counterfactual answers given within the same context. As emphasized in the introduction, such consistency is necessary to identify causal relationships such will describe various methods for generating
necessity and sufficiency. Later, in Section datasets that aim to optimize either of these metrics.
3.1 CORRECTNESS
Ignoring relationship between factual and counterfactual effects, the correctness of individual answer a = l q(u) lo Oz (u) f zz (u) can be characterized by the factual error rate (F-ER) and the counterfactual error rate (CF-ER) respectively
= #Y}
CF-ER P
Ye # Yo Yv
if x
if X x=x
where Y, Ýz. and Y. represent the binary values implied by the answer a. Using these two metries, we define the average error rate as Avg-ER- = (F-ER + CF-ER)/2.
Why are factual and counterfactual correctness alone not enough? Being able correctly estimate factuals (cf. F-ER) or counterfactuals (cf. CF-ER) is, of course, an important step in causal reasoning. However, what we ultimately want to characterize the relationship between cause and its effect. For instance, is the cause necessary for the effect to occur? Is it sufficient? Or do the cause and the effect only occur together (necessary and sufficient)? Identifying such relationships rely on the estimated factuals and counterfactuals collectively-ont not always lead to a correct characterization of the cause-effect relationship. By measuring the factual and counterfactual accuracy separately, F-ER and CF-ER fail capture any dependencies between the two answers and how they might be describing a larger relationship together.
getting one right but not the other might
As conerete example, consider necessity. According Pearl (1999), when cause X and an effect Y occur together (i.e. = I and = y). the cause is said to have been necessary for the effect if the effect would not have occurred in the absence of the cause (i.e. Y. y'). Making an accurate judgement regarding whether there is a necessity relationship between X and Y requires both Ý and Y. to be correct when X and Y = However, no factual or counterfactual estimate needs to be correct when X = (as it is immediately apparent that cases where = x' do not affect necessity), and similarly, only the factual estimates needs to be correct when X = but Y = y/. F-ER and CF-ER do not account for this complex requirement at all. In particular, depending on how X and Y are distributed, language model can achieve F-ER and CF-ER as high as 1/2 by always estimating either Y, or Ys correctly (but not both together) while never reaching an accurate conclusion regarding necessity.
4

---

Published as a conference paper at ICLR 2025
3.2 CAUSAL CONSISTENCY
Previous work (González and Nori, 2024) has considered the of "probabilities of causation" together with F-ER and CF-ER to provide a set metrics that fully characterize the relationship between cause and its effect. Similar to necessity, Pearl (1999) also provides causal definition of sufficiency: whether the cause would have produced the effect (i.e. Yz y) when both the cause and the effect are absent (i.e. X x' and Y =/). The probability of necessity (PN) and the probability of sufficiency (PS) are defined as:
PN = PY y' X
v}
PS: P y/ x - =y}
The answers given by the language model to factual and counterfactual questions and the effects Yz, Y estimated from those answers naturally induce an empirical pair of PN and PS values:
PN P(Y v/ X z, Y = y)
PS
PP y =z' Y =x}
(5)
Why are PN and PS correctness alone not enough? To evaluate reasoning in language models, González and Nori (2024) use (1) probabilistic measure (7-overlap) to assess how well the distri- butions of PN and PS match the true PN and PS, and (2) the factual and counterfactual error rates. We refine this approach by defining unifying metrics that simultaneously take both aspects of the problem into account, thereby simplifying the evaluation process.
Due to the averaging done by probabilities, achieving a perfect PN-PS with the language model only requires identifying correct vs. predicted marginal frequencies, without needing individual units to be accurate. Although this is captured by the factual and counterfactual error rates F-ER and CF-ER, it is convenient to have a single metric that encapsulates both dimensions of the problem. We address this by requiring the necessity or sufficiency relationships identified by the language model to be accurate on a unit-by-unit basis. A unit is a realization of the exogenous variable U. It induces the values of X and Y well as the counterfactual outcome Yx'. where X' represents the complement of the observed X regardless of its value. Note that Yx- = Y is the factual outcome.
We focus on necessity where a unit/context might exhibit one of three situations: (i) Necessity occurs, denoted by "N", meaning that both X and Y occur, X = x and Y = y, and the cause was necessary for the effect, Yx = y/ (ii) Necessity does not occur, which we denote by "N"", meaning that both X and Y occur but the cause was not necessary for the effect, Yx' + y'. (iii) Not relevant case as necessity concerned, which we denote by Ø. when neither X nor Y (or both) did occur. Since value of the context variable U fully characterizes the unit, we can define unit-wise necessity as
N if X - x AA - y A yx - yf N' if X==AAY= Yx /V
N(X,Y,
(6)
x
The necessity inconsistency rate (N-IR) is the frequency with which the language model estimates the unit-wise necessity N inaccurately (see Appendix G for an alternative interpretation):
N-IR = (N(X, Y, 8X:; U) + (X Y, Yx:: U)},
where EP(U)) denotes the expectation over U and Y, Yx are the analogous factual and counterfactuals to Y, Yx' estimated from the model. Remark that PN Ep(U) W NWW / Ø by construction. Also note that N-IR = 0 implies that PN = PN. However, erTors made in different units can no longer 'balance each other out' to achieve N-IR- O. We can also define context-wiss sufficiency S in an analogous way: (i) S = S if = x', Y = y', Yx y, (ii) S = S' if X = x', Y = y Yx # and (iii) S = Ø otherwise. This induces the sufficiency inconsistency rate S-IR = P + S}.
Neither PN and PS nor the inconsistency rates N-IR and S-IR are sensitive to all answers given by the language model. This is because necessity and sufficiency only concern cases where X = x, Y Y= and X 1/, = y/- For instance, when X = x' and Y = y and the factual effect has been estimated correctly such that Ý = Y, the counterfactual estimate Y, has no impact on PN, PS, N-IR, or S-IR. Regardless of whether Y, Yz. all four quantities stay the same. To cover all possible counterfactualls we can ask a language model for, it makes sense to also evaluate counterfactuals of the type Y, X x, Y y/ and Y, v x', Y y. Of course, the probabilities of these counterfactuals can be defined by means of PN and PS by changing the default observed state. However, here we name them absent necessity and absent sufficiency to be explicit about the
5

---

Published as a conference paper at ICLR 2025
two extra cases where language model can make mistakes." 9 In context- based framework, the corresponding context-wise AN and AS are defined in a similar fashion as N and S, which induce the inconsistency rates AN-IR P(AN & AN} and AS-IR P(AS * AS}. As a reasoning metric, we define the average inconsistency rate as Avg-IR = (N-IR + S-IR + AN-IR + AS-IR) /4. This metric has the following properties: (i) it accounts for all characterizations of the necessity and sufficiency of the target causal effect, and (ii) it is unit-dependent, so factual and counterfactual accuracy errors cannot be balanced out.
An Illustrative Example. We illustrate
the difference between correctness and
arieetel corter
causal consistency as follows. Consider
the following language models: (i) Factu- ally Correct answers all factual questions correctly (F-ER- 0) but makes occasional mistakes in answering counterfactual ques- tions. This represents an extreme version of the imbalance highlighted in the intro- duction. (ii) Uniformly Correct makes both factual and counterfactual mistakes at equal rates (F-ER = CF-ER), but these mis- takes happen independently of each other. (iii) Causally Consistent reasons on a unit- by-unit basis (as opposed question-by- question) and either gets both the factual question and counterfactual question right or gets both of them wrong. Suppose the cause never prevents the effect such that (X.Y..Ya)) {(..8/. (2,, ,)), (I,, M.,), (x', y', n)), (x', w), (x', y. v) (with equal probabilities).
Pork bocociy
82
0.5
2.4
(a) PN PS
(b) N-IR & S-IR
Figure 3: Causal consistency vs. correcmess. Despite having the same Avg-ER, different types of error distributions lead to widely different PN & PS characteristics.
Figure 3 shows the PN & PS as well as N-IR S-IR of these models for fixed levels of Avg-ER as the error shifts between contexts where the cause may be necessary (i.e. X = a)) vs. contexts where it may be sufficient (i.c. X = z'). Despite having the same Avg-ER, the three models induce widely different PN& & PS values, representing different causal interpretations. While Figure 3a might suggest that the factually correct models are the best performing, this is purely coincidental. Due to the averaging done by PN & PS, the mistakes made in different units end up balancing each other out. Looking at N-IR & S-IR in Figure 3b reveals that the causally consistent models are actually the best, even outperforming models with significantly smaller Avg-ER.
4 FINE-TUNING WITH COUNTERFACTUAL FEEDBACK
Despite the significant differences between correctness and causal consistency, success in either metric relies on accurate estimates of counterfactual outcomes. Therefore, to solve the fine-tuning problem in (2), it is essential to leverage the counterfactual information available in demonstrations D, irrespective of the metric we aim to target as V. We present a data-centric approach to achieve this and propose three methods for generating datasets using counterfactual feedback. These datasets can then be utlised by existing algorithms for fine-tuning such as SET or DPO. These methods are summarized in Figure 4.
Supervised Counterfactual Feedback. Recall that we assumed access to an extractor h that can reduce answers given in natural language to binary outcomes ÿ h(a)) E {y. Now, further suppose that we can perform this extraction in reverse, denoted as H: Given a question q and the true H1 (q. Mtre). In practice, we achieve this by prompting a language model to provide an answer to question that starts with "Yes" " or "No" " (see the appendix for the full prompt). Based on these answers, we generate a dataset D' of both factual and counterfactual questions and their answers:
outcome Shree corresponding to this question, we can form natural language answer
D = { q = g(U): ay = H(g Y), = (U), aef H (g Yx.) JU.X.YY
This dataset can directly be used with any SFT algorithm to fine-tune the target model to-
'Note that the use of these quantities is an alternative but equivalent characterization of all possible counter- factual outcomes to the one in Pearl (1999), where the probabilities of disablement Y,, = = y and the probability of enablement Yz- y/Y y' are introduced.

---

Published as a conference paper at ICLR 2025
Preference-based Counterfactual Feedback. SFT can be limited by the quality of answers generated as ground-truth and their similarity to the model's original answers. Without access to a language model that is already better at reasoning than our target model, it might challenging to build an answer generator H that provides high quality samples. In that case, it is desirable to provide direct feedback to the answers generated by the target language model. We do so by first generating multiple answers to different questions (using high sampling temperature to get sufficient variation);
D = { U, q = q(U), a,11 ~ lo(a))
af[M] ~ lo(g).
Yef (U), ag(1] fo(dg). ..... dg|N fo(ug) JU.XYYY..-D
(8)
Then, we form a preference-based dataset where correct answers are preferred over incorrect answers:
ayli] > a/bil = 1(b(a(())
> 1(h(af(i) = Y
(9)
agrli] > aزابه = 1(h(ae(t)) = Yx 1(h(arli)) Yx-}
(10)
The DPO algorithm can directly be used with this dataset to maximize the likelihood of preferred answers (i.e. ali)) relative the answers they are preferred over (i.e. a(j]).
Preference-basee Causal Consistency Feedback. Running DPO with preferences determined by reward function, where alternatives with higher rewards are preferred over those with lower rewards, is equivalent to maximizing that reward function (Rafailov et al., 2024). In our case, this means that running DPO with the above preferences would, in effect, minimize the average error rate (i.e. Avg-ER), as these preferences are generated by treating correctness (i.e. 11 (1((aa) = Y = YJ) as a reward function. To target the inconsistency rates introduced in Section 3.2, we propose to (i) pair factual and counterfactual questions, (ii) prompt the target language model to answer them simultancously, and then (iii) clicit preferences based on the joint answer. Formally, (a.li), aglli) > aglli) = R(h(a)))), h(agli)) R(h(a/bi)), U(aoD)):U) (1))
where R(Ý N
+ 1 (AN = AV] + 1 = ÁS). We call this causal consistency
Foro: lbbe
(CCF) (see Fig-
ure 4b vs. 4c). CFF explicitly targets Avg-IR rather than Avg-ER and can still be used directly with the DPO algorithm.
35 LL
(a) Supervised CF
EXPERIMENTS
We begin with a proof-of-concept case study. We analyze hand-crafted puzzle assess the effectiveness of all fine-tuning techniques introduced in Section when trained on dif- ferent types of datasets within the context of the in-domain causal reasoning scenarios ($5.1). We also address the research question posed in Section 1, i.e., to what extent the performance improvements in causal reasoning achieved through the fine-tuning process gener- alize across all the generalization modes ($5.2). Subsequently, we use three additional real-world problems examine our findings ($5.3).
Wi had beitennnnk A.
Forit, (rad boon X
(b) Preference-based CF
For ti will li
Oo will &
boon wou di hile bean
Fie
would El Ehate been
ho
You
U
5.1 IN-DOMAIN REASONING
(c) Preference-basee CCF
We evaluate all fine-tuning techniques when trained on various types of demonstrations in a synthetic in-domain reasoning problem.
Figure 4: Summary of the proposed fine taning methods. Supervised and preference- based counterfactual feed- back (CF) target correctness: the former by generating correct answers given each question and the latter by sampling answers and preferring the correct ones over the others. Causal consistency feodback (CCF) targets causal consistency instead: Asking both the factual and the counterfactual questions within the same dialogue allows to elicit preferences according to relationships between the factual and counterfactual answers.
Experimental Setup. See Figure 5. The puzzle describes a candy party. The context is defined by the four-dimensional random vector U = (NA, NB, Nc,ND) ND) where each element follows the same uniform distribution (11 12). The causal structure, derived from the narratives,
7

---

Published as a conference paper at ICLR 2025
is presented in the middle section of the figure. We selected A: Anna is happy or not as the cause
Question: Anna, Bill Cory, and Dave are going perty, where the going distribute candies.
happy gets least candies. Cary be
(X), and D: Dave is happy or not as the effect
canndees Bill will happy happy Anna Bill both happy or he gets at least candles. Dave will be happy Anna and Bill are both happy or if ho gets at loast 10 candies. Atter distribulling the candles, Anna gets Na. Bil gets No. Cory gets No, and Dave gats No. Dave happy?
gets
(Y). The factual questions ((u) are obtained by
randomly drawing values for the four numerical variables from the distribution 24, The counter- factual questions qx (u) are generated by intro- ducing an assumption that negates the cause i.e. if in the context A is "Anna is happy" based on the value of NA. the injected assumption would be "suppose that Anna is not happy", and vice versa. Since we are assessing the in-domain rea- soning scenario, the cause-effeet demonstration used during the fine-tuning phase are likewise employed in the evaluation phase.
Anna happy?
Cory is happy?
No. No,, 20(1...................
A Na 24
B Nu 2
(A BJ (No 2) (A (NN 2010)
Bill happy?
Dave is happy?
Counterfactual Question: Now. suppose that Anna is (not) happy ro gardlees of the candy distribution. With this assumption, is Dave happy?
Figure 5: Hand-crafted puzzle with the original factual question, causal model with structural equations, and a counterfactual question. Blue and orange arrows show the cause-effect interventions demonstrated to the model during fine-tuning and evaluation phases.
We first generate dataset D = ((((u). U (ūxx(u), ] for each fine-tuning techniques introduced in Section 4 following the algo- rithms shown in Appendix C. Then we fine- tune the mini version of Phi-3 (Abdin et al., 2024) on D. We include five baselines: the base language model (Phi-3 mini) without fine- tuning (Base), the base model fine-tuned using the SFT and DPO methods on factual examples ((g(u), (a) exclusively (SFT-OnlyF and DPO- OnlyF), ) and the base model fine-tuned using the counterfactual exam- ples { (w), acr)] exclusively (SFT-OnlyCF and DPO-OnlyCF). For OnlyF and OnlyCI, we have doubled the number of contexts sampled so that every method still has access to the same number of question-answer examples. As our proposed methods, we include the base model fine-tuned using SFT, DPO, and CCF methods on both factual and counterfactual examples (SFT-F&CF, DPO-F&CF, and DPO+CCF).
043
C2 X20
(a) S-IR w.r.t. F-ER
(b) S-IR w.r.t CF-ER
SFT and DPO methods
Figure 6: In-domain results in the candy party puzzle. The y-axis in both figures represents S-IR, while the x-axes represent F-ER and CF-ER, respectively. We focus on S-IR because, in this puzzle, the cause is more sufficient than necessary for producing the effect.
shows the sufficiency inconsistency rate (S-IR) in relation to the fac- tual/counterfactual eгror rates (F/CF-ER) across all approaches. SFT and DPO models, trained exclusively on either factual or counterfactual examples OSFT+OnlyF, OnlyO,, Onl+F and DPO+OnlyCF) do not improve S-IR, even though they manage to reduce the corresponding F/CF-ER. However, when given access to both types of examples, DPO+F&CF shows an improve- ment in S-IR, though this improvement is not as pronounced as the reduction observed in F/CF-ER, particularly in CF-ER. The SFT+F&CF model shows a significant enhancement in both S-IR and F-ER, but it fails to make progress in CF-ER. Finally, by directly addressing causal consistency, with S-IR factored into the reward during fine-tuning, the DPO+CCF model achieves substantial improvements across F-ER, CF-ER, and S-IR. These results highlight the crucial role of effectively coordinating factual and counterfactual feedback for advanced reasoning tasks.
Results. Figure 6
5.2 MODES OF GENERALIZATION
In this section, we answer the question "to what extent the performance improvements in causal reusoning achieved through the fine-tuning process generalize across all the generalization modes defined in Section 2n. As mentioned in Section 2, an in-domain evaluation alone is inadequate for fully assessing the success of fine-tuning for reasoning and differentiating it from basic recall. Therefore, we evaluate all fine-tuning methods in the generalization modes introduced in Section 2.
Experimental Setup. To allow for the problem in Figure 5 to reflect all possible generalization modes we made slight modifications to the puzzle context, creating two variations: chain NDE and chain WDE (refer to Structure-2 and Structure-3 in Appendix D.1). The top section in Figure 7
WWhen evaluating models, we sample 10 answers for each question, which gives us distribution over FR//R (rather than just a point estimation).
8

---

Published as a conference paper at ICLR 2025
Comman-CCre ("CC")
Cammen ("CE")
No Dinnct Effict ("NDE")
With Direet Blfiet ("WDE")
No Dinact Elfect ("NDE")
With Dirret Blfiet ("WDE")
No Dinas Elfect ("NDE")
Wish Direct Elfiet ("WDE")
am
AB
500 progge
400
(a) Ceamonn-Cause/EFig
(b) Inductive
(c) Deductive, Cause-B. (d) Deductive, Effect-B.
Eight scenarios involving three different causal structures: the bipartite graph B} - {C, D) (Structure-1 in Appendix D.I) well the chain A- + B + with and without direct effect from to C. (Structure-2 and Structure-3 Appendix D.I). Blue and orange arrows show the cause-effect interventions demonstrated the model during fine-tuning and evaluation phases. Bottom: The causal reasoning ability of the fine-tuned models generalizes most effectively in inductive demonstrations. However, with canmon-cause/effect and deductive demonstrations, they no longer show the same reasoning improvements observed in the in-domain setting.
Figure 7: Generalization results in the candy party puzzle. Top:
displays all the causal structures used for each generalization mode, along with the cause-effect interventions demonstrated during the fine-tuning and evaluation phases. Based on the findings from the in-domain reasoning experiments (Section 5.1), where both SFT and DPO fine-tuning methods showed significantly better performance when provided with both factual and counterfactual examples, we include here only the methods SFT-F&CF, DPO-F&CF, DPO+CCF, and the Base model.
Results. The bottom section in Figure 7 presents the causal reasoning performance of all systems across the different generalization modes. We observe that: (i) based on demonstrations that involve just the target cause or the target effect (but not both as in the in-domain case) no longer leads to improvements in S-IR (unlike the in-domain case). While we do see improvements in N-IR, this can be attributed to better recall and not necessarily to better reasoning. The common--effec case leads to the greater improvement in N-IR precisely because the task of identi- fying factuals remains the same in this mode of generalization. (ii) Induction. Fine- -uning generalizes best when performed inductively. This is because relationships involving both the target cause and the target effect have been demonstrated, albeit not together. (iii) Deductions. While harder than induc- tion, deduction is also possible as long as there are direct effects that circumvent the intermediate variable. If there are such effects, deduction based on a shared cause becomes virtually impossible: Without any intervention on the intermediate variable, it is challenging to tell how much of the shared cause's effect is mediated through the intermediate variable vs. how much of it is not. Meanwhile, this seems be identifiable to some extent when interventions the intermediate variable demonstrated as in deduction based on a shared effect (see Appendix H for a more detailed analysis).
Fine-tuning
5.3 REAL-WORLD PROBLEMS
Experimental Setup. We present three real-world causal reasoning problems: in the Healthcare domain, we examine breast cancer treatment and develop a simplified problem that determines how different treatment options-namely. padiotherapycchmmohhrapp and surgery-are are assigned to patients based on cancer type. tumor size, and nodal involvement. This model is grounded in real-world guideline (MD Anderson Cancer Center) and published statistics on the disease (Orrantia- Borunda et al., 2022; Sezgin et al., 2020; Carey et al., 2006). In the Engineering domain, we implement an automatic fault detection algorithm for transmission lines (Reddy et al., 2016). This algorithm aims to identify the type of fault occurring measurements. In the Math Benchmarking domain, we select a math question from GSM8K (Cobbe al., 2021), widely used benchmark for evaluating language models on grade school math problems. A detailed explanation of these three problems, including the context, factual and counterfactual questions, causal structures, and the cause-effect interventions demonstrated during the fine-tuning and evaluation phases across different generalization modes, can be found in Appendix D.2, D.3, D.4 respectively. For the real-world problems, we sample the same number of contexts for each method as this is more likely to be the case in real-world application, where each context would correspond to an individual entry (e.g. a single patient in the healthcare domain, see Appendix for details).
transmission line using three different

---

Published as a conference paper at ICLR 2025
Results. The results for all three prob-
Table I: Average generalization performance across three real- world causal reasoning problems. The scores are normalized relative to the Base approach's 's scores in cach generalization mode, Higher scores indicate greater number of errors made by the approach, with scores above 1.0 meaning that the approach makes more mistakes than the Base model, which has not undergone any fine-tuning.
lems across in-domain and different gen- eralization modes available in Table 2 in Appendix A. Given the extensive num- ber of experiments in this table, we have summarized the Average Error Rate (Avg- ER) and Average Inconsistency Rate (Avg- IR) scores in Table 1. For this summary, we first normalized the scores of each ap- proach relative to the scores of the cor- responding Base approach. Then, for each generalization mode (including the in-domain scenario), we calculated the av- erage score of each tested method across all applicable problems. Not all generaliza- tion modes can be tested for every problem due to differences in causal structures, and the average score includes only the prob- lems that were tested for each generaliza- tion mode. In Table 1, higher scores indicate more errors. and scores above 1.0 signify that the approach makes more mistakes than the Base model. We observe that: (i) In the in-domain scenario, when the fine-tuning is guided by both factual and counterfactual examples (-F&CF) the language models show significant improvement in causal reasoning ability. (ii) Similar to what we observed in previous experiments, this improvement generalizes to most generalization modes, with the exception of common-cause and effect-based deduction. (iii) In most of modes, language models trained with causal consistency feedback (DPO+CCF) demonstrate a lower error and inconsistency rate.
OnlyF'
F&CP (Ours)
Mode
Metric
Rase
SFT
DPD
SFT
DPO
DPO:CCF
Domain
Avg-ER
Avg-IR
1.00
0.82
0.86
0.72
0.44
0.53
01488
Common-Cause
Avg-ER
Arg IR
1.00
1.00
1.30
1.35
1.75
1.49
1.17
1.14
1.62
1.42
2.04
1.80
Common-
Avg.ER wgg-ER
Avrg
1.00
0.60
0.71
0.60
0.53
0.66
0.64
Inductive
Avg-ER
Avg. IR
1.000
1.00
0.86
0.74
0.79
0.62
0.51
0.60
0.58
0.50
0.59
0166
Deductive, Cause-B.
Avg-ER
0.67
Avg-
1.000
0.62
0.59
0.53
0.53
0.51
Deductive, Effect-B.
Avg. ER 1.00
0.83
0.83
0.90
0.95
0.77
IR
1.00
0.91
0.87
0.93
0.89
0,76
6 RELATED WORK
Reasoning Evaluation. While our work focuses on reasoning elicitation, there is a plethora of work on reasoning evaluation (Frohberg and Binder, 2021; Wu et al., 2023; Chang et al., 2024).
Parmar et al. (2024) evaluate logical reasoning, Cohn and Frrmandez-Orallo (2023) evaluate spatial reasoning, Gandhi et al. (2024) evaluate social reasoning, Li et al. (2022); Jin et al. (2023); Ashwani et al. (2024); Li et al. (2024); Wang (2024) evaluate causal reasoning, and Maasch et al. (2025) evaluate compositional reasoning. Being able to determine which models are better at reasoning is an important aspect of reasoning elicitation. We have explored this aspect in Section 3 building on the work of González and Nori (2024), which has allowed us to consider relationships like necessity and sufficiency that require a higher level of reasoning than simply answering counterfactual prompts.
Counterfactual Frameworks. Counterfactual frameworks have been employed to explore aspects of LLMs other than reasoning elicitation. For instance, Lin et al. (2024b) formulate preference align- ment as a causal inference problem and develop an alternative approach algorithms like DPO. In Wu et al. (2021); Nguyen et al. (2024), counterfactual inputs are used to explain a model's ' predictions. Additionally, Kandpal et al. (2023); Zhang al. (2023b) model memorization through counterfactuals.
Fine-tuning with Factual Feedback. While we fine-tune language models for reasoning with counterfactual feedback, previous work has considered fine-tuning for factuality -that is providing factually correct answers to questions which do not involve interventions (Tian et al., 2023; Tong et al., 2024: Lin et al., 2024a). Khalifa et al. (2020); Korbak et al. (2022a;b) propose methods for controlled generation, which aim to constrain a model's answers using binary reward functions. While they suggest using correctness as the reward function to improve factuality, we have shown that targeting metrics like causal consistency require more fine- -en-gind feedback (beyond a binary reward).
7 CONCLUSION
This work introduced the problem of fine-tuning for reasoning, along with (i) taxonomy for general- ization modes, (ii) multiple metrics that address the limitations of existing performance measures, and (iii) methods for generating fine-tuning data with counterfactual feedback. We showed that fine-tuning for reasoning requires both factual and counterfactual examples, and that paring examples related through shared context can lead to improvements. A key limitation of our approach is the restriction of causes and effects to binary variables, which allows us to focus on high-level relationships like necessity and sufficiency present in human reasoning (see Appendix I for further discussion).
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
Author Maasch acknowledges the National Science Foundation Graduate Research Fellowship under Grant No. DGEE-1139899
REFERENCES
Abdin, M., S. A. Jacobs, A. A. Awan, J. Aneja, A. Awadallah, H. Awadalla, N. Bach, A. Bahree, A. Bakhtiari, H. Behl et al., "Phi-3 technical report: highly capable language model locally on your phone," arXiv preprint arXiv:2404.142/11 2024.
Ahn, J., R. Verma, R. Lou, D. Liu, R. Zhang, and W. Yin, "Large language models for mathematical reasoning: Progresses and challenges," arXiv preprint arXiv:2402.001 2024.
Ashwani, S., K. Hegde, and A. Chadha, "Cause and effect: Can large language models truly understand causality?" arXiv preprint arXiv:2402. 18139, 2024.
Mannuru, M. Jindal, D. Sengar, C. R. Kathala, D. Banga, V. Jain,
Bubeck, S., V. Chandrasekaran, R. Eldan, Gehrke, E. Horvitz, E. Kamar, Lee, T. Lee, Y. Li, S. Lundberg, H. Nori, H. Palangi, M. T. Ribeiro, and Y. Zhang, "Sparks of artificial general intelligence: Early experiments with GPT-4,"" arXiv preprint Xxxx:2333.12711 2023.
Carey, L. A., C. M. Perou, C. Livasy, L. G. Dressler, D. Cowan, K. Conway, G. Karaca, M. A. Troester, C. K. Tse, S. Edmiston et al., "Race, breast cancer subtypes, and survival in the carolina breast cancer study," JAMA, vol. 295, no. 21, pp. 2492-2502, 2006.
Chang, Y., X. Wang, J. Wang, Y. Wu, L. Yang. K. Zhu, H. Chen, X. Yi, C. Wang, Y. Wang el al., "A survey on evaluation of large language models," ACM Transactions on Intelligent Systems and Technology, vol. 15, no. 3, pp. 1-45, 2024.
Cobbe, K., V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano et al., "Training verifiers to solve math word problems," arXiv preprint arXiv:21 10.10 14168, 2021.
Cohn, A. G. and J. Hemandez- "Dialectical language model evaluation: An initial appraisal of the commonsense spatial reasoning abilities of LLMs," arXiv preprint arXiv:2304.1 1164, 2023.
Dai, A. M. and Q. V. Le. "Semi--suprrvii sequence learning," in Conference on Neural Information Processing Systems, 2015.
Frohberg, J. and F. Binder, "CRASS: A novel data set and benchmark to test counterfactual reasoning of large language models," arXiv preprint urXiv:21 2021.
Gan, W., Qi, Wu, and Lin, "Large language models in education: Vision and opportunities," in IEEE International Conference on Big Data, 2023.
Gandhi, K., J.-P. Fränken, Т. Gerstenberg, and N. Goodman, "Understanding social reasoning in language models with language models," Advances Neural Information Processing Systems, vol. 36, 2024.
González, J. and A. V. Nori, "Does reasoning emerge? Examining the probabilities of causation in large language models," arXiv preprint arXiv:2408. 08210, 2024.
González, J., C. Wong, Z. Gero, J. Bagga, R. Ueno, I. Chien, E. Orakvin, E. Kiciman, A. Nori, R. Weerasinghe, R. S. Leidner, B. Piening, T. Naumann, C. Bifulco, and H. Poon, "TRIALSCOPE: unifying causal framework for scaling real-world evidence generation with biomedical language models," " arXiv preprint arXiv:2311.0133 2023.
Halpern, J. Y. and J. Pearl, "Causes and explanations: structural-model approach. Part I: Causes," The British journal for the Philosophy of Science, 2005.
Howard, J. and S. Ruder. "Universal language model fine-tuning for text classification," in Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2018
11

---

Published as a conference paper at ICLR 2025
Huang, J. and K. C.-C. Chang, "Towards reasoning in large language models: A survey," in The 6lst Annual Meeting of the Association for Computational Linguistics, 2023.
Jin, Z.,, Y. Chen, Leeb, L. Gresele, O. Kamal, Z. Lyu, K. Blin, F. Gonzalez, M. Kleiman-Weiner, M. Sachan, and B. Scholkopt, "Cladder: Assessing causal reasoning in language models," in Conference on Neural Information Processing Systems, 2023.
Jin, Z., J. Liu, L. Zhiheng, S. Poff, M. Sachan, R. Mihalcea, M. T. Diab, and B. Schölkopf, "Can large language models infer causation from correlation?" The Twelfth International Conference on Learning Representations, 2024.
Kandpal, N., H. Deng, A. Roberts, E. Wallace, and C. Raffel, "Large language models struggle to learn long-tail knowledge," in International Conference on Machine Learning. 2023.
Kelley, D., The Art of Reasoning with Symbolic Logic. WW Norton Co, 1992.
Khalifa, M., H. Elsahar, and M. Dymetman, "A distributional approach to controlled text generation." arXiv preprint arXiv:2012.11635, 2020.
Khandelwal, U., K. Clark, D. Jurafsky, and L. Kaiser, "Sample efficient text summarization using single pre-trained transformer," arXiv preprint arXiv:1905, 088366 2019.
KIcıman, E., R. Ness, A. Sharma, and C. Tan, "Causal reasoning and large language models: Opening new frontier for causality," arXiv preprint arXiv:2305.0003. 2023.
Korbak, T., H. Elsahar, Kruszewski, and M. Dymetman. "Controlling conditional language models without catastrophic forgetting," in International Conference on Machine Learning. PMLR, 2022, pp. 11 499-11 528.
Korbak, T., H. Elsahar, G. Kruszewski, and M. Dymetman, "On reinforcement learning and distri- bution matching for fine-tuning language models with catastrophic forgetting." Advances in Neural Information Processing Systems, vol. 35, pp. 16 203-16 220, 2022.
Lewis, D., "Causation," Journal of Philosophy, vol. 70, pp. 556-567, 1973.
Li, A. and J. Pearl, "Probabilities of causation with nonbinary treatment and effect," " in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 18, 2024, pp. 20 2045-20044 472.
Li. J. L. Yu, and A. Ettinger, "Counterfactual reasoning: Do language models need world knowledge for causal inference?" in NeurlPS 2022 Workshop on Neuro Causal and Symbolic Al (nCSI), 2022.
Li, J., L. Yu, and A. Ettinger, "Counterfactual reasoning: Testing language models' understanding of hypothetical scenarios," in The 61 Annual Meeting Of The Association For Computational Linguistics, 2023.
Li, Y., W. Tian, Y. Jiao, J. Chen, and Y.-G. Jiang, "Eyes deceive: Benchmarking counterfactual large language models," arXiv preprint wrXiv:2404.12966 2024.
reasoning abilities of
Lin, S.-C., L. Gao, B. Oguz, W. Xiong, J. Lin, W.-t Yih, and X. Chen, "Flame: Factuality-aware alignment for large language models," arXiv preprint 45X5v22405.01525, 2024.
Lin, V., E. Ben-Michael, and L.-P. Moreney, "Optimizing language models for human preferences is causal inference problem." arXiv preprint arXiv:2402. 2024.
Maasch, J. R. M. A., A. Hüyük, X. Xu, A. V. Nori, and J. Gonzalez, "Compositional causal reasoning evaluation in language models," arXiv preprint arXiv:2503.045: 04555500.0455.... 2025.
MacColl, H., "Symbolic reasoning," Mind, vol. 6, no. 24, pp. 493-510, 1897.
Mackie, I.., "Causes and conditions," American Philosophical Quarterly, vol. no. 4, pp. 245-264, 1965.
MD Anderson Cancer Center,, Cancer treatment algorithms breast cancer invasive stage i-iii. [Online]. Available: http://wwwwww.eeoooooo/////nn/na www.yyyyicaaalleeeiimmmmmmeeeeee breaeemeeeeeettttttteeeeeeeeeese -nw-ssie--weeeb-aggorttm
12

---

Published as a conference paper at ICLR 2025
Nguyen, V. B., P. Youssef, J. Schlötterer, and C. Seifert, "LLMSs for generating and evaluating counterfactuals: A comprehensive study," arXiv preprint arXiv:2405.00722 2024.
Orrantia-Borunda, E., Anchondo-Nuñez. L. E. Acuña-Aguilar, F. O. Gómez- -Valle,, and C. A. Ramírez-Valdesping "Subtypes of breast cancer," Breast Cancer [Internet], 2022.
Ouyang, I., J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, Schulman, J. Hilton, Kelton, L. Miller, M. Simens, A. Askell, Welinder, P. F. Christiano, J. Leike, and R. Lowe, "Training language models to follow instructions with human feedback," Conference on Neural Information Processing Systems, 2022.
Parmar, M., N. Patel, N. Varshney, M. Nakamura, M. Luo, Mashetty, A. Mitra, and C. Baral, "LogicBench: Towards systematic evaluation of logical reasoning ability of large language models," in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2024.
Pearl, J., "Probabilities of causation: Three counterfactual interpretations and their identification," Synthese, vol. 121, no. 1-2, pp. 93-149, 1999.
Pearl, J., Causality: Models, Reasoning, and Inference. Cambridge University Press, 2000.
Peters, M. E., M. Neumann, M. lyyer, M. Gardner, C. Clark, K. Lee, and L.. Zettlemoyer, "Deep contextualized word representations," in Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume 1 (Long Papers), 2018.
Radford, A., K. Narasimhan, T. Salimans, and I. Sutskever. (2018) Improving language understanding by generative pre-training. [Online]. Available: httpss//s3--s- https:/kS-usswest-2aamaae www.-wtaaaaaaaawsss
Rafailov, R., A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn, "Direct preference optimization: Your language model is secretly reward model," Conference Neural Information Processing Systems, 2024.
Reddy, M. J. B., P. Gopakumar, and D. Mohanta, "A novel transmission line protection using dost and svm," Engineering Science and Technology, an International Journal, vol. 19, no. 2, pp. 1027-1039, 2016.
Seals, S. and V. Shalin, "Evaluating the deductive competence of large language models," Proceed- ings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume I: Long Papers), 2024.
Sezgin, G., M. Apaydın, Etit, and M. K. Atahan, "Tumor size estimation of the breast cancer molecular subtypes using imaging techniques," Medicine and Pharmacy Reports, vol. 93, no. p. 253, 2020.
Simmering, P. F. and Huoviala, "Large language models for aspect-based sentiment analysis," arXiv preprint arXiv:2310. 18025, 2023.
Tian, K., F. Mitchell, H. Yao, C. D. Manning, and C. Finn, "Fine-tuning language models for factuality," arXiv preprint arXiv:231 1.08401, 2023.
Tong, Y., D. Wang, Y. Wang, Teng, and J. Shang, "Can Ilms learn from previous mistakes? investigating Iims' errors to boost for reasoning," arXiv preprint arXiv:2403.200446 2024.
Toshniwal, S., I. Moshkov, S. Narenthiran, D. Gitman, F. Jia, and I. Gitman, "Openmathinnstruttl:: A 1.8 million math instruction tuning dataset," arXiv preprint arXiv:2402.10176, 2024.
Wang, 7., "CausalBench: A comprehensive benchmark for evaluating causal reasoning capabilities of large language models," in Proceedings of the 10th SIGHAN Workshop on Chinese Language Processing (SIGHAN- 2024.
Wei, J., X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou al., "Chain-of- thought prompting elicits reasoning in large language models," " Conference on Neural Information Processing Systems, 2022.
13

---

Published as a conference paper at ICLR 2025
Wong, C., S. Zhang, Y. Gu, C. Moung, J. Abel, N. Usuyama, R. Weerasinghe, B. Piening, T. Naumann, C. Bifulco, and H. Poon, "Scaling clinical trial matching using large language models: case study in oncology." in Machine Learning for Healthcare Conference, 2023.
Wu, T., M. T. Ribeiro, J. Heer, and S. Weld, "Polyjuice: Generating counterfactuals for explaining. evaluating, and improving models," in Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing ( (Volume 1: Long Papers), 2021.
Wu, Z., L. Qiu, A. Ross, E. Akyürek, B. Chen, B. Wang. N. Kim, J. Andreas, and Y. Kim, "Reasoning or reciting? exploring the capabilities and limitations of language models through counterfactual tasks," arXiv preprint arXiv:2307.0224.
2023.
Zhang, C., S. Bauer, P. Bennett, J. Gao, W. Gong, A. Hilmkil, J. Jennings, C. Ma, T. Minka, N. Pawlowski et al., "Understanding causality with large language models: Feasibility and oppor- tunities," arXiv preprint arXiv:2304.05524, 2023.
Zhang, C., D. Ippolito, Lee, M. Jagielski, F. Tramèr, and N. Carlini, "Counterfactual memorization in neural language models," in Conference on Neural Information Processing Systems, 2023.
Ziegler, D. M., N. Stiennon. J. Wu, T. B. Brown, A. Radford, D. Amodei, P. Christiano, and G. Irving, time-tuning language models from human preferences," arXiv preprint arXiv:1909.08593, 2019.
14

---

Published as a conference paper at ICLR 2025
A BREAKDOWN OF THE RESULTS IN SECTION 5.3
Table 2: Results of Healthcare, Engineering, and Math Benchmarking kinn problems. For some scenarios in Math Benchmarking. N-IR and AS-IR are equal to 0.00 for all algorithms because the target cause X is never present without an intervention due to how these scenarios structured.
Correctness
Causal Consistency
Secnario
F-ER
C-ER
Avg ER
N-IR
S-IR
AN-IR
AS
Avg IR
Rose
23.57(0.00)
28.93/0.00)
26 20.25(0000)
20/62/0.00)
4.34(0.00)
11.47(0.0)
32,0100.00)
17.1100.00
SFTT-OnlyF
3,12/0.04)
30.8200.22
11.9700.0)
1.500.01>
2.09(0.02)
19.95/0.1) 300
6.6((001))
DPO-Only
10.77(188)
28.52(2,07)
19.671.941 941
9.10(222)
4.76/0.10)
6.000.16)
25.32(1.10)
11.29(0.66)
In-domain
SPT-FCCF
1,64/0.01)
0.0810.00)
0.6600000)
0.8500.00)
0.8000.00
0.3000000)
0.8700000)
0.84(0.00)
DPD-F&CY
8.2411.99) 12554.04)
10.633.14)
8.653.04)
4.3200.00
5.000.25)
11,3744,99)
2.26:1.58)
DPO
9.55(111))
5.16/0 15)
7.3660.53)
7.97(1.21)
1,70(000)
3.50/0.04)
9.31(1911)
5 5.64(0.3))
Breat Cancer Treatmert
SFT--OnlyF
38.82(0.11) 48.86(3.27)
(2.58(2.87)
10,5500110 10)
(306(0011
45.71(2.06)
15006))
DFoyOnlyP
67.95(0.19) 69.56(1993)
66 66.2510770 70)
53,011.45)
13 (3.20)0006
14 (4.25(0.00)
64.770055)
36.3110.26)
Healthaare
Com- Couss
SFT-F&CF
25.990.55)
38, 140.81)
$2.07:0.07) 07)
25.8300.35)
4. 4.1000004
13.0700.01
41,4200.36)
21.10:0.06)
DPD.F&CF
53.2222455
63.6244.52)
58.772.10 (0)
46.270.44)
14.090.001)
14 14.44(
53.232.34)
32.010.333
DRO+CCF
80.18(4.78)
76. 22
1.55)
70.55(1.67)
15 96(0.03)
93(0000
68.67(3667
42,53(069)
SPT-
3,1600.04
12.00000)) 20.8700.02
14500.01>
2.210.02)
25400.00)
20.0640.01
6.6710.11)
DRO-DnlyF
5.550.02)
11.7522244
3.77(0.00)
4.000.00
12)
Com- Efficct
SPT-&&CF
1.400.01)
20.4340.02
109400000 00)
16000,00
0.530.000
14000.00)
19.60/01113
80I0
DROFFECF
2.900.03)
02)4
127000.00
1.000111
2,080.03)
4.140.01)
14/0400000)
6.8G)000)
DFO+CCF
2,17(0.04)
22.78/0.11)
12.48(0.07)
0.3660.00)
2.58/0007)
3.1000.05
19.5000.00)
6.4210.02)
SPT-OnlyF
1.500.01)
20.3010.01)
105960.01)
1,1400.00)
0.6SI0DD)
1.400.013
20.12/0.00)
5.8510.00)
Deductive
DRO-OslyF
18.52(4.41)
32,93(4 16
25.77/4 14)
130822.76)
6, 6.52(0.21)
6.000 20)
10. 202.82)
14.16(1.10 11)
(Cunse-Raval)
SPTRRACF
D&O-
1.32/0.00)
4,25(0.14) 24.37/0.14)
22.
14.3100.22
11 (1.54(0.0)
144:0.04)
1.280.000
3.2100.09)
0.15(0.00)
5.990.09)
3.700.00)
20/4000000)
20.36/0.03)
7.75(0.04)
6.24(0.00)
DPD:CCF
5,05/0.04)
25.1710.17)
15.1100.07)
2.950.03)
3.970.077)
6.200.00)
20.12/0.00)
8.3600.22)
16.3900.00)
21.75 27.10/0.00)
13.7000.00)
27. 77.
2
13.9600.00)
14.4700.00)
SFT-OnlyF
2,94/0.05)
29. 26.64)0.06)
16 29/000011)
4.840.02)
26.
2.16:0001)
8.6700.00 00)
DPO:Oalyy
3.75/0.85)
31,3010.06)
17.5900.0)
66600.11
26.4210.00)
10000.000
3,25/0.55)
9.32(0.0)
In domain
SFT
1.3900000
6.4310.05)
3.9100011)
4.32/0.03 3.270.04)
0.6600.00)
0.930.00)
2.25(0.00)
DRO-F&CF
33,4811 10.23/2.0))
15)
6.340.290 2116611.76)
30.13/0.77)
S.3411.25)
6.870.82)
12.32/0.72) 72)
Thansniscion Line Profection
DRO+CCF
7.64(0.89)
23.00/0.60)
15.33/0.71)
6.610.12)
(9.61(0.77)
4.4.0.32)
4.03:0.37)
8.67(0.32)
SFT--nnlyP
14.61(0.05)
30.28/0.11 17)
22.4N0.11 ID)
13 83,00.005)
27. 27.2310.00)
0.8L(0.00)
14.02/0.07;
13 (3.66/0.02)
DRO-OnlyF
11.2500.1)
30,7840.200
21 02/0 18)
11.04/015)
26 60/0.00)
0.210..00)
11.75/0.16)
12.39(0..4)
Enginoring
Com Couse
SPTE&CF
14,05/0.10)
24.4310.15)
24.2440.2)
13.130.05)
32. 18.6410.88)
0.9200.01)
13.800.06)
15.0710.05)
DRO-F&CF
12.52/0.17)
31.46/0.10)
999990011
10.54/0.22)
31 11.4))212)
1.730005)
11.41(0.11
DPO+CCF
14.2000.09
35.72(0.00)
266600000)
13.2600.07)
35 68(0.00))
09400.01)
13.560.006
(3.8000.22)
SFT-OnlyF
2.71/0.05)
29.26(0.04
159990.01
4.390..33)
26.73(0.00)
0.700.011
2.07/0.02)
8.49(0.11)
DPO-OnlyF
2.710.05)
29. 29.2610.04)
05.9000001
4.390.03
26 73.000
0.880.011
2.07:0002)
8.49(0.01 (1)
Com Effect
SPFR&CF
0.95/0.00)
26.4110.00)
13.880.00)
0.820.000
26.4110.00)
0.1560.00)
0.80X0.00)
7.0410.00)
DPO-
2.28/0.09)
24,30( 84)
18:29/0.15)
300460.10)
25.7110.02)
0.46(0.00)
8.74(1.32)
9.1910.13)
DPO
1,51(0.00)
32.
(7.25(0.49)
2.4100.04)
26, 26,55/0.00)
0.2300.00)
6,50(1.25)
8. 8.92(0.11)
SPT-OnlyF
31.10/0.00 2,580012)
16 16.84(0.00)
5.850.07)
26 26.41(0.00)
0.57(0.00)
2.58/0.05)
8.80(O.01)
Dnl-OnlyF
2.680.05)
31,36(0.07
17.0200.0 (0)
6.2000000
25 69(0.0))
04660.00)
2 2.8900066
8.81(0.00)
Inductive
SFT-F&CF
20.2500111 1.450.00)
20. 0.910.000
0.53/0.0
5:72/0.01 0.9100000
DPO
2.8500.02)
22.2310.69)
12.5400 22)
2.530.033
22.2710.67)
0.3300.00
2.56/0.03)
6,9210.07)
DPO#CCF
2.34/0.00)
26.2310.01)
14 (4(002)
1.37(0.11)
26.03(0.01)
10500.04)
1.47/0.01)
7.48/0.01)
Dase
47.35(0.00)
26.32(0.00)
36.8400.00)
0.030.90)
61. 61.50/0.00)
48.2400.00
0.0000.00
27.4010.00)
Math Benchmarking
Onyy
DRO-OslyF
37,21(0.58)
28.25/0.55)
28.9110.08)
57.39(3.22)
33.0600 25)
42.6311 54)
0.000.00)
0.000.000
66. 66.84().15)
52.1710.41)
40 10.0.99
20 7860.99)
0.000.000
0.000.00)
24.15/0.24 24)
23.0710.14 14)
In In-domain
SPERECF
22.5200.18)
14.05/0.17)
18.29 (8.277
(4K00.00)
32.07(0.22)
25 15
(.000.00)
13 13.8800.05)
Texting on. S
DRO-F&CF
14,0600100 26.87(1 71)
20.4700.33
0.000.000
37. 78..58
9000077
0.000.000
13 13.42(0.07)
DPO-CCF
19.38/0.04)
26.1340.44
27.75(0.19)
0.000.000
30.89(1.13)
21.03(0.01
0,0000.00)
17. $7.9610.09)
Only
37.25(0.78)
35.3640.17 33.4610.02)
0.000.000
53.3910.46
(2062)
0.0000000
23. 03.8810.1
DPO-OnlvF
15.31(0.27)
42.9310.013
12/0 10)
0.900.900
$2.54(0.31)
18.060028)
0.9000.00)
17.6510.07)
Inductive
SPTEFCCF
24.27(0:08)
27.1310-20)
25.000.04
00000300
40:0810.05)
1000111
16.9900.22) B.000.00)
DPO-F&CF
16.13(1.16)
26.86(1 39)
21 21 31)
00000.000
38 28(0.95)
44(1
0.000.000
14.1800 18)
DRO-CCF
22.21(1.355
29.94( 19)
26.07/0.22 34)
0.000.00)
48 44(14()))))
23. (1(116))
0.0000.00)
17.99(0.23)
Rose
50.2500.00)
51.1100.00
50.6800.00)
0.000.000)
67. 30(00000)
S% 65(0000)
0.0000000)
31.49(0.00)
Moth Benchmurking
Teolin on. S -x Rt.
SFT-OnlyF
DPO-OslyF
45.1600.01)
39.11(0.06)
56 36.90/0113 13)
72. 45)
55 700 16)
0300000.0)
0.000.00)
00000.000
68. 02(0.22)
76 30(0.06)
40 07(0
52 95(0.05)
0.000.000
D.000.00
29 22.0000)
30.24/0.000 00)
In domain
SFT
46.5500.11)
32.8610.99
38.7600.1)
0.000.900
62, (81000000)
47.15
0.000.000
27.3310.00)
DRO-F&CF
15.43(3.99)
6.5600600
11.99(199)
0.000.00)
20.06/5001)
18(399
(.000.00)
9,14(1.12)
DPO-CCP
17.12(1.78)
8.060087)
12.99(126)
0.030.900
23 .M14.44
0.300.000
10.3410.75)
SPT:OnlyF
46.17(0:13)
42.4610.18)
44.31(0.05)
66.3100.04
40(0.10)
(.KY0.00)
28, 28.9200.00)
Deductive (Effect-Bosed)
SFT
DPO-OnlyF
39.5640.88) 78)
46.5700.04
48 962- 24)
48.58(
47.57/0
1660 38)
0.000.000
0.000.000
65 (38870041)
66.0510.100 (0)
$2 63(005)
21(1.15)
0.000.000
29.67/0.01) 0.0000.00
(7.51(0.06)
DPO-F&CF
39.500..00
61 61)
50 1.5
0.000.000
43.43(0.47)
39 (1.000))
0.0000.00
28, 26(0000)
DPD+UCF
31,05(0.96)
30.2714.58)
$0.06(1553)
0.0000.00)
63 63.75(2.23)
52 52.9200000
(.0000.00)
24.1710.34)
Base
47,3500.00
58.8200.00
53.0800.00)
28. 77.770000
50, 10.000
25 69/0.00)
22.44(0.00)
31.76(0.00)
SFT-OnlyP
36.07(0.97)
63.9 63.9610.02)
51.010 30)
21.32/0.09)
54.1100.2)
26 27)
9.970.255
28, 23.0010.15 13)
Math Benchmarking
Texling on: H. + ?
DPO-OnlyF
28.28/0.95)
22.110.12)
25
32.5000.31)
28800.013
0.400.000)
24 28.460.92)
15.9710.1
In-domain
SFT-FACF
23.12.0.12)
45.03/01131
16.8700.02)
39(0.09)
07
19.9200.04) 4.760.044
DRO-F&CF
24.33(2.42)
23. 16.
23.35/0 47)
30.27/0.00
6.1100455
3.400466
21.18(1.63)
15.32/0.39 39)
DPO+CCF
21.0340.05)
20.91(0.55)
20.970.161
27.75(0.11)
5.52
08300.01
20.55/0.06)
13.67(0.03)
Onl-
37.870.81)
62.75/0.10)
50.310 36)
51.6300.7)) 23.38/0.11)
26 27
12.13(0.25)
28.3510.14 14)
(Cusse Based)
Deductive
SPTE&CF
DRO-OalyF
DPO-F&CF
25.41(0.10)
24.35(1.01)
15.51(0.27)
53.6310155
19.98/0.03)
38.6266 13)
39 39.52(0.05)
51 (8(.44))
17.65/0.06) (6)
23.77(0.25)
29) 1600.35
19.12/0.04)
22. 646445595 (39
44.40:0.05)
4. 4.040.02)
18 22(0007)
06600.000
(12400000
15.61(0.28)
24.69(1077)
7.35/0.12)
22. 22.3210.02)
10.91/0.06)
10. 20.29 08)
DPC-CCF
15.97/0.425
453.76)
28.21(1445
23.140.25)
25. 25.35(3.48)
1.500.02)
14.850056)
16.3110.46)
15

---

Published as a conference paper at ICLR 2025
B NUMBER OF CONTEXT SAMPLES VS. QUESTION-ANSWER EXAMPLES
When collecting datasets, we always sample the same number of context variables U across all feedback types, whether it be OnlyF, OnlyCF, F&CF or CCF. This is with the exception of Section 5.1 where we double the number of contexts for OnlyF and OnlyCF to end up with the same number question-answw examples as other methods. In other words, for those experiments, F&C has access N context samples with two question- answer examples for each context sample (one factual, one counterfactual). Meanwhile, OnlyF and OnlyCF have access to 2N context samples with a single question-answer example for each context (either factual or counterfactual).
This means, for the remainder of the experiments, F&CF has more question-ansye examples in its training dataset, but crucially, each method still sees the same number of problem instances. For example, in the healthcare domain, OnlyF and F&CF sees the same set of breast cancer patients (each patient corresponding to context); the training data for F&CF does not include any new patients; and the additional CF feedback is provided only for the existing patients. Therefore, the comparison between OnlyF and F&CF remains fair, which would not have been the case if the context variables for the factual dataset and the counterfactual dataset were sampled independently.
In Section 5.1, we preferred the setup with the same number of question-answer examples across methods to be absolutely certain that any improvement we see is due to the type of feedback being provided. In Sections 5.2 and 5.3, we preferred the setup with the same number of context variables across methods because this is likely to be the case in practice (for instance, if we apply these methods in healthcare setting, the number of patients would be constant for each method).
That being said, for completeness, we report here supplementary generalization results where OnlyF has access twice the number of context variables, hence the same number of question-answw examples (as in Section 5.1). We call these new baselines OnlyFx2 to differentiate them from the original setup, see Table 3 for a detailed breakdown and Table 4 for a summary.
Table 3: Detailed breakdown of the generalization results for SFT-OnlyFx2 and DPO-OnlyFx2.
Correctuess
Causal Consistency
Scenario
FER
C-ER
Avg- Dvg-ER
N-IR
S-IR
AN-IR
AS-IR
Avg-IR
Base
23,5700:00)
28.93/0.00)
26 25025())00))
20.62/6:00)
4,34(0.00)
11.47/0000)
32,0100.000
17.1100.00)
Treamest
In-deettinn
SFT-DblyP22
DPO-OnlyF32
5.18(0.3)
3.0510.01
21 21.43/0.01)
24.700066)
12.25(0.01)
14.54(0.45)
2.39/0.01)
4.21(0.40)
2.7500.05)
1.1500.00
5 5.090.261
232000013
21,06/0.011
21.68/0.13) 13)
6,866(01)
8.400 19)
Bealtheare
Com Cuuse
SFT-OnlyFx2
51.481157)
26.4820.66)
28.97(1011
24.840079)
7.850.21)
9.75(0.21)
34.450042)
19.2340.35
76.27/4144)
68.7400.09)
72.5100999
65.14/3.45)
13.270.00
14.69/0000)
70.4300.61
40.5800-55)
Com-Fiffcet
SFT-DhlyP52
DRO-OnlyFs2
2.5810.01
2.230.002
21.35/0181))
22.43(0.5)
12.3500.8)
11,82/0011
2.010.0113
1,2300.01)
1.0000000
(.800.00)
34400.03)
2.580002)
20.54/0.000
19.7600.00
6.54(0.0))
6.560001)
Dedicctive
SFT-OnlyFx2
2.260.000)
22.260000))
12.36/0.11)
1.9500.00)
0.630.000)
1000022
20.9100.00
6,550000)
(Cuuse Based)
Dnly
337(0.03)
22.990.32)
13.280000)
166/0.011
2.6700322
440001L
20.30/0:021
7.27(0.00)
16.3900:00)
27.1000.000
21.35(0000)
13.7000.00)
5225200000
13.96/1.000
14.47(0.00)
Enginerrig Line Protecioon
In-domtsin
DR0-OnlyF32
SFT-Dhlyk22
4.2100.06
0.42(0.00)
30. 13.000)
31.650802
17.1700.11
160050000)
5.59H0.01)
5.34(0.00)
20.4000000
26.770.01)
0 09/00.33
3. 3.49(0.02)
0.4006.00)
9.18/0011
Com Cuuse
DP0-Unlybx2
SFT-OnlyFx2
13.220.02)
36.3000000
33.530001)
35.8000)
26.0000.00
25.5300000)
13.2900.055
18.0000000
26.00(0,00
26 86/0.00
O 2400L00
04710000
15.1500.00
66,0500.01 000
13,530000)
(4.020000)
Com-Elfict
SFT44iyyS2
DPO-OnlyFs2
0.4510.00)
30100.09
29.51(0..00
32.3821.00)
16.3 (6.0))
6:110-00
5.130.088
26.4000.000
26 660
O 4400.00
2. 2.690.099)
0,44/0.00)
0.28(0.00)
8.74.0.02)
Inductive
SFT4hdyy52
DPO-OnlyFx2
09310000)
1.720.01)
32.55(1.00)
31. 18.98.0000)
16.750000)
16.500000))
6. K1100.00)
6.510-000
26 3740000
26. 52H0,00
O 0860000
O A30OL00
0,75/0.01
16800.01)
0.50(U.00)
B. 0.5
Base
47.35/0:00)
26.32/0.00)
30.84(0000))
61 500000))
4% 2400U00
27.44(0.00)
In dostain
SFT-OnlyTx2
DPU-Unlybx2
24.4710.41)
26,5910.38
26.57(0.01)
48.01(1330)
36.24(001)
36.56/0.1 (3)
0.0000.000
42 85/0.13
59, 290290
28 28.08003399
28 33/007747>
0.00X0.00)
0.00115000
21 21.54(0.44)
17.50K0. 10)
Math
Inductive
DP0-OnlyPs2
SFT-Dhlye22
22 22.300)08)
16.22(.55)
28 28.04400.055
46 46.6522011
25.17(0.05)
31,6911.0
0U000.00
0.0000000
401630100)
52.9911611
18. 49(1.333
0,0000.00)
16. 16.1400.22)
17.87H0.37)
Base
50.2000000:
51,1160.00)
50,6800.00)
0.0000000
67.3000.00
SS. 65.6500.00
0,0000.00)
31,49.0.0)
la domain
D8044alyP32
SFT-OnlyPx2
35 35.9510.36)
39.06/0.84)
56 36.060008)
60.64:1.50)
4935(0.56)
66.15 (6)
0L0000.00
0.0000.00
69.2400.99
61.39.0.16
45. 45.74)
44.1913.08)
0,6000.00)
28.3640009)
26.7840.06)
Marh
Dedictive
SFT-DulyP22
45.920001)
45.91:0121)
05.04(0)0))
66.490005
50.64/0.07)
20)2540000)
(Eiffect
DR0-OnlyF52
24.75(266)
52.65(2.00)
38.72(1
0.0000.00)
47(44.66)
0.0000.00)
22.2440.55
47.3510.00:
38.62/0.00)
5300600000
28.7700000
50.11K0.00)
25. 69(0.00)
22.44(0.00)
31,7660000)
domainn
DP0-CalyyF32
Sny-OnlyEx2
24.47(0.41)
7.8011.14)
64.0300.55
24.54/0.15)
24.50(0008)
92/0..3)
19.4300.17
29.80/0.211
7.4000.44)
51,9400000
20.630026))
1.740000))
23.25(0.64)
7.58/0.36)
24.9000.17)
15.55(0.4))
Math
Diductive
SFT-ChiyPR2
22 22.8400:9))
59 (9.30)
41.37(0007)
17.28/0:011
49.08(0.05)
09.490055)
5.84/0.03)
22.4100.0)
(Canse-Based)
D80-OnlyF32
16.72(1.55)
22. (2..77)
19.79(0.70)
24.01(0.63)
6.87.0.16 16)
1.1500.11
15.90(1.9)
11 11,96/0 27)
16

---

Published as a conference paper at ICLR 2025
Table 4: Average generalization performance, including SFT-OnlyFx2 and OnO-OnlyFx2.
Base
OnlyF
OnlyFx2
F&CF (Ours)
Mode
Metrie
Base
SFT
DPO
SFT
DPO
SFT
DPO
DPO+CCF
Avg-ER
1.00
0.82
0.86
0.74
0.76
0.42
0.53
0.48
In-Domain
Avg-IR
1.00
0.73
0.72
0.65
0.66
0.44
0.51
0.47
Avg-ER
1.00
1.35
1.75
1.14
1.92
1.17
2.04
Common Cause
Avg-IR
1.00
1.30
1.49
1.07
1.67
1.14
1.42
1.80
Avg-ER
1.00
0.60
0.71
0.60
0.60
0.53
0.66
0.64
Common-Efie Effeon
Avg-IR
1.00
0.49
0.60
0.47
0.49
0.42
0.53
0.50
Avg-ER Avg-IR
1.00
0.86
0.79
0.72
0.81
0.60
0.58
0.69
Inductive
1.00
0.74
0.62
0.58
0.62
0.51
0.50
0.59
Avg-ER
1.00
0.70
0.67
0.62
0.43
0.61
0.58
0.57
Deductive, Cause-B.
Avg-IB IR
1.00
0.62
0.59
0.54
0.40
0.53
0.53
0.51
Avg-ER Avg-IR IR
1.00
0.83
0.83
0.95
0.78
0.90
0.95
0.77
Deductive, Effect-B.
1.00
0.91
0.87
0.82
0.73
0.93
0.89
0.76
C FORMAL DESCRIPTION OF THE ALGORITHMS IN SECTION 4
Algorithm Supervised Counterfactual Feedback
Inputs: Demonstrations D (Px.-r.)- question templates 9, q. answer generator
2: Output: Dataset D = {q.a) of questions and answer pairs
3: ++0
4: Px, E D,NE ,I, do
5: Y, Px;-Y
Sample context, cause, and potential effects
6: g @ t 1544xx10 (u)
Generate questions given context
7:
ar f- Híg. ax) t
b Generate answers given correct outcome
8:
T + To U ((g: a²). (g))
9:
end for
Algorithm 2 Preference-basee Counterfactual Feedback
1: Inputs: Demonstrations D (Px, +Y }. question templates q, q. answer extractor h, target model to
2: Output: Dataset {ig, a) > (q
of preferences over guestion-answer pairs
3: D +
4: for Px, E D.nE NL. do
5:
Y, ~ Px:-V:
Sample a context, cause, and potential effects Generate questions given the context
6:
q q(), 2r + (v)(u)
7:
for E (I,..., ... do
8:
arim] ~ (o(gr), am/m] ~ fo(gr) [m[m] - h(a//m]) 8x m] h(a/(m)) end for
> Collect answers from the language model
9:
Extract outcome estimates from answers
11:
for m E (1...., [1...,,M, M). m² E f1,...,My do
YF = Yx [m] * Vx [m') then ID < D ((g, fmmmm > (g, a/mm)))
13:
b Elicit factual preferences
14:
end if
15:
if Yx
Yx [m] # [m'] then ag/mm) x (46, a anr
Elicit counterfactual preferences
17:
end
18:
end for
19:
end for
17

---

Published as a conference paper at ICLR 2025
Algorithm 3 Preference--ased Causal Consistency Feedback
1: Inputs: Demonstrations D = (Px,-M J. question templates 4, q, answer extractor h. target model & (g/, dj, Gy add) of preferences dialogues
2: Output: Dataset D {19/ dy, Pot ag))
3: D ++0
4: for Px,-x, E DD E fI,..., do
5:
u, N,YYYYYY ~ Px,-
Sample a context, cause, and potential effects Generate questions given the context
6:
@ @+ q(u), 9o t (u)(u)
7:
for E (I....,hu (1,...... ......Merr do
8:
ar/m] ~ to(()) arfm] Eolg, afm)) [m[m] - h(ar[m)) m)) fx ml h(a/(m))
b Collect answers the language model Extract outcome estimates from answers r(m] R(Vx mm), Yx [m]] u), rlm") + (mm') Yx (m']; u) b Compute consistency scores
9:
10:
end for
for wn E (1...., M). m² E (I......M. M. do
13:
if rfr[ 2 r[m'' then
14:
ID <- U ((gn. afmm), Yos as(mi) > (as ag/mm) 9os aglmm)))
Elicit preferences
15:
end if end for
17:
end for
D DETAILS OF THE EXPERIMENTS
In all our experiments, we specifically used the version of Phi-3 mini available on https: / Thuggingfacee co/miiieeeeoooooooooo.... -asssfttPhh----mnnn-18------- context length of 128k and 3.8B parameters, and has been fine-tuned for instruction following. Table summarizes its performance on various reasoning benchmarks in comparison with other language models, as reported in the Hugging Face repository. Overall, Phi-3 mini demonstrates competitive performance on these benchmarks.
This version has
Table 5: Reasoning performance of Phi-3-Mimi-128K-N in comparison other language models.
Benchmark
Phi-3-Mini-I286e
Gemma-7B
Mistral-7B
Mixtral-&x7B
Llama-3-BB-Ins
OPTAA--Turbo-1
ARC Challenge 10-shot
85.
78.3
78.6
87.3
82.8
87.4
RoolQ (--hot
77.1
66i
72.2
76.6
80.9
79.1
MedQA 2-shot
56.4
49.6
62.2
60.5
63.4
OpenBookQA 10-shot
78.8
78.6
79.8
85.8
82.6
86
PIQA
78.1
GPQA O-shot
29.7
29
15
6.9
32.4
29,9
Social
TruthfulQA (MC2) 10-shot
64.8
52.1
53
60.1
63.2
67.7
WinoGrande 5-shot
71.0
55.6
54.2
65
68.8
When collecting datasets, we always sample 100 contexts per causal relationship and generate 10
answers for each question per context (see Table 6). This with the exception of Section 5.1 where we double the number of contexts for OnlyF and OnlyCF to end up with the same number question-answer examples as other methods, and with the exception of OnlyFx2 in Appendix B.
Table Summary of Healthcare, Engineering, and Math Benchmarking problems. report number of context samples for each generalization mode if that generalization made is available within the corresponding problem domain. While we sample 100 contexts for each causal relationship but some generalization modes than causal relationship.
(namely induction and deduction) involve training
Math Benchmarking
Generalization Mode
Healthcare
Engineering
S 5 T
S
+
R
R +
DoDaiii
100
100
100
100
Common Cause
100
100
(n/a)
(n/a)
(n/a)
Common-Effe
100
100
(n/a)
(n/a)
(n/a)
Inductive
(n/a)
200
200
(n/a)
(n/a)
Deductive (Cause-Based)
400
(n/a)
(n/a)
(n/a)
200
Deductive (Effect-Basee
(n/a)
(n/a)
(n/a)
200
(n/a)
18

---

Published as a conference paper at ICLR 2025
In order to obtain enror bars, repeat each experiment five times. The extractor h is implemented using Llama 3 8B with the following prompt:
will give you question and its answer. Determine whether the meaning of the answer is 'POSITIVE' or 'NEGATIVE'. An answer is 'POSITIVE' if it contains phrases like 'yes', holds', 'correct', 'true', or similar affirmations. An answer is "NEGATIVE" if it contains phrases like 'no', 'it does not hold', 'incorrect', 'false', or similar negations. Respond only with one word: 'POSITIVE' or 'NEGATIVE'. Question: '(q)' Answer: "{a}' Is the meaning 'POSITIVE' 'NEGATIVE'?
Similarly, the answer generator H, in the case of supervised counterfactual feedback, is implemented using Llama 3 8B with the following prompt:
will give you a question and the initial word of its answer. Complete the answer starting form the provided word. Respond only with the complete answer. Question: {q} Answer: [No/Yes),
D.1 REASONING PROBLEM: PUZZLE
This hand-crafted puzzle, centered around a candy party, has been used in the experiments conducted in Section 5.1 and 5.2. Based on different generalization modes, we developed three variations of this puzzle, each featuring distinct causal structures:
Structure-1: 1: Bipartite Graph
Anna, Bill, Cory, and Dave are going to party, where the host is going to distribute candies. Anna will happy if she gets least 4 candies. Bill will happy if he gets least candies. Cory will be happy if Anna and Bill are both happy if he gets at least 8 candies. Dave will happy if Anna and Bill are both happy or if he gets at least 10 candies. After distributing the candies, Anna gets {Na). Bill gets {Nu}. Cory gets {Nc}. and Dave gets (ND}
Context:
Factual Question: Is [Anra/Bil/CoryyDavee happy? Be as concise as possible.
Interventional Question: Now, suppose that (Anna/Bil/Cory/Dawa {is/is not} happy regardless of the candy distribution. With this assumption, (Anna/Bil/CoryyIDawe) happy?
Be as concise as possible
Causal Relationships:
A = NA 24
(12)
B No 2 6
(13)
= (AAB) V (No 2 8) D (AAB) V (ND 2 10)
(14)
(15)
Causal Structure:
Anna is happy
Cory is happy
happy
happy
Structure-2: Chain with No Direct Effeet (NDE)
Context: Anna, Bill, and Cory going party, where the host is going to distribute candies. Anna will be happy if she gets at least candies. Bill will be happy if Anna is happy or if he gets at least 7 candies. Cory will be happy if Bill is happy if he gets at least 9 candies. After distributing the candies, Anna gets {. Bill gets (N and Cory gets (Nc).
Factual Question: Is (Anna/Bill/Cory) happy? Be concise as possible.
19

---

Published as a conference paper at ICLR 2025
Interventional Question: Now, suppose that (Anna/Bill/Cory) {is/is not) happy regardless
of the candy distribution. With this assumption, is (Anna/Bill/Cory) happy? Be as concise as possible.
Causal Relationships:
A NA 2 5
(16)
B = A (NB 2 7)
(17)
C B (Ne 9)
(18)
Causal Structure:
Anna is happy
Bill is happy
Cory is happy
B
Structure-3: Chain With Direct Effect (WDE)
Anna, Bill, and Cory are going party, where the host is going distribute candies. Anna will be happy if she gets at least candies. Bill will be happy if Anna is happy or he gets least candies. Cory will be happy if Annd and Bill are both happy if he gets least candies. After distributing the candies, Anna gets {NA}, Bill gets {NB), and Cory gets {Nc}.
Context:
Factual Question: Is I (Anna/Bill/Cory) happy? Be as concise as possible.
Interventional Question: Now, suppose that (Anna/Bill/Cory) {is/is not} happy regardless of the candy distribution. With this assumption, (Anna/Bill/Cory) happy? Be as concise as possible.
Causal Relationships:
A NA 25
(19)
B= AV (NB 27)
(20)
C = (A B) v (Ne 29)
(21)
Causal Structure:
huppy
happy
Cory hupgy
D.2 REAL-WORLD REASONING PROBLEM: HEALTHCARE
In Section 5.3 we introduced three real-world problems to validate our experimental findings from the proof-of-concept puzzle reasoning problem. Here, offer more detailed explanation of one of these real-world reasoning problems-tu the Healthcare problem.
Context: There are four types of breast cancer patients (based their ERPR and HER2 indicators): (1) If a patient is ERPR positive and HER2 negative, they are 'Luminal A', All luminal A patients should undergo surgery. (2) If patient is ERPR positive and HER2 positive, they are 'Luminal 8 Luminal B patients should undergo surgery if their tumor is smaller than 1 em and there no nodal involvement. Luminal B patients should undergo therapy if their tumor is larger than 1 em or if there is nodal involvement. (3) If a patient is ERPR negative and HER2 positive, they are 'Enriched', Enriched patients should undergo surgery their tumor smaller than cm and there is no nodal involvement. Enriched patients should undergo therapy only if their tumor is larger than I cm (even there nodal involvement). (4) a patient is ERPR negative and HER2 negative, they are 'Basal'. Basal patients should undergo surgery their tumor is smaller than cm and there is no nodal involvement. Basal patients should undergo therapy only if their tumor is larger than I em (even if there is nodal involvement). Jane is ERPR [negative/posiitve) and HER2 (negative/positive). Her tumor is (Tem) cm and there is (nodal involvement/no nodal involvement).
20

---

Published as a conference paper at ICLR 2025
Factual Question: Will she undergo /surgery/therapy? Be concise possible.
Possible Interventional Questions: If [Jane had been ERPR positive/Janc had been ERPR neg ative/Jane had been HER2 positive/Jane had been HER2 negative/the tumor had been larger than 1 cm/the tumor had been smaller than cm/there had been nodal tevettmmet/there had been no nodal involvement), would she have undergone /surgery/therapy? Be concise possible.
Causal Relationships:
(1,0)
with probability 0.50
(1,1)
with probability 0.15
HERPR, HHER2
(22)
(0,
with probability 0.20
(0,0)
with probability 0.15
Luminal A
if HERPR A HHER2
Luminal B
if HERPR A HIERZ
Crype
(23)
Enriched
if - HERPR A HHER2
Basal
if - HERPR A HIERZ
Ni 3.07, a = 2.22)
Luminal
N( = 2.96, α = 1..5)
if Luminal
Tom
(24)
N(u 2.42, 1.03)
if Enriched
(N 3.32, 33.64)
Basal
T
(Tem 1)
(25)
Bip 86/251]
if Luminal A
B(p 35/79)
if Luminal B
N
(26)
B(p 18/32) if Enriched if Basal if Luminal A if Luminal B -I N otherwise
B(p 41/99)
1
Y Surgory
-I -
(27)
0
if Luminal A Luminal otherwise
Yiherapy
TVN
(28)
T
Causal Structure and In-domain Fine-tune/Evaluate Relations:
Haase Potient BRPR pocitive
positive
Toree lange than
invivement
Vnewry Perwee dbvald undengn
N
Yiuroy Parint danald undargo cherapy
Causal Structure and Cammon-Caaue Fine-tune/Evaluate Relations:
Hexen Pattett EMIA postiie
Recre
Myena Paatect TIERa
Terrer largertion can
Them is rould insovurene
Yowwry Parrect dhvilt undenge enrgrry
HaLie
Yohwe Patert shvald undergo thenayy
21

---

Published as a conference paper at ICLR 2025
Causal Structure and Common-Effect Fine-tune/Evaluate Relations:
Rassin
Hearn ERIttpotttoo
Harm Paatent EEEl poethve
larger
them is zodal invelvreent
Foeper shoil
Yruos Prisct undego therupy
Causal Structure and Cause-Based Based Deduction Fine-tune/Evaluate Relations:
Harre Patiect's ERPRppoltle
Haru Paliox a BERs pesitive
Tuntoe laggrtien
nodil Insobveuent
Frisco
Fivers Patieer should
Youwyy andergo thamapy
D.3 REAL- WORLD REASONING PROBLEM: ENGINEERING
In Section 5.3 we introduced three real-world problems to validate our experimental findings from the proof-of-coneept puzzle reasoning problem. Here, offer more detailed explanation of one of these real-world reasoning problems the Engineering problem.
Context: The type of fault transmission line is determined through three factors X, Y. and Z. These factors are "close to zero' if they are less than 0.1. (1) If only one of the factors is close to zero, it line-to-line fault. When there is line-to-line fault, it is BC fault if factor X close to zero, AC fault if factor Y is close to zero, and AB fault if factor 7. is close to zero. a line-to-ground fault. When there is line-to-groume fault, it is AG fault if factors Y and Z are both close to zero, BG fault if factors and Z. are both close to zero, and CG fault factors and Y are both close to zero. For some faulty transmission line, X = X, Y Y, and Z = Z.
(2) If exactly two of the factors are close zero,
Possible Factual Question: {Is there a line-to- to--iine/ineetteee toouudd fault? Is the fault type BC/AC/AB//A//G//CCT) Be as concise as possible.
Possible Interventional Questions: factor X/Y/Z had been/had not been close to zero, {would there have been a line-to-line/lne-tto tooguud fault? / would the fault have been type BC/AC/AB/AG/BGICG} Be as concise as possible.
Causal Relationships:
X N(p X, 0.1)
(29)
Y N(p Yll= 0.1)
(30)
Z N(p 2.o 0.1)
(31)
X, (X <.1.)
(32)
Yn (Y 0.1)
(33)
Zu [Z 0.1)
(34)
LL (Xo -Y, A V V-X A 7n)
(35)
LG = (-X, A Yo Zo) VX von (X, - 2n) (X A Yo Zo)
(36)
BC LLA Xo
(37)
AC LLA Yo
(38)
AB LLA 7
(39)
22

---

Published as a conference paper at ICLR 2025
AG LG A Yo A Zo
(40)
BG LG Xo Zo
(41)
CG LG LG
(42)
where X, Y, and are drawn randomly from the values reported in Reddy et al. (2016).
Causal Structure and In-domain Fine-tune/Evaluate Relations:
Factor clase
II.
L1. Line-to-Linee
EXG There is fime-tn-Grrundd fault
BC/AC//BB/M//B0/CCC
Causal Structure and Common- Common-Cause Fine-tune/Evaluat Relations:
Factor vero
close
2g Factor zero
Yo
LG
11. There is Line-tu-Line fuult
DG There is Line-to-Groomdd fault
BC/AC/AB/AG/BGGCCC
BCJAC/AB/AC/ /BC//G
Causal Structure and Common- Common-Effect Fine-tune/Evaluatt Relations:
Xg
Y4 Factor is clase
Yn
Zn
LL
LG
L1.
EAG /'here is Line-to Aa--ono-- fault
BC/AC/AB/AG/BB/CC
There BC/AC/AB/AC/ AAC/A/CCCCCCCCCC fault
Causal Structure and Induction Fine-tune/Evaluate Relations:
Xa
Y1 Factor is close
X
YA
2
LL
LG
L1.
LG TThere is Line to Ground fault
BC/AC/ABB/A/BB//CC:
There BC/AC/AB/AG// /BC/CC
D.4 REAL-WORLD REASONING PROBLEM: MATH BENCIMARKING
In Section 5.3 we introduced three
problems to validate our experimental findings from the proof-of-concept puzzle reasoning problem. Here, offer more detailed explanation of one of these real-world reasoning problems-tee the Math Benchmarking problem.
Context: Carla downloading {Nam)) GB file. Normally she download 2 GB/minute, but in 100 minutes, Windows will force a restart to install updates, which takes (Nminutes minutes. After the restart, Carla resume her download.
Possible Factual Question: (Will Windows force a restart before the download is com- plete?/ Will the download take longer than 120 minutes?) Be concise as possible.
23

---

Published as a conference paper at ICLR 2025
Possible Interventional Questions: If (she were downloading file twice the size / Windows had forced a restart before the download was complete Windows had not forced restart before the download was complete), would (Windows have forced restart before the download was complete? / the download have taken longer than 120 minutes?) Be as concise as possible.
Causal Relationships:
Nsize L100, 300)
(43)
N Minutes ~ L((1, 30)
(44)
Bpp = 0.5)
(45)
N downtoad time [Nsize 2 Nsize S)]/22
(46)
R (N (Nowwtlodd time 2 100)
(47)
T (Nownoood time + R Nminutes) 120)
(48)
Causal Structure and Induction Fine-tune/Evaluee Relations:
She diverlonding file tvize
Winskons donnkead
Comeepan Seappe
Thee dewnbood siber mone than
Causal Structure and Deduction Cause-Based Fine-tune/Evaluate Relations:
Delustics Sarss
doweleeding
R: Windows fore rectort before download completeee
Comovesiian in domiit
The donutoal las lakes mure than nio
Causal Structure and Deduction Effect-Basse Fine-tune/Fvaluate Relations:
Deloctie brlet Setri
St Shew wan deorrlooding file tvize ebs
It: Windons freara notert before the donnlead cempleao
Comepeadingg In-dumain Seapp
E METHODS BESIDES FINE-TUNING
E.1 CHAIN-OF-THOUGHT PROMPTING
Responses of a language model can be improved in different ways, not just through fine-tuning, For instance, one could pursue prompt engineering instead. When eliciting reasoning, our focus has been fine-tuning while chain-of-thouge prompting is form of prompt engineering. These two approaches mainly differ in terms of what point they intervene on: Prompt engineering modifies the inputs passed to the language model whereas fine-tuning modifies the language model itself. Importantly, this means that the use of these two techniques are largely orthogonal to each other and definitely not mutually exclusive. In our case, Sfain-of-thougee prompting not competing method against ours but rather separate avenue for further improvement that could be pursued in conjunction with fine-tuning. Because of this, did not consider it baseline in our experiments.
24

---

Published as a conference paper at ICLR 2025
Notably, when used to elicit reasoning, chain-of-though prompting would be subject to the same evaluation challenge that our fine-tuning approach faces: Examples provided in a setain-of-thougeee prompt might lead to more accurate answers for "in-domain" questions, however, this does not mean the examples was successful in eliciting reasoning unless the accuracy generalizes to new problems (that require forming novel chains using the same concepts).
E.2 FORMAL REPRESENTATIONS AND DEDICATED SOLVERS
Since the reasoning problems we consider are straightforward to solve once translated into formal representation (like the structural equations in Appendix D), an alternative solution strategy could be to extract such formal representations from the natural language descriptions of the problem and rely on dedicated solvers to compute outcomes in an exact manner. However, the extraction of formal representations would remain a bottleneck for performance in such a strategy as off- thf-the-sheff language models are not effective at this task. For instance, OpenMathhnstruuttt (Toshniwal et al., 2024) achieves an accuracy of 84.6% on GSMSK by generating crode-interpreter" solutions, which list the computations performed to arrive at the final answer in formal language. However, this required fine-tuning CodeLlama- 70B to generate responses in the style of these code-interpreter solutions using custom dataset with 18 million problem-to-code examples.
In short, although extracting formal representations (like inte-interpreter solutions) could enable exact computation of answers thereafter, representations is likely a more complex task than fine-tuning the model to provide direct answers. This challenge is especially pronounced for small language models like Phi-3 mini, which has 3.8B parameters compared to the 70B of OpenMathInstruct-11 This makes our approach even more suited the domain of small language models.
a language model to accurately extract such
Additionally, our setting poses an even further complication for the formal-reprseentation route: The reasoning problems we consider are not simple forward computations as in math problems but rather involve causal systems with multiple possible interventions, each intervention altering the forward computation necessary to obtain the final answer. In order to highlight all these challenges, we report qualitative evaluation of Phi-3 mini in terms of its ability to extract formal representations of causal scenarios. We designed the following one-shot prompt using the problem Figure as an example. We then used this prompt to extract structural equations for our other problem domains: Healthcare, Engineering, and Math Benchmarking.
want you to generate structural equations from a causal scenario described in English. Respond only with the structural equations. Here is an example:
Scenario:
Anna, Bill, Cory, and Dave going a party, where the host is going to distribute candies. Anna will be happy she gets at least 4 candies. Bill will be happy if he gets at least candies. Cory will be happy if Anna and Bill are both happy or if he gets at least 8 candies. Dave will be happy if Anna and Bill are both happy or if he gets at least 10 candies.
Structural Equations:
NA: The number of candies Anna gets
NB: The number of candies Bill gets
NC: The number of candies Cory gets
ND: The number of candies Dave gets
HA: Happiness of Anna
HB: Happiness of Bill
HC: Happiness of Cory
HD: Happiness of Dave
HA (NA >= 4)
HB = (NB >= 6)
HC = (HA and HB) or (NC >= 8)
HD (HA and HB) or (ND >= 10)
Scenario:
TTest Scenario]
Structural Equations:
25

---

Published as a conference paper at ICLR 2025
Results for Healtheare.
Test scenario:
There are four types of breast cancer patients (based on their ERPR and HER2 indicators): (1) If a patient is ERPR positive and HER2 negative, they are 'Luminal A. All luminal A patients should undergo surgery. (2) If patient ERPR positive and HER2 positive, they are 'Luminal B'. Luminal B patients should undergo surgery if their tumor is smaller than 1 cm and there is no nodal involvement. Luminal B patients should undergo therapy if their tumor is larger than I cm or if there nodal involvement. (3) If a patient is ERPR negative and HER2 positive, they are EEriched'. Enriched patients should undergo surgery if their tumor is smaller than 1 cm and there is no nodal involvement. Enriched patients should undergo therapy only if their tumor is larger than 1 cm (even if there is nodal involvement). (4) If a patient is ERPR negative and HER2 negative, they are 'Basal'. Basal patients should undergo surgery if their tumor is smaller than cm and there is no nodal involvement. Basal patients should undergo therapy only if their tumor is larger than cm (even if there is nodal involvement).
Response of Phi-3 mini:
NA: ERPR status of the patient
NH: HER2 status of the patient
TA: Tumor size of the patient
TN: Nodal involvement of the patient
SA: Surgery for the patient
TH: Therapy for the patient
LA: Luminal A patient
LB: Luminal B patient
EN: Enriched patient
B: Basal patient
SA = (NA 'ERPR positive' and NH 'HER2 negative'and and TA <1 cm and TN-
NNo nodal involvement')
LB = (NA- = 'ERPR positive'and NH = 'HER2 positive'and (TA <1 cm and TN-
NNo nodal involvement'or TA >1 cm or TN = 'Nodal involvement')
EN = (NA = 'ERPR negative'and NH 'HER2 positive'and (TA <1 cm and TN
No nodal involvement''rr TA >1 cm))
B = (NA= = 'RRPR negative'and NH = 'HER2 negative'and (TA <1 cm and TN
No nodal involvement'or TA > cm))
LA = (NA = 'ERPR positive'an and NH 'HER2 negative')
LB (NA= 'ERPR positive'and NH 'HER2 positive')
EN = (NA- = 'ERPR negative'and and NH = 'HER2 positive')
B = (NA- = 'ERPR negative'ant and NH 'HER2 negative')
SA = (LA or LB or EN or B)
TH = (not SA)
Beyond whether it is accurate not, this response lacks clear formal interpretation, many of the variables are defined multiple times (causing ambiguity).
Results for Engineering.
Test scenario:
The type of fault on transmission line is determined through three factors X, Y. and Z. These factors are "close to zero' if they are less than 0.1. (1) If only one the factors is close to zero, it a line-to-line fault. When there is a line-to-line fault, it is BC fault if factor X is close to zero, AC fault if factor Y is close to zero, and AB fault factor is close to zero. (2) If exactly two of the factors are close zero, it is a tine-to-groum fault. When there is fault if factors Y and Z are both close to zero, BG fault if factors X and are both close zero, and CG fault if factors X and Y are both close to zero.
too-uuo fault, it is AG
26

---

Published as a conference paper at ICLR 2025
Response of Phi-3 mini:
XC = (X <0.1) and (Y >= 0.1) and (Z>=0.1)
YC = >= 0.1) and (Y <0.1) and (Z. 0.1))
ZC = (X >= 0.1) and (Y >= 0.1) and (Z <0.1)
BC.fault = XC
AC fault YC
AB fault = ZC
YZ.clo (Y <0.1) and (Z <0.1)
XZ.close (X <0.1) and (Z <0.1)
XY_close (X <0.1) and (Y <0.1)
AG_fault = YZ.close
BG fault XZ close
CG fault = XY close
This response is accurate except that it marginalizes out the general fault types, namely line-to-line and line-to-ground. While is possible compute factual values for specific fault types using these equations, the same is not true for the general fault types. Additionally, this limitation prevents us from performing any interventions on the general fault types.
Results for Math Benchmarking.
Test scenario:
Carla is downloading 250 GB file. Normally she download 2 GB/minute, but in 100 minutes, Windows will force a restart to install updates, which takes 20 minutes. After the restart, Carla can resume her download. Consider how long it will take to download the file.
Response of Phi-3-Mini:
NA: The amount of data Carla downloads before the restart
NR: The amount of data Carla downloads after the restart
T: Total time to download the file
NA 2 GB/minute 100 minutes
NR = (250 GB NA) / 2 GB/minute
T 100 minutes 20 minutes NR/2 minutes
This response best underlines the additional challenges involved in extracting causal equations. First of all, the calculations provided in the response are not entirely accurate, even for the specific instance of the scenario represented. Setting aside these small inaccuracies, the response assumes that restart will necessarily occur. With these equations, it is not possible to reason about outcomes under interventions such as "What if Windows had forced restart?" "What Carla had been downloading file half the size?" experiments. Even if we were to modify these equations explicitly include whether a restart occurs or not variable, it may not always be possible to anticipate what other kinds of interventions will be queried. For instance, these equations also fail to account for potential interventions on the download speed.
of which are types of interventions considered in our
F DISAMBIGUATION OF "GENERALIZATION"
Generalizing from a few examples to larger input space, potentially involving never-seen-before inputs in the case of 2-4 digit multiplication to 5-7 digit multiplication, is useful and non-trivial property for language model to have. This type generalization remains challenge within what term the "in-domain" problem. In fact, the results in Figure show how DPO+CCF outperforms other baselines in this regard.
While there many notions of generalization, what argue is that reasoning fundamentally involves breaking problem down to its basic components and synthesizing them in novel ways that enable generalization entirely new problems. In causal graph, this may correspond to
27

---

Published as a conference paper at ICLR 2025
understanding of individual edge relations in way that would enable to make estimations for any composite relationship. This is the type of generalization we are interested in. Here, sample efficiency in learning to solve a fixed problem would not necessarily be indicative of reasoning ability.
G AN INTUITIVE EXPLANATION OF OUR INCONSISTENCY RATES
An alternative interpretation of the inconsistency rates we introduce is to frame reasoning about necessity sufficiency classification problem. For instance, focusing on necessity, each context variable U. along with the cause X and the potential effect Y, Yxx induced by U, needs to be assigned to one of three classes:
(i) N: The cause and the effect both occurred together, and the cause was necessary for the effect to occur, meaning the effect would not have occurred otherwise if the cause was prevented. In case like this, one might say "For the observed effect, the cause was a necessary condition."
(ii) N': The cause and the effect both occurred together, but the cause not necessary for the effect to occur, meaning the effect would have occurred anyways even if the cause was prevented. In a case like this, one might say "For the observed effect, the cause was a necessary condition."
(iii) 0: It is not meaningful to talk about necessity as either the cause or the effect did not occur. Note that the previous cases take the occurrence of the cause and the effect as a given, and make statements about what would have happened to the effect if the cause were to be prevented. This is consistent with Pearl's definition of PN as a conditional probability with the condition being X and Y y.
According to the underlying causal model, each context has a corresponding ground-truth "label" as defined in Equation 6 Given a language model, its answers provide estimates for the unknown potential outcomes, which in turn imply prediction of this ground-truth "label". The necessity inconsistency rate (N-IR) is the error rate for this classification task (how often the predicted label differs from the ground-truth label).
H FURTHER ANALYSIS OF THE GENERALIZATION RESULTS
In this section, we expand upon our analysis in Section 5.2, giving more detailed explanations as well as share new observations.
In Figure 7a, we see that common-effee generalization leads to an asymmetric improvement in N-IR and S-IR. This because, in common-effe effecc generalization, the effect of interest is fixed between training and testing, hence the factual questions regarding this fixed effect remain the same well (factual questions do not depend on the cause as there are no interventions involved). Meanwhile, the cause that is being intervened on changes, hence the fine-tumed model now needs to answer different set of counterfactual questions during test time than the ones seen during training time. Therefore, we would expect common-effect generalization to be more challenging in terms of CF-ER opposed to F-ER. Since in our constructed examples, causes tend to be sufficient for effects rather than necessary, performing well in terms of S-iR more dependent on counterfactual question answering while N-IR is more dependent on factual question answering. Most contexts can be dismissed as irrelevant as far as necessity is concerned (i.e. N = 0) by estimating the factual outcome to be negative, whereas determining sufficiency requires determining whether the counterfactual outcome is positive or it is also negative. Therefore, we see greater improvement in terms of N-IR.
Across all generalization modes presented in Figure 7, by far the most challenging one is cause-based deduction with direct effect ("WDE"), where no method strictly improves the base model. This is due to the problem structure: When A affects C both directly as well as indirectly through B, it is not possible to separate the two types of effects without seeing any interventions on the intermediary B. Notice how this stops being an issue for cause-based deduction with no direct effect ("NDE"). Interestingly, we still see an improvement in terms of N-IR (at the cost of worse performance in S-IR). This is similar to the previous example: Seeing demonstrations for A - C at least improves the model's 's ability to answer factual questions regarding C, which translates to better F-ER when reasoning about B C. which in turn translates better N-IR as before (B is sufficient for C' hence
28

---

Published as a conference paper at ICLR 2025
most contexts can readily be dismissed irrelevant far as necessity is concerned just by estimating the factual outcome correctly).
Finally, when there is no direct effect, see that cause-based deduction and effect-based deduction are similarly effective. However, cause-based deduction is slightly better in terms of S-IR whereas effect- based deduction is slightly better in terms of N-IR. This is again the result of each variable being sufficient for the next variable. When reasoning about B + C' (cause-basee determining sufficiency requires estimating the outcome under the intervention B 0 when B - O. Since A is sufficient for B, this intervention can indirectly be performed through the intervention A - However, determining sufficiency requires estimating the outcome under the intervention B = 1 - 0 when B = 1. This can no longer be achieved consistently through A (it could still be the case that B even after the intervention A - 0). Therefore, S-IR the easier metric for cause-based deduction. A similar thing happens when reasoning about A -> B (effect-based deduction). Determining necessity requires observing B 1 + 0 if we perform the intervention A = + 0. After performing A 1 + 0, if we observe C = 1 + 0 instead, we can still conclude this must be because B + 0 as is sufficient for (i.e. B = C). However, determining sufficiency requires observing B = 0 1 after the intervention A = 0 1. Here, it can already be the case that C' = as B is not a necessary condition for B, preventing us to reason about B indirectly through C.
deduction),
I FURTHER DISCUSSION
1.1 WHAT TO Do WHEN CAUSES AND EFFECTS ARE NON-BINARYS
The math benchmarking domain (detailed in Appendix D.4) is good example of how non-binary causes and effects can be handled within our formulation. In this domain, we consider particular math problem within the GSM8K dataset, which is about Carla downloading file with a certain size (denote with Nx) and how long it would take to download this file in minutes (denote it with Ny-). Note that both of these variables would be non-binary. As effect, we consider whether the download takes longer than 120 minutes, that is = 1 Ny > 120 and Y otherwise. As an intervention, we consider what would have happened if Carla was downloading file that is twice the size. For this, we defined X such that X = 1 the intervention has occurred and X = 0 if it has not; and the instevntervnntion file size is given by N = X (2Nx) (1 X) Nx.
Generally, when interested in a non-binary outcome Ny, one can consider binary events dependent on that outcome instead. Similarly, when interested in interventions on non-binary target Nx, one can decide on a particular intervention and let the cause be the binary indicator of whether that intervention has occurred not. We mainly focused on binary causes and effects because the concepts of necessity and sufficiency are the most meaningful for binary variables. However, it is worth mentioning there is also work that aims to extend these concepts categorical variables (Li and Pearl, 2024).
I.2 DEFINING FALSE NECESSITY AND FALSE SUFFICIENCY
We can breakdown necessity and sufficiency inconsistency rates to finer components such as false necessity and false sufficiency. In our framework, false necessity and false sufficiency can be interpreted as the events A NAN + N and $ S S + respectively. However, notice that both of these events are the same event Ý + Y v yx * Yx,, that is either factual counterfactual error has been made. These failures are already covered by factual and counterfactual error rates.
Overall, the individual inconsistency rates we introduce, N-IR, S-IR, AN-IR, and AS-IR, together with the individual error rates, F-ER and CF-ER, are already finer breakdown of aggregated metries like Avg-IR IR and Avg-ER, both of which represent general accuracy (the former accounts for correlations between factual and counterfactual errors whereas the latter looks at them independently).
29


Reviews:
[{"review_id": "9SbSUrLfsi", "reviewer": "Reviewer_uHCM", "summary": "This paper studies causal reasoning in LLMs through counterfactual question answering.\n\nThe authors propose a number of metrics: necessity inconsistency rate (N-IR), sufficiency inconsistency rate (S-IR), absent necessity inconsistency rate (AN-IR), absent sufficiency inconsistency rate (AS-IR), and the average of all those (Avg-IR). These go beyond simple accuracy by assessing the consistency between factual and counterfactual answers within a causal framework.\n\nThey perform a number of experiments where they finetune a small LLM (Phi-3 mini, 3.8B parameters) with SFT and DPO and present results on the extent to which their fine-tuning methods effect different generalization modes (common-cause, common-effect, inductive, deductive).", "questions": "1. Could you provide a more intuitive explanation of the inconsistency rate metrics in the paper?\n2. Can you include the specific architecture and number of parameters of the Phi-3 mini model and discuss the limitations introduced by this choice? Can you include the Phi-3 mini performance on common reasoning benchmarks (compared to what top models are capable of)?\n3. Have you considered comparing your methods with chain-of-thought prompting (see weakness above)?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* Introducing more finegrained metrics: The proposed metrics for causal consistency are a valuable contribution, addressing a limitation of existing evaluation methods that focus primarily on accuracy.\n* Generalization modes:  The proposed classes for generalization modes provide a structured framework for evaluating reasoning transfer.\n* Experiments:  The paper includes a comprehensive set of experiments, including a hand-crafted puzzle and real-world problems.", "weaknesses": "* The choice of a very small LLM (Phi-3 mini) with limited reasoning capabilities makes it difficult to draw any conclusions for stronger models that are more commonly used for reasoning tasks.  \n* Clarity and Presentation:  While the paper focuses on an interesting problem and makes interesting suggestions, the presentation could be significantly improved. I found the formal definitions and descriptions of the methods a bit difficult to follow.\n* Analysis of generalization: The analysis of the generalization results could be more in-depth.\n* Comparison to CoT: The authors claim (in a footnote) that the effectiveness of CoT \"can be attributed to\nsuccessful imitation of the provided examples\". It would have been interesting to test that claim with the analytic tools they developed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KLegZtFu9d", "reviewer": "Reviewer_fqJm", "summary": "This paper is interested in LLM's ability of doing causal reasoning (that follows Pearl's counterfactual methods). For this they propose novel metrics that balance accuracy in factual and counterfactual questions and then propose several fine tuning approaches and evaluate them using their proposed metrics with respect to several examples. They show that their fine tuning methods lead to better performance with respect to their proposed metrics.", "questions": "1. See the question in the weakness part.\n\n2. Are the examples that are studied in the paper original to this paper?  (If so, they are a good contribution.) Or are they published in some other papers cited in this paper. \n\n3. In the Introduction you say: \"When the goal of fine-tuning is specifically to improve reasoning, a unique problem arises in evaluating the fine-tuned LLMs: we cannot just measure performance for a held-out set of test samples within the same reasoning task. If we do, it would be impossible to tell whether the LLM actually learned to reason or whether it is still recalling the demonstrations we have made during fine-tuning.*2* \"\n\n*2* For instance, chain-of-thought prompting aims to improve reasoning by providing examples of how a problem can be solved in smaller steps. While such prompting is effective, its effectiveness can be attributed to successful imitation of the provided examples and is not necessarily the result of true reasoning (Wei et al., 2022).\n\nIn Page 3 you say: \"Modes of Generalization. As we have discussed in the introduction, an in-domain evaluation is not sufficient alone to assess the success of fine-tuning for reasoning.\"\n\nI am a bit confused by these. Consider an analogy. Say I have examples of how to multiple \n2, 3, and 4 digit numbers and using that a model learns to be able to multiple 5, 6, 7 ... digit numbers. Is that a good generalization?\n\nSimilarly your model may learn from a few examples and if it is able to address causality in new examples with a much more complicated graph, and more variables, will that not be a good generalization?\n\nKindly clarify the distinction you are making between in-domain evaluation and generalization, and how your approach differs from the type of generalization mentioned above.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Causal reasoning is an important problem and Pearl's approach of using counterfactuals to do causal reasoning is sound and well studied. There is indeed necessity of more fine grained metrics when evaluating reasoning ability of LLMs and the papers efforts towards that is a plus. \nThe paper presents several fine tuning techniques and evaluate them using their proposed metrics with respect to several examples. The evaluations show that their fine tuning methods lead to better performance with respect to their proposed metrics.", "weaknesses": "The examples that are studied in the paper are very direct in terms of language and LLMs can translate them to a formal representation (LLMs are good at that) and then call a dedicated solver and thus achieve near 100% accuracy. What is the motivation then of doing it the way that is proposed in the paper?\n\nWhy is your approach preferable to using a formal representation and a dedicated solver?\n\nThat motivation must match with the way the examples and evaluation data are presented and studied.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "15nckZ6BVM", "reviewer": "Reviewer_dEY8", "summary": "This paper presents a method for fine-tuning large language models by means of counterfactual feedback. The authors present four categories of causal reasoning (common-effect, common-cause, inductive, and deductive). They also propose two new metrics to measure necessity and sufficiency under a reasoning framework (these metrics measure necessity and sufficiency inconsistency rates), by first presenting an example justification of these metrics. They then introduce three methods for fine-tuning LLMs with counterfactual feedback: supervised, preference-based and causal consistency feedback.", "questions": "- Can you give some hints on how the proposed method could be adapted to the case of non-binary representations of causes and effects?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "I see a lot of strengths in this paper, including: \n\n- The paper presents an interesting framework for fine-tuning LLMs with counterfactual feedback, which is highly relevant and original. \n- The quality of the paper is very good: it is very well structured, organized, and it integrates a multiplicity of concepts, scenarios and metrics for the problem at hand. In this sense, the paper is beyond complete. \n- The significance of the paper appears to be very relevant for researchers in causal reasoning (with or without LLMs).", "weaknesses": "I see very few weaknesses in this paper. Here is a couple of things I would improve:\n\n- Organization of the paper: the placement of related work is a bit odd - perhaps moving it to the end of the paper would help. \n- Conclusion: the limitations in the conclusion feel a bit 'incomplete', in the sense that these leave room open for further research of non-binary representations of causes and effects. Can you give some hints on how the proposed method could be adapted in this case?\n- There is a typo in line 144 of the paper: \"interesting\" --> \"interested\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "93ONE3ziYr", "reviewer": "Reviewer_7FpA", "summary": "This paper explores the potential of fine-tuning language models (LLMs) to improve their causal reasoning abilities, particularly through counterfactual feedback. Standard LLMs exhibit a discrepancy between their strong recall abilities and their limited reasoning abilities. The authors propose a framework that incorporates new metrics (necessity and sufficiency inconsistency rates) to assess reasoning more comprehensively. They explore fine-tuning methods, including supervised and preference-based approaches, to improve understanding of causal dependencies and generalization across tasks. The study finds that fine-tuning is more effective at improving reasoning in inductive scenarios than in deductive ones, suggesting different patterns of generalization in reasoning tasks. Also, the paper suggests that using factual and counterfactual reasoning examples together can lead to better reasoning abilities in models.", "questions": "Section 6.3 lacks any information about the real-world data used. It is important to discuss the training and test data used and how many samples were rejected due to incorrect or unclear counterfactuals. It is mentioned at the end of the paper *Note that not all generalization modes were tested for each problem due to differences in causal structures, so the average scores were calculated using only the problems that were tested for each generalization mode.* which needs to be discussed. \n**Provide a table summarizing the datasets used, including sample sizes, data sources, and which generalization modes were tested for each problem.** This will clear the confusion in this section. \n\nAuthors need to provide a discussion section where the type of error each type of fine-tuned model makes is discussed. Currently, just from the numbers, it is hard to understand what are the errors made by each model and how preference tuning can solve some of them. **Some error analysis examples or visualizations that would illustrate the different types of errors made by each model and how preference tuning addresses them would be helpful here**.\n\nSome results need to be discussed more like statements like: *However, with common-cause/effect and deductive demonstrations, they no longer show the same reasoning improvements as observed in the in-domain setting.*  Inductive vs. Deductive results are not clear and need to be discussed why one approach performs better than the other. \n\nFinally, the most important result to discuss is the data imbalance between the factual or counterfactual training set alone and a combination of the two. Both experiments need to be balanced for a fair comparison, otherwise it could just be the additional data points that lead to improved performance.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The authors present a framework for fine-tuning based on causal reasoning, which provides a structured approach to understanding how reasoning can generalize across different problems. They formally categorize this generalization into four distinct types: common-effect, common-cause, inductive, and deductive reasoning, which helps to better understand reasoning in LLMs. This classification provides a systematic way to analyze and apply causal reasoning to different tasks. \n\nThe authors propose new metrics-Necessity Inconsistency Rates (N-IR) and Sufficiency Inconsistency Rates (S-IR)-for evaluating reasoning in large language models (LLMs) based on causal principles. They also introduce absent necessity and absent sufficiency to capture cause-effect relationships beyond traditional necessity and sufficiency. These metrics provide a more detailed framework for evaluating causal reasoning in LLMs.\n\nThe authors fine-tuned the models by including the counterfactual information with the factual information, which proved useful in improving the model's understanding of causal dependencies. This focus is consistent with human reasoning processes, where understanding \"what-if\" scenarios often plays a critical role in causal inference. Preference training using DPO was also used in the paper, which showed that learning from comparative feedback can be more effective than standard supervised approaches in some cases.", "weaknesses": "My main concern with the paper relates to the way the experiments are conducted. The paper claims that using factual and counterfactual examples leads to a performance gain compared to the base model or the model trained only on factual examples of counterfactual examples. However, this could be due to the increased number of training examples, which leads to an increase in overall performance. \nIn addition, a model tuned with both factual and counterfactual examples tends to perform well on metrics that measure both properties compared to either one. This may simply be due to an over-fitting of causal specific patterns. In addition, the experiments are limited to one model and one synthetically generated dataset, and the results are difficult to generalize across models and datasets. The other datasets used later are missing all the training/test splits used and how the model was trained. Section 6.3 is hard to understand because it lacks information on the number of samples used. \n\n**Actionable Suggestion** : It would be great to see an experiment where you randomly sample equal number of factual and counterfactual examples and fine-tune the model and compare it with the individual factual and counterfactual only models. That would be a fair comparison.\n\nThe experiments also lack deeper analysis to better understand where fine-tuning fails. For example, a breakdown of errors by task type, complexity, or specific reasoning challenge (e.g., false sufficiency vs. false necessity errors) would provide insight into model weaknesses. The errors for which the model struggles to perform well need to be analyzed, and this is not discussed in the paper. \n\n**Actionable Suggestion:** Please provide the task specific error categories that provides deeper insights into the models' strengths and weaknesses.\n\nFinally, the approach of using counterfactual reasoning to improve model performance in reasoning tasks is not new and has been studied previously by https://arxiv.org/abs/2307.02477 and https://aclanthology.org/2022.lrec-1.229/. \nThe novelty of this paper lies in the way the two metrics are proposed to understand causal reasoning in LLMs. However, due to the weaker experiments, it is not clear why this paper stands out. It is similar in many ways to papers suggesting that more data can improve reasoning (such as from https://arxiv.org/abs/2309.12284).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)