Generalization of RLVR Using Causal Reasoning as a Testbed
Authors: Brian Lu
Hongyu Zhao
Shuo Sun
Hao Peng
Rui Ding
Hongyuan Mei
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Models
Reinforcement Learning with Verifiable Rewards
Generalization
Causal Reasoning
Primary area: foundation or frontier models, including LLMs

Abstract: Reinforcement learning with verifiable rewards (RLVR) has emerged as a promising paradigm for post-training large language models (LLMs) on complex reasoning tasks. Yet, the conditions under which RLVR yields robust generalization remain underexplored. This paper provides an empirical study of RLVR generalization in the setting of probabilistic inference over causal graphical models. 
This setting offers two natural axes along which to examine generalization: (i) the level of the probabilistic query---associational, interventional, or counterfactual---and (ii) the structural complexity of the query, measured by the size of its relevant subgraph. We construct a dataset of causal graphs and queries spanning these difficulty axes and fine-tune Qwen-2.5-Instruct models using RLVR or supervised fine-tuning (SFT). We vary both the model scale (3B-32B) and the query level included in training. We find that RLVR yields stronger within-level and across-level generalization than SFT, but only for specific combinations of model size and training query level. Further analysis shows that RLVR's effectiveness depends on the model's initial reasoning competence.
With sufficient initial competence, RLVR improves an LLM's marginalization strategy and reduces errors in intermediate probability calculations, producing substantial accuracy gains, particularly on more complex queries. These results show that RLVR can improve specific causal reasoning subskills, with its benefits emerging only when the model has sufficient initial competence. Our code and data is available at https://github.com/zhichul/rlcausal.

Published as a conference paper at ICLR 2026
GENERALIZATION OF RLVR USING CAUSAL REASON- ING AS A TESTBED
Brian Lu 1+ Hongyu Zhao Shuo Sun
Hao Peng
Rui Ding
Hongyuan Mei
"Johns Hopkins University University of Maryland, College Park Universiit of Illinois Urbana- Champaaign *Microsoft Research Asia Toyota Technological Institute at Chicago
ABSTRACT
Reinforcement learning with verifiable rewards (RLVR) has emerged as a promis- ing paradigm for post-training large language models (LLMs) on complex rea- soning tasks. Yet, the conditions under which RLVR yields robust generalization remain underexplored. This paper provides empirical study of RLVR general- ization in the setting of probabilistic inference over causal graphical models. This setting offers two natural axes along which to examine generalization: (i) the level of the probabilistic ausriaaasoiiationnal, interventional, or counterfactual (ii) the structural complexity of the query, measured by the size of its relevant subgraph. We construct a dataset of causal graphs and queries spanning these dif- ficulty axes and fine-tune Qwen-2.5-Instruct models using RLVR supervised fine-tuning (SFT). We vary both the model scale (3B-32B) and the query level included in training. We find that RLVR yields stronger within-level and across- level generalization than SFT, but only for specific combinations of model size and training query level. Further analysis shows that RLVR's effectiveness de- pends on the model's initial reasoning competence. With sufficient initial com- petence, RLVR improves an LLM's marginalization strategy and reduces errors in intermediate probability calculations, producing substantial accuracy gains, particularly on more complex queries. These results show that RLVR can im- prove specific causal reasoning subskills, with its benefits emerging only when the model has sufficient initial competence. Our code and data is available at attps://githus. com wtt.c///hhhccccccrcaeete..seeeee
INTRODUCTION
Reinforcement learning with verifiable rewards (RLVR) (Lambert et al., 2025 DeepSeek-Al et al.] 2025) is promising paradigm for post-training large language models (LLMs) complex rea- soning tasks. RLVR leverages automatic correctness signals from domains equipped with reliable verifiers, and has enabled substantial progress in mathematical problem solving (Shao et al.] 2024
Lambert et al., (025 DeepSeek-Al et al. 2025), formal theorem proving (Xin et al.) 2024: Ren et al. 2025 Wang al.] (025), code generation (Le al.) 2022, Liu & Zhang (025), and in biomedi-
cal and chemistry applications (Biomni & Sky RL 2025) Narayanan et al. (025). Despite rapid
progress across diverse domains, the conditions under which I.LMs trained with RLVR exhibit reli-
able generalization beyond their training data remain underexplored.
Recent work has begun to examine the generalization behavior of caaaaforreemett-eeaninn fine-tuning (RL) relative to supervised fine-tuning (SFT) or hybrid approaches (Chu et al. 2025 Chen et al., 2025 Swamy et al. 2025 Qiu al. 2025). Particularly relevant is Chu al. 2025), which evaluates the generalization of RLVR and SFT I on novel variants of text and visual reasoning tasks. Our work differs from these prior work by focusing on a challenging and essential task: causal inference.
Causal inference provides structured setting for examining RLVR generalization, because its three levels of inference-associinn associational, interventional, and counterfactual, known collectively as the causal ladder (Bareinboim et al. 2022 Pearl & Mackenzie 2018) form a hierarchy that supports
"Correspondence zlu39 @jhu.edu.

---

Published as a conference paper at ICLR 2026
Task Formulation and RL Training
Input: Causal Graph Marginal Inference Question
Oslput: Derivations & Calculation Final Answer
Samp1e Answer
7200000 2HOCES5
THOUGHT 1OOCE2S
IBOCGIE FROCZOS
Bace's canaai graph:
modidy the graph
so graph change
inccherent steps
Variables: vi, M5,
Owen2.5
Helle Focmola
digcnssion
false assuxption
Valsee: (0, 11
Gzagl::
vubro tute valoe
copy waong eyt
no calealations
Vi -2 22, v2
V1
// vore edged
compute ansvez
Feramecrization:
PC
Stons v1 interwened (te pro2lv2e1) [0.3. (0.7)
ANANER: [0. [0.66.)) 66.
p1o21vie0; 15.3, 0.7)
Aicle all caleciatiions,
pis2jri-1) 15.0, 0.23 piv3jvi-0, // vore opts
10,70, 6.221
gledee (d) m2) 6.78
ARENER: [0.70, 0.22]
Auswen: 10.3, 0.72
Question:
Nhat's marginal distribution of inbervene net 19
1.0
0.2
0.0
Compute Rewards
Reward:
r(y)
0.8 p2)
)+
0.2 (s)
Data Generation
Graph Sampler
Query Sampler
Query Sampler
Sampler
Association
AVi [ m
p ((
Intervemion
Distribution of DAGs
Distrburion Moclanisms
Distrbutton of Queries
Graph Variable Elimination
Counterfictul
XVitty
Vx Ws)
Three Query Levele
Figure 1: Top: Our causal inference task for investigating generalization of RLVR (see section system prompt (fig. [ omitted for space. Bottom Left: Generative process for sampling task stances, and solver for computing the reference (see section 3). Bottom Right: We generate associa- tion, intervention, and counterfactual queries to study RLVR's within-/acroos-levell generalization
both within- and across-level generalization tests. CLadder (Jin et al. 2023) covers this hierarchy and serves as holistic causal inference benchmark, requiring models to interpret natural language scenarios, identify query types, and formalize causal expressions, in addition to performing the derivation and caleulation needed answer the query. In our setting, focus on the derivation and calculation, which require more steps on larger and more interconnected graphs, making them useful for probing how LLMs fine-tuned with RLVR perform as required number of reasoning steps increases. We therefore construct our own dataset, RI. RI.Causal, whose questions are about fully speci- fied causal graphs instead of natural language scenarios. We also vary the graph structure, extending beyond CLadder's manually curated topologies to larger, randomly generated 10-node graphs.
Our task illustrated fig. top. Concretely, the input & contains description and a query. For RLVR, the LLM outputs intermediate reasoning steps followed by probability distribution P as the final answer For SFT, the LLM directly outputs the distribution p We define processes for sampling instances of our task, and compute for each instance reference answer p* via variable climination (Zhang & Poole 1994).
causal graph
We then RLVR and SFT fine-tuning experiments starting from representative LLM family, Swee2..5-Instruet (Qwen et al. 2025), varying model size and the level of query seen during training. Our RLVR training uses variants of GRPO (Shao al.) 2024) and DAPO (Yu et al. (025b), )), and our SFT baseline is trained to maximize the probability of p" conditioned on task input Ø.
'The reader would be correct to point out that the causal ladder, contrasts the different knowledge required answer each level of questions (Bareinboim al. 2022). but setting with full SCM parametrization as input, eliminates this difference. However, queries from each of levels still need different modes of reasoning- abduction for association, deduction for intervention, and abduction followed by deduction for counterfactual. We discuss in section 3 how our setting affects the difficulty ordering of the three levels.
Additional ablations studying SFT with dndection-sampllle reasoning chains included in appendix D.S
2

---

Published as a conference paper at ICLR 2026
We present the following findings from our experiments and analysis:
1. Within- and Across-level Generalization When trained and evaluated on the same query level, RLVR achieves stronger generalization than SFT association and intervention queries for models 2 7B, but under-periorms on 3B (for all levels) and counterfactual level (for all sizes): When measuring generalization a different query level from training, RLVR outperforms SFT on sizes 2 7B (figs. and 4). RLVR is often more precise than SFT and better on more complex queries (fig.
2. Scaling and LLM's Reasoning Prior We trace the effectiveness of RLVR partly back to the strength of the LLM reasoning competence prior to fine-tuning, Scaling up the size of the LLM improves reasoning significantly: 3B models fail to reason before and after RLVR, while an 32B model prompted to reason zero-shot beats 32B model that is fine-tuned but predicts the answer directly (fig. bottom).
3. RLVR improves marginalization strategy and reduces derivation and calculation rors. Overall, when there is sufficient initial reasoning competence, RLVR shifts the marginalization strategy of LLMs towards incremental marginalization (fig. reduces abstract probability/causal derivation errors (e.g. dropping dependencies, confusing inter- vention with observation) (fig. 5 bottom) well as calculation errors (fig. 27
Overall, our findings contribute to the understanding of RLVR's generalization behavior as well as
its effectiveness on enhancing L.LMs' reasoning capabilities on formal causal reasoning tasks.
METHOD
We are interested studying the limits of generalization of RLVR using the task of probabilistic inference in causal graphical models. In this section, we discuss the task definition (section 2.1 training objectives (section 2.2). and the main factors we will vary in our experiments (section 2.3
2.1 TASK DEFINITION
Please refer fig. (top) for illustration of the task input and output.
Input
We use Qwen2.55Instruet (Qwen et al. 2025) as our base model. The input x for our
task consists of system message Tays containing short instructions for task and format (fig. [8]
appendix) and a user message Tuser describing a concrete task instance (fig. [3] appendix), including
description of the causal graphical model and query. The description of the causal graph includes
variable definitions, the graph structure, and mechanism parametrizations.
Output For RLVR, the output y is reasoning chain followed by probability distribution. For SFT. the output y is directly a probability distribution. See fig. [ (top) for an illustration of the task for RLVR. We perform extraction of the answer Py from y using regular expression. Each training instance x comes with a reference answer p
2.2 TRAINING OBJECTIVES
Reinforcement Learning with Verifiable Rewards We optimize the RL objective Ex-TlEy-pu(() [r(y), where T is the distribution over training instances. Following typical
RLVR setups (DeepSeek-AI et al. 2025). we use a combination of format and accuracy reward, specifically r(y) 0.8 Tams (py, PT + 0.2 (tomma(y).. where
Vans(p 9) = 1D( 4) < 4 Tiormat (v) = 0.5 1 [f extractable] + 0.5 [f length correct] (1)
and D(p,g) q) : 1 (p(x) ((x)) do is the total variation distance. We round f.y and p to the nearest two decimal points and use 0.01.
Supervised Fine-tuning For our supervised fine-tuning baseline, we directly maximize the con- ditional likelihood of the reference answer ve for input x, Eg log pe(v x).
è write intervention queries as p(Vi(1 for consistency in notation with the counterfactual queries. equivalent to p(v: do(v, = c)) for readers familiar with the do notation (Pearl 2009
3

---

Published as a conference paper at ICLR 2026
3
3
5
Illustration of graph modifications corresponding to each query level and its relevant (solid) and irrelevant subgraph (dashed). x denotes dependencies removed due to an intervention. Relevant nodes are defined as ancestors of either the observation or the query variable, after graph modifications are performed to account for any interventions. Left: original graph. Mid-left: relevant nodes for association query pVV2 Vs uz). Mid-right: 2 relevant nodes for intervention query p( (v2)))) Right: 10 relevant nodes for counterfactual query p p(Va(va=e) V2 = 02).
Figure 2:
2.3 STUDY DESIGN
Data Setup We stratify the task instances across two axes of difficulty: first, by the query's 's level (association, intervention, or counterfactual), and second, by the complexity of the query mea- sured by its relevant subgraph. We vary the source of the training data between individual levels to measure across-level generalization, and breakdown within-level generalization by complexity.
We vary the model size between 3B, 7B, and 32B within the @wen2.55-Insrru family, using variants GRPO (Shao al.) 2024) and DAPO (Yu et al. 2025b) for RL, and maxi- mum likelihood of p" conditioned on æ for SFT.
Fine-tuning Setup
3 DATA GENERATION
In fig. (bottom left) we provide a diagram for the generative process of synthesizing data for our task. We first describe the objects that we sample, then the step by step process for sampling them.
Structural Causal Models We use structural causal models (SCMs) (Pearl et al. (016) with bi- nary variables no cycles, and independent noise variables as our causal graphical model family. Such structural causal model M = (G, F. Q) is defined by DAG G (V, E), where each node i V is associated with a variable v, and deterministic function fa that defines its relationship with its parents pa(v;) and noise variable u, following distribution qa Q This gives
2 = (pa(v)), u.), u, ~ q
which induces conditional distributions p(v; pa(v:)) for all variables Vi.
fig. 2 illustrates queries of each level and their graph modifications. Following notation in Pearl et al. 2016). an association level query p(v; Vy 0y) concerns statistical dependence, asking about the distribution of v; given an observed value Vj Vy-- An intervention level query p( p(Vi(y,=e) concerns causal effects, asking about the distribution of v: under external intervention that sets V to c. counterfactual level query p( p(Y(())=e)
Queries
V4 = v2) concerns hypothetical alternatives, asking about the distribution of had vy been set in world where we in fact observed V; 1%
DI: Graph Sampler The first step of sampling SCM is sampling DAG y. Given the desired size N, adopt Lampinen et al. (2023'ss 's workflow generate a graph, which first samples random number of independent nodes between 1 and N. Additional nodes are then introduced iteratively until reach N total nodes. Each added node has either uniformly from the existing nodes. Finally, the nodes are renamed with a random permutation.
parents chosen
Details on this relevant subgraph metric are in section 5 We choose binary variahles for speed of computing the ground-truth solution- inference in graphical models NP-hard in general, and slows down significantly in practice with cardinality and graph size. 'The tbservation//inteeveniiisss in our data always on single variable for simplicity, and L.MMs already struggle in this simple setting. Future work could explore vector-valued observations and interventions.
4

---

Published as a conference paper at ICLR 2026
D2: Mechanism Sampler For each V, and for each joint assignment to its parents pa(v:), we sample a binary distribution gv uniformly from the simplex. We then define one noise variable up dv per each and define the mechanism fü to simply select one particular noise variable's 's value to take on based on v, namely 8 fi(v, the noise distributions Tv onto rows of the conditional probability table p(v; pa(y:)).
((v,4;:) u,) = u." This simple mechanism directly maps
D3: Query Sampler Given a SCM, we then sample queries for a chosen level. Association level queries contain one observation, intervention level one intervention, and counterfactual level one of each (fig. (2). We sample the target variable v, uniformly. For association and intervention level queries, we also uniformly sample a variable v, to condition on or intervene on, respectively. For the counterfactual query, we sample the intervention variable v, first, and then sample the observation VA from its descendants. Observations are drawn from the SCM, while interventions uniform (0, 1)-
D4: Solver Given the full specification of SCM M (G, F, Q) and a query q of association, intervention or counterfactual level, we reduce to exact inference of some o' in a possibly modified SCM M' = (G', IF,, Q). We then use variable elimination (Zhang & Poole 1994) to compute the answer. The modified graphs are illustrated in fig. 2 with additional details in appendix A.I
Difficulty Metric We first stratify queries by their level, as they represent different modes of rea- soning. In our setting where we provide the fully parametrized SCM as input, association queries of the form p(v. V2 vz) requires abduction (summing out ancestors in the posterior), intervention queries of the form ((o)) requires deduction (sum out ancestors after fixing Vz), and counter- factual requires abduction followed by deduction (infer posterior of noise variables, then out noise variables and ancestors in an alternative world after fixing Vý). This changes the difficulty ordering from the usual association intervention in the causal ladder association intervention in our setting, since computing posterior given vy usually requires more work than fixing V, at 7
Within each level, we also measure difficulty by [vell the complexity of the query, defined as the size (number of nodes) of the subgraph relevant to the query. Relevant nodes are ancestors of either the observed variable, or the query variable, in the modified graph G². Factors on irrelevant nodes in V \ VeS sums out to I during variable climination and can be ignored. See fig, 2 for examples.
4 EXPERIMENTS
4.1 EXPERIMENT SETUP
For each level in {association, intervention, counterfactual), we generate training, development, and test set, consisting of 8000, 2000, and 8000 examples respectively. Each example is query over parametrized causal graph over 10 binary variables. We ensure that the SCMs in training, development and test sets are disjoint. Refer to appendix A.2 for more details.
Dataset Construction
|Vral Distribution See fig. 7 (appendix) for histograms of query complexity metric |Vial- When presenting results, we group examples by ranges of [Viel with the following cutoffs: 1-3 (small), 4-6 (medium), 7-10 (large) for association, 1-2 (small), 3-4 (medium), 5-10 (large) for intervention, and 1-7 (small). 8-15 (medium), 16-30 (large) for counterfactual. The complexities are meant to be used within-level and is generally not comparable across levels.
Metrics Given input x for a language model, its reference solution pt, and a LLM output y, we measure its correctness by the following metric based on total variation distance and a format requirement. Let P,, be solution extracted from y. Let P.y and pz be rounded to 0.01,
CORRECT, (x. y) = {
if format error, failed to extract Py
(3)
if (py, pt)) S t
where =
((x) dr is the total variation distance, and 0.01.
Filtering Due to rounding, on many instances the intervention and observation may not result in a
measurable change in the marginal distribution of the query variable, making the instance insensitive to faithful reasoning. Therefore, we filter out such examples in our main analysis. We also include the unfiltered results in appendix D
IIf we had also included intervention queries with conditions, then it would require graph modification fol- lowed by abduction, and the difficulty ordering would be reverted back the usual association < intervention.
5

---

Published as a conference paper at ICLR 2026
Isanetiion Quere
A-V Phonutt
1em Durnee
AU% Chansa ineett
cot 2P1
Figure 4: Top: Accuracy (y-axis) vs. LLM size (x-axis) when evaluated intervention (left), association (middle), and counterfactual (right) queries. Red curves correspond to RLVR, blue curves correspond to SFT. Solid (-) curves LLMs fine-tuned on the same level as evaluation, dashed (--) curves are trained on a different level from evaluation. Bottom: Reasoning (RLVR) vs non-reasoning (SFT) strategies, before and after fine-tuning. As scale increases, both reasoning and non-reasoning prior improve, though the reasoning prior benefits more from scaling.
Fine-tuning And Inference Setup For RLVR, We use GRPO with the token-level normalization, and DAPO without the overlong buffer for simplicity. We use batch size of 8, with 32 roll-outs per example, a learning rate of 10-6, and other hyperparameters default from DAPO implemented in the VERL library (Sheng et al. 2024). We train for 7.5k steps for 3B and 7B models, and 2.5k steps for 32B models, as they are roughly third slower to train compared to 7B. For SFT, We train using maximum likelihood on pt for 5k steps, with learning rate 10-6 and pick best checkpoint (saved every 200 step) by picking best loss on development set. For inference, we decode at temperature Additional details on nyper-parameters are included in appendix B.I
4.2
MAIN RESULTS
We focus our discussion and analysis on GRPO as the represen- tative RLVR algorithm in the main text, as it simple, widely used, and its results are not significantly different from DAPO in our experiments. DAPO results are included in appendix D
Inter
Assc Counter
latev Asse Counter
3B
SET
SET
SFT
SFT
SFT
SET
7B
REYR RLYR
SET
REVR RIVK RLVR
In fig. we compare the accuracy of LLMs fine-tuned via RLVR and SFT on the three query sxpes-intervention association, and counterfactual. We vary the model size between 3B and 32B, and we vary the query type that the model was fine-tuned on.
30R
RIVR RIVR
SFT
REVR
RIVR
RIVR
Within-lvel
Acme-lovel
Figure 3: The algorithm with higher aithin-/across-level aceu- racy for different sizes and query types. Significant cells (paired- perm test at 0.05) bolded and colored.
Within-level Generalization: RLVR outperforms SFT on only a subset of (model size, query type) configurations. In fig- 3 left, we show the size and query type configurations for which RLVR outperforms SFT, when trained and evaluated on the same query type RLVR significantly outperforms SFT on intervention and association queries, for sizes 2 7B.
Across-level Generalization: RLVR outperforms SFT beyond 3B models. Larger LLMs per- form better across levels for both RLVR and SFT. In fig. 3 right, we see when evaluating on different level from training, RLVR outperforms SET on models 7B. In fig. we find that for both SFT and RLVR, the performance gap between LLMs fine-tuned on in- and out-of-level queries gen- erally decreases as model scale increases, suggesting better cross-level generalization with scaling.

---

Published as a conference paper at ICLR 2026
Intervention Queries
Asseciation Queries
Counterfactual Queries
1.0
0.-
0.0
ma
inl
vie 717bb
1.326
d ma
d 3b
A me
7b
1.330
#3 mi
1.38
e rte
7b
#330 into
322
st
Norginsiization Strategy
Incremental (Comert)
Brute fome (Comeet
Neightors (Conrett)
No Marginsizatton [Caneet)
Incremental (Incorrect)
inncomect
incorrecti
Horginalizstion (ncorrect)
Intervention Queries
Asseciation Queries
Counterfactual Queries
Proportion
Proputtion
117b
# #22b ver
328
# #32b 12300
3b
# 3233
r130
r
Probbility Derivation
No Probabiliyy Erne [Conect)
Contart hobsbiilty Eroo (Corrett)
Net Apolicable (Conect) Not Applicaaiee (nnorrect)
No Probability Error Incorrezt)
Contains Probabliity Emor (ncorrect)
Figure 5: LLM judge (o4-mini) analysis of the marginalization strategy (top) and the existence of
derivation errors (bottom) before and after RLVR. Derivation errors and marginalization strategies are annotated (the same) 80 samples per level. Judge prompts (including category definitions) are included in fig. Example traces of marginalization strategy are included in figs. 14
4.3 ANALYSIS
Analysis-I: RLVR ineffective when the reasoning capability of the base model too poor prior to fine-tuning. Why is RLVR ineffective on 3B and counterfactual queries,
in fig. 3
3B models attempt explicit marginalization before fine-tuning, but succeed rarely; After fine-tuning it outputs answer directly without explicit marginalization. We reviewed subset of reasoning traces across levels and sizes, before and after RLVR fine-tuning. All models attempt explicit marginal- ization before fine-tuning, but only models > 7B continue attempting explicit marginalization after fine-tuning. Our analysis of reasoning traces in fig- fisow that at 3B, traces that attempt to marginal- rarely correct-a possible explanation
ize step by step (incremental, brute force, and neighbors) for their regression to directly predicting the answer after fine-tuning. See fig. [8] for an example trace from 3B model with errors annotated, and Analysis- III for more detailed discussion.
On counterfactual level queries, models did not attempt to build twin-networks or perform inference over exogenous variables, both before and after RLVR. We reviewed a subset reasoning traces from models of different sizes, but did not observe any attempts to create twin-network-gnae -rrahh. We conducted an oracle experiment where additional hints on solving the counterfactual query via twin network graph is provided in the system prompt (fig. appendix). However, its accuracy is not very different from the original prompt without hints (fig. This result is in contrast to more positive findings in previous evaluations of LLMs counterfactual reasoning in the commonsense settings (Kıcıman et al. 2024) or formal settings with continuous mechanisms Tu et al. (2024). This contrast may be partly due our strict metric and more numerically challenging task.
Overall, these findings suggest that RLVR is sensitive to the LLM's reasoning success rate prior to fine-tuning, echoing the cold start problem (DeepSeek-Al) AI et al. 2025) in RLVR post-training. Since we start from Qwen2.5-Inmg Instrucct (Qwn et al. (225).. which is already instruction-tune and has some reasoning-tune -uuiig too, the limitations seen here is likely more specific the causal domain.

---

Published as a conference paper at ICLR 2026
Isanwtiin Queres
Adtal aronn OLarie
5.2 0.4
6-l
6-:41
020 5200
5079
Ca0 DOG odl 8200 sic cale 621 1200
120 0.00 EGG 021
Figure 6: Top: Within each query level, accuracy vs. query complexity Víe increases. Note that complexities are not comparable across levels. Bottom: Accuracy as the threshold for correctness t E (0.01, 0.2] is relaxed for 7B models. r--xis plots threshold for accuracy t (the lower the stricter), and y-axis plots accuracy at t. Across all levels (left to right), we see that RL models are often more precise than SFT models. See figs. [2] to 24 for the same plot but for all query levels and model sizes. The observable staircase pattern is due to rounding to two digits.
Analysis-II: LLM's success rate prior to fine-tuning improves with scale and especially with reasoning. The prior is major source of RLVR's effectiveness. In fig. 4 (bottom), we show the initial accuracy of LLMs prompted to reason (used in RLVR) versus LLMs prompted to directly predict the answer (used in SFT). Across all levels, both the reasoning prompt and the direct predic- tion prompt's s accuracy prior to fine-tuning increases substantially with scale, but reasoning benefits more from scaling. Furthermore, the reasoning prompt prior to fine-tuning achieves a higher aceu- racy than non-reasoning prompt across the board. Notably, on intervention and association queries, 32B models fine-tuned with SFT still under-performs zero-shot reasoning (fig. 4 bottom). This demonstrates the substantial benefit of the reasoning prior, which we next show RLVR improves on.
Analysis-III: RLVR favors incremental marginalization strategy and reduces abstract reason- ing errors and calculation errors. Answering queries from our causal inference task requires abstract reasoning about probability and causality (e.g. marginalization and graph modification) as well as low-level caleulations (e.g. substitution of values and arithmetic). See example in fig. T0
To qualitatively understand LLMs' performance on these sub-tasks, we manually examined a sub- set of reasoning traces across all levels. We found that incorrect answers often due errors in abstract reasoning about probability or causality (e.g. falsely assuming independence, dropping terms in marginalization formula, treating a counterfactual query an intervention / association query), and can sometimes also be due to low-level calculation errors (e.g. copying wrong probabil- ity values from the problem statement or making numerical mistakes in addition multiplication).
To quantitatively understand LLMs' improvement on these sub-tasks after RLVR, we analyzed the reasoning traces from our RLVR models with o4-mini. We focus here on presenting analyses of abstract reasoning errors and marginalization strategies (and leave low-level calculations to ap- D.7) abstract reasoning errors were salient in our qualitative analysis, and marginalization influences the high-level solution |
Abstract Reasoning Errors: In fig. 11 (bottom) we show our prompt to 04-mini for analyzing ab- stract reasoning errors in an LLM's reasoning chains. The prompt focuses on two sub-types of
'The first author validated the prompts for analyzing reasoning traces with 10 samples per category of marginalization strategy / reasoning emror found 25/30 agreement with o4-mini reasoning error detection and 37/40 agreement on marginalization strategy detection.
8

---

Published as a conference paper at ICLR 2026
errors, one where some probability identity was used incorrectly, such errors in the formula for the chain rule/Bayes rule, and another where some false assumption was made, such as assuming unwarranted independences when computing joint probabilities falsely treating interventions as observations. In fig. 5 (bottom) we show the comparison of the rate of abstract reasoning errors detected by 04-mini (indicated with color) in reasoning chains before and after RLVR, and whether the final answer was incorrect (indicated with shading). Overall, we see that larger models make much less probability derivation errors, RLVR fine-tuning significantly reduces probability deriva- tion errors, and the existence of such errors correlate highly with incorrectness of the final answer.
Marginalization: In fig. [1 top we show our prompt for o4-mini to annotate the marginalization strategy used in a reasoning chain. We identified three distinct marginalization strategies during our qualitative analysis: brute-force summation, where the solution involves large summation formulas (e.g. p (V4,, V5 = =) En.wa.y p(V1, V2, V3, V4, V5 = 1)), incremental marginalization, which sums out one or two variables at a time (e.g. first marginalize out V3. Va then v2), and no-marginalization, which can be due to trivial query like -()) or due LLM predicting a numeric answer directly (often incorrectly) without calculations. To address ambiguity between "brute-force" and "incremental marginalization" on the base-case of summing over the immediate neighbors (as it can be seen as special case of both), we add a fourth category "neighbors" to avoid ambiguity during o4-mini's ss analysis. See figs. 14 to 17 for an example of each category.
In fig. (top), we see that for 7b and 32b models, RLVR shifts the marginalization strategy towards incremental marginalization, and we found that this effect is especially strong more difficult queries (see fig. 2T topee with a larger number of relevant nodes to sum over. A possible explanation for this shift could be that brute-force summation often introduces large summations with terms that are long products of conditional probabilities, which can give rise to more chances of errors, causing incorrect answers (green shaded), and resulting in RLVR learning avoid it. In fig. (top). we also see that for 3b models, instead of learning to marginalize correctly, they learned to avoid marginalization after RLVR (grey), frequently predicting the answer without any calculations (see fig. 17 for an example). The high rate of failure (indicated by shading) of marginalization-bass solutions (incremental, brute-force, and neighbors) prior to training is a possible explanation for why 3b LLMs avoided it after RLVR.
How does RLVR models behave differently from SFT models? In fig. 6 (top), we show that within each query type, on 7B and 32B LLMs, SFT tend to perform better on the less complex queries of that type, while RLVR performs better on more complex queries of that type. In fig. (bottomm) we plot the proportion of correct answers as we relax the criterion of correctness from exact match to within 0.2 in total variation distance. We show that LLMs fine-tuned with RLVR are more precise, while SFT models often is only able to get the solution approximately correct. Both RLVR and SFT significantly improve the precision of the LLMs relative to base.
Analysis-IV:
5 RELATED WORK
RLVR for LLM Post-Training Reinforcement learning is becoming a common step of LLM post-training. Early works RL align LLMs using human preference data (Ouyang et al., 2022]. Reinforcement learning with verifiable rewards (RLVR) (Lambert et al., 2025 DeepSeek- AI et al. (025) teach reasoning on domains where reward (often a combination of correctness and format) can be verified automatically. This led to significant progress in domains such as math problem solving theorem proving, and code generation (DeepSeek-Al al. 2025 Ren al. 2025
Liu ZZang 2025). In this paper, we attempt to understand the successes and limitations of RLVR'S generalization, using a controlled family of causal reasoning tasks.
Understanding RLVR's Generalization Several recent studies explored the respective contribu- tion of SFT and RL to the generalization of the resulting LLM (Chu al. 2025 Chen et al. 2025] Swamy et al. 2025). with evidence that SFT is useful for warm-starting, while Rl. can significantly
improve generalization. Another line of work investigates how SFT and RL should be ordered or combined to achieve the best generalization result (Liu et al. 2025b Qiu et al. 2025). Swamy et al. (2025) and Qin et al. (2025) study the sources of RLVR's 's effectiveness and where it improves mod- els, and Yue et al. (2025): ( Sun et al. (2025); Liu et al. (2025a) and Wu & Choil (2025) study changes on the generalization boundary of LLMs fine-tuned with RLVR. Similar to the earlier works, we investigate generalization of RLVR and SFT, and similar to the latter works, we focus under-

---

Published as a conference paper at ICLR 2026
standing the limits and the sources of RL's effectiveness itself. Different from these works, we focus on the causal reasoning domain, a less explored but important area that remains challenging.
LLM for Causal Reasoning LLMs' understanding of causality has been receiving increasing in- terest Yu et al. 2025a), as LLMs make their way into domains like medicine and law where causal reasoning is essential. One line of work investigates the real world causal knowledge and reasoning of pretrained LLMs (Kiciman et al.) 2024 Zéčević et al.) 2023] Chi et al. 2024). These benchmarks are valuable for assessing whether LLMS encode plausible causal facts about the world, but they are less suited for probing RLVR, which focuses more on step-by-step reasoning over such facts to draw new conclusions, a separate skill from memorization of the facts or conclusions. A line of datasets that focus more on formal causal reasoning capabilities include CLadder, Corr2Cause, and Carl-gt (Jin al. 2023 2024, Tu et al. 2024). Among these more formal benchmarks, the closest to ours CLadder (Jin et al. (023). whose questions and answers are algorithmically generated by running causal-inference engine over structured graphical models and expressed as natural-language scenar- ios spanning all rungs of the causal ladder. Similar to CLadder, our benchmark is built on synthetic casual graphical models, focusing more on reasoning over memorization. Different from CLadder, benchmark isolates causal reasoning from natural language understanding by providing a full specification of the abstract causal graphic in the question rather than converting it to a natural lan- guage scenario. We also introduce substantially more structural diversity by using larger graphs (10 nodes) and a wider range of randomly generated graph topologies (80 distinct for training and 200 distinct for dev/test). These two features make our dataset focused and challenging dataset suitable for studying within-level and cross-level generalization of RLVR.
6 CONCLUSION AND FUTURE DIRECTIONS
In this paper, we investigated the generalization behavior of RLVR and SFT using probabilistic infer- ence in causal graphical models as a testbed. Our main findings are that RLVR is less effective when the task is too challenging for the model prior to fine-tuning, but when the LLM has a basic level of successful reasoning rate on a problem level, RLVR significantly improves generalization, by fixing domain-specific reasoning errors and boosting systematic marginalization strategies, outperforming SFT, and especially on more complex queries. We also find that the L.LM's prior (both reasoning and direct prediction) scales with model size, and scaling gains stronger with reasoning.
Our findings add to an emerging line of investigation into RLVR's generalization (Chu et al. 2025 Chen et al. 2025 Swamy et al. 2025). A future direction is to explore more deeply the roles of execution quality and strategy quality in reasoning and RLVR (e.g. see recent works Qin et al. 2025
Sinha et al.
2025). This may provide more insights into the mechanism of RLVR's generalization.
Our findings also add to an increasing body of work studying LLMs for causal reasoning (Kıcıman ct al. 2024 Jin et al. |223 2024). Their effectiveness some commonsense counterfactual causal reasoning settings (Kicmman et al. 2024) contrasts with our results on more challenging formal counterfactual reasoning problems. A possible future direction is to understand this gap better.
Recent work has started building comainn-specife task suites for specializing LLMs to solve complex tasks in real-world science and engineering domains via RLVR fine-tuning. For example, Biomni-RO (Biomni& & Sky RL 2025) uses RLVR to fine-tune LLMs on a range of biomedical tasks that requires step-by-step reasoning. and etherO (Narayaaan et al., 2025) does so similarly for chemistry. Beyond supporting our analysis of RLVR generalization, our dataset can be used to train and probe sub- skills (e.g. organizing marginalization of latent variables, applying probability identities, performing arithmetic) that are useful to solving more practical causal and probabilistic problems, even though our problems are abstracted away from real-world scenarios. Our data generation process can also be configured to scale up the number of problems as well as their difficulty, by increasing graph size and the cardinality of each variable as appropriate.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
This work was supported by a research gift to the last author by Adobe Research. This work is also supported by NSF Grant No. CHE2505932, an Amazon AICE Award, gift funding from A12, and a grant from Coefficient Giving. The views and opinions of authors expressed herein do not necessarily state or reflect those of the United States Govermment or any agency thereof. We thank the anonymous ICLR reviewers and mevierevi for their constructive feedback.
REFERENCES
Elias Bareinboim, Juan D Correa, Duligur lbeling, and Thomas F. Icard. On pearl's hierarchy and the foundations of causal inference. Probabilistic and Causal Inference, 2022. URL https:
7/api. wwepeessmanticcccoo..o wttp://o/o10111022223000909
Biomni and Team Sky RL. Biomni-rO: Using sl to hill-climb biomedical reasoning
agents to expert-level, Sep 2025. URL https://bar omni
biomni mecii-rr-tcche i ca reportt/
Hardy Chen, Haoqin Tu, Fali Wang. Hui Liu, Xianfeng Tang, Xinya Du, Yuyin Zhou, and Cihang Xie. Sft or rI? an early investigation into training rl-like reasoning large vision-language models,
2025. URL
abs ab5. 2504. 11468
Haoang Chi, He Li, Wenjing Yang, Feng Liu, Long Lan, Xiaoguang Ren, Tongliang Liu, and Bo Han. Unveiling causal reasoning large language models: Reality or mirage? In A. Glober- son, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang (eds.), Advances in
Neural Information Processing Systems, volume 37, pp. 96640-96670. Curran Associates, Inc., pwwp/ files/paper/2020
2024. URL httt:sss/ppoceeeingg...eerestrie
f
Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V Le, Sergey Levine, and Yi Ma. SFT memorizes, RI. generalizes: comparative study of founda- tion model post-training. In Forty-second International Conference on Machine Learning, 2025. URL htttps:://ppereeeeeeee
net/ forum? new./orummmeeeemem id=dYur3yabby
DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang. Han Bao, Hanwei Xu, Haocheng Wang. Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang. Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang. Meng Li. Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei, Tianyu Sun. T. Wang, Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An. Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu. Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li. Xiangyue Jin, Xiaojin Shen, Xi- aosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang. Xinxia Shan, Y. K. Li, Y. Q Wang, Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo. Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha. Zhe Fu. Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu,
11

---

Published as a conference paper at ICLR 2026
Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang. Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepscek-rl: Incentivizing reasoning capability in llms via reinforce- ment learning, 2025. UR
org/ http:/bs/255000 12948
Zhijing Jin, Yuen Chen, Felix Leeb, Luigi Gresele, Ojasv Kamal, Zhiheng LYU, Kevin Blin, Fer- nando Gonzalez Adauto, Max Aleiman-Weiner, Mrinmaya Sachan, and Bernhard Schölkopf. CLadder: A benchmark to assess causal reasoning capabilities of language models. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https:
/ //openreview.oo http:/otmmtodddm/eeeeeer.. jxxxqq
Zhijing Jin, Jiarui Liu, Zhiheng Lyu, Spencer Poff, Mrinmaya Sachan, Rada Mihalcea, Mona Diab, and Bernhard Schölkopf. Can large language models infer causation from correlation?, 2024.
URL /tttps:///axtttt/ .rg. abs/ 2.3 05836
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017. URL
https:://arat http:///xxvv.r///////// abs/ /1ss//11.... 6980
Emre KIemann Robert Ness, Amit Sharma, and Chenhao Tan. Causal reasoning and large lan- guage models: Opening a new frontier for causality, 2024. URL http://:///xxii..rg/////////
2335.00035
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brah- man, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Ma- lik, Victoria Graf. Jena D. Hwang, Jiangjiang Yang. Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luca Soldaini, Noah A. Smith, Yizhong Wang, Pradeep Dasigi, and Hannaneh Ha- jishirzi. Tulu Pushing frontiers in open language model post-training, 2025. URL https:
15124
Andrew Lampinen, Stephanic Chan, Ishita Dasgupta, Andrew Nam, and Jane Wang. Pas- sive learning of active causal strategies in agents and language models. In A. Oh, T. Nau- mann, Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances Neu- ral Information Processing Systems, volume 36, pp. 1283-1297. Curran Associates, Inc., 2023. URL htttps:/ www.eediiinnnn..uuieee.ee neurips httpppsscecpppppee....eeeeeeeee fites/pape//////////////// pdf
E
Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven Chu Hong Hoi. Coderl: Mastering code generation through pretrained models and deep reinforcement learning. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 21314-21322
Curran Associates, Inc., cc wwppppppppppp..eeeeeeeeeeeeeeeee files/papez/2022/ Sore
2022. URL
File/ 0ee//6664111 9deal
Jiawei Liu and Lingming Zhang. Code-rl: Reproducing rl for code with reliable rewards. 2025.
Mingjie Liu, Shizhe Diao, Ximing Lu, Jian Hu, Xin Dong, Yejin Choi, Jan Kautz, and Yi Dong. ProRL: Prolonged reinforcement learning expands reasoning boundaries in large language mod- els. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025a. URL httppp::///peeeevie.... net
fine-tuning, 2025b. wttp://wwwww..eeeooooo/iia//a//a httpte////oooommmaaa.aa.a.aaaaa.
Mingyang Liu, Gabriele Farina, and Asuman Ozdagaar. Uft: Unifying supervised and reinforcement 16984
Siddharth M. Narayanan, James D. Braza, Ryan-Rhys Griffiths, Albert Bou, Geemi Wellawatte, Mayk Caldas Ramos, Ludovico Mitchener, Samuel G. Rodriques, and Andrew D. White. Training scientific reasoning model for chemistry, 2025. URL htttps://arxiiiii http://aaxxxvv...raaa/// wtgp.bs///bb.sso/b./sssg///////e
17238
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kel- ton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback, 2022. URL https: ptttps://arxiiin //tpx//avvvv////////a///a org. wttp:/or//aba./..../// 02155
12

---

Published as a conference paper at ICLR 2026
Judea Pearl. Causality: Models, Reasoning and Inference. Cambridge University Press, USA, 2nd edition, 2009. ISBN 052189560X.
Judea Pearl and Dana Mackenzie. The Book of Why: The New Science of Cause and Effect. Basic Books, Inc., USA, 1st edition, 2018, ISBN 046509760X.
Judea Pearl, Madelyn Glymour, and Nicholas P. Jewell. Causal Inference in Statistics: Primer. John Wiley & Sons, 2016.
Tian Qin, Core Francisco Park. Mujin Kwun, Aaron Walsman, Eran Malach, Nikhil Anand, Hidenori Tanaka, and David Alvarez-Melis. Decomposing elements of problem solving: "math" does teach?, 2025. URL htt
Haibo Qiu, Xiaohan Lan, Fanfan Liu, Xiaohu Sun, Delian Ruan, Peng Shi, and Lin Ma. Metis- rise: RI incentivizes and sft enhances multimodal reasoning model learning, 2025. URL https:
//axxiv.org/ass/23000. 2000500066116666
Qwen, An Yang. Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang. Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yugiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025.
URL https: www.xaxxivv..... org/ abs/ /0fss/2442 2412 15115
Z. Z. Ren, Zhihong Shao, Junxiao Song, Huajian Xin, Haocheng Wang, Wanjia Zhao, Liyue Zhang, Zhe Fu, Qihao Zhu, Dejian Yang. 7. F. Wu, Zhibin Gou, Shirong Ma, Hongxuan Tang, Yuxuan Liu, Wenjun Gao. Daya Guo, and Chong Ruan. provveck--ooee--2:: Advancing for- mal mathematical reasoning via reinforcement learning for subgoal decomposition, 2025. URL
https://arxiv: org/
wttg//bs/////o/ abs 2504 21801
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathe- matical reasoning open language models, 2024. URL/ttps: //tp://aavvvvv//////ae org/ abs/20as 2402.
03300
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 19256, 2024.
Ilya Shpitser and Judea Pearl. What counterfactuals can be tested, 2012. URL https://arxii borg/abs// 120e 5294
Akshit Sinha, Arvindh Arun, Shashwat Goel, Steffen Staab, and Jonas Geiping. The illusion of diminishing returns: Measuring long horizon execution in lims, 2025. URL htttt:://///xiii
org ht///bb//5555 09677
Yiyou Sun, Yuhan Cao, Pohao Huang, Haoyue Bai. Hannaneh Hajishirzi, Nouha Dziri, and Dawn Song. DELTA: How does RL unlock and transfer new algorithms in LLMs? In The 5th Workshop Mathematical Reasoning and Al NeurIPS 2025, 2025. URL https::/openrev//// het.fforum www.mmmmmmmmfffmmmmmmmmmm LA5
Gokul Swamy, Sanjiban Choudhury, Wen Sun, Zhiwei Steven Wu, and J. Andrew Bagnell. All roads lead likelihood: The value of reinforcement learning in fine-tuning, 2025. URL https:
01667
Ruibo Tu, Hedvig Kjellström, Gustav Eje Henter, and Cheng Zhang. Carl-gt: Evaluating causal reasoning capabilities of large language models, 2024. URL https: // //aaxivvv... wwg//ba.......
2412 2, 17970)
13

---

Published as a conference paper at ICLR 2026
Haiming Wang, Mert Unsal, Xiaohan Lin, Mantas Baksys, Junqi Liu, Marco Dos Santos, Flood Sung, Marina Vinyes, Zhenzhe Ying. Zekai Zhu, Jiangiao Lu, Hugues de Saxcé, Bolton Bailey, Chendong Song, Chenjun Xiao, Dehao Zhang, Ebony Zhang, Frederick Pu, Han Zhu, Jiawei Liu, Jonas Bayer, Julien Michel, Longhui Yu, Léo Dreyfus-Schmidt, Lewis Tunstall, Luigi Pagani, Moreira Machado, Pauline Bourigault, Ran Wang, Stanislas Polu, Thibaut Barroyer, Wen-Ding Li. Yazhe Niu, Yann Fleureau, Yangyang Hu, Zhouliang Yu, Zihan Wang, Zhilin Yang. Zhengying Liu, and Jia Li. Kimina-prover preview: Towards large formal reasoning models with reinforce- ment learning. 2025. w https: arxiv V. org/ abs/ 2504. 11354
Fang Wu and Yejin Choi. On the limits of RLVR: Support, entropy, and the illusion of reasoning.
In 2nd Al for Math Workshop ICML 2025, 2025. URL
Forum? id- K-tKKXLLJJAAA
Huajian Xin, Daya Guo, Zhihong Shao, Zhizhou Ren, Qihao Zhu, Bo Liu, Chong Ruan, Wenda Li, and Xiaodan Liang. Deepseek-prover: Advancing theorem proving in Ilms through large-scale synthetic data, 2024. Ittps::/:
org/ abs/ 2405 14333
Longxuan Yu, Delin Chen, Siheng Xiong, Qingyang Wu, Dawei Li, Zhikai Chen, Xiaoze Liu, and Liangming Pan. CausalEval: Towards better causal reasoning in language models. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguisties: Human Language Tech- nologies (Volume Long Papers), pp. 12512-12540 Albuquerque, New Mexico, April 2025a. Association for Computational Linguistics. ISBN 979-8-89176--18-6. doi: 10. 10.18653/v 1/2025. wwwp:///2222.....aaacaca......
naacl-long.66 622.
622/
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang. Hongli Yu, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, Wei- Ying Ma, Ya-Qin Zhang, Lin Yan, Mu Qiao, Yonghui Wu, and Mingxuan Wang. Dapo: An open-source IIm reinforcement learning system at scale, 2025b. URL. https:
warxivv.rgg/aba. 2503 14476
Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Yang Yue, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in LLMs beyond the base model? In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URL https:::/:/// /ottts:://operrviee... ew www.://ee/eoeee.eemeo fom 400g000007eeee
Matej Zečević, Moritz Willig, Devendra Singh Dhami, and Kristian Kersting, Causal parrots: Large language models may talk causality but are not causal, 2023. URL https::/arxiv.org
abs/ 00ss/2308.22 13067
Nevin Lianwen Zhang and David L. Poole. A simple approach to bayesian network computations. 1994. URL https:::: / http://pp......issseeseeeeeeeeee
semant wew.nticcccolar..rooo.... wtt.://ccooooo....oooooooaaaaaaa
14

---

Published as a conference paper at ICLR 2026
A ADDITIONAL DETAILS ON DATA
association intervention counterfactual
Frequency
0.00
42345678
101112131411161
relevant q graph size
Figure 7: Distribution of query difficulty within each level, as measured by size of relevant subgraph defined in section
A.1
GRAPH MODIFICATIONS FOR EACH LEVEL
For an association level q of the form p(v, Vy By), we keep the same graph and query: M' M and q' = q.
For intervention level query of the form p( (vv
modify the graph and query. We make p(v),, and M' modification of M where we replace the mechanism for v, with constant function /j = c, resulting with F' = F U {ʃz = c). We also remove any incoming edges to Vy, resulting with G' (V, {(k k)
For a counterfactual query ((y VE = vk), we modify the graph and query. We create M' by first augmenting with twin copy of each endogenous node denoted ywin, This results with G" (V U Vtwin E. U E.tinn. Frwi mirrors F. but connects twin copy nodes. The mechanisms for twwi uses parents in /twin but share noise u, with the original V;- Then the same graph surgery and mechanism replacement is applied
the twin copy vi to account for the intervention. We then define q' p(yturin V& = vi) in M'. See (Shpitser Pearl 2012 for more discussions on twin network graph that handles two possible worlds (enough for setfing) and its generalization parallel worlds graph that handles more.
All the modified queries inference, e.g. variable elimination Zhang & Poole (1994), in M' to compute the answer.
association level queries,
standard graphical model
See fig. for example graph modifications of each level.
A.2 ADDITIONAL DATASET DETAILS
The training questions is sampled on 80 different graphical models, with 100 queries sampled per model. The development and test are sampled each 200 different graphical models, with 10/40 queries per model for dev/test.
B ADDITIONAL DETAILS OF EXPERIMENT SETUP
B. ADDITIONAL HYPERPARAMETERS
RLVR For GRPO variant, train 3B and 7B models for 7.5k steps, 32B models for 2.5k steps. For DAPO variant, we train 3B and 7B models for 2.5k steps, and 32B models for 850 steps. We the final checkpoint for all. These step heuristically chosen roughly control the amount of gpu
15

---

Published as a conference paper at ICLR 2026
hours spent each (32B is about 3x slower than 7B, and DAPO is roughly another 3x slower in terms of time per step due to filtering). For 3B models with GRPO and DAPO, the dev performance did not train further beyond
already plateaued early due to regression to direct prediction, 7.5k/2.5k steps, respectively.
Additional hyper-parameters parameters for GRPO variant include 0 coefficient the KL term, PPO clip ratio low of 0.2, high of 0.28, and с of 10, and token level averaging when computing the advantage function, following enhancements described in DAPO (aside from dynamic sampling) Yu et al. (2025b) and its implementation in VeRL (Sheng et al., 2024) We use Adam optimizer (Kingma & Ba 2017) with default parameters, weight decay of and no warmup steps.
For DAPO runs, we focus on its curriculum aspect. disabling the overiong buffer, and setting stricter filtering of accuracy between (0.09, 0.91) which corresponds more than out of 32 roll- outs being correct and more than 3 out of 32 rollouts being incorrect. We chose this filtering range hopes of seeing stronger effects with filtering compared to (0, 1) range originally used in DAPO Yuu et al.] (2025b). However, our results showed significant difference compared to GRPO. See results in appendix D with rows labeled with "cur" for curriculum.
C PROMPTS AND REASONING OUTPUTS
System Prompts fig. we include the system prompt for RLVR and SFT. fig. we include the system prompt with hint for counterfactual level.
User Prompt In fig. 13 we show full example of task input.
Reasoning Traces In figs. cremental", "Brute Force", "Neighbors", and "No Marginalization".
show reasoning traces of the four strategy categories "In-
16

---

Published as a conference paper at ICLR 2026
System Prompt RLVR
You are an expert on graphical models and causal inference. You task is to compute probability queries over structural causal model.
Format your solution follows:
THOUGHT PROCESS
All intermediate steps analysis, reasoning, and computation.
ANSWER
Only state your final answer to the query, via a list of numbers such as [0.1, 0.7, 0.2] or [0.1, 0.9]. Round you answer to the nearest two digits.
System Prompt SFT
You are an expert on graphical models and causal inference. You task is compute probability queries over a structural causal model.
Format your solution as follows:
Only state your final answer to the query, via a list of numbers such as [0.1, 0.7, 0.2] or [0.1, 0.9]. Round you answer to the nearest two digits.
Figure 8: System Prompt for RLVR and SFT.
System Prompt RLVR Hint For Counterfactual
You are an expert on graphical models and causal inference. Your task compute probability queries over a structural causal model (SCM).
## Model Specification
You will be given the specification of an SCM:
* The graph structure will be provided.
* The parametrization of mechanisms will specified with conditional probability tables (CPTs).
Each endogenous variable v.i is associated each possible joint assignment of its parents-that follows the same conditional distribution P(v par- ents(v_i)). Given joint assignment of its parents and the values of the corresponding exogenous vari- ables, the mechanism assigns vii the value of the appropriate exogenous variable.
collection of independent exogenous variables-one
## Counterfactual Queries
compute a counterfactual query:
Construct a twin network:
Copy all endogenous variables.
For each pair (v i. ioriginal, v_i.ooy). they share the same exogenous variables but connect their (original copied).
respective parent
Interpret the query as standard probability inference problem:
Observations are applied to the original variables.
Interventions are applicd to the copied variables.
The query target is also copied variable.
## Format
Format your solution follows:
THOUGHT PROCESS
All intermediate steps of analysis, reasoning, and computation.
ANSWER
Only state your final answer the query, via list of numbers such [0.1, 0.7, 0.2] [0.1, 0.9]. Round you answer to the nearest two digits.
Figure 9: System Prompt for RLVR with hint.
17

---

Published as a conference paper at ICLR 2026
THOUGHT PROCESS
1. Intervention c2: We need to compute the marginal distribution of el after setting v2 = 0. This need consider the distribution of e1 given x2 and sum over all possible values of v5 (since u5 a parent vl). (graph modification unnecessary for this case)
2. CPT for 81 given x2 0: The CPT for vll given v2 and v5 is: (copy probability values)
-P(vlle2 = 0, 25 = 0) = [0.83, 0.17]
P(vllee2 0, v5 1) j0.22, 0.788
need to sum over probabilities of (which (((5) = [0.49, 0.51J) Compute the marginal distribution: (marginalize over 225)
Marginalize over v5:
P(v1 = 0/ = 0) = P(v1 = 0/r2 = 0, v5 = 0) P(c5 = 0) + P(r1 = 0/ = 0, u5 1) P(55 = 1)
P(v1 = 1x2 0) P(r1 1|n2 0, z5 0) P(c5 0) + P(+1 = 1/ 0, v5 1) P(05 1)
Let's compute these values step by step. (substitution and arithmetic)
P(ul = 0/ = 0) = 0.83 0.49 + 0.22 0.51 = 0.4067 + 0.1112 = 0.5179
P(v1 1/v2
0.17 0.49 0.78 0.51 0.0833 + 0.3978 0.4811
Rounding to two decimal places. we get:
P(ul 0/e2 0) 0.52 P(e1 1/b2 0) 0.48
ANSWER [0.52, 0.48]
Figure 10: Illustration of sub-tasks involved in computing p(Vicogo;): V2 has no parents in this particular graph so p(v) = p(vi V2 = 0)). This an example "neighbors" marginalization strategy, performing summations over Vi 's immediate parents.
18

---

Published as a conference paper at ICLR 2026
LLM Prompt, Marginalization Strategy Categorization
You are an expert grader that never makes mistakes.
A4ब Input
You will be given LLM's SOLUTION for QUESTION that asks for the marginal distribution of some random variable #; under intervention, observation, or counterfactual (hypotherical intervention under an observation).
HM Analysis
correct strategy
marginalize over
correctly.
determine
strategy
of the following:
immediate: attempt to marginalize only over immediate neighhors
incremental: attempt marginalize neighbors well other distant variables, performing
marginalization incrementally, often following the graph structure and performing summations locally over ese subset of variables many
brute: attempt to marginalize over neighhons as well other more distant variahles, hut does sa by explicitly writing out a main formula that sums over the joint probubility distribution over ALL relevant variables together (which often involves many terms), instead summing over smaller subscts many times.
none: no attempt at explicit marginalization.
### Formatting Response
You need to ourput ane judgement specified helow: for that judgement, put any relevant EVIDENCE, which are excerpts from SOLUTION, within <ovidence>> </evidenee>> tag. then your EXPLANATION, any, within the eeaplaaaiios>-ceexpaatiioo tag. and linally put your judgement in </judgemnttt tag.
Overall Stralegy
Use <evidence, strategyy a/nnteeyy=</eevdeeeee h/eee//eeeeeeeeeetteeteee strategy'> amplanation_stratege olaeammmnn <judgement stratesyssrrrrr strjudgeeennttsrrateey""" Choose judgement between "immediate"" "Incremental", "bruce", "none",
<lexplamationnntrree strategy>. and
LLM Prompt. Probability Derivation Errors
You expert grader that makes mistakes.
atā Tnput
You will be given LLM's SOLUTION for QUESTION that asks the marginal distribution of random variable es under an intervention, observation, or counterfactual (hypethetical intervention under an observation).
### Analysis
A correct SOLUTION needs to perform derivations correctly and perform calculations conrectly. Your task is to identify any prob- ability derivation emrors derivation. you believe there are any erross precise identified.
solution
Probability derivation emcrrs include: errors when applying probability identities (eg. applying chain nule or bayes rule incorrectly, summing over too many too few variables marginalizing)
false assumptions (e.g. ignoring dependencies between variables when performing inference, ignoring observations interven- tions)
Probubility derivation emorrs does NOT include:
* in copying CPT values
numeric errrrs (e.g. incorrectly performing addition or multiplication)
##d Formatting Response
You need to ourput one judgement specified below: for that judgement, put any relevant EVIDENCE, which are excerpts from SOLUTION, within <evidence>> </evideace>> tag. your EXPLANATION, any. the <explamation </explantio>> tag.
lug. and limally put your judgement in
Derivation
Use <evidence derivation emor's <levidence derivation emor <explanation derivation emrer> c/explaatinn derivation emtrr> and jjuigement_lerivattoon derivation _mmor> _tjuuagemeetddervvvtttooooooope deerivation_emmr
Choose your judgement from "yes" (solution contains derivations, and contains probubiltty derivation emrors. "no" (solution contains derivations, no probability derivation emoons detected), "ula" (salution not contain any derivations).
Figure 11: Prompts for LLM Judge for strategy categorization and detecting probability derivation errors.
19

---

Published as a conference paper at ICLR 2026
LLM Prompt, Copy Error Detection
You are an expert grader that never makes mistakes.
A4ब Input
You will be given LLM's SOLUTION for QUESTION that asks for the marginal distribution of some random variable #; under intervention, observation, or counterfactual (hypothetical intervention under observation).
HHd Analysis
correct SOLUTION needs perform derivations correctly perform calculations conectly. task identify any copy- paste errors the conditional probability values used in the derivation. If you believe there are any errors the procise error loca- tion in the solution must identified.
### Formatting Response
You need to ourput one judgement specified helow: for that julgement, put any relevant EVIDENCE, which are excerpis from SOLUTION, within <evidence's </evidenee>> then your EXPLANATION, any, within the <explamation>> </expaanttt> tag. and finally put your judgement in soodgemmant=</fdddgeeee tag.
Conditional probaiiitty copy-paste Emor
Use <ovidence.cops wwwrroorrrrooooooooooooo w/wwiidenee.cop..oneee. <explanation.cop .eror> wtwp::///ppptttioooooooooooc and
<judgement.copy _erer>> wtw.in//ww//gtemmm...oo.....mm.m
Choose your judgement from "yes" (solution contains dorivations, and contains conditional probability copy-paste erroes), "no" (solution contains derivations, conditional prob- ability copy pasle eruis detected), or "nld" (solution does not contain derivations).
LLM Prompt, Arithmetic Error Detection
an expert grader never mistakes.
##ă Input
marginal distribution under intervention, observation, or counterfactual (hypotherical intervention under observation).
will given
SOLUTION for QUESTION
variable es
Analysis
A correct SOLUTION needs to perform derivations correctly and perform calculations correcll. Your task is to identify arith- metic errors in the derivation (when adding subtracting multiplying dividing numbers). It not considered an ervor to per- Form rounding intermediate steps. you helieve there any emois the precise location solution he identi- fied.
#Hd Formatiing Response
You need to ourput ane judgement specified helow: for that judgement, put any relevant EVIDENCE, which are excerpts from SOLUTION, within <evidence>> </evidence"> then EXPLANATION, any, within the <eexplanation>> </explanaiion tag. and finally put your judgement in <judeemaat></jddemmeet>>>> tag.
Arithmetic Error
Use <ovidence, arithmetic. ertoeerrrrrssoo <ievideneeeedeeeeeee...e.e miodd--cloiddeee..aiinieeee cettmmettic.cror>..... <explanation arithmetic antttnnicccnooc-cckkpaaaooooo.no http://saaaiioariiiieeccc.cooooo crror>>, and <judgement
etrer>> <fjudgement arithmetic emor>>.
Choose your judgement from "yes" (solution contains derivations, and contains arithmetic errors), "no" (solution contains dariva- tions, arithmetic detected), "a'a" (solution does any derivations).
Figure 12: Prompts for LLM Judge for detecting copy error and arithmetic error in solution.
20

---

Published as a conference paper at ICLR 2026
User Prompt Levels
Here's strictural causal midel over discrete random variables. The Variahles are vO, vl, v2, v3, v4, v5, vó, v7, v8, v9. Here
are Values they can take on.
v0 can take values in [0. !]
can
can take values can take values in
11
v5 can take values in [0. 1
v7 can take values in [0. [] can take values take values
Here's causal
acyclic graph (DAG):
strict digraph
v0: vI: V2; v3: v4: v5; vo; v7: v8; v9: vO: v3 v5: v4 + vl: v3;
v7; v8: 19: v6
Here are the causal conditional probahility tables (CPT) associated with the DAG:
CPTs for
P(v4) = [0.51, 0.49]
CPTs
P(v8) vd=0) 10.02, 0.98
P(v8) v4=1) [0.36, 0.64]
CPTs for v7:
P(y7) v8=0,y4-0) [0.94, 0.06]
P(v7 v8=0,y4=1) [0.25. 0.75]
P(v7 x8=1,x4=0) [0.49, 0.51]
v8=1,y4=1) [0.58, 0.42]
CPTs for v2:
v8=0) [0.11.
P(v2) v8=1) [0.97. 0.03)
P(v9) v4=0) [0.95, 0.05]
P(v9) y4=1) [0.42, 0.58]
CPTs for v3:
P(v3) v9=0,y44=0 [0.46, 0.54]
P(v3) v9=0,yda1) [0.61, 0.39]
P(v3) 19=1,y4=0) [0.7. 0.3]
P(v3) v9=1,v4=) (0.77. 0.23]
CPTs for v6:
v7=0,y8=0)
P(vó) v7=0,y8=1) [0.5, 0.5]
P(y6) v7=1,v8=0) [0.47, 0.53]
P(v6) v7=1,v8=1) [0.53. 0.47]
CPTs
P(v5) v3-0) [0.38, 0.62]
P(v5) v3=1) [0.66, 0.34]
CPTs for
P(v0) v3=0) [0.96. 0.04)
P(v0) v3=1) [0.86, 0.14)
CPTs for vl:
P(y]] v4=0) [0.59, 0.41]
P(y]] v4-1) [0.7. 0.3]
(Included Only Counterfactual
Furthermore, variable assumed depend deterministically on parents
pa(v) and collection of indopendent exogenous selector variables, one for each possible joint assignment to pa(v), whose marginal distribution defined to be ptv paív)). Given particular assignment to pa(v), v takes on the value of the selector variable corresponding that particular assignment pa(v).)
Here's your Question: (Question)
Now start your solation process. Be precise.
Association Question
What is the marginal distribution of iven it observed that v8=1?
Intervention Question
What is the marginal distribution of given we intervented to v8 to 1?
Counterfactual Question
What is the marginal distribution of given we first observed vó and then intervened set v8 to
Figure 13: Example user prompt Tuser containing causal graph and query.
21

---

Published as a conference paper at ICLR 2026
Incremental Marginalization
THOUGHT PROCESS
Given the intervention do(v1 0), the distribution of el is fixed O. This means need to consider the conditional probability tables (CPTs) for the other variables given O.
CPT for is given by:
P(16 vl = 0) = P(u6 v0 = 0, =11=0 0) if a0 0
P(u6 v0 = 1, = 0)
if v0 =
From the CPTs provided:
P(v6/00 0,
0.35,..33
P(u6 1, v1 = [0.04,
Next, we need to find marginal distribution s0 given the intervention do(ul = 0). The intervention does not change the distribution of e0 directly, so we use the original CPT for v0.
The CPT for e0 is given by:
pvv0/01 0, v5 [0.61,0.39)
P(v0/e1 v5 = (0.47,0.53
We need to consider the distribution of e5:
P(t5) 00.24, 0.76
Given vl 0, we need to find the marginal distribution of vO:
P = 0) P(v0 0'v1 0, c5 0) P(55 0) + P(v0 = 0/v1 0, 05 1)P(15 1)
P(v0 = 0) (0.61 x 0.21) + (0.47 x 0.76)
P(e0 0) 0.1464 + 0.3572 0.5036
P(v0 = P 0) - 0.5036 0.4964
Now, we can find the marginal distribution of e6:
P(u6 = 0) P(v6 0/e0 ((v0 + P(u6 0/o0 1)P
P(u6 = 0) (0.65 x 0.5036) + (0.04 x 0.4964)
P(u 0) = 0.32734 0.019856 0.347196
P(u6 1) Pvv6 0) 0.347196 0.652804
ANSWER 00.3472, 0.6528)
Figure 14:
Example trace with incremental marginalization. This example is from system x1 32b init, medium size intervention level query. Trace marginalizes first over v5, then over vO.
22

---

Published as a conference paper at ICLR 2026
Brute Force Marginalization
THOUGHT PROCESS
marginal distribution e1 given we compure P(u1/09 0). can be done using the law total probabdity and the given conditional probability tables (CPTs).
1. **Identity the relevani CPTs and variables:**
P(v1[07, v9)
P(v7]
P(v9)e0, v6)
P(u6)10)
P(v0)
2. MExpress P(e1/o9
the law of total probubility**
P(v1je9 =
Σ
P(cljeT, P(07|u6) P(09 o/b0, P(v6]e0) P(v0)
08,08,00
3. "oBreak down the computation:
Compute P(v9 00v0, v0) from the CPTs.
Compute P(c6 v0] from the CPTs.
Compute P(e0)
the
Compute P(e7 from the CPTs.
Compute P(el 17, og 0) from the CPTs.
4. **Compute each term:
P(v9 0/v0, v6);
P(v9 = 0|00 0, = 0.95, P(c9 0/60 06 1) 0.6,
P(w9 0/v0
P(19 0/00 1,06 0.32
P(v6)=0):
0.33. P[e6=1|v0 0.68,
P(e6=0 0.03, P(e6 1/60 = 1) = 0.97
P(v0):
P(v0= = 0.65,
P(00= 1) = 0.35
P(v7]16))
0[07=0;;==0)
P(07= 1|16=1166 1/16 0.88,
P(07 0/66 = P(07= 1/v6 1) = 0.29
P(u1 = 0/07 09 02 P(v1 = 1|177 U,
P(el 0/07
P(el 1|67 1. 0) 0.97
5. **Combine all terms
P(ul 0le9
Σ
P(v1 0) P(e7 P(e9 0,60, v6) P(v6/o0) P(00)
07.08,00
P(ul 1|-9 =
Σ
P(ul = 17, 09 0) P(07 P(09 0/.0 .0) P(v6)o0) P(v0)
w7,
6. **Caleulate the sums:es
P(u1 djva 0) (0.2-0.12.0.58) 0...0.22........ 0.32 0.600 0...0.......... 0.68 0.60 0.60-40.03
0.03 03 0.1.....11....33111.... 0...000.....3.....0000330333 0.03-0.88 38.00.320.97.00..
P(ul 1/ (0.-0.12-0.95-0.33 0...0..9....220006---.--------1. 0--.88---66600 0.1.06.00..00000110000010.000000 0.97 00888.08888.. 0.8.-0888-800000 www.0002-..00000.0.
7. **Perform the eM
P(rl 0/60 0.002496 0.030912 0.0000324 0.0078528 0.0412932
P(v1 1|29
0.0003816 0.285633
8. **Roand decimal places:**
P(el 0) 0.01, P(el 1/v9 0.66
ANSWER [0.04,0.m 0.66]
Figure 15: Example Trace with brute force marginalization. This example from system rl 7b init medium size association level query.
23

---

Published as a conference paper at ICLR 2026
Neighbor Marginalization
THOUGHT PROCESS
1. *"Intervention *2:** We need compute the marginal distribution of vl after setting v2 0.
This means we need to consider the distribution el given v2 and sum over all possible values of v5 (since v5 a parent v1).
2. #*CPT for vl given v2 = 0:++ The CPT for el given x2 and v5 is:
P(v1 [:2 = 0, v5 0) [0.83, 0.17)
PPvllee 0, v5 1) [0.22, 0.78]
**Marginalize over v5, We need sum over probabilities of v5 (which P(v5) [0.49. 0.51).
4. **Compute the marginal distribution:**
-P(v1 = 0/ 0) P( 0/v2 0, x5 0) P(c5 = + P(e1 0/:2 0, 1) P(05 1)
Pvvl = n22 0) P(r1 1|u2 0, v5 = P
P(r1 1/ 0, v5 1) P(05 1)
Let's compute these values step by step.
P(ul = 0 0) = 0.83 0.49 + 0.22 0.31 = 0.4067 + 0.1112 = 0.5179
P 1/02 = 0) = 0.17 0.49 + 0.78 0.51 0.0833 + 0.3978 = 0.4811
Rounding to decimal places, get:
P(e1 = 0/ = 0) = 0.52
P(el 1|22 0.48
ANSWER [0.52, 0.48
Figure 16: Example trace with summation over immediate parents. This example is from system x1 7b init. on medium size intervention level query. These solutions introduce ambiguity between brute-force summation and incremental summation since they are the trivial cases of both. Thus, we treat them separate category.
No Marginalization
THOUGHT PROCESS
The question asks for the marginal distribution of v7 given that v4=1. This means we need to compute the probability distribution of v7 without conditioning v4, but we can use the conditional probability of v4 given v5 find the marginal distribution of v7. ANSWER (0.31, 0.69)
Figure 17: Example trace with no explicit marginalization. This example from system rl 3b, on a small size association level query. Trace is inconsistent, and contains an error about ignoring v4.
24

---

Published as a conference paper at ICLR 2026
Reasoning Traces are Error Prone
THOUGHT PROCESS
marginal distribution given possible values (0 and 1) under the intervention. This involves marginalizing out the effect of v8 from the joint distribution of v2 and v8.
compute probahiiity distribution
The formula for the marginal distribution of v2 given v8=1 is:
P(e2 0/08 1) Σ P(o 0, P(e4)
P(e2= 1/08 1)
P(22 = 1, u= 1/w4)
We compute conditional probabilities P(v2 1/w4) and P(w2 1/e4) using CPTs for va.
Let's with P(u2 0, 1/ef): treated intervention observation
P(#2 0, 1|m4) 0/e8 1, s()8 1
P(v2 1, 1/w4) P(v2 1/e8 1, P(e8 1(44)
From the CPTs:
P(v2
P(v2 1 1|08
04) 0.03
P(u8 1/04) 0.64
This: assumed v4 I here and didn't do v4
P(v2 = 0, 1144 0.97 0.84 0.6208
P(v2 = 1, 1/11 0.03 0.64 0.0192
Now, need marginalize these
P(c2 0/18 1) = Σ
P(#2 = 0, v8 = P(e4)
P(02= 1018 1) Σ P(#2 08 ((o4)
Given the CPT for v4:
P(11= = 0) 0.51
P(14 = 1) 0.49
Socinvalid marginalization formula
P(n2 = 0/88 1) 0.6208 0.51 + 0.0192 0.49 0.316608 0.009408 0.326016 16
P(22 = 1/08 = 1) = 0.0192 0.51 + 0.6208 0.49 0.009792 0.299192 0.308984
arithmetic
Rounding to the nearest two digits:
P(v2 18 1) 0.33
P(v2 v8 1)
ANSWER
0.33, 0.31
Figure 18: Example trace illustrating probability errors and arithmetic errors made by model before RLVR for intervention query p(V2 ) Errors are annotated in red.
25

---

Published as a conference paper at ICLR 2026
D ADDITIONAL RESULTS
D.1 ACCURACY RESULTS
Accuracy Results In table we show within-level generalization on fil- tered
test
set.
In table we show within-level generalization on unfiltered we show results for across-level generalization for filtered results for across-level generalization for unfiltered set.
test
set. In table
3
set.
In table we
show
interv
Interv
Intery
soace nilee niov
0590C
as5Dc
assnc
councc
coume
couste
allv
n10v2
nitv3
niov3
nlov3
61052
any
medium
hard
eary motiun
hard
modiun
hard
(n-1006)
(n-044)
(b-39410)
il 32b
100Л001
99.315
85633
97.426
96.73
05,201
45.299
77,970
4.107
17.068
16,450
16-433
oar
101001
99,196
84.155
95.883
03.395
39,63
76.191
0,000
16:521
16,853
16,292
45.934
21:155
24.195
234554
20.367
19:87
15,152
18,118
st thh al 1h
99.948
91.566
58.621
96.419
81.354
46.635
33.515
$7,340
4,167
13.345
15,152
13624
09.724
91.566
54,823
88.561
80.523
48.896
35,309
56,984
0,000
14,804
14,719
13766
36 <0.7b
99.079
13.102
16.379
55,151
67.221
21.305
15,979
40,406
20,533
14,800
15.554
15,309
4.167
Table 1: Within level generalization (test, filtered). System accuracy (average CORRECT, see eq. (3) when training and evaluating on queries from same level. Stratified by query level, and difficulty within each level, measured by [Vall. the size of the relevant subgraph to the query variable. Note that difficulty is not comparable across different levels. The models are trained on a mix of small/medium/kare questions. Systems not significantly worse than the best (with a monte-carlo paired permutation test with n= are bolded.
interv
2000
21002
20002 s10b/?
Cousii e10k2
coust2 ri0l
COSSEE niox?
n1De2
nt0k2
rayy
mediia
Iaa
cayy
median
Tard
dl
raey
melinn
had
(om4861))
(sm2222)
Sem99111
8000003
(am3336))
(on1831)
cam7733
(om9000)
(mm2440)
(am4535)
(027)
(an90C0)
Tb i Th
996677
8E.583
$4.236
90.185
98,318
64.233
46,304
73,575
37.245
61.156
38.367
66.212
99,378
X3.979
51811
50 257
$9.670
64.161
48.50
73,525
36.763
62.453
38.663
66.757
$5.518
19:225
22.722
57.975
77.340
52.745
26.03
53.458
301 401
41.582
22.590
$1.000
Table 2: Within level generalization (test, unfiltered). System accuracy (average CORRECT, see eq. (3)) when training and evaluating on queries from same level. Stratified by query level, and difficulty within each level, as measured by [Vell. the size of the relevant subgraph variable. Note that difficulty is not comparable across different levels. The models are trained on mix of small/medium//arre questions. Systems not significantly worse than the best (with monte- carlo paired permutation test with n=10000) are bolded.
the query
26

---

Published as a conference paper at ICLR 2026
interv easy
interv medium
intorv
interv
assoc
assoc
assoc
assoc
m10v2
m10v2
n10v2
m10v2
m10v2
hard
all
mediom
hard
all
(n=1086)
(B=664)
(a=2098)
(n=1684)
(n=1868)
(s=388)
(==3940)
r1 32b asso
99,398
85,920
97.474
rl inte
95.487
62.152
38,660
74.086
rl 32b
100,000
98.946
80. 172
96.378
rl 3.b coun
96.378
62.152
34.794
74.086
32b rl init
99.908
91.867
52.011
89.418
32b
41.717
68.398
326 inte
72.031
26.927
24.485
45,964
*lt 32b coun
99.908
44.428
26.437
70.162
32h coun
57.423
26.927
22.938
39.569
32b
32h sft init
58.314
11.991
10.825
31,675
counte easy
counte
counte hard (n=211)
counte
n10v2
medium (n=457)
all
(n=24)
(n=712)
32b inte
0.000
17.287
16.883
16.573
32h
4.167
17.943
19.913
18.118
32h init
0.000
12.473
9.091
10.955
sft 8.333
14.004
13,420
13:624
sft 32b asso
4.167
11,160
12.554
11.376
32b sft
4.167
8.315
7.359
7.865
interv
interv
interv
interv
ass0c easy
assoc
ass0c
assoc
m10+2
m10x2
m10v2
m10v2
n10v2
easy
medium (n=664)
hard (n=348)
all
medium
hard
all
(n= 1086)
(n=2098)
(n=1684)
(n=1868)
(n=388)
(a=3940)
rl 7b asso
98.619
87.500
52.874
87.512
rl into
48,337
34,904
25.258
39.695
87.199
86.273
rl
55.641
31,156
23.454
40.863
7b rl init
61.510
51,958
16.092
50.953
7b el init
29.513
11.135
6.701
18.553
asso
99.263
19.729
18.391
60.677
97.514
18.825
22.126
60.105
sft 7b coun
31.116
18.737
18,299
23.985
sft init
24.862
4.217
0.287
14.252
2.677
4.797
counte
counte n10v2
counte n10v2
counte
n10v2
m10v2
easy
medium (n=457)
hard
all
(n=24)
(n=231)
(n=712)
r 7b inte
0.000
13,567
11,688
12,500
4.167
14.747
7b rl init
4.167
8.753
4.762
7,303
sft inte
8.333
6.127
6.494
6,320
sft
8,333
9.409
9.957
9.551
7b sft init
4.167
1.751
0.433
1.404
interv
n00v2
n10v2
m10v2
m10v2
m10v2
m10v2
n10v2
n10v2
medium
easy
medium
hard
all
(n= 1086)
(b=664)
(n=348)
(n=2098)
(n=1684)
(n=1868)
(n=388)
(n=3940)
r1 3b
77.072
9.337
5.747
43.804
rl 3b inte rl coun
41.627
11.456
11,340
24.340
r1 3b coun
74.125
8.434
5.172
41.897
9,636
9.794
20.88
zl init
24.309
10.392
16.492
3b el init
9.561
5.621
4.897
7.234
slt 3b asso
60.866
10.994
12.931
37.131
3b inle
38.717
15.953
15.979
25.685
3b coun
16.983
14.507
15.979
15.711
3b sft init
22.744
8.283
2,874
14,871
3b sft init
14.014
8,458
10.567
11.041
counte
n10v2
m10v2
10v2
n10v2
easy
mediumi
(n=24)
(n=457)
(n#231)
(n#712)
3b inte
0.000
5.689
8.225
6.320
3h asso
4.167
6.565
7.359
6.742
0.000
2.165
sft 3h inte
0.000
10.284
11.688
10.393
asso
4.167
10.390
9.551
3b sft init
0.000
4.814
3.030
4,073
Table 3: Across-level generalization (test, filtered). Row specify which level trained column specify which level evaluated on. System accuracy (average CORRECT, see eq. (3) on evaluation the query
sets of different difficulties, measured by |Veil, the size of the relevant subgraph variable. Note that difficulty is not comparable across different levels. The models are trained on mix of easy/medium/hard questions. Systems not significantly worse than the best (with monte- carlo paired permutation test with n=10000) are bolded.
27

---

the query Published as a conference paper at ICLR 2026
intery
interv
Interv
interv
assoc
a550c
assoc
Bssoc
al0v2
m1Uv2
m1Uv2
n10v2
m10+2
n10v2
m10v2
easy
medium
hard
all
easy
medium
hard
(n=4661)
(==2428)
(t=911)
(a=8000)
(a=3398)
(p=3831)
(g=771)
(1=8000))
asso
98.048
93.287
77.827
94.300
97.705
74,915
46,433
$1.800
32b
97.147
91.269
72.338
92.537
32b coun
98.175
73,532
41.634
80.925
32b init
95.559
77.636
44.566
84,312
init
93.320
sft 32b
27.442
sit 326 into
83.196
45.784
35.279
60.662
sit 326 coun
83.201
39.621
31.065
64.038
sft 32h coun
75.544
46.150
34.112
57.475
init
76.636
32h sft init
67.039
23.727
16.083
41.388
counte
counte
counte hard
counte
n10v2
n10v2 medium
n10v2
easy
all (n=8000)
(n=2440)
(n=4533)
(n= 1027)
r1 32b inte
92.131
68.564
41.967
72.338
r1 32h
93.033
70.042
43.720
73.675
32h r1 init
88.074
55, 306
140
61. 812
32b inte
79.426
48.577
28.627
55,425
sft 32b asso
76.967
48.68 68.687
29.309
$4.825
32b
64,590
31,370
14.411
39.325
interv
interv
interv
interv
8880c
assoc
8650c
assoc
m10v2
m10v2
m10v2
m10v2
m10v2
m10v2
n10v2
easy
mediumm (n=2428)
hard
easy
medium (m=3831)
hard
all
(n=4661)
(n=911 l)
(n=8000)
(m=3398)
(n=771)
(a=8000)
rl 7b assa
98.777
79.572
46.652
87.013
7b inte
72.543
54.346
33,982
60.113
7b coun
98.219
39.627
7b coun
75.044
49.987
30,999
58,800
7b init
62.969
32,80
14.380
48,300
7b init
37,905
14.748
7.134
23,850
sit 7b asso
79.897
858
21.625
55.950
si 7b
83.802
20.099
26.015
57.888
sft 7b coun
$8.711
35.787
29.053
44.875
7b <ft init
18.537
1.689
0.110
11,325
7h init
12,919
3.211
1.686
counte n10v2 medium (n=4533)
counte n10v2 hard
counte
n10v2
n10v2
rasy
all
(n=2440)
(n=
(n=8000)
7b inte
84.672
62,850
34.761
65.900
64,263
7b init
60.615
31,568
12:56]
37,988
xft 7b inte
73.361
35,892
19.182
45.175
sft 7b
65.533
40.393
19.669
45.400
7b sft init
31.148
9.618
1.753
15.175
interv
interv
n10v2
m10v2
m10v2
m10v2
n10v2
m10v2
n10v2
m10v2
easy
medium
hard
easy
medium
hard
(n=4661)
(n=2428)
(n=91 l)
(n=8000)
(m=3398)
(n=3831)
(n=771)
(n=8000)
3b
73.868
7.784
6.476
46.137
3b inte
60.153
23.388
17.121
38.400
3b coun
73.203
6.755
6.257
45.413
36 coun
57,004
21.874
15.045
3b init
35,250
6.260
4,281
22.925
3b init
26.574
10.232
7.393
16,900
sft 3b asso
71.680
12.891
17.124
47.625
sit 3b inte
60.212
28.974
24.125
41.775
coun
79.446
sft 3b
49.205
27.982
21.401
36.362
3b sft init
42.502
5.890
5.488
27.175
3b sft init
35.227
16.967
12.192
24.262
counte m10v2 mediumi (n#4533)
counte n10v2 hard
counte
n10v2
m10v2
(nw2440)
(n= 1027)
(n=8000)
3b inte
62.459
29.164
12.658
37.200
3b asso
60.615
29.274
12,074
36.625
init
6.134
18.200
sft 3b inte
62.459
31.348
15.774
38.838
sft asso
59.713
32.120
16.456
38,525
36 sft init
20.902
10.170
3.603
12,600
Table 4: Across-level generalization (test, unfiltered). Row specify which level trained on, column specify which level evaluated on. System accuracy (average CORRECT, see eq. (3) on evaluation sets of different difficulties, measured by |Veil, the size of the relevant subgraph variable. Note that difficulty is not comparable across different levels. The models are trained on mix of easy/medium/hard questions. Systems not significantly worse than the best (with monte- carlo paired permutation test with n=10000) are bolded.
28

---

Published as a conference paper at ICLR 2026
D.2 HINT RESULTS
Hint Results In fig. 19 we show the performance of LLMs after RLVR with hint in system p for the counterfactual level. It did not improve the simple system prompt (fig.
(fig. 9
Figure 19: Counterfactual level with hint. Prompting with hint about how to solve counterfactual queries by twin-network-graph is not enough induce genuine solutions and improve performance Post-RIVR. From left to right is accuracy breakdown on all, small, medium, and finally large problems. Having hint in the prompt did not significantly improve RLVR's performance on coun- terfactual level.
D.3 LLM JUDGE RESULTS
LLM Judge Results
In table 5 we show numerical results for strategy categorization, visualized in fig. 5 In table 6 we show numerical results for strategy categorization on unfiltered test set,
visualized in fig. 20 In table we show numerical results for strategy categorization on the hard subset of the filtered test set, visualized in fig. 21
reasoning incremental (n=80)
reasening bruie (n=80)
reasoning immediate (n=80)
reusoning Done (n=80)
derivation yes (n=80)
derivation error no (n=80)
derivation error na (a=80)
el 3b
0.000
0.000
20.000
80,000
41.250
31.250
27.500
rl 3b init
8.750
3.750
51.250
36.250
rl 3b init
80.000
17.500
2,500
r1 7h
32,500
0.000
35.000
32,500
7b
25,000
1,250
46.250
27.500
rl 7b init
40.000
57,500
2,500
rl 32b
33.750
0.000
36.250
30.000
rl 326 rl 326 init
5,000
95.000
0.000
28.750
5.000
33,750
32,500
5,000
95,000
0.000
rensoning incremental
reasoning hrute (n=80)
reasoning immediate (n=80)
reasoning none
derivation error yes (a=80)
emror
emoor na
(n=80)
(n=80)
(n=80)
(a=80)
0.000
17.500
r1
46.250
23,750
28.750
r1 3b init
6,250
8.750
66.250
18.750
rl 3b init
85.000
10.000
5.000
32,500
0.000
37.500
30.000
el 7b
12.500
18.750
67.500
1.250
el 7b init
67.500
32,500
0.000
r1
32b
46.250
3.750
28.750
21.250
32h
32b
rl 32b init
28.750
68.750
2,500
reasoning incremental (a=80)
reasoning brute (a=80)
reasoning immediate (a=80)
reasoning Done (a=80)
derivation errue yes (a=80)
derivation error (n=80)
enoo na (a=80)
0.000
0.000
17.500
82,500
r1
78.750
6,250
13.750
el 3b init
7,500
6.250
46.250
40.000
rl init
92.500
2.500
5,000
47.500
rl
88.750
11,250
0.000
r1 7b init
26.250
5.000
55 000000
13.750
rl 7b init
93.750
6.250
0.000
r 32b
45.000
8.750
38.750
7,500
32b
22,500
r1 32b
32,500
11.250
43.750
11.250
el 32b init
83.750
15.000
1,250
Table 5: LLM Judge Numerical Results Filtered Set. See fig. 5 for visualization. Top bottom: intervention, association, counterfactual.
29

---

Published as a conference paper at ICLR 2026
rcasoning incremental (==80)
reasoning brute (n=80)
reasoning immodiate (a=80)
reasoning
derivation (a=80)
derivation error no (n=80)
Done
eroor pa
(a=80)
(a=80)
0.000
0.000
12.500
87.500
rl
50,000
28.750
20.000
rl 3b init
10.000
5.000
51.250
33.750
e1 3b init
92,500
5.000
2.500
r1 7h
25.00 35.000
3,750
36.250
35.000
7b
20.000
7,500
46.250
25.000
rl 7b init
60.000
40,000
0,000
r1 32b
27.500
1,250
41.250
28. 750
#1 326
6,250
92.500
1.250
32b
s1 326 init
11.250
88.750
0,000
reasoning incremental
reusoning
reusoning immediate (a=80)
reusoning nane (a=80)
derivation
derivation emrer (m=80)
derivation
brute (a=80)
error yes (a=80)
emoir ma (a=80)
(a=80)
0.000
92.500
#1
53.750
27.500
18.750
r1 3b init
11.250
7,500
52,500
750
e1 init
88.750
8.750
2,500
28.750
0.000
37.500
33.750
40.000
57,500
r1 7b init
20.000
23.750
46.250
10.000
el 7b init
72,500
27.500
0.000
r1 3:2b
36.250
2,500
31.250
30.000
32h
r1 32h
30.000
11.250
27.500
31,250
rl 32b init
16.250
80.000
3,750
reasoning incremental (h=80)
reasoning brute (n=80)
reasoning
reasoning
derivation error yes (n=80)
derivation error (n=80)
derivation
immediate (n=80)
DOne
na
(n=80)
(a=80)
rl 3b
0,000
0.000
10.000
90.000
rl 3b
45.000
22,500
30,000
el 3b inat
6.250
0.000
46.250
47.500
r1
81.250
10.000
40.000
41.250
rl
46.250
52,500
1,250
r 7h init
17,500
6.250
36.250
40.000
rl 7b init
70.000
26.250
2,500
326
31.250
5,000
33.750
30.000
326
36.250
63.750
el 32b init
18.750
10,000
25.000
46.250
e1 326 init
45.000
47,500
6.250
Table 6: LLM Judge Numerical Results Unfiltered Set. bottom: intervention, association, counterfactual.
See fig.
20 for visualization. Top to
reasoning incremental
reasoning hrute (n=80)
reasoning immediate (n=80)
reasoning none
derivation yes
derivation error no (n=80)
derivation
emair na
(n=80)
(n=80)
(n=80)
(n=80)
el 3b
0.000
0.000
3.750
96.250
61.250
3b
11,250
12.500
48.750
27.500
rl 1.3 init
95.000
0.000
5,000
e 7b
97.500
0.000
2,500
0.000
47.500
52,500
1,250
rl init
85,000
13,750
1,250
rl
32h
93.750
5.000
1,250
0.000
rl 32h
32.500
67.500
0.000
32b
82,500
15.000
1,250
1.250
326 init
28.750
70.000
1.250
reasoning incremental (n=80)
reasening
reasoning
reasoning
derivation erfor yes (a=80)
dorivation error no (n=80)
derivation
bruie (n=80)
immediate (n=80)
Done
enror na (a=80)
(n=80)
0.000
0.000
6.250
93.750
48.750
1.250
48.750
c1 3b init
11,250
10.000
47.500
31.250
rl 3b init
91.250
0.000
8.750
r1 7h
67.500
1.250
20.000
11,250
10.000
r 7b
11,250
62.500
23.750
2,500
rl 7b init
91.250
8.750
0.000
rl
32b
63.000
27.500
5.000
2,500
rl 322
73.750
26.250
0.000
32b
26.250
rl 326 init
68.750
25,000
6,250
rensoning incremental (a=80)
reasoning hrute (a=S0)
reusoning immediate (n=S0)
rensoning none (a=80)
derivation errur yes (n=80)
error
na
(n=80)
(a=80)
0.000
0.000
8.750
91.250
r1
66.250
1,250
30.000
el 3b init
13.750
2.500
28.750
55.000
rl init
85.000
2.500
12.500
c
$7.500
3.750
35.000
3.750
85.000
15,000
0.000
rl 7b
23.750
13.750
50.000
12.500
el 7b init
96.250
3.750
0.000
r1 32b
71.250
7.500
18.750
2,500
3.32
r 32b
45.000
17.500
22.500
15.000
rl 32b init
75.000
15,000
10.000
Table LLM Judge Numerical Results on Large Complexity Split Filtered Test Set. See fig. for visualization. Top to bottom: intervention, association, counterfactual.
30

---

Published as conference paper at ICLR 2026
Intervention Queries
Association Queries
Counterfactual Queries
1.0
0.-
0.0
st ma
inl
7177b
33s vie
1.326
d ma
3b
n me
7b
1.330
#3 mi
1.38
e ene
7b
3330 into
3220
Norginsiization Strategy
Incremental (Comert)
Brute fome (Comett
Neightors (Conrett)
No Marginsizzatin [Caneet)
Incremental (Incorrect)
inncomect
Marginalizstion (ncorrect)
Intervention Queries
Asseciation Queries
Counterfactual Queries
Proputiio
117b
n 328
r1
# #32b 12300
# 3233
r130
# #22b
3b
r
Probsility Derivation
No Probabiliyy Erne [Conect]
Contant Pobubillty Eroo (Corrett)
Net Apolicatle (Conect) Not Applicaate innorrect)
No Probability Error Incorrezt)
Contains Probabliity Emor (ncorrect)
Figure 20: LLM judge analysis of reasoning strategy (top) and existence of derivation errors (bot- tom) before and after RLVR on unfiltered test set. Marginalization strategies are annotated on 80 samples per level. Derivation errors are also annotated on the same 80 samples per level.
Intervention Queries
Association Querles
Courterfactual Queries
0.6
0.4
11.6
32th ni 32b
117b d
328 32b
3
TD d
3.300 1226
Morginalizstion Stratogy
Incremental (Comect)
Brute Force (Corrett Brute Forre (ncomeett
Neighoors (Correct)
No Morginsitation (Comecti No Marginsiizet ion (ivormet))
Incremental (Inoomet)
Neightors Oncomeett
Intervention Queries
Association Queries
Courterfactual Queries
1.0
0.8
0.6
Propee
0.4
0.2
0.0
R1 mit
1:36
17 Init
lar
32b
Ini
Probabiiity Derivation d soit mit
1766 mit
1.10
1.30
iiit
3b
4.326 Init
320
Apolicable
Probability (nnormect)
Contann Probabiiiy (rcomeett)
Not Applicable (nncomect)
Figure 21: LLM judge analysis of reasoning strategy (top) and existence of derivation errors (bot- tom) before and after RLVR on the hard complexity queries. Marginalization strategies are anno- tated on 80 samples per level. Derivation errors are also annotated on the same 80 samples per level.
31

---

Published as a conference paper at ICLR 2026
D.4
PRECISION RESULTS
Precision Results
We plot precision of SFT vs. RL for all sizes and all levels in figs. 22 to 24
201
110
<20 052
C20 c20 1210
028 C20
620
Couce Duntes
Caddetataa
422
Figure 22: Accuracy by threshold E (0, 0.2] for 3B Models. æ-axis plots threshold for accuracy (the lower the stricter). y-axis is accuracy. From top to bottom different levels. From left to right are query complexities small, medium, large within each level.
32

---

Published as a conference paper at ICLR 2026
300
110 L10
<20 C.2
020: 600 1210 020 (he AND 200 CIN
Courerterss Oueres
Cou VaTiaaa
Co Covmertccial
120
KIC
020
www
Figure 23: Accuracy by threshold / E (0, 0.2] for 7B Models. æ-axis plots threshold for accuracy / (the lower the stricter). y-axis is accuracy. From top to bottom are different levels. From left right are query complexities small, medium, large within each level.
33

---

Published as a conference paper at ICLR 2026
18.81
020
CAD
200 1:0 sid 110 RIS C2B
022 CS2
2013
500
020 6.0
020 CUs
801 620
Courra Oueree
Cou Vaciual ComBe
Co Covmericcial Comte
130
020
www.
Figure 24: Accuracy by threshold / E 0.2] for 32B Models. --axis plots threshold for accuracy t (the lower the stricter). y-axis is accuracy. From top to bottom are different levels. From left right are query complexities small, medium, large within each level.
34

---

Published as a conference paper at ICLR 2026
D.5 SFT WITH REJECTION-SAMPLES COT
SFT with Rejection-sampled COT In our main experiment (section 4.2 fig. 3}. we found that 7b and 32b LLMs fine-tuned with RLVR generalizes significantly better than those fine-tuned with SFT on association and intervention level problems. How much of this gap is due to RLVR training on reasoning chains (which our SFT setup did not have)? How much of this gap is due to RLVR training on reasoning chains that are generated on-policy?
To understand the relative contributions of fine-tuning with reasoning chains and fine-tuning with on-policy data, we additionally fine-tuned Qwen2.5-7B-Insttee models on correct reasoning chains collected via rejection sampling from Qwen2.5-7B-Instuun itself. We perform rejection sampling across intervention, association, and counterfactual level problmes, and build supervised fine- tuning dataset for each level with reasoning chains for the same 8000 training problems that are used to perform RLVR and SFT in section 4 For each problem, we independently sample 32 reasoning chains at temperature 1.0 from
1aem2.5--B-Insttruee and keep only thains-offthoughhe that have a correct final answer. See fig. 25 for scatter plot of the proportion of correct reasoning chains (out of 32) for a problem versus the difficulty of the problem. We then fine-tune Qwen2.5-775 20-25-----Instrucc for epochs, and pick the checkpoint that has the highest accuracy on the dev set (described in section 4
Pranarior Leve
Aunderir Leve
Conterleder
Figure 25: Scatter plot of accuracy by [Vial, the size of the relevant subgraph to the query variable. Each point is a particular query from the (unfiltered) training set.
Within-level generalization of SFT on rejection-sampled reasoning chains (RS32) across three levels are included table Comparing Rs32 with COT INIT shows that fine-tuning on correct reasoning chains improves the within-level accuracy of the LLM. Comparing Rs32 with RL shows that it nonetheless underperforms RLVR, which uses online rather than offline data, and especially so on more difficult splits. Comparing RS32 with direct prediction SFT (DP), we see that fine-tuning with reasoning chains outperforms slightly on association and intervention levels, and is comparable on counterfactual. Overall, these results show that fine-tuning on reasoning -hainss and the on on-policy nature of RLVR both contribute to its superior performance on within-level generalization.
20022
2000::
200e
200c
nl0v2
n1052
s10v2
n1
61002
n10V2
n10k2
at0k2
n10V2
n1013
a10/2
x1bb2 all
onez
modian
hand
4ll
easy
mtddiuss
Itad
all
aay
madian
hard
(m:100))
(e:36-)
(n:54)
(1223000)
(694)
(s:1856)
(n:188)
(s:34)
(ro457)
(E:231
(x:2133
75
99.008
91.568
SK421
98.419
48.455
57,308
4.167
13.345
15,152
13624
003
20:311
21.352
75.834
50
19:47
14:3)
(U800)
18.941
9.987
10.235
ieit
61.510
16000
29.513
11.135
19.553
4.307
7.303
7b-dp 2b-dp-it
99,079
13.102
16.379
$8.151
67. 221
21.306
15:979
40.405
28.335
14,880
15.584
15,309
db-dp
24,962
4.277
0.287
14,252
7.895
2,577
1.546
4.797
4.167
1.551
0.433
1.404
Table 8: Within level generalization (test, filtered) of 7b models fine-tuned with various supervision sources: online chains-of-thouge (rl), offline chains-of-thoughe (rs32), answer only (dp). System accuracy (average CORRECT, see eq. (3) when training and evaluating on queries from same level. Stratified by query level, and difficulty within each level, as measured by |Veel, the size of the relevant subgraph to the query variable. Note that difficulty is not comparable across different levels. The models are trained on a mix of small/medium/lage questions. Systems not significantly worse than the best (with a monte-carie paired permutation test with n=10000) are bolded. rs32 is fine-tuned on chains -ftthuught collected via sampling 32 times per training problem and keeping only correct ones.
35

---

Published as a conference paper at ICLR 2026
interv easy
interv
interv
interv
m10v2
m10v2
n10v2
medium
hard
all
(n=1086)
(1=664)
(m=348)
(a=2098)
asso
98.619
87.500
52.874
87.512
7b sft rs32
89,963
63.404
21.264
70.162
7b rl coun
99.079
87.199
44,540
86,273
94,383
73.343
22.701
75.834
7b cut init
61.510
$1,958
16,092
50,953
7b sft asso
99.263
19.729
18.391
60.677
7b dp coun
97.514
18.825
22.126
60.105
7b init
24.862
4.217
0.287
14.252
n10v2
n10v2
m10v2
n10v2
medium
(n-1684)
(n= 1868)
(n=388)
(n= 1940)
7b inte
48.337
34,914
25,258
39.695
sit rs32 inte
67.043
24.732
17.784
42.133
7b cown
55.641
31,156
23.454
40.863
7b sft 1532 coun
43.943
22.645
15,464
31,041
dp sft
56.116
18.094
21.907
34.721
7b dp sit coun
31.116
18.737
18.299
23.985
7b dp init
7,898
2.677
1.546
4.797
counte
counte
counte m10v2 hard (n=231)
counte
m10v2
n10v2
m10v2
casy
medium (n=457)
all
(n=24)
(n=712)
7b el inte
0,000
13,567
11,688
12,500
rs32 inte
4.167
11,597
10.390
el asso
4.167
14.880
15.584
14.747
all r33 asso
4.167
10.503
7.359
9.270
init
4.167
8.753
4.762
7,303
sil 7h dp sft asso
8,333
6.494
8,333
9.409
9.957
9,551
dp
0.433
Table 9: Across-level generalization of 7B models fine-tuned with RLVR, SFT with reasoning chains, and SFT with direct prediction (test, filtered). Row specify which level trained on, column specify which level evaluated on. System accuracy (average CORRECT, see eq. (3) on evaluation sets of different difficulties, variable. Note that difficulty is not comparable across different levels. The models are trained on mix of easy/medium/hard questions. Systems significantly worse than the best overall (with monte-carlo paired permutation test with 1=10000 are bolded.
measured by |Vral, the size of the relevant subgraph
the query
Across-level generalization SFT on rejection-samples reasoning chains (RS32) across three levels included table 9 Comparing Rs32 with COT INIT shows that fine-tuning correct reasoning chains improves the performance of the L.M when evaluated on a different level from training. Comparing Rs32 with RL shows that it underperforms RLVR on across-level generalization except slightly on the intervention to association direction, and again the gap is especially large on more difficult queries. Rs32 outperforms direction prediction SFT (DP) on average on intervention levels and association levels. Overall, these results show that fine-tuning on geasoning-chain and the on- policy nature RLVR both contribute to its superior performance on across-level generalization.
36

---

Published as a conference paper at ICLR 2026
D.6 DETERMINISTIC COUNTERFACTUAL PROBLEMS
Deterministic Counterfactual Problems In our main experiment (section 4.2 fig. we found that no method generalizes reliably the counterfactual level. Our counterfactual level problems require abduction and marginalization, which is challenging for the LLMs even at small graph sizes. To understand the limitations of LLMs on this level better, we perform an evaluation of the fine- tuned models on a set of simpler counterfactual queries that are have small graphical models with deterministic mechanisms that remove the need of marginalization. See fig. 26 for an example problem.
User Prompi Deterministic Counterfactual
Here's structural causal model discrete random variables. The Variablos vO, vl, v2, Here the Values they can lake on.
vo take values in 10. 1]
vl can take values in |0. 11
v2 take values [0, 1]
Here's the causal directed acyclic graph (DAG)) striet digraph {
vo: vl:
v2:
vl:
V2 vI:
Here the causal conditional probability tables (CPT) associated with the DAG:
CPT's for v2:
P(v2) 10.71, 0.29]
CPT's for wo:
P(v0) 10.08, 0.92]
CPTs for wt:
Pivl
v2=0.v0=st 11. 0]
Ptvl
v2=0.0==) = [0. 1]
10.
P(vl
v2=1,v0=1 [0, !]
Furthermore, variable assumed depend deterministically parents po(v) collection independent
exogenous selector variables, one for each possible joint ussignment to pa(v), whose marginal distribution is defined to be r(v pa(v)). Given particular assignment pa(v), on the selector variable correpponingg particular assignment pa(v).
Here's your Question: 0?
marginal distribution of vl given observed
Now start your solution process, Be precise.
Figure 26: Example user prompt Zuser containing causal graph and query converted from CLadder [Jin al. 2026) deterministic counterfactual subset. The original question was "We know that blowing out the candle or candle with wax causes dark room. We observed the candle is out of wax. Would the room is dark if blowing out the candle instead of blowing the candle?" (question ID 9538). The answer is "no"", or [1, 0] in our format.
Specifically, construct the evaluation data by selecting a subset of problems from the "deter- ministic counterfactual" subset of the CLadder dataset (Jin et al.) 2023). The det-counterfactual subset consists of 1422 counterfactual queries on small graphs (3 to 4 endogenous nodes) with de- terministic mechanisms. We keep only questions that have exactly one observation and action (e.g. p((,==)) z = 1)) to match problems. Instead of the natural language format, represent the problems formal problems, to match our training and test sets.
setting of our training and test sets. This leaves us with 790
Comparing RLVR and SFT models in the deterministic setting (table 10 left), find that LLMs fine-tuned with RLVR (with reasoning) are significantly better than those fine-tuned with SFT (with direct prediction), which is opposite to the ordering of models tual test set. Overall, regardless the fine-tuning method. LLMs also generally perform much better on the simpler setting of deterministic counterfactuals (table T0 left) than our counterfactuals that
more challenging counterfac-
37

---

Published as a conference paper at ICLR 2026
require abduction and marginalization (table rightt This contrast suggests that the requirement to perform nontrivial abduction followed by marginalization is a major challenge of the counterfactual problems in our dataset.
claikler det-counteriactual
counte n10v2
(n=790)
(n=24)
#1 326
99.747
32h
4.167
32b curriculum
0.000
r1 32h init
98.987
e1 32b init
0.000
7b
84.937
4.167
7b curriculum
84.937
7b curriculom
0,000
7b init
67.468
7b init
4.167
3b
64.051
3h
4.167
:1 3b curriculum
57.722
3h curriculom
0.000
3b init
0.000
sft 32b
70.633
sit 32b
29.167
xiI 32b inil
81.392
sft 32b init
4.167
sft 7h
61.139
sit 7b
20.833
sft 7b init
43.038
xi 7b init
4.167
sft 3h
0.000
sit 3b init
47.089 sft 3b init
0.000
Table 10: Performance of models trained with our counterfactual training split, and tested on the CLadder (Jin et al., 2023) det-counterfactual subsplit which consist of counterfactual queries on small graphs with deterministic mechanisms. We include queries with exactly one observation and action (790 out of 1422), and represent the questions formally rather than in natural language (in CLadder) to match our setting. Systems not significantly worse than the best (with a monte-carlo paired permutation test with n=10000) are bolded.
D.7 COPY AND ARITHMETIC ERRORS
LLMs to analyze the reasoning trace of fine-tuned models, and categorize their high level marginalization strategy as well detect probability derivation errors (e.g. falsely assuming independence among variables, missing terms in marginalization formulas). To have a more complete understanding of the error patterns, extend the LLM-aided error analysis to copy errors and arithmetic errors. Copy errors are errors where probability values are copied incorrectly from the problem statement (e.g. substituting an incorrect value for a term like p(v1 numerical errors in addition, subtraction, multiplication, or division. We use 04-mini to analyze on the reasoning traces, using prompts in fig. [2] to detect copy and arithmetic errors respectively. The first author checked 10 sample annotations per category for copy (has errors, no errors, not applicable) and arithmetic errors (has errors, no errors, not applicable) and found that they agreed with the annotation 25/30 and 23/30 respectively for copy error and arithemtic eπrоr.
Copy and Arithmetic Error Analysis In section (Analysis-III),
V22 1) during calculation). Arithmetic errors are
The analysis of RLVR traces are visualized in fig. 27 First, fig. 27 (top) show that copy er- rors generally decrease after training expected. We also that for 7b and 32b models and especially on the association and counterfactual levels, examples without copy errors (pink) are nonetheless often incorrect (pink with shade), suggesting these examples suffer other kinds of rors that lead to wrong answers. Next, fig. 27 (bottom) show that arithmetic errors reduce but not by large magnitude after fine-tuning. We also that for 7b and 32b models on intervention and association levels, traces with arithmetic errors (green) are nontheless often marked correct (green without shade). This likely due our correctness function (used during training and
We re-ran the rl 7b training due a lost checkpoint, and report the cladder det-counterfactual performance of this new
38

---

Published as a conference paper at ICLR 2026
evaluation)
check answers rounded to the nearest 0.01, which allows for slight numerical errors.
Interventian Queries
Association Queries
Caunterfactual Queries
1.0
0.8
0.6
Poportion
Prpponttn
0.4
6.
0.0
r1 tut
mee
326
7666
n178 coet
d mit
d.30b
3: loue
3b
I1 ins
re
3330 Init
320
Derivation
Abplicable
Copy Incorrec)
Uncomecti
Asplicable (Inoorect)
Intervention Queries
Association Queries
Courterfactual Queries
1.0
0.8
proporton
Poparton
8.30
7h Iole
77
32th mit
326
# 3b ma
1778 sout
3.2h mit
a 1.3 t0le
hii
313b
Arthmetic Derivation Enorr
No Arithmetic Erer (Conect)
Cortans Arthmmti Eror (Corrett)
Not Asplicale (Corret)
Arthmeticc
Arthmetic
Acpliciable
Figure 27: LLM judge (o4-mini) analysis of copy (top) and arithmetic errors (bottom) be- fore and after RIVR, on 80 samples per level. Judge prompts (including category definitions) are included fig. 12
39

---

Published as a conference paper at ICLR 2026
D.8 ERROR IMPROVEMENT RLVR vs. SFT WITH REASONING CHAINS
Error improvement of RLVR vs. SFT with Reasoning Chains In fig. 28 we compare the re- duction of various kinds of emrrs achieved by RLVR versus SFT with reasoning chains based 7B models (see appendix D.5 for details about the setup), and find that RLVR is generally more effec- tive at reducing all of probability, copying, and arithmetic errors, and especially so for probability derivation emrors.
Intervention Queries
Association Queries
Courterfactual Queries
1.0
0.8
0.6
0.4
0.2
11 ma
m33
#' Hit
Ya r3
d
11
Derivation Probability
Probability
Apolicable
Pribailiiy (nnerrect)
Cortans Probibiiiy froor Orcorrett)
Net Applicable (Inoorect)
Intervention
Association Queries
Counterfactual Queries
1.0
0.8
Propertion
117b
7b
n1
77
m3
dt To ro32
copy Derivation Ener Contain (Corect)
No (Conect) (nnorrest)
Not Arplicable (Corect) Not Applicable (Inoorect)
(inconecti
Intervention Queries
Association Queries
Courterfactual Queries
Proparton
Proput
0.2
0.0
777b mi
J8 76 raz
7b
77 mi
7b m33
#7b
n wit
7h reaa
m
Arthmetic Derivation
No Arthmmetic Fror (Conect)
Contans Arthmmeie frre (Conect)
Not Applinable (Corect) Not Asplicable (ncomrett
No Arthimetic Erer Incorrezt)
Contains Arthmmttc Emor linoomecti
Figure 28: LLM judge (o4-mini) analysis of derivation (top), copy errors (mid), and arithmetic errors (bottom) before and fine-tuning a 7B LLM, on 80 samples per level. Judge prompts (including category definitions) are included in fig.
and fig. 12
40

---

Published as a conference paper at ICLR 2026
D.9 3B RLVR WITH MORE TRAINING STEPS
Training 3B models for longer with RLVR to match 7B and 32B compute leads to only minor improvements and no qualitative change in marginalization strategy. We trained 3B models for 7500 steps in our main experiments, which uses about a quarter of the compute compared to 7B and 32B training. We perform an experiment extending their training steps to 30000 steps match the compute. The results are included in table TT we can see the additional steps lead to minor improvements in aceuracy across levels. In fig. 29 we also compare the marginalization strategy of 3B model trained with 30000 steps of RLVR versus those trained with 7500 steps and find little qualitative difference.
Inary
3007
2000
05100
E5500
ESSC
couste
nf0k2
610x2
010.2
ni0x2
z10k2
nl042
al0+2
a10v2
n1042
c10r2
nI0N2
a10i
clsy
modimm
bart
al1
cady
mnlium
hsed
<B1
methm
bart
(m-1
(cx064)
(mm34)
(mm2958)
(c+1604)
(mm1855)
(e=358)
(ne3
(ax24)
(a-457)
(mx21)
(p-712)
Akkk
20048
47.208
Table 11: Within level generalization (test, filtered) of 3b models fine-tuned with rl for 7500 and 30000 steps. System accuracy (average CORRECT, see eq. (3) when training and evaluating on queries from same level. Stratified by query level, and difficulty within each level, measured by |eel the size of the relevant subgraph to the query variable. Note that difficulty is not comparable across different levels. The models are trained on a mix of small/medium/large questions. Systems not significantly worse than the best (with a monte-carlo paired permutation test with n=10000) are bolded.
Intervention Queries
Association Queriess
Counterfactual Queries
1.0
0.6
Propertion
6.-
0.0
mnt
36 stane
3b stepd
mi
3b 1.kk stace
#e cteps
miit
te steps
r13b steee
Manginelication Strategy
Incremental (Conect)
(Correct)
Naighoors (Correct)
Marginalization
Incremental (Incorrec)
Brute Force Uncomeect
Naighbors Oncorrecti
No Marginalization (inorrect)
Figure 29: LLM judge (o4-mini) analysis of reasoning strategy of 3B RIVR trained with 7.5k steps and 30k steps, on 80 samples per level. Judge prompts (including category definitions) included in fig.
41


Reviews:
[{"review_id": "rnaoBOd4FL", "reviewer": "Reviewer_yW6b", "summary": "This paper investigates the generalization capabilities of RLVR for post-training LLMs, using probabilistic inference in causal graphical models as a testbed. The authors fine-tune Qwen-2.5-Instruct models (3B-32B parameters) using both RLVR and SFT on datasets of causal graphs and queries spanning three difficulty levels and complexity. The work contributes to understanding the conditions under which RLVR effectively generalizes, highlighting both its strengths and limitations in challenging formal reasoning tasks.", "questions": "- Can you provide results for SFT trained on reasoning chains (even if just for 7B/one level)?\n- Can models solve trivially simple counterfactuals (e.g., 3-node graphs, no marginalization needed)?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The choice of probabilistic inference in causal graphical models as a testbed is genuinely innovative. Unlike prior RLVR generalization studies that focus on text/visual reasoning tasks, this formal mathematical domain enables precise control and analysis.\n- The findings have practical implications: practitioners should check if their base model has sufficient reasoning capability before investing in RLVR. The identification that counterfactual reasoning remains unsolved even with RLVR and 32B models highlights a key challenge for the field.", "weaknesses": "- SFT is trained only to predict final answers while RLVR generates full reasoning chains. This creates an asymmetric comparison that conflates two factors: (1) reasoning vs. direct prediction and (2) RL vs. supervised learning. A fair strategy is to include an SFT baseline trained on optimal reasoning chains (generated by the solver or sampled from successful RLVR rollouts). This would isolate whether gains come from RL exploration or simply having reasoning chains.\n- The paper observes 3B models fail to benefit from RLVR and regress to direct prediction after training, but doesn't investigate what specifically these models lack. \n- The reward is simply r = 0.8 · accuracy + 0.2 · format with threshold t=0.01. Is the 0.8/0.2 weighting optimal? Would shaped rewards (partial credit for intermediate steps) help?\n- The writting is vague. For example, the description of the third finding \"What did RLVR learn? \" is too abstract, containing too many technical terms. They should present this result in a concrete style.\n- Why not compare RLVR with other RL-based post-training paradigm, such as RLHF and RLAIF?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1w1Vz8gthN", "reviewer": "Reviewer_5AJc", "summary": "The paper investigates the relationship between large language models’ (LLMs) post-training mechanisms and their ability to perform causal reasoning tasks. The author introduces a data generation process to create and evaluate causal question-answering scenarios. The study then compares models post-trained using Supervised Fine-Tuning (SFT) and Reinforcement Learning from Verbal Reward (RLVR, including GRRO and DAPO variants) across different evaluation dimensions.", "questions": "N/A", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well-structured and clearly written, with a logical flow that makes it easy to follow the author’s reasoning. The analysis sections are particularly strong: insightful, well-grounded, and supported by detailed experiments. The results are extensive and could serve as a valuable reference for future researchers studying the intersection of LLM post-training and causal reasoning.\nI was initially debating between a rating of 6 and 8 and currently lean toward the former, though I remain open to adjusting this based on the rebuttal and other reviewers’ feedback.", "weaknesses": "1. The RLVR experiments use 7.5K and 2.5K samples, while the SFT model is trained on 5K samples. This discrepancy makes the quantitative comparison between models less reliable. I suggest adding an ablation study where models (or checkpoints) are trained on the same amount of data and for comparable GPU hours to mitigate this concern.\n\n2. LLMs learn differently from humans as they rely primarily on language pattern recognition rather than true causal inference. The proposed causal reasoning tasks implicitly require two skills: (a) retrieving or identifying the correct numerical values from context, and (b) performing basic computations. It would strengthen the paper to include a detailed error analysis comparing SFT and RLVR models on these sub-tasks, in addition to the LLM-judge results that emphasize human-like problem-solving performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "82sOzi0utq", "reviewer": "Reviewer_Cf3X", "summary": "This paper studies the generalization of reinforcement learning with verifiable rewards (RLVR). The authors selected causal reasoning as the probing task. Specifically, authors generate a benchmark of causal graphs alongside corresponding questions. With respect to the experiments, the authors make a comparison between SFT and RLVR and find that RLVR outperforms SFT on some subsets of the benchmark. Finally, the authors conclude that RLVR indeed enhances the generalization abilities of causal reasoning at the association and intervention level.", "questions": "Please refer to the weaknesses part.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "1. The experiments are comprehensive and sufficient to verify the authors' claims.\n\n2. The analysis is comprehensive, authors provide in-depth analyses.", "weaknesses": "1. I think the first weakness lies in the writing of the abstract. The current version seems a bit colloquial rather than a formal academic paper. Specifically, the sentence \"We choose this setting because causality is an important area that LLMs still struggle with, and because this setting ...\" is too colloquial and lengthy. I suggest authors consider rewriting the abstract thoroughly and consider using shorter sentences.\n\n2. Similarly, in the introduction, the sentence \"However, we focus on identifying situations in which RLVR itself generalizes effectively\n(versus not), and we focus on the causal reasoning domain.\" may be informal. Authors employ \"we focus on\" twice and \"(versus not)\" may be a little informal. Besides, \"in which RLVR itself generalizes effectively\" can be confusing, what dose RLVR itself mean here? I would encourage authors to revise the introduction throughly. For example, \"our work differs from prior studies by focusing on an essential and challenging task: causal reasoning.\" \n\n3. I would suggest that authors include a discussion on the practical value of the studied formal causal reasoning. Since I believe LLMs are more targeted at the commonsense setting, and there already exist lots of formal causal tools in the area of causal inference.\n\n4. Authors should consider assigning a formal name for their datasets (e.g., RLCausal or other names), since it is an important contribution of this work.\n\n5. As there already exist other formal causal reasoning benchmarks (e.g., the CLADDER [1]), I would suggest that authors add an individual section on the differences between their newly proposed datasets and existing benchmarks. Why can't other benchmarks test the generalization abilities of RLVR?\n\n6. The colors in Figure 1 are too light; the authors should consider deepening them. The text is too small, and it's unnecessary to show every detail of the sample; simplifying the content would allow for a larger font size. Besides, I do not quite get the main structure of the Task Formulation in this version. Authors should revise Figure 1 to a more abstract and summarized representation.\n\n> [1] Jin Z, Chen Y, Leeb F, et al. Cladder: Assessing causal reasoning in language models[J]. Advances in Neural Information Processing Systems, 2023, 36: 31038-31065.\n\nHappy to raise the score if the authors could address my concerns.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)