RuleReasoner: Reinforced Rule-based Reasoning via Domain-aware Dynamic Sampling
Authors: Yang Liu
Jiaqi Li
Zilong Zheng
Venue: ICLR 2026 Poster (2026)
Keywords: logical reasoning
rule-based reasoning
reinforcement learning
language models
Primary area: foundation or frontier models, including LLMs
TL;DR: RuleReasoner, a reinforcement learning method for reasoning models, beats frontier models at rule-based reasoning (ID/OOD) using effective curriculum learning and is more efficient.

Abstract: Rule-based reasoning is acknowledged as one of the fundamental problems of reasoning. While recent studies show that large reasoning models (LRMs) have remarkable reasoning capabilities enhanced by reinforcement learning (RL), real applications still face severe challenges due to variations in rule formats, types, and complexity. To mitigate this issue, we introduce RuleReasoner, an effective method for rule-based reasoning via a wide collection of curated tasks and a novel domain-aware dynamic sampling approach in RL. Specifically, RuleReasoner resamples each training batch by updating the domain weights based on historical rewards. This facilitates domain balance and active learning schedules for RL, obviating static mix-training engineered by humans. Evaluations of in-distribution (ID) and out-of-distribution (OOD) benchmarks reveal that RuleReasoner outperforms frontier LRMs by a significant margin ($\Delta$4.1% on eight ID tasks and $\Delta$10.4% on three OOD benchmarks over OpenAI-o1). Notably, our approach also exhibits higher computational efficiency compared to prior methods.

Published as a conference paper at ICLR 2026
RULEREASONER: REINFORCED RULE-BASED REA- SONING VIA DOMAIN- AWARE DYNAMIC SAMPLING
Yang Liu' Jiaqi Li Zilong Zhengi
State Key Laboratory of General Artificial Intelligence, BIGAI {liuyang, 1ijiiiijee jiagi, z Azhhnng/bbiaal
ABSTRACT
Rule-based reasoning is acknowledged as one of the fundamental problems of rea- soning. While recent studies show that large reasoning models (LRMs) have re- markable reasoning capabilities enhanced by reinforcement learning (RL), real ap- plications still face severe challenges due to variations in rule formats, types, and complexity. To mitigate this issue, introduce RULEREASONED method for rule- bale-based reasoning via a wide collection of curated tasks and a novel domain-aware dynamic sampling approach in RL. Specifically. RULEREASONER resamples each training batch by updating the domain weights based on historical rewards. This facilitates domain balance and active learning schedules for RL, ob- viating static mix-training engineered by humans. Evaluations of in-distribution (ID) and auttof-ddisrrbuttinn (OOD) benchmarks reveal that RULEREASONER out- performs frontier LRMs by significant margin (44.1 % on eight ID tasks and A A10.4% on three OOD benchmarks over OpenAl-ol). Notably, our approach also exhibits higher computational efficiency compared to prior methods.
effective
RulcReasener-8B
RuleReasoner-4B
OpenAI-ol
OpenAI-o3-mini
Tlaude33.--Somnc
DeepSock-RI
100
99.6
95.4
92.6
97.9
93.8
Pass @ (%)
78.0
76.6
74,2
80.0
69.1
78.7
68.9
65.5
55.0
59.2
49.2
50
40.6
40.8
BigBench Hard
ProverQA
BigBench Extra Hard
performance comparison between RULEREASONER (8B and 4B) and other frontier reasoning models on challenging cule-based reasoning benchmarks.
Figure 1:
INTRODUCTION
Rule-based reasoning (Xu et al., 2024b; Wang al., 2024c; Servantez et al., 2024h; Morishita al., 2024; Wang et al., 2024d) is an ability to draw new conclusions or make decisions based on provided facts and predefined logical rules, which requires a strong ability of reasoning. It emulates human reasoning and mirrors the structured deductive processes that humans employ in domains such as law, finance, and medical diagnostics (Liu et al., 2023b; Xiong et al., 2024; He et al., 2025b). The need for rule-based reasoning increasingly grows in scenarios that require transparency, explainabil- ity, and adherence to domain constraints. Moreover, deviations from rules for different scenarios lead to significant changes in reasoning process, which requires reasoning capabilities under ever-changing circumstances (Saparov et al., 2023; Tang et al., 2023a; 2024; Li al., 2025b).
controllable and adaptable
Recent work has demonstrated the remarkable reasoning capabilities of large reasoning models (LRMs) with an intermediate thinking process, chain-of-thought (CoT) (Wei et al., 2022b), notably
'Equal contribution. *Corresponding author.

---

Published as a conference paper at ICLR 2026
the long thinking ability stimulated by reinforcement learning (RL) (Xie al., 2025; Meng al., 2025; Guo et al., 2025; Hu et al., 2025). However, conventional approaches rely closely on scaling to larger model sizes supervision distilled from superior models. What's more, as the contextual window expands, language models encounter difficulties like "lost in the middle" (Liu et al., 2024; An al., 2024) that leads to performance collapse in long-dependency reasoning tasks (Li al,, 2024; Mao et al., 2025). The limit exhibits relatively weak instruction following abilities to under- stand and associate rules and facts provided in the context (Srivastava al., 2025; Li al., 2025d; Guan et al., 2025) required for task completion.
With this in mind, we aim investigating whether is feasible and effective to enhance the rule- based reasoning ability of language models. We also want to validate that this improved capability can generalize to various and unseen reasoning tasks, forms, and difficulties. Therefore, we propose RULEREASONER, which trains language models to be rule-based reasoners. It performs on on-policy RL and mitigates its limitations of sample inefficiency and the application of rigid rules in dynamic contexts. Specifically, RULEREASONER leverages task reward to estimate domain sampling weights without requiring human prior knowledge or high computational costs from repeated rollouts. First, RULEREASONER initializes to train model in standard Reinforcement Learning with Verifiable Rewards (RLVR) way. Second, within a training iteration, RULEREASONER updates the domain weights based on historical rewards of previous training steps. Finally, RULEREASONER samples training batch based on the domain weights to perform policy optimization.
Our innovations summarized three folds:
Novel Rule-centric Dataset:
We release a large and diverse dataset, RULECOLLECTION-E which spans eight rule-based reasoning tasks with explicit or implicit contextual rules tied each question. These rules vary in format, reasoning forms, and complexity, allowing training and ID/OOD evaluation generalizable rule application rather than memorization.
RLVR for Rule-based Reasoning: ing regularization to achieve stable RI. training dynamics on complex rules, even with model size under 8B. It encourages exploring and exploiting valid reasoning steps instead of imitation, improving generalization to unseen rules in the RULECOLLECTION-321
We design RLVR framework that introduces useful train-
Domain-aware Dynamic Sampling: To harmonize language models' proficiencies across imbal- anced domains, we present an adaptive sampling algorithm that dynamically reweights training domains based on their degree of under-optimizzatinn. This ensures balanced learning dynamics across tasks, enhancing both in-distribution (ID) and aut-of-distribution (OOD) performance.
We perform extensive evaluations on RULEREASONER, introducing two best rule-based rea- soning models: RULEREASONER-88 and RUTLEREASONER-44 Empirical results show that: (1) As depicted in Figure 1, RULERRASONEER-8 outperforms OpenAT-ol, Claude-3.7-Sonnet, and DeekSeck-R1, achieving higher performance than trivial RLVR methods. Specifically, RULEREASONER-SD achieves 144% and A49% OOD pass@1 respectively over ol and the base model. (2) RULEREASONER
further demonstrates that language models can effectively learn rules even with smaller model size, achieving an average pass of 78.3% on three OOD bench- marks (A7.3 over ol). (3) Surprisingly, RULEREASONER achieves comparable task performance with notably fewer training steps than existing RLVR methods. This suggests that RULEREASONER not only enhances ID and OOD performance but also improves training efficiency.
RELATED WORK AND PRELIMINARIES
RL plays a critical role in improving the reasoning capabilities of large language models (LLMss (Silver & Sutton, 2025), particularly through approaches such as RLVR (Guo et al., 2025; Li et al., 2025c; Zuo et al., 2025). In this section, we introduce the task definition of rule-based reasoning ($2.2) and discuss key components of prior RLVR methods ($2.3) and their limitations.
2.1 RELATED WORK
It has been demonstrated that LLMs can learn and apply rules, with promising potential (Zhu al., 2023; Servantez et al., 2024a). Early works start from predefined rules in symbolic forms with an emphasis on scalability and compositionality in specific tasks (Tang et al., 2023b; Luo et al., 2024;
2

---

Published as a conference paper at ICLR 2026
Jia et al., 2024; Gui et al., 2024; He et al., 2025a) while recent works are dedicated to perform rule- based reasoning in natural language that are more applicable for real scenarios (Zhou et al., 2024; He et al., 2024; Tang al., 2024). It is also worth noting that recent advances in logical reasoning, such as Logic-RL. (Xie et al., 2025), are generally considered rule-free for reasoning, which differs from task definition. These methods explore the potential of rule learning through diverse prompting methods (Diallo et al., 2025; Peng et al., 2024), supervised distillation (Wang et al., 2024a), and external memory augmentation (Wang al., 2024b;d). However, they spend less effort adapting the reasoning capability of LLMs to unseen tasks with limited task types and formats. Inspired by recent advancements in RLVR methods focused on mathematical reasoning and code generation (Zhang et al., 2025; Chen et al., 2025; Wei et al., 2025; Li et al., 2025b; Zhao et al., 2025; Li et al., 2025a), we further optimize their limitation on data efficiency with dynamic data sampling along with curated collection of diverse rule-centric training data. Our method improves model performance across both ID and OOD reasoning tasks with higher generalization and computational efficiency. Conventional reweighting and loss-based sampling methods (Luo et al., 2025; Shi et al., 2025b) are unsuitable for RLVR because the direction of optimization for surrogate loss of RL does not correlate with policy model performance. Moreover, compared with prior methods, dynamic sampling strategy into RL for reasoning without requiring proxy models to obtain domain weights (Xie et al., 2023; Liu et al., 2025a), instead leveraging only the verifiable rewards in self- adaptive paradigm.
unifies domain-leved
2.2 PRELIMINARIES! I: RULE-BASSE REASONING
Given a question, a set of facts, and associated rules as context, the model asked to answer the question by applying and reasoning the facts with rules. In this paper, we refer to the rules as contextual logic rules, which are expressed in natural language and given specifically for each question. The rules provided can be generated either explicitly or implicitly as principles or premises to solve the question. For grouped datasets D == {(d, qt. Fc, }=] where d: E (d,, is specific domain, q: is question, r; is reasoning trajectory, and y/ is verifiable answer.
2.3 PRELIMINARIES II: ON-POLICY REINFORCEMENT LEARNING
Reward Shaping. To teach models to learn reasoning, we design rule-based exact match (EM) reward function to evaluate the response according to the final answer, ensuring both the correctness of the answers and the adherence to the format. We define (q, ú) D, To(-11), and
{ otherwisc.
is equivalent (ū. v),
REM(Ú.V)
Policy Optimization. We adopt the basic form of GRPO (Shao et al., 2024) but discard the part of KL loss, encouraging the model to explore various solutions. For each question-answer pair (g, y), policy model Tew, samples a group of responses {yı, .,, ,G } and calculates the associated re- wards {r1, r2, r.), given the oracle answer y, using the aformentioned reward function REM-
[(0)= (( D. (vil M (1l
6
| min (rr.(0)A,,,, (r..())).))))))))))))))))))))))) -,,10)) )
Iyil
Σ
where (0) is the rate of importance sampling for domain de at the t-th token for y/ and A, is the advantage as the critic obtained by normalizing the rewards within each group. We strietly follow the on-policy training method, performing only one gradient update after the policy model mew. generates a group of rollouts, enable stable RL training and prevent entropy collapse.
(i.)(0) = To [y (.Bi..t)
mcan ((r1, F2, ... .......,,,. ,rra)) A, F3 std (fr .*, ,rc))
(3)
nea (v1,1 9, V1,ct))
Limitations of RLVR on Training Efficiency. Though current RLVR elicits the long chain-of: thought reasoning ability based on the policy gradient RL algorithm like PPO (Schulman et al.,
3

---

Published as a conference paper at ICLR 2026
2017) and GRPO (Shao et al., 2024), the efficiency of training data for RLVR remains relatively unexplored. Existing works like DAPO (Yu et al., 2025) oversamples and filters out prompts with the accuracy equal to 1 and 0 to enhance training efficiency. However, it does not push the limits of training efficiency due the large recompute cost in the rollout stage. SRPO (Zhang et al., 2025b) shows the gains via epoch-level re-sampling wtih RLVR without exploring the agile sampling meth- ods for fine-grain control. Moreover, ADARFT (Shi et al., 2025a) explores an efficient batch-level sampling method using curriculum learning; however, it relies on human priors or an empirical suc- cess rate by models on sample difficulty. In the following sections, we expend great effort to further leverage training examples to achieve higher reasoning performance effectively.
3 DOMAIN-AWARE POLICY OPTIMIZATION WITH DYNAMIC SAMPLING
In data sampling, increasing the number of samples an individual domain po- tentially harms the performance of other domains without timely control (Albalak et al., 2024) or causes obvious tradeoff across domains (Xie ct al., 2023).
Damaatt-oaa are Donamii Samnling
Grouped D
Sanipe by W.
DADS
Domain Revent P.ck.
Sampplt
Domuin Weigl W
To address similar issues in RLVR, shown in Alg. I and Figure 2. we pro- pose Domain- (DADS), an effective sampling method for RLVR aiming to improve the performance of a policy Ty for solving multi-domain rule-based reasoning tasks. DADS dynam- ically adjusts the probability of sampling data from different domains based on their historical rewards. By prioritizing domains that yield lower verifiable rewards or those lagging behind a target reward, DADS enhances sample efficiency in training batch B, and mitigates the domain imbalance issue, leading to faster and more stable learning of policies that satisfy reward specifications. We instantiate RULEREASONER with the gra- dient policy algorithm of GRPO variant this work to demonstrate its effectiveness and efficiency.
RM m(-))
Dynamic Sampling
Reward Advantage n,. &
Updaaed Weight W.
(a) On-policy RLVR with DADS
(b) Dumain Weights Estimution
Figure 2: RULEREASONER training recipe.
3.1 DOMAIN- DYNAMIC SAMPLING (DADS)
Domain- Rewards. each training step 5, to evaluate the proficiency for domain d: E D. we define fa.ss as the algebraic mean, calculated by domain rewards {ra.d..)) Im ~ REM 8) of m previous samples in the domain, which correspond to genersations V,-1 (Wa,d:.) ~ T and the set of ground truth P.-1- Note that may vary across different domains and training steps due to the batch-level domain sampling strategy. Domain-aware rewards calculation over batch (Alg. 1, line 6 and 7) is computed as: Fs.d, m Σ "'sd;.-- We employ target reward,
Ttarget A 1, to define the upper bound for the underoptimization estimation, Us.di of a domain. Thus, we have the estimation:
Ts.d; where target reward quantifies the extent to which a domain performance lags behind the desired level.
Decaying Importance Sampling. Furthermore, given the utilization of past rewards for do- main d,, we introduce a decaying importance-sampling strategy, which employs the exponentially weighted moving average (Holt, 2004) that considers both current and the historical estimated re- wards. The historical rewards (F.-1,4; 1 are involved with the smoothing factor to produce normalized rewards fF.dd J We have Ts.d. = añ,-id, + (1 (1-a) FF,,,,, where a E [0, 1] serves as smoothing factor that creates more stable estimate of the performance for domain over time, rather than relying solely on the most recent reward Fs.d,-
Domain Re- Re-witthting. Consequently, we establish domain weight, Wa.d,, which is then normal- ized using a standard softmax function (as detailed in Alg. 1, lines 11 to 13). In this normalization, hyperparameters 7 and are used: ensures a minimum sampling weight for all domains, even well-learned ones, and 7 adjusts how strongly the sampling prioritizes domains based on their re- wards. We discuss the sensitivity of a, r, and in $C.5. After obtaining the re-sampling weights
4

---

Published as a conference paper at ICLR 2026
W := {wi, W2, ..... W.n across domains, use W3 to construct batch B, for the subse- quent policy optimization iteration. The process will be implemented iteratively during the training and more details described in Algorithm 1.
3.2 TRAINING REGULARIZATION
We want to ensure that the model follows, matches, and implements the rules and does not just learn to identify specific datasets and perform correctly on them. To do this, we introduce different kinds of noise during RL training. This stops the models from recog- nizing partieular datasets, recalling specific rules, or memorizing only the similar seen facts the context.
Algorithm Domain-aware Dynamic Sampling
Input: Policy model: ma : : + y/
Reward model: REM(-; :)) V. " - {0, 1):
Last weight: Wa-1 := {11, W2, ................................ Wn
Grouped data: D := [dd, to rs y:)
where domain: d, E {dl
d
Hyperparameters: {a, c, C R
Output: Constructed batch of samples: B,.
1: procedure TRAIN STEP SAMPLING
Disabling Entropy Bonus. We discard the entropy bonus employed by Schulman al. (2017) and Shao et al. (2024), to avoid the issue of entropy explosion in RL without the coldstart bootstrap.
Initialize: B3-1 W.-1 x D; Fo.d. 0.
3: V--1 t (B
4 ROLLOUT
D f- REM (EM((e-,,)e--1)
A# Update estimated rewards */
(Fadd + (
my Σ}= re.d..j li=1
Discarding KI. Divergence. Similary to Liu al. (2025b), we eliminate the KI. term since the concerns on distributional shift of my eliminated by rule-based reward function de- fined in Eq. 1. This saves memory and com- putation in training while encouraging more
Fadd. + fafr-1dd +1 a 1 1* Calculate weights by rewards $/
8:
for 1.2.....7! do
10:
Us t 1 - Fa.ds
11:
Wa,dd + cxp(Us, (a,at/T) + +
exploration required by Tg.
12:
W. = wodd Fim1 D NORMALIZING
13:
{wadd// Σj= Ws,d, HE
Rules Order Shuffling. To prevent mem- orization of ordered rules appeared in RULECOLL.ECTION-33
14:
1* Re- Re-sample w.r.t optimized weights */
the order of con- textual logical rules are randomly shuffled for 16: each training sample.
15:
B, + W, D
SAMPLING BY W
return B,
3.3 RULECOLLECTION- 32K: LOGICAL RULES DATA CURATION
We follow the following principles to collect our training data, namely RULECOLLECTION
Varying Depths. We collect 0-7 hop reasoning data for curriculum learning (Bengio et al., 2009)
across complexity levels and forms (deductive, inductive, analytical).
Different Formats. Collected data includes explicit or implicit rules as premises or constraints, enhancing the model's flexibility in recognizing, parsing, and applying rules in diverse contexts. Multiple Inference Rules. We focus the diversity of rules inference, which implies in the dataset facilitates learning dynamic rule employment and length generalization further.
Context Dependency. We focus on applying contextual rules adaptively for different questions. Correctly answering requires more than memorizing rules, reasoning, or using common sense. Robust Evaluation. We prioritize boolean and multiple-choice questions over free text to make it more conducive to obtaining rule-based outcome rewards and precise evaluation results.
These principles are critical to ensure training data captures the complexity and diversity inherent in rule-based reasoning tasks. The statistics of training data are presented in Table We highlight the importance and necessity of COLECCLLECTION-32 with further analyses in $C.9.
EXPERIMENTS
4.1 EXPERIMENTAL SETUP
5

---

Published as a conference paper at ICLR 2026
Task
Context (Explicit or Implicit Rules)
Question
Answer
RL LISS suu bear waok dog and sagg strtu deer that tur ouar thir eacy
ProofWriter
The bear neels
Tie
Evegshop corty essges i separ. Sveryling 1s exmptu
PrentoQA
Taue
FACTS: Saly Soly tniopur.
Cluter
FACTS: Pesko biling his wili Donnilyy ou to sleserffe Ohe dee seught Tracy lowes cooding fo fer Tinry woer the sove wil sizce Dencolty
RULES:
and grosdhoater of
motler of
Skarel relied Padre?
Shannel mother -i-low Pedro.
P'chor lesst people beollggee dken Restie is pegile and Jowe is flvee
RCT
LogicNLI
Landon is ror serliignat
Estuilment
FACTS: Johne lo Peogle Phitp and rociable, Junne broie.
FOLJO
Rak Kashi mo oven en Mathea Aajo Nodal is op pamaiouniaai tusets wlene Modal's was raoo
playery
are prnforrnnnt who hnre
Nada was not born Mallerzs
Falso
Logical Deduction
Ou ckey' showe are for boalc Bie kook a and book people hoot a gray food and
Which od the fellovizg baad
wkie vile and The
right ofe had
lifhnood
seonad
D
addad clie which dight
Sight A
Zeaate loong chised swoung rocety dhoe gais
AR-LSAT
Mikg diegue
Gropptodt
tokay?
LogiQA
is tur out and Papplis Tareoday dly wive Sod
Monday in)
bataa conee ded.
Tharsday Sendayy
Figure 3: Demonstration overview of RULECOLLECTIOND22X
Datasets and Benchmarks. We assess the generalization of models on unseen tasks using subsets from BigBench Hard (Suzgun al., 2023), BigBench Extra Hard (Kazemi et al., 2025), and ProverQA (Qi et al., 2025), as de- tailed in Table 1. We also employ AIME 2025 (AIME, 2025), GPQA (Diamond) (Rein et al., 2024), and Coin Flip (Wei et al., 2022b) as additional evaluation to assess whether RULEREASONER can extrapolate to more general reasoning tasks. More details on RULECOLLECTION 32K (Figure 3) are presented in $B.1.
Table 1: OOD benchmarks statistics.
OOD Test
Examples
Levels
BBH
750
BBEH
400
ProverQA
1,500
Compared Baselines. We include five types of baselines: (1) Prior rule-based reasoners (RBRs):
Hypotheses-to-Theorris (Zhu et al., 2023), Logic-of-Rule (Servantez et al., 2024a), and Rule-Guided Feedback (Diallo et al., 2025); (2) Frontier reasoners: OpenAI-ol (ol-2024-12-17) (Jaech et al., 2024), 03-mini (c3-mini-2025-014 Claude-3.7-Sonnet
(Zhang et al., 2025a), DeepSeek-RI (Guo et al., 2025), and with thinking mode) (Anthropic, 2025) with stan- dard zero-shot CoT prompting (Wei et al., 2022b); (3) Behavioral cloning (Pomerleau, 1988)": SFT without CoT (Wei et al., 2022a). SFT with short CoT (Yeo et al., 2025), and SFT with distilled long CoT (Yco et al., 2025) from 03-mini; (4) Advanced RLVRs: we compare RLVR approaches including GRPO (Shao et al., 2024), Dr. GRPO (Liu al., 2025b), and DAPO (Yu et al., 2025): (5) Curriculum Learning: we also introduce recent curriculum learning approaches for RL: ADARFT (Shi et al., 2025a), and balanee and easy-to-har strategies from Parashar et al. (2025).
Evaluation Metries. All tasks in the work are evaluated using the algebraic mean of hard exact match, which also equivalent to pass@1 @ accuracy under strict extraction and comparison.
Implementation Details. Training Setup:
We use Qwen3 (4B and 8B base) (Yang et al., 2025b) as the base models and employ veRL. (Sheng et al., 2024) for RL post-training. We set train and mini batch sizes to 64 for strict on-policy updates, and a rollout size of 64 per question. For hyperparam- eters DADS, we use a T of 0.5 for moderately frequent domain weight updates and an e of 0.1 for minimum sampling probability per domain, with smoothing factor of 0.5. Inference Setup: We employ random sampling (temperature т = 0.6 and top-p = 0.95 with a maximum output length of Owen3-8B-Bass for full-parameter SFT or RL. We perform
2,048. For the rest of baselines, five runs per test set and report the mean and standard deviation of the performance.
4.2 RULEREASONER IMPROVES RLVR PERFORMANCE AND EFFICIENCY
In-Distribution Performance, As shown in Table 2, we report the ID tasks performance to de- pict the effectiveness of RULEREASONER, Compared with the frontier LRMs, RULEREASONER-
FFllowing RI. literature nomenclature, we refer models trained with the negative log-likelihood loss as behavioral cloning and perform task-focused supervised training maximize baseline performance.

---

Published as a conference paper at ICLR 2026
Table 2: Comparison with all baselines on eight ID benchmarks. RULEREASONER significantly outperforms most of other methods. Average is the macro mean across all samples of domains.
Induction
Deduction
rOL
Others
Clutrr
ProntoQA ProofWriter
FOLIO
LogiNNLI AR-LSAT Logic. Dedu.
LogiQA
Results
PRIOR RBRS
HrT (Zhu 2023)
40.3
92.0
88.0
71,0
54.0
97.0
100.0
79.1
77.7
RGFB (Diallo et al, 2025)
31.3
94.0
88.0
74,0
55.0
95.0
100.0
79.1
77.1
Chain-of-Logic (Servantez ol,, 2024a)
44,8
91.0
92.0
80.0
54.0
97.0
100.0
80.6
80.0
FRONTIER REASONERS
OpenAl al (lecch el all, 2024)
52.2
91.0
91.0
77.0
60.0
98.0
88.0
82.1
79.9
OpenAl 03-mini (Zhong
2025a)
94,0
93.0
74.0
55.0
96.3
100.0
77.6
78.8
Clande-3.7 Sonoon- Sonnet
(Anihropic, 2025)
65.7
92.8
90.0
74.7
58.0
76.2
97.0
81.5
79.5
DoepSoek-RI (G00 al,, 2025)
71.6
40.0
27.0
72.7
49.0
89.7
98.3
85.0
66.7
BEHAVIORAL CLONING
SFT w/o CoT (Wei et al., 2022n)
37.5
96.0
88.8
73.4
74.8
37.5
85.9
76.1
71.2
SFT Short CoT (Yoo
2025)
92.6
87.0
82.9
54.8
80.9
SFT w/L.ng Long CoT (Yeo et al., 2025)
83.5
95.6
89.2
83.4
76.6
68.6
79.6
79.1
81.9
ADVANCED RIVRS
GRPO (Shan et all, 20244)
73.1
95.4
96.4
72.3
66.6
36.3
90.3
70.1
75.0
Dr. GRPO (1 (uin et ol., 20256)
68.6
96.0
95.6
73.9
75.4
32.1
84.3
65.6
73.9
DAPO (Yu et 2025)
96.0
80.9
79.2
CURRICULUM LEARNING
Dato-balance (Parashar 2025)
86.5
95.8
95.6
76.8
64.4
45.6
95.3
73.1
79.1
Easy-ta-hard RI. (Parushar ol,, 2025)
$8.0
96.2
96.8
78.9
66.6
46.3
96.0
74.6
80.4
ADARFT (Shi ot all, 2025a)
92.5
96.0
97.4
81.8
64.4
44.6
96.6
80.5
81.7
RULEREASONER (Ours)
RULEREASONER- 4B
82.0g 4
95.0g.6
96.30.
78.9p.s
66.60.4
38.6io.s
96.30.2
80.50.7
79.2p.
RULEREASONER-SE
95.5
96.40.4
97.0uu
84.70.s
70.4u.1
46.8o.3
98.30.4
83.5u.s
84.0.s
8B surprisingly outperforms with a large performance gap. Notably, on eight ID tasks, OpenAI- ol lags behind RULEREASONER-88 with 4.1% point, whereas Claude-3.7-Somnet underper- forms with 4.5% point. Also, RULEREASONERR-8T outperforms prior strong RBRs such as HtT and Chain-of-Logic, which built directly on top of OpenAl 03-mini for all tasks, cept for AR-LSAT and Logical Deduction. This implies that RULEREASONER benefits from RLVRs to obtain higher improvement in rule understanding and utilization. In addition, RULEREASONER-81 also outperforms recent RLVR methods which are trained with lower intra- task performance variance in eight tasks, for instance, higher performance of 84.0% (+4.8%) yet with a lower variance of 3.1% (-0.5%), comparing to DAPO (79.2% with a variance of 3.6%). This demonstrates that RULEREASONER not only develops impressive task performance, but also maintains the domain performance balance.
Table 3: Comparison of average improve- ment. % denotes ID and % denotes OOD performance, respectively. Unlike the task-focused settings in $4.1, f indi- cates a full-set mix supervised training to obtain stronger OOD performance for SFT.
Distt--Distiibttion Performance. As illustrated in Figure 1, RULEREASONER-SB surpasses fron- tier LRMs across three OOD benchmarks. Specif- ically, it shows a remarkable 10.4% improvement compared to OpenAl-ol. As depicted in Table 3, RCLEREASONERR-B consistently increases perfor- mance across the three OOD benchmarks, achieving the highest average performance gains of A56.0%. in- cluding 77 .44 on BBH, 448.4% on ProverQA, and A48.2% on BBEH. These findings highlight the effec- tiveness of RULEREASONER in enhancing the general rule-based reasoning capabilities of models. As shown in Table the SFT baseline lags behind RULEREA- SONER in both ID and OOD evaluations. Notably, while SFT improves ID performance to closely match RULEREASONER (81.9% versus 84.0%), ), its OOD performance remains significantly lower (34.4% versus 54.5%). We conclude that, in contrast to RLVR, SFT does not effectively generalize to unseen rules or OOD scenarios, which also aligned to Chu et al. (2025). Extended evaluations on other reasoning tasks (AIME 2025, GPQA, and Coin Flip) and the test-time scaling are discussed in SC.6 and SC.3, respectively. Moreover, a relative OOD demonstration in Table reveals that RULEREASONE elicits models to extrapolate to new tasks by applying unseen rules, through a concise and logically rigorous reasoning trajectory.
Model
Pass
Avg. A
Qwen3-8B
27.4 34.2
+ SFTT
81.9 66.6
54.5
34.4
+ GRPO
75.0 75.8
47.6 / 41.6
Ours
84.0 81.7
56.6 / 47.5

---

Published as a conference paper at ICLR 2026
5
ANALYSES
5.1
ADVANTAGES OF DOMAIN-AWARE DYNAMIC SAMPLING
Reward
Pass
Domain Weights
ProofWriter
ProntoQA
Clutrr
FOLIO
1.0
Reward Passia
0.2
Domain Weights
0.8
0.6
0.1
LogieNLI
AR-LSAT
Logical Deduction
LogiQA
Reward @ass(
1.0
0.2
Domain Weights
0.8
0..
0.15
0.4
0.1
30 60
180
30 60
180
180
Training Steps
Training Steps
Training Steps
Training Steps
Figure 4: Learning dynamics by domains. "Reward" represents the training reward obtained from tasks and " 1" denotes validation pass @ performance. We employ exponential moving aver- age smoothing for clearly displaying the curves "Reward", "Pass I", and "Domain Weights".
As depicted in Figure 4, we find that RULEREASONER enhances task performance across all do- mains without tradcoff. Specifically, RULEREASONER takes care of the underperformed task such as AR-LSAT and increases domain weights consistently. Even for the low-portion domains (e.g., ProofWriter) and the decreased domains such as Clutrr, RULEREASONER still steadily improves their training rewards and validation pass@ without reaching a plateau. Interestingly, analogous to the phenomenon described by Zucchet et al. (2025), the knowledge acquisition period during pre- training is accelerated fast on transition, but led to overfitting by the imbalanced data distributions. As shown in Table 6, compare the OOD performance with DAPO and ADARFT to directly demonstrate that DADS achieves superior generalization without requiring external difficulty esti- mation (as in ADARFT) or additional rollout compute (as in DAPO).
Furthermore, we perform curriculum learning baselines as stated in SC.8. In summary, DADS sur- passes these baselines by significant margin (see Table and 10). Static curriculum methods fail because they treat easy and hard domains equally or experience vast distribution shift during train- ing, while DADS domains (e.g., ProntoQA) to under-optimized (hard) domains (e.g., AR-LSAT). This prevents over- optimization on solved tasks while preventing under-fitting on complex ones. summary, we con- clude that DADS serves as an online data scheduling strategy, stablizing the dynamics of on-policy RL training and mitigates over-optimization.
online scheduler, shifting compute resources from converged (easy)
5.2 IMPACT OF TASK MIXING RECIPE
Figure illustrates the impact of mixing recipes of incremental tasks in OOD tasks across models of different sizes. The incremental mixing strategies show consistent gains the model size increases, indicating that a larger model capacity generally contributes to better generalization. The mixed col- lection of various reasoning types enhances the OOD performance nearly 80% while the baseline "w/o All" at around 25%. further validates the benefit of our training data curation principles and the effectiveness of task mixing for learning generalization.
8

---

Published as a conference paper at ICLR 2026
80
80
70
22 (1
OOD Performance (%)
0OD Performane (%)
OOD Perfomaanee (%)
60-
60-
60
50-
40-
Others
#0-
40-
Rule Reasoner
Rule Reasomer
DAPO
20
wio (Base)
Deduction
Induction
20
GRPO
SFT
30
Dr: GRPO
GRPO
4B Model Size (# of Parameters)
kb
52 Proportion of each Domain (%)
84 T2S
100
200
300
Training Steps
Figure 5: Impact on incre-
Figure 6: Impact on training sample efficiency.
Figure 7: Comparisons on different RLVR methods.
mental task mixing recipes.
5.3 INVESTIGATION ON TRAINING SAMPLE EFFICIENCY
We assess OOD performance by scaling training sample sizes among three methods at step 500, sampling data with varying domain proportions (Figure 6). as the proportion of each domain increases, RULEREASONER consistently outperforms GRPO and Mix SFT every proportion level. For the same proportion of data, RULEREASONER achieves higher performance than Mix SFT as well as a slight advantage over GPRO. For example, when the domain proportion is 1 1%, RULEREASONER achieves an OOD performance of about 60% point, while the other two is around 59% point and 37% point. This implies the effectiveness of the online data scheduling stated in Section
log scale. It is evident that for all methods
As shown in Figure 7, we also prove that RULEREASONER improves sample efficiency with fewer training steps than the existing RLVR methods. It achieves comparable OOD performance with ~6..8% on average, roughly fewer 72 steps and approximately 1.4x acceleration compared to DAPO within the same training steps. Similarly, GRPO requires more steps to match RULEREA- SONER'S performance, which indicates that the dynamic batch-level data curricula in RULEREA- SONER benefit RLVR and accelerate it to achieve higher performance with fewer costs.
5.4
CASE STUDY OF RULE APPLICATION
Table 4: Reasoning chains before and after training by RULEREASONER for comparison.
Case Type
Input
Before Training
After Training
furry. something it green. Cold, smast things are grees. All furry things are white. If something green,
Rules: If something is smart
Step I: Amalyse the given rules and facts.
sumething smart, then firry. Fact
Rule
Rule something cold, then is Since green. the given that Dave green. Froes Rule
Step-hy-Step Ressoning: 1. Davee is cold. From
Dave blue.
cold. Step
consistent with
Desermine Dave is furry based on the given rules and fass. From Faet 2 and Rule we know
something is green. then it smart, Since Dave is
Explicit Rules (PioofWriter)
furry thangs are green. All cold, green things are rough. Cold things green. Facts: Dave is blue. Dave cold. greas. Question: Dave is furry?
that Dove
From Fact and Rule 8, that Dave green. However,
Dave is smort. From Rule I: If something smart, Dave
also have
funr. Since
Rule wating that all furry things are white, and Dave is blu (Fact 1). This creatos contrufiction,
fury. From Rule All furry things are green. This is consistent with the given fact that Dave is green. 3. Conclusion: Since Dave smort (from Rule 5), from Rule something smart, funy. we can conclude Dovee is furry.
Dave cannot be buth furry and blue. Since conteatiction,
cunnot determine Dave furry The inswer Unknown. Answer: Unknown. X
Answer: True. /
The following pact- graphis describe of thene objects arranged fised order. The statements are logically consistent within each parapmihh. On branch, there three binds: robin, an owl, and a hawk, The robin is to the right of the bask. The owl is left- most.
Context:
answer question. let's andlyse the informa- tion powided are three binds on branch: Now; let's evaluste the options: (A) The robin is the rightmost: This option the huwk. because the rubin the right rightmost burd.
desormine
order the three birds branch. analyze information peovided:
contest step by step: I. There
The owl is the loflmust. This means the owl in
the first position on the branch. 2. The robin to the right the hawk. This Now, determine
the nobin
necessarily
hawk the posiiices of the haw', and the robin: Since the owl is the lefimosi, the oud must be in the first position. hawk, the hawk in second position,
Implicit Rules (BBH)
(B) The und is the rightmoet: This option is not true because the ued is the lefimest hird, not the right- options (A. B. information provided, none true. However. since we must chiuse one of the given options, we can conclude that the correct answer is: (B) Please note cotirely accurate. The question might have a mistake or missing information. Answer: B.
Based
The robin the right
must in the thinl position. So, the under from to right is: I. Owl 2. Hawk 3. Robin Now, let's evain- uptions: (A) The robin the rightmost this statenent true. (B) The is the rightmest. So, this statement false. Answer: A.
Options:: (A) The rohin the righmmost (B) The owl is the
that this
rightmost (C) bawk rightmost.
actually leftasoer
Question: Which of the fol- kowing options true?

---

Published as a conference paper at ICLR 2026
We show examples from RULEREASONER 8B in various test sets in Table 4 to demonstrate how our training recipe improves the performance of the model. Before training, the model fails to solve the problems because of various reasons, such poor reasoning logic, reasoning distracted by the noisy rules, etc. After training, the model is able to derive a valid reasoning chain and answer rigidly. Additionally, we provide more successful and failed cases with their trajectories in Table 13 and 14.
5.5 GENERALIZATION MECHANISM FOR COMPOSITIONAL RULES
Motivated by the efficacy gap in Table 3, we analyze the origin of RULEREASONER' generalization capability. The OOD performance gains stem not just from the dynamic curriculum via DADS but also from the emergence of meta introspection during training, fostering self-simulation and self- verification abilities. RULEREASONER explores candidate paths and verifies logical consistency against inferred rules before concluding. This mirrors abstract introspection, where the model cri- tiques intermediate outcomes and corrects flawed steps, enabling generalization beyond seen tasks.
In the first ProverQA example (Table 15), the untrained model makes unverified and speculative assumptions (e.g.. "that good intentions imply being lovable"). In contrast, the trained model adheres strictly to objective facts, noting that "lovable" is not explicitly defined or linked to those attributes in context. It analyzes available clues contextually, performing velf-veriiiiation and consistency checks throughout the reasoning process. This rigorous behavior enables model to eliminate cognitive biases and logical fallacies, arriving well-supported conclusions that align with established facts.
6 CONCLUSION
We introduce RULEREASONER, training framework tailored to enhance both effectiveness and efficiency of multi-domain training for RLVR. It harmonizes diverse rule-based reasoning capabili- ties across various tasks, resulting in higher performance efficiently compared to existing rule-based reasoners, frontier LRMs, strong supervised baselines, and prior RLVR methods. We aim to explore further research into data-centric approaches to facilitate reasoning efficiency in future work.
Limitations Future Work. In this study, due to the scarce and imbalanced nature of rule-basse based reasoning data, current methods may not cover all rule formats and complexities found in real- world applications, which hinders task generalization. Besides, our method is constrained by the quality of rule filtering, particularly when dealing with noisy redundant rules that can negatively impact reasoning, Furthermore, while effective with smaller models (4B and 8B), its scalability to large-scale modeling remains unverified due to computational limitations, despite potentially higher effectiveness in complex scenarios. Beyond these, the effectiveness and utility of our rule-applying in domain-spectin scenarios remains be explored and further enhanced. To meet the inherent nature of rule reasoning, the current methods presumably relies on static pre-defined set of rules. lacks mechanism for dynamic, ever-changing rule discovery and updating from new data or interactions, limiting its long-term adaptability in evolving environments. It could be studied in the future for moving towards a self-improving system. These limitations highlight areas for future improvement in expanding rule diversity and robustness to support longer reasoning trajectories.
ETHICS STATEMENT
We adhere to ethical principles to ensure the responsible development and application of our pro- posed techniques. The research conducted in the article every respect in accordance with the ICLR code of ethics guidelines. Our work focusses on enhancing the rule-based reasoning abilities of models without directly involving human subjects or sensitive information. The study acknowl- edges ethical implications, such as the transparency of rule-based systems being advantageous for interpretability but raising concerns about accountability if rules are misapplied in high-stakes do- mains. We advocate for rigorous validation of rules against diverse datasets to avoid human biases in manually crafted rules. We also recognize the potential broader impacts, including the environ- mental and computational costs of L.LM training, and strive optimize methods for efficiency.
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of our research, we provide detailed information regarding our method- ology and experimental setup. The exact prompt templates utilized for datasets with both explicit and implicit rules during training and evaluation are detailed in Appendix A. Furthermore, Appendix B.I provides comprehensive list of the data sources for both training and OOD evaluation, along with specifics on how the datasets were curated for the RULECOLLECTION 32K. Our evaluation metries are defined $4.1, with additional evaluation results and analyses presented Appendix C. Finally, a complete breakdown of the computational infrastructure and all hyperparameter assign- ments for both training and inference can be found in Appendix D. The code, model, and data will be made publicly available upon completion of the review process.
ACKNOWLEDGMENTS
We would like to thank Zixia Jia, Hengli Li, and Xubo Qin from BIGAI for their valuable contri- butions in discussing the project, and Tong Wu from BIGAI for his initial trials, helpful discussions on RLVR training, and the infra-system operation and maintenance. We also thank the reviewers for their insightful suggestions to improve the manuscript. This work presented herein is supported by the National Natural Science Foundation of China (62376031).
REFERENCES
AIME, AIME problems and solutions, 2025. URL
com/
wi kik //nindie....///// phppp AMPPP//MMEBBTA Probl ems sedtoeeeeaan___tttionnnn
Alon Albalak, Yanai Elazar, Sang Michael Xie, Shayne Longpre, Nathan Lambert, Xinyi Wang, Niklas Muennighoff, Bairu Hou, Liangming Pan, Haewon Jeong, Colin Raffel, Shiyu Chang, Tatsunori Hashimoto, and William Yang Wang. A survey data selection for language mod- els. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL https: / //tpeneeview.neeeee. net. wtr/m//rummmmm id-x-ddfio XfW.cNHH..... Survey Certification.
Shengnan An, Zexiong Ma, Zeqi Lin, Nanning Zheng, Jian-Guang Lou, and Weizhu Chen. Make your Ilm fully utilize the context. In A. Globerson, L. Mackey, D. Bel- grave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (eds.), Advances Neural In- formation Processing Systems, volume 37, pp. 62160-62188. Curran Associates, Inc., 2024. URL https: http::///ooceeddeeeeeeeee wewpeinnnnnnuuuueeeeee neurips http://ppepccaee.oe..oooooooooas
Anthropic. Claude 3.7 sonnet.
https: /ttp:://wwwwwaaa/ http://wwww.t..pp...co news//
claude- 3- sonnnet, 2025. Accessed: 2025-04-11.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pp. 41-48, 2009.
Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V Le, Christopher Ré, and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling. arXiv preprint arXiv:2407.21787 2024.
Yang Chen, Zhuolin Yang, Zihan Liu, Chankyu Lee, Peng Xu, Mohammad Shoeybi, Bryan Catan- zaro, and Wei Ping. Accreason-nemmotron:: nemotron: Advancing math and code reasoning through rein- forcement learning. arXiv preprint arXiv:2505.16400 2025.
Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Sergey Levine, and Yi Ma. SFT memorizes, RL generalizes: A comparative study of foundation model post-training. In The Second Conference on Parsimony and Learning (Recent Spotlight Track), 2025. URL https: / //openreview net./ / forum? d=d3333..Wmma
Aissatou Diallo, Antonis Bikakis, Luke Dickens, Anthony Hunter, and Rob Miller. Rule-guided
feedback: Enhancing reasoning by enforcing rule adherence in large language models. arXiv preprint arXiv:2503. /336, 2025.
11

---

Published as a conference paper at ICLR 2026
Mehdi Fatemi, Banafsheh Rafice, Mingjie Tang, and Kartik Talamadupula. Concise reasoning via reinforcement learning. arXiv preprint arXiv:2504.0518. 2025.
Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Ying- han Shen, Shengjie Ma, Honghao Liu, et al. survey on llm-as-a-judge. arXiv preprint WrXiv:2411.15594, 2024.
Xinyu Guan, Li Lyna Zhang, Yifei Liu, Ning Shang, Youran Sun, Yi Zhu, Fan Yang, and Mao Yang- rstar-math: Small Ilms can master math reasoning with self-evolved deep thinking. arXiv preprint arXiv:2501.045) 2025.
Jiayi Gui, Yiming Liu, Jiale Cheng, Xiaotao Gu, Xiao Liu, Hongning Wang, Yuxiao Dong, Jie Tang, and Minlie Huang. Logicgame: Benchmarking rule-based reasoning abilities of large language models. arXiv preprint 1.XX:::2008.1778., 2024.
Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501. 2025.
Simeng Han, Hailey Schoelkopf, Yilun Zhao, Zhenting Qi, Martin Riddell, Wenfei Zhou, James Coady, David Peng, Yujie Qiao, Luke Benson, Lucy Sun, Alexander Wardle-Solano, Hannah Szabó, Ekaterina Zubova, Matthew Burtell, Jonathan Fan, Yixin Liu, Brian Wong, Malcolm Sailor, Ansong Ni, Linyong Nan, Jungo Kasai, Tao Yu, Rui Zhang, Alexander Fabbri, Wo- jeiech Maciej Kryscinski, Semih Yavuz, Ye Liu, Xi Victoria Lin, Shafiq Joty, Yingbo Zhou, Caiming Xiong, Rex Ying, Arman Cohan, and Dragomir Radev. FOLIO: Natural language rea- soning with first-order logic. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Pp. 22017-22031, Miami, Florida, USA, November 2024. Association for Computational Lin- guistics. doi: 10. 10.18863/1/2224 camnlpmmainn11 URL attps://aclanttaaaeee 2024. wwwnlpppem in 1229/
org/
Kaiyu He, Mian Zhang, Shuo Yan, Peilin Wu, and Zhiyu Zoey Chen. Idea: learning ability of large language model agent through induction, deduction, and abduction. arXiv preprint arXiv:2408. 10455, 2024.
Enhancing the rule
Qiyuan He, Jianfei Yu, and Wenya Wang. Large language sadel-enhanced symbolic reasoning for knowledge base completion. arXiv preprint arXiv:2501.01246, 2025a.
Ziyuan He, Yuxuan Wang, Jiaqi Li, Kexin Liang, and Muhan Zhang. Loogle v2: Are Ilms ready for real world long dependency challenges? arXiv preprint arXiv:2510.225:: 2025b.
Charles C Holt. Forecasting seasonals and trends by exponentially weighted moving averages. International journal of forecasting, 20(1):5-10, 2004.
Jingeheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung- Shum. Open-reasonnr--zero: An open source approach to scaling up reinforcement learning on the base model, 2025. URL https://arxiv. org/abs/2503.24290 2025.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky. Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint 24Xiv:2412.16722 2024.
Zixia Jia, Junpeng Li, Shichuan Zhang, Anji Liu, and Zilong Zheng. Combining supervised learning and reinforcement learning for multi-label classification tasks with partial labels. In Lun-We Wei Ku, Andre Martins, and Vivek Srikumar (eds.). Proceedings of the 62nd Anmual Meeting of the Associ- ation for Computational Linguistics (Volume 1: Long Papers), pp. 13553-13569, Bangkok, Thai- land, August 2024. Association for Computational Linguistics. doi: 13653//1/2024.acl-bo 731. URL ht attpps: / //a/ccaaaa wwllaanthhoogyy. org/ / 024 acl-long: 731/.
Mehran Kazemi, Bahare Fatemi, Hritik Bansal, John Palowitch, Chrysovalantis Anastasiou, San- ket Vaibhav Mehta, Lalit K Jain, Virginia Aglictti, Disha Jindal, Peter Chen, et al. Big-bench extra hard. arXiv preprint 2502 /9187, 2025.
12

---

Published as a conference paper at ICLR 2026
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng. Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Hengli Li, Chenxi Li, Tong Wu, Xuekai Zhu, Yuxuan Wang, Zhaoxin Yu, Eric Hanchen Jiang, Song- Chun Zhu, Zixia Jia, Ying Nian Wu, et al. Seek in the dark: Reasoning via test-time instance-level policy gradient in latent space, arXiv preprint arXiv:2505. 13308, 2025a.
Jiaqi Li, Mengmeng Wang, Zilong Zheng, and Muhan Zhang. Loogle: Can long-context language models understand long contexts? In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2024.
Jiaqi Li, Xinyi Dong, Yang Liu, Zhizhuo Yang. Quansen Wang, Xiaobo Wang, Song-Chun Zhu, Zixia Jia, and Zilong 7heng. Reflectevo: Improving meta introspection of small Ilms by learning self-reflection. In Findings of the Association for Computational Linguisties: ACL 2025, pp-
16948-16966, 2025b.
Xuefeng Li, Haoyang Zou, and Pengfei Liu. Limr: Less is more for rl scaling. arXiv preprint 1xXiv:2502.118866. 2025c.
Yuetai Li, Xiang Yue, Zhangchen Xu, Fengging Jiang, Luyao Niu, Bill Yuchen Lin, Bhaskar Ra- masubramanian, and Radha Poovendran. Small models struggle learn from strong reasoners. arXiv preprint arXiv:2502.12144 2025d.
Hanmeng Liu, Jian Liu, Leyang Cui, Zhiyang Teng,
Nan Duan, Ming Zhou, and Yue Zhang. Logiqa 2.0-an an improved dataset for logical reasoning in natural language understanding. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 31:2947-2962, 2023a. doi: 10.1109/ TASL.P2023.399344.
Junling Liu, Peilin Zhou, Yining Hua, Dading Chong, Zhongyu Tian, Andrew Liu, Helin Wang, Chenyu You, Zhenhua Guo, Lei Zhu, et al. Benchmarking large language models on cmexam- comprehensive chinese medical exam dataset. Advances in Neural Information Processing Systems, 36:52430-52452. 2023b.
Nelson Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models long contexts. Transactions of the Association for Computational Linguistics, 12:157-173, 2024. doi: 10.1162/tacl a 00638. URL
https:/ /ccl antho logy. www.ooyy..oooooooooooo.oo.ooo.oo 2024 tacl-1.11..
Qian Liu, Xiaosen Zheng, Niklas Muennighoff, Guangtao Zeng, Longxu Dou, Tianyu Pang, Jing Jiang, and Min Lin. Regmix: Data mixture as regression for language model pre-training. In The Thirteenth International Conference on Learning Representations, 2025a. URL https:
//ppnrevieww
jOOUXq77.
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding rl-zero-like training: A critical perspective. arXiv preprint arXiv:2503. 20783, 2025b.
I. Luo, Ju, B Xiong, YF Li, G Haffari, and S Pan. Chatrule: Mining logical rules with large language models for knowledge graph reasoning, arxiv 2024. arXiv preprint arXiv:2309.01S3 2024.
Zheheng Luo, Xin Zhang, Xiao Liu, Haoling Li, Yeyun Gong, Qi Chen, and Peng Cheng. Ve- locitune: A velocity-based dynamic domain reweighting method for continual pre-training. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Pro- ceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Vol- ume I: Long Pupers), pp. 16644-16656 Vienna, Austria, July 2025. Association for Com- URL
putational Linguistics. ISBN 179-8-891788 16--251--- doi: 10. https: ///aaaaaaantoooooo..ooo org/ 2025. acl- 1anll-oonn.... 813/.
Yansheng Mao, Yufei Xu, Jiaqi Li, Fanxu Meng, Haotong Yang, Zilong Zheng, Xiyuan Wang, and Muhan Zhang. Lift: Improving long context understanding of large language models through long input fine-tuning, arXiv preprint 250i 14644, 2025.
13

---

Published as a conference paper at ICLR 2026
Fanqing Meng, Lingxiao Du, Zongkai Liu, Zhixiang Zhou, Quanfeng Lu, Daocheng Fu, Tiancheng Han, Botian Shi, Wenhai Wang. Junjun He, et al. Mm-eureka: Exploring the frontiers of multi- modal reasoning with rule-based reinforcement learning. arXiv preprint arXiv:2503.07365. 2025.
Terufumi Morishita, Gaku Morio, Atsuki Yamaguchi, and Yasuhiro Sogawa. Enhancing reason- ing capabilities of Iims via principled synthetic logic corpus. Advances in Neural Information Processing Systems, 37:73572-776666 2024.
Shubham Parashar, Shurui Gui, Xiner Li, Hongyi Ling, Sushil Vemuri, Blake Olson, Eric Li, Yu Zhang, James Caverlee, Dileep Kalathil, et al. Curriculum reinforcement learning from easy to hard tasks improves IIm reasoning. arXiv preprint arXiv:2506.000 06632, 2025.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lever, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary VeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- Buché-Bu,,, E. Fox, and R. Garnett (eds.). Advances in Neural Information Processing Systems, volume 32. Cur- ran Associates, Inc., 2019. URL https:tpps.......... httpttpss/p/rooeeeeeeeeeeeeeeeee nwurips......... www.nnuuwrr.....aappeeeeeee.. wacp/pppp_filleeeeee
Xiangyu Peng, Congying Xia, Xinyi Yang, Caiming Xiong, Chien-Sheng Wu, and Chen Xing. Regenesis: Lims can grow into reasoning generalists via self-improvement. arXiv preprint arXiv:2410.02100, 2024.
Dean A. Pomerleau. Alvinn: An autonomous land vehicle in neural network. In D. Touret- zky (ed.), Advances in Neural Information Processing Systems, volume Morgan- Kaufmannn, 1988. URL https: http:///ccceeen..c/coe/. neuriin neurips cc ccppppprr_...eeeeeeeeeeeeeeeeree files/ httes//paprr198e file/ B1 2522 ott.aeeee..eectteao..oeee-eeeeee
Chengwen Qi, Ren Ma, Bowen Li, He Du, Binyuan Hui, Jinwang Wu, Yuanjun Laili, and Con- ghui He. Large language models meet symbolic provers for logical reasoning evaluation. In The Thirteenth International Conference on Learning Representations, 2025. URL https:
//openreview. wtt.:/e///oooooee http://orrmmdddd222geeexxi.i
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang. Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling, 2024. URL https: huttps:://ppeeeeee //ppnr/vee..eeeee// net/forung i 4dT.16784b99..
Abulhair Saparov and He He. Language models are greedy reasoners: A systematic formal analysis In The Eleventh International Conference on Learning Representations,
of chain-of-thouge http///ppeeeeeeeccceoe net.//eeeetteeeeeeee 2023. URL https:
wttp://wwwwe/foommmeeeeee http/formmmiide /t
Abulhair Saparov, Richard Yuanzhe Pang, Vishakh Padmakumar, Nitish Joshi, Mchran Kazemi, Najoung Kim, and He He. Testing the general deductive reasoning capacity of large language models using ood examples. Advances in Neural Information Processing Systems, 36:3083-3105, 2023.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: (707.067770 2017.
Sergio Servantez, Joe Barrow, Kristian Hammond, and Rajiv Jain. Chain of logic: Rule-based reasoning with large language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics: ACL 2024, pp. 2721-2733, Bangkok, Thailand, August 2024a. Association for Computational Linguistics. doi: 1/.284. madings-acl.159 URL https: https//////aaaahooog/.ooo. org/ wtgg//224 findings-acl.199
Sergio Servantez, Joe Barrow, Kristian Hammond, and Rajiv Jain. Chain of logic: Rule-based reasoning with large language models. In Lun- Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguisties: ACL 2024, pp. 2721-2733, Bangkok, Thailand, August 2024b. Association for Computational Linguistics. doi: 10.18653/112024. findings-acl. 159. URL https:ttt///o //t////aaaattaa ogy. org/2024 f inddi indinggs 159/.
14

---

Published as a conference paper at ICLR 2026
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song. Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXtv:2402.03300 2024.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibín Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprini arXiv: 2409. 100.9922 19256, 2024.
Taiwei Shi, Yiyang Wu, Linxin Song, Tianyi Zhou, and Jieyu Zhao. Efficient reinforcement finetun- ing via adaptive curriculum learning. arXiv preprint arXiv:2504.05520. 2025a.
Weijie Shi, Jipeng Zhang. Yaguang Wu, Jingzhi Fang, Shibo Zhang, Yao Zhao, Hao Chen, Ruiyuan Zhang. Yue Cui, Jia Zhu, Sirui Han, Jiajie Xu, and Xiaofang Zhou. DIDS: Domain impact-aware data sampling for large language model training. In Christos Christodoulopou- los, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng (eds.), Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 3330-4350, Suzhou, China, November 2025b. Association for Computational Linguistics. ISBN 979-8-89176-332- 6. doi: 18.13v53/v 1/0025 mmnlp- URL https: /awlaannhhlooyy...... org/2025. munnlp-main... 215/.
David Silver and Richard S Sutton. Welcome to the era of experience. Preprint of a chapter to appear in Designing an Intelligence, edited by George Konidaris, MIT Press (forthcoming), 2025.
Koustuv Sinha, Shagun Sodhani, Jin Dong, Joelle Pineau, and William L. Hamilton. CLUTRR: A diagnostic benchmark for inductive reasoning from text. In Kentaro Inui, Jing Jiang, Vincent Ng. and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Process- ing (EMNLP- J,MNLP,JJCNPP), pp. 4506-4515, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18833/v1/D194 1458, URL https: hatp::s:/aacaathhoooggo org/D19-145B/
Gaurav Srivastava, Shuxiang Cao, and Xuan Wang. Towards reasoning ability of small language models. arXiv preprint arXiv:2502. 1569, 2025.
Zhiqing Sun, Longhui Yu, Yikang Shen, Weiyang Liu, Yiming Yang, Sean Welleck, and Chuang Gan. Easy-to-hard generalization: Scalable alignment beyond human supervision. In The Thirty- eighth Annual Conference on Neural Information Processing Systems, 2024. URL https:// openreview n forum? feeimmid=-wwee hdggggf
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoe Le, Ed Chi, Denny Zhou, and Jason Wei. Challenging BIG-bench tasks and whether chain-of of-thought can solve them. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: ACL 2023, pp. 13003-13051, Toronto, Canada, July 2023. Association for Computational Linguis- tics. doi: 1/208653//1/20233 htt.://w//2.....nngg---accll fnddnggs-all.844 URL https: http///aaaantttoooo.oooeo org/2023. findings - acl. woc.... 824/.
Oyvind Tafjord, Bhavana Dalvi, and Peter Clark. ProofWriter: Generating implications, proofs, and abductive statements over natural language. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 3621-3634 Online, August 2021. Association for Computational Linguistics. doi: 10.18653/ v1/2021 win.dings--al.311.... URL ht //acl http://cclancoooogy.ooooooooo wwtpgy..rgg222222 findings-ard acl. 317/.
Xiaojuan Tang, Zilong Zheng, Jiaqi Li, Fanxu Meng, Song-Chun Zhu, Yitao Liang, and Muhan Zhang. Large language models in-context semantic reasoners rather than symbolic reasoners. arXiv preprint arXiv: 2305.14825, 2023a.
Xiaojuan Tang, Zilong Zheng, Jiaqi Li, Fanxu Meng, Song-Chum Zhu. Yitao Liang, and Muhan Zhang. Large language models are in-context semantic reasoners rather than symbolic reasoners. arXiv preprint a305v:2305.1 2nte:23555..... /4825, 2023b.
15

---

Published as a conference paper at ICLR 2026
Xiaojuan Tang, Jiaqi Li, Yitao Liang. Muhan Zhang, and Zilong Zheng. Mars: Situated inductive reasoning in an open-world environment. In 38th Conference on Neural Information Processing Systems (NeurIPS 2024) Track Datasets and Benchmarks, 2024.
Jidong Tian, Yitian Li, Wenqing Chen, Liqiang Xiao, Hao He, and Yaohui Jin. Diagnosing the first- order logical reasoning ability through LogicNLI. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 3738-37882 -377, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 10.18653/11/2021. emolp-main.303. URL ht:t::: / antho log. wwwpor//202 2021 emn p-ma n.. 303//
Hemish Veeraboina. Aime problem set 1983-2024. Kaggle Dataset, 2023. URL ht httpp://ww/ kaggle. com/ datasets //cesssheeeee hawamm/sieeeraaoaaaaaaaaeeeeeeee Accessed 24 November 2025.
Siyuan Wang. Zhongyu Wei, Yejin Choi, and Xiang Ren. Can LLMs reason with rules? logic scaffolding for stress-testing and improving LLMs. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 7523-755 3523-7543, Bangkok, Thailand, August 2024a. As- sociation for Computational Linguistics. doi: 10. 110553//1//222.allloogg //aclantho ogy. wwg.//2//222 acl- aal-lonlllllll 406/.
URL https:
Siyuan Wang, Zhongyu Wei, Yejin Choi, and Xiang Ren. Symbolic working memory enhances lan- guage models for complex rule application. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 17583-17604, Miami, Florida, USA, November 2024b. Association for Computa- tional Linguistics. doi: 10..8853/111/// http//222..mmmmmmaaae 9744 / 974/.
org/2024,
Siyuan Wang, Zhongyu Wei, Yejin Choi, and Xiang Ren. Can LLMs reason with rules? logic scaffolding for stress-testing and improving LLMs. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.). Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 7523-7543, Bangkok, Thailand, August 2024c. As- sociation for Computational Linguistics. doi: 10.18653/v h/wp/2..alllooon..cccc. 1eeel-oong..000
URL https:
//aclanthology org/
Siyuan Wang, Zhongyu Wei, Yejin Choi, and Xiang Ren. Symbolic working memory enhances language models for complex rule application. arXiv preprint arXiv:2408. 02022408..3554, 2024d.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, 2023. URL https: htppss//ppeereveeee net
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoe V Le. Finetuned language models are zero-shot learners. In Interna- tional Conference on Learning Representations, 2022a. URL /tttps://ppenres
net./
forum?id idd-g22GGozzde..
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoe V Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Ad- vances in Neural Information Processing Systems, 2022b. URL pettps://openreeee ii= wit.aiidddVViMMBB..___
Yuxiang Wei, Olivier Duchenne, Jade Copet. Quentin Carbonneaux, Lingming Zhang, Daniel Fried, Gabriel Synnaeve, Rishabh Singh, and Sida I Wang. Swe-rl: Advancing Ilm reasoning via rein- forcement learning on open software evolution. arXiv preprint arXiv:2502. 18449, 2025.
Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy Liang, Quoe V Le, Tengyu Ma, and Adams Wei Yu. Doremi: Optimizing data mixtures speeds up language model pretraining. In sevethh Conference on Neural Information Processing Systems, 2023. URL tettts://oppneet
n forum? d= X=xXuByDehhd....
16

---

Published as a conference paper at ICLR 2026
Tian Xie, Zitian Gao, Qingnan Ren, Haoming Luo, Yuqian Hong, Bryan Dai, Joey Zhou, Kai Qiu, Zhirong Wu, and Chong Luo. Logic-rl: Unleashing Ilm reasoning with rule-based reinforcement learning. arXiv preprint arXiv:2502.14768, 2025.
Guangzhi Xiong, Qiao Jin, Zhiyong Lu, and Aidong Zhang. Benchmarking retrieval-augmented generation for medicine. In Findings of the Association for Computational Linguistics ACL 2024, pp. 6233-6251, 2024.
Jundong Xu, Hao Fei, Liangming Pan, Qian Liu, Mong-Li Lee, and Wynne Hsu. Faithful logi- cal reasoning via symbolic chain-of-thought. In Lun-Wei Ku, Andre Martins, and Vivek Sriku- mar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pp. 13326-133 13365,, Bangkok, Thailand, August 2024a. As- sociation for Computational Linguistics. doi: 10.18653/v 1/202..cl-2222-1--111 long. URL https: //aclanthology wwt. htt.//222...ca111o1aee/ acl-long 720/.
Jundong Xu, Hao Fei, Liangming Pan, Qian Liu, Mong-Li Lee, and Wynne Hsu. Faithful logi- cal reasoning via symbolic chain-of-thought. In Lun-Wei Ku, Andre Martins, and Vivek Sriku- mar (eds.), Proceedings of the 62nd Anmual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pp. 13326-13365, Bangkok, Thailand, August 2024b. As- sociation for Computational Linguisties. doi: 18653/v1/2024.ccl //aalaattolooooo
URL https:
wrr.or/////// 2024..22.... acl- 720/.
An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jian- hong Tu, Jingren Zhou, Junyang Lin, et al. Qwen2. 5-math technical report: Toward mathematical expert model via self-improvemmnt. arXiv preprint arXiv:2409.12122. 2024.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 093888 2025a.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao. Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint 0rXiv:2505.09388, 2025b.
Wenkai Yang, Shuming Ma, Yankai Lin, and Furu Wei. Towards thinking-optimal scaling of test- time compute for IIm reasoning. arXiv preprint arXiv:2502.18080, 2025c.
Edward Yeo, Yuxuan Tong, Xinyao Niu, Graham Neubig, and Xiang Yue. Demystifying long chain- of-thought reasoning in LLMs. In ICLR 2025 Workshop on Navigating and Addressing Data Problems for Foundation Models, 2025. URL https: http:://///pereeee... net forum?id-
Agg011MO
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An open-source IIm reinforcement learning system at scale. arXiv preprint arXiv:2503. 14476. 2025.
Brian Zhang, Eric Mitchell, Hongyu Ren, Kevin Lu, Max Schwarzer, Michelle Pokrass, Shengjia Zhao, Ted Sanders, Adam Kalai, Alexandre Passos, Benjamin Sokolowsky, Elaine Ya Le, Erik Ritter, Hao Sheng, Hanson Wang, Ilya Kostrikov, James Lee, Johannes Ferstad, Michael Lampe, Prashanth Radhakrishnan, Sean Fitzgerald, Sébastien Bubeck, Yann Dubois, Yu Bai, Andy Ap- plebaum, Elizabeth Prochl, Evan Mays, Joel Parish, Kevin Liu, Leon Maksin, Leyton Ho, Miles Wang, Michele Wang, Olivia Watkins, Patrick Chao, Samuel Miserendino, Tejal A. Patward- han, Antonia Woodford, Beth Hoover, Jake Brill, Kelly Stirman. Neel Ajjarapu, Nick Turley, Nikunj Handa, Olivier Godement, Akshay Nathan, Alyssa Huang, Andy Wang, Ankit Gohel, Ben Eggers, Brian Yu, Bryan Ashley, Chengdu Huang. Davin Bogan, Emily Sokolova, Eric Ho- racek, Felipe Petroski Such, Jonah Cohen, Joshua Gross, Justin Becker, Kan Wu, Larry Lv, Lee Byron, Manoli Liodakis, Max Johnson, Mike Trpcic, Murat Yesildal, Rasmus Rygaard, R. J. Marsan, Rohit Ram-chandani, Rohan Kshirsagar, Sara Conlon, Tony Xia, Siyuan Fu, Srinivas Narayanan, Sulman Choudhry, Tomer Kaftan, Trevor Creech, Andrea Vallone, Andrew Duber- stein, Enis Sert, Eric Wallace, Grace Zhao, Irina Kofman, Jieqi Yu, Joaquin Quiñonero Candela, Made laine Boyd, Mehmet Ali Yatbaz, Mike McClay, Mingxuan Wang. Sandhini Agarwal, Saachi Jain, Sam Toizer, Santiago Hernández, Steve Mostovoy, Tao Li, Young Cha, Yunyun Wang.
17

---

Published as a conference paper at ICLR 2026
Lama Ahmad, Troy Peterson, Carpus Chang, Kristen Ying, Aidan Clark, Dane Stuckey, Jerry Tworek, Jakub W. Pachocki, Jo hannes Heidecke, Kevin Weil, Liam Fedus, Mark Chen, Sam Alt- man, and Wojeiech Zaremba. Openai 03-mini system card. In OpenAl Blog Post, 2025a. URL httpss: cdn .ecdnoopen opena com/ o3 -
0. pdf.
Xiaojjang Zhang, Jinghui Wang, Zifei Cheng, Wenhao Zhuang, Zheng Lin, Minglei Zhang, Shaojie Wang, Yinghan Cui, Chao Wang. Junyi Peng, al. Srpo: cross-domain implementation of large-scale reinforcement learning on Ilm. arXiv preprint (iXix:2504.1422500 2025b.
Yifan Zhang. Yifeng Liu, Huizhuo Yuan, Yang Yuan, Quanquan Gu, and Andrew C Yao. On the Design of KL-Regularized Policy Gradient Algorithms for LLM Reasoning. arXiv e-prints, art. arXiv:2505. 17508, May 2025.
Andrew Zhao, Yiran Wu, Yang Yue, Tong Wu, Quentin Xu, Matthieu Lin, Shenzhi Wang, Qingyun Wu, Zilong Zheng, and Gao Huang. Absolute zero: Reinforced self-play reasoning with zero data. arXiv preprint arXiv:2505.0333 03335, 2025.
Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, Alban Desmaison, Can Balioglu, Pritam Damania, Bernard Nguyen, Geeta Chauhan, Yuchen Hao, Ajit Mathews, and Shen Li. Pytorch fsdp: Ex- periences on scaling fully sharded data parallel. Proc. VLDB Endow. 16 16(12):3848-366) August 2023. ISSN 2150-8097. doi: 10. 1147778/3611540331116660 URL https: /doi.ii../// org/10. (4778/3611543 3611569.
Wanjun Zhong, Siyuan Wang, Duyu Tang, Zenan Xu, Daya Guo, Yining Chen, Jiahai Wang, Jian Yin, Ming Zhou, and Nan Duan. Analytical reasoning of text. In Marine Carpuat, Marie- Catherine de Marneffe, and Ivan Vladimir Meza Ruiz (eds.), Findings of the Association for Computational Linguistics: NAACL 2022, pp. 2306-2313 Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10.18653/v /2022.inndnngs---ieiee maadinngsnnall..1.. URL 2org/2022. https::/aaee hatpsss///clathhloogg.. httppss///ellattholloooo...///22 findings-nannn maa.l.17//............
Ruiwen Zhou, Wenyue Hua, Liangming Pan, Sitao Cheng, Xiaobao Wu, En Yu, and William Yang Wang. Rulearena: A benchmark for rule-guided reasoning with 1lms in real-world scenarios. arXiv preprint arXiv:2412.0890 2024.
Zhaocheng Zhu, Yuan Xue, Xinyun Chen, Denny Zhou, Jian Tang, Dale Schuurmans, and Hanjun Dai. Large language models can learn rules. arXiv preprint arXiv:2310.00 (tl.:2330.00044, 2023.
Nicolas Zucchet, Jorg Bornschein, Stephanie C.Y. Chan, Andrew Kyle Lampinen, Razvan Pascanu, and Soham De. How do language models learn facts? dynamics, curricula and hallucinations. In Second Conference Language Modeling, 2025. URL pestps::/openreeet forum? ddvBcGnragkr.
net/
Yuxin Zuo Kaiyan Zhang, Shang Qu, Li Sheng, Xuekai Zhu, Biqing Qi, Youbang Sun, Ganqu Cui, Ning Ding, and Bowen Zhou. Ttrl: Test-time reinforcement learning. arXiv preprint a50iv:2504. 16084, 2025.
18

---

Published as a conference paper at ICLR 2026
Appendix
TABLE OF CONTENTS
A Prompts
20
A.I Prompts for Dataset with Explicit Rules.
20
A.2 Prompts for Dataset with Implicit Rules.
20
Data Details
20
B.1 Data Sources
20
B.2 Dataset Curation Details
Additional Evaluation Results
C.1
Challenges of Different Rule Settings
21
................................
C.2 RULEREASONER can Adapt Varying Rule ................................
22
C.3 Test-time Scalability
22
C.4 Comparison with More Baselines
23
C.5 Robustness Analyses on Hyperparameters of DADS.
23
C.6 Extrapolating Unseen Reasoning Tasks.
24
C.7 Effectiveness of DADS on Extremely High Difficulty.
24
C.8 Comparison with Data Scheduling and Balancing Baselines
24
C.9 Impact of Rule-based Reasoning Data Curation
25
Computational Environments
26
D.I Infrastructure & Hyperparameters
26
Example Trajectories
26
E.1 In-distribution Positive Examples
27
E.2 In-distribution Negative Examples
28
E.3 Out-of-distribution Examples
29
19

---

Published as a conference paper at ICLR 2026
A PROMPTS
In this work, we use the same prompt template for each dataset for model training and evaluation.
A.1 PROMPTS FOR DATASET WITH EXPLICIT RULES.
Instruction: Please answer the question based on the given rules and facts using either of [A/B/C/D] (or TTrue/aals//nnnoowwi Fill in the answer between <answer> and </answer>. Provide your step-by-step reasoning process between <think>> and </think>.
Input:
-Rules: {Rules))
-Facts: {[Facts]
Question: {[Question}} }
Options: {{Options)} (OPTIONAL)
A.2 PROMPTS FOR DATASET WITH IMPLICIT RULES.
Instruction: Please answer the question based on the given contexts using either of [A/B/C/D] (or [True/False/Ukknown)) Fill in the answer between <answer>> and </answer>. Provide your step-by-step reasoning process between <think> and </think>.
Input:
Context: {[Context)) }
Question: {[Question)}
Options:
{{Options)} (OPTIONAL)
B
DATA DETAILS
B.1
DATA SOURCES
We list the training and evaluation data sources associated with the urls used the paper as below. The followings are the training and validation data sources:
ProofWriter (2021): http:p/p/aaleaa..oooo/////////t/
ProntoQA (2023): httear//ghthaacccc///sppptttttgg
Clutrr (2019): http:://gghtta.coo/yyyyyyatttttp
alld//mateerDatt/cllttt
AR-LSAT http::///gghtha..../../.aaaaaaea
blob/masee Dlobb/nsster/Dtaa
FOLIO (2024):
httes///hthhoocom//aaaall//////2
data/v0.0 0
LogicNLI (2021): main/dataset
LogicalDeduction
main/data
LogiQA (2023): https://gittab..ccm/ciiitit http://///tttttoooo//aaaat//////
logi gattia///AAAATAO LOOTTA
20

---

Published as a conference paper at ICLR 2026
The followings the OOD test data sources:
BigBench-Hard (2023): http://///aaggaaae//////aaaaa///
ProverQA (2025):
ProverQA
B.2
DATASET CURATION DETAILS
Table 5: Data statistics of curated tasks. f denotes it can be deemed deduction reasoning since provide rules explicitly. The abbr. in the table indicate Modus Ponens (MP), Universal Instantiation (UI) Hypothetical Syllogism (HS), Disjunctive Syllogism (DS), Modus Tollens (MT), respectively. "FOL", "AR", "CS", and "CCR" denotes First-Order Logic, Analytical Reasoning, Constraint Satis- faction, and Categorical & Conjunctive Reasoning, respectively. "MC" represents multiple choice.
Task Reasoning Reasoning Fiction Form
Rule of Inference
Dataset
Train/Test
Format
Depth
Rule
ProofWriter (Tafjord al., 2021)
7,997/500
Boolean Deduction
[0, 5]
MP. UI UI, Conjunction Simplification
ProntoQA (Saparov He, 2023)
8,000/500
Boolean Deduction
(1,3,5)
Clutrr (Sinha al., 2019)
268/67
Free Text Induction!
X
HS
FOLIO (Han et al., 2024)
1,208/242
MC
FOL
[0. 71
V
MT, DS, UI
LogicNLI (Tian et al., 2021)
8,000/500
MC
FOL
[1,
V
MP. MT
AR-LSAT (Zhong al., 2022)
1,636/410
MC
AR
V
MP. MT
Logic. Dedu, (Xu et al., 2024a)
1,200/300
MC
CS
(L.3.5)
/
MT
LogiQA (Liu et al., 2023a)
264/67
MC
CCR
/
MP. MT
For ProntoQA, we randomly negate some of the proof questions avoid learning the shorteut of answer "True". For PritfWriter, we randomly sample ten percent of the orginal source data con- sidering the imbalance nature of the whole training data. Then we use DeepSeck-PR RI to gener- ate the reasoning process including short CoT and long CoT sequences for each QA sample. For LogiQA, we use data with the reasoning type both categorical reasoning and conjunctive reasoning that leverages the implicit rule application and reasoning. For BigBench- Hardd we use the subset of "dogical.deduutionn deduction" with three, five, and seven objects with varing levels of difficulties and lect BoardgameQA and ZebraPuzzles from BigBench- Eatteacc--xtta-Haadd to keep consistent with our task definition OOD evaluation.
C ADDITIONAL EVALUATION RESULTS
C.I CHALLENGES OF DIFFERENT RULE SETTINGS
We investigate the task performance Clutrr, with three-
Noisy
92.5
level rule settings of Figure in the following: 1) Ordered
Rules
Rules: rules arranged in their application order; 2) Shuf- fled Rules: rules are provided in random order; 3) Noisy Rules: rules are shuffled and include irrelevant ones. aligns with real-world scenarios, rules may contain distractors. To our expectation, the task with ordered rules achieves the best performance among them likely due to the logical sequence aiding in task execution. Shuffled Rules, while still contain only the relevant rules but in random order, show a moderate performance drop. Noisy rules result in the most significant performance reduction with the added complexity of redun- dant rules as distractors, highlighting the negative effect on task performance.
Shuffled Rules
95.7
Ordered Rules
99.5
0
20
40
60
80
Rules Ordering Performance (%)
Figure Comparison of perfor- mance on challenging rule settings.
21

---

Published as a conference paper at ICLR 2026
C.2 RULEREASONER CAN ADAPT TO VARYING RULE COMPLEXITY
As depicted in Figure 9, we present the extended OOD eval- uation results, with test sets separated by rule complexity (i.e. task difficulty). The BBH, ProverQA, and BBEH benchmarks consist of questions requiring reasoning up to various difficul- ties that hinge on the diverse factors of query complexity.
100
80
Poss (%)
60
Specifically, for BBH, we divide the original test set into three difficulty levels based on the multi-hop number of the query. For ProverQA, we adopt the original difficulty levels from its source, which is separated by the number of reasoning steps. For BBEH, the test set was categorised into three levels ac- cording the length of the query (in tokens): Easy [0, 1068), Medium [1068, 2175), and Hard [2175, 2741). Thus, we test the generalization capabilities on the three subsets of each OOD benchmark and report their performance on the higher difficulty questions. Not surprisingly, performance in easy subset exhibit substantially stronger than corresponding medium- and hard-level subsets, with an average pass@1 of 86.7% compared to the 63.0% (-23.7%) of hard subset across benchmarks. In- terestingly, we notice that RULEREASONEES difficulties, while we maintain the still performance in BBH. One possible explanation that the base model, Qwen3-8B- Base. might have encountered test set leakage, given that BBH was pub- lished late 2022 but Qwen3 models were released in 2025 (Yang et al., 2025a). Therefore, we suggest assessing our models to more challenging benchmarks to achieve more reasonable results. We leave this direction for future work.
40
BBH
20
ProverQA
BDEH
Easy
Medium
Hard
Rule Complexity
Figure 9:
Comparison of perfor- mance on varying task complexity.
drops significantly along with subsets in different
C.3 TEST-TIME SCALABILITY
In the cutting-edge discussion on the essence and usefulness of a longer thinking process, Fatemi al. (2025) find that the extra generated tokens do not help improve the final prediction accuracy, while Yeo et al. (2025); Yang et al. (2025c) hold the opposite positions which claim that accurate results are not necessary with the long reasoning process. To investigate this interesting question for rule-based reasoning, we study parallel test-time scaling strategies with RUTERFASONER. We perform repeated sampling to investigate the upper limit of performance for each collection of rule- based reasoning problems in the way of Brown et al. (2024), illustrated in brown curves. Concretely, we take the majority vote and normalized weighted sum methods in the way of Wang et al. (2023).
BigBench Hard
ProverQA
BigBench Extra Hard
100
80
76
Performance (%)
Pass@al (Baseline)
99-
78-
74-
98-
76-
97-
74-
Passal (Baseline)
70
Passi (Baseline)
96
72
68
2
32 Number of Samples
4
8
16
64 128
2
8
16
32
64
128
4
16
32 64 128
Number of Samples
Number of Samples
K Coverage
Pass@K Majority
Pass@K Weighted
Figure 10: Comparison of OOD performance parallel test-time scaling methods.
As depicted in Figure 10, test-time scaling demonstrates different effectiveness across benchmarks. For BigBench Hard, all Pass @ K methods achieve near-perfect performance (close to 100%) with minimal scaling, indicating the limited complexity of the testbed to distinguish scaling benefits. In ProverQA, Pass Coverage consistently outperforms both the majority and weighted approaches,
22

---

Published as a conference paper at ICLR 2026
with performance gaps widening the sample size increases (~1.7% at 128 samples). BigBench Extra Hard reveals the most substantial scaling benefits, where the coverage method achieves 73.1% pass@k performance compared to ~72% for alternative approaches at 128 samples. The consistent superiority of Coverage sampling across challenging benchmarks (ProverQA and BigBench Extra Hard) suggests that diverse solution exploration outweighs consensus-based aggregation for com- plex reasoning tasks. These findings support the position that extended reasoning processes, when properly sampled, enhance the prediction accuracy of difficult problems in rule-based reasoning.
C.4 COMPARISON WITH MORE BASELINES
We compare the differences and advantages of DADS versus DAPO and ADARFT on three OOD benchmarks. DAPO's s dynamic sampling employs an online filtering scheme, which repeatedly sam- ples and discards rollout generations until rewards fall within a target range, ensuring mix of par- tially correct and incorrect answers. DADS is more efficient than DAPO with its dynamic sampling. DADS samples training problems before responses are generated, preventing wasted computation of creating and then discarding unwanted responses. ADARFT relies on an opaque deterministic difficulty estimation based Qwen2.5-Math-77B (Yang et al., 2024), which limits the precision of initial difficulty judgement and introduces inductive bias from the L.L.-as-aajuu
model (Gu et al., 2024). For direct comparison, DADS only needs coarse-rrained metadata labels from original data sources (it can be the form of
for domain partitioning, reducing the need for prior dependencies like domain annotation or problem difficulty scoring.
Model
BBH
ProverQA
BBEH
Average (A)
Qwen3-8B-Bal
21.2
13.4
8.0
14.2 (-)
+ DAPO (w/ dynamic sampling)
95,4
68.8
62.0
75,4 (+61.2)
ADARFT
96.4
73.4
64.5
78.1 (+63.9)
+ DADS (Ours)
99.6
76.6
68.9
81.7 (+67.5)
Table 6: Performance comparison across different methods on BBH, ProverQA, and BBEH.
C.5 ROBUSTNESS ANALYSES ON HYPERPARAMETERS OF DADS.
To investigate the robustness and sensitivity of the hyperparameters, we conduct concise analyses of the sensitivity and effect of the hyperparameters т, and in DADS. Specifically, we keep two of the three hyperparameters constant and incrementally adjust the remaining hyperparameter. For each hyperparameter combination, we observe the model's 's performance on the BBH, ProverQA, and BBEH, three OOD benchmarks after convergence.
Table 7: Hyperparameter sensitivity analysis for different parameters
Hyperparameter Sensitivity Analyses
Avg. Pass@1 (OOD)
Avg. Pass@1 1 (OOD)
Avg. Pass 1 (OOD)
0.1
68.6
0.1
67.6
0.1
70.4
0.3
69.3
0.3
69.4
0.3
70.2
0.5
70.4
0.5
70.1
0.5
69.9
70.4
0.8
69.3
1.0
69.8
1.0
69.7
1.0
68,9
As depicted in Table 7. We list the observations and effects as below.
Smoothing factor Performance shows inverse "U-shape curve", initially increasing and then decreasing as a increases. Optimal performance observed around = 0.5 with 70.4% @ 1. Larger means more dependence historical rewards. An excessively high or low a can degrade performance, suggesting a sweet spot for balancing historical and current rewards.
23

---

Published as a conference paper at ICLR 2026
Temperature for magnitude control: Performance generally improves T increases, with slight dip at r = 1.0. The best performance is observed at T 0.8 with 70.4% pass@1 1. Smaller t indicates more sensitivity to reward fluctuations. Higher (up point) seem to improve performance, suggesting that less sensitivity to individual fluctua- tions can be beneficial.
Minimum sampling weight Observation: Performance generally decreases in- creases. The best performance is at 0.1 with 70.4% pass@1 Larger e indicates more average sampled number for each domain. This suggests that increasing the minimum sam- pling weight leads to a slight degradation in performance, implying that a lower e (allowing for more varied sampling) is generally better.
C.6 EXTRAPOLATING TO UNSEEN REASONING TASKS.
As presented in Table 8, we further explore the generalization to other verifiable domains, such as math, multi-disciplinary, and symbolic reasoning. We complement expanded inference-only OOD evaluation on more general reasoning tasks and show the extended OOD evaluation results including: (1) math (AIME 2025) (AIME, 2025); (2) multi-disciplinary (GPOA-Diamond) (Rein et al., 2024); and (3) symbolic (Coin Flip) (Wei et al., 2022b) reasoning benchmarks based on RULEREASONER which is only trained on the rule-based reasoning tasks described in $3.3.
Table 8: Additional evaluation results AIME 2025, GPQA (Diamond), and Coin Flip.
Model
AIME 2025
GPQA (Diamond)
Coin Flip
Average (A)
Qwen3-8B-Base
3.3
16.6
44.7
21.5 (-)
DADS (Ours)
23.3
44.9
95.1
54.4 (+32.9)
C.7 EFFECTIVENESS DADS ON EXTREMELY HIGH DIFFICULTY.
To investigate whether DADS, the dynamic sampling approach, can still be a highly difficult learn- able set of tasks by assigning lot of computations to learn these samples, we identify the existence of an unlearnable set of extremely difficult tasks within our training domains, such as AR-LSAT. As Figure 4 illustrates, the training rewards for AR-LSAT start at approximately 0.11 and steadily increase to approximately 0.42. This improvement correlates with an increase in the training sample size, indicating more assigned training computation.
Interestingly, even after Step 120, when the AR-LSAT domain was allocated nearly 20% of the com- putation in each training batch, other domains maintained their upward slope in reward/validation pass@ @1 1 curves. This leads to conclude that the DADS algorithm achieves optimal balance in computation assignment within multi-domain reinforcement learning dynamics. Although the ex- perimental dashboard (Figure might suggest an occasional over-emphasis one domain, overall performance indicates an effective allocation strategy.
To empirically demonstrate it, we provide additional comparison experiment between RULEREA- SONER with DADS and without DADS on the data mixing training (infusing the two-curve compar- ison of AR-LSAT).
We perform GRPO-style RLVR training with or without DADS on our curated rule-based reasoning dataset. We conduct interval evaluations every 50 training steps to assess the reasoning accuracy (Pass 1) on AR-LSAT, which we assume is an extremely hard domain to optimize with DADS. Finally, we compare the results of two setups and showcase the step-wise gains of validation per- formance. The comparison results between "w/o DADS" and "w/ DADS" assessed on AR-LSAT are shown in Table 9.
C.8 COMPARISON WITH DATA SCHEDULING AND BALANCING BASELINES
We provide additional simple data scheduling baselines including Easy-to-hard SFT (Sun et al., 2024), Easy-to-hand hard RL (Parashar et al., 2025), and mata-balance RL (Parashar et al., 2025) for comprison purposes with DADS.
24

---

Published as a conference paper at ICLR 2026
Table Performance comparison of Pass @ with and without DADS across different training steps.
Training Steps
w/o DADS
w/ DADS (A)
30.0
32.6 (+2.6)
100
30.9
36.8 (+5.9)
150
34.3
37.0 (+2.7)
200
35.1
38.5 (+3.4)
250
34.8
39.0 (+4.2)
300
38.0
41.7 (+3.7)
Baselines setup. Easy-to-hard SFT RL: GRPO-style RL partitioned data subsets, respectively. First, adopt LLM--s-a-judge with Qwen3-8B-BBl BaBee to obtain difficulty scores of problems. This allows to partition the training data into eight domains based on difficulty. We then train Swen3--B-Base model without DADS on this offline-partiionn partitioned data, progressively moving from easier to harder problems with the same portion of training epochs for each data partition. Data-balance RL: We perform GRPO-style policy gradient- based RL training using balanced domain setup. First, based on the training data statistics Table 5, we calculate the average samples per domain across all eight domains as 3,571 problem samples. Second, we apply repeated Nomain-balancee sampling to ensure 3,571 sampled problems for each domain. This involves down-sampling domains with more samples than the average and up-sampling those with fewer. Finally, disable DADS and initiate RL training from scratch with this lata-balanced setup.
We perform progressive curriculum training using SFT
Results. As shown in Table 10, we demonstrate that training with DADS outperforms other three baselines of Easy-to-hard SFT / RL and Data-balance RL recipes both the three OOD rule-based reasoning benchmarks and ODD general reasoning benchmarks, such as AIME 2025, GPQA (Diamond), and Coin Flip.
Table 10: Performance comparison across different benchmarks and training methods
BBH
ProverQA
BBEH
Average (Á)
Qwen3-8B-- Base
21.2
13.4
8.0
14.2 (+)
Easy-to-hard SFT
89.2
31.8
17.7
46.2 (+32.0)
Easy-to-hard RL
96.0
72.2
63.5
77.2 (+63.0)
Data-balance RL
94.8
69.4
60.5
74.9 (+60.7)
DADS (Ours)
99.6
76.6
68.9
81.7 (+67.5)
AIME 2025
GPQA (Diamond)
Coin Flip
Average (A)
Qwen3-8B-BB
3.3
16.6
44.7
21.5 (-)
Easy-to-hard SFT
6.6
33.8
48.7
29.7 (+8.2)
Easy-to-hard RL
20.0
39.8
91.4
50.4 (+28.9)
Data-balance RL
16.7
38.3
92.8
49.2 (+27.7)
+ DADS (Ours)
23.3
44.9
95.1
54.4 (432.9)
C.9 IMPACT OF RULE-BASED REASONING DATA CURATION
To the best of our knowledge, there is no accessible dataset used for training rule-based reason- ers. We argue that current approaches over-specialize in math and code (Yu et al., 2025), omitting the scalability of general rule-based and natural language reasoning. This narrow data focus hin- ders progress on broader reasoning capabilities, a critical bottleneck we aim to address. Moreover, ($3.3), is integral to evaluate DADS,
collected multi-domain dataset, RULECOLLECTION-3 which specifically mitigates online training problem scheduling for such data in RLVR for LLMs.
Without these intrinsically linked multi-domain datasets, the full benefits of DADS not be com- prehensively demonstrated. As depicted in Table 11, additional experiments (RL on math-only data, RL. on rule-only data, and RL jointly on both of them) using naive GRPO objective (without DADS)
25

---

Published as a conference paper at ICLR 2026
Table 11: Performance comparison on AIME 2025 and ProverQA across different training datasets.
AIME 2025
ProverQA
Qwen3-8B-Bat
3.3 (-)
13.4 (-)
AIME 1983-2024
63.3 (+60.0)
38.8 (+25,4) .))
RULLCOLLLECTIO-222 (Ours)
23.3 (+20.0)
73.6 (+60.2)
AIME 1983-2024 + RULECOLLECTION-32K (Ours)
66.7 (+63.4)
80.4 (+67.0)
further underscore the irreplaceable necessity of our curated data, proving can not be substituted by training solely existing math problems such as AIME 1983-2024 (Veeraboina, 2023).
D COMPUTATIONAL ENVIRONMENTS
D.1 INFRASTRUCTURE & HYPERPARAMETERS
We list the details of the computational environments of training and inference in Table 12.
Table 12: Hyper-parameters RULEREASONER-44 and RULEREASONER-S on-policy RL train- ing and inference.
Computational Infrastructure 4 x A100-SXM4-8000 GPU
Hyperparameter
Assignment
Hyperparameter
Assignment
Base model
RULEREASONER--H
Base model
RULEREASONER-3B
(Qwen3-4B-Base
(Qwen3-8B-Bi)
Training strategy
FSDP (Zhao al., 2023)
Training strategy
FSDP (Zhao al., 2023)
Training epochs
3
Training epochs
3
Training precision
bfloat16
Training precision
bíloat16
Mini batch size
65,536 tokens
Mini batch size
65,536 tokens
Micro batch size
16,384 tokens
Mícro batch size
16.384 tokens
Max input length
1,024
Max input length
1,024
Max output length
2,048
Max output length
2,048
Max learning rate
le -
Max learning rate
le -
Optimizer
AdamW
Optimizer
AdamW
Adam beta weights
0.9, 0.95
Adam beta weights
0.9, 0.95
KI. loss coefficient
0
KI. loss coeffccient
0
Rollout per sample
64
Rollout per sample
64
Clip ratio
0.2
Clip ratio
0.2
Gradient clipping
1.0
Gradient clipping
1.0
Evaluation steps
10
Evaluation steps
10
Save steps
10
Save steps
10
Training engine
PyTorch Paszke et al. (2019)
Training engine
PyTorch (Paszke et al., 2019)
Decay factor a
0.5
Decay factor
0.5
Base reward
0.1
Base reward
0.1
Scaling factor
0.8
Scaling factor
0.5
Rollout engine
vLI.M (Kwon al., 2023)
Rollout engine
vLIMM (Kwon al., 2023)
Rollout temperature
0.8
Rollout temperature
0.8
Rollout top-p
1
Rollout top-p
1
Random seeds
[21, 42, 84)
Random seeds
[21, 42,
E EXAMPLE TRAJECTORIES
We provide additional positive and negative examples (i.e.. failed cases) with fine- mieeggraned trajec- tories from various tasks to demonstrate clear step-by-step logical reasoning patterns and well- organized thought processes of RULEREASONER after training in Table 13, Table 14, and Table 15.
26

---

Published as a conference paper at ICLR 2026
E.1 IN-DISTRIBUTION POSITIVE EXAMPLES
Table 13: in-distribution positive examples of RULEREASONER on various datasets.
Dataset
Input
RULEREASONER
To determine how Natasli is related to Pronces, let's snalyze the information provided and trace family relationships slep by step. Given Informatio:: Jamex his soe Fruncisco out husting trip. This menns James Francisco's falherr Step-hy-Siep Remsoning: I. James is Francisoa's i fulter, Francisoo is Inmes's is son. 2. Since Frances is James's wife. she is also Francisco's 's mother. 3. Since Francisco is James's son and France''s son, Natnshu is Frincssoa's daughter, making Natacla James's grankäangher Frances's granddlughter Couclusion: Natacha the grundduightr of Friness. Answer grankluugher /
Francisco out hunting trip. Natssha Frinison's diughter, wanted to come, but they thought she wis tno young. Inssend, James's wife Frances stayel hor.
Confest:
James look
Clutrr
Question: How Natasha related to Fronces?
Diamond Mine is professional wreuting stable, formed WWF, Roderick Stooe leads Diamtond Mine. Damond Mine includes the Creed Brothers, and try Nile. Ims- periums his fend with Diamood Mine.
Confext:
To determine whether the statement "Koderick Strong leads professional westiing stable" is false, unknown besed on the given Thee contest states: ... 3. From the contest, know that Rodarick Strong leads Diamand Mine. and Dianood Mine is grofessional wresting stable. 4. Therefore, since Koderick Strong leads Dianced Mine, and Diamond Mine a professional wreatlng stbble, follows that Rederick Strong leads professional evestling stable. Answer: true
analyse infommation providal::
Folio
Question: Roderick strogg leads professional wrestling stable.
Contest: A fruit stand sells live fruits: apples, loquass, cantaloupes. watermelons, and kiwis. The watemelooss loquat. watermelons are less expensive than the kiwis. The loquals are more expensive than the apples. The cantaloupes are the cheup-
To determine the correct order of the fruits from most expensive to least expersivee let's analyze the information provided: The cantaloupes are the cheapest This means cuntaloupes are the less expensive. 2. The loquits watermelons are more expensive than loquaes. Watermelons Loquats ons are less expensive than the kiwis. Kiwis ; Watermelins Now, let's picce this information together to establish the order from most expensive least expensive: the order from moost expensive least expensive Kiwis; Watermelons Loquats & Apples Cantaloupes apples are the thind most expensive. Apples are
expensive than
more expensive thin the apples. Loquit Apples 3. The watenuel-
Logical Deduction
Options: A) The apples are the most pensive. B) The loquats are the thurd-most ex- pensive. C) The cantaoouees are the third most expensive. The watermelons most expessive. bi The liwis are the third most expensive.
let's evaluate the options: order, so this is false. B) The loquats ure the third expeusive. Loquzts are third is the order, so this true. C) The cuntaloupes are the third most expensive. Cantaloupes are fifth in the under, so this false. D) The watermelons the third most expensive. Watermelons kiwis are the thor-moot expensive. Kiwis are finst
the third-
second in order, so this false. in the ooder, so this is false Answer: B /
Question: Which of the following is trae?
To devermine whether the statement "Queemie free-Jiving", let's amalyoe the information pro- vided
options step by step Given Information: Every protouea cither purasite free living, but not both. This means if protozos is parasitic. x cannot be free living, and vice vuma Slep-ly by Soep Ressuning: I. Queenic lives in a host From the given information, any protoron that lives in host or feeds nutrients is puravitic, Since Queemic lives host, sho falls into the category of protesoa that lives feeds on nutrients is panasilic Since Queenic lives a hust, she is pursoltic. Every protozon is either parastic or free living. but not both. If Queenie is parasitic, she cannot be free living. 4. Conclusion: Since Queenie parasitic, not free living.
Context: Every protozoa either parasitic free living, bui not both. Predzrick lives in bost. Any protosica thit lives in host or feeds hest.
ProverQA
Queenic does not Options: A) True B) False CJ Useertain Question: Queenie is free-living.
parasitic. Queenie nutrients.
2.
protuzoa
lives
or
To determine the correct answer, we need to analyze the role of castoms offior based on the givas contest and vee which option with their duties. Rensoning: 1. The prinary role of customs officer is to spprrvie, check, and collect customs duties on goods and articles entering leaving the country.
Custams is a state that super- vises, chocks and collects casioms duties all goods and articles entering and lesving country.
Context:
compliance with may need inspact laggage or packugos to verlly the contents and ensure they are properly declared. Checking the body of others (Option B) not a standard dury of customs offices. as it would involve invaiive procedures that are not typically part of castems inspection. daries
regulations,
officers
LogiQA
Options: A) Chock other people's luggage B) Check the body ethers C) pantiits of others D)Desann others for one day
"Taiing portnits others (Oycion
Detaining others for one day (Option D) more severe ausion and would require specific legail grouds. which are not mentioned in the context of customs dutee. Conclusion: The most appropriate action for csstoess officer, bosed on their roll, is check other people's laggage (Option A) to ensure compliance with customs regulations. Answer A /
unselated
Question: Accoeding to the showe delinition, when dary, customs officer can?
We need to analyze the informatio provided and trace Alex's lo''s properties through the relation- ships defined. Given Rulss: I. Every lempus Given Facts: Alex brimpus and lempus.
Rules:
Every vumpus is rumpus and a ster- Every rompus is brimpus and a lampus. Zumpuses wumpuses
is rompu and strrps. 2. Every rompus a
dumpuses, Each grimpus a shumpus and tumpus. Each lor- pus is mufiled and an impus, Every numpus a zumpuses and jumpuses. Every jampus is vumpus
brimpus gorpos. Siip- Reasaning: 1. is zumpus and a jompus. Since Alex is gorpus. Alex is nampus and a jompus. From mll 3: Zumpuses are wumpuses and dumpuses. Since Alex is zumpas, Alex wumpus and a dumpus, From is vanpus and a grimpus.
zumpus and zumpus a garuus. From rule 7: Every gorpes
ProntoQA
gorpus. Gorpuses
grimpus. Since Alex a jompus, Alex a vampus. From above ressoning. since joaupus, and every jempus is vumpus, Alex must be vumpus. 3. Conclasion: The statemeat "Alex is not vunpus"" would be false becouse we have determined that Alex indeed vom- pus.
Every jompus sumpus
grimpus. Facts: Alex a brimpus and lempus. Alex is zumpus and gurpus
Question: Alex vampus??
Answer:
27

---

Published as a conference paper at ICLR 2026
E.2 IN-DISTRIBUTION NEGATIVE EXAMPLES
Table 14: In-distribution negative examples of RULEREASONER on various datasets.
Dataset
Input
RULEREASONER
Context: As ftr the role of individual and nation
cuncerned, calieral quality mainly infiltrates people's mind and improwes people's realm by ferming a spe- eifle caltaml atmospherre; In other words, the role of cultural quality is indireet and absunet
Comeex Analysis:
The role of cultural quality deseribed indirect abstract. It mainly infilrales people's minds and impeoves realm by forming specific cultural annosspeee. 'hee faaction of humanistic spint described as more dircet and specifte. Question Andlysis: The question asks us tn compure the importance cultural quality and humanistic spirii based their functions. suggests that since the humanistic spirii more direct specitic, should consider insortance in relation to cultural quality. Reasoning: The context tells us that cultural quility is indirect and abstract. while the hemanistic spicit discet and specific. Since the humanistic spiit more direct and specific, likell
important humaniatic sputit R)Humnsistic spirt more impoe- tant than culural quality C)Culural quality and hu- manistic spirit are one thing DyWhen vigorously im- geuving quality national culture, we should at- tach great importince to the construction of humaani-- ric spirit
Optiions: AlCulural quality
LogiQA
unmeditte and tungible impact compared cultural quality. which more indirect abstract. Therefore, would reasonable conclude henanistic spiit is more important thas caltural quality in terms of its direct and specific function. Conclusion: Given that the humanistic spirit more direct and specitic, it impoctans than cultural quality. Answer: B
Question: The function of humanistic spirit is direct and specific
Step-hy-Step Renooning 1. The flrst message is not from Lula, Both the first and list messages are from the same person Therefore, the last message most also be from the same person the message. So, the
While vacation, Sukanya receives o-nail messiges from work, each message fromt one of three associates: Hilary, Jerome, and Lula. Sakanya receives st lesst one and no more thin two messages from
Contest:
fourth messages only Jerone message first three messages, the Jemome message must be oue of the first three messiges. The Jerome message on the day after Hillary must be the fourth meseage (since there are only Messiges
either Hilary
Jerome. 2. Messages from Jenome: Since
Sukasya receives message on the day is sent. more than one mees- sage is sent each day. The messages are rezeived in manner consistent with the following. The first mes- sage
Lula: Sukusyo receives least and two messages from each associate. Since misi he in the second, third, or foorth maseage If the first and fourth messages are from Hilary, then the second and third messages ausst inclade Luln and Jerome. If the first from Jerome, then Anilysing Options: Option Since Sakanya meceives lenst one
messges))
message not from
Lulo
Lala, Both the fisst and last messages are from some person. Exactly once Sukanya re- ccives message from Jerome on the day after receiv- ing from Hillry. Of the first three massgges actly
and fourth messages Hilary and message from each associate and there are four messeges, it's possible that Lala sends two messages. This oprion is necessmily trre. Option B: The first and fourth messages same person (either Hiltey Jerome). If and fourth messoges are from
include third messages
AR-LSAT
and founth messages
messages is frous Lula. B) Extetly two of the messages are from Jervene The second message is from Lala. D) third message from Hilary. sage from Jerome
Options: Exactly one
from Jarome, then Jeronse scads two messages. If Hilbry, then Jerome must send one message in the fins three messiges and one in the fourth message. Therefore, Jerome could send either one or twa messages. This oytion Option
fourth
secoad messige could frosh Lala, necessarily trw based on the constraints, This aprian not necescarily true. Oytion D. The third message could he from Hiliry, but it's not necessurily tmie hased on the constraints. Option E: Since evacily once Sukanya receives message from Jerome the day receiving one Hilary, on day after Hillay must the foutth message. Therefore, the fourth mossage mosst be from Jerume. Answer: F.X
necessarily
Question: If Sukenys receivess exnetly four e-mail messages, (hen which the following must true?
only messages, Jerome messge
Rules: If there least people who popu- lar, then Curt not mnahle. If there somoote loyal, the Brock is timid and Lucille not popu- lar. If there someone who timid, than Marcns shiny unable and Licille loyal. Ganth Marcus being equivalent Jed being not unable light. somoone who is not timid is also loyal, thas he is not undble. If there shiny, then Silas timid. Locille light, than Brock not popular. If Curt is light, men Marcus is populaz. Silis is light if and only if Silas is loyal and Brock is not unable. if someone is timid, he is loyal. these someone who unable and not timid, then is shiny. Somcone who is hoch not shiny and unable is always light.
Given Fists: 1. Silas is light. 2. Mancus is not timid. Given Rules: I. there is leust person who popular, then loyal,
lenst one people who
unable. 2. there someone who Lucille not popular.
given facts, we can see that **fad 7 dineclly states: Brock is inable. This meins that the statement 'Brock is unable" is *enaalled"* by the facts provided. Conclusion: Since the fact "Brock is unable" is explicitly given the facts, statement Brock is unablets ""nttaled" by the information. Answer: entalimentt x
Analyss:
LogicNLI
Facts: Silas light. Matens is nor timid, Marens is popular. Mascus light. Lucille
unable. unbble
loyal. Garth light. Brock is not shiny. Jed is papellar. Question: Brock unable.
loyal, Brock unuble,
28

---

Published as a conference paper at ICLR 2026
E.3
OUT-OF-DISTRIBUTION EXAMPLES
Table 15: OOD trajectories before and after training by RULEREASONER for comparison.
Dataset
Input
Before Training
After Training
To deteminne whether the statement Colten lov- able true, or uncertain
on the given context, let's anilyse the informstion step by step- From statement I: From statemest 3: Colten is either cocedinated or clumey, but not both, 3. F.Fmmm Collen good intentioons. From statement If Coltes has good inteations. then he either guod-maturan or well-mmening (bui not both). S. From statement 4: Collen is good- natured. Thersfore, from stasement 2. since is good he comnot be well-meaning (be- cuuse it's either good-tstuned well--enning but not buth), We know this Collen good- natured and good intentions. However, the not provide any direct information about whether Collen is locable. The attributes of being cour- dinated, good monueed and having good intention not necessarily imply that Colten lovable, as lovable is not explicily defined or linked these attributes in the given context. Therefore, bused the provided information, cannot dediniivelly determine whether Coluen lovable. Answer:
Step: 'Tii tells us that Colten is not clumey,
he cither coordinated or clumsy. but not both.
2. This conditional statement.
tion (good intentions) is trme, then the condi-
Coutext: Coltea is eedinated. If Cohtea has
the outcomes (good natured or well meining) mus be true. bur not both. This tottloggy based doesn't provide Colten good-astured. S. This the condition in the sec- ond statement, we can apply it. We know from the fourth statement that Colsen is good natured.
good intentions, thes he either good-aatured well meannng (but not both). Colten is either coardiaated or clumsy, but not both. good natured, has good inteutions. Options: A) 'Tue Bi False Cy Uncestain Question: Colten lovable?
the
formation, This directly
ProverQA
Colien
The torm lovable is subjective and not divectly fined in the given contest. However based on the information provided, we can infer thet Colten is good
does
person good inteutions ture. These trats geserally contribute somsone being considered lovable. Answer: A.
Contest: Tommy has fine Boece. For all sheep. sheep has fleece. either prized by farmers or produces quality wool toe both). Nolan has line fleece. Tommny is not prized tana- ens Question: Based an the above information. is the fol-- lowing statement trae, false, or uncestain? Tommy pro- duees quality wool. A) True B) Fulse C)
Tos dittermin whether the statement Tommy pro- duees quality wool is truc. false, mncertainn based given
The answer C) Uncertain. Step 1: Analyze fine fleece
given information sheep. a sheep fine flooce, then ei- ther prized by farmers or produces quality wool (oe both). Nolan has fine fleece. Tommy is not prized furmors. Step Apply the given infonmation qusstia We know that Tommy fine flooee. and acoveding to the given information, a shoep has fine floced, is cither prized by farmers pro- both). We also know Tommy not prized by firmers Step Determine the answer Since Tommy not prizod by farmars, cannot determine be produces quality woul bosed the given information.
let's analyze the information stop by step. Ressoning: "*Frces statement 1:* Tomany has fine fleese.2. reproan statement 2 (gea- eral raley=s If shoep has fine flocce, then is either prized by
produces quality wool
both). Since Tommy has fine flaoce (frcen state- ment 1), we can apply this mile to Tommy: Tommy is cither prized by fammers or produces quality wool both),3 ""Piom statement 4.7 Tommsy prized by firmens This means the prived by famm- ers post of the rule is false for Tommy-, Ap- plying the rule Tommy:* Since Tummy is prized farmers, the remuining possibility that Tommy produces quality wool.. Answer: A. /
ProverQA
quality
quilly
Options:
Uncertain
Question: Tommy produces
quality wool?
Answer: Uncertain. x
Context: The following pace- graphi each describe of thene objects arranged fixed order. The statements are logically consisten within each paragraph. On shelf, there are three books: yel. low book, a brown book, and red book. The red book to the book. The yellow book to the right of the brown book.
To determine which of the options moe, let's on- alyze given context step by step. Step Ua- derstand relationchipa between books. From statement 2: The red book is the left of the brown book. This means: Red book: Brown book From yellow book
The answer is C. Since the red book is to the len of the brown book, we place the red book first and the brown book second. the right of the brown book, so it must be placed third. New, lat's evaluate the options: (A) The yel- low book is the righemost: This
yellow book
statement This means: Brows boown book Step 2: Arming the books in onder. Combiii ing the two relarionships: Red book Brown book is: Red book, Rrown book, Yallow book. Step De- termine the rightioot book. In the order Red book, Browa book, Yellow book, the rightmost book is Yellow book. Stop 4: Einhatt the oprions. rightmest From our arrange meni, the yellow book is indoed the rightmoot. The rod book is the leftmost, not the rightmost. This tion "Taase"
right Yellow
BigBench Hard
the brown
the yel- placed third trangement. brown book the rightmost: This is false, as the
Yellow book So, the order from
Options: (A) The yellow book is rightmast (B) The boown book is rightmest
brown book placed second in the araaggemett
(C) The red book the rightmust This false, as
book placod
arragement.
The yellow
(C) The red book the right- most
Answer: C.
Question: Which the lowing options true?
Answer /
29


Reviews:
[{"review_id": "uVGn6SGCRH", "reviewer": "Reviewer_x9cg", "summary": "The paper addresses the challenge of enhancing the *rule-based* reasoning capabilities of language models by introducing a new training framework, RuleReasoner. The framework first introduces a new dataset (RuleCollection-32K), consisting of 8 different domains covering different logical reasoning forms. Then, the paper introduces DaDS, which dynamically samples batches by reweighting over the domains, then performing RLVR with rule-based exact match reward. The efficacy of this is validated with Qwen3 base models (4B and 8B) and with numerous ablations.", "questions": "1. Why -1 and +1 instead of the \"usual\" 0-1 reward for RLVR? At least the 0-1 reward is what I usually see in RLVR literature.\n2. In the abstract and throughout, the authors claim that the proposed method is more sample-efficient. And indeed the figures do support that claim. But to my eyes, at least in terms of efficiency, the gap between Rule Reasoner and other RLVR approaches (GRPO, Dr. GRPO, DAPO) seems to be a bit small...? It would be helpful to see the error bars for Figures 6 and 7.\n3. As in R1, does the response length significantly increase with training? Are there any complex behaviors emerging beyond improved rule applications?\n4. What do the authors mean by \"the order of contextual logical rules are randomly shuffled for each training sample.\"? Some concrete examples would be very helpful here.\n5. Here, is it usual to consider only pass@1? How about pass@k or BoN? How do RuleReasoner-4B and 8B interact with standard test-time scaling methods?\n6. Have the authors considered other models? Perhaps stronger ones, such as Qwen3-Instruct or Qwen3-Thinking? How about other families like Llama? I'm not suggesting that the authors run all of these ablations (obviously due to time and resources constraints), but I am particularly curious about whether one can do the RuleReasoner framework with a stronger model and whether this would yield a better logical reasoning model.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Impressive empirical performance\n- New logical rule data that would be helpful for future research", "weaknesses": "- Mathematical notations should be improved overall:\n   - I believe that this notation is the clearest: $\\mathcal{D}$ represents a fixed (offline) collection of $(d, q, r, y)$, where $d \\in \\\\\\{d_1, \\cdots, d_n\\\\\\}$ represents the domain. (In Algorithm 1, it states that $\\mathcal{D} = \\\\\\{d_1, \\cdots, d_n\\\\\\}$, which seemd a bit weird and contradictory to prior notations)\n   - In Algorithm 1, initializing $\\tilde{r}_{0, d_i} \\gets 0$ is missing; for notational clarity, $m$ should be written as $m_i$\n   - It should be explicitly stated that $\\alpha \\in [0, 1]$.\n- When $r_{target} = 1$, isn't $r_{target} - \\tilde{r}$ always nonnegative? This is because $|\\bar{r}\\_s| \\leq 1$ always, and thus, $|\\tilde{r}\\_s| \\leq \\alpha |\\tilde{r}_{s-1}| + (1 - \\alpha)|\\bar{r}_s| \\leq 1$ by induction. Thus I don't think that $max(0, 1 - \\tilde{r}_s)$ is not necessary; simply writing $1 - \\tilde{r}_s$ is enough.\n- In Section 3.2, please include why eliminating the KL term is okay for this situation. As written by Liu et al. (2025), I believe the crucial reason is that the reward is rule-based, thereby eliminating concerns about distributional shift.\n- Discussions related to prior approaches on (iterative) reweighting and difficulty-aware approaches [1,2,3] are lacking. Why can prior approaches not be used directly here? Is the proposed Dads algorithm really as novel as the authors claim? What are the differences? I'm especially interested in knowing the difference between this work's reweighting scheme and [1].\n   - Could it be that the importance sampling is equivalent to some form of reweighted RLVR?\n- The authors claim that $\\epsilon$ ensures a minimum sampling weight for all domains. But, due to the translation invariance of softmax, we have that (ignoring $s$)\n$$ w_i^{norm} = \\frac{\\exp((v_i + \\epsilon) / \\tau)}{\\sum_j \\exp((v_j + \\epsilon) / \\tau)} = \\frac{\\exp(v_i / \\tau)}{\\sum_j \\exp(v_j / \\tau)}$$\nand thus, $\\epsilon$ plays no role.\n- The above point leads to the following suspicion: the authors claim that in Appendix C.5, $\\epsilon = 0.1$ leads to the best performance. Considering how $\\epsilon$ should not impact the performance in principle (maybe floating point level discrepancy, but nothing significant), the conclusion from this ablation seems to be flawed.\n\n[1] https://openreview.net/forum?id=lXuByUeHhd\n\n[2] https://arxiv.org/abs/2408.09849\n\n[3] https://openreview.net/forum?id=GLUIuli3Sm", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eCiOMUPoiI", "reviewer": "Reviewer_NRKF", "summary": "This paper introduces RULEREASONER, a framework for reinforced rule-based reasoning that utilizes a \"domain-aware dynamic sampling\" approach. The authors claim that RULEREASONER enhances language models' ability to perform rule-based reasoning, mitigates challenges from varying rule complexities, and improves efficiency, outperforming frontier large reasoning models (LRMs) on both in-distribution (ID) and out-of-distribution (OOD) benchmarks.", "questions": "see above", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This paper introduces a dynamic sampling approach to stabilize RL training across imbalanced domains, which is a practical consideration.\n2. RULEREASONER achieves quantitative performance improvements.", "weaknesses": "1. Lack of comparison with existing adaptive sampling methods: Since the core contribution of this paper appears to be the \"domain-aware dynamic sampling\", it is better to include prior curriculum learning or adaptive sampling methods for comparison.\n2. It can be seen from Table 2 that RL-based methods show relative low performance on LogicNLI, AR-LSAT compared to SFT w/ CoT methods. Could authors explain this phenomenon? Besides, according to the avg results, the performance of models after SFT w/ Short/Long CoT seems to be higher than models after RLVRs, which is contrary to popular belief. The authors should provide a brief analysis about the reasons and explain why they still select to design a data sampling method for RL instead of SFT in this situation.\n3. Limited Generalizability and Scalability: All experiments are conducted on relatively small Qwen3 models (4B and 8B). This severely undermines the method’s scalability to \"larger-scale modeling,\" especially when the authors themselves acknowledge scalability as a future work limitation. It's unclear if the observed benefits hold for state-of-the-art larger models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Rz29ZG7yag", "reviewer": "Reviewer_7sWJ", "summary": "The authors create a model (RuleReasoner) for rule-based reasoning through the creation of a data collection which spans eight logical reasoning tasks with various rules and rule types. Then, a modified RLVR framework is used to train their RuleReasoner model that is designed to explore reasoning steps and improve generalization. Within the training process is a dynamic sampling algorithm aimed at reweighting domains that are undertrained. The result is that on several benchmarks, their RuleReasoner (4B, 8B) outperforms or performs competitively against significantly larger models and baseline RLVR methods both in-domain and OOD. Comprehensive analysis and ablations demonstrate the effect of the domain-aware sampling, the benefits of using their assembled data collection, as well as additional insights.", "questions": "1. Is RuleCollection too easy? It can be a useful dataset for SFT/GRPO but is it still useful for benchmarking? A related question is what is the human average across the collection?\n\n2. DADS is applicable to other multitask/generalized RLVR problems. While this paper makes no strong claims of generalizability, let’s speculate a little. Would this approach (along with no KL/Entropy bonus) work on a collection of math or coding problems? Or are there substantial limitations why it may not directly transfer?\n\nMisc.\n```\n103: “It has demonstrated” -> “It has been demonstrated”\n178: “RLVR, aim” -> “RLVR aiming”\n250: grammar/phrasing confusing\n252: Correctly answering require*s*\nL373: “tranining”\n```", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "* A large collection of datasets for rule reasoning of varying formats difficulties/depth, inference types, etc. The experiments demonstrate that these are useful for training.\n* The paper proposes a domain-adaptive algorithm for data sampling for RLVR, named DADS, which modifies the reward to be domain-normalized. The rewards per domain are also tracked and used for sampling data in the next batch.\n* Comprehensive empirical results on a large suite of both logical inference and mathematical reasoning demonstrate the effectiveness of DADS.\n* Detailed analysis and ablations showing how various design decisions contribute to the learning dynamics or performance.", "weaknesses": "These are minor suggestions. I did not find major weaknesses given this topic and area.\n\n* The evaluations and collection is ultimately limited by the available data from other logical reasoning datasets; the authors did not collect or generate their own. As a result, they could be missing some rule reasoning types or be over-indexed on some. While dynamic sampling addresses the latter, it cannot fill gaps in missing reasoning types.\n* Thanks for including the case studies (and C.2) – however they are not too illustrative of what is being learned. In other words, while there is learning dynamics per-domain, there is no analysis around rule/reasoning type (depth, format, chaining, etc). That would be insightful if, for example, something like C.7. was carried out for the rule types rather than domains.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bunCRE3bt2", "reviewer": "Reviewer_fFpE", "summary": "The paper targets rule-based reasoning—an area where general language models especially struggle—and proposes RuleReasoner: a reinforcement learning with verifiable rewards (RLVR) framework augmented by Domain-aware Dynamic Sampling (DADS). DADS adaptively shifts training probability toward domains with lower historical rewards, balancing progress across tasks (domains) and mitigating over-specialization. The dataset spans eight domains with varied rule types and depths; training further removes KL loss to encourage diversity and shuffles rule order to discourage memorization. Experiments on BBH, ProverQA, and related suites show that RuleReasoner-8B surpasses strong baselines (including several frontier LLMs on selected tasks), while RuleReasoner-4B approaches or exceeds peer models at similar scale—suggesting practical sample/compute efficiency and SLM-friendliness.", "questions": "1. Why didn’t the authors include AdaRFT (and the DAPO dynamic sampling variant) in the main results? To allow a fair comparison, the performance of AdaRFT and DAPO (with dynamic sampling) on the full set of benchmarks (as in Table 2) should be reported, since it likely provides a strong baseline for adaptive sampling in RL training.\n\n2. How exactly does dynamic domain sampling lead to better OOD generalization? For example, does the learned sampling policy encourage learning more abstract rules or reduce overfitting to training-task patterns? Without clarifying why reweighting domains yields better generalization, the connection between the proposed training strategy and the OOD success feels tenuous—we see the improvement in numbers, but not the underlying cause.\n\n3. Why is DADS necessary or superior to simpler data-balancing schemes? A more thorough analysis (e.g., ablations of DADS components instead of entirely removing DADS in Table 9, or comparisons to a static mixed-curriculum baseline) would strengthen the justification. As it stands, we know DADS yields better results, but we don’t know what unique role it plays in training or why alternative scheduling methods underperform, leaving the rationale for DADS somewhat weak.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* **Addresses a real challenge:** Domain imbalance in multi-domain RL training is a genuine problem. DADS provides an intuitive solution using historical reward signals to guide sampling without requiring human-designed curricula or expensive recomputation.\n\n* **Strong OOD generalization:** The out-of-distribution results (Δ 10.4% over OpenAI-o1 on average across BBH, ProverQA, BBEH) demonstrate generalization beyond the training data. Consistency across multiple OOD benchmarks and additional tasks (AIME, GPQA, Coin Flip) strengthens this claim.\n\n* **Comprehensive evaluation:** The paper includes appropriate baselines across multiple categories (frontier models, prior rule-based reasoners, SFT variants, RLVR methods), statistical testing with multiple seeds, ablation studies, and insightful learning dynamics analysis (Figures 4–6).\n\n* **Reproducibility:** Detailed hyperparameters, data sources, and prompts are provided. The authors commit to releasing code and data.", "weaknesses": "1. **Missing competitive baseline AdaRFT:** The main results table (Table 2) includes many baselines from different viewpoints, but it omits AdaRFT and DAPO (with dynamic sampling), which is arguably the most competitive related work. AdaRFT (Adaptive Curriculum Reinforcement Finetuning) also uses a dynamic sampling strategy to balance training difficulty [1], making it directly comparable to the proposed approach. In the paper, AdaRFT appears only in Appendix Table 6 (evaluated on three OOD datasets) instead of the comprehensive results in Table 2.\n\n2. **Unclear OOD generalization mechanism:** The method (RuleReasoner with DADS) demonstrates improved out-of-distribution performance (a 10.4% gain over three OOD benchmarks), but the reason for this improvement remains unclear. The authors attribute the ID gains to balancing training across domains (preventing any single task from dominating; Line 344) while attributing the OOD gains to extrapolation capability (Line 364), but this explanation is superficial and doesn’t reveal what the model actually learns. It’s uncertain whether the model is acquiring more general reasoning patterns or simply benefiting from a data augmentation effect (i.e., seeing more diverse examples). No qualitative analysis or insight into the model’s behavior on OOD inputs is provided beyond the aggregate accuracy numbers. This lack of explanation is problematic because robust generalization is a central claim of the paper. Prior work has shown that even advanced LLMs have significant gaps in handling complex or compositional rules [2], so the reader expects a deeper discussion of how this approach addresses those gaps.\n\n3. **Insufficient explanation for DADS’s effectiveness:** The paper provides empirical evidence that it improves performance—e.g., Table 9 shows higher accuracy at various training steps with DADS compared to without DADS. However, the analysis stops at reporting metrics and does not explain why DADS is so effective. The authors state that DADS “stabilizes the dynamics of on-policy RL training and mitigates over-optimization,” but this claim is made without deeper investigation or evidence. It remains unclear how DADS achieves these benefits: Does it prevent the model from overfitting on easier tasks by continually challenging it? Does it ensure more evenly distributed gradient updates across tasks, acting as an implicit curriculum? These possibilities are not explored. The paper essentially shows that “DADS works better” but offers little insight into the conditions or mechanisms that make it work. Can you demonstrate DADS’s necessity by comparing it to simpler baselines like uniform domain sampling, round-robin scheduling, or periodic domain rebalancing against your reweighting from historical rewards?\n\n[1] [2504.05520] Efficient Reinforcement Finetuning via Adaptive Curriculum Learning\n[https://arxiv.org/abs/2504.05520](https://arxiv.org/abs/2504.05520)\n[2] Can LLMs Reason with Rules? Logic Scaffolding for Stress-Testing and Improving LLMs — ACL\n[https://aclanthology.org/2024.acl-long.406/](https://aclanthology.org/2024.acl-long.406/)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oAqJytfkg6", "reviewer": "Reviewer_VMSU", "summary": "This paper introduces RuleReasoner, a reinforcement learning framework designed to enhance large language models’ rule-based reasoning ability. The authors construct a new dataset, RuleCollection-32K, covering eight logic and reasoning domains (e.g., ProofWriter, ProntoQA, AR-LSAT, LogiQA). The model is trained with Reinforcement Learning with Verifiable Rewards (RLVR) and a new sampling strategy called DADS (Domain-aware Dynamic Sampling), which adjusts training focus dynamically based on domain performance. Experiments on both in-distribution and out-of-distribution benchmarks show consistent improvements over baselines such as GRPO, DAPO, and even OpenAI o1-mini on some tasks.", "questions": "See Questions", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The Domain-aware Dynamic Sampling methods is simple and experiments show that the effectiveness on enhancing the logical reasoning ability with RL.", "weaknesses": "The Domain-aware Dynamic Sampling also is not really something new. There are many works related to sampling data regarding to the weights of domains. It'd be more comvincing for the authors to discuss related works or the technical contribution of the paper is not enough.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)