RuleReasoner: Reinforced Rule-based Reasoning via Domain-aware Dynamic
  Sampling
Authors: name: Yang Liu, name: Jiaqi Li, name: Zilong Zheng
Organization: Beijing Institute for General Artificial Intelligence

Rule-based reasoning has been acknowledged as one of the fundamental problems
in reasoning, while deviations in rule formats, types, and complexity in
real-world applications pose severe challenges. Recent studies have shown that
large reasoning models (LRMs) have remarkable reasoning capabilities, and their
performance is substantially enhanced by reinforcement learning (RL). However,
it remains an open question whether small reasoning models (SRMs) can learn
rule-based reasoning effectively with robust generalization across diverse
tasks and domains. To address this, we introduce Reinforced Rule-based
Reasoning, a.k.a. RuleReasoner, a simple yet effective method to conduct
rule-based reasoning via a wide collection of curated tasks and a novel
domain-aware dynamic sampling approach. Specifically, RuleReasoner resamples
each training batch by updating the sampling weights of different domains based
on historical rewards. This facilitates domain augmentation and flexible online
learning schedules for RL, obviating the need for pre-hoc human-engineered
mix-training recipes used in existing methods. Empirical evaluations on
in-distribution (ID) and out-of-distribution (OOD) benchmarks reveal that
RuleReasoner outperforms frontier LRMs by a significant margin (Delta4.1%
average points on eight ID tasks and Delta10.4% average points on three OOD
tasks over OpenAI-o1). Notably, our approach also exhibits higher computational
efficiency compared to prior dynamic sampling methods for RL.