Title: Enhancing Language Model Reasoning with Structured Multi-Level Modeling
Primary area: foundation or frontier models, including LLMs
Keywords: Chain-of-Thought reasoning, Direct Preference Optimization, Process supervision, Twisted Sequential Monte Carlo, Large language models

Abstract: Inference-time scaling enhances a model’s reasoning by extending its chain-of-thought (CoT). However, existing approaches typically rely on a single policy trained with outcome-reward RL fine-tuning, which often suffers from CoT derailment, i.e., the reasoning trajectory drifts off-course due to compounding errors. This problem is particularly severe for smaller LMs with long CoTs due to their limited capacity. To address this, we propose Multi-Level Reasoning (MLR), a framework that augments single-policy methods with high-level policy modeling. Specifically, MLR incorporates two policies: a high-level planner that generates step descriptors (abstract sub-goals) and a low-level executor that produces detailed content conditioned on these descriptors. The planner then summarizes the content to guide the next sub-goal, forming an alternating plan–execute loop. To reduce cost, we adopt a minimal design, where the base model serves as the low-level policy and a lightweight LoRA module implements the high-level policy. We further observe that outcome-reward RL is inefficient for long trajectories (e.g., exceeding 4K tokens). To overcome this, we introduce online Step-DPO, a process-level preference optimization scheme that leverages Twisted Sequential Monte Carlo (TSMC) to provide scalable stepwise supervision. This yields more efficient training, improved stability, and higher accuracy. Extensive experiments on challenging math, science, and logical reasoning benchmarks demonstrate that, with only 10% SFT data and 5% of preference pairs, our method outperforms both the DeepSeek-R1 distillation baseline and the outcome-reward RL baseline across multiple base models and tasks.