Title: Pruning Long Chain-of-Thought of Large Reasoning Models via Small-Scale Preference Optimization
Primary area: foundation or frontier models, including LLMs
Keywords: Large Language Models, Large Reasoning Models

Abstract: Recent advances in Large Reasoning Models (LRMs) have demonstrated strong performance on complex tasks through long Chain-of-Thought (CoT) reasoning. However, their lengthy outputs increase computational costs and may lead to overthinking, raising challenges in balancing reasoning effectiveness and efficiency. Current methods solving this problem often compromise reasoning quality or require extensive resources. In this paper, we investigate how to reduce the generation length of LRMs with limited tuning. We analyze generation path distributions and filter generated trajectories through difficulty estimation. Subsequently, we analyze the convergence behaviors of the objectives of various preference optimization methods under a Bradley-Terry loss based framework. Based on the analysis, we propose Length Controlled Preference Optimization (LCPO) that directly balances the implicit reward related to NLL loss. LCPO can effectively learn length preference with limited data and training. Extensive experiments demonstrate that our approach significantly reduces the average output length by over 50\% across multiple benchmarks while maintaining the reasoning performance. Our work highlights the potential for computationally efficient approaches in guiding LRMs toward efficient reasoning. Code available at https://anonymous.4open.science/r/anonymous\_code\_74CD.

TL;DR: This paper analyzes existing preference optimization method and proposes a computationally efficient method to mitigate the problem of overly lengthy outputs for Large Reasoning Models.