Title: Exploring the Entropy Mechanism in LLM Agents On-policy Optimization
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Reinforcement Learning; Large Language Models; LLM Agents

Abstract: Reinforcement Learning has emerged as a critical technique for training Large Language Model (LLM) agents in multi-turn interactive environments. However, sparse reward structures in long-horizon tasks—where agents receive feedback only after completing episodes spanning 30+ turns—create fundamental training instabilities that existing entropy regularization methods fail to address. We identify a critical oscillation pattern: early steps rapidly converge to low-entropy states due to limited feedback signals, triggering compensatory entropy spikes in later steps that compromise task completion. To address this exploration-exploitation dilemma, we propose Entropy-regularized Policy Optimization (EPO), a novel framework designed for stable on-policy training in multi-turn settings. EPO introduces two key innovations: (1) an entropy smoothing regularizer that mitigates abrupt entropy fluctuations across extended trajectories, and (2) an adaptive weighting scheme that dynamically balances exploration and exploitation based on training progression. We provide theoretical analysis demonstrating how entropy regularization prevents premature convergence in sparse reward environments. Extensive experiments on ScienceWorld and ALFWorld benchmarks show that EPO achieves up to 152\% performance improvement over baseline methods while exhibiting significantly more stable training dynamics. Our results validate that properly designed entropy mechanisms are essential for successful multi-turn LLM agent optimization.