Information Gain-based Policy Optimization: A Simple and Effective
  Approach for Multi-Turn LLM Agents
Authors: name: Guoqing Wang; name: Sunhao Dai; name: Guangze Ye; name: Zeyu Gan; name: Wei Yao; name: Yong Deng; name: Xiaofeng Wu; name: Zhenzhe Ying
Published: 2025-10-16 17:59:32

Abstract: Large language model (LLM)-based agents are increasingly trained with
reinforcement learning (RL) to enhance their ability to interact with external
environments through tool use, particularly in search-based settings that
require multi-turn reasoning and knowledge acquisition. However, existing
approaches typically rely on outcome-based rewards that are only provided at
the final answer. This reward sparsity becomes particularly problematic in
multi-turn settings, where long trajectories exacerbate two critical issues:
(i) advantage collapse, where all rollouts receive identical rewards and
provide no useful learning signals, and (ii) lack of fine-grained credit
assignment, where dependencies between turns are obscured, especially in
long-horizon tasks. In this paper, we propose Information Gain-based Policy
Optimization (IGPO), a simple yet effective RL framework that provides dense
and intrinsic supervision for multi-turn agent training. IGPO models each
interaction turn as an incremental process of acquiring information about the
ground truth, and defines turn-level rewards as the marginal increase in the
policy's probability of producing the correct answer. Unlike prior
process-level reward approaches that depend on external reward models or costly
Monte Carlo estimation, IGPO derives intrinsic rewards directly from the
model's own belief updates. These intrinsic turn-level rewards are combined
with outcome-level supervision to form dense reward trajectories. Extensive
experiments on both in-domain and out-of-domain benchmarks demonstrate that
IGPO consistently outperforms strong baselines in multi-turn scenarios,
achieving higher accuracy and improved sample efficiency.