Title: Group-in-Group Policy Optimization for LLM Agent Training
Venue: NeurIPS 2025
Accepted: yes
Keywords: Large Language Model

Agent

Group-Based Reinforcement Learning
Abstract: Recent advances in group-based reinforcement learning (RL) have driven frontier large language models (LLMs) in single-turn tasks like mathematical reasoning. However, their scalability to multi-turn LLM agent training remains limited. Unlike static tasks, agent-environment interactions unfold over many steps and often yield sparse or delayed rewards, making credit assignment across individual steps significantly more challenging. In this work, we propose Group-in-Group Policy Optimization (GiGPO), a novel RL algorithm that achieves fine-grained credit assignment for LLM agents while preserving the appealing properties of group-based RL: critic-free, low memory, and stable convergence. GiGPO introduces a two-level structure for estimating relative advantage: (i) At the episode-level, GiGPO computes macro relative advantages based on groups of complete trajectories; (ii) At the step-level, GiGPO introduces an anchor state grouping mechanism that retroactively constructs step-level groups by identifying repeated environment states across trajectories. Actions stemming from the same state are grouped together, enabling micro relative advantage estimation. This hierarchical structure effectively captures both global trajectory quality and local step effectiveness without relying on auxiliary models or additional rollouts. We evaluate GiGPO on challenging agent benchmarks, including ALFWorld and WebShop, as well as tool-integrated reasoning on search-augmented QA tasks, using Qwen2.5-1.5B/3B/7B-Instruct. Crucially, GiGPO delivers fine-grained per-step credit signals, achieves performance gains of > 12\% on ALFWorld and > 9\% on WebShop over GRPO, and obtains superior performance on QA tasks (42.1\% on 3B and 47.2\% on 7B): all while maintaining the same GPU memory overhead, identical LLM rollout, and incurring little to no additional time cost.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to address the credit assignment problem in multi-step reinforcement learning and discovers the anchor state phenomenon. The authors introduce step-level fine-grained advantage computation, which enables better reuse of existing data and combines with episode-level advantages to obtain more accurate advantage signals. The experimental results are impressive, demonstrating significant performance improvements.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. High writing quality and clarity: The paper is well-written and easy to understand, with well-formatted figures, tables, and equations that enhance comprehension.\n2. Accurate problem identification with strong motivation: The paper effectively targets the credit assignment problem in multi-step reinforcement learning with well-justified motivation.\n3. Clever method design: The authors first discover the anchor state phenomenon during experimentation, then introduce step-level advantage computation to fully utilize this data information, incorporating more fine-grained advantage signals. The computational overhead is negligible, requiring no additional rollouts or complex calculations.\n4. Impressive experimental results: The results demonstrate the effectiveness of the proposed method design.\n\nWeaknesses:\n1. Method design relies on task characteristics, raising concerns about generalizability: The method depends on identical states to construct anchor groups, which may only be applicable to environments where identical states can be detected. For more complex and open multi-step environments (such as multi-turn dialogue, web queries, textgame: SOTOPIA[1], GAIA[2], NegotiationArena[3], LMRL Gym[4]), this method may not be applicable. The two environments validated in the paper, ALFWorld and WebShop, are relatively simple with limited state spaces, making it easy for identical states to reappear during multi-step interactions.\n2. Limited hyperparameter sensitivity analysis: While I understand that multiple experiments are computationally expensive, conducting experiments on the weight parameter w to explore hyperparameter sensitivity would make the work more solid.\n3. Missing LRM baselines: Since the method employs the thinking-action paradigm, it should also compare against some LRMs such as DeepSeek-R1.\n4. Missing classic multi-step RL baselines: It is recommended to include classic baselines for multi-turn RL such as ArCHer[5] and Raven[6]. Raven may be recent work within three months before submission, so it's not mandatory if conditions don't permit.\n5. High computational resource requirements, difficult to reproduce: Based on the submitted code and my experience, the proposed approach requires substantial CPU (at least 64 cores) and GPU resources, resulting in significant computational overhead.\n\n[1] Zhou X, Zhu H, Mathur L, et al. SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agents[C]//The Twelfth International Conference on Learning Representations.\n\n[2] Mialon G, Fourrier C, Wolf T, et al. Gaia: a benchmark for general ai assistants[C]//The Twelfth International Conference on Learning Representations. 2023.\n\n[3] Abdulhai M, White I, Snell C, et al. Lmrl gym: Benchmarks for multi-turn reinforcement learning with language models[J]. arXiv preprint arXiv:2311.18232, 2023.\n\n[4] Bianchi F, Chia P J, Yuksekgonul M, et al. How well can LLMs negotiate? NEGOTIATIONARENA platform and analysis[C]//Proceedings of the 41st International Conference on Machine Learning. 2024: 3935-3951.\n\n[5] Zhou Y, Zanette A, Pan J, et al. ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RL[C]//International Conference on Machine Learning. PMLR, 2024: 62178-62209.\n\n[6] Wang Z, Wang K, Wang Q, et al. Ragen: Understanding self-evolution in llm agents via multi-turn reinforcement learning[J]. arXiv preprint arXiv:2504.20073, 2025.\n\nQUESTIONS:\nDuring the actual rollout process, in each turn, how are $r_t^{(i)}$ and $r_k^{(i)}$ given? My understanding is that only the final step receives a reward, while all other steps have a reward of 0?\n\nLIMITATIONS:\n1. The method is only applicable to environments with limited or simple state spaces where repetitive environmental states are easily identifiable or frequently occur.\n2. The method incurs high training overhead and requires substantial computational resources.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces Group-in-Group Policy Optimization (GiGPO), a novel reinforcement learning technique for training large language model (LLM) agents in long-horizon tasks. GiGPO uses a two-level advantage estimation structure: episode-level advantages for overall trajectory quality (original GRPO) and step-level advantages for fine-grained credit assignment (based on anchor state). This hierarchical approach enables precise optimization without additional rollouts or auxiliary models, maintaining efficiency and scalability. Evaluations on ALFWorld and WebShop show performance gains over existing methods, with over 12% improvement on ALFWorld and over 9% on WebShop.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n* The method proposed in this paper is simple to experiment with and intuitively useful.\n* The paper is well-written with clear exposition and reasonable experiments.\n* This paper reports good performance for ALFWorld and WebShop.\n\nWeaknesses\n* As mentioned in the paper, state matching itself is a challenging problem, and the current algorithm is mainly tested on tasks with a limited number of states. This leads to two issues: first, when reading the paper's introduction, I mistakenly thought this was a universal GRPO improvement, and I believe the writing here is a bit flawed; second, the technical depth discussed in the paper is relatively average.\n\nQUESTIONS:\n1. I think Section 5.5 of the paper is too obvious, and the content of E.3 or E.4 might be more important. How do the authors comment on this?\n2. Is it possible to further enhance the technical section of the paper, perhaps by including at least one task demonstrating GiGPO's effectiveness under non-precise anchor state matching?\n3. Can the improvements proposed by the author be applied to PPO?\n4. Could the experiments be supplemented by applying similar advantage calculations in other reinforcement learning frameworks to strengthen the generality of this idea?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper extends the group-based reinforcement learning method from single-turn tasks to long-horizon multi-turn agent training and proposes a Group-in-Group Policy Optimization method, named GiGPO. The method introduces two-level structure to achieve fine-grained credit assignment for LLM agents. At episode level, GiGPO samples a group of trajectories and compute advantage following vanilla GRPO. At step level, GiGPO splits actions into groups based on anchor states, enabling step-level guidance for long-horizon agent refinement.  Experiements on ALFWorld and WebShop with two base models demonstrates the effectiveness of GiGPO.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n1. The paper extends the group-based RL method to multi-turn tasks and proposes GiGPO, which designs rewards from both solution-level and step-level, enabling both global and local credit assignment. This significantly improves the agent’s performance over long decision sequences.\n2. The step-level credit assignment is constructed retroactively using existing trajectories, thereby avoiding additional rollouts or costly value model estimations, which helps preserving the strengths of group-based RL methods.\n3. Experimental results demonstrate that GiGPO achieves substantial improvements over existing baselines across both benchmarks and model scales. \n4. The paper is clearly written and easy to follow.\n\n\n**Weaknesses**\n1. Although GiGPO is designed to minimize additional computational overhead, the method requires constructing step-level groups to estimate relative advantages. This process may demand a substantial amount of data to ensure group diversity and representativeness.\n2. GiGPO is primarily designed for long-horizon tasks and relies on the presence of anchor states. This design limits its applicability in certain scenarios—for example, in short-horizon or simple tasks, its advantages may not be as significant.\n3. GiGPO builds step-level groups by identifying and grouping repeated environment states across trajectories. This requires that environment states be repeatable and comparable across different rollouts. However, in real-world settings, environment states may suffer from noise, variability, or observation inaccuracies, which could hinder accurate step-level grouping and affect the reliability of relative advantage estimation.\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nThe authors discuss their potential limitations in their paper.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present GiGPO, a new RL algorithm aimed at tackling the tricky problem of credit assignment for LLM agents in long-horizon tasks. They argue that standard group-based methods like GRPO fail to distinguish good from bad actions within a single long trajectory. The key idea here is to supplement the usual episode-level advantage with a more fine-grained, step-level signal. This is achieved through what they call \"anchor state grouping\"—a  process that groups together actions taken from the exact same state across different rollouts. This allows for a localized relative advantage calculation without needing extra models or environment interactions. The authors demonstrate on ALFWorld and WebShop that their method provides significant gains over existing approaches while remaining just as efficient as the GRPO baseline.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths \n1. The core contribution, \"anchor state grouping,\" is a practical way to get fine-grained credit signals without the usual costs. It's an elegant solution that avoids the prohibitive expense of extra rollouts or auxiliary critic models, which is a big win when training these large-scale agents.\n2. A convincing set of experiments. They don't just test on one toy problem; they show consistent, significant gains on tough benchmarks like ALFWorld and WebShop against a strong field of baselines, including GRPO itself. The numbers speak for themselves, with reported boosts of over 12% on ALFWorld and 9% on WebShop over the GRPO baseline, which is quite impressive. I also appreciate the deeper analysis provided; the ablation studies effectively demonstrate that both the macro and micro advantage components are necessary for the method's success, and the plot showing how group sizes evolve during training was a nice touch that gives real insight into the learning dynamics.\n3. the paper is very well-written.\n# Weaknesses\n1. My main concern is the robustness of the anchor state grouping. The current method relies on exact state matching, which seems to work well enough in the clean, simulated environments tested6. But how would this hold up in the wild? In messier, noisier settings with minor but constant state variations (e.g., dynamic content on a webpage), this hashing approach would likely fail often, causing the method to simply fall back to GRPO. The authors do acknowledge this in their conclusion. but the paper would be much more compelling if it included an experiment that actually tests this degradation, perhaps by introducing controlled noise into the state representations. That would give us a much better sense of the method's practical limits.\n2. Secondly, the paper is missing a key ablation study for the hyperparameter ω. This coefficient in Equation (8) balances the macro (episode) and micro (step) advantage signals and seems quite important to the whole setup. Just setting it to 1 \"with no further tuning\" feels like a missed opportunity. It's very likely that the optimal balance is task-dependent, and an analysis of how performance changes with different values of  ω would add a lot of value for anyone looking to apply this method.\n3. Finally, the assumption of identical initial states for all trajectories in a group, while standard for this kind of RL, does limit the method's immediate applicability. It's fine for simulations, but it's a real barrier for many real-world scenarios where you can't just reset the world to the exact same starting point. A more explicit discussion on how this limitation might be relaxed in future work would be a welcome addition to the paper.\n\nQUESTIONS:\n1. Equation (8) introduces the hyperparameter ω to trade off the global, episode-level signal (AE) against the local, step-level signal (AS). In the manuscript, ω is fixed at 1 without any tuning or ablation. Could you elaborate on how sensitive the reported results are to this choice? In particular, might certain values of ω shift the optimal solution of the original problem or induce undesirable agent behaviours?  If there are some theoretical analysis, it will be helpful.\n2. The proposed anchor-state grouping relies on exact state matches, a condition that may be fragile in noisy or high-dimensional environments. Have you analysed how performance varies with increasing state-space noise or complexity? For example, how does the performance gap between GiGPO and GRPO evolve as the probability of encountering an exact state match diminishes? Any empirical or theoretical insights into this degradation would strengthen the contribution.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}