Group-in-Group Policy Optimization for LLM Agent Training
Authors: Lang Feng
Zhenghai Xue
Tingcong Liu
Bo An
Venue: NeurIPS 2025 poster ()
Keywords: Large Language Model
Agent
Group-Based Reinforcement Learning
Primary area: deep_learning

Abstract: Recent advances in group-based reinforcement learning (RL) have driven frontier large language models (LLMs) in single-turn tasks like mathematical reasoning. However, their scalability to multi-turn LLM agent training remains limited. Unlike static tasks, agent-environment interactions unfold over many steps and often yield sparse or delayed rewards, making credit assignment across individual steps significantly more challenging. In this work, we propose Group-in-Group Policy Optimization (GiGPO), a novel RL algorithm that achieves fine-grained credit assignment for LLM agents while preserving the appealing properties of group-based RL: critic-free, low memory, and stable convergence. GiGPO introduces a two-level structure for estimating relative advantage: (i) At the episode-level, GiGPO computes macro relative advantages based on groups of complete trajectories; (ii) At the step-level, GiGPO introduces an anchor state grouping mechanism that retroactively constructs step-level groups by identifying repeated environment states across trajectories. Actions stemming from the same state are grouped together, enabling micro relative advantage estimation. This hierarchical structure effectively captures both global trajectory quality and local step effectiveness without relying on auxiliary models or additional rollouts. We evaluate GiGPO on challenging agent benchmarks, including ALFWorld and WebShop, as well as tool-integrated reasoning on search-augmented QA tasks, using Qwen2.5-1.5B/3B/7B-Instruct. Crucially, GiGPO delivers fine-grained per-step credit signals, achieves performance gains of > 12\% on ALFWorld and > 9\% on WebShop over GRPO, and obtains superior performance on QA tasks (42.1\% on 3B and 47.2\% on 7B): all while maintaining the same GPU memory overhead, identical LLM rollout, and incurring little to no additional time cost.

Group-in-Group Policy Optimization for LLM Agent Training
Lang Feng
Zhenghai Xue!
Tingeong Liu'
Bo An
Nanyang Technological University, Singapore
Skywork AI, Singapore
(lang005, zhenghai001, tingoong001 1θe .nt. edu.se boanêntu. leannntu.ddu...
Abstract
Recent advances in group-based reinforcement learning (RL) have driven frontier large language models (LLMs) in single-turn tasks like mathematical reasoning. However, their scalability multi-turn LLM agent training remains limited. Un- like static tasks, agent-environment interactions unfold over many steps and often yield sparse delayed rewards, making credit assignment across individual steps significantly more challenging. In this work, we propose Group-in in-Group Policy Optimization (GiGPO), novel RL algorithm that achieves fine-grained credit signment for LLM agents while preserving the appealing properties of group-based RL: critic-free, low memory, and stable convergence. GiGPO introduces level structure for estimating relative advantage: (i) At the episode-level, GiGPO computes macro relative advantages based on groups of complete trajectories; (ii) At the step-level, GiGPO introduces an anchor state grouping mechanism that retroactively constructs step-level groups by identifying repeated environment states across trajectories. Actions stemming from the same state are grouped together, enabling miero relative advantage estimation. This hierarchical structure effectively captures both global trajectory quality and local step effectiveness without relying on auxiliary models or additional rollouts. We evaluate GiGPO on challenging agent benchmarks, including ALFWorld and WebShop, as well as tool-integrated reasoning on nearch-augmented QA tasks, using Qwen2.5-1 1.B2/....BB////BB Instrucc.. Crucially, GiGPO delivers fine-grained per-step credit signals, achieves perfor- mance gains of > 12% on ALFWorld and 9% on WebShop over GRPO, and obtains superior performance on QA tasks (42.1 1% 3B and 47.2% on 7B): all while maintaining the same GPU memory overhead, identical LLM rollout, and incurring little to no additional time cost.
Introduction
Large Language Models (LLMs) [1, 2, 3, 4] have leapt from static question-ansee answer systems to versatile agents that perceive, reason, and open-ended environments. For instance, they now power embodied assistants that navigate simulated homes [5, 6]. mobile and web navigators that plan multi-step browsing sessions [7, 9, 10], and autonomous explorers in interactive games [11. 12]. In these settings, L.M agents need to perceive, reason, and act in multi-turn loops, which requires not only language understanding but also long-horizon planning and decision-making
Reinforcement learning (RL) [13] has become a crucial recipe for post-training LLMs, leading to frontier models like OpenAl ol [14] and DeepSeek RI [15]. In particular, group-based RL algorithms such as RLOO [16, 17] and GRPO [18] have proven especially effective in large-scale training. These
Corresponding author Code: https://ee t/gbthbb.bttt.t/////ee/er..b.. wtt.aau.....llaagffaeaaae
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

methods replace value-function estimation with simple yet powerful relative advantage estimation within groups of rollouts. This group-based advantage computation enjoys favorable properties such as low memory overhead, critic-free optimization, and scalability to large models. However, their successes have so far been largely limited to single-turn tasks such as math problem solving [19, 20] and code generation [21], where reward arrives immediately and credit assignment is straightforward.
In contrast, LLM agents operating in external environments face fundamentally different learning landscapes. Their behavior unfolds over long episodes with tens of decision steps and tens of thousands of tokens (e.g.. an ALFWorld [5] episode may include up to 50 steps and over 20k tokens). Rewards are typically sparse (sometimes arriving only at the end of an episode), and the impact of any individual action may only manifest much later in the trajectory. This substantially complicates the credit assignment for individual steps and increases the challenge of policy optimization. Naively applying existing group- RI. algorithms in such settings collapses step-level distinctions, undermining their effectiveness. Hence, these limitations raise core question:
Can we preserve the critic-free, low-memory, and stable convergence properties of group-based RL while introducing fine traiggr credit assignment for multi-turn LLM agent training?
To address this, we introduce Group-in- Group Policy Optimization (GiGPO), new group-based RL algorithm that nests two-dimensional notions of credit assignment, better-suited for the multi-turn optimization of LLM agents. (i) At the episode level, GiGPO samples group of complete trajectories under identical task and initial-state conditions, and computes macro relative advantages based on total returns like vanilla GRPO [18]. This captures the overall effectiveness of each trajectory and reflects the completeness of task execution. (ii) At the step level, GiGPO introduces a novel anchor state grouping mechanism for fine-grained relative ad advantages estimation. Specifically, it retroactively identifies the repeated environment states, termed as anchor states, across trajectories and uses them as anchors to construct step-level groups that allow for localized credit assignment.
The key insight behind GiGPO is that, under identical tasks and initial environment conditions, many trajectories within the group encounter the same states multiple times due to ineffective actions or loops, such as revisiting the same webpage, room, or game scene. These shared states provide a natural basis for step-level group construction and computing more granular advantage estimates. GiGPO uses these step-level groups to assign localized credit to actions based on their relative performance a common state, enabling more precise optimization while avoiding the cost explosion of per-step extra rollouts. As such, GiGPO remains fully critic-free and requires no auxiliary value models while dramatically introducing finer credit signals for training LLM agents.
We first evaluate GiGPO on long-horizon agent benchmarks: ALFWorld [5], which tests embodied task planning in simulated household environments, and WebShop [22], which simulates complex, goal-driven web interactions. In addition, we study multi- tool-integrated reasoning on search- augmented QA tasks. Our experiments with Qwen2.5-1. 1.../../..B///BBBB Instrruct [3] show that GiGPO consistently outperforms prompt-based based agents, actor-critic baselines, and prior group-based RL methods. In particular, GiGPO injects fine-grained, step-level credit signals that sharpen policy learning of agents over horizons and achieves performance gains of > 12% on ALFWorld and > 9% on WebShop over GRPO, along with remarkable performance search-based QA tasks (42. 1% on 3B and 47. 47.2% on 7B). These gains come without compromising the core strengths of group-based RL (only < 0.002% time cost), making GiGPO versatile and high-utility algorithm for LLM agents.
2 Related Work
LLMs as decision-making agents. The use of large language models (LLMs) as autonomous agents has expanded rapidly across domains such program generation [23]. smart device operation [24, 25,
26, 27], interactive gameplay [11], and robot behavior control [28]. Early works typically relied on leveraging pre-trained, frozen models through carefully designed prompting methods (like ReAct [29] and Reflexion [30]), enhanced memory and retrieval systems [12, 31], and integration with external tools [32, 33, 34). More recent research has shifted toward adapting model parameters with supervised fine-tuning (SFT) [24] or RL [13], enabling agents to learn directly from environment interaction rather than static prompts handerafted workflows, which we introduce below.
Reinforcement learning for LLM agents. RL has played a pivotal role in enabling LLM agents to operate in dynamic, open-ended environments. Early work applied classical RL algorithms such as
2

---

DQN [35] to train LLM agents in text-based games [36] and later research [37, 38, 39, 40, 41] started to employ value-based methods, such PPO [42] and AWR [43], in more diverse and interactive agent scenarios including Android device control [44], embodied ALFWorld [5], and card games [45]. More recent approaches have extended RI training to complex web-based and application-centerel tasks. For instance, ArCHer [46] and AgentQ [47] target the WehShop benchmark [22], but require intricate designs and computation overhead such as additional value networks or Monte Carlo Tree Search (MCTS) [48]. CoSo [10] introduces an entropy-based RL method that enhances the performance of agents, Going further, LOOP [49] introduces a hybrid method combining REINFORCE leave-one- out (RLOO) [16, 17] with PtO-style updates, achieving state-of-the of-the-art art results in Applorld [50]. RAGEN [51] introduces a trajectory-level GRPO that concatenates all states, intermediate reasoning, and actions into unified episode-level response. However, it faces scalability challenges in long- horizon tasks (e.g., in ALFWorld, which involves up to 50 steps).
Reinforcement learning for large language models. An early and influential application of RL in LLMs is the Reinforcement Learning from Human Feedback (RLHF) [52, 53, 54, 55]. which focuses on aligning LLMs to human preferences. Most recent works have explored using RL to enhance the capabilities of reasoning and logical deduction in LLMs [56, 15]. In particular, group-based RL algorithms have emerged as promising alternatives to traditional methods like PPO [42]. These methods, such as RLOO [16, 17]. GRPO [18]. Dr. GRPO [19]. DAPO [20], and CPPO [57], avoid introducing extra value functions by leveraging group of samples from the same query and estimate the advantages accordingly. This enables the large-scale RL training and has shown strong results in tasks such mathematical reasoning [15], search [58, 59], and tool use [60, 61]. Our work is closely related to this line of research, with a focus on training I.M agents. We aim to retain the benefits of group-based RL, such critic-free learning and efficiency, while introducing finer-grained credit assignment. Moreover, the hierarchical core of GiGPO is orthogonal to existing group- based RL approaches, making it fully compatible and capable of incorporating them to enhance performance.
3 Preliminaries
Problem setup. We consider a general setting in which an LLM agent interacts with an environment to complete multi-step tasks based on a task description x E p(X). At each discrete time step t = 1,2,...,77 T, the agent observes a state 8, E S and generates a textual action a, E V", where V denotes the token vocabulary and n is the maximum generation length. The environment then returns scalar reward Ti E IR and the next state S1+1- A full episode consists of a trajectory = {( 11. r1), (52, a2, r2),... (ST, ar, rT)]. The agent's 's behavior is governed by an LLM policy Te [ st, z). parameterized by 6, which defines a distribution over outputs conditioned on the current state s; and the task prompt æ. In many realistic scenarios, the environment may provide sparse or delayed rewards (e.g., success and failure signals at the end of an episode) or weak feedback signals for intermediate steps. As the agent generates T consecutive textual actions (a1.... ar), each potentially spanning thousands of tokens, it becomes particularly challenging to assign credit to individual tokens over the course of an episode.
Group-based RL. Recent RL works converge on simple recipe for training LLMs: for a given task description x, the LLM samples a group of N candidate trajectories {Ti, ....... TN }. each corresponding to one full episode rollout under Tew.. Each trajectory receives a scalar reward R(T,) reflecting the overall quality or success of the generated outcome. Instead of estimating advantages using separate value functions like PPO [42], group-based RL methods compute advantages purely based on the statistics internal to the sampled group:
A(T,) GroupComputationn ((R(:)))
(1)
For example, in GRPO [18), the advantage of each trajectory estimated by normalizing its reward with respect to the group's mean and standard deviation. This design is highly memory-efficied and can scale effectively to the large batch sizes and model sizes typical in modern L.LM training, making it a practical and scalable choice for large-scale RI. training.
4 Training LLM Agents with GiGPO
While group-based RL. algorithms (18, 15) have proven highly effective for training LLMS in single- turn tasks, their extension to multi-step agent settings faces critical challenges in credit assignment.
3

---

Step
Step 2
Step 3
Step 4
Step
Step 2
Step 3
Step 4
Step
Step 2
Step
Stop d
Group
Group Comp.
Group
Comp
Comp
Trajectory
Trajectory
Trajectory
Grroup:
Group Comp.
(04, di p0r))
Figure I: Comparison of multi-turn LLM agent training. Left: Vanilla GRPO rolls out group of full trajectories and computes episode-level advantages. Middle: Constructing step-level groups via additional per-state rollouts (e.g., a d", a²" ..... enables fine-graines feedback but incurs prohibitive computational cost. Right: GiGPO efficiently achieves fine-grained credit assignment by aggregating distinct actions (a', a", a". a taken from the same environment state á across the trajectories.
Figure 1 illustrates this gap. Vanilla GRPO (left) treats each trajectory as whole and computes single relative advantage for the entire episode, which fails to provide actionable feedback for individual steps. natural remedy to roll out multiple single-step actions for each state 8; via Thu shown in Figure 1 (middle). However, this approach quickly becomes impractical due to the substantial overhead of extra LLM forward passes and the difficulty of evaluating rewards for hypothetical actions never actually executed.
To overcome these challenges, we propose our Croup-in-Group Policy Optimization (GiGPO) in this section. Similar to prior works [49, 51], GiGPO begins by sampling groups of trajectories under identical tasks and initial environment states. then introduces a two-level grouping structure: pre- serving episode-level grouping for holistic performance comparison, while dynamically constructing an additional set of step-level groups by retroactively aggregating actions encountering the same environment states. This "grouppin-groud" innggrpppp" construction yields two complementary advantages: (I) Episode relative advantages capture the holistic effectiveness of each trajectory, providing a stable, global training signal. (2) Step relative advantages zoom in which actions outperform their peers within the same state, endowing the gradient with fine- geeigraindd credit.
Figure 2 presents an overview of the GiGPO training pipeline. In the remainder of this section, we will detail the computation of episode-level relative advantages, elaborate on the anchor state grouping mechanism, describe the derivation of step-level relative advantages, and finally present the overall GiGPO objective.
4.1 Episode Relative Advantages
We first introduce the episode-level relative advantages, which represent the coarse grariee component of GiGPO and mirror the naive application of GRPO at the trajectory level. We roll out the agent's policy Tew in the environment to collect N complete trajectories under fixed task x and identical initial states. Formally, this process yields group of trajectories [T,) where each trajectory is
denoted as Ti {(s aj (".),,.., (s%)
} 8 (N) For each trajectory, we utilize the total return R(T,) Σ,rr rt
ax (4) (P)) and the initial states satisfy s s
holistic measure
of how effectively the agent completes the task. In settings where only binary reward is given at the end of the episode, this simplifies to R(T,)) 1 for success and R(T,) = 0 for failure. Then, organize the trajectories and their corresponding returns into an episode-level group:
== - R(T))) (T2, R(T:)). (TN. R(TN))
(2)
To evaluate the global relative quality of each trajectory within the group, we compute an episode relative advantage A (T;)) for each t; by normalizing the total return with the group's 's mean and normalization factor:
A R(TT)) - mean (IR(T)))
Frorm
In GRPO [18], the default normalization factor is defined as the standard deviation, i.e., From std. However, this may introduce a difficulty bias [19], where trajectories from low-variance groups (c.g., very casy or hard tasks) receive disproportionately large gradients. In the context of the LLM

---

Apent-Savvnm ment Interned Ion
Rollout
Episode Level
(1
1
F2
R(rs>
Mouading
Group
Inkling
Computaiion
(+22
Group
[M
R(tw>
4 (t,)
State Roward
Action
Anchor State Grouping
Step-kasel
Group Computation
C (1((gc)
Group Computation
(5g)
Group Computation
(s())
(4L
A
(41
00
...
A (4
(+
L. Agent
Figure 2: Overview of GiGPO. The agent interacts with a group of environments initialized with identical states to generate of trajectories (Ti) States with the same color represent the same environment state. GiGPO performs two-dimensional group computations (episode-level A and step-level A to produce hierarchical relative advantages that guide fine-grained policy optimization.
agent, where tasks often involve very long horizons, this effect tends to emerge frequently, potentially affecting the stability of updates. As an alternative, we also consider fixed normalization factor Fnomm which yields an unbiased Leave- Saaee-OOe-uuu estimator [16] (see Appendix C for details). This simple adjustment helps stabilize training in some challenging agent scenarios.
Overall, the episode relative advantage captures whether the agent successfully completes the assign- ment across the entire decision horizon T. Similar to the vanilla GRPO for multi-step optimization shown in Figure I (left), primarily focuses on macro credit assignment, encouraging the policy to develop coherent, trajectory-wide behaviors that maximize overall task performance.
4.2 Step Relative Advantages
While the episode relative advantage offers macro, trajectory-wide signal, it cannot distinguish between the contributions of individual actions within the trajectory. To obtain this fine-grained feedback, we need to form step-level groups: for the same state, we gather the different actions and compare their outcomes, thereby learning which choices are relatively better or worse. naive way to do so would be to roll out fresh actions from every state (Figure 1, middle), but that is prohibitively expensive. Instead, we introduce anchor state grouping below, avoiding extra LLM overhead.
Anchor state grouping. As all trajectories {T..... TN) } arise from the same task x and identical initial conditions, many enviromment states naturally recur across episodes and even across time steps within single trajectory. We leverage this redundancy by identifying and grouping identical states across trajectories, thereby dynamically constructing step-level groups. Specifically, let U (81, . &t) denote the set of all distinct environment states appearing in the trajectory group (TI...., ). We treat each such unique state E L as an implicit anchor and use to gather all matching occurrences of that state, and therefore call "anchor state". Based on this, we can construct [ step-level groups (one for each unique anchor state 8), which defined as follows:
(4) Unlike per-state rollout, this procedure incurs no extra rollouts: it is entirely offline and requires only lightweight key-based grouping using hashmaps. Each group G (8) contains multiple instances of the same environment state paired with potentially different actions. Hence, this structure effectively builds the step- group. forming the basis for subsequent step-level advantage estimation.
F {(a !2)
s. 3, 13 S N. ST.T).
Relative advantage computation. Although each tuple (a, (((()) contains an immediate reward r
(t) it may be sparse, especially in long-horizon tasks. To better capture long-term impact, we associate a discounted return with each step. Let > E (0, 1] be the standard RL discount factor. For
each element in 8 (8), we compute its discounted return R((0 by
R =
k
(5)
k==
5

---

This quantity captures the future impact of action a on subsequent rewards, rather than relying solely on the immediate reward ri Accordingly, the step-level group for each & E U becomes:
GO { (a. R St ( =, 1 i 1 sttST)
(6)
Once these step-level groups formed, we compute the step relative advantage for each l and each action a; in ($)):
R mean ((R()) | (aa , ) E G3
^ Aaa (8)
Fnom (( (at ()) R( G3
A provides micro credit assignment and fine-grained feedback on the relative quality of individual actions taken from the same state. In contrast to the coarse, trajectory-wide signal of A it offers step-level guidance that is essential for refining decisions in long- long-horizon agent tasks.
How does step-level group work? We present an intuitive illustration in Figure to show the utility of the step relative advantages. We consider two ex- ample trajectories from the set {T: In TI, the agent first selects the 2nd Item (incorrect), then re- turns to the previous page and selects the Ist Item (correct), successfully completing the task. Due to temporal discounting (Equation (5)), the earlier ac- tion (2nd Item) receives a lower discounted return than the later correct one (Ist Item). In 72. the agent clicks the Next Page, ultimately failing to find the target and receiving no reward. By aggregating these actions into the same step-level group based on their shared anchor state, GiGPO computes their relative advantages and yields a clear preference ordering: A$ (Ist Item) (2nd Item) A Page). This ranking successfully captures fine-grained dis- tinctions in long-term utility that are missed by prior group-based RL methods [17, 18, 20]. While this example illustrates GiGPO's effectiveness in sparse- reward environments, its advantages extend naturally to dense-reward scenarios, where per-step rewards can be fully leveraged to assess the relative quality of individual actions within shared states.
Next
Pagg
Step
F2
Figure 3: Illustration of step-level grouping in WebShop. Both 7i and T2 encounter the same environment state multiple times: a search results page (highlighted by the red border). Top: Ti eventually succeeds. Bottom: T2 leads failure.
4.3 Group-in-Group Policy Optimization
We finally combine the two levels of advantage signals into a single group-in-group advantage to assign credit both global (episode) and local (step) scales:
A(a A (T:))) + (a
(8)
where w E Rzo is weighting coefficient that balances episode relative advantage and step relative advantage. A (T())) captures how good an episode is compared to others in the group, while A (a refines step-level performance within shared environment state conditions. Jointly, they provide hier- archical supervision for the policy optimization of LLM agents. Then the clipped policy optimization objective of GiGPO is:
N ΣΣ min (oa(a)))(aa) ) cip((p(aa))) (,) {())((
(o) = xxp(x)
NT
t=1
[Te( x)) (- m)).
where pe(a) (0) = mx(a (() is the importance sampling ratio, 3 controls the strength of the KL
no (a
penalty encouraging proximity to reference policy Ned.- We present the pseudo code in Appendix D.

---

5 Experiment
In this section, we present empirical evaluations of GiGPO across a variety of agentic tasks. Specifi- cally, we aim to demonstrate: (1) the strong ability of GiGPO in training LLM agents; (2) the ablation study of GiGPO; (3) the dynamic trend of step-level group G (8) over the course of training; (4) the computational budget of GiGPO.
5.1 Experiment Setup
Benchmarks. We first train the LLM agents on two challenging benchmarks: ALFWorld [5] and WebShop [22]. ALFWorld is an embodied environment designed to assess the ability of LLM agents to perform multi-step decision-making. In each episode, the agent receives a text goal and must accomplish it through multi-turn interaction with the environment. It includes 3,827 task instances across six categories of common household activities: Pick & Place (Pick), Examine in Light (Look), Clean & Place (Clean), Heat & Place (Heat), Cool Place (Cool), and Pick Two & Place (Pick2). WebShop is a complex, web-based interactive environment designed to test the LLM agents in realistic online shopping scenarios. To complete the task, the agent must interact with simulated HTML-- based shopping website to search for, navigate to, and ultimately purchase a suitable item. It contains over 1.1 million products and 12k user instructions, providing a rich and diverse action space. In addition, we also evaluate the multi-turn tool calling performance of GiGPO on search-augmented QA tasks, including single-hop QA datasets (NQ [62], TriviaQA [63], and PopQA [64]) and multi-hop QA datasets (HotpotQA [65], 2Wiki [66], MuSiQue [67], and Bamboogle [68]).
Baselines. For ALFWorld and WebShop, we compare our approach with a range of competitive baselines: (1) Closed-source LLMs: GPT-4o [1] and Gemini-2.5-PP [2], which represent otate-of- the-art capabilities in general-purpost reasoning and language understanding. (2) Prompting agents: ReAct [29] and Reflexion [30], which rely on in-context prompting to guide mulli--tep behavior without parameter updates. (3) RL training methods: PPO [42], widely used actor-critic algorithm that requires an additional value model, and group-based critic-free methods RLOO [16, 17] and GRPO [18), which perform advantage estimation over trajectory groups For search-augmented QA, we compare GiGPO with R1-Instruct, Search-R1 [58], ZeroSearch [59], and StepSearch [69].
Training details. We use Qwen2.551..5B33B7BBBBme
[3] our base models. The weighting coefficient is set to with no further tuning. For ALFWorld and WebShop, all RL training methods (including ours and the baselines) use exactly the same hyperparameter configurations. The rollout group size N for group-based RL methods is set to 8. For search-augmented QA, we follow the same settings in Search-R [58]. We use E5 [70] as the retriever. The rollout group size N is set to 5 and the max turn is set to 4. Moreover, we incorporate similarity-based GiGPO, where anchor state grouping performed by grouping two states if their similarity (longest matching subsequence) exceeds the threshold of 0.9. Full training settings and hyperparameter details are provided in Appendix E.I.
5.2 Performance on ALFWorld and WebShop
Table demonstrates the strong performance of GiGPO across both ALFWorld and WebShop. As shown, closed-source LLMs offer only moderate performance: Gemini-2.5 reaches 60.3% success on ALFWorld and 35.9% on WebShop, while GPT-4o lags further behind. Open-source prompt-only agents (e.g., ReAct and Refiexion) show marginal improvements over vanilla prompting but still underperform, underscoring the difficulty of long- control without post-training. RL training brings substantial gains: PPO improves average ALFWorld success to 54.4% on the 1.5B model and 80.4% on the 7B model, with WebShop scores also increasing significantly. However, this comes at the expense of increased complexity: requiring a separate critic network, hyperparameter tuning, and longer training durations [71, 49]. GRPO and RLOO also yield strong performance while being more computationally efficient, demonstrating the effectiveness of group- RL in large-scale scale LLM training. Nevertheless, their lack of fine-grained per-step feedback limits their ability to provide precise credit assignment across long horizons. In contrast, GiGPO overcomes this limitation with a two-level advantage estimation, enabling both GiGPOw sad and GiGPO sud to consistently surpass GRPO and RLOO. In particular, GiGPO std surpasses GRPO by 13.3% on ALFWorld and 10.6% on WebShop at 1.5B, and by 12.6% and 9.1% respectively, at 7B. These results highlight GiGPO's superior ability to train LLM agents more effectively and more efficiently. We also find that GiGPO enables agents to exhibit emergent reasoning behavior (see Appendix F).
7

---

Table 1: Performance on ALFWorld and WebShop. Results are averaged over 3 random seeds. For ALFWorld, we report the average success rate (%) for each subtask as well as the overall result. For WebShop, we report both the average score and the average success rate (%). GiGPOw/ sid denotes using momm std, while GiGPO wlo sid uses F Formm
ALrld Heat
WebShop Score
Type
Method
Pick
Clean
Cool
Pick2
All
Source Model
Prompting
GPT-4o
75.3
60.8
31.2
56.7
21.6
49.8
48.0
31.8
23.7
Prompting
Gemini- 2.5-Pro
60.3
Qwen2.5- S-15B: Instratctct
Prompting
Qwen2.5
5.9
5.5
3.3
9.7
4.2
0.0
4.1
23.1
5.2
Prompting
ReAct
17.4
15.7
6.2
7.7
2.0
12.8
40.1
Prompting
Reflexion
35.3
22.2
21.7
13.6
19.4
21.8
55.8
21.9
RI. Training
PPO (with critic)
64,8ga5
40.5a69
57.1
60.62s55
46.4240
47.4219
54.4.3.1
73.8g20
51 51,5,29
RL Truining
RLOO
88.3u30
52.8
71.0,59
62.8
66.4,55
56.9147
69.7.25
73.9
52.1
RI. Training
GRPO
85.3215
53.723.
84.5
78.2
59.7,5.
53.5256
72.8
75.8g15
56.843.
RL Training
GIGPO
94.4a59
67.544
94.8,13
94.4.7%
79.8.47
76.425
86.7.17
83.1416
65.0233
Training
GiGPOwls
96.0a14
76.5
81.8
91.326.
71.7
79.5
83.5
67.4g4.
@wen2.5-785 Si--trurtttttt
Prompting
Qwen2.5
33.4
21.6
19.3
6.9
2.8
3.2
14.8
26.4
7.8
Prompting
ReAct
48.5
34.3
13.2
17.6
31.2
46.2
19.5
Prompting
Reflexion
62,0
41.6
44.9
30.9
23.8
42.7
58.1
28.8
RL Training
PPO (with critic)
92.3s4p
64.0,a
92.5.14
89.5.10
80.3,20
68.8
80.4.a7
81.4.1
68.7
RL Truining
RLOO
87.6s4.
78.2,8.3
87.3
81.3
71.9552
48.9284
75.5
80.3
65.7.40
RI. Training
GRPO
90.825.1
66.1
89.3254
74.7
72.5
64.7g73
77.6.52
79.3228
66.1
RL Truining
GIGPO
97.7.1.6
82.7a79
98.8.16
83.7272
89.3182
79.2268
90.8
84,4
72,8
Training
GiGPO
91.825.4
88.6
95.9
90.2,26
86.5
85.2g75
90.2g23
86.2236
75.2
Table 2: Performance on search-augmented QA tasks. GiGPO is trained on NQ and HotpotQA with Frorm std. f and indicate in-domain and out-of-domain datasets, respectively.
Single-Hop QA TriviaQA* PopQA*
Multi-Hop QA
Type
Method
NQ
HotpotQA
2Wiki MuSiQue*
Bamboogle"
Avg.
Qwe32.5-3BB 5-3B-Insiree
RL Training
R1-Instruct
27.0
53.7
19.9
23.7
29.2
7.2
29.3
27.1
RI. Training
Search-RI
34.1
54.5
37.8
32.4
31.9
10.3
26.4
32.5
RL Training
ZeroSearch
41.4
57.4
27.4
30,0
9.8
31.7
RL Training
StepScarch
34.5
32.0
17.4
34.4
RL Training
GiGPO
42.0
59.5
42.4
36.9
37.0
12.6
64.1
42.1
Owen2.5-7B-Insstee
RL Training
R1-Instruct
21.0
44.9
17.1
20.8
27.5
6.0
19.2
22.4
RI Training
Search-RI
39.3
61.0
39.7
37.0
40.1
14.6
36.8
38.5
RL Training
ZeroSearch
43.6
61.8
51.5
34.6
35.2
18.4
27.8
39.
RL Training
StepScarch
38.6
36.6
22.6
40.0
RL Training GiGPO 46.4
64.7
46.1
41.6
43.6
18.9
68.9
47.2
Lastly, we observe that the normalization factor Fnom is task-dependent rather than universally helpful. On relatively diffieult tasks (such Look, Pick2, and WebShop), standard-deviation scaling (Foom std) could exaggerate gradients from overly difficult samples or highly imbalanced groups, harming update stability: fixing noom 1 therefore yields higher success. Yet, Foom offers no clear advantage on other tasks and both variants perform similarly, which aligns with findings in [72]. This suggests that From std can still be beneficial when reward variance is stable.
5.3 Performance on QA tasks
As shown in Table 2, GiGPO achieves strong and consistent gains on multi-turn search- augmented QA tasks, reaching 42.1 % 3B and 47.2% 7B, and significantly outperforming prior strong baselines such Search-R1 and StepSearch. Although nearch-augmenty step-level signals of GiGPO still yield meaningful improvements. Furthermore, we observe that GiGPO is markedly more nool-efficient. Under limit of at most 3 tool calls per query, the 7B model requires only ~0.9 calls on average for single-hop tasks and 1166 calls on average for multi-hop tasks: matching the superior performance of OTC [61], which achieves 1.0 and ~1.7 calls, respectively, on the same tasks. This efficiency likely stems from GiGPO's ability to effectively identify and suppress redundant queries in multi-turn turn decision-making, For instance, in repetitive patterns such queryl
QA is relatively
hort-oori the

---

Trsining Curve
Distribttion Lovel
Distribution of Step Level Grougs
Distrbution of Step Level Groups
Iteraiin
Doration 75
Birutiin 140
Proportion (%1
30
20
6.6
20
20
Itcutiia 75
Iermiton 10
10-
1AG
5+
10
12A
12.1
10
JA3123
Berution
6.0
Grouu Size
Group Size
Group
Figure 5: Dynamics of step-level groups during the training in ALFWorld. Left: Success rate over training iterations. Vertical red markers denote selected checkpoints (Iterations 10, 75, 140). Right: Distribution of step-level group sizes at those checkpoints. For each anchor state å E U, the group size is given by (x))-
+ infol queryl - infol - queryl - infol, GiGPO groups them into the same step-level group and eliminates unnecessary repeated calls during the training, similar to the example we introduced in Figure 3.
5.4 Ablation Study
Next, we conducted an ablation study, comparing GiGPO w/o std (Foor= 1). GiGPOw sud (Faem std). GiGPO wio A (without step relative advantages), and GiGPO we A (without episode relative ad- vantages) to evaluate the impact each component on performance. We use
WebShop
Pick2
80
Pick
11.5BBInsstuu as the policy of the agent. The results are presented Figure 4.
As illustrated, eliminating either component of the two-level advantage significantly degrades performance. Removing the episode relative advantages (GiGPO w/o ==) leads to substantial drop across all tasks, as the policy no longer receives a stable, trajectory-wide signal en- courage long-range coherence. Similarly, discarding the step relative advantages (GiGPO w/o (5) results in pronounced declines, particularly on more complex and demanding tasks such as Cool, Pick2, and Web- Shop, which may require nuanced training feedback at each decision step. In such cases, precise per-step credit assignment is essential for effective learning and policy refinement. Moreover, we can see that the relative performance gap between GiGPO w/ std and GiGPOwo std comparatively minor compared to that observed structural ablations. This suggests that the combination of episode- and step-level signals is the primary driver of performance gains, and that each component is crucial for training LLM agents effectively.
Cool
Look
Heat
Clean
GiGPO
GiGPO
GiG
GiGPO
Figure 4: Ablation results. The y-axis shows success rate (%).
5.5 Dynamics of Step-Level Group
In this part, we examine how the distribution of step-level groups evolves throughout training to better understand the utility of GiGPO. We use 1...2..5-1..B-Intrru as the base model. We train the LLM agent in ALFWorld and track changes in step-level group sizes throughout training.
As shown in Figure 5, we observe that step-level groups of size (i.e., those with (G) 1) only account for < 35% throughout training. This indicates that the majority of states (over 65%) recur across trajectories and therefore contribute to anchor state grouping. Moreover, at iteration 10, large group sizes S (8) 2 10 account for over 20%, reflecting behavioral redundancy in the early stages of training. This is consistent with the fact that immature policies often produce invalid actions or fall into repetitive loops. As training progresses, the group size distribution changes markedly. By iteration 75, we observe substantial reduction in extreme group sizes: 10- < ( < 50 drops from 16.2% to 12. 1% and (G$( 2 50 drops from 5.6% to 3.1%. This shift suggests that the agent is learning to avoid previously common dead ends and invalid actions, and is beginning to exhibit more diverse and purposeful decision-making
At iteration 140, the distribution becomes tightly concentrated around group sizes of 6 to 8. Given that N is set to 8, this convergence implies that

---

all 8 trajectories within an episode group are now behaving consistently: LLM agent has learned coherent and robust policy for completing the task, aligning with the plateau in success rate 80%).
5.6
Computational Budget
Lastly, we analyze the computational budget of GiGPO. GiGPO shares the same architecture GRPO, includ- ing multi-turn rollouts, computation of old and reference probabilities, and clipped policy updates. Both approaches are critic-free and operate with a single actor LLM, thereby resulting in identical GPU memory usage and L.L.M rollout costs. The primary additions introduced by GiGPO are the step-relative advantage estimation components, as described in Section 4.2. To evaluate their costs, we train an LLM agent in ALFWorld using Qwen2.5-11.B-IIssmue and record a detailed breakdown of per-iteration training time.
200
Time (6)
100-
80.85
24,99
25,96
1.05
0,5
0.05
0.01
0.0
Rollent
Gld Prob
Rob
Update
A
Grouping
Figure 6: Per-iteration training time breakdown of GiGPO, Blue bars indi- cate shared components with GRPO. Orange bars show GiGPO-specific ad- ditions. The y-axis uses a broken scale to accommodate small values.
As shown in Figure 6, the additional components incur little to no additional time cost compared to dominant operations such rollouts, computation of old and reference proba- bilities, and policy updates, whose total time cost reaches 362.83s per iteration. In contrast, anchor state grouping (in- volving hashmap lookups) takes only 0.01s per iteration, and the step-relative advantage computation (involving simple arithmetic) adds just 0.53s. These operations account for < 0.002% of the total per-iteration training time, demonstrating that GiGPO shares the same high computational efficiency GRPO.
6 Conclusions and Limitations
In this work, we proposed GiGPO, a novel group-based RL algorithm to tackle the credit assignment challenge in long-horizon LLM agent training. GiGPO introduces hierarchical advantage estimation that enables fine-grained per-step credit assignment while retaining the efficiency and stability of group-based RL. By retroactively grouping steps that share the same state across trajectories, achieves this without incurring additional LLM rollout or GPU memory overhead. Empirical evaluations across complex agentic environments (ALFWorld and WebShop) and search-augmented QA tasks demonstrate that GiGPO significantly outperforms both prompt-based agents and prior RL methods. A potential limitation of GiGPO is its reliance state matching for anchor group construction. In highly complex environments, identical states may be hard to detect due to noise or subtle differences. Despite this, GiGPO still retains strong performance lower bound: in the extreme case where no states are repeated across trajectories (i.e., A = 0), it naturally degrades to GRPO, preserving GRPO's effectiveness and stability in credit assignment. Although this issue is partly mitigated by incorporating similarity- based grouping, exploring more robust state- matching strategies, such as embedding- based representations or domain domain-specific structural equivalence, remains an important direction.
Acknovledgements
This research is supported by the Ministry of Education, Singapore, under its Academic Research Fund Tier 1 (RG18/24).
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. GPT-4 technical report. arXiv preprint 23Xv:2330330874,,, 2023.
[2] Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean- -aatiste Alayrac Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: A family of highly capable multimodal models. arXiv preprint arXiv:2312.11880 2023.
10

---

[3] An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, al. Qwen2. 5 technical report. arXiv preprint arXiv:2412. 15115, 2024.
[4] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. DeepSeek-V. V3 technical report. arXiv preprint arXiv:2412. 19437, 2024.
[5] Mohit Shridhar, Xingdi Yuan, Mlec-Alexandre Cote, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. ALFWorld: Aligning text and embodied environments for interactive learning. In International Conference on Learning Representations, 2021.
[6] Manling Li, Shiyu Zhao, Qineng Wang, Kangrui Wang, Yu Zhou, Sanjana Srivastava, Cem Gokmen, Tony Lee, Erran Li Li, Ruohan Zhang, et al. Embodied agent interface: Benchmarking I.MMs for embodied decision making. Advances Neural Information Processing Systems, 37:100428-1005344 2024.
[7] Hiroki Furuta, Kuang-Huei Lee, Ofrr Nachum, Yutaka Matsuo, Aleksandra Faust, Shixi- ang Shane Gu, and Izzeddin Gur. Multimodal web navigation with instruction-finetune foundation models. In The Twelfth International Conference on Learning Representations. 2024.
[8] Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. GPT-4V (ision) a generalist web agent, if grounded. arXiv preprint MXXi::4411001144 2024.
[9] Boyu Gou, Ruohan Wang. Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, and Yu Su. Navigating the digital world humans do: Universal visual grounding for GUI agents. In The Thirteenth International Conference on Learning Representations, 2025.
[10] Lang Feng. Weihao Tan, Zhiyi Lyu, Longtao Zheng, Haiyang Xu, Ming Yan, Fei Huang, and Bo An. Towards efficient online tuning of VLM agents via counterfactual soft reinforcement learning. In International Conference on Machine Learning, 2025.
[11] Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. Trunsuctions on Machine Learning Research, 2024.
[12] Junyang Wang, Haiyang Xu, Haitao Jia, Xi Zhang, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, and Jitao Sang. Mobile-Agent-v2: Mobile device operation assistant with effective navigation via multi-agent collaboration. Advances in Neural Information Processing Systems, 37:2686-2710, 2024.
[13] Richard S Sutton and Andrew G Barto. Reinforcement Learning: An Introduction. MIT press, 2018.
[14] OpenAl. Introducing OpenAl o1, 2024.
[15] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning. arXiv preprint urXiv:2501. /2948, 2025.
[16] Wouter Kool, Herke van Hoof, and Max Welling. Buy 4 reinforce samples, get baseline for free! In ICLR 2019 Workshop, 2019.
[17] Arash Ahmadian, Chris Cremer. Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting reinforce style optimization for learning from human feedback in LLMs. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers), pages 12248-12222 12267, 2024.
[18] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Wu, et DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint ar 40Xiv:2402.033 03300, 2024.
11

---

[19] Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding rl-zero-like training: A critical perspective. arXiv preprint wrXiv:2503.20783 2025.
[20] Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. DAPO: An open-source LLM reinforcement learning system at scale. arXiv preprint arXiv:2503. 14476, 2025.
[21] Yuxiang Wei, Olivier Duchenne, Jade Copet, Quentin Carbonneaux, Lingming Zhang, Daniel Fried, Gabriel Synnaeve, Rishabh Singh, and Sida Wang. SWE-RL: Advancing I1m reasoning via reinforcement learning on open software evolution. arXiv preprint arXiv:2502. 18449, 2025.
[22] Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. WebShop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Processing Systems, 35:20744-207511 2022.
[23] Kechi Zhang, Jia Li, Ge Li, Xianjie Shi, and Zhi Jin. CodeAgent: Enhancing code generation with tool-integrated agent systems for real-world repo-level coding challenges. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13643-13658, 2024.
[24] Zhuosheng Zhang and Aston Zhang. You only look at screens: Multimodal chain-of-action agents. In Findings of the Association for Computational Linguisties ACL 2024, pages 3132- 3 3149, 2024.
[25] Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang. Yuxiao Dong, Ming Ding, et al. CogAgent: A visual language model for GUI agents. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14281-14290, 2024.
[26] Izzeddin Gur, Hiroki Furuta, Austin V. Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, and Aleksandra Faust. A real-world webagent with planning, long context understanding, and program synthesis. In The Twelfth International Conference on Learning Representations, 2024.
[27] Siyuan Hu, Mingyu Ouyang, Difci Gao, and Mike Zheng Shou. The dawn of GUI agent: A preliminary case study with claude 3.5 computer use. arXiv preprint arXiv: 10323, 2024.
[28] Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Stefan Welker, Ayzaan Wahid, et al. RT-2: Sasion-language-actto models transfer web knowledge to robotic control. In Conference on Robot Learning, pages 1165-2183. PMLR, 2023.
[29] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, 2023.
[30] Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
[31] Weihao Tan, Wentao Zhang, Xinrun Xu, Haochong Xia, Gang Ding, Boyu Li, Bohan Zhou, Junpeng Yue, Jiechuan Jiang, Yewen Li, et al. Cradle: Empowering foundation agents towards general computer control. In NewrIPS 2024 Workshop on Open-World World Agents, 2024.
[32] Timo Schick, Jane Dwivedi-YY Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems, 36:68539- 68551, 2023.
[33] Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongehan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. OSWorld: Benchmarking multimodal agents for open-ended tasks in real computer environments. In The Thihty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024.
12

---

[34] Chaoyun Zhang, Liqun Li. Shilin He, Xu Zhang, Bo Qiao, Si Qin, Minghua Ma, Yu Kang, Qingwei Lin, Saravan Rajmohan, al. UFO: A Ul-focused agent for windows OS interaction. arXiv preprini arXiv:2402.07933, 2024.
[35] Volodymyr Mnih. Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, al. Human-level control through deep reinforcement learning. Nature, 518(7540):529-535 2015.
[36] Karthik Narasimhan, Tejas Kulkarni, and Regina Barzilay. Language understanding for text- based games using deep reinforcement learning. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1-11, 2015.
[37] Weihao Tan, Wentao Zhang, Shanqi Liu, Longtao Zheng, Xinrun Wang, and Bo An. True knowl- edge comes from practice: Aligning large language models with embodied environments via reinforcement learning. In The Twelfth International Conference Learning Representations, 2024.
[38] Muning Wen, Ziyu Wan, Jun Wang, Weinan Zhang, and Ying Wen. Reinforeing LLM agents via policy optimization with action decomposition. In The ehirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[39] Simon Zhai, Hao Bai, Zipeng Lin, Jiayi Pan, Peter Tong. Yifci Zhou, Alane Suhr, Saining Xie, Yann LeCun, Yi Ma, et al. Fine-tuning large vision-language models as decision-making agents via reinforcement learning. Advances in Neural Information Processing Systems, 37:1 10935- 1 110971, 2024.
[40] Hao Bai, Yifei Zhou, Jiayi Pan, Mert Cemri, Alane Suhr, Sergey Levine, and Aviral Kumar. DigiRL: Training in-the-wild wild device device-control agents with autonomous reinforcement learning. Advances Neural Information Processing Systems, 37:12461 12495, 2024.
[41] Taiyi Wang, Zhihao Wu, Jianheng Liu, Jianye HAO, Jun Wang, and Kun Shao. DistRL: An asynchronous distributed reinforcement learning framework for on-device control agent. The Thirteenth International Conference Learning Representattons, 2025.
[42] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
[43] Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. wdvintage-weighhed regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv: 191.00011100017.,, 2019.
[44] Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy Lillicrap. Android in the wild: A large-scale dataset for android device control. Advances in Neural Information Processing Systems, 36, 2024.
[45] G Brockman. OpenAl Gym. arXiv preprint arXiv: TXXv:16660015444 2016.
[46] Yifei Zhou, Andrea Zanette, Jiayi Pan, Sergey Levine, and Aviral Kumar. ArCHer: Training language model agents via hierarchical multi-turn rl. International Conference on Machine Learning, pages 62178-62209. PMLR, 2024.
[47] Pranav Putta, Edmund Mills, Naman Garg, Sumeet Motwani, Chelsea Finn, Divyansh Garg, and Rafael Rafailov. Agent Q: Advanced reasoning and learning for autonomous ai agents. arXiv preprint arXiv:2408.00 07140088.79., 2024.
[48] David Silver, Julian Schritwieserr, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. Nature, 550(7676):354-353 2017.
[49] Kevin Chen, Marco Cusumano-Towner Brody Huval, Aleksei Petrenko, Jackson Hamburger, Vladlen Koltun, and Philipp Krähenbühl. Reinforcement learning for long-horizon interactive Ilm agents, arXiv preprint 2rXiv:2502.01600, 2025.
13

---

[50] Harsh Trivedi, Tushar Khot, Mareike Hartmann, Ruskin Manku, Vinty Dong, Edward Li, Shashank Gupta, Ashish Sabharwal. and Niranjan Balasubramamian. AppWorld: controllable world of apps and people for benchmarking interactive coding agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 16022-16076, 2024.
[51] Zihan Wang, Kangrui Wang, Qineng Wang. Pingyue Zhang, Linjie Li, Zhengyuan Yang, Kefan Yu, Minh Nhat Nguyen, Licheng Liu, Eli Gottlieb, et al. RAGEN: Understanding self-evolution in LLM agents via multi-turn reinforcement learning. arXiv preprint arXiv:2504.20073, 2025.
[52] Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford. Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv: 1909.08593, 2019.
[53] Nisan Stienmon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-302) 2020.
[54] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray. et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 15:27730-27744, 2022.
[55] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems, 36, 2024.
[56] Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, al. Kimi k1.5: Scaling reinforcement learning with LLMs. arXiv preprini arXiv:2501. 12599, 2025.
[57] Zhihang Lin, Mingbao Lin, Yuan Xie, and Rongrong Ji. CPPO: Accelerating the training of group relative policy optimization-base reasoning models. arXiv preprint arXiv:2503.22342, 2025.
[58] Bowen Jin, Hansi Zeng, Zhenrui Yue, Dong Wang, Hamed Zamani, and Jiawei Han. Search-R1: Training LLMs to reason and leverage search engines with reinforcement learning. arXiv preprint STti::2503.9551// 2025.
[59] Hao Sun, Zile Qiao, Jiayan Guo, Xuanbo Fan, Yingyan Hou, Yong Jiang, Pengjun Xie, Fei Huang, and Yan Zhang. ZeroSearch: Incentivize the search capability of Ilms without searching. arXiv preprint arXiv:2505.0m 045888 2025.
[60] Cheng Qian, Emre Can Acikgoz, Qi Hc, Hongru Wang, Xiusi Chen, Dilck Hakkani- Tür, Gokhan Tur, and Heng Ji. TooIRL; Reward is all tool learning needs, arXiv preprint 3rX1v:2504.13958 2025.
[61] Hongru Wang, Cheng Qian, Wanjun Zhong, Xiusi Chen, Jiahao Qiu, Shijue Huang, Bowen Jin, Mengdi Wang. Kam-Fai Wong, and Heng Ji. OTC: Optimal tool calls via reinforcement learning. arXiv preprint arXiv:2504.14870 2025.
[62] Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453-466, 2019.
[63] Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.035521 2017.
[64] Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Ha- jishirzi. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. arXiv preprint arXiv:2212210511. 2022.
14

---

[65] Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhut- dinov, and Christopher Manning. HolpotQA: A dataset for diverse, explainable multi-hop question answering. arXiv preprint arxivv 201 8.
[66] Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps. arXiv preprint arXiv:2011.01060 2020.
[67] Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. MuSiQue: Multihop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 10:539-554, 2022.
[68] Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis. Measuring and narrowing the compositionality gap in language models. arXiv preprint arXiv:2210.03350, 2022.
[69] Ziliang Wang, Xuhui Zheng, Kang An, Cijun Ouyang. Jialu Cai, Yuhang Wang, and Yichao Wu. StepSearch: Igniting LLMs search ability via step-wise proximal policy optimization. arXiv preprint 2rXiv:2505.15100 2025.
[70] Liang Wang, Nan Yang. Xiaolong Huang. Binxing Jiao, Linjun Yang. Daxin Jiang, Rangan Majumder, and Furu Wei. Text embeddings by weakly-supervised contrastive pre-training. arXiv preprint 1rXiv:2212.03533, 2022.
[71] Amirhossein Kazemnejad, Milad Aghajohari, Eva Portelance, Alessandro Sordoni, Siva Reddy, Aaron Courville, and Nicolas Le Roux. VinePPO: Unlocking rl potential for Ilm reasoning through refined credit assignment. arXiv preprint arXiv:2410.01679, 2024.
[72] Xiangxiang Chu, Hailang Huang, Xiao Zhang, Fei Wei, and Yong Wang. GPG: A simple and strong reinforcement learning baseline for model reasoning. arXiv preprint arXiv:2504.02546, 2025.
[73] Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang. Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. HybridFlow: A flexible and efficient rlhf framework. arXiv preprint arXiv:2409. 19256, 2024.
[74] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505009388, 2025.
[75] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
[76] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824-24837, 2022.
[77] Max-Philipp B. Schrader. gym- sokoson, 2018.
Gym-Sokoban.
https://github: htt.ub..oommpcchreeeee.
[78] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang, et al. Qwen2.5-V LL technical report. arXiv preprint wxXiv:2502.13922 2025.
15

---

A
Open Source Codebase: verl-agent
As part of the new assets released with this work, we propose verl-agent (https://gi thub. com/ BangfengQ//a verl-agent), a highly scalable RL training framework for long-horizon, multi-turn LLM agent training.
Our verl-agent is built upon the veRL. framework [73] and extends it with several features to enable scalable reinforcement learning for long-horizon LLM agents. Key capabilitiess of our framework include: (1) wise multi-turn interaction paradigm that avoids concatenating full interaction histories (as in Search-RI [58] and RAGEN [51]), ensuring efficient memory control and scalability for very long-horizon optimization; (2) customizable memory module that allows developers to flexibly determine which historical information to include at each step (e.g., key events, summaries, or external knowledge); (3) parallel and group-basedd environments with gym-style interface supporting high-throughpir rollouts; (4) broad model compatibility, including Qwen3 [74], Qwen2.5, and LLaMA3.2, along with LoRA-bar A-based fine-tuning [75] for efficient large-model adaptation; (5) support for multimodal (vision-language) agents such as Qwen2.5-VL: (6) diverse suite of environments, including Search (tool use), ALFWorld, WebShop, Sokoban, and Gym Cards; (7) comprehensive RL algorithm support, encompassing GiGPO. GRPO, PPO, DAPO, RLOO, etc.
RAGEN/Scarch-Bl
lagent
-Multti- Turn Rollout Phase
-MMtii-Trre Roliout Phase
Sep
>x Update Phose
> Update Phose
Figure 7: Open-souree agentic training framework comparison. Left: RAGEN/Search-RR concate- nates the full history at every step, leading to rapidly expanding context. Right: verl- adopts a step-wist wise multi-turn turttttuunnnn rollout with flexible per-step input construction and memory control.
B Broader Impacts
This work introduces an efficient, group-based RL algorithm, GiGPO, that enables fine- geaiggdi credit assignment for long-horizon LLM agents. By combining episode-level and step-level advantages, GiGPO improves training stability and agent performance without increasing computational or memory overhead.
The algorithm holds promise for a wide range of applications, including virtual assistants, web automation, educational tools, and embodied Al systems. Its critic-free and scalable design lowers the barrier to training effective multi-step agents, making advanced agent capabilities more accessible to both academic and industrial practitioners. While GiGPO is methodological contribution with no direct downstream deployment, the improved agent training techniques may indirectly enable more autonomous behavior. As with any advancement in agent capabilities, it is important to consider appropriate safeguards and responsible usage. Overall, GiGPO contributes a valuable tool to the RI. and I.LM communities, with the potential to accelerate progress in building more intelligent, efficient, and adaptable AI agents.
C Unbiasedness
We show that setting From leads to unbiased estimator up to constant scaling factor. Our derivation follows the approach in [19]. For comparison, the standard REINFORCE Leave-One-Out (RLO0) [16, 17] is defined as:
A ALLO0 (T.) -R(T)) N
Σ R(T;)) jzL
(10)
16

---

We can relate A (T.) nFoom = 1) and A A (T:) by introducing scaling factor of
N
N-1²
N
N
N
N
A (T
N - R(T))
R(Ty)
(11)
N
N
N 1
N
1
N R((T))
R(T))
Σ
R(T,)
(12)
N
N
-R(T.)
Σ R(T,) jti
(13)
N-1
=AROOO (T.)
(14)
Thus, setting Fnom = 1 corresponds to a rescaled version of A (T.) and scaling the advantage by constant does not affect the dynamics of policy gradient (it can be absorbed into the learning rate).
D Pseudo Code
Algorithm summarizes the full GiGPO training procedure. Compared to vanilla GRPO, we highlight the additional parts introduced by GiGPO in italics. In particular, building step-level groups G is implemented by treating anchor states as keys and aggregating corresponding data into hash table, which incurs minimal overhead. Furthermore, computing step relative advantages and combining advantages involve only simple arithmetic operations, both of which are lightweight. As such, GiGPO preserves the critic-free, low-memory, and stable convergence properties of group-based RL, while introducing fine-grained credit assignment that is particularly beneficial for training long-horizon LLM agents.
Algorithm 1 Training LLM Agents with GiGPO
1: Require: Initial policy new.. task distribution p(X), discount factor 7, weighting w, clipping parameter KL penalty B. group size N
2: for each training iteration do
3: Update the old policy model: Gold
0
4: // Multi-step rollout phase
5: Sample task x ~ p(X) and initialize N identical environments
6: for = to T do
7:
Sample actions {a," (5) Temu (-1 | s! {1,),))) x))
8: Execute actions, observe rewards (r( ( Jim1 next state (2(2 (
9:
end for
10:
// Grouping phase
11:
Compute episode relative advantages (r.) via Equation (3)
12:
Build step-level groups G
G*( (8) via the anchor states
13:
Compute step relative advantages A (a(aa)
(1) via Equation (7)
14:
I/ Policy update phase
15:
Combine advantages: A(a, (3) = A ) + wA (a (8)
16:
Update policy 0 by maximizing objective JGIGPO
17:
end for
E Experiment Details
E.1 Details of Training
Hyperparameters for ALFWorld. All methods are configured with identical hyperparameters: maximum prompt length is 2048 tokens, and the maximum response length is 512 tokens. Each episode allows up 50 environment steps. The learning rate is set to le-6 for the actor and le-5 for the critic (used only in PPO). We adopt a rule-based reward, assigning a reward of 10 for success and 0 for failure. To handle invalid actions generated by the agent, apply reward penalty of
17

---

-0.1. For all group-based RL methods, we use a group size of 8 and sample 16 different groups per rollout, resulting total of 16 environments for rollouts. The rollout temperature is set to 1.0, while the validation temperature is set to 0.4. The mini-batch size is 256, and the KI--divegeence loss coefficient set to 0.01. For GiGPO, the weighting coefficient is fixed at I without further tuning, and the discount factor is set to 0.95.
128 environments. In contrast, PPO uses 128 separate
Hyperparameters for WebShop. All methods configured with identical hy hyperparameters: the maximum prompt length is 4096 tokens, and the maximum response length is 512 tokens. Each episode limited to 15 environment steps. The learning rate is le-6 for the actor and le-5 for the critic (used only in PPO). We adopt a rule- based reward, assigning a reward of 10 for success and 0 for failure. Invalid actions are penalized with reward of -0.1, As with ALFWorld, all group-based RL methods use group size of 8 and sample 16 groups per rollout, totaling 16 x 8 = 128 environments. PPO, on the other hand, uses 128 distinct environments for rollouts. The rollout temperature set to 1.0, while the validation temperature is set to 0.4. The mini- size is 64, and the KI.-divergence loss coefficient is set to 0.01. For GiGPO, the weighting coefficient set to 1 without additional tuning, and the discount factor is set to 0.95.
Hyperparameters for Search-Augmented QA. The maximum prompt length is 4096 tokens, and the maximum response length is 512 tokens. The max turn is set to 4. The learning rate is le-6 for the actor. We adopt rule-based reward, assigning a reward of I for success and for failure. Invalid actions are penalized with a reward of -0.01. We set the train data size to 256 and use group size of Rollout and validation temperatures are to 1.0 and 0.0, respectively. The mini-batch size 512, and the Kl--divergence loss coefficient is set to 0.001. The weighting coefficient w is set to 1 without additional tuning, and the discount factor y is set 0.95.
Computing Details. For ALFWorld and WebShop, Qwen2.5-1.5D experiments are run on 2xH100 GPUs and Qwen2.5-7B on 4xH100 GPUs, each for 150 iterations. For search-augmented QA, Qwen2.5-3B uses 4xH100 GPUs and Qwen2.5-7B uses 8xH100 GPUs, each for 200 iterations.
E.2 Prompts
The prompts we for LLM agents are presented in Figure 8, Figure 9, and Figure 10. These prompt templates are constructed using Python-style string formatting, where placeholders enclosed in curly braces (()) represent semantic slots. These placeholders, such as (task description), (step_count) count), and (current_ observation), are dynamically populated at runtime via Python's. format () function. To enrich the agent's context, we use historical information and set the history length for ALFWorld and WebShop and the full history for seareh-augmented QA experiments.
The <think </think> block instructs the agent explicitly perform step-by-ster step reasoning, thereby promoting th-hugt [76] style deliberation. The <action> </action> block is used to clearly indicate the final action decision. The search agent outputs reasoning traces within <think> </think>, issues search queries within <search> </search>, provides anwsers within <anwser> </anwser>. Retrieved evidence from the retriever is presented in <information> </information> tags.
Prompt Template for ALFWorld
are an expert agent operating in the ALFRED embodied Environment. Your task to: (task deseription), Prior this step, you have already taken [step_count] step(s). Be- observations and the corresponding actions you
low are the most recent (history_ took: {action_ {action_history_, history). You now step (current step) and your current observation is: [current_observeee obbsevatioo.)) Your admissible actions of the current situation [[admissi- ble actioons]]
take action. You should first reason step-by-ste about the current situation. This reasoning process MUST be enclosed within <think> </think> tags. Once you' ve finished your reasoning, you should choose an admissible action for current step and present within <action> </action> tags.
Now it's your
Figure 8: The prompt template of ALFWorld agents.
18

---

Prompt Template for WebShop
an expert autonomous agent operating in the WehShop e-commerce environment. Your task [task, description]. Prior this step. you have already taken [step_count] step(s). Below the most recent [history_length) observations and the corresponding actions you took: faction [action_history]] history). J. You are at step [currentssepp step) and your current observation is: (ourrent_obervaatio)) Your admissible actions for the current situation are: [[avalable_actiinn actiionttss
Now it's your turn take action for the current step. You should first about the current situation, then think carefully which admissible action best advances the shopping goal. This reasoning process MUST be enclosed within <think> </think> tags. Once you'te ve finished your reasoning. should choose admissible action for current and present it within <action> </action> tags.
step-by-step
Figure 9: The prompt template used for WehShop agents.
Prompt Template for Search
You expert agent tasked answering the given question step-by-step. Your question: (task description). Prior this step, you have already taken step__coml step(s). Below the interaction history where <search>> </scarch> wrapped your past search queries and <information> </information> wrapped the corresponding search results returned by the external search engine. History: [memory_contex!] context)
Now it's your turn respond for the current step. You should first conduct reasoning process. This process MUST be enclosed within <think> </think> tags. After completing your reasoning, choose only one of the following actions (do not perform both):
(1) If you find lack some knowledge. you can call search engine to get more external information using format: <search> your query </search>.
(2) If you have enough knowledge to answer the question confidently, provide your final answer within <answer> </answer> tags, without detailed illustrations. For example, <an- swor:Beijnge/answer..
Figure 10: The prompt template of Search agents.
E.3 Performance Vision-Language Agents
We conduct additional experiments in vision- language model (VLM) settings where agents must reason over both visual and textual inputs. We use
two interactive game environments: Sokoban [77] with a 6x6 size and EZPoints in Gym Cards [39] as shown in Figure 11. Sokoban is a classic grid- based puzzle game where the agent must push boxes onto designated goal positions. Solving Sokoban re- quires spatial reasoning and long-tern planning. In EZPoints, the agent is presented with two playing cards and must construct arithmetic formula step- by-step such that the final result equals a target value of 12. This setting emphasizes symbolic reasoning and multi-step planning. In these tasks, the agent receives RGB image representing the current environment state along with textual instruction, and must take sequential actions to complete the task. All methods are built upon Qwen2.5-VL-3B- Instruct [78].
Formula:
(1) Sokoban
(2) EZPoints
Figure 11: Visual observation VLM agents in Sokoban and EZPoints.
As shown in Table 3, GiGPO significantly outperforms the prompting baseline and GRPO on both tasks. Notably, GiGPO achieves 81.0% success on Sokoban and 100% success EZPoints, suggesting its ability to generalize beyond language-only settings.
E.4 Orthogonality to Single-Tury Group-Based RL
GiGPO remains orthogonal to other advancements in single-turn group-based RL,, allowing it to incorporate complementary techniques without loss of generality. To verify this, we consider
19

---

Table 3: Success rates (%) of VLM agents using Qwen2.5-VL-3B-Imssie
Type
Method
Sokoban [6x6]
EZPoints
Prompting
Qwen2.5-VL GRPO
11.7
3.1
RL Training
67.1
86.9+3.4
RL Training
GiGPO w/ std
76.9
100.0+0.0
RL Training
GiGPO w/o
81.0
100.020.0
DAPO [20], which builds on GRPO with dynamic sampling and clip-higher techniques. We integrate both into GiGPO, yielding a variant denoted as GiGPO GiGPOdynamic- dynamic-
As shown in Table 4, DAPO improves over GRPO, confirming the benefits of its techniques. Impor- tantly, GiGPOdynami further outperforms DAPO, demonstrating that it can effectively benefit from and amplify such improvements, achieving better results (75.0%) on WebShop.
Table 4: Performance on WebShop using -wen2.5-1..B-Insttee
Type
Method
Score
Success Rate (%)
Prompting
Qwen2.5
23.1
5.2
RL Training
GRPO
75.8
56.843.8
RL Training
DAPO
84.6+29
66.1
RL Training
GiGPO dynamic
87.5
75.0g3.5
E.5
Sensitivity Analysis
We further analyze GiGPO's sensitivity to the key hyperparameter w, which balances the episode-level advantage A and the step-level advantage A We conducted experiments the WebShop task, using 1..cn2.5-1..B-Instruue
Table 5: Sensitivity analysis on w for the WebShop task.
0.0
0.2
0.4
0.6
0.8
1.0
1.2
1.4
Score Success Rate (%)
76.2
79.6
82.4
83.5
84.9
83.5
82.6
77.0
56.6
63.1
65.8
67.2
68.3
67.4
66.5
56.3
As shown in Table 5, GiGPO needs an appropriate w to work best. Increasing w initially improves performance due to the added fine- fineggrained step-level reward. However, performance declines beyond the optimum (w = 0.8), suggesting that excessive emphasis on step-level signals can suppress useful trajectory-level guidance. Furthermore, GiGPO is relatively insensitive within the range [0.4, 1.2]. demonstrating reasonable degree of robustness to w.
20

---

Reasoning Behavior
We next present the complete trajectory of multi-turn interaction between agent and environment, and demonstrate reasoning behavior of LLM agents (Qwen2..--7B-Insttuu trained by GiGPO.
F.1
ALFWorld
Environment (Step 1)
Welcome to TextWorld, ALFRED! =-
Your task is to: heat some egg and put in countertop. You are the middle of room. Looking quickly around you, you see a cabinet 9, cabinet a cabinet cabinet 6, cabinet 5, cabinet 4, cabinet 3, cabinet 2, cabinet 1, coffeemachine 1, countertop
countertop I, drawer 13, a drawer 12, drawer drawer 10, a drawer 9, drawer drawer a drawer 6, drawer drawer microwave
1, fridge I, garbagecan drawer drawer 2, stoveburner a stoveburner 4, stoveburner 3, sinkbasin 1, a stoveburner
stovebumer 2, stoveburner 1, and toaster I.
Agent (Step
<think> To heat an and put it on the countertop, need to first find an egg. Given that eggs are commonly stored in the fridge, should check the fridge first to see there is an egg available. </think>
<action>go to fridge 1</action>
Environment (Step 2)
You arrive fridge The fridge is closed.
Agent (Step 2)
<think>
The task is heat egg and put it the countertop.
I currently at the fridge and the fridge closed. This means need to open the fridge to check if there egg inside.
3. If there is egg in the fridge, will take it out, heat it, and then it the countertop.
4. If there is no egg in the fridge, I will need to check other locations such as the cabinets, countertop, and garbagecan, but the fridge is the most likely place to find an egg-
5. After opening the fridge and finding the egg, will need to take it out, heat it using the microwave or stovetop. and then put it on the countertop. </think>
<action>open fridge I</action>>
Environment (Step 3)
You open the fridge The fridge is open. it, you bowl pan plate potato
Agent (Step
<think>
The task is to heat egg and it the countertop.
I have already opened the fridge I and found bowl 1, pan 1, plate and
potato inside. There is egg the fridge.
3. Since the egg is not in the fridge. need to check other locations where an egg might be found, such as the cabinets, countertop. and garbagecan.
4. The countertop is mentioned in the task, so should check the countertop to see there is an egg there.
5. If the egg is not found on the countertop. will check the cabinets and the garbagecan.
</think>
<action>go to countertop 1</action>
21

---

Environment (Step 4)
You arrive countertop On the countertop you apple butterknife dishsponge fork 1, lettuce pun pot 1, soupbottle soupbottic spoon 2, tomato tomato window 3, and window 1.
Agent (Step
<think>
The task is to heat an egg and put it on the countertop.
2. To heat an egg. I need to first find an egg.
3. The most likely places to find an egg would he the fridge or the countertop, as
eggs are often stored in cool places to keep them fresh.
4. have already opened fridge and found contains a bowl, pan, plate, and
potato, but no eggs.
The current observation shows that I am at countertop 1. which contains various food items no eggs.
Since have not yet found the egg. should check the fridge again case missed it, should check the countertop 2, as is another common place to store eggs.
After checking fridge and countertop should consider the sinkbasin 1, as eggs might washed there before being used.
If still cannot find egg. should check cabinets, as eggs are sometimes stored in cabincts keep them organized and out of the way.
</think>
<action>go countertop 2</action>
Environment (Step 5)
You arrive at countertop 2. On the countertop 2, see bread 2, bread 1, cellphone 1, a egg 3, lettuce I, a pot 2, potato 2, spatula 2, and a tomato 3.
a egg I, a fork 2, kettle I, knife 1, ladle 1,
Agent (Step 5)
<thinks
The task is to heat an egg and put it on the countertop.
have already opened fridge and observed its contents, which include a bowl, pan, plate, and potato, no eggs.
current observation an egg from countertop proceed with task.
there are eggs on countertop need to take
taking the egg. will need to heat
heat egg. will need to go to the most common appliance for heating food.
the microwave,
The available actions include taking different eggs from countertop 2, well going the mierowave heat the egg.
</think>
<action>take egg from countertop 2</action>>
Environment (Step 6)
You pick the egg from the countertop 2.
Agent (Step 6)
<thinks
have just picked up the egg from countertop 2.
My next step to heat egg. To do this, need to place the egg in a suitable heating device, such as microwave or a stovetop.
The most convenient and common device heating an egg is microwave. Therefore, I will go the microwave
heat the egg. After heating the egg, need put it the countertop. Since I have already picked up egg, will go the countertop to place heated egg there. </think>
<action>go microwave l</action>>
22

---

Environment (Step 7)
You arrive microwave The microwave closed.
Agent (Step 7)
<think>
have just taken the egg from countertop 2.
My current location microwave and microwave is closed.
To heat the egg. need to open microwave place egg inside then heat
After heating the egg, I need to put the countertop.
</think>
<action>open microwave 1</action>
Environment (Step 8)
You open the microwave 1. The microwave is open. In it, you see nothing.
Agent (Step 8)
<think>
have just opened the microwave and found nothing inside.
The ege is on the countertop 2. and have already gone to the microwave to open
next step to heat the egg Since the egg opened the microwave, can now heat egg using the microwave 1.
countertop
After heating the ege. need to put it on countertop. The action 'heat egg with microwave available, which is the appropriate action to take next.
</think>
<action>heat egg with mierowave I</action>>
Environment (Step 9)
You heat the egg using the microwave
Agent (Step
<think>
have just heated the egg 1 using the microwave
My next step is to put the heated egg on the countertop.
To do this, need to go to the countertop where I can place the egg.
The admissible actions include gg to countertop and 'o to countertop 2', which are the appropriate actions to take to reach the countertop.
Since have not specified which countertop to use, can choose either one. For simplicity, I will choose 'go countertop
</think>
<actionzgo countertop 1</action>
Environment (Step 10)
You arrive countertop On the countertop you apple butterknife dishsponge fork 1, lettuce pan pot 1, soaphottle soaphottle spoon 2, tomato tomato 1, window 3, and window 1.
Agent (Step 10)
<think>
have just heated the egg 1 using the microwave
My current location is countertop 1, where see various kitchen items including a window and a tomato, but no egg. The action 'move egg countertop available, which the appropriate action to take next. This will place the heated egg on the countertop. </think>
<action>-move egg to countertop I</action>
23

---

F22
WebShop
Environment (Step 1)
Instruction: Find me loose fit, slim fit men's 's tuxedo shirts with long sleeve. short sleeve, polyester cotton, elastic waist, regular fit for gym workout with color: b-blue, and size: xx la-large, and price lower than 40.00 dollars. Your current observation is: "Search".
Agent (Step
<think> To find loose fit, slim fit men's tuxedo shirts with long and short sleeves, polyester cotton, clastic waist, regular fit for gym workout, in blue color (b-blue), and size xx-large, with a price lower than $40.00, I need to construct search query that includes all these criteria. The current observation is 'Search', which means the scarch function is available. Therefore, the appropriate action is to use the 'scarch' action to input the desired query. </think>
<action> scarch[color: b-blue, size: xx-large, price: 0-40, product type: tuxedo shirt, fit: regular, sleeve: long and short] </action>
Environment (Step 2)
'Back to Search'
'Page (Total results: 50)'
"Next>"
'B09Q67H373'; 'JSPOYOU 2022 Newly T-Shirt for Mens Funny 3D Graphies Pattern Crewneck Short Sleeve Tees Big and Tall Summer Casual Comfy Top',
'$1.99 $8.99"
'B09QQP3356': 'HAUKLIE Men's Sports Waffle Ribbed Polo Shirts Summer Short Sleeve Cotton Muscle Quarter-Zip Henley T-Shirt Tunics Tops', '$10.99 "B09NNM99.N: JSPOYOU Mens Short Sleeve Crewneck 3D Graphic Tunic Shirts Big & Tall Tie Dye Summer Top Basic Designed Classic Cotton Shirt'.
'$3.99 to $10.99'
"B09S3BN15C': "Mens Linen Shirt,Men's Striped Shirts Casual Short Sleeve Button Down Shirts Regular Fit Hawañan Shirts Beach Tees Tops', ; '53.78 to $1 1,38 "B08SWQKRRJ": S.LANMIUMMS Men's Classic Comfort Soft Regular Fit Short Sleeve Henley T-Shirt Tee", '$17.99 to $19.99'
"B078WSND96:: "Clearance Deals Retro T-Shirt Bee Printed Blouse, 2021 Summer Fashion Boys Men 3D Round Neck Short Sleeve Tee Tops'. '99.99 to $11.99' BB8JD14GJJF: "COOrun Women's 's UPF 50+ Sun Protection Shirts Quick Dry Long Sleeve Shirts Lightweight T-Shirt Outdoor Hiking Runing Fishing', '$9.98 to $16.98'
"B09175KKQ5:: "WENKOMGI Men's Crewneck Goth Tee Shirts Skull Print Tops Spring/Summeer Long Sleeve Sports T-Shirt Baggy Y2K Soft Streetwear' '$4.09 to $8.09'
'B09R9YCM6R': "XXBR Summer T-shirts for Mens. Soldier Short Sleeve Street Vintage Printed Shirt Slim Musele Casual Tee Tops', '$8.98 to $11.99 'B09Q5ZHRVM": "Yinimo Mens Gym Shorts Fashionable Loose Beach Drawstring Turkey Print Beach Cool Shorts'. ''19.19'.
Agent (Step
<think> Given the current page results, it clear that the search is not yet filtered to include only tuxedo shirts. Therefore, the first step should be to click the product that seems to be the most relevant, which is 'B09Q67H373' This product t-shirt, which is close but not the exact item we are looking for However, it is the first item that matches the search criteria in terms of color and price range. The product description also mentions "Mens Funny 3D Graphics Pattern Crewneck Short Sleeve Tees". which suggests it might be tuxedo shirt. We will click on to see if it is indeed a tuxedo shirt and if meets the other criteria, </thinks (actiooncccckk(09944//333333333

---

down the search and then Environment (Step 3)
"Back Search" << Prev'
"color': "a-yellow", 'b-blue', 'c-blue' 'd-black' 'd-gray" 'd-navy'
"size': 'small' 'medium' 'large", 'x-large" 'xx-large' "SPOYOUU 2022 Newly T-Shirt for Mens Funny 3D Graphics Pattern Crewneck Short Sleeve Tees Big and Tall Summer Casual Comfy Top" 'Piice: $1.99 to S8.99' "Rating: N.A.' "Description' 'Features', 'Reviews', "Buy Now".
Agent (Step
<thinks> Based on the previous obscrvations and actions, is clear that the search is not yielding any results that match the specific criteria of loose fit, slim fit men's 's tuxedo shirts with long and short sleeves, regular fit for gym workout, and the desired color (b-blue) and size (xx-large). The current observation shows that the search results are not matching the specific product type (tuxedo shirt) and the specific (regular fit for gym workout).
Given current options, the best action to refine search by first checking are not matching, will 'b-blue" and then check the options.
color and size options. Since the product type first click
This approach based on the assumption that product type and fit might be the search results but explicitly laheled tuxedo shirts or regular for gym workout. By clicking 'b-blue' color. can check the size options see if the product is tuxedo shirt and if it regular fit for gym workout. </think> Cactionnccick((--lleej/aadtied
Environment (Step 4)
"Back to Scarch"
<< Prev'
"color': 'a-yellow", 'b-blue' 'e-blue", 'd-black', 'd-gray", 'd-navy' "size": "small", 'medium' ''larg'. , 'x-large', 'xx-large" "ISPOYOU 2022 Newly T-Shirt for Mens Funny 3D Gruphics Pattern Crewneck Short Sleeve Tees Big and Tall Summer Casual Comfy Top' 'Price: $1.99 to $8.99' 'Rating: N.A.' 'Description' FFeatrrs' 'Reviews'. "Buy Now".
Agent (Step 4)
<think>> The task find loose fit, slim fit men's tuxedo shirts with long and short sleeves, polyester cotton, elastic waist, regular fit for gym workout, blue color (b-blue), size xx-large, and with price lower than $40.00. From the current observation, it clear that the search is already filtered include the color b-blue. The next step to filter the size to xx-large and check the price range. </think> <action=click/se (aaeee//ex/ae///tttttttt
Environment (Step 5)
'Back to Search' "< Prev"
"color': 'a-yellow", 'b-blue' 'e-blue', 'd-black', 'd-gray', 'd-navy' 'small" 'medium' 'large' 'x-large', 'xx-large' "ISPOYOU 2022 Newly T-Shirt for Mens Funny 3D Graphics Pattern Crewneek Short Sleeve Tees Big and Tall Summer Casual Comfy Top' 'Price: $1.99 to S8.99" 'Rating: N.A." 'Description' 'Feature', 'Reviews", "Buy Now*.
"size':
25

---

Agent (Step 5)
<thinks The task find lonse slim fit men's tuxedo shirts with long and short sleeves, made of polyester cotton, with clastic waist, regular fit for gym workouts, in color b-bluc, size xx-large, and with price lower than $40.00. From the current observation. it is clear that the search is already filtered to include the color b-blue, size xx-large and price lower than $40.00. Therefore, the next step is click the 'Buy Now" button. </think> <action>click(buy </actiino
F.3
Search
Environment (Step 1)
Who is the guy in the video of the performer of One Last Time?
Agent (Step 1)
<think> To answer this question, need to determine who the guy in the video is. The question refers to the performer of the song "One Last Time." The song "One Last Time" was performed by the boy band #NSYNC. The video likely features the members of *NSYNC. Without more context, cannot determine which specific member is referred to. I will need to search for the video to determine the specific guy. </thnnk> wscarch>-ho the guy in the video the performer One Last Time</search>>
Environment (Step 2)
<information> "result": "Doc 1: "Just One Last Time" uploaded Guetta's 's official VEVO channel on December 3, 2012. The music video for the track was filmed in Los Angeles during October 2012 by director Colin Tilley. It features man going in his half-burned house and lying on the bed and rememhering the death of his girlfriend, reliving and trying change the past in his mind: he is sneaking of his girlfriend's house to go and play a game of pool with Guetta, only to hear that the house is on fire. He races back to save his girlfriend, this time managing to do so, Doc 2: "One Last Time (Ariana Grande song)" rest of the video then is presented through the use of a handheld viden camera. Grande is in a Fiat car as the passenger, with her boyfriend (played by Matt Bennett), who is recording. They are stuck in traffic as they try to leave the city, implicd to be Los Angeles, California. While everyone around them is panicking, Grande seems to be calm as she wants to get a closer look at the comet. Impatient, she leaves the car while Bennett tries to stop her, and then follows. They make their way through the crowds, but are stopped by the Doc 3: "One Last Time (Ariana Grande song)" and the United States, where became Grande's 's sixth top-twenty single on the US "Billboard" Hot 100. The single's ' peak number 13 on the US "Billhoard"" Hot 100 ended Grande's streak consecutive top ten hits from 2014. was later certified platinum by Recording Industry Association of America (RIAA) Australian Recording Industry Association (ARIA). song's accompanying music video was released February 15, 2015 and was directed by Max Landis. The music video visually presented as found fontage, similar Landis' previous work." work.") </informationn
Agent (Step
<think>> From the information provided, there are songs titled "One Last Time" mentioned. The first one by David Guetta, and the second one is by Ariana Grande. The question usks about the video of the performer of "One Last Time". The information provided for the Ariana Grande song includes details about the music video, which was released on February 15, 2015, directed by Max Landis, The video is described as found fontage. The question might be referring to the video of Ariana Grande's s "One Last Time". The information does not provide specific name for the guy in the vidco, so will assume the question is about the video of Ariana Grande's 's song. </think> wso is the guy in the video of Ariana Grande's 's One Last Time</search-
26

---

Environment (Step 3)
<information> "result": "Doe "One Last Time (Ariana Grande song)" of video then is presented through the Fiat the passenger, with her boyfriend (played by Matt Bennett), who is recording. They stuck traffic they try leave the city, implied to be Los Angeles, California. While everyone around them is panicking. Grande calm as she wants get closer look at the comet. Impatient, she leaves the while Bennett tries to stop her, and then follows. They make their way through the crowds, but stopped by the Doc 2: "One Last Time (Ariana Grande song) and the United States, where it became Grande's ' sixth top-twenty single on the US ""Billboard" Hot 100. The single's 's peak of number 13 on the US "Billboard" Hot 100 ended Grande's 's streak of consecutive top ten hits from 2014. It was later certified platinum by the Recording Industry Association of America (RIAA) and Australian Recording Industry Association (ARIA). The song's 's accompanying music video was released on February 15, 2015 and was directed by Max Landis. The music vidco is visually presented as a found footage, similar to Landis' previous work. The video faced controversy due to its similarity to the Doc 3: "One Last Time (Ariana Grande song)" later dismissed after both sides filed Joint Stipulation for Dismissal. Grande came up with idea of the music video and contacted Max Landis, known for his 2012 film ""Chronicle" "" ""It's very unique, it's very different, called Max [Landis] up and said 'hey want something
handheld video camera. Grande in
to be
interview, Grande explained:
ends exactly like this' and described ending shot to to he take [music video] 2026 and I ending shot look like this, And described those things") </information>
him and then said also
Agent (Step
<think> From the information provided, it seems the question is about the music video for Ariuna Grande's song "One Last Time". The video was directed by Max found footage. The video shows Ariana Grande and her boyfriend (played by Matt Bennett) in a Fiat car, stuck in traffic trying to leave the city. The "guy" in the video is Matt Bennett, who plays Grande's 's boyfriend in the video. </think>
Landis and presented
<unswer>Matt Bennett</answer>
27

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims in the abstract and introduction accurately reflect this paper's contributions and scope: the proposed GiGPO algorithm that preserve the critic-free, low- memory, and stable convergence properties of group-based RL. while introducing fine- grained credit assignment for long-horizon LLM agents.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: See Section 6.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
28

---

Answer: [NA]
Justification:
This work does not involve theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We provide all necessary materials to reproduce the main results, including: (I) detailed training setup in Appendix E.I, (2) full source code with out-of-the-box scripts and dependency files at https: happs://gitiin //tpt/ggtttt/tt//// com/ aoon//agffff enge/ene/eeeeeeeeee code details in Appendix A.
and (3) more
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
29

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: We provide the complete source code at https: https://gitie //ghthub com/ com/land langf engQ/ verl aggltt along with detailed installation guidelines and out-of-the- too-thh-boxx scripts in Ap- pendix A for reproducing our paper results.
Guidelines:
The answer NA means that paper does not include experiments requiring code. Please see the NeurIPS code and data submission guidelines pictps://nips. cc/
onPolicy) for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to rum to reproduce the results. See the NeurIPS code and data submission guidelines (https: //n //ipps.coo wwtpcc/pppllc.. wttggguids///eee https//ooeeSubiiiseeee wwssssnolliiooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: See Appendix E.I.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification: We report both the mean and standard deviation (std) across independent runs with different random seeds for all key results in Table 1, Table 3, and Table 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
30

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Appendix E.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: //tearr/////....//////e/ea cc/ cttppipppcccpcc///cccccccc ittpliieeecccssciiilleiieeeeee
Answer:
[Yes]
Justification: Our research fully complies with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
See Appendix
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
31

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification:
This paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We properly cite benchmarks (e.g., ALFWorld, WebShop, Search-RL Sokoban, and EZPoints), pretrained models (e.g., Qwen2.5), and APIs (c.g.. GPT-4o and 2emini-2.5-Pro) with acknowledgment of their licenses and terms use.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
32

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The paper provides new code assets at verl agrl:ttent: an anonymized version of the code is provided with the submission, includ- ing README with setup instructions, training and evaluation procedures, and license information (see details in Appendix A).
cwm/langf engQ/
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: No human subjects were involved in this work.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: No human subjects were involved in this work.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
33

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core methodology of this paper is fully original and developed by the authors without the use of LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https: pettps:/neetiee //neruurpp.... ccc/onnereences/20000 for what should or should not be described.
34


Reviews:
[{"review_id": "NurOZIozCN", "reviewer": "Reviewer_yaqn", "summary": "The authors present GiGPO, a new RL algorithm aimed at tackling the tricky problem of credit assignment for LLM agents in long-horizon tasks. They argue that standard group-based methods like GRPO fail to distinguish good from bad actions within a single long trajectory. The key idea here is to supplement the usual episode-level advantage with a more fine-grained, step-level signal. This is achieved through what they call \"anchor state grouping\"—a  process that groups together actions taken from the exact same state across different rollouts. This allows for a localized relative advantage calculation without needing extra models or environment interactions. The authors demonstrate on ALFWorld and WebShop that their method provides significant gains over existing approaches while remaining just as efficient as the GRPO baseline.", "questions": "1. Equation (8) introduces the hyperparameter ω to trade off the global, episode-level signal (AE) against the local, step-level signal (AS). In the manuscript, ω is fixed at 1 without any tuning or ablation. Could you elaborate on how sensitive the reported results are to this choice? In particular, might certain values of ω shift the optimal solution of the original problem or induce undesirable agent behaviours?  If there are some theoretical analysis, it will be helpful.\n2. The proposed anchor-state grouping relies on exact state matches, a condition that may be fragile in noisy or high-dimensional environments. Have you analysed how performance varies with increasing state-space noise or complexity? For example, how does the performance gap between GiGPO and GRPO evolve as the probability of encountering an exact state match diminishes? Any empirical or theoretical insights into this degradation would strengthen the contribution.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strengths \n1. The core contribution, \"anchor state grouping,\" is a practical way to get fine-grained credit signals without the usual costs. It's an elegant solution that avoids the prohibitive expense of extra rollouts or auxiliary critic models, which is a big win when training these large-scale agents.\n2. A convincing set of experiments. They don't just test on one toy problem; they show consistent, significant gains on tough benchmarks like ALFWorld and WebShop against a strong field of baselines, including GRPO itself. The numbers speak for themselves, with reported boosts of over 12% on ALFWorld and 9% on WebShop over the GRPO baseline, which is quite impressive. I also appreciate the deeper analysis provided; the ablation studies effectively demonstrate that both the macro and micro advantage components are necessary for the method's success, and the plot showing how group sizes evolve during training was a nice touch that gives real insight into the learning dynamics.\n3. the paper is very well-written.\n# Weaknesses\n1. My main concern is the robustness of the anchor state grouping. The current method relies on exact state matching, which seems to work well enough in the clean, simulated environments tested6. But how would this hold up in the wild? In messier, noisier settings with minor but constant state variations (e.g., dynamic content on a webpage), this hashing approach would likely fail often, causing the method to simply fall back to GRPO. The authors do acknowledge this in their conclusion. but the paper would be much more compelling if it included an experiment that actually tests this degradation, perhaps by introducing controlled noise into the state representations. That would give us a much better sense of the method's practical limits.\n2. Secondly, the paper is missing a key ablation study for the hyperparameter ω. This coefficient in Equation (8) balances the macro (episode) and micro (step) advantage signals and seems quite important to the whole setup. Just setting it to 1 \"with no further tuning\" feels like a missed opportunity. It's very likely that the optimal balance is task-dependent, and an analysis of how performance changes with different values of  ω would add a lot of value for anyone looking to apply this method.\n3. Finally, the assumption of identical initial states for all trajectories in a group, while standard for this kind of RL, does limit the method's immediate applicability. It's fine for simulations, but it's a real barrier for many real-world scenarios where you can't just reset the world to the exact same starting point. A more explicit discussion on how this limitation might be relaxed in future work would be a welcome addition to the paper.", "was_revised": true, "final_justification": "The response addresses my concerns, and the paper can meet the acceptance standards. Thus, I keep my scores.", "rebuttal": "We sincerely appreciate your in-depth and encouraging feedback.\nThank you for highlighting that our method provides “an elegant solution” for obtaining fine-grained credit signals and finding the experimental results convincing. \nWe also appreciate your kind comments on the clarity of the paper and the “deeper analysis” offered through ablation studies and training dynamics visualizations.\nRegarding your concerns, we provide our responses below:\n\n> (Weakness 1 & Question 2) The robustness of the anchor state grouping.\n\nThank you for this insightful suggestion. As discussed in Sec. 6 (Conclusions and Limitations), we can use **state similarity** to better capture structurally equivalent states in noisy scenarios.\nWe have explored this perspective and conducted experiments to evaluate how GiGPO performs under realistic noise, where exact state matches are rare. We find **using similarity for GiGPO works well**.\n\nSpecifically:\n- (Noisy WebShop): To simulate realistic noise, we used GPT-4o to generate 551 diverse advertisements, which are **randomly inserted** into the states returned by WebShop to **simulate noise from ad pop-ups**. This substantially increases the state spaces.\n- (Similarity‐based grouping): We use Ratcliff–Obershelp similarity (ranges from 0 to 1) to construct step-level groups: **grouping the states whose similarity** $\\geq$ `sim_thresh`. Exact match is the special case `sim_thresh=1.0`.\n\nThe results are shown as follows:\n\n||Score|Success Rate |Avg. Step Group Size|\n|:-:|:-:|:-:|:-:|\n|GiGPO (sim_thresh=1.0)|74.5|56.7|1.01|\n|GiGPO (sim_thresh=0.9)|82.6|66.0|4.50|\n|GiGPO (sim_thresh=0.8)|83.2|66.2|4.96|\n|GiGPO (sim_thresh=0.5)|80.7|64.8|6.69|\n|GiGPO (sim_thresh=0.2)|78.7|58.6|12.15|\n\nWe can see that:\n- **Exact-match failure**. With `sim_thresh=1.0`, the avg. step group size collapses to $\\approx 1$, indicating that almost no identical states are found. As a result, GiGPO **degrades to GRPO** and performance drops.\n- **GiGPO remains robust and effective when using similarity-based grouping** even under noisy conditions. As shown, `sim_thresh=0.9/0.8` works well in such a setting. As `sim_thresh` decreases further (e.g., 0.5 or 0.2), group size increases, but similarity quality diminishes, revealing a trade-off between group generalization and precision.\n\n> (Weakness 2 & Question 1) Ablation study for the hyperparameter $\\omega$\n\nThanks for your valuable feedback. We conduct the sensitivity study on the hyperparameter $\\omega$ in WebShop.\n| $\\omega$           | 0.0  | 0.2  | 0.4  | 0.6  | 0.8  | 1.0  | 1.2  | 1.4  |\n|--------------------|------|------|------|------|------|------|------|------|\n| Score              | 76.2 | 79.6 | 82.4 | 83.5 | 84.9 | 83.5 | 82.6 | 77.0 |\n| Success Rate | 56.6 | 63.1 | 65.8 | 67.2 | 68.3 | 67.4 | 66.5 | 56.3 |\n\nWe find that \n- **GiGPO needs an appropriate $\\omega$ to work best**.\n- Increasing $\\omega$ **initially improves** performance due to the added fine-grained step-level reward. However, performance **declines beyond the optimum** ($\\omega=0.8$), suggesting that excessive emphasis on step-level signals can suppress useful trajectory-level guidance.\n- GiGPO is **relatively insensitive** to $\\omega$ within the range $[0.4, 1.2]$, demonstrating a reasonable degree of robustness to $\\omega$.\n\n> (Weakness 3) The assumption of identical initial states\n\n- The assumption of identical beginning is a **standard setting in all group-based RL methods** such as GRPO, RLOO, and DAPO. GiGPO is designed for group-based RL and therefore **follows this established setting**.\n- We also explored the **non-identical** initial state case in our response to (Weakness 1 & Question 2), where we introduced randomized ad pop-ups to perturb the initial states. GiGPO remained effective using similarity-based grouping, suggesting that **exact initial state matching might not be strictly necessary**.\n- This indicates that knowledge sharing is feasible among similar tasks, providing promise for relaxing this assumption in future work.\n\n\nThank you again for all your constructive feedback. We will include these clarifications, especially the experiments on robustness to noise and the hyperparameter sensitivity analysis."}, {"review_id": "3H0AXR2UPB", "reviewer": "Reviewer_vKcT", "summary": "The paper extends the group-based reinforcement learning method from single-turn tasks to long-horizon multi-turn agent training and proposes a Group-in-Group Policy Optimization method, named GiGPO. The method introduces two-level structure to achieve fine-grained credit assignment for LLM agents. At episode level, GiGPO samples a group of trajectories and compute advantage following vanilla GRPO. At step level, GiGPO splits actions into groups based on anchor states, enabling step-level guidance for long-horizon agent refinement.  Experiements on ALFWorld and WebShop with two base models demonstrates the effectiveness of GiGPO.", "questions": "See weaknesses.", "limitations": "The authors discuss their potential limitations in their paper.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "**Strengths**\n1. The paper extends the group-based RL method to multi-turn tasks and proposes GiGPO, which designs rewards from both solution-level and step-level, enabling both global and local credit assignment. This significantly improves the agent’s performance over long decision sequences.\n2. The step-level credit assignment is constructed retroactively using existing trajectories, thereby avoiding additional rollouts or costly value model estimations, which helps preserving the strengths of group-based RL methods.\n3. Experimental results demonstrate that GiGPO achieves substantial improvements over existing baselines across both benchmarks and model scales. \n4. The paper is clearly written and easy to follow.\n\n\n**Weaknesses**\n1. Although GiGPO is designed to minimize additional computational overhead, the method requires constructing step-level groups to estimate relative advantages. This process may demand a substantial amount of data to ensure group diversity and representativeness.\n2. GiGPO is primarily designed for long-horizon tasks and relies on the presence of anchor states. This design limits its applicability in certain scenarios—for example, in short-horizon or simple tasks, its advantages may not be as significant.\n3. GiGPO builds step-level groups by identifying and grouping repeated environment states across trajectories. This requires that environment states be repeatable and comparable across different rollouts. However, in real-world settings, environment states may suffer from noise, variability, or observation inaccuracies, which could hinder accurate step-level grouping and affect the reliability of relative advantage estimation.", "was_revised": true, "final_justification": "Most of my questions and concerns have been addressed during the rebuttal, I will keep the score and recommend the acceptance of the paper.", "rebuttal": "Thank you very much for your detailed and thoughtful review. \nWe truly appreciate you highlighting our credit assignment design as effective and pointing out that GiGPO enables fine-grained guidance while avoiding additional rollouts or costly value model estimations.\nWe also thank you for your kind comments on the clarity of the paper and the substantial performance gains across benchmarks and model scales.\nRegarding your questions, we provide our responses as follows.\n\n> (Weakness 1) Constructing step-level groups demands a substantial amount of data.\n\nWe agree that increasing data volume can lead to more robust step-level groups. However, we would like to clarify that our experiments on GiGPO were actually conducted using **relatively small data settings** `(batch_size=16, group_n=8)`. \nFor comparison, prior agentic training work Search-R1[1] uses `(batch_size=512, group_n=5)`.\n\nDespite this, GiGPO was able to form effective step-level groups and maintain stable training, demonstrating its **efficiency and practicality even under small data conditions**.\n\n> (Weakness 2) Short-horizon or simple tasks.\n\nThanks for your valuable insight. To explore this, we evaluated GiGPO on a short-horizon agentic task using the **Search-R1 environment[1]**:\n\nSpecifically:\n- This task involves answering questions via **search engine tool-calling**, with a maximum of 4 steps (short-horizon).\n- We tested both GRPO and GiGPO using Qwen2.5-1.5B-Instruct.\n- Since search results are inherently variable, with extremely large state spaces, we used similarity-based grouping for GiGPO as introduced in our response to (Weakness 3).\n\nThe results are shown as follows (*Note: The “Search-R1” results are sourced from the original paper [1]*):\n||NQ|TriviaQA|PopQA|HotpotQA|2wiki|Musique|Bamboogle|Avg.|\n|-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|Search-R1 (Qwen2.5-3B-Instruct)|0.341|0.545|0.378|0.324|0.319|0.103|0.264|0.325|\n|GRPO (ours)|0.380|0.541|0.423|0.303|0.269|0.088|0.317|0.331|\n|GiGPO (ours)|0.406|0.574|0.441|0.294|0.284|0.103|0.312|0.345|\n\nWe can see that, despite the short-horizon nature of the task, GiGPO's step-level signals still provided **meaningful improvements** over GRPO. Naturally, this advantage becomes more pronounced as the number of steps increases.\n\n> (Weakness 3) State noise and variability.\n\nThanks for your valuable insight. As discussed in Sec. 6 (Conclusions and Limitations), we can use **state similarity** to better capture structurally equivalent states in noisy scenarios.\nWe have explored this perspective and conducted experiments to evaluate how GiGPO performs under realistic noise where exact state matches are rare. We find **using similarity for GiGPO works well**.\n\nSpecifically:\n- (Noisy WebShop): To simulate realistic noise, we used GPT-4o to generate 551 diverse advertisements, which are **randomly inserted** into the states returned by WebShop to **simulate noise from ad pop-ups**. This substantially increases the state spaces.\n- (Similarity‐based grouping): We use Ratcliff–Obershelp similarity (ranges from 0 to 1) to construct step-level groups: **grouping the states whose similarity** $\\geq$ `sim_thresh`. Exact match is the special case `sim_thresh=1.0`.\n\nThe results are shown as follows:\n\n||Score|Success Rate|Avg. Step Group Size|\n|:-:|:-:|:-:|:-:|\n|GiGPO (sim_thresh=1.0)|74.5|56.7|1.01|\n|GiGPO (sim_thresh=0.9)|82.6|66.0|4.50|\n|GiGPO (sim_thresh=0.8)|83.2|66.2|4.96|\n|GiGPO (sim_thresh=0.5)|80.7|64.8|6.69|\n|GiGPO (sim_thresh=0.2)|78.7|58.6|12.15|\n\nWe can see that:\n- **Exact-match failure**. With `sim_thresh=1.0`, the avg. step group size collapses to $\\approx 1$, indicating that almost no identical states are found. As a result, GiGPO **degrades to GRPO** and performance drops.\n- **GiGPO remains robust and effective when using similarity-based grouping** even under noisy conditions. As shown, `sim_thresh=0.9/0.8` works well in such a setting. As `sim_thresh` decreases further (e.g., 0.5 or 0.2), group size increases, but similarity quality diminishes, revealing a trade-off between group generalization and precision.\n\nThank you again for your valuable and encouraging feedback. We will incorporate all these clarifications and improvements into our revision.\n\n---\nReferences:\n\n[1] Jin, Bowen, et al. Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning. arXiv, 2025."}, {"review_id": "GPSbPIWJmN", "reviewer": "Reviewer_aEAm", "summary": "The paper introduces Group-in-Group Policy Optimization (GiGPO), a novel reinforcement learning technique for training large language model (LLM) agents in long-horizon tasks. GiGPO uses a two-level advantage estimation structure: episode-level advantages for overall trajectory quality (original GRPO) and step-level advantages for fine-grained credit assignment (based on anchor state). This hierarchical approach enables precise optimization without additional rollouts or auxiliary models, maintaining efficiency and scalability. Evaluations on ALFWorld and WebShop show performance gains over existing methods, with over 12% improvement on ALFWorld and over 9% on WebShop.", "questions": "1. I think Section 5.5 of the paper is too obvious, and the content of E.3 or E.4 might be more important. How do the authors comment on this?\n2. Is it possible to further enhance the technical section of the paper, perhaps by including at least one task demonstrating GiGPO's effectiveness under non-precise anchor state matching?\n3. Can the improvements proposed by the author be applied to PPO?\n4. Could the experiments be supplemented by applying similar advantage calculations in other reinforcement learning frameworks to strengthen the generality of this idea?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n* The method proposed in this paper is simple to experiment with and intuitively useful.\n* The paper is well-written with clear exposition and reasonable experiments.\n* This paper reports good performance for ALFWorld and WebShop.\n\nWeaknesses\n* As mentioned in the paper, state matching itself is a challenging problem, and the current algorithm is mainly tested on tasks with a limited number of states. This leads to two issues: first, when reading the paper's introduction, I mistakenly thought this was a universal GRPO improvement, and I believe the writing here is a bit flawed; second, the technical depth discussed in the paper is relatively average.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your constructive review and positive feedback.\nWe truly appreciate you highlighting the strengths of our method: the simplicity and intuitive design that make it easy to experiment with, the clarity and structure of the paper, and the strong empirical performance.\nBelow, we provide our detailed responses to your remaining questions.\n\n> (Weakness 1) Universal GRPO improvement.\n\nWe appreciate you pointing out this confusion. To clarify: GiGPO is **not intended for single-turn tasks** (e.g., math or question answering), but is designed as a **universal GRPO improvement for LLM agent scenarios**, which involve multi-turn or multi-step interactions.\nWe will revise the text to reflect the intended scope and applicability of GiGPO more accurately.\n\n> (Question 1) Section E.3 or E.4 might be more important.\n\nThank you for this helpful suggestion. We agree that Sections E.3 and E.4, which demonstrate the generality of GiGPO, offer important insights and may be more impactful than Section 5.5. We will consider moving them into the main paper to better highlight the broader applicability of GiGPO.\n\n> (Question 2) Demonstrating non-precise anchor state matching.\n\nThanks for your valuable suggestion.\nWe have explored this perspective and conducted experiments to evaluate how GiGPO performs under realistic noise where exact state matches are rare. We find **using similarity for GiGPO works well**.\n\nSpecifically:\n- (Noisy WebShop): To simulate realistic noise, we used GPT-4o to generate 551 diverse advertisements, which are **randomly inserted** into the states returned by WebShop to **simulate noise from ad pop-ups**. This substantially increases the state spaces.\n- (Similarity‐based grouping): We use Ratcliff–Obershelp similarity (ranges from 0 to 1) to construct step-level groups: **grouping the states whose similarity** $\\geq$ `sim_thresh`. Exact match is the special case `sim_thresh=1.0`.\n\nThe results are shown as follows:\n\n||Score|Success Rate|Avg. Step Group Size|\n|:-:|:-:|:-:|:-:|\n|GiGPO (sim_thresh=1.0)|74.5|56.7|1.01|\n|GiGPO (sim_thresh=0.9)|82.6|66.0|4.50|\n|GiGPO (sim_thresh=0.8)|83.2|66.2|4.96|\n|GiGPO (sim_thresh=0.5)|80.7|64.8|6.69|\n|GiGPO (sim_thresh=0.2)|78.7|58.6|12.15|\n\nWe can see that:\n- **Exact-match failure**. With `sim_thresh=1.0`, the avg. step group size collapses to $\\approx 1$, indicating that almost no identical states are found. As a result, GiGPO **degrades to GRPO** and performance drops.\n- **GiGPO remains robust and effective when using similarity-based grouping** even under noisy conditions. As shown, `sim_thresh=0.9/0.8` works well in such a setting. As `sim_thresh` decreases further (e.g., 0.5 or 0.2), group size increases, but similarity quality diminishes, revealing a trade-off between group generalization and precision.\n\n> (Question 3 & Question 4) Can similar advantage calculations be applied to other RL frameworks?\n\n1. **Yes**. As discussed in Section E.4, the core ideas of GiGPO (anchor state grouping & step-level advantage) are **general** and can be applied to **all group-based RL frameworks**.\nWe have already extended GiGPO to DAPO in Section E.4 and additionally applied it to RLOO for this response. Results are summarized below:\n||Score|Success Rate|\n|-|-|:-:|\n|DAPO (Sec. E.4)|84.6|66.1|\n|DAPO + Step-Level Adv. (Sec. E.4)|87.5|75.0|\n|RLOO|73.9|52.1|\n|RLOO + Step-Level Adv. |83.3|63.2|\n\n2. Regarding PPO: As PPO is not a group-based RL method and does not require multiple rollouts for the same task, applying anchor state grouping to PPO is **not straightforward**. However, one potential way is to **introduce task grouping into PPO by rolling out a set of trajectories under the same task**. In such a setting, the core idea of GiGPO can be incorporated into PPO.\n\n\nThank you again for your valuable feedback. We will incorporate the above points in our revised version to enhance the technical contributions."}, {"review_id": "4UFxGj7TxL", "reviewer": "Reviewer_fP6w", "summary": "This paper aims to address the credit assignment problem in multi-step reinforcement learning and discovers the anchor state phenomenon. The authors introduce step-level fine-grained advantage computation, which enables better reuse of existing data and combines with episode-level advantages to obtain more accurate advantage signals. The experimental results are impressive, demonstrating significant performance improvements.", "questions": "During the actual rollout process, in each turn, how are $r_t^{(i)}$ and $r_k^{(i)}$ given? My understanding is that only the final step receives a reward, while all other steps have a reward of 0?", "limitations": "1. The method is only applicable to environments with limited or simple state spaces where repetitive environmental states are easily identifiable or frequently occur.\n2. The method incurs high training overhead and requires substantial computational resources.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. High writing quality and clarity: The paper is well-written and easy to understand, with well-formatted figures, tables, and equations that enhance comprehension.\n2. Accurate problem identification with strong motivation: The paper effectively targets the credit assignment problem in multi-step reinforcement learning with well-justified motivation.\n3. Clever method design: The authors first discover the anchor state phenomenon during experimentation, then introduce step-level advantage computation to fully utilize this data information, incorporating more fine-grained advantage signals. The computational overhead is negligible, requiring no additional rollouts or complex calculations.\n4. Impressive experimental results: The results demonstrate the effectiveness of the proposed method design.\n\nWeaknesses:\n1. Method design relies on task characteristics, raising concerns about generalizability: The method depends on identical states to construct anchor groups, which may only be applicable to environments where identical states can be detected. For more complex and open multi-step environments (such as multi-turn dialogue, web queries, textgame: SOTOPIA[1], GAIA[2], NegotiationArena[3], LMRL Gym[4]), this method may not be applicable. The two environments validated in the paper, ALFWorld and WebShop, are relatively simple with limited state spaces, making it easy for identical states to reappear during multi-step interactions.\n2. Limited hyperparameter sensitivity analysis: While I understand that multiple experiments are computationally expensive, conducting experiments on the weight parameter w to explore hyperparameter sensitivity would make the work more solid.\n3. Missing LRM baselines: Since the method employs the thinking-action paradigm, it should also compare against some LRMs such as DeepSeek-R1.\n4. Missing classic multi-step RL baselines: It is recommended to include classic baselines for multi-turn RL such as ArCHer[5] and Raven[6]. Raven may be recent work within three months before submission, so it's not mandatory if conditions don't permit.\n5. High computational resource requirements, difficult to reproduce: Based on the submitted code and my experience, the proposed approach requires substantial CPU (at least 64 cores) and GPU resources, resulting in significant computational overhead.\n\n[1] Zhou X, Zhu H, Mathur L, et al. SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agents[C]//The Twelfth International Conference on Learning Representations.\n\n[2] Mialon G, Fourrier C, Wolf T, et al. Gaia: a benchmark for general ai assistants[C]//The Twelfth International Conference on Learning Representations. 2023.\n\n[3] Abdulhai M, White I, Snell C, et al. Lmrl gym: Benchmarks for multi-turn reinforcement learning with language models[J]. arXiv preprint arXiv:2311.18232, 2023.\n\n[4] Bianchi F, Chia P J, Yuksekgonul M, et al. How well can LLMs negotiate? NEGOTIATIONARENA platform and analysis[C]//Proceedings of the 41st International Conference on Machine Learning. 2024: 3935-3951.\n\n[5] Zhou Y, Zanette A, Pan J, et al. ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RL[C]//International Conference on Machine Learning. PMLR, 2024: 62178-62209.\n\n[6] Wang Z, Wang K, Wang Q, et al. Ragen: Understanding self-evolution in llm agents via multi-turn reinforcement learning[J]. arXiv preprint arXiv:2504.20073, 2025.", "was_revised": true, "final_justification": "The authors have supplemented some experiments, which addressed my main concerns, if integrated into the revised paper, can meet the acceptance standards.", "rebuttal": "We sincerely appreciate your valuable comments and your positive remarks about our paper’s clear writing, strong motivation with accurate problem identification, and the effective design based on the anchor state grouping. \nWe're also glad you found the step-level advantage computation efficient and the experimental results impressive.\nRegarding your questions, we provide our responses below:\n\n> (Weakness 1) Generalizability to complex and open multi-step environments.\n\nThanks for your valuable insight. As discussed in Sec. 6 (Conclusions and Limitations), we can use **state similarity** to better capture structurally equivalent states in complex and open scenarios.\nWe have explored this perspective and conducted the following **two** experiments to evaluate how GiGPO performs under realistic noise where exact state matches are rare. We find **using similarity for GiGPO works well**.\n\n1. **Experiment 1**\n\t- (Noisy WebShop): We used GPT-4o to generate 551 diverse advertisements, which are **randomly inserted** into the states returned by WebShop to **simulate noise from ad pop-ups**. This substantially **increases the state spaces**.\n\t- (Similarity‐based grouping): We use Ratcliff–Obershelp similarity (ranges from 0 to 1) to construct step-level groups: **grouping the states whose similarity** $\\geq$ `sim_thresh`. Exact match is the special case `sim_thresh=1.0`.\n\t\n\tThe results are shown as follows:\n\t\n\t||Score|Success Rate|Avg. Step Group Size|\n\t|:-:|:-:|:-:|:-:|\n\t|GiGPO (sim_thresh=1.0)|74.5|56.7|1.01|\n\t|GiGPO (sim_thresh=0.9)|82.6|66.0|4.50|\n\t|GiGPO (sim_thresh=0.8)|83.2|66.2|4.96|\n\t|GiGPO (sim_thresh=0.5)|80.7|64.8|6.69|\n\t|GiGPO (sim_thresh=0.2)|78.7|58.6|12.15|\n\t\n\tWe can see that:\n\t- **Exact-match failure**. With `sim_thresh=1.0`, the avg. step group size collapses to $\\approx 1$, indicating that almost no identical states are found. As a result, GiGPO **degrades to GRPO** and performance drops.\n\t- **GiGPO remains robust and effective when using similarity-based grouping** even under noisy conditions. As shown, `sim_thresh=0.9/0.8` works well in such a setting. As `sim_thresh` decreases further (e.g., 0.5 or 0.2), group size increases, but similarity quality diminishes, revealing a trade-off between group generalization and precision.\n\t\n2. **Experiment 2**\n\t- To further demonstrate GiGPO's applicability in open environments, we applied it to the **Search-R1 environment[1] (web query task)**, where the agent interacts with a search engine (tool-calling) to answer questions. \n\t- We tested both GRPO and GiGPO using Qwen2.5-1.5B-Instruct.\n\t- Since search results are **inherently variable, with extremely large state spaces**, we used similarity-based grouping for GiGPO.\n\t\n\t||NQ|TriviaQA|PopQA|HotpotQA|2wiki|Musique|Bamboogle|Avg.|\n\t|-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n\t|Search-R1 (Qwen2.5-3B-Instruct)|0.341|0.545|0.378|0.324|0.319|0.103|0.264|0.325|\n\t|GRPO (ours)|0.380|0.541|0.423|0.303|0.269|0.088|0.317|0.331|\n\t|GiGPO (ours)|0.406|0.574|0.441|0.294|0.284|0.103|0.312|0.345|\n\n\tWe can see that, even in this open setting, GiGPO's step-level signal provides benefits over GRPO, demonstrating its effectiveness in open agentic tasks.\n\n> (Weakness 2) Hyperparameter sensitivity analysis.\n\nThanks for your valuable suggestion. We conduct the sensitivity study on the hyperparameter $\\omega$ in WebShop.\n| $\\omega$           | 0.0  | 0.2  | 0.4  | 0.6  | 0.8  | 1.0  | 1.2  | 1.4  |\n|--------------------|------|------|------|------|------|------|------|------|\n| Score              | 76.2 | 79.6 | 82.4 | 83.5 | 84.9 | 83.5 | 82.6 | 77.0 |\n| Success Rate | 56.6 | 63.1 | 65.8 | 67.2 | 68.3 | 67.4 | 66.5 | 56.3 |\n\nWe find that \n- **GiGPO needs an appropriate $\\omega$ to work best**.\n- Increasing $\\omega$ **initially improves** performance due to the added fine-grained step-level reward. However, performance **declines beyond the optimum** ($\\omega=0.8$), suggesting that excessive emphasis on step-level signals can suppress useful trajectory-level guidance.\n- GiGPO is **relatively insensitive** to $\\omega$ within the range $[0.4, 1.2]$, demonstrating a reasonable degree of robustness to $\\omega$.\n\n> (Weakness 3) LRM baselines.\n\nWe conduct experiments for `DeepSeek-R1-0528` and the results are provided below:\n\n||Pick|Look|Clean|Heat|Cool|Pick2|All|WebShop Score|WebShop Succ.|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|DeepSeek-R1|64.2|52.1|46.5|47.0|54.3|55.2|54.5|42.2|33.6|\n\nCompared with Table 1 in the paper, DeepSeek-R1 achieves performance between GPT-4o and Gemini-2.5-Pro across most tasks.\n\n> (Weakness 4) Classic multi-step RL baselines.\n\nThanks for your valuable suggestion. We conduct experiments for RAGEN[2] on WebShop and the results are provided below:\n\n||Score|Success Rate|\n|-|-|:-:|\n|RAGEN (Qwen2.5-1.5B-Instruct)|79.7|59.6|\n|GiGPO (Qwen2.5-1.5B-Instruct)|83.5|67.4|\n\nGiGPO shows improvements over RAGEN, further supporting its effectiveness in multi-step settings.\n\n> (Weakness 5) High computational resource requirements.\n\nWe appreciate the concern, but we would like to clarify that the cited resource demand is **ONLY the code-level optimization choices**, which are **fully independent of the GiGPO's algorithmic core** and can be adjusted based on available hardware.\n\n- In our experiments, we used 2 H100 GPUs and 64 CPUs **only to shorten wall-clock time ($\\approx$ 9h) via high parallelism**. This setup is an implementation convenience, not an algorithmic requirement.\n- The same training can be run on 1 H100 GPU + 32 CPUs (or even smaller setups) by, for example, leveraging the accumulation steps or allowing workers to execute sequentially. This change lengthens training time ($\\approx$ 18h) but leaves results unchanged.\n\n\n> (Question 1) How are $r_t^{(i)}$ and $r_k^{(i)}$ given?\n\n- This depends on the specific task and environment. Some environments provide dense rewards, while others are sparse-reward scenarios. In the case of **sparse-reward environments**, your understanding is correct: $r_t^{(i)} = 0$ for $t < T$; $r_T^{(i)} = 1$ if the task is successful, and $0$ otherwise.\n- However, it is worth noting that when constructing step-level groups, we do **not** directly use the raw per-step reward $r_t^{(i)}$ from the environment. Instead, we use the **discounted return** $R_t^{(i)}$, as defined in Equations (5) and (6). This allows us to assign **non-zero values to earlier steps** ($t<T$) and its effectiveness is illustrated in Figure 3 of the main paper.\n\n\nFinally, thank you again for your valuable and encouraging feedback. We will incorporate these clarifications and improvements into our revision.\n\n---\nReferences:\n\n[1] Jin, Bowen, et al. Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning. arXiv, 2025.\n\n[2] Wang, Zihan, et al. RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning. arXiv, 2025."}]

Author rebuttal:

Meta-review: This paper introduces Group-in-Group Policy Optimization (GiGPO), a simple, critic-free RL algorithm for long-horizon LLM agents that combines (i) episode-level relative advantages with (ii) step-level advantages computed by retroactively grouping identical environment states. The combined objective (Eq. 8) provides hierarchical credit signals without extra rollouts or an auxiliary value model, and delivers strong gains on ALFWorld and WebShop across 1.5B and 7B backbones.

**Strengths**
* **Clear problem framing & elegant method.** The paper targets the key gap in group-based RL for multi-turn LLM agents—fine-grained credit assignment—via a two-level advantage that is conceptually simple and implementation-friendly (critic-free, no extra rollouts).
* **Compelling empirical results.** On ALFWorld and WebShop, GiGPO consistently beats strong prompting, PPO, RLOO, and GRPO baselines.
* **Ablations that isolate what matters.** Removing either episode- or step-level advantages notably degrades performance, supporting the necessity of both macro and micro signals.
* **Clarity and presentation quality.** Reviewers uniformly praise the writing and structure; the paper also details environments, baselines, and training settings.

**Weaknesses raised in reviews & how the rebuttal addressed them**
* **Fragility of exact-match anchor grouping in noisy states.** Reviewers worry the hash-match assumption would collapse in real-world noise. The authors run a Noisy WebShop variant and switch to similarity-based grouping. Results show robustness with similarity thresholds 0.8–0.9, improving success vs. exact match. This both diagnoses failure at exact match and offers a practical fix with a precision-vs-group-size trade-off.
* **Sensitivity to the macro/micro weight $\omega$**. The revised analysis on WebShop finds an optimum around $\omega\approx0.8$ and robustness for $\omega\in[0.4,1.2]$, providing concrete guidance for practitioners and showing that performance is not knife-edge sensitive. 
* **Applicability beyond long-horizon / identical initial states.** The rebuttal adds short-horizon Search-R1 experiments where GiGPO still outperforms GRPO, showing benefit even with only ~4 steps. The authors also note that similarity-based grouping relaxes strict initial-state identity.
* **Missing/stronger baselines and compute practicality.** The authors add DeepSeek-R1 and RAGEN comparisons; GiGPO remains stronger. They also clarify compute: the H100×2 setup shortens wall-clock but is not required; training can run on a single H100 (+ longer time) with unchanged results.

The paper presents a clear, scalable solution to long-horizon credit assignment for LLM agents with solid evidence and constructive rebuttal additions. As a result, I recommend accept.
Decision: Accept (poster)