The Lighthouse of Language: Enhancing LLM Agents via Critique-Guided Improvement
Authors: Ruihan Yang
Fanghua Ye
Jian Li
Siyu Yuan
Yikai Zhang
Zhaopeng Tu
Xiaolong Li
Deqing Yang
Venue: NeurIPS 2025 poster ()
Keywords: Large language model
Language agent
Agent learning
Primary area: deep_learning

Abstract: Large language models (LLMs) have recently transformed from text-based assistants to autonomous agents capable of planning, reasoning, and iteratively improving their actions. While numerical reward signals and verifiers can effectively rank candidate actions, they often provide limited contextual guidance. In contrast, natural language feedback better aligns with the generative capabilities of LLMs, providing richer and more actionable suggestions. However, parsing and implementing this feedback effectively can be challenging for LLM-based agents. In this work, we introduce Critique-Guided Improvement (CGI), a novel two-player framework, comprising an actor model that explores an environment and a critic model that generates detailed nature language feedback. By training the critic to produce fine-grained assessments and actionable revisions, and the actor to utilize these critiques, our approach promotes more robust exploration of alternative strategies while avoiding local optima. Experiments in three interactive environments show that CGI outperforms existing baselines by a substantial margin. Notably, even a small critic model surpasses GPT-4 in feedback quality. The resulting actor achieves state-of-the-art performance, demonstrating the power of explicit iterative guidance to enhance decision-making in LLM-based agents.

The Lighthouse of Language: Enhancing LLM Agents via Critique-Guided Improvement
Ruíhan Yang 1 Fanghua Ye Jian L Siyu Yuan 1,0 Yikai Zhang 2.7, Zhaopeng Tu Xiaolong Li Deqing Yang!.
1.9 School of Data Science, University Tencent Hunyuan 2, College of Computer Science and Artifcial Intelligence, Fudan University (rhhyang1,, yangdeqingj)fudan edu.cn cn (syyuan21, ykzhang22Jem. fudan. .dd...du.com cn {fanghua. ye. 21, lijianjack, tuzhaopeng3_@maill.co com
Project Page: https: / thub.hhub..o. http://thuu..com///y//////2//2//
Abstract
Large language models (LLMs) have recently transformed from text-based sistants to autonomous agents capable of planning, reasoning, and iteratively improving their actions. While numerical reward signals and verifiers effec- tively rank candidate actions, they often provide limited contextual guidance. In contrast, natural language feedback better aligns with the generative capabilities of LLMs, providing richer and more actionable suggestions. However, parsing and implementing this feedback effectively can be challenging for LLM-based agents. In this work, we introduce Critique-Guided Improvement (CGI), a novel two-player framework, comprising an actor model that explores an environment and a critic model that generates detailed nature language feedback. By training the critic to produce fine -raiigr assessments and actionable revisions, and the actor to utilize these critiques, our approach promotes more robust exploration of alternative strategies. Experiments in three interactive environments show that CGI outperforms existing baselines by a substantial margin. Notably, even small critic model surpasses GPT-4 feedback quality. The resulting actor achieves state- of-the-art performance, demonstrating the power of explicit guidance to enhance docision-making LLM-based agents.
Introduction
Recent advances in large language models (LLMs) [1; 2; 3] have broadened their applicability from conventional chat and text-based interactions to more dynamic, agentic roles. In these roles, LLM- based agents [4; 5; 6] harness the reasoning and planning capabilities of LLMs to automate complex tasks across diverse domains, including code generation [7; 8J, software engineering [9; 10], and web applications [11; 12]. A key requirement in these agentic tasks is the ability to iteratively acquire, store, and use new information to improve performance. Thus, a central challenge emerges: How can high-quality feedback be obtained and utilized efficiently?
common approach for providing feedback is to rely on numerical signals, such as verifiers [13; 14] reward models [15; 16]. In this paradigm, LLMs typically trained as discriminators to evaluate and select the best action from a set of candidates. One widely used method is Best-of-N (BoN) [17; 18], where the agent generates N possible actions and a trained reward model picks the
WWok done during an intersship at Tencent Hunyuan. CCoresponding authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

highest-scoring one. However, numerical feedback has limited informativeness: it reduces guidance to scalar scores that merely filter candidate actions, overlooking context-specific suggestions, avenues for exploration, or strategies for refining suboptimal behavior.
more direct and flexible approach involves using natural language feedback. Rather than providing only a "best" action, LLMs can generate detailed critiques and explicit, context-aware recommen- dations for improvement. However, transitioning to natural language feedback presents two main challenges: 1) Weak Feedback: Many techniques rely on self-refinemen
self-correction [19; 20; which heavily depend on the LLM's inherent capacity. This reliance can lead to degraded performance. especially when models hallucinate encounter complex problems [20; 22]; 2) Poor Utilization: Unlike numerical feedback, which directly selects an action to execute, verbal critiques require the agent accurately interpret and integrate the guidance into its decision-making process. L.LM agents often struggle with this, sometimes demonstrating limited flexibility in adjusting or discarding previously generated plans [23; 24].
In response, we propose Critique- Guided Improvement (CGI), a two-player framework in which an actor model interacts with the environment while critic model provides supervisory feedback. CGI features two main stages: Critique Generation and Action Refinement. In the Critique Generation stage, we train critic model to produce precise evaluations and actionable revision instructions. In the Action Refinement stage, the actor model learns to effectively apply these critiques through iterative supervised fine-tuning, thereby improving both its reasoning and its ability integrate external feedback.
We conduct extensive experiments in three interactive environments [11: 25; 26], demonstrating that CGI substantially enhances performance over baseline approaches. Notably, our critic model, when trained small backbone (e.g.. Llama-3-8B), outperforms even GPT-4 as a critic by large margin (+29.16%) Additionally, the action-refinemmnt process further boosts results, surpassing state-of-the-ar metrics by 26.74%.
In summary, our contributions are follows: 1) We tackle the challenge of obtaining and utilizing high-quality nature language feedback in agentic tasks. 2) We present Critique-Guided Guided Improvement (CGI), a novel two-player framework that provides more informative feedhack L.L.-based agents by combining a specialized critic model with an actor model. 3) We develop training strategy that iteratively refines the actor's actions based on natural language critiques, enabling both improved reasoning and enhanced adaptability. 4) We conduct extensive experiments across three diverse interactive environments. By leveraging CGI, the agent can continuously improve its performance in long-horize tasks and enhance task efficiency by achieving higher scores in fewer steps.
2 Related Work
Learning from Feedback Current feedback can be categorized into numerical and natural language
feedback. Numerical feedback [27; 28; 29; 30] typically provided by training a reward model (RM) or verifiers. In this approach, the model learns to predict either the correctness of solution [13: 29] or the preference between multiple solutions [17: 31]. Specifically, the RM generates a continuous numerical score, which is then incorporated into a classification objective. In contrast, natural language feedback offers denser rewards by evaluating the model's actions using natural language. This feedback can be generated in two main ways: by prompting an off-the-shelf LLM to act verifier (i.e., -uddge)) [32; 33], or through self-refinement and self-critique [20; 34], where the model evaluates and improves its own outputs. Our trained critic model provides natural language feedback, offering denser and more accurate rewards.
Agent Learning in Interactive Environments. Previous approaches to agent learning in interactive
environments can be classified into three main categories: 1) Prompt-based methods [21: 35; 36: 37] utilize human-wrinn written prompts to guide LLMs in summarizing experiences. These summaries, which may include causal abstractions from both successful and failed attempts [20; 38] or transferable skills [39], are integrated into the model's memory to enhance its knowledge and performance. 2) Training-based methods [40; 12; 41] rely on techniques such as Supervised Fine-Tuning (SFT) [42; 43] or Direct Preference Optimization (DPO) [44; 45]. to train LLMs. The training data may come from expert models or generated through exploration strategies like Monte Carlo Tree Search (MCTS). 3) Inference-ting time sampling methods [14; 15; 16] employ techniques such as Best-of-N

---

(BoN) [17; 18] and Tree-of-Thought (ToT) [46] to identify optimal actions during inference. These methods leverage the prior knowledge LLMs,, enabling more efficient search processes. Our CGI method introduces novel variation of inference-time sampling by using trained critic that evaluates candidate actions and suggests revisions to enhance inference performance.
3 Preliminary
Partially Observable Markov Decision Process We define the collection of enviromments as E. For specific environment E E, the embodied tasks for LLM agents are typically modeled as Partially Observable Markov Decision Process (POMDP): (X, (X,S, S, O, T) Here, X' denotes the set of instructions, S represents the set of environment states, A is the set of available actions at each state, and O represents the observations available to the agent. The transition function T S is defined by the environment, while the reward function R S x A + [0, 1] specifies the reward received by the agent from the environment. For an actor model Th parameterized by 6, the actor selects the next action ~ Tn [a] Tr, e) at each time step t. based on the interaction history T,, which defined as
S
= { 0n. On, ................................ 00.00........................... 01-1,
= {x)}
The trajectory then represented
== (I, 0o Oo,
ar.,...,,,rr or) ~ Ta(T e).
Iterative Supervised Fine- Fine-Tuning in Agentic Task Iterative Supervised Fine-Tuning (SFT) is process of exploration and learning [47; 48; 49]. iteratively utilizes the correct responses from the actor model to enhance the model's problem-solving abilities. The process involves K iterations, each consisting of two steps: exploration and learning. For the collection of environments &, in the exploration step of iteration k. the model T k-1 from the previous iteration is applied to each environment e, resulting in |E| trajectories { These trajectories are then filtered using the reward function R(T), retaining only the correct ones. The filtered trajectories form new dataset Deorrect {- 1-1+ where N' is the number of trajectories retained. In the learning step of iteration k, this new dataset is used to fine-tune the actor model my obtain T
j=1--
4 Methodology
In this section, we first provide an overview of the proposed Critique-Guided Guided Improvement (CGI) framework ($ 4. 1), explaining how the actor and critic collaborate to enhance performance. We then detail the two key stages of CGI, critique generation and action refinement, which respectively address the challenges of critic's weak feedback and the actor's poor utilization. In the Critique Generation stage ($ 4.2), the critic model learns to evaluate the actor's 's candidate actions and provide actionable feedback. In the Action Refinement stage ($ 4.3), the actor refines its actions by integrating critiques from the critic model into its interaction with the environment. Algorithm summarizes the CGI framework (see Appendix C for definitions of all notifications).
4.1 Overview of the CGI Framework
The CGI framework follows a two-player setting, where the actor model wo generates multiple candidate actions, and the critic model ne provides feedback refine them. For a specific environment E ε, at each time step t, given the refined trajectory history
{2, d/1 .u...., ...
the actor generates M candidate actions, stored in the action buffer A,= (ar,i (at,i) i=1- The critic then
I.M.
evaluates each candidate and generates corresponding critiques c = [T T/. C4.;, e) Using these critiques, the actor refines decision and generates the final action af TT (a T Ct, e). which is then executed in the environment. This iterative process produces the refined trajectory:
=
T' {x, a/s 00, ... a',, oT).
In this collaborative framework, higher-quality critiques e and better utilization of feedback lead to improved refined actions at, ultimately enhancing the final performance R(T').
3

---

Critique Generation
Overview of CGI
Action Refinement
Target at Weak Feedback
Turget at Poor Utllization
located
[-]
Action: Go to bathroom
Critique:: fick up ther mometer
Critique: current
highly
Action: Look around. Obs: This room is called the kitchan. In you sear door bettwoom, therrmmmtee [-]
Thoughts have to go to buthroom to get close substaence
Critio
Critique
Action Buffer A
Collect Training Data by g
Al: Pick thermometer AB: kichhen
Dimention: Contrussion: [--].
Phoxcbiiitty [-]] Merolwnnyy
AZ: Focus on Substance
bathroom
Prefte
Input:
Proerect
Haltrr:
Took
Grade: (occalien (Goodd/ Poony PoonyVery Poor]
Critique e
Contribution: Agent the crucill step of picking the thermometer [.]
Crttiue:....
Outgru:
Suggested Revision: [..]]
Rwlined Action
Suggested Revision: The agent should Pirk up thermometer before attempting
Output:
Step Distilled Expert Critique
Gritic
bathroom.
Refined Action
Refined Action
Currect step:
Critique Utliization
Enhance Reasoning
Refined Action of
Expert Tra?
+
Thought: Based the Critquue, need to thermometer
Update Actor Pollcy
Step 2: Training Critic modell by
distilled critique Portique Hif Cp)1
Action:
temperature.
Pick up thermometer
Actor
Improve Actor Policy by Drefne Doorecct
Filttrd Drgfne Oearnest R
Figure 1: An overview of CGI, illustrated with ScienceWorlde example. The actor first generates candidate actions, and the critic then assesses each action and generates suggested revisions. The actor refines its actions based the critique. The Critique Generation stage enhances the critic model's iss ability to provide effective feedback, while the Action Refinement stage helps the actor model better utilize the critique.
4.2 Critique Generation
The Critique Generation stage trains the critic model to assess the alignment between LLM-- generated actions and optimal ones at each step, and convert this assessment into structured natural language critiques. We first define the critique structure, then outline the training method for critic.
Critique Structure The critique generated by the critic model n.. consists of two components: Discrimination, which involves analyzing and assessing the quality of the candidate action, and Revision, which provides actionable suggestions for improvement.
For the discrimination component, the critic model evaluates the candidate action in three predefined dimensions:
Assessing how well the candidate action contributes solving the overall task. Irrelevant actions receive a low score.
Contribution:
Feasibility: Determining whether the candidate action valid according to the agent's 's prede- fined action list.
Efficiency: Evaluating whether the candidate action approaches the task optimally, avoiding unnecessary steps or redundancy.
For the revision component, the critic model first assigns an overall grade to each candidate action (i.e., haal/Poor//eeeeee /Very Poor]), then generates concise and actionable suggestions based on its analysis of the candidate actions (see Appendix for detailed prompts).
Fine-tuning the Critic Model To collect high-quality step-level expert critiques, employ an expert critic annotator (c.g., GPT-4 [1]) to generate critiques for each candidate action based on a reference expert trajectory. For specific environment E E, at time step f, the actor generates candidate actions aj Given the refined history trajectory T/ and the expert trajectory 5 eex, the expert critic Texp assesses the alignment between candidate actions and the optimal ones, converting this assessment into structured critiques according to a predefined format. Formally, the critiques for each candidate action represented q. Texp (a, T(: -exp,e).
Subsequently, the actor refines next step by generating a' based on the critique et Critiques are collected only when R(7')= = ensuring they originate from correct trajectories. We collect step-level

---

Algorithm 1 Critique-Guided Improvement (CGI)
1: Input: Initialized policy actor model Tæ, critic model T.. environment set &, full instruction set
X, reward function R. the expert critiques D Deritique. the expert path Dexpert> the general dataset D Dgeneral, max time step T, and the iteration rounds K.
2: Procedure Fine-tune the Critic Model:
3: Minimize the following objective to obtain the critic model ng.
Leitii (Ø) = Bea ()e))))) log F (ce, Té az,
4: Procedure Iterative Action Refinement:
5: TT f- T
6: for iteration = 1 to do
7: Drrain + Dexpert: // Initialize Derain with Dexpert.
8: Perform Exploration Step
Actor = interacts with e under the guidance of critic nei
9:
// Collect critique-action pairs with R(r') =
Collect Drefine Unge Deefine where D Doffin {(T Ct af)
11:
// Update training set with correct trajectory.
Collect Dooreet = Ureg D coorectt Derain < Derain Deorrecti
13:
Perform Learning Step
14:
Minimize the following objective to obtain actor model mp:
Luctor (0) 3
{ (.(r.,e)DDen
log TT(T (z,e) ee) + [(aa) log lognola' T ( (()
+(1 (3 E (
log To [z)]:
15:
end for
expert critiques from each environment, forming the dataset D Peritiuee = =Uege Dr is then fine-tuned using supervised learning with the collected expert critiques. Specifically, we apply the standard language modeling loss, defined as:
critigus The critic model
Caritic ee) log To (ce Téi az, e)
This approach enables the critic model generate structured, step-level critiques that provide both discrimination and revision, which are crucial for guiding the actor model toward more effective decision-making.
4.3
Action Refinement
Although the critic model trained to provide high-quality critiques, the actor may not fully utilize the feedback. Therefore, action refinement is necessary to enhance the actor's ability to effectively leverage critiques for improving actions. key challenge this process is policy misalignment. During training, the actor model learns to incorporate critiques based on its current policy. However, after training, the model's policy may evolve, making it difficult to integrate critiques effcetively for newly generated candidate actions. This misalignment
lead to suboptimal action refinement.
To address this issue, we propose an iterative action refinement method based on supervised fine- tuning (SFT). This method consists of two main components: exploration and learning. In the exploration step at iteration k. for specific environment
E, the actor model T - interacts with the environment under the guidance of the critic model kg. At each time step t. the refined action is given by d: (a T Ct. e),e, e), forming set of critique-action pairs {(T(, C, d)) Here, T/ represents the previously refined trajectory, and denotes the current critique. To ensure the quality of the trajectories, we filter them based on the environment reward R. Only trajectories for which R(r') = 1, along with their corresponding tritique-action pairs, are retained. We collect correct trajectories and critique-action pairs from each environment, yielding two datasets: D Doomect- Uneg De De eorreet and Drefine Ungg Dr The dataset Deamet enhances the model's reasoning ability by providing correct trajectories, while Drefine improves its capacity to utilize critiques and generate refined actions.
5

---

In the learning step of iteration k; the newly collected datasets, combined with the expert path D Dexpent for each environment, are used to fine-tune the actor model тg- To avoid overfitting, we follow previous work [50] by fine-tuning the original model gy rather than the previous iteration model x k-1 Furthermore, following AgentTuning [51], we incorporate general datasets such as ShareGPT3' to improve generalization. The training objective is defined as:
Lactor (0) = 3 { (Trr,,)))DDm To r e) + Blchit log Te I ct e)}
+ (1 B) ((x
log T 1r)]
where Dunin = D Dexport U D Domect. After this refinement step, new dataset with higher-quality samples generated for further training. Overall, this iterative action refinement process allows the actor model to progressively improve its reasoning capabilities and better integrate critiques through continuous interaction with the environment.
Experiment Settings
In this section, we conduct extensive experiments in three interactive environments to demonstrate the effectiveness of the critic model and our CGI framework.
5.1 Interactive and Agentic Environments
Following previous work [40; 52], we conduct experiments on three types of representative interactive environments:
WebShop [11], which is an interactive web environment for online shopping. It contains 12K instructions and offers over one million real products from amazon.com com. Agents can click buttons on the webpage or perform searches using the search engine.
ScienceWorld [25], which a text-based scientific environment designed to evaluate agents' scientific reasoning abilities. It includes 30 types of scientific tasks at the standard elementary science curriculum level.
TextCraft [26], which a text-based environment to create Minecraft items. It constructs a crafting tree based Minecraft's recipes. Each task provides target item and list of crafting commands generated by the tree. Agents receive a reward of when they successfully craft the target item.
Evaluation Metries Following the setup of AgentGym [40], we evaluate our model on the test sets for these three environments (200 simulations for ScienceWorld and WebShop, 100 for TextCraft). For ScienceWorld and WebShop, we use the success rate as the evaluation metric. Further details can be found in Appendix E.
the average final score the evaluation metric. For TextCraft,
5.2 Training Settings
We use Llama-3-88 --8--Insttuct [3] as the backbone model for both the actor and critic models. To collect training data, we randomly sample 500 simulations from WebShop, 350 from ScienceWorld and 374 from TextCraft. We train the critic model by using the expert critic (i.e., GPT-4o) to guide the actor in interacting with the environment three times, collecting expert critiques during each interaction. For action refinement, we perform three iterations and report the results of the third iteration in Table 2. The training data for the critic model and each iteration of action refinement are provided in Table 4. Additional details can be found in Appendix F.
5.3 Baselines
To evaluate the effectiveness our critic model, we fix the actor model Miama-3-8B-Instruue [3]. We then compare our critic model against two types of approaches (see Appendix G for imple- mentation details): /) Numerical based: We use DGAP [14], discriminator trained assess the alignment between actor actions and expert actions at the step level, and Explicit RM [15], which
Pnttos://huggingface wtwp://gwww..e///ttat//////eeeee

---

Table We compared the natural language feedback provided by the trained critic model with other methods, including numerical-basd bbased (i.e., DGAP, Explict RMM, and verbal- baseel (i.e.. self-critique, GPT-4o). Our critic offers better guidance the actor model in three interactive scenarios.
Model
Method
WehShop
ScienceWorld
TextCraft
Average
Critique
13.49
14.48
10.00
12.65
DGAP
30.41
19.52
21.00
23.64
Explicit
14.21
18.41
15.87
Liumu-3-88 3-BB AB-ntust
Self-Critiqur
1.50
10.06
19.00
10.19
GPT-4o
17.78
33.06
46.00
32.28
Critic Model (Ours)
56.80
68.51
59.00
61.44
Critique
8.35
49.20
2,00
19.85
DGAP
11.17
55.86
9.00
25.34
Explicit RM
11.18
48.41
14.00
24.53
LJama-3-700
Self-Critique
3.00
28.17
40.00
23.72
GPT-40
16.24
43.73
56.00
38.65
Critic Model (Ours)
52.20
72.44
73.00
65.88
Critique
32.85
DGAP
73.97
38,26
56.00
56.08
Explicit RM
74.33
41.57
52.00
55,97
Liama-3-8B- B-a------BB----eee SFT
Self-Critique
50.18
31.22
21.00
34.13
GPT-4o
55.65
48.48
58.00
54,04
Critic Model (Ours)
74.68
55.94
56.00
62.21
is trained to predict the Q-value, i.e. expected accumulated rewards each time step. 2) Verbal based: We employ a self-critique method where the actor model itself generates critiques for each candidate action the step level. Additionally, which serves as a strong general-purpose evaluator. The structure of the critiques aligns with that in Appendix Each approach guides the actor's 's inference, and we evaluate their effectiveness based on the actor's 's performance".
use GPT-4o (gpt-40-2024-08-066 [1] as critic,
For the assessment of CGI, consistent with the methodology used in AgentGym [40], select combination of closed-source models, including GPT-3.5-turbo [53], GPT-4o [54], Claude [55], and DeepSeek-Chan Chat [56], well open-source models such Alama-3-70B-Insttre
[3]. Additionally, we include agents trained on expert trajectories, such as AgentL.M (13B and 70B) [57] and Agent- Flan [42]. We also compare our approach to Iterative SFT, which iteratively refines the actor model using correct trajectories collected from interactions with the environment, and Reflexion [20], self-reinforcemeet method that concludes each iteration with a summary to guide decision-making in subsequent iterations. We report the results from the third iteration for both approaches.
6
Main Results
We first evaluate our critic model against numerical- and verbal-based methods to assess its effective- ness. shown in Table 1, our critic model significantly outperforms GPT-4o on both the Llama-3 (8B and 70B) and Llama-3-8B's fine-tuned variant trained expert data. Furthermore, Table demonstrates that through iterative action refinement the actor model, our CGI notably enhances actor performance in interactive environments. It outperforms both advanced closed-source models (e.g., GPT-4o) and agents trained on expert trajectories (e.g., AgentLM-700 and Agent-FLAN). Based on these results, identify three key findings.
Finding 1: Verbal critique feedback is more effective than numerical signal. As shown in Table the Critic Model consistently outperforms numerical feedback from the discriminator. For the Llama-3-8B model, it achieves an average improvement of 42.89% over the no-critique baseline, while the discriminator yields only 5.09% gain. The advantage persists with the stronger Llama-3- 70B and the fine-tuned Llama-3-8B models, where the Critic Model delivers a 46.03% and 10.55% improvement, respectively, compared to 5.49% and 4.42% from discriminator. These results indicate that the discriminator's numerical feedback relies heavily on the model's 's inherent capabilities, as it essentially performs action-level filtering. For example, in the Science the discriminator enables Llama-3-70B to reach 55.86%, but only 18.52% for Llama-3-8B. In contrast, the Critic
In the subsequent experiments of this paper, we set the number of candidate actions sampled at inference
time M
7

---

Table 2: Results from three interactive environments. The CGI here employs two-player setting with a fine-tuned LLama-3-8B critic model and an 8B actor model refined through three iterations. We compare its performance with various models.
Method
WebShop
ScienceWorld
TextCraft
Average
Closed-source Models
SeepScek-Chat
11.00
16.80
23.00
16.93
Chaude-3HHaid
5.50
0.83
0.00
2.11
Cluude-3-Sonnel
1,50
2.78
38.00
14.09
GPT-3.5--Trro
7.64
47.00
22.38
GPT-4o
25.48
46.91
64,00
45.46
Open- source Models
Llama-3-70B-
1-70B-Instrrct
8,35
49.20
2.00
19.85
AgenLLM-13B
39.50
2.75
0.00
14.08
AgentLM-70B
10.68
4,00
21.39
Agent-FLAN
40.35
28.64
16.00
28.33
-lama-3--8--Innee Instructt
13.49
14.48
10.00
12.66
w/ Reflexion
14.08
12.55
8.00
11.54
lterative SFT
78.21
41.42
$5.00
58.21
w/ CGI (Ours)
76.17
78.43
68.00
74.20
Llama3 Critique
IterSFT
Reflexion
CGI
Average Final Score
Webshop
ScienceWorld
TextCraft
75
75
68:51
59.00
56.80
50
50
40
25
25
Vanilla #iter1 #Iter2
#iter3
Vani la
#iterl # Iter
#Iter3
Vanilla
#iterl
#
liter3
lteration
Iteration
Iteration
Figure 2: Performance of different iterative methods across iterations in three agentic tasks.
Model shows minimal sensitivity the base model, achieving 68.51% with Llama-3-8B and 72. 44. with Llama-3-70B. However, the self-critique approach, which also relies on verbal feedback, shows a negative effect on both Llama-3-70B, 8B and fine-tuned variant. This suggests that while verbal feedback can convey richer information, untrained self-critique methods often produce low-quality or unstructured feedback, failing to provide effective guidance.
Finding 2: Fine-tuned models struggle utilize critiques. While supervised fine-tuning signifi- cantly improves baseline performance (+33.11%) models struggle to effectively incorporate critique feedback. In the ScienceWorld scenario, Vanilla Llama3-8B and Llama3-70B show substantial improvements with Critic guidance, achieving average scores of 68.51 and 72.44%, respectively. However, despite having a stronger base performance, fine-tuned Llama3-8B only reaches 55. 55.94% with the same critique mechanism. This issue persists in the WebShop scenario, where critique guidance even causes a performance decline in fine-tuned Llama3-8B, dropping from 12% 74.68%. These results suggest that while fine-tuning enhances base performance, it may reduce the model's responsiveness to external feedback. For example, in the Science World scenario, despite the critic model advising the actor to "look around" and assess the environment for possible actions, the actor persists in executing "Go to Bathroom", an action that seems closest the task description. This behavior disregards the fact that the actor cannot reach the bathroom from its current position, leading to deadlock (see Appendix H. Our iterative action refinement approach helps mitigate this issue; after three iterations, the model shows significant improvement in critique utilization, with its performance surpassing that of the actor model with only supervised fine-tuning by +15.99% achieving state-of-ther of-the-rrt results compared to various baselines.
experiments reveal key limitation: fine-tuned
Finding 3: CGI continuously enhance model performance via action refinement. As shown in Figure 2, compared to other iterative methods (e.g., Reflexion and vanilla iterative SFT). CGI consistently supports model performance improvement. In contrast, Reflexion shows minimal improvement and can even cause performance degradation. This aligns with previous findings [20], where self-critique struggles to escape local minima in agentic tasks that require significant diversity and exploration. Iterative SFT achieves a notable improvement only the first iteration, with minimal progress in the following 2-3 iterations. This issue is more pronounced in long-horizon

---

#iterl
#lter2
#iter3
ScienceWorld
WebShop
TextCraft
Revision Ratio
80
60
+8.02
75
17..8
+9.12
50
40
40
25
20
1
3 Stage
4
4
Stage
Stage
Figure 3: Revision Ration of actor model at different trajectory stages across three tasks, #Iter-k
denotes the k-th iteration of action refinement. Revision Ratio represents the proportion of actions that differ after critique compared to those without critique.
tasks (e.g.. Science and TextCraft), suggesting that the model may over-sample simpler tasks while under-sane sampling more complex ones. Consequently, the training set for subsequent iterations becomes dominated by casier tasks, with few solutions for harder ones. As iterations progress, this bias deepens, leading to long-tail distribution where solutions to more challenging tasks are scarce, ultimately causing the model to plateau or even degrade. In contrast, CGI leverages high-quality critiques and enhances the actor's 's utilization, enabling the model to consistently sample solutions to more difficult tasks, thus supporting sustained performance improvement.
7 Further Analysis
7.1 Qualitative Analysis of CGI
In this section, we conduct a qualitative analysis examine how CGI enhances performance in agentic tasks. Specifically, we focus on two key aspects: 1) Trajectory-level: Identifying the stages in the trajectory where CGI provides the most significant improvement; 2) Task-level: Evaluating CGI's performance across tasks of varying trajectory length (see Appendix E for categorization).
CGI significantly improves etrly-stage performance. To determine at which stage of the trajec- tory CGI contributes most to performance improvements. we divide the trajectory into five stages based on its length and compute the extent to which the critique influences the actor model's behavior, measured by the Revision Ratio. As shown in Figure 3, across three tasks, actor models exhibit the highest action revision frequency in stage 1, with the revision ratio dropping sharply in later stages. This suggests that the critique primarily guides the actor during early exploration, helping reduce ineffective searches. Notably, as the actor undergoes more refinement iterations, its revision ratio in stage 1 increases (+8.02% indicating that better critique utilization accelerates effective exploration. This, in tum, enhances CGI's s efficiency, enabling it to achieve higher scores in fewer steps (Figure 8).
ScienceWorld +17.81 % on WebShop, +9.12% on Tex(Craft),
Vanilla
w/ Critique
#iter1
#iter2
#iter3
CGI helps the model to continuously improve long-horizon tasks. To investigate the effect of CGI on tasks of varying trajectory length, we categorize the ScienceWorke World scenarios into three difficult-level groups based on the average length of the oracle agent's trajec- tories. Longer trajectories correspond to higher diffi- culty levels. As shown in Figure 4, the vanilla model (Llama3-8B) performs poorly, with performance de- creasing the task length inereases. Critique-Guided Inference leads to significant improvement, particu- larly on easy tasks, where performance increases by +59.64%. Although there is also a notable improvement on harder tasks (+43.95%) the gain is less pronounced compared to easier tasks. However, with iterative action refinement, model exhibits the greatest improve- ment on hard tasks, with +28.75% increase after three iterations. These results suggest that models
Average Final Score
80
40
20
Easy
Medium
Hard
Figure 4: Performance of models across varying trajectory length. Vanilla refers to the Llama3-8B model, w/ Critique refers to the critique-guided inference with our trained Critic-Llamaa3, and #lter-k repre- sents the k-th iteration of action refinement.

---

without action refinement quickly adapt to critiques for easy and medium-level tasks. However, for longer tasks, the model's ability to leverage critiques diminishes. Action refinement helps the model continuously follow critiques, thereby enhancing performance on Long-horizon tasks.
7.2 Effects of Number of Candidate Actions
To further evaluate the scalability of CGI, we varied the number of candidate actions (M) sampled at each step during inference. As shown in Figure 5, we ob-
Self-Critique
DGAP
GPT-4
Critique
serve the following trends: CGI achieves significant improvement with just one candidate action, reaching an average score of 56.89%. As M increases, perfor- mance improves further, rising from 56.89% to 61.72%. However, when M 7, performance plateaus, indi- cating saturation. Other numerical- and verbal-based methods exhibit similar trends. For GPT-4 critic, performance increases with M, improving from 32.28% to 38.13% DGAP is particularly sensitive to the num- ber of candidate actions, with performance rising from 15.02% to 25.07%. This highlights that numerical meth- ods are highly dependent on the quality of the sampled candidate actions, as they only discriminate but not suggest meaningful revisions. In contrast, self- critique does not benefit from an increase in M. This suggests that low-quality critiques can degrade performance, regardless of the number of candidate actions.
Average Score
28
30
20
10
5
Candidate Actions (M)
Figure 5: Performance of different meth- ods across varying candidate actions. The average score is the mean final score across three environments.
8 Ablation Study
To analyze the contribution of the three types of data (i.e., Deureet, Drefine and 'D geeneral) used to enhance the ac- tor model during the action refinement stage, we conduct an ablation study. Specifically, we remove each dataset from the training data used the first iteration of action refinement and evaluate the final performance under the guidance of the critic model. As shown in Figure removing any one of the three datasets results in per- formance drop compared to the actor model trained with the full dataset (CGI #lter).. ). The most significant de- crease occurs when the critique-action pairs are removed (w/o Drefime), where the average performance across the three tasks drops from 68.50% to 50.37%. This decline is more pronounced for longer tasks, with performance reductions of 5.4% on WebShop, 22% on TexICraft, and 26.94% on ScilnceWorld. These results highlight that enhancing the actor's critique utilization ability is the most critical factor in improving overall performance. less severe drop is observed when the generalization data set is excluded (w/o D geaeerat), highlighting the importance of general instructions for model generalization, which helps the model adapt to the unseen test set.
CGlasar
w/o Deanee
wio Dudir
w/o Oggnnna
Average Final Score
80
600
40
20
Webshop
ScienceWorld
TextCraft
Figure 6: Ablation study of the three data types used in action refinement: Deorreet (correct trajectories to improve the actor's reasoning ability), Drelin (critique-action pairs to enhance critique utilization), and D geeeraal (data to maintain the model's gen- eralization capability). CGI #htrl refers to the actor model trained in the first iteration of action refinement using the full dataset.
Conclusion
We have introduced Critique- Criiiue-Guided Improvement (CGD), a two-player framework that emphasizes nature language feedback for iterative refinement of LLM-based agents. By separating the roles of an actor, which proposes actions, and a critic, which provides verbal guidance, CGI circumvents the limitations of purely numerical signals and addresses the challenges arising from self-refinement. Experimental results in three interactive and agentic environments confirm the effectiveness of CGI, with small critic model outperforming GPT-4 in providing feedback.
10

---

Acknowledgement
We appreciate the support from the Chinese NSF General Program (No.62572129), Major Research Plan (No.9227012)). We also acknowledge the of icon from Flaticon' and thank its creators for providing this visually appealing design.
References
[1] OpenAI. GPT-4 Technical Report, 2024.
[2] Gemini Team. Gemini: A family of highly capable multimodal models. Technical report, Google, 12 2023.
[3] Meta. Llama 3 model card. 2024.
[4] Zane Durante, Qiuyuan Huang, Naoki Wake, Ran Gong, Jae Sung Park, Bidipta Sarkar, Rohan Taori, Yusuke Noda, Demetri Terzopoulos, Yejin Choi, Katsushi Ikeuchi, Hoi Vo, Li Fei-Fei, and Jianfeng Gao. Agent AI: Surveying the Horizons of Multimodal Interaction, 2024.
[5] Jiangjie Chen, Xintao Wang, Rui Xu, Siyu Yuan, Yikai Zhang, Wei Shi, Jian Xie, Shuang Li, Ruihan Yang. Tinghui Zhu, Aili Chen, Nianqi Li, Lida Chen, Caiyu Hu, Siye Wu, Scott Ren, Ziquan Fu, and Yanghua Xiao. From Persona to Personalization: A Survey on Role-Playing Language Agents, 2024.
[6] Boyu Li, Haobin Jiang, Ziluo Ding, Xinrun Xu, Haoran Li, Dongbin Zhao, and Zongqing Lu. Selu: Self-learning embodied milms in unknown environments. arXiv preprint arXiv:2410.03303, 2024.
[7] Siyu Yuan, Kaitao Song, Jiangjie Chen, Xu Tan, Yongliang Shen, Ren Kan, Dongsheng Li, and Deqing Yang. EASYTOOL: Enhancing L..M-based Agents with Concise Tool Instruction, 2024.
[8] Yujia Qin, Shihao Liang Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. TooILLM: Facilitating Large Language Models Master 16000+ Real-world APIs, 2023.
[9] Yue Hu, Yuzhu Cai, Yaxin Du, Xinyu Zhu, Xiangrui Liu, Zijie Yu, Yuchen Hou, Shuo Tang, and Siheng Chen. Self-Evolving Multi-Agent Collaboration Networks for Software Development. In The Thirteenth International Conference on Learning Representations, 2025.
[10] Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. SWE-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, 2024.
[11] Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. WebShop: Towards Scalable Real-Worid Web Interaction with Grounded Language Agents, 2023.
[12] Pranav Putta, Edmund Mills, Naman Garg, Sumeet Motwani, Chelsea Finn, Divyansh Garg, and Rafael Rafailov. Agent Q: Advanced reasoning and learning for autonomous agents, 2024.
[13] Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-shene shehhsshephed:: Verify and reinforce LLMs step-by-step without human annotations. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 9439, Bangkok, Thailand, August 2024. Association for Computational Linguistics.
hhttps://www flaticon.com

---

[14] Haofu Qian, Chenjia Bai, Jiatao Zhang, Fei Wu, Wei Song, and Xuelong Li. Discriminator- Guided Embodied Planning for LLM Agent. In The Thirteenth International Conference on Learning Representations, 2025.
[15] Yu Xia, Jingru Fan, Weize Chen, Siyu Yan, Xin Cong, Zhong Zhang, Yaxi Lu, Yankai Lin, Zhiyuan Liu, and Maosong Sun. AgentRM: Enhancing agent generalization with reward modeling, 2025.
[16] Zhenfang Chen, Delin Chen, Rui Sun, Wenjun Liu, and Chuang Gan. Autonomous agents from automatic reward modeling and planning. In The Thirteenth International Conference on Learning Representations, 2025.
[17] Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback, 2022.
[18] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens. Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[19] Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uni Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36, 2024.
[20] Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language Agents with Verbal Reinforcement Learning, 2023.
[21] Ruihan Yang, Jiangjie Chen, Yikai Zhang, Siyu Yuan, Aili Chen, Kyle Richardson, Yanghua Xiao, and Deqing Yang. SELFGOAL: Your Language Agents Already Know How to Achieve High-level Goals. In NeurlPS 2024 Workshop on Open-World World Agents, 2024.
[22] Aviral Kumar, Vincent Zhuang, Rishabh Agarwal, Yi Su, John D Co-Reyes, Avi Singh, Kate Baumli, Shariq lqbal, Colton Bishop, Rebecca Roelofs, Lei M Zhang, Kay McKinney, Disha Shrivastava, Cosmin Paduraru, George Tucker, Doina Precup. Feryal Behbahani, and Aleksandra Faust. Training Language Models to Self- Correct via Reinforcement Learning. In The Thirteenth International Conference on Learning Representations, 2025.
[23] Alejandro Cuadron, Dacheng Li, Wenjie Ma, Xingyao Wang, Yichuan Wang, Siyuan Zhuang, Shu Liu, Luis Gaspar Schroeder, Tian Xia, Huanzhi Mao, Nicholas Thumiger, Aditya Desai, Ion Stoica, Ana Klimovic, Graham Neubig, and Joseph E. Gonzalez. The danger of overthinking: Examining the reasoning-action dilemma in agentic tasks, 2025.
[24] Yannick Metz, Andras Geiszl, Raphaël Baur, and Mennatallah El-Assady. Reward Learning from Multiple Feedback Types. In The Thirteenth International Conference on Learning Representations, 2025.
[25] Ruoyao Wang, Peter Jansen, Mare-Alexandre Côté, and Prithviraj Ammanabrolu. Science Is your Agent Smarter than a 5th Grader?, 2022.
[26] Archiki Prasad, Alexander Koller, Mareike Hartmann, Peter Clark, Ashish Sabharwal, Mohit Bansal, and Tushar Khot. ADaPT: As-Needed Decomposition and Planning with Language Models, 2024.
[27] Binghai Wang, Rui Zheng, Lu Chen, Zhiheng Xi, Wei Shen, Yuhao Zhou, Dong Yan, Tao Gui, Qi Zhang, and Xuanjing Huang. Reward modeling requires automatic adjustment based on data quality. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Findings of the Association for Computational Linguistics: EMNLP 2024, pages 4041 4041-4064, Miami, Florida, USA, November 2024. Association for Computational Linguistics.
12

---

(28) Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichhro Nakano, Christopher Hesse, and John Schulman. Training Verifiers to Solve Math Word Problems, 2021.
[29] Fei Yu, Anningzhe Gao, and Benyou Wang. OVM, outcome-supervise value models for planning mathematical reasoning. In Kevin Duh, Helena Gomez, and Steven Bethard, editors, Findings of the Association for Computational Linguisties: NAACL 2024, pages 858-872 -75, Mexico City, Mexico, June 2024. Association for Computational Linguisties.
[30] Liangchen Luo, Yinxiao Liu, Rosanne Liu, Samrat Phatale, Meiqi Guo, Harsh Lara, Yunxuan Li, Lei Shu, Yun Zhu, Lei Meng, Jiao Sun, and Abhinav Rastogi. Improve Mathematical Reasoning in Language Models by Automated Process Supervision, 2024.
[31] Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christo- pher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. WebGPT: Browser-assisted question-answwringg with human feedback, 2022.
[32] Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hermandez, Dawn Drain, Deep Ganguli. Dustin Li. Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer EI Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Conerly, Tom Henighan, Tristan Hume, Samuel Bowman, Zac Hatfield- -oodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. Constitutional AI: Harmlessness from AI Feedback, 2022.
[33] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing. Hao Zhang. Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-Judge with MT Bench and Chatbot Arena, 2023.
[34] Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang. Shashank Gupta, Bodhisatiwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self- Refine: Iterative Refinement with Self-Feedback, 2023.
[35] Shunyu Yao, Jeffrey Zhao, Dian Yu. Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. ReAct: Synergizing Reasoning and Acting in Language Models. In The Eleventh International Conference on Learning Representations, 20233.
[36] Yu Shang, Yu Li, Keyu Zhao, Likai Ma, Jiahe Liu, Fengli Xu, and Yong Li. AgentSquare: Automatic LLM Agent Search in Modular Design Space, 2025.
[37] Minghao Chen, Yihang Li, Yanting Yang, Shiyu Yu, Binbin Lin, and Xiaofei He. AutoManual: Constructing Instruction Manuals by LLM Agents via Interactive Environmental Learning, 2024.
[38] Bodhisattwa Prasad Majumder, Bhavana Dalvi Mishra, Peter Jansen, Oyvind Tafjord, Niket Tandon, Li Zhang, Chris Callison-Burch, and Peter Clark. CLIN: A Continually Learning Language Agent for Rapid Task Adaptation and Generalization, 2023.
[39] Kolby Nottingham, Bodhisattwa Prasad Majumder, Bhavana Dalvi Mishra, Sameer Singh, Peter Clark, and Roy Fox. Skill Set Optimization: Reinforcing Language Model Behavior via Transferable Skills. arXiv, 2024.
[40] Zhiheng Xi, Yiwen Ding, Wenxiang Chen, Boyang Hong, Honglin Guo, Junzhe Wang, Dingwen Yang, Chenyang Liao, Xin Guo, Wei He, Songyang Gao, Lu Chen, Rui Zheng, Yicheng Zou, Tao Gui, Qi Zhang, Xipeng Qiu, Xuanjing Huang, Zuxuan Wu, and Yu-Gang Jiang. AgentGym: Evolving Large Language Model-based Agents across Diverse Environments, 2024.
13

---

[41] Siyu Yuan, Zehui Chen, Zhiheng Xi, Junjie Ye, Zhengyin Du, and Jiecao Chen. Agent-R: Training Language Model Agents to Reflect via Iterative Self-Training, 2025.
[42] Zehui Chen, Kuikun Liu, Qiuchen Wang, Wenwei Zhang, Jiangning Liu, Dahua Lin, Kai Chen, and Feng Zhao. Agent-FLAN: Designing data and methods of effective agent tuning for large language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguistics: ACL 2024, pages 9354 9366, Bangkok, Thailand, August 2024, Association for Computational Linguistics.
[43]
Yifan Song, Weimin Xiong, Xiutian Zhao, Dawei Zhu, Wenhao Wu, Ke Wang, Cheng Li, Wei Peng, and Sujian Li. AgentBank: Towards generalized LLM agents via fine-tum tne-ttnnig on 50000+ interaction trajectories. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Findings of the Association for Compulational Linguistics: EMNLP 2024, pages 2124-2141, Miami, Florida, USA, November 2024. Association for Computational Linguistics.
[44] Wentao Shi, Mengqi Yuan, Junkang Wu, Qifan Wang. and Fuli Feng. Direct multi-turn preference optimization for language agents. Yaser Al-Onaizam, Mohit Bansal, and Yun- Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 2312-2324, Miami, Florida, USA, November 2024. Association for Computational Linguistics.
[45] Yifan Song, Da Yin, Xiang Yue, Jie Huang, Sujian Li, and Bill Yuchen Lin. Trial and Error: Exploration-Basee Trajectory Optimization for LLM Agents, 2024.
[46] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of Thoughts: Deliberate Problem Solving with Large Language Models. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[47] Renat Aksitov, Sobhan Miryoosefi, Zonglin Li, Daliang Li, Sheila Babayan, Kavya Kopparapu, Zachary Fisher, Ruiqi Guo, Sushant Prakash, Pranesh Srinivasan, Manzil Zaheer, Felix Yu, and Sanjiv Kumar. ReST meets ReAct: Self-Improvement for Multi-Step Reasoning LLM Agent, 2023.
[48] Jiaxin Huang, Shixiang Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. Large language models self-improve. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 1051-1068, Singapore, December 2023. Association for Computational Linguistics.
[49] Yuxiao Qu, Tianjun Zhang, Naman Garg, and Aviral Kumar. Recursive introspection: Teaching language model agents how to self-improve, 2024.
[50] Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah D. Goodman. STaR: Bootstrapping Reasoning With Reasoning, 2022.
[51] Aohan Zeng, Mingdao Liu, Rui Lu, Bowen Wang, Xiao Liu, Yuxiao Dong, and Jie Tang. AgentTuning: Enabling Generalized Agent Abilities for LLMs, 2023.
[52] Archiki Prasad, Alexander Koller, Mareike Hartmann, Peter Clark, Ashish Sabharwal, Mohit Bansal, and Tushar Khot. ADaPT: As-needed decomposition and planning with language models. In Kevin Duh, Helena Gomez, and Steven Bethard, editors, Findings of the Association for Computational Linguistics: NAACL 2024, pages 4226-4252, Mexico City, Mexico, June 2024. Association for Computational Linguistics.
[53] OpenAl. Chatgpt blog post. 06.
2022, Accessed: 2024-09-
[54] OpenAL. Gpt-4 technical report, 2023.
[55] Anthropic. Introducing claude 2.1, Nov 2023. Available from Anthropic: https://www. anthropic.com hwwp:///..ccco//////aaaee-e---
14

---

[56] DeepSeek-Al, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongji Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui J. L. Cai, Jian Liang, Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jiawei Wang, Jin Chen, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, Junxiao Song. Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang. Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Litong Wang, Liyue Zhang, Meng Li, Miaojun Wang. Mingchuan Zhang, Minghua Zhang. Minghui Tang. Mingming Li, Ning Tian, Panpan Huang, Peiyi Wang, Peng Zhang, Qiancheng Wang. Qihao Zhu, Qinyu Chen, Qiushi Du, R. Chen, R. I.. Jin, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang. Runxin Xu, Ruoyu Zhang, Ruyi Chen, S. S. Li, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaoqing Wu, Shengfeng Ye, Shengfeng Ye, Shirong Ma, Shiyu Wang, Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Shuting Pan. T. Wang, Tao Yun, Tian Pei, Tianyu Sun, W. L. Xiao, Wangding Zeng, Wanjia Zhao, Wei An, Wen Liu, Wenfeng Liang Wenjun Gao, Wenqin Yu, Wentao Zhang, X. Q. Li. Xiangyue Jin, Xianzu Wang, Xiao Bi, Xiaodong Liu, Xiaohan Wang, Xiaojin Shen, Xiaokang Chen, Xiaokang Zhang. Xiaosha Chen, Xiaotao Nie, Xiaowen Sun, Xiaoxiang Wang, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu. Xingkai Yu, Xinnan Song, Xinxia Shan, Xinyi Zhou, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, Y. K. Li, Y. Q. Wang, Y. X. Wei, X.X. Zhu, Yang Zhang, Yanhong Xu, Yanhong Xu, Yanping Huang, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Li, Yaohui Wang, Yi Yu, Zheng, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Ying Tang, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yu Wu, Yuan Ou, Yuchen Zhu, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yukun Zha, Yunfan Xiong, Yunxian Ma. Yuting Yan, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Z. F. Wu, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhen Huang, Zhen Zhang, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhibin Gou, Zhicheng Ma, Zhigang Yan, Zhihong Shao, Zhipeng Xu, Zhiyu Wu, Zhongyu Zhang, Zhuoshu Li, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Ziyi Gao, and Zizheng Pan. Deepseek-v3 technical report, 2025.
[57] Aohan Zeng. Mingdao Liu, Rui Lu, Bowen Wang. Xiao Liu, Yuxiao Dong, and Jie Tang. AgentTuning: Enabling generalized agent abilities for LLMs. In Lun-Wot Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguistics: ACL 2024, pages 3053-3077, Bangkok, Thailand, August 2024. Association for Computational Linguistics.
[58] Diederik P Kingma. Adam: A method for stochastic optimization. arXiv:1412.6980, 2014.
arXiv preprint
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Abstract and Section 1
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Appendix A
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
16

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced
statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Section 3, Section and Appendix D, E,
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
17

---

Answer: [Yes]
Justification: Section 4
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
public/ public/guides/Code Sutieee//CoeSSumiisiiiiiii
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (ht / //nis. cc/ppc/ http//ullcc/cc ctgii//iiiiie gigguidess/// des/CooeSubmi ssiossiiois ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Section 4, Section 5 and Appendix F Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification:
We set temperature to 0 for all models in experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
18

---

It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative enror rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Appendix E
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: Section 5, Appendix and F
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Appendix D
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
19

---

generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:eecionn 5, Appendix and F
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
20

---

Answer: [Yes]
Justification: Section 5, Appendix E and F Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
21

---

Answer: [No]
Justification:
The paper does not use LLMs for writing, editing, or formatting.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer our LLM policy /tttts::/neerpps..... / for what should or should not be described.
22

---

Appendix
A Limitations
CGI incurs higher computational time compared to baseline methods but remains within reasonable range. As shown in Table 3, CGI requires approximately four times the computational time of the baseline. However, this additional cost results in substantial performance improvement, with CGI achieving 43.31% % gain over No Critique. This indicates that the extra computational time is effectively utilized. In contrast, other verbal-based methods, such GPT-4o and Self-Critique, do not produce significant improvements and require more computational time than CGI.
B Broader Impacts
Positive Societal Impacts. By generating and leveraging natural language critiques, CGI enhances the performance in embodied Al tasks (e-g.. Science World) and real-world scenarios (e.g.. Web- Shop), leading to improved task performance. This contributes to enabling agents to better understand and align with human goals, ultimately supporting more helpful, interpretable AI systems. Moreover, our critic model demonstrates strong performance despite using relatively small backbone (Llama- 3-8B), which significantly reduces the computational and environmental costs compared to larger proprietary models. This also promotes broader accessibility to advanced feedback mechanisms.
Negative Societal Impacts and Risks. Natural-language critiques may inadvertently incorporate biases from the training data or expose private information through the agent's s reasoning process. Moreover, as these critiques provide explicit, step-by-step explanations, they could be misused by bad actors to facilitate disinformation, social engineering, or the generation of unsafe code. The relatively small size of the critic model further reduces the cost of replication and misuse, increasing the associated risks.
C
Notations
Dataset Definitions
Symbol
Meaning
Deriiique
Expert critique dataset generated by expert critic (GPT-40)) used for training the critic model as descrihed in $4.2
Doorness
Correct trajectory dataset collected during each iteration of the action refinement process
D Dreline
Critique-action pairs dataset used for learning how utilize critiques effectively
D general
General conversational dataset (e.g., ShareGPT) used to maintain general language modeling capabilities during fine-tuning
Loss Function Parameters
Symbol
Meaning
Castur
Actor model loss function with three components: learning from expert/correct trajec- tories, learning to utilize critiques, and maintaining general capabilities
8
Weight parameter balancing agentic task learning vs. general capabilities (typically 3 = 0.8)
y
General responses from datasets like ShareGPT for maintaining language modeling abilities
23

---

D Instruction Prompt Examples
The system prompts for three agentic environments are presented in Listing 1. The instruction for critique generation introduced in $4.2 is presented in Listing 2.
Listing 1: Prompt details for ScienceWorld, WebShop, and TextCraft.
ScienceMorld Instruction:
You are an agent for the science world. Every round I will give you observation, you have respond with action based the observation
to finish the given task.
Here are the actions you may take:
("action": "open/close OBJ", "description": "open/close container",, ("action": "de/activate OBJ", "description": activate/ddacttivart device",)
("action"; "connect OBJ to 08J", "description": "connect electrical components",
("action": "disconnect OBJ", "description": "disconnect electrical
components",
("action": "use OBJ [on 0BJ]", "description": "use a device/item".)
("action": "look around", "description": "describe the current room",
("action": "look at OBJ", "description": "describe an object in detail", ("action": "look in OBJ", "description": "describe container's contents ".)
("action": "read OBJ", "description": "read a note or book",
("action": "move OBJ to OBI", "description": "move an object to
container"
("action": "pick up OBJ", "description": "move object the inventory
("action": "put down OBJ", "description": "drop an inventory item",) ("action": "pour OBJ into 08J", "description"; "pour liquid into container"
("action": "dunk liquid",
OBJ", "description": "dunk container into a
("action": "mix OBJ", "description": "chemically mix container",,
("action": "go to LOC", "description": "move to new location", ("action": "eat 0BJ", "description": "eat food".)
("action": "flush OBJ", "description": "flush toilet",)
("action": "focus on OBJ", "description": "signal intent on a task object ",}
("action": "wait", "description": "take action for 10 iterations",
("action": "wait1", "description": "take action for iteration", ("action": "task", "description": "describe current task",) ("action": "inventory", "description": "list your inventory") Your response should use the following format:
Thought: your thoughts.
Action: your next action
WebShop Instruction:
You web shopping. will give you instructions about what to do. You have to follow the instructions. Every round will give you an observation and a list of available actions, you have to respond an action based on the state and instruction.
You can use search action if search is available. You can click one of action should be of the following structure
the buttons in clickables. search click[value].
the action is not valid, perform
nothing. Keywords in search are up to you, but the value in click must be value in the list of available actions. Remember that your keywords in search should be carefully designed. Your response should the following format:
Thought: I think
Action: click[something] search[somethingg

---

TextCraft Instruction:
You are given a few useful crafting recipes to craft items in Minecraft. Crafting commands are of the format "craft [target object] using [input ingredients]". Every round will give you observation, you have to respond to an action based on the state and instruction. You can "get" object (ingredients) from the inventory or the environment, look up the game "inventory" by inventory, or "craft" (target) using any of the crafting commands. You can use ONLY these crafting commands provided, do not your crafting commands. However, if the crafting command a generic ingredient like "planks", you can use special types the same ingredient e.g. dark oak "planks" in the command instead. Your response should use the following format:
Thought:
Action:
Listing The instruction prompts for Critique Generation.
Critique Generation Instruction:
Your task to critique candidate 's next step action based on agent's task goal and interaction history.
[available_acctons)
Critique Steps
Step 1: Analyze Candidate Action
Examine the candidate 's action based the following criteria and assign an overall grade using this scale: Excellent, Good, Neutral, Poor, Very Poor
Critique Dimensions
Contribution: Assess whether the action contributes to completing the agent task. This includes both direct actions (e.g., picking up the target 0BJ) and indirect actions (e.g., reasonable exploration that can provide additional environmental information and facilitate future progress).
Feasibility: Assess whether the action is valid according the agent predefined Allowed Action Types list. Efficiency: Analyze whether the action optimally achieves the task without unnecessary steps or redundancy.
Step Provide Revision Suggestions
Suggest modification to align the candidate action better with task or the agent's 's action capabilities. Note that the suggested revision should be based on the Allowed Action and Object Types.
Critique Format
Please structure your critique in the following format:
## Contribution: [Analysis of Contribution].
## Feasibility: [Analysis of feasibility].
## Efficiency: [Analysis of efficiencyl.
Overall Grading: [Overall grade: Excellent/Good/ httpt//Gooo///erraa///oo// /Very Poor
].
## Suggested Revision: [Brief revision suggestion, if applicable].
Inputs:
The agent's task goal and interaction history:
(history)
Candidate next step action: (candidate_action)
Now, please provide your critique:
25

---

Critique Generation Instruction (with expert path):
Your task is to critique the candidate next-step action based on the agent's task goal and interaction history. The gold path for current task is provided as reference to guide your critique, but do not explicitly mention it in your critique.
[available__cctoos)]
Critique Steps
Step 1: Analyze Candidate Action Examine the candidate 's action based on the following criteria, then assign overall grade using this scale: Excellent, Good, Neutral, Poor, Very Poor
Critique Dimensions
-Contribution: Assess whether the action contributes to completing the agent's 's task. This includes both direct actions (e.g., picking up the target OBJ) and indirect actions (e.g., reasonable exploration that provide additional environmental information and facilitate future progress).
-Feasibilityy Assess whether the action is valid according to the agent's s predefined Allowed Action Types list. -Efficiency: Analyze whether the action optimally achieves the task without unnecessary steps or redundancy.
Step 2: Provide Revision Suggestions
Suggest modification align the candidate's 's action better with the task or the agent's action capabilities. For example, if the action not allowed, recommend an alternative from the action list that aligns better with the task goal.
Critique Format
Please structure your critique in the following format:
4# Contribution: [Analysis of Contribution].
## Feasibility: [Analysis of feasibilityl.
## Efficiency: [Analysis of efficiency].
Overall Grading: [Overall grade: Excellent/Good// Neettel/ /oo /Very Poor
J.
## Suggested Revision: [Brief revision suggestion, if applicable].
Referenced Gold Path for Current Task:
(gold_path)
Inputs:
The agent's task goal and interaction history:
(history)
Candidate next step action: (candidate_actiio)
Now, please provide your critique:
26

---

Table 3: Computational efficiency of different methods in three agentic environments per task.
WebShop
Sciworld
Textcraft
Method
Inference Time
Performance
Inference Time
Performance
Inference Time Performance
No Critique
0.38 min
13.49
1.49 min
14.48
1.02 min
10.00
DGAP
1.13 min
30.41
3.13 min
19.52
min
21.00
w/ Self-Critique
1.36 min
1.50
3.42min
10.06
2.17 min
19.00
w/ GPT-4o
2.34 min
17.78
6.28 min
33.06
3.28 min
46,00
w/ Critic Model
1.22 min
56.80
3.41 min
68.51
1.98 min
59.00
E Evaluation Settings
In the experiments presented in this paper, used three agentic environments: WebShop, SciWorld, and TextCraft.
E.1 Details of Environments
WebShop is interactive web environment designed for web shopping. In this environ- ment, agents are given instructions and must purchase product that meets specified criteria. Agents either click button the webpage performance using the success rate, with a maximum of 10 rounds per task.
WebShop
the search engine to find the product. We evaluate
ScrenceWorld ScienceWorld is benchmark enviromment for testing agents' scientific reasoning abilities, based on a standard elementary science curriculum. It includes 30 types of tasks, such using measurement instruments and conducting mechanics experiments. The action space is task-specific, with the environment simulator providing the effects of actions. We use reward as the evaluation metric, with maximum of 30 rounds per task. Task lengths classified based the Science World environment, which provides gold-standard trajectories from 30 hand-coded oracle agents. We adopt the following categorization: Short (11.76), Medium (28.58), and Long (94.30). These lengths correspond to the average number of steps taken by the oracle agent.
TextCraft TextCraft text-only environment for crafting Minecraft items. It constructs a crafting tree based on Minecraft's crafting recipes, consisting of 544 nodes, each representing target item. For each task, the agent given target item and list of crafting commands generated by the tree. Tasks vary in complexity, ranging from 1 to 4 steps. The environment supports three valid actions:"crafi <item> using <ingredients>", "get <item>". and "inventory"". After each round, the environment checks the agent's actions and returns the execution state. Agents are rewarded with point only upon successfully crafting the target item. We evaluate performance using the success rate, with maximum of 20 rounds per task.
E.2 Computational Efficiency Analysis
We evaluated the computational efficiency of CGI by conducting experiments in three agentic environments, using Llama-3-8B as the actor model. We measured the average inference time per task. As shown in Table 3, although CGI incurred higher computation times compared inference without critique guidance (i.e., No Critique), these times remained within an acceptable range. Furthermore, CGI demonstrated substantial reduction in inference time compared to using GPT-4 as the critic, while achieving significant performance improvements.
27

---

F
Implementation Details
In our experiments, we use the Llama-3-8B-Insteeee [3] as the base model for both the actor and the critic.
E1 Training Data
The training data for the critic model consists of expert critiques generated by the expert critic (i.e., GPT-4o) in the Science Webshop, and TexICraft environments, as described in $ 4.2. The specific training set sizes during the SFT phases for Llama3-8B-Instrus are 14K from ScienceWorld, 10K from Webshop, and 8K from TextCraft. Examples of our SFT training data are provided below:
Critique Generation
SFT Example:
("coovvrsations".
frofr"m": "human", "value": "Your task is to critique the candidate next-step action based on the agent 's task goal and interaction history. Allowed Action Types:- search[KENWORDD]: perform web search click[oB] link
webpage. Step 1: Analyze Candidate Action Examine the candidate 's action based on the following criteria and assign an overall grade using this scale: Excellent, Good, Neutral, Poor, Very Poor. Critique Dimensions: Contribution: Assess whether the action contributes to completing the agent's task. This includes both direct actions and indirect actions. Feasibility: Assess whether the action is valid according to the agent's predefined action list. Specifically, check if the action is part of the Allowed Action Types, whether the OBJ/ KKEYOODDS within the action is clearly specified, and whether it is the system's requirements. Efficiency:
]: click object
formatted correctly according Analyze whether the action optimally achieves the task without unnecessary steps or redundancy. Step 2: Provide Revision Suggestionn: Suggest modification to align the candidate action better with the task the agent's action capabilities. For example, if the action is not allowed, recommend an alternative from the action list that aligns better with the task goal. Critique Format: structure your critique in the following format ## Contribution [Analysis of Contribution]. ## Feasibility: [Analysis of feasibility]. ## Efficiency: [Analysis of efffiiincy]. ## Overall Grading: [Overall grade: Excellent/Good /Neutrra// Suggested Revision: [Brief revision suggestion, applicablel. #u* InputsThe Agent's Task Goal and Interaction History:
Poor//erry Poor].
WebShop [SEP] Instruction: [SEP] Find me machine wash, moisture wicking
men's t-shirts tanks with polyester spandex [SEP] Search Candidate Next
Step Action: search[machine wash moisture wicking
t-shirts with
polyester spandex long sleeve red blue XXL price: under 50]. Now, please provide your critique:"),
("from": "gpt", "value": "## Contribution: The candidate's 's action contributes to the task by attempting to perform search for the specified product. It includes most of the necessary attributes such as material, sleeve length, color, size, and price, which are crucial for finding the desired item. # Feasibility: The action is feasible as it However, there are some
uses the allowed action type "search[[EvwoRDS]]] discrepancies in the keywords used. The candidate action uses "XXL" instead of "xx-large" and "price: under 50" instead "price 50.00". While these variations might still yield relevant results, they do not exactly match the specified format. #s Efficiency: The action is relatively efficient as it attempts to gather relevant search results in keyword formatting could
one step. However, the slight variations potentially lead to less precise search results, which might require additional steps to refine the search. ## Overall Grading: Good. ## Suggested Revision: Modify the search keywords to exactly match the specified format:
wash moisture wicking men's 's t-shirts & tanks polyester spandex long sleeve red blue size xx-large price 00]. This ensures consistency and potentially more accurate search results. ")],
"system": ""}}
28

---

Table 4: Training dataset size for each iteration of Action Refinement stage.
Gold
#Iter
#Iter 2
#Iter 3
Env
Path
Critique
Path
Critique
Path
Critique
Path
Critique
ScienceWorld
378
4106
133
1464
158
1803
193
2205
WebShop
427
1659
106
663
283
1637
344
2010
TextCract
441
1701
233
1423
310
2070
340
2049
OverÅll
1246
6740
472
3550
751
5510
877
6264
Action Refinement As described in $ 4.3, we performed action refinement three times. Each training dataset consists of two parts: Correct Path, which includes the correct trajectories obtained by the actor from interactions with the environment in the previous iteration; and Correct Critique, which includes the <critique, action> pairs that the actor correctly followed during the previous iteration. The dataset size for each iteration of Llama-3-88 ---B----B---. 8--Instruct detailed in Table
F.2 Finetuning Details
We ran SFT experiments using 8 NVIDIA A100-40GB GPUs. For action refinement, to prevent policy drift, we only trained the base model (i.e., Llama-3-SB-Instruct) each time. We conduct experiments with the LlamaFactory code base". The configurations of our hyper-parameters are detailed Table 5.
Table 5: Fine-tuming hyper-pppreeeee parametera for Critique Generation and Action Reinement stage.
Configuration
Critique Generation
Action Refinement
Model
Llama-3 1eaaa------Insntuu Instructt
Llama-3-88 3aama---BB-Instruuee Instruct
Number of epochs
3
3
Devices
A100 GPU (40 GB)
8 A100 GPU (40 GB)
Total Batch size Optimizer
64 samples
64 samples
Adam [58]
Adam [58]
(8, = 0.9, B2 = 0.98, = x 10-8)
(31 = 0.9, 82 = 0.98, E = 1 80
Learning rate
2 10-5
10
Warmup Ratio
0.05
0.05
Cutoff Length
4096
4096
Training Time
5h 46m 24s
lh 1 1l 55s
F.3
Effect of Iteration Count
Vanilla
#iter1
#iter2
#Iter3
#iter4
Average Final Score
Here we analyze the impact of the number of it- erations on the performance of CGL. As shown in Figure 7, the fourth iteration leads to performance drop CcienceWorld and TextCraft, while the im- provement in WebShop is marginal (+0.67%). The third iteration achieves the highest average perfor- mance across all tasks. These results suggest that ad- ditional iterations not yield further gains. There- fore, we report the results from the third iteration in main results section ($ 6).
80
60
20
WebShop
SciWorld
TextCraft
Figure 7: Performance models with varying iteration counts across three tasks.
F.4 Effect of Training with Expert Trajectories and Updating the Critic
In this section, we analyze two key factors: 1) whether our critic heavily depends on expert trajectories; 2) whether updating the critic model through iterations can continue improve performance. To evaluate the first factor, we replace the gold-standard expert trajectories with HDTT-4o-generate
29

---

Table 6: Performance comparison across different methods and datasets
Method
WebShop
Science
TextCraft
Average
Llama3-88 8B
13.49
14.48
10.00
12.66
w/ 4PT-40
17.78
33,06
46.00
32.28
w/ Critic (Trained on GPT-4o Traj)
43.21
56.93
52.00
50.71
Critic (Trained on Expert Traj)
56.80
68.51
59.00
61.44
w/ Updated Critic
62.51
74.56
64.00
67.02
trajectories across three environments, following the exact pipeline described in $4.2 for training the critic model. For the second factor, we perform iterative updates on the critic model after action refinement to adapt to the updated actor's policy. Specifically, we use the critic from the first iteration to generate new critique data and retrain the critic.
As shown in Table 6, CGI remains highly effective even without gold-standard expert data. Our method achieves an average performance of 50.71, significantly outperforming both the baseline and the GPT-4o model that generated the initial data. This highlights the robustness of our framework. Additionally, the critic can be dynamically updated. The results indicate that updating the critic leads further performance improvements (from 61 14.44% 67 37.02%), demonstrating that the critic is not limited to being static and can evolve alongside the actor.
G Baselines
G.1 Critique Methods
In Section 6, we compare our trained Critic-Llama3 model with other verbal- and numerical- -based methods. For the verbal-based approaches, such self-critique and GPT-4, generate critiques by prompting the model with the instructions provided in Appendix D. For the numerical method (i.e., DGAP), following [14], compute the cosine similarity between the actor model (Llama-3-8B) and expert data at each step. We collected 23K data points for Scienceworld, 16K for WebShop, and 12K for TextCraft. Examples of the reward model training data are provided below:
DGAP Example:
("input": "Your task is to measure the melting point of lead, which is located around the kitchen. 10. look around. Action: to hallway",
Score": "10"}
("input": "Your task is to
the melting point of lead, which is
located around the kitchen. 10. look around. Action: look at art studio", "Score": "ø")
("input": "Your task is to measure the melting point of lead, which is located around the kitchen. 18. look around. 9. go to hallway. Action: put down orange", "Score": "9"}
("input"; "Your task is to measure the melting point of lead, which is
located around the kitchen. 10. look around. Action: look at hallway", Score": "9.03"}
("input": "Your task is to measure the melting point of lead, which is located around the kitchen. 10. look around. 9. look at hallway. Action: open door to outside", "Score": "6.13")
("input"; "Your task is to measure the melting point of lead, which is located around the kitchen. 18. look around. 9. look hallway. 8. open door to outside. Action: teleport kitchen". "Score": "8.87")
G.2 Iterative Methods
In Section we compare CGI with two iterative baselines: Reflexion and Iterative SFT. Following [20], we prompt the model for self-reflection (Listing 3) at the end of each iteration. The insights gained from current round are then incorporated into the system prompt to guide the model's decision-making in the subsequent round. For Iterative SFT, we use the actor model nz from the previous round to interact with the environment. The newly collected correct trajectories are combined with the expert path to form the training data for the next round. Specifically, for iterations to 3,
30

---

used 1,676, 1,719, and 1,750 trajectories, respectively, mixing them with the general dataset (i.e., ShareGPT) at a 1:4 ratio for training. To prevent policy drift, we update To at each iteration. Each iteration consists of epochs. The learning rate for Iterative SFT is and cosine scheduler. We use the AdamW optimizer with a weight decay of 0 and apply a maximum gradient clipping norm to prevent gradient explosion. The batch size is set per device, with sequence length of 4,096 and an accumulation count of 4, adjusted based on sequence parallelism.
2c-5, with a 3% warm-up
Listing 3: The instruction prompt for Reflexion.
Reflexion Instruction:
Your task is to provide next plan of action based the previous attempt s interaction history. The next plan should reflect the lessons learned from the failure and adjust for better results.
Previous Trial: (history)
Next Plan:
H Case Study
H.1
Failure Cases of Fine-tuned Llama3
We provide negative examples where the fine-tuned Llama3-8B does not follow the critique.
Listing 4: Failure Cases of Fine-Tuned Llama3-8B in ScienceWorld
Task Description:
the temperature of unknown substance B, which is located around the bathroom. First, focus on the thermometer. Next, focus on the unknown substance B. If the unknown substance В temperature is above 100.0 degrees celsius, place it in the orange box. If the unknown substance temperature is below 100.0 degrees celsius, place it in the yellow box. The boxes located around the bathroom.
Your task is to
History:
Action: open door to kitchen
Observation: The door is now open.
Action: go kitchen
Observation: You move the kitchen.
Action: open door to hallway
Observation: The door is now open
Action: go to hallway
observation: You move the hallway. Critique:
### Feasibility: The candidate action "open door to bathroom" is efficient in terms of task progression, it directly addresses the need to access the bathroom. However, the agent should consider checking the current state first to ensure the action is feasible by "Look Around".
### Overall Grading: Netual
Action: go bathroom
Observation: No known action matches that input.
H.2 Critic-Llama3 Outperforms GPT-4o
We provide the following examples to explain why our critic outperforms GPT-4o when the actor model is fixed as Llama3-8B. Compared GPT-4o, our critic demonstrates superior spatial reasoning and higher degree of goal-directed behavior.
Listing 5: Comparison between Critic-Llama? and GPT-4o as Crities
Example 1: Spatial Reasoning
31

---

#CGI Iterl
#CGI Iter2
CGI Iter3
Vanilla
4-4
3-1
3-2
3-3
4-2
4-3
Figure 8: Visualizing trajectories CGI, and base model (Llama-3-8B-Instte (BB-Innstucct), the x-axis repre- sents time steps, ranging from 0 T, while the y-axis denotes scores, which vary from 0 100. Each graph illustrates the trajectories corresponding to different tasks in test variation. Task identifiers are positioned at the bottom right of each figure.
State: Agent is in the workshop. Task requires an object the bathroom. Action: focus on thermometer
GPT-4o (Incorrect): Overall Grading: Excellent.
It approves the action based the task description, ignoring the agent location.
Our Critic (Correct): Overall Grading: Very Poor
It correctly identifies the thermometer is not in the current room and suggests navigating first.
Example 2: Goal-Directedness
Task: Find living thing.
Action: move to the closet (an exploratory but inefficient action). GPT-4o (Inefficient): Overall Grading: Neutral. It allows this exploratory step.
Our Critic (Efficient): Overall Grading: Poor
It rejects the inefficient action and suggests a more direct one, like opening the door to explore other rooms.
H.3 Trajectory Visualization of ScienceWorld
We visualize the cumulative scores of the actor performing different numbers of action refinements (#lter k) and the base model in the ScienceWorld scene". As shown Figure 8, our CGI method shows notable improvement in efficiency. achieving higher scores in fewer steps. Furthermore, as the number of action refinements increases, efficiency also improves.
Detaaledd information of each task can be found in https:: https///gttuubb.cclaaeeaaaa////s
32


Reviews:
[{"review_id": "ef1XwABpIl", "reviewer": "Reviewer_5w5J", "summary": "This paper propose CGI a two-player framework consisting of an actor module, which explores and takes actions in an environment, and a critic module, which provides natural language critique that provides action quality judgement and feedback for revision. The critique is distilled from a larger model GPT4 by selecting trajectories containing actions and critiques leading to successful outcome. The actor seems to be trained by jointly optimizing the probability of the policy taking improved actions given observations and revising proposed actions incorporating the critique. Results showed that CGI robustly outperforms other critique methods and iterative-SFT-based method alone in three different tasks (ScienceWorld, WebShop, TextCraft). Further analysis suggest that the critic usually takes effect early on during actors' explorations.", "questions": "- Does table 1 result come from the 3rd iteration as Table 2 and 4?\n- $L_{actor}$ term is quite unclear, and does not track with the previous text. Please fix it so it's more clear.\n- minor suggestion: change “critique” to CGI to avoid confusion in FIgure 5.", "limitations": "No, the author did not talk about any limitations of this work and i have mentioned two in the Weaknesses.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n- As the field is moving towards building more agentic capabilities in open-ended domains where verification is non-trivial, this work provide a way to improve LLMs' ability to provide and improve upon natural language (NL) feedback is timely.\n- The results are very solid. Comparison with bases for both critic and actor components in three non-trivial domains are provided, and the results showed the two-player framework led to significant and robust improvements across different models.\n- I appreciate the further analysis section provided in section 6, it is meaningful to see how and where the most of the gains comes from. Also interesting to see a mostly positive scaling of performance w.r.t number of candidate actions. Although seems to decrease a bit after 7.\n\nWeaknesses:\n- while I find most of the paper clear, I have a hard time following the method section, particularly the part describing how the actor is trained. Per state of this paper, there is not enough info to reimplement the method. Here are a list of my confusions:\n  - there is multiple versions of datasets D mentioned, but not all of them are clearly defined (D_general) whereas some annotations are confusing D_critique (but it's from the expert)\n  - $L_{actor}$ formula does not reflect description btw L132 and L143, and also $y$ and $\\beta$ are not defined anywhere in text\n- this general direction is important but there's a few weaknesses in CGI:\n  - the existence of expert critic is required, which limits the ceiling of the critic performance, and also will not apply to domains where cheap expert critic is hard to find,\n  - the critic stays static during the actor improvement phase, while realistically it should evolve alongside the actor policy changes. However, the current implementation prevents that.", "was_revised": true, "final_justification": "The original draft proposed a novel idea with strong results, although the writing in certain sections, particularly in method section, are not clear. The confusion I had regarding to the draft is clarified during the rebuttal, hence, I am raising my score.", "rebuttal": "Thank you for the positive feedback on the timeliness of our work and the robustness of our results. We address the concerns about clarity and the critic model below.\n\n>  **Weakness 1:** Method clarity, undefined symbols ($D_{general}$, $D_{critique}$, $y$, $\\beta$), and unclear actor loss $L_{actor}$.\n\nThank you for pointing out the lack of clarity. We will add a comprehensive symbol table to the appendix and clarify all definitions in the main text.\n\n**Dataset Definitions**\n\n| Symbol |Meaning|\n|---------|---------|\n| $\\mathcal{D}_{\\text{critique}}$ | Expert critique dataset generated by expert critic (GPT-4o), used for training the critic model as described in Section 3.2 |\n| $\\mathcal{D}_{\\text{correct}}$ | Correct trajectory dataset collected during each iteration of the action refinement process |\n| $\\mathcal{D}_{\\text{refine}}$ | Critique-action pairs dataset used for learning how to utilize critiques effectively |\n| $\\mathcal{D}_{\\text{general}}$ | General conversational dataset (e.g., ShareGPT) used to maintain general language modeling capabilities during fine-tuning |\n\n**Loss Function Parameters**\n\n| Symbol | Meaning |\n|---------|---------|\n| $\\mathcal{L}_{\\text{actor}}$ | Actor model loss function with three components: learning from expert/correct trajectories, learning to utilize critiques, and maintaining general capabilities |\n| $\\beta$ | Weight parameter balancing agentic task learning vs. general capabilities (typically β = 0.8) |\n| $y$ | General text responses from datasets like ShareGPT for maintaining language modeling abilities |\n\nWe will revise Lines 132-143 to ensure the mathematical formulation aligns with the textual description and add these definitions explicitly in the main text.\n\n\n> **Weakness 2:** The expert critic requirement limits the performance ceiling, and the critic is static.\n\nThis is an excellent point. Our framework has two key strengths here:\n\n1. **The expert does NOT create a performance ceiling.** As shown in Table 1, our fine-tuned Llama-3-8B critic **already surpasses the GPT-4o expert by a large margin (+29.16%)**. This demonstrates that our method distills and refines the expert's knowledge, rather than being capped by it.\n2. **The critic can be dynamically updated.** Following your insightful suggestion, we ran a **new experiment** with an alternating critic-actor update scheme. We used the first iteration's critic to generate new critique data and retrained the critic.\n\n**Results on ScienceWorld and TextCraft:**\n\n| Method | ScienceWorld | TextCraft |\n|:--------|:--------------:|:-----------:|\n| Llama-3-8B (baseline) | 14.48 | 10.00 |\n| w/ Static Critic | 72.91 | 62.00 |\n| w/ Updated Critic | 74.56 | 65.00 |\n\nAs the results show, an updated critic yields further performance gains. This confirms the critic is not limited to being static and can evolve with the actor. We will add these results and the methodology to the final paper.\n\n**Questions:**\n\n> Does Table 1 result come from the 3rd iteration as Table 2 and 4?\n\n**No.** Table 1 is designed to isolate and evaluate the quality of the critic model itself. It shows the performance of a base Llama-3-8B actor guided by different critics, without any iterative actor refinement. This is in contrast to Tables 2 and 4, which show the full CGI framework with iterative actor improvement. We will state this explicitly in the caption of Table 1 to make this distinction clear.\n> Minor suggestion: change \"critique\" to CGI to avoid confusion in Figure 5.\n\nThank you for the suggestion. We agree this improves clarity and will update Figure 5 in the revised version."}, {"review_id": "9lfARPDCE1", "reviewer": "Reviewer_jS2Z", "summary": "This paper introduces Critique-Guided Improvement (CGI), a novel two-player framework designed to enhance the performance of large language model (LLM)-based agents. The framework consists of an actor model that interacts with an environment and a critic model that generates detailed, natural language feedback. In the Critique Generation stage, the critic model provides precise evaluations and actionable suggestions. In the Action Refinement stage, the actor model learns to integrate this feedback through iterative fine-tuning, improving both reasoning and decision-making capabilities.", "questions": "Please refer to the weaknesses section. Specifically: 1. It is suggested that the authors provide a detailed explanation of how the environmental reward (R) is calculated, particularly whether it only considers task completion or also includes factors such as action efficiency, path optimization, etc. (Is there a critique-based reward mechanism?). 2. Could the authors clarify the model’s handling mechanism when encountering incorrect candidate actions, especially how it backtracks and corrects erroneous actions? 3. The numerical definition of long-term tasks and ablation studies on action refinement should also be added in the experimental section to help further validate the model's performance across different tasks and iterations.", "limitations": "Although the paper demonstrates the effectiveness of the CGI method, it lacks a discussion of its limitations or potential issues in real-world applications. For example, can the CGI method effectively handle extreme situations or environments with highly complex tasks? What about the model's scalability and computational complexity?", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths\n\nThe paper introduces a novel Critique-Guided Improvement (CGI) method, which significantly enhances the model's performance in complex interactive tasks by combining a critique model with action refinement. The structure of the paper is clear, logically rigorous, and easy to understand. Each section is thoroughly developed, and the transitions between sections are smooth, improving the overall readability of the paper.\n\nWeaknesses\n\n1、Ambiguity in the Definition of Environmental Reward: The paper does not clearly explain the specific calculation of environmental reward (R), particularly whether the reward is solely based on task completion or if it also considers factors such as action efficiency, path optimization, and other aspects.\n\n2、Lack of a Structured Reward Function: In Section 3.2, the paper discusses the structure of the critique model and feedback generation, but it does not further address how to utilize a structured reward function to optimize and standardize the critique generation process. If a format-based reward function were introduced, would the generated feedback be more standardized? It is suggested to provide further clarification on this.\n\n3、Inadequate Handling of Incorrect Candidate Actions: The paper focuses on how to improve the critique model's matching of candidate actions with ideal \"annotated answers,\" but it does not address how the model should handle cases where no accurate candidate actions are generated. How should the model deal with incorrect actions? How should backtracking corrections be applied? It is suggested to provide more clarification on this aspect.\n\n4、In the experimental section, although the paper mentions that task difficulty is related to trajectory length, it does not provide a clear numerical threshold or boundary to define long-term and short-term tasks.\n\n5、While action refinement is shown to improve the model's performance, there is a lack of clear ablation comparison experiments. It is recommended to include relevant explanations or experiments.", "was_revised": true, "final_justification": "I have read the rebuttal and the comments of the other reviewers. My final rating is borderline reject.", "rebuttal": "> **Weakness 1:** Ambiguity in the Definition of Environmental Reward (R).\n\nThank you for asking for clarification. The environmental reward `R` is defined **solely by task completion**, following the standard setup of the AgentGym benchmark [1] we use.\n\n- **ScienceWorld**: A score from 0 to 100 based on sub-task completion.\n- **TextCraft & WebShop**: A binary reward (1 for success, 0 for failure).\n\nOur reward function does not include factors like efficiency or path length. This is a deliberate choice to ensure a fair and direct comparison with prior work on these standard benchmarks. We will add this explicit definition to Section 4.1.\n\n[1] AgentGym: Evolving Large Language Model-based Agents across Diverse Environments\n\n> **Weaknesses 2&3:** Lack of a structured reward function and inadequate handling of incorrect candidate actions.\n\nWe respectfully clarify that our framework explicitly handles both of these points.\n\n1. **Structured Critiques:** As described in Section 3.2, our critiques are **highly structured**. The critic is trained to generate feedback across five dimensions: **Contribution, Feasibility, Efficiency, Overall Grading, and a Suggested Revision**. We enforce this structure by filtering the training data, ensuring the critic learns to produce standardized and effective feedback.\n2. **Handling Incorrect Actions:** Our framework is designed for the common scenario where **all candidate actions might be incorrect.** The critic's role is not just to pick the best action, but to provide a `Suggested Revision` that guides the actor toward a better, potentially entirely new action.\n\nTo make this concrete, we will add the following example to the appendix, where all initial actions are flawed:\n\n- **Task:** Use a thermometer in the bathroom.\n- **State:** The agent is in the kitchen.\n- **Bad Candidate Action:** `focus on thermometer` (it's in another room).\n- **Our Critic's Feedback:** `Overall Grading: Very Poor. Suggested Revision: The agent should first 'open door to hallway' to begin navigating toward the bathroom...`\n\nThis shows how CGI provides actionable guidance even when the initial proposals are unworkable.\n\n\n>**Weakness 4** In the experimental section, although the paper mentions that task difficulty is related to trajectory length, it does not provide a clear numerical threshold or boundary to define long-term and short-term tasks.\n\nThank you for requesting this clarification. In our experiments, task length classification is based directly on the **ScienceWorld** environment [1], which provides gold-standard trajectories from 30 hand-coded oracle agents. Following prior work [2, 3], we adopt the established categorization:\n\n| Task Type | *Len |\n|-----------|---------------------------|\n| **Easy** |              11.76 |\n| **Medium** |         28.58 |\n| **Hard** |              94.30 |\n\nThese lengths correspond to the **average number of steps** taken by the oracle agent for each task difficulty level. This classification enables us to analyze how CGI performs across varying task complexities, as shown in Figure 4. We will include this detailed information in Section 4.1 and ensure consistent terminology throughout the paper.\n\n[1] ScienceWorld: Is Your Agent Smarter than a 5th Grader? *EMNLP 2022*\n\n[2] SWIFTSAGE: A Generative Agent with Fast and Slow Thinking for Complex Interactive Tasks. *NeurIPS 2023*  \n\n[3] Discriminator-Guided Embodied Planning for LLM Agent. *ICLR 2025*\n\n\n> **Weakness 5:** While action refinement is shown to improve the model's performance, there is a lack of clear ablation comparison experiments. It is recommended to include relevant explanations or experiments.\n\nThank you for raising this concern. We want to clarify that we have conducted comprehensive ablation studies, which are presented in the appendix:\n\n**Ablation Study 1 - Data Component Analysis (Appendix H)**: We conducted an ablation study on the contribution of three types of data (`D_correct`, `D_refine`, and `D_general`) used to enhance the actor model during action refinement. Our results confirm that critique-action pairs (`D_refine`) provide the most significant contribution to performance improvement.\n\n**Ablation Study 2 - Iteration Count Analysis (Appendix F.3)**: We analyzed the effect of different numbers of action refinement iterations, demonstrating that three iterations achieve optimal performance while a fourth iteration provides no additional benefit.\n\nWe believe these ablation studies provide sufficient evidence for the effectiveness of our approach. If you have specific aspects you would like us to investigate further, we would be happy to include additional ablation experiments.\n\n**Limitations**\n\n> Although the paper demonstrates the effectiveness of the CGI method, it lacks a discussion of its limitations or potential issues in real-world applications. \n\nThank you for this point. We analyze computational cost in Appendix E.2, showing that CGI provides a +43.31% performance gain for a ~4x inference time cost, a much more favorable trade-off than using a larger model like GPT-4o (+4.29% gain for a 7x time cost). While we selected complex, long-horizon environments to approximate real-world challenges, we agree that scaling to fully open-ended tasks is a key direction for future work. We will add a dedicated Limitations section to discuss this."}, {"review_id": "JHNnd0Q6vP", "reviewer": "Reviewer_3ZoY", "summary": "This work proposes Critique-Guided Improvement (CGI), a two-player framework that upgrades large-language-model agents with explicit, step-level natural-language feedback. An actor model first samples several candidate actions; a separately trained critic model then issues structured critiques that rate each action’s contribution, feasibility, and efficiency and includes concrete revision instructions. The actor is iteratively fine-tuned to incorporate these critiques, eliminating the brittleness of purely numerical rewards and the low quality of self-generated feedback. Using Llama-3-8B for both roles, CGI is evaluated on WebShop, ScienceWorld, and TextCraft. The 8-B critic outperforms GPT-4 as a feedback source, and after three refinement iterations the CGI agent achieves state-of-the-art results, surpassing strong closed- and open-source baselines and demonstrating especially large gains on long-horizon and hard tasks.", "questions": "Would the generated natural language feedback be still useful to powerful models that already have a good baseline performance?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strength:\n1. The paper focuses on critique-guided improvement, which is an important perspective to enhance the reflection capabilities of LLMs\n2. The performance is great! Table 1 shows that the critic model outperforms the no-critique baseline by 40%+, and GPT-4o by 30%+.\n\nWeakness:\n1. In the essential, the idea of using feedback to improve LLMs is not entirely new.\n2. There lacks analysis on why the proposed critic model can significantly outperform GPT-4o. Maybe two or three examples would make it more clear.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for highlighting the importance of our work's perspective and its \"great\" performance.\n\n> **Weakness 1:** The idea of using feedback to improve LLMs is not entirely new.\n\nWe agree that feedback-based improvement is an established direction. However, as we state in Lines 26-32, our novelty lies in **how we generate and utilize this feedback**. Prior work has struggled with two key challenges that our CGI framework is designed to solve:\n\n1. **Low-Quality Feedback:** Relying on self-correction [2,3] or simple numerical rewards [1] provides limited guidance. Our key contribution is a specialized, trained critic that generates **high-quality, structured, and actionable natural language critiques.**\n2. **Poor Utilization:** Agents often struggle to effectively integrate verbal feedback. Our second key contribution is the **iterative action refinement stage**, which explicitly fine-tunes the actor to understand and apply these critiques.\n\nOur work is the first to systematically address both the generation and utilization of high-quality language feedback in a unified framework.\n\n[1] Discriminator-guided embodied planning for LLM agent. ICLR 2025\n\n[2] Self-refine: Iterative refinement with self-feedback. NeurIPS 2024\n\n[3] Reflexion: Language agents with verbal reinforcement learning. NeurIPS 2023\n\n\n> **Weakness 2:** There lacks analysis on why the proposed critic model can significantly outperform GPT-4o.\n\nThis is an excellent question. Our fine-tuned critic excels because it becomes a specialist for the task domain. Compared to the generalist GPT-4o, our critic develops superior **spatial reasoning** and **goal-directedness**. We will add the following examples to the appendix to illustrate this.\n\n- **Example 1 (Spatial Reasoning):**\n    - **State:** Agent is in the `workshop`. Task requires an object in the `bathroom`.\n    - **Action:** `focus on thermometer`\n    - **GPT-4o (Incorrect):** `Overall Grading: Excellent`. It approves the action based on the task description, ignoring the agent's location.\n    - **Our Critic (Correct):** `Overall Grading: Very Poor`. It correctly identifies the thermometer is not in the current room and suggests navigating first.\n- **Example 2 (Goal-Directedness):**\n    - **Task:** Find a living thing.\n    - **Action:** `move to the closet` (an exploratory but inefficient action).\n    - **GPT-4o (Inefficient):** `Overall Grading: Neutral`. It allows this exploratory step.\n    - **Our Critic (Efficient):** `Overall Grading: Poor`. It rejects the inefficient action and suggests a more direct one, like opening the door to explore other rooms.\n\n> **Q1:** Would the generated natural language feedback be still useful to powerful models that already have a good baseline performance?\n\nThank you for pointing this out. To address this concern, we conducted experiments using powerful actor models (GPT-4o and Llama3-70B) guided by our critic model. The results are shown below:\n\n| Model | ScienceWorld | WebShop |\n|:-------|:--------------:|:---------:|\n| **Llama3-70B** | 49.20 | 8.35 |\n| **w/ Critic** | 72.44 | 52.20 |\n| **GPT-4o** | 46.91 | 25.48 |\n| **w/ Critic** | 75.91 | 58.59 |\n\n**Key findings:**\n- **Llama3-70B** shows substantial improvements: +23.24 points on ScienceWorld and +43.85 points on WebShop\n- **GPT-4o** also benefits significantly: +29.00 points on ScienceWorld and +33.11 points on WebShop\n\nThese results demonstrate that even powerful models with strong baseline performance can benefit from our critic's guidance. We will add these results to Table 1 in the final version."}, {"review_id": "HxwyRJxbM1", "reviewer": "Reviewer_v7iq", "summary": "The authors propose to improve an LLM agent using a two-player approach. First, they pre-train an LLM critic to evaluate the actions of an LLM actor and suggest refinements to its actions. Then, they fine-tune the LLM actor so that it proposes better actions next time. They compare their approach to 7 LLM training baselines and 9 zero-shot models in three textual environments and provide a few side analyses and ablations.", "questions": "- The proposed framework is very close to SELU [1], apart from the fact that SELU addresses multimodal LLMs rather than purely text-based ones. In SELU they also have a critic-guided improvement mechanism.\n[1]:Li, B., Jiang, H., Ding, Z., Xu, X., Li, H., Zhao, D., & Lu, Z. (2024). SELU: Self-Learning Embodied MLLMs in Unknown Environments. arXiv preprint arXiv:2410.03303.\n\n- The impact of initializing the training dataset with expert trajectories is not studied. An ablation where this initialization is removed is missing, as the expert paths may play a key role in the good performance of the method, but the need for collecting such expert paths is a key weakness of the approach. Can the authors provide such an ablation in their rebuttal?\n\n- To insist on this potential weakness, if the authors have a way to generate expert trajectories, why learn another agent that will perform the same trajectories? Is it just a matter of distilling into a smaller size agent?\n\n- The results presented in Tables 1 and 2 seem to result from a single run for each condition. Is there some variability in the results if one performs the same experiment several times? Is the experiments are too costly to be run several times, could the authors measure the variability on a small subset of the setup?\n\n- The comparison between verbal feedback and numerical feedback is potentially very interesting but the proposed study does not adequately focus on this specific question, as it might also be the critic accuracy that is different between DGAP, Explicit RM and CGI, and not only the feedback modality.\n\n- Expanding on the above point, isn't the point that CGI proposes (potentially not proposed by the actor) action refinements and the actor is fine-tuned on the difference to these actions, whereas in DGAP and Explicit RM, the actor will be fine-tuned on the difference to the best-ranked action among the ones proposed by the actor?\n\n*More local points*\n\n- line 117: why not also train the agents from refined versions of failed trajectories? How is performance influenced by this design choice of only training on successful trajectories?\n\n- line 145: To avoid overfitting, we follow previous work [29] by fine-tuning the original model πθ rather than the previous iteration model... -> Does this mean that the datasets are incremented at each iteration with the content of the previous iterations, or does CGI purely forget all improvements from the previous iterations?\n\n- did the authors make sure that trajectories from the test set are never seen in the training set?\n\n- \"our 8B critic model significantly outperforms GPT-4o on both the Llama-3 (8B and 70B) and Llama-3-8B’s fine-tuned variant trained on expert data.\" -> the sentence does not make sense to me.\n\n- would it make sense to merge Tables 1 and 2 into a single table?\n\n- line 275 : are the authors sure that longer trajectories are always the most difficult? It is not the case in general...\n\n- putting all related work in the Appendix is a bad practice\n\n*Typos:*\n- In Fig. 1: Dimentions -> Dimensions\n- Lama-3-8B-Instruct and Lama-3-Instruct-8B: please use uniform naming\n- \"(200 for ScienceWord and Webshop, 100 for textCraft)\" -> 200 simulations\n- in Table 1 \"Critic Model (Ours)\" -> CGI (Ours)\n- Figure 4 should not appear before Fig. 3.\n- caption Fig.3 \"Ration\" -> Ratio\n- in references, Agentrm -> AgentRM (use \"{AgentRM}\" and check all papers)", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- the method seems to work well in practice, outperforming a large number of baselines in 3 representative environments\n- the extra computational cost seems counter-balanced by the significant gain in performance\n\nWeaknesses:\n- some very related work is missing (see below)\n- a key ablation is missing (see below)\n- several points in the methods and empirical study are questionable (see below)", "was_revised": true, "final_justification": "The rebuttal process helped a lot clarifying what the authors are truly doing and why their method improves over just fine-tuning with an expert model. I now believe that the work is good, but I see a risk that the authors do not leverage the clarification discussion to write a much clearer paper. Hence my borderline accept.", "rebuttal": "> **Weakness 1:** The proposed framework is very close to SELU [1], apart from the fact that SELU addresses multimodal LLMs rather than purely text-based ones. In SELU they also have a critic-guided improvement mechanism.\nThank you for pointing out this related work. We would like to clarify the key differences between our approach and SELU[1]:\n\nThank you for bringing this concurrent work to our attention. While there are high-level similarities, CGI and SELU differ in fundamental ways:\n\n- **Feedback Type:** SELU's critic provides **trajectory-level binary feedback** (`{yes, no}`). In contrast, CGI's critic provides **step-level, rich, structured linguistic feedback** (grading, feasibility, revision suggestions). This is a core design difference.\n- **Critic Objective:** SELU's critic is a simple classifier to filter trajectories for training data. Our critic is a generative model trained to produce **actionable guidance**.\n- **Problem Setting:** SELU is designed for settings without expert data and works by simplifying tasks when the agent fails. Our work focuses on demonstrating that high-quality language feedback itself is a powerful mechanism for improving performance on complex, fixed tasks.\n\nWe will add a detailed discussion of SELU to our related work section.\n\n> **Weakness 2:** A key ablation is missing: removing the initialization with expert trajectories.\n\nThis is a critical point about the reliance on expert data. We have conducted the requested ablation study by replacing AgentGym's gold paths with GPT-4o generated trajectories on ScienceWorld, following the exact pipeline described in Section 3.2 to construct $D_{critique}$ for training the critic model.\n\n| Critic Training Data | Performance |\n| --- |:---:|\n| Llama3-8B (baseline) | 14.48 |\n| GPT-4o (for comparison) | 46.91 |\n| **CGI w/ Critic (trained on GPT-4o paths)** | **56.93** |\n| CGI w/ Critic (trained on expert paths) | 68.51 |\n\nThe results are clear: **CGI remains highly effective even without gold-standard expert data.** Our method still achieves a performance of 56.93, significantly outperforming both the baseline and the GPT-4o model that generated the initial data. This shows the robustness of our framework. We will add this to the appendix.\n\n>**Weakness 3:** To insist on this potential weakness, if the authors have a way to generate expert trajectories, why learn another agent that will perform the same trajectories? Is it just a matter of distilling into a smaller size agent?\n\nThis is not just distillation. The expert trajectories from AgentGym [2] were collected via many expensive sampling runs with GPT-4o. Our framework achieves two things:\n\n1. **Superior Performance:** Our final CGI agent (72.91 on ScienceWorld) **significantly outperforms the expert model (GPT-4o at 46.91)** used to generate the initial data.\n2. **Efficiency:** Our framework uses much smaller models (Llama-3-8B), making it far more efficient at inference time than running a large, proprietary model like GPT-4o.\n\n> **Weakness 4:** The results presented in Tables 1 and 2 seem to result from a single run for each condition. Is there some variability in the results if one performs the same experiment several times?\n\nOur main results use greedy decoding (temperature=0), which is deterministic. However, to address your valid concern about stochasticity, we ran a **new experiment** on ScienceWorld and WebShop with temperature=1 for 3 separate runs.\n\n| Environment | Mean ± Std |\n| --- |:---:|\n| ScienceWorld | 78.17 ± 1.33 |\n| WebShop | 76.17 ± 1.82 |\n\nThe standard deviation is very low, confirming that **our method is robust and its performance is stable across runs.** We will add this analysis to the appendix.\n\n**Weakness 5: Verbal vs. numerical feedback comparison**\n> The comparison between verbal feedback and numerical feedback is potentially very interesting but the proposed study does not adequately focus on this specific question, as it might also be the critic accuracy that is different between DGAP, Explicit RM and CGI, and not only the feedback modality.\n\nThank you for this insightful comment. As mentioned in Appendix G.1, we ensured fair comparison by using the same number of gold paths for training CGI, DGAP, and ExplicitRM. Our goal is precisely to compare different critic capabilities, which is why we selected two major categories: numerical methods (DGAP, Explicit RM) and verbal methods (CGI, self-refine), while keeping the actor model fixed as Llama3-8B.\n\n> Expanding on the above point, isn't the point that CGI proposes (potentially not proposed by the actor) action refinements and the actor is fine-tuned on the difference to these actions, whereas in DGAP and Explicit RM, the actor will be fine-tuned on the difference to the best-ranked action among the ones proposed by the actor?\n\nWe would like to clarify the process: For CGI, the actor samples 5 candidate actions, the critic provides critique for each action, and the actor takes the next step based on the critic's feedback. For numerical methods, we also sample 5 candidate actions, and the critic selects the best-ranked action for the next step. Both approaches are completely aligned in terms of sampling count and critic evaluation frequency.\n\n\n>**Other Points:**\n- **Training on Failed Trajectories:** This is an interesting direction. Our current design focuses on learning from success to ensure the quality of the critic's training data. Learning from failure is a complementary research problem, as seen in methods like SELU that modify the task itself. We believe our focus on high-quality critique generation is a valid and important contribution.\n- **Fine-tuning Original Model (`πθ`):** As shown in Appendix F.1, Table 4, we use three types of data to train the actor at each iteration: (1) correct paths (mixture of $D_{gold}$ and $D_{correct}$ from model-environment interaction), (2) $D_{refine}$ (critic-action pairs from model-environment interaction), and (3) $D_{general}$ (general datasets to maintain general capabilities). As model capability improves, $D_{correct}$ and $D_{refine}$ increase, while $D_{gold}$ and $D_{general}$ remain constant.\n- **Train/Test Split:** **Yes**, we strictly followed the official splits from AgentGym [2]. There is no data leakage.\n- **Confusing Sentence:** We will rephrase to: \"Our CGI approach, using an 8B critic, significantly boosts the performance of various actor models (including Llama-3-8B, Llama-3-70B, and fine-tuned variants) far beyond what GPT-4o can achieve alone.\"\n- **Longer Trajectories:** We will rephrase the analysis in Section 6.1 to be about \"trajectory length\" rather than \"difficulty\" to be more precise.\n- **Related Work & Typos:** We will move the related work to the main body and fix all typos you kindly pointed out.\nWe initially did this due to space constraints. In the final version, we will move the Related Work section to the main text."}]

Author rebuttal:

Meta-review: This paper introduces Critique-Guided Improvement (CGI), a two-player actor–critic framework where the critic provides structured natural language feedback and the actor is iteratively fine-tuned to incorporate it. Evaluated on ScienceWorld, WebShop, and TextCraft, CGI achieves strong gains, even surpassing GPT-4o as a feedback source.

Strengths: The work tackles an important direction for LLM agents, and demonstrates convincing improvements across three benchmarks. The use of structured critiques and iterative refinement is well-motivated and clearly impactful. Reviewers appreciated the thorough ablations, scaling analyses, and additional experiments provided in rebuttal.

Weaknesses: The main concern across reviews is clarity. Several features (e.g., ablations, structured critique design, explanation for critic > GPT-4o) were already in the submission but buried in the appendix or insufficiently emphasized. This led to confusion about novelty and completeness. Reviewers also noted overlap with related work (e.g., SELU), though rebuttal clarifications distinguished CGI’s contributions.

After rebuttal, most concerns were resolved, and residual issues are presentation-related.

Final Recommendation: Accept. The work is technically solid and impactful, but the final version must integrate rebuttal clarifications and highlight key details (datasets, ablations, inference-time benefits of the critic) in the main text for clarity and reproducibility.
Decision: Accept (poster)