Enhancing Decision-Making of Large Language Models via Actor-Critic
Authors: Heng Dong
Kefei Duan
Chongjie Zhang
Venue: ICML 2025 poster (2025)
Keywords: Large Language Models
Decision-Making
Actor-Critic
Primary area: deep_learning->large_language_models
TL;DR: We propose a novel LLM-based Actor-Critic framework that enhances LLMs' decision-making through long-term action evaluations and efficient policy improvements

Abstract: Large Language Models (LLMs) have achieved remarkable advancements in natural language processing tasks, yet they encounter challenges in complex decision-making scenarios that require long-term reasoning and alignment with high-level objectives. Existing methods either rely on short-term auto-regressive action generation or face limitations in accurately simulating rollouts and assessing outcomes, leading to sub-optimal decisions. This paper introduces a novel LLM-based Actor-Critic framework, termed LAC, that effectively improves LLM policies with long-term action evaluations in a principled and scalable way. Our approach addresses two key challenges: (1) extracting robust action evaluations by computing Q-values via token logits associated with positive/negative outcomes, enhanced by future trajectory rollouts and reasoning; and (2) enabling efficient policy improvement through a gradient-free mechanism. Experiments across diverse environments -- including high-level decision-making (ALFWorld), low-level action spaces (BabyAI-Text), and large action spaces (WebShop) -- demonstrate the framework’s generality and superiority over state-of-the-art methods. Notably, our approach achieves competitive performance using 7B/8B parameter LLMs, even outperforming baseline methods employing GPT-4 in complex tasks. These results underscore the potential of integrating structured policy optimization with LLMs’ intrinsic knowledge to advance decision-making capabilities in multi-step environments.

Enhancing Decision-Making of Large Language Models via Actor-Critic
Heng Dong" Kefei Duan
*2
Chongjie Zhang? ?
Abstract
1. Introduction
Large Language Models (LLMs) (Touvron et al., 2023; Jiang et al., 2023; Team al., 2024) have demonstrated impressive capabilities across various natural language pro- cessing tasks, including text generation (Li et al., 2024a), question answering (Li et al., 2024b), and summarization (Jin et al., 2024). The successful application of LLMs in these areas, coupled with their extensive internal knowl- edge, has generated significant interest in leveraging LLMs to tackle complex derision-makingg problems, particularly in data-scarce environments.
Large Language Models (LLMs) have achieved remarkable advancements in natural language processing tasks. yet they encounter challenges in complex decision-making scenarios that re- quire long-term reasoning and alignment with high-level objectives. Existing methods either rely on short-term auto-regressive action genera- tion face limitations in accurately simulating rollouts and assessing outcomes, leading to sub- optimal decisions. This paper introduces a novel LLM-based Actor-Critic framework, termed LAC, that effectively improves LLM policies with long- term action evaluations principled and scal- able way. Our approach addresses two key challenges: (1) extracting robust action evalu- ations by computing Q-values via token logits associated with positive/negative outcomes, en- hanced by future trajectory rollouts and reason- ing; and (2) enabling efficient policy improve- ment through a gradient-free mechanism. Ex- periments across diverse environments includ- ing high-level decision-man maaknn (ALFWorld), low- level action spaces (BabyAA-Text), and large ac- tion spaces (WebShop) demonstrate the frame- work's 's generality and superiority over otate-of- the-art methods. Notably, our approach achieves competitive performance using 7B/SB parameter LLMs, even outperforming baseline methods em- ploying GPT-4 in complex tasks. These results underscore the potential of integrating structured policy optimization with LLMs' intrinsic knowl- edge to advance decision-making capabilities in multi-step environments.
Early works (ichter et al., 2023; Huang et al., 2022a) have employed LLMS as policies, generating actions in an auto- regressive way that directly utilizes the models' prior knowl- edge for decision crtision-mmaing.... While these methods are simple and effective for short-term action generation, they often lack the capacity for long-term planning. Although reason- ing techniques such as Chain-of-Thoone Thouuhh have been intro- duced to enhance the reasoning capabilities of LLMs (Yao et al., 2023; Shinn et al., 2024) and improve action selection, the ability to engage in comprehensive long-term planning remains underdeveloped. Consequently, decisions may ap- pear locally optimal but fail to meet the overall objectives in more complex, multi-step environments.
Other approaches (Hao et al., 2023; Liu et al., 2023; Fu et al., 2024; Brooks et al., 2024) incorporate planning, either through interaction with real environments or by leveraging the LLMs' internal imaginative capabilities, followed by action evaluation based on the planning outcomes. These evaluations are then used to select the actions that yield the best results. However, such methods heavily depend on the model's accuracy in performing rollouts and evaluating outcomes, leading to sub-optimal action selection when the model's 's rollouts diverge from reality or when action evaluations are inaccurate.
The fundamental limitation of these two lines of research lies in the decoupling of LLM's prior policy and action evaluations. They either (1) rigidly follow LLM priors without sufficient planning (2) over-rely on potentially flawed simulated rollouts, both fail to reconcile the LLM's inherent knowledge with error-corrcted planning insights.
'Equal contribution 'HS. Tsinghua University 'Washington University in St. Louis. Correspondence to: Heng Dong <drd- hxi xxi@gmail.coms
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
To address the above problems, we propose novel LLM-

---

LLM-based Actor-Critic
Trajectory
[Reflection]: Now I find saltshaker (1). Next, I need take it. take the saltshaker after finding it. The thought is reasonable. This step is GOOD,
[Instruction]: (...) Your task to: put saltshaker in drawer.
(...)
Action Evaluation (Sec. 4.1)
[Obs You open the cabinet 2. The cabinet 2 is open. In it, you see plate 1, a saltshaker 1, and soapbottle 1.
Prior Action Distribution LTL
take saltshaker from cabinet 2 [Imagined Traj u']: You pick up the saltshaker 1 from the cabinet 2.
[Candidate a']:
[Reflection]: have taken saltshaker in this place. This step is GOOD.
a²
a²
a
[Critic )']: 0.81355 by QLM = log POo)
POw) (Fq. 3)
[LAC Inference 4]
Updated Action Distribution Nnew
[Candidate a²]: go to
[Candidate a a²]: go to
drawer 1
shelf 1
(...) [Critic @²1: 0.79087
(...)
[Act :]: take saltshaker 1. from cabinet
[Critic @²]: 0.59058
[Obs 5]: up the saltshaker from cabinet 2.
pick
a²
a²
a
Policy Optimization (Sec. 4.2)
Nnew TILM exp(aQuuM) (Eq. 6)
Figure 1. Framework of our LAC. At each time step, LAC optimizes the policy via two steps: (1) the critic QLL evaluates each candidate action sampled from the policy MLAMi (2) the policy ALLM is optimized according to the action evaluations using a gradient-fre free method.
based Actor-Critic (LAC) approach that leverages action evaluations to optimize the LLM's prior policy under con- straints, facilitating the effective integration of decision- making information inherent in the LLM's prior knowledge with insights derived from robust action evaluations that incorporate long-term planning. While there have been at- tempts to simultaneously utilize both types of information (the LLM's prior knowledge and action evaluations) for decision-making these efforts have typically resulted in simplistic combinations (Zhang et al., 2023).
uation insights to optimize the policy. A straightforward method might involve direct gradient-based actor-critic up- dates; however, this approach is inefficient for LLMs due to high computational demands, leading to increased de- cision latency. To solve this challenge, we formulate the policy improvement problem KI-diivegeenc optimiza- tion problem and derive closed-form solution, allowing us to optimize the policy in gradient- free manner and improve the accuracy of decision-making
Empirically, we demonstrate the effectiveness and generality of LAC across diverse environments, including high-level decision-making tasks (ALFWorld, (Shridhar et al., 2021)), low-level action space (BabyAI-Text. (Carta et al., 2023b)) and potentially infinite action space (WebShop, (Yao et al., 2022)). The results show that our approach consistently outperforms state-of-the-art methods, e.g., RAP (Hao et al., 2023), LATS (Zhou et al., 2024a). Notably, in several chal- lenging tasks, LAC using 7B/8B L.MSs significantly sur- passes ReAct (Yao et al., 2023) with GPT-4 (Achiam et al., 2023).
Implementing this idea involves two major challenges. First, how to extract action evaluation information from the LLM. A simple approach is directly prompt the LLM to output evaluation values that indicate the quality of action; however, this direct prompting method can be unstable and casily influenced by the instructions and examples provided to LLMs. To address this issue, we use the logits of specific tokens to calculate Q values (action values). Specifically, we employ tokens that convey positive or negative meanings as the LLM's internal beliefs about the action's 's quality or its likelihood of successfully completing the task. We derived simple yet effective formula to use these logits for Q value computation. Additionally, to allow long-term planning ability and enhance the accuracy of this calculation, we perform rollouts for each candidate action to predict future trajectories, conduct brief reasoning process based on these trajectories, and then collect positive/negative token logits to compute Q values.
Our contributions are twofold: (1) We propose a novel Q- function estimation approach to extract action evaluation information from LLMs that leverages LLMs' internal belief' about success or failure of the current task; (2) We formu- late the policy improvement problem as a KL-divergence constrained optimization and derive a closed-form solution, allowing us to optimize the policy in gradient- free manner using the action evaluation.
The second challenge is how to utilize these action eval-
2

---

LLM-based Actor-Critic
2. Related Work
wards like ALFWorld and BabyAl-Text. Retroformer (Yao et al.) trains a smaller LLM with PPO generate suit- able prompis for larger LLM for specific task, while our method only needs a small model. ICPI (Brooks et al., 2024) uses L.MSs to implement policy iteration by predicting fu- ture trajectories and accumulating future rewards, which may also struggle with sparse reward settings. We have compared it empirically in Section 5.
Large Language Models for Sequential Decision-Making Sequential decision makion is a key ability of intelligent agents, involving generating actions to achieve goals (Barto al., 1989 Littman, 1996; McCarthy et al., 1963; Bylander, 1994). Recently, LLM-based agents have gained popular- ity in patisiioommkiing across various fields, requiring only instructions or few-shot examples to adapt to new tasks (Huang et al., 2022b; Singh et al., 2023; Ding et al., 2023) due to pre-training on large datasets. Previous work pri- marily categorizes LLMs into two roles: policies, which generate actions from trajectories, and crities, which eval- uate actions based on trajectories and actions. Research falls into two categories: the first uses LLM--genratedd ac- tions directly from previous trajectories (ichter et al., 2023; Huang et al., 2022b; Yao et al., 2023; Huang et al., 2022a; Shinn et al., 2024). However, the auto-regressive nature of LLMs limits their long-term planning capabilities, making them struggle with complex tasks (Huang & Chang, 2022; Mialon et al., 2023). The second category employs another LLM to evaluate actions by simulating outcomes, choosing the best one (Hao et al., 2023; Liu et al., 2023; Fu et al., 2024; Koh et al., 2024). While these methods enable long- term planning, they heavily depend on evaluation accuracy, which can lead to sub-optimal solutions.
3. Preliminary & Previous Work
In this section, we describe the task setting and previous LLM decision-making methods for better understanding.
Task setup. Consider a general setup of an agent interact- ing with an environment for achieving a given goal g, e.g.. g = ""put clean egg in microwave" (from ALFWorld) or g = =pick up the green ball" (from Bl-xxA- At time step t. the agent receives a natural language described ob- servation or E O from the environment. The agent then takes an action O E A sampled from policy Taalg. h2), where fg := (01, a1, 02, a2 ... ,r) is the history to the agent. During execution, there is no immediate reward and only at the end of each episode, the environment will give signal to evaluate the completion of the task. For each testing task, the agent can only try once and cannot conduct improvements through repeated trials.
To address these issues, we propose optimizing LLM's prior policy using action evaluations under constraints, enhancing long-term planning while mitigating evaluation inaccuracies. Previous attempts to combine LLM's prior policy and action evaluation (Zhang et al., 2023) have been simplistic and lacked theoretical guarantees, resulting in unsatisfactory performance.
Methods that directly use LLM's prior as a policy. To solve the above tasks with LLMs, one simple method is to directly use LLM's prior as policy: as + arg max, (al, he). which can be implemented by sim- ply injecting instructions or few-show examples to the prompt as suggested in Yao et al. (2023). Despite its sim- plicity, the policy TLLM generates actions solely relying on its auto-regression ability and does not conduct long- term planning explicitly, which is typically necessary for sequential prcision-making tasks. Additionally, this issue will be exacerbated when using lightweight models like CodeLlama- 7 (Roziere et al., 2023) and Mistral-7B (Jiang et al., 2023). This problem is verified in Section 5.
Large Language Models with Reinforcement Learning Classical sequential decision- decision-making methods, such as Re- inforcement Learning (RL), have been widely adopted in embodied environments (Schulman et al., 2017; Fujimoto et al., 2018; Huang et al., 2020; Dong et al., 2022). However, these RL-based methods are typically cample-ineficieen and require lots of samples for training. On the other hand, LLMs that contain rich prior knowledge about the world may alleviate this burden. To combine RL and LLM, one straightforward way is to use LLMs as base models and add policy/value heads on top of LLMs (Carta et al., 2023a; Tan et al., 2024). Then classical RL methods like PPO (Schul- man et al., 2017) can be used for training (Szot al., 2023: Zhou al., 2024b). However, these methods still require lots of training samples of the same tasks, which reduces the benefits of using LLM to some extent and contradicts our settings. There are also other paradigms for combining. RLEM (Zhang et al., 2024) adopts Q-learning (Watkins& & Dayan, 1992) and an experience memory to update policies, but may get stuck in the tasks with extremely sparse re-
Methods that incorporate planning and action evalua- tions. To handle the issue of lack of long-term planning, another line of research incorporates planning and action evaluations into decision-making (Hao al., 2023; Liu et al., 2023; Fu et al., 2024). The basic idea is to first sample several candidate actions from policy {a}, at ......................... . ] TLLM [gg h2), then evaluate each candidate action by other LLMs and finally select the action with the highest evalu- ation value. The evaluation procedure is the key to these methods, and many approaches can be adopted. For exam- ple, directly ask an LLM to evaluate the action candidate (Fu et al., 2024), predict the future trajectory ut of each action candidate by regarding an LLM forward world model
3

---

LLM-based Actor-Critic
Algorithm 1 LAC: LLM-based Actor-Critic algorithm.
receives a reward only upon achieving the goal. By linking QulM to the agent's success probability of task comple- tion, we enable more effective guidance toward maximizing expected returns. Furthermore, we demonstrate how this evaluation can be derived directly from token logits associ- ated with positive and negative outcomes.
1:
Input: current task goal g. history he, actor TLLM. for- ward model fLLM. value-based critic QLLM. hyperpa- rameter a, candidate action size n.
2: Output: selected action oj
M.IM(=l9, ha); 4: for i = 1 to n do
3: {ai]
D candidate actions
5: u² + fLLiIg he, af);
>. predict future trajectory
4.1.1. CONNECT CRITIC TO SUCCESS PROBABILITY
6: LLLM(G, he, ab, ui) +- log Plgulg.b...f.a))) D action P(unlo.h..a..u)))
Let QuLM(g, he; at) be the value-based evaluation of each candidate action aj given the task goal g and history ha- We consider scenarios with sparse rewards, which are only provided at the end of each episode. Considering the binary outcomes, we hope QLIM(g, Ive, af) could reflect the possi- bility of achieving success effectively. We employ a logistics function (Jordan et al., 1995) to relate QLMM(g he, af) to the success probability.
evaluation (Sec. 4.1)
7: end for
8: Tnew (ailg, he) + TLLM(ajIg. ha) exp(a he, b policy optimization (Sec. 4.2) 9: ag + arg m Tew (a (a ht)
a; u;));
fUm and use the future outcome as evaluations, or use tree-search methods like Monte Carlo Tree Search (MCTS) (Kocsis& & Szepesvári, 2006; Coulom, 2006) to expand each action candidate (Hao al., 2023). Despite this progress, these approaches heavily depend on the model's accuracy in performing rollouts and evaluating outcomes. When rollouts diverge from reality and evaluations are inaccurate, which could be common when using lightweight LLMs, the action selection could be sub-optimal.
Let P(y h,, at) E [o, 1] denote the probability of suc- cessfully completing the task goal g after executing action o/. where you represents a success signal at the end of the episode. Similarly, let P(wigg he, at) represent the failure probability. We use the following formulation to connect P(y 9, h2, at) with QLIM(g, h, af):
P(ywlg, he, ai) 1+ exp (- QLLM(M a;))
With this formulation, QLIM(g. he, of) is positively corre- lated with the success probability P(ywlg, he, Higher QuLM(g he, af) values map to a greater likelihood of suc- cess, allowing the critic to guide the policy toward actions that maximize long-term returns. In this way, maximiz- ing the Q-function corresponds to maximizing the success probability for the current task.
4. Method
In this section, we introduce our novel LLM- Actor- Critic (LAC) algorithm, which effectively integrates the gen- erative capabilities of LLMs (as the actor) with their eval- uative reasoning capabilities (as the critic) in a principled and scalable manner. The actor, represented by the LLM- based prior policy (TLLM), generates potential actions, while the critic (QLLM) evaluates these actions by incorporating long-term reasoning. This synergy enables effective policy improvement through an innovative gradient-free optimiza- tion method.
While other formulations could be used, we found that Equa- tion (1) is both simple and effective for wide range of tasks. For a comparison of alternative formulations, refer to Ap- pendix A.2 and Appendix A.3.
4.1.2. ESTIMATE CRITIC WITH LLMS
We first explain how to leverage LLMs to extract meaning- ful action evaluation information (Section 4.). Building on this, we propose gradient-free policy optimization method that efficiently refines the policy based on the critic's feed- back (Section 4.2). The overall workflow of the algorithm is summarized in Algorithm 1.
To estimate QuLM(y, hy, at) using LLMs, we perform an equivalent transformation on Equation (1):
P(y he, at) P(y g, he, at)
Qum(g, he, at) log
(2)
which uses the equation P(y.) + P(y)= = With Equa- tion (2), we can use the LLM obtain QLIM(g, hr af) via first estimating P( (m.1)) [g, a;). The basic idea is to prompt the LLM to predict the outcomes given the cur- rent trajectory (g, ht) and action aj. Specifically, we use special paired tokens with positive/negative meanings, i.e., to indicate
4.1. Critic Development for Long-term Action Evaluation
We propose novel method for constructing the critic QIIM. designed to provide numerical evaluations of candidate ac-
tions sampled from the policy (-1, h.). This approach targets base- decision-maan makinn problems characterized by sparse rewards and binary outcomes, where the agent
"GOOD" or "SUCCESS" success/failure outcomes. The corresponding generated
4

---

LLM-based Actor-Critic
AlfWorld
(ours)
RAFA
BabyAl-text (task: go to)
0.73
0.06
GPT-4-
0.7
0.6
1.34
0.5
Rate
0.5
Succeess Rate
2.42
0.46
0.4
0.34
0.72
0.3
0.3
0.25
0.25
0.25
0.25
0.24
0.2
0.18
0.36
0.1
0.1
0.22
0.0
0.0
CoceLiama-78
Uamai-3-83 Different Large Language Models
Gemma-78
Mistrai-78
Codeliama-78
Gemma-78
3-86
Mistral-7B
Large Language
Figure Performance of our LAC compared with various baselines benchmarks ALFWorld and BabyAl-Text
probabilities of LLMs for those special tokens reflect LLMs' internal belief in success/failure after taking action a². We let the generated probabilities of "GOOD" and "BAD" rep- resent (v gg. h4, at) and P(u h4, at) respectively. Fi- nally, using Equation (2), we can calculate QL lı, of) for action a;. Though our action evaluation is designed for binary outcomes, we empirically show that this formulation also effective continuous-rewar settings in Section 5.
generate a short reflection such as "I have found object-X. This step GOOD" or should take object-X instead of object-Y Y first. This step is BAD." These reflections provide judgments about whether and why the previous actions were appropriate. For more examples of these reflections, please refer to Table 27 and Table 28 of Appendix B.
These reflections serve as simple reasons, akin to Chain- of- Thought (CoT) (Wei et al., 2022; Kojima et al., 2022; Prystawski al., 2024), allowing the policy to sample better candidate actions by avoiding past mistakes. These reflec- tions also enable the critic evaluate candidate actions with greater accuracy, ultimately enhancing decision-making per- formance. For the difference comparison between our re- flection and CoT, please refer to Appendix A.4.
To improve the accuracy of QL hy; af), we introduce future trajectory rollouts using forward world model fLLM- which can be implemented by prompting LLMs, e.g., adding few-shot examples, or by fine-tuning on these examples. For each candidate action aj. we rollout sev- eral future steps to predict the resulting trajectory ot ~ fg he, a.). By considering the future trajectory uf, we obtain more informed estimates of the success and fail-
4.2. Gradient-free Policy Optimization
ure probabilities (()(w.1) Ig, /r, at, ut). This approach ac- counts for the delayed consequences of actions and ensures that QLLM(g, he, at, ut) reflects the long-tern value of each action:
In this subsection, we derive gradient-free policy opti- mization method using the above estimated value-based evaluation. To effectively improve the LLM's prior policy, we propose to use the following Kl.-constrained policy opti- mization problem maximize the expected value function:
P( Wy hes ai, uf) CLiM(g, he, of, ut) llog P(yllg g hy ot. u()
max
maxXaate (1..
[LLM(g he at, u4)]
Trajectory rollouts are especially important in tasks where the outcomes of actions may unfold over several steps. By simulating the future impact of actions, the critic provides accurate assessment, guiding the policy towards actions that maximize the probability of long-term success.
DEL [ h.) Mmmmm(aaa ha)]
(4)
where is hyperparameter controlling the deviation from the original policy MIlM. The KL-divergence term prevents the new policy nnew from deviating too far from the origi- nal policy, balancing the policy's 's prior knowledge and the critic's s guidance.
Empirically, we also found that contextual reflections on previous actions can be helpful to improve performance. Specifically, given the task goal and history h1 which may contain the predicted future trajectory, we prompt L.MSs to
Following prior work (Rafailov al., 2024; Go et al., 2023:
5

---

LLM-based Actor-Critic
lourst
0.7
0.4
0.39
0.39
0.c3
0.00
0.64
0.62
0.03
6.37
0.37
0.0
0.59
2.33
0.30
Succiss Rate
0.20
0.20
0.27
0.12
0.15
0.14
0.1
0.0
0.0
CodeLlama-78
Lama--388 Different: Large Language Models
Gemma-78
Mistral-78
CodeLiama-78
Cemma-78
Llama-3 3-88
Mistral-78
Different Large Langusge Models
Figure 3. Performance of LAC in benchmark WebShop
Peng et al., 2019; Jain et al., 2013; Peters & Schaal, 2007), we can show that the optimal solution to the KL-constrained maximization objective in Equation (4) takes the following form:
5. Experiments
In this section, we benchmark our method LAC on three benchmarks that cover high-level action space (ALFWorld (Shridhar al., 2021)), low-level action space (Baby Al-Text (Chevalier- -ooisvett et al., 2018)) and potentially infinite ac- tion space (WebShop (Yao et al., 2022)). We evaluate the effectiveness LAC by answering the following questions: (I) Can LAC outperform other decision-making with LLM- based methods? (Section 5.2) (2) How does each component of LAC contribute its performance? (Section 5.3) (3) How do different large language models influence performance? (Section 5.2 and Section 5.3) (4) Is our method computation- ally consuming? (Section 5.4) (5) Why is LAC effective? (Section 5.5). The code of LAC is publicly available on GitHub and website?.
1
Thow (a/lg, h1)
ALLM(ai he)
((g, h.)
eexp (a QLLLI, he, oi ut)),
(5)
where (gg, (b)) = Ea; ALLM h1) exM(a he, at, u²)) is the partition function. Please refer Ap- pendix B.1 for complete derivation. As the partition function does not depend action at, ignore it in practice:
Tnew (a'lg, he) x mLIM(a h.) exp (d.Li ba, at w()).
(6)
5.1. Experiment Setup
We simply take the action with maximum proportion a;
We compare our method with various seciisionmmaingg with LLMs baselines. Here we briefly introduce these methods, and for more details, please refer to Appendix C.2.
arg max Tnew (nilg, ht). It is worth mentioning that if we let in Equation (6), recover the methods that directly use LLM's prior a policy, and if we let +00, we recover the methods that incorporate planning and action evaluations.
Various baselines: (1) ReAct (Yao et al., 2023) combines reasoning and acting in the interaction with the environ- ment and leverages the reasoning capabilities of L.LMs to increase the probability of the LLM acting correctly as policy. (2) RAP (Hao et al., 2023) utilizes LLMs as policy and world models and adopts tree-search planning methods to evaluate each possible action candidate. (3) ICPI (Brooks et al., 2024) implements policy iteration using LLMs by predicting future trajectories and selecting the action with
There are two key advantages of using Equation (6). Firstly, updates the action distribution of policy TLLM the direc- tion suggested by critic QLLM a gradient-free way, which achieves policy improvement with much lower computa- tion burden compared to gradient-based methods, especially when the actor is realized by LLMs. Secondly, the action distribution of the new policy nnw is a balanced integra- tion of the policy's 's prior based on past information and the critic's posterior based on predicted future information.
hnttps://github com/ droh/LAChcC Pnttps://siten www.aiteeesssseoeeeoeeoooooooooe com/voo/vvii/oo/oooo wwto//viewwaaggg./
6

---

LLM-based Actor-Critic
AlfWorld
citli-anly
BabyAl-text (task: go to)
3:18
6:73
3.64
0.42
0.99
0.6
0.62
2.54
AM
SUccese Ra1e
147
Succesi Romt
1.45
2.42
0-32
0.3
2:0
0.1
Codelano 78
Gomito 78
Lama 38
Nisrral 78
CodeUlamo 78
Gemma 78
Upmo 388
Mistrol 78
Diffeeent Lange Langunge Models
Different Large Language Modes
Figure 4. Ablation studies in benchmarks ALFWorld and BabyAl-Tere
the highest predicted cumulative rewards. (4) RAFA (Liu al., 2023) evaluates each action candidate by tree-search and selects the action that may complete the most sub-goals. (5) LATS (Zhou et al., 2024a) combines the reasoning, act- ing, and planning capabilities of LLMs with MCTS (Kocsis & Szepesvári, 2006) and external feedback mechanisms to enhance decision-making across various domains, achieving competitive results in web navigation.
Potentially infinite actions: WebShop (Yao et al., 2022) requires an agent to purchase product based instructions (e.g. "I need long clip-in hair extension which natural looking, and price lower than 20.00 dollars") through web interactions (e.g. search "long clip-in hair extension", click buttons such as "[item IDJ" or "back to search"). Within this context, the "search" and "click" actions can indeed lead to an unbounded set potential actions, as the agent can con- tinuously refine its queries and selections based on dynamic web results. Different from ALFWorld and BlbyAl-Text, the final reward in this benchmark is continuous value between 0 and 1, depending the degree to which the final purchased product meets the requirements.
We evaluate LAC on three decision decision-making benchmarks with sparse rewards and distinct types of action space.
High-level actions: ALFWorld (Shridhar al., 2021) is a widely used text-based household environment with 134 different evaluation tasks, which require the agent to achieve a goal through a sequence high-level actions, e.g. "go to place-X", XX", "take object-Y from place-X", etc. The agent gets reward of if it achieves the goal, and otherwise. The main challenge of this benchmark is to locate the tar- get object and fulfill household work with commonsense knowledge of LLMs. Following ReAct, we evaluate all 134 unseen evaluation games a task-specific setup.
To show the stability of LAC, we adopt four open- source large language models from different organizations: CodeLlama-7B (Roziere et al., 2023), Mistral-7B (Jiang et al., 2023), Gemma-7B (Team et al., 2024), and Llama-3- 8B (Meta, 2024a).
5.2. Main Performance
We report the results of method LAC compared with other baselines in Figure 2 (ALFWorld and Baby Al-xet),, Figure 3 (WebShop) and Figure 6 (more BabyAI-Texx tasks), For all experiments, we set the temperature of LLMs to 0, hence the generation is deterministic. For this reason, there is no error bar in the figure.
Low-level actions: Baby Text (Carta et al., 2023b) is Grid World environment that extended from the BabyAl plat- al., 2018), in which the agent and objects placed in room of x 8 tiles. The agent has 6 primitive actions: turn left, turn right, go forward, pick up, drop, toggle, to solve a task described in natural language (e.g. "Pick up the red box"). Similar to ALFWorld, this benchmark also has binary rewards of 1 or These tasks could be difficult because agents have to make long-term plan, avoid obstacles, and find short path to target objects based partial observations that are described in natural language.
form Seevvalier-Boiveer
LAC outperforms all other baselines in both ALFWorld and BabyAl-Text and even better than GPT-4+ReAct in most settings, which validates our method's effectiveness and stability. In WebShop, LAC consistently outperforms other baselines, in terms of both accumulated reward and success rate across various base models. This further demonstrates

---

LLM-bbsed Actor-
bouse
Plakc
DAP
Icht
TARA
Success Rate
Staps lins
Time Usard lask
Toeen Coal Teak
0.3
33
21.39
3.0
2.5
1,49
0.5
20-
6.19
15
3.2
0.2
0.5
Cfftrent Pathecs
Offinen:
Difleam:
Figure 5. Computational cost analysis of LAC and baselines.
the robustness of method in handling more complex and open-ended action spaces.
flection procedure when sampling and evaluating candidate actions; (4) critic-only only uses critic's action evaluation information for action selection.
LAC'S superior performance stems from its effective inte- gration of the decision-making information contained in the LLM prior with the decision maasing insights derived from action evaluations that incorporate long-term planning. For better understanding, we have also provided illustrative examples for ALFWorld and Al-TTT- in Figure 10 and Figure 11 respectively. In summary, the policy and critic alone may make mistakes at different time steps, our LAC can select the correct action through their integration.
We report the results in Figure 4. LAC is better than all other variants in both ALFWorld and BabyAI-Texi Specifically, the performance decrease in LAC w/o critic and critic-only compared with LAC verify the effectiveness LAC'S policy optimization with critic's action evaluations. And the result that LAC w/o reflection and LAC w/o rollout perform worse than LAC also suggests the necessity for incorporating re- flections and future trajectory predictions.
Regarding the performance of LAC with different base mod- els, we highlight two key findings: (1) Our method is general and can adapted various base models, and (2) stronger base models, such Gemma-7B. demonstrate higher per- formance when integrated with approach. However, due to the incomplete public availability of training details for these base models, further in-depth analysis will require additional investigation.
5.4. Computational Cost Analysis
Our method conducts action evaluations by predicting future trajectories, which may bring extra computational cost per step. In Figure 5, we compare computational costs concern- ing the number of tokens spent and running time between LAC and other baselines. Specifically, though LAC has higher computational cost per step due to the extra infer- ence procedure of critics and the forward model, the total cost of LAC is still lower than most LLM-based baselines because LAC requires fewer steps to finish each task. This is due to LAC'S higher success rate, enabling it to complete tasks within the maximum step limit, while other baselines often reach this limit without completing the tasks. If we only consider successful tasks, the step cost is similar across methods: LAC: 15.32 steps, ReAct: 17.75 steps, and RAP. 16.36 steps. For more computational cost analysis of other basclines, please refer to Appendix A.6.
Here we only compare LAC with LLL--based methods, for more results regarding other baselines, e.g., finetuning the policy, LLM-based RL variants, and (Chen et al., 2021), 1), please refer to Appendix A.5.
5.3. Ablation Studies
To investigate the contributions of each component of LAC, we conduct elaborate ablation studies. There are several main components that characterize our method: (1) policy optimization step using action evaluations provided by critic: (2) action evaluations by extracting internal information of L.MMs. Therefore, to show the contribution of each compo- nent, design the following ablation studies: (1) LAC w/o critic removes the policy optimization step from LAC; (2) LAC w/o rollout does not predict future trajectories during action evaluations; (3) LAC w/o reflection removes the re-
5.5. Statistical Analyses
Directly interpreting LLM-based decision-making is inher- ently challenging due to two factors: (1) LLMs are black- box models, making it difficult to directly explain their outputs; (2) In sequential decision-making tasks, ground-
8

---

LLM-based Actor-Critic
Table 1. Pearson correlations between Q-values and timesteps.
Successful Trajectory (1843 steps in total)
Failed Trajectory (1092 steps in total)
log P("GOOD")
0.35 = 0.18
-0.37 + 0.19
log P("BAD")
-0.32 0.19
0.38 + 0.19
Q-value of chosen action
0.34 # 0.18
-0.41 0.18
Table Confidence analysis of policy improvement.
Case (Proportion)
Prior Policy Conf.
Q-function Conf.
the Improved Policy Conf.
Both agree (49.81%) Prior agrees, Q disagrees (29. 12% cases)
0.49 (V)
0.21 (V)
0.74
0.34 (V)
0.06 (X)
0.28
Prior disagrees, Q agrees (19.28% cases)
0.21 (X)
0.34 (V)
0.22
Prior disagrees, Q disagrees (1.79% cases)
0.06 (X)
0.03 (X)
0.07
truth actions are typically unavailable for analyzing each action the policy selects-one task-level success or failure signals are observed.
more confident model; (3) In low-confidence cases, the pol- icy remains conservative. These results suggest our method implicitly aggregates decision knowledge from both sources by weighting based on model confidence, enabling effective policy improvement without explicit gradients.
Therefore, we provide statistical analyses to shed light on why the two key components of LAC Q-function estima- tion and gradient-free policy improvement-wort effec- tively in practice.
6. Discussion
In this work, we introduce a novel LLMM-based Actor-Critic algorithm LAC that leverages action evaluations to optimize the prior policy of LLM, enabling an effective integration of the decision decision-making insights derived from LLM prior and action evaluations that incorporate long-term planning. Compared with previous methods, LAC achieves high per- formance on three benchmarks that cover various action spaces even using lightweight open-source LLMs..
Q-function estimation reflects task progression We esti- mate Q-values using LLM-internal beliefs (log-probale bog-probabsllties of "GOODD""BAD" tokens). To validate that these Q-values meaningfully track task success, we compute the Pearson correlation between Q-values and timesteps within each trajectory. The intuition is: if an agent is successful path, Q-values should inerease otherwise). The results in Table 1, averaged across 134 ALFF World tasks, confirm this trend: successful trajectories show increasing Q-values, while failed ones show the opposite. This demonstrates that our Q-function captures the evolving success likelihood of the policy throughout trajectory.
progresses (and decrease
Despite the advancement, our method also has limitations and inspires possible future directions. Firstly, the reflection of LAC only used before action generation, which can also be applied after action generation. For example, it can provide reflections for predicted future trajectories to re- sample candidate actions if the previous candidate actions all fail to complete the target task. Secondly, though we only expand one node for each candidate action for simplicity and efficiency and find it works effectively, LAC can also adopt tree-search to provide more accurate assessment of candidate actions. Thirdly, though LAC could deal with scenarios with continuous final reward empirically by only treating getting the highest reward as success, it is an ex- citing future direction to develop a more principal method to deal with such situations. Last but not least, the effec- tiveness of LAC when applied larger and more powerful LLM models needs further investigation.
Policy improvement reflects model confidence Our gradient-free policy improvement balances the prior pol- icy and the Q-function based relative confidence. Specifically, we define model's confidence as the dif- ference between its top two scores: (1) For the prior pol- icy: log ((a]ls)= - log P(az ((az (2) For the Q-function: - Q(ayls)-Q(azla s). where a1 and 02 are the actions with the highest and aecond-highest probability (or Q-value), respectively, under the corresponding model.
We analyze which action (from the prior policy Q- function) is selected by the improved policy and what confi- dence each model had in its choice. Table 2 summarizes the outcomes. This analysis shows that: (1)When both models agree, the confidence of the improved policy is highest; (2) When they disagree, the improved policy tends trust the
9

---

LLM-based Actor-Critic
Impact Statement
Cuadron, A.,. Li, D., Ma, W. Wang, X., Wang, Y., Zhuang, S., Liu, S., Schroeder, L. G., Xia, T., Mao, H., et al. The danger of overthinking: Examining the reasoning-action dilemma in agentic tasks. arXiv preprint arXiv:2502.08233, 2025.
Our method is built upon open-source large language mod- els (LLMs). Like other methods that use LLMs, our method also inherits some benefits and challenges from LLMs. For the benefits, method directly exploits the prior knowl- edge from LLMs, which may reduce potential carbon costs compared with training policies from scratch. For the chal- lenges, our method might be susceptible to producing unin- tended output when confronted with harmful input, such as unethical text or input intended for adversarial attacks. To solve this problem, we suggest a thoughtful deployment of method, such as adding filtering component.
DeepSeek-AN Deepseek-rl:
Incentivizing reasoning ca- pability in Ilms via reinforcement learning, 2025. URL https: http://aaxxvv....bbb//aoaaa hbtprg/bb//20000 12948.
Ding, Y., Zhang, X., Paxton, C., and Zhang, S. Task and motion planning with large language models for object rearrangement. 2023 IEEE/RSJ International Con- ference on Intelligent Robots and Systems (IROS), pp. 2086-2092. IEEE, 2023.
References
Dong, H., Wang, T., Liu, J., and Zhang, C. Low-rank modular reinforcement learning via muscle synergy. Ad- vances in Neural Information Processing Systems, 35: 19861-19873, 2022.
Achiam, J., Adler, S., Agarwal, S., Ahmad, L.,, Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S.,
Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
Fu, D., Huang. J., Lu, S., Dong, G., Wang, Y., He, K., and Xu, W. Preact: Predicting future in react enhances agent's planning ability. arXiv preprint arXiv:2402. /134, 2024.
Barto, A. G., Sutton, R. S., and Watkins, C. Learning and sequential decision making, volume 89. University of Massachusetts Amherst, MA, 1989.
Fujimoto, S., Hoof, H., and Meger, D. Addressing function approximation error in actor-critic methods. In Interna- tional conference on machine learning, pp. 1587-1596. PMLR, 2018.
Brooks, E., Walls, L., Lewis, R. L., and Singh, S. Large lan- guage models can implement policy iteration. Advances in Neural Information Processing Systems, 36, 2024.
Bylander, T. The computational complexity of propositional strips planning. Artificial Intelligence, (9(1-)):165-2 1994.
Go, D., Korbak, T., Kruszewski, G., Rozen, J., Ryu, N., and Dymetman, M. Aligning language models with prefer- ences through f-divergence minimization. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), Proceedings of the 40th In- ternational Conference Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 11546 1 11583. PMLR, 23-29 Jul 2023. URL https: proceed wwaceedingg..o ml r. press / 222. go233a httll.
Carta, T., Romac, C., Wolf, T., Lamprier, S., Sigaud, O., and Oudeyer, P.-Y. Grounding large language models in interactive environments with online reinforcement learn- ing. In International Conference on Machine Learning, pp. 3676-3713. PMLR, 2023a.
Carta, T., Romac, C., Wolf, T., Lamprier, S., Sigaud, O., and Oudeyer, P.-Y. Grounding large language models in interactive environments with online reinforcement learning, 2023b.
Hafner, D. Benehmarking the spectrum of agent capabilities. In International Conferencee on Learning Representations, 2022. URL https:/ / htt.hhttttt/t/nntaeeeet wptp//einnneeeeeeeeepeeeeoo net/ forum? id=1W0z 6MFEoH.
Chen, L.,, Lu, K., Rajeswaran, A., Lee, K., Grover, A., Laskin, M., Abbeel, P., Srinivas, A., and Mordatch, I. De- cision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:1 5084-15097, 2021.
Hao, S., Gu, Y., Ma, H., Hong, J., Wang. Z., Wang, D., and Hu, Z. Reasoning with language model is planning with world model. In Bouamor, H. Pino, J., and Bali, K. (eds.). Proceedings of the 2023 Conference on Empérical Meth- ods in Natural Language Processing, pp. $154-8173. Singapore, December 2023. Association for Computa- tional Linguistics. doi: 10. 10.18663w.1/0023.mmmmm 507. URL emnlp-main..
-hevalier-Boisvert, M., Bahdanau, D.,, Lahlou, S., Willems, L., Saharia, C., Nguyen, T. H., and Bengio, Y. Babyai: A platform to study the sample efficiency of grounded lan- guage learning. arXiv preprint arXiv:1810.00 1111.10000.22... 2018.
2023..
ma m.
Coulom, R. Efficient selectivity and backup operators in monte-carlo tree search. In International conference on computers and games, pp. 23. Springer, 2006.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models, 2021.
10

---

LLM-based Actor-Critic
Huang, and Chang, K. C.-C. Towards reasoning in large language models: A survey. arXiv preprint arXiv:2212.10403 2022.
Koesis, L. and Szepesvári, C. Bandit based monte-carlo planning. In European conference on machine learning, pp. 282-293. Springer, 2006.
Huang, W., Mordatch, I., and Pathak, D. One policy to con- trol them all: Shared modular policies for agent-agnostic control. In International Conference Machine Learn- ing, pp. 1455-44666 PMLR, 2020.
Koh, J. Y., McAleer, S., Fried, D., and Salakhutdinov, R. Tree search for language model agents, arXiv preprint arXiv:2407.00 00000000476.... 2024.
Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwasawa, Y. Large language models are zero-shot reasoners. Ad- vances in neural information processing systems, 35: 22199-22213, 2022.
Huang, W., Abbeel, P., Pathak, D., and Mordatch, I. Lan- guage models as zero-shot planners: Extracting ac- tionable knowledge for embodied agents. In Interna- tional Conference on Machine Learning, pp. 9118-9147. PMLR, 2022a.
Li, J., Tang. T., Zhao, W. X., Nie, J.-Y., and Wen, J.-R. Pre- trained language models for text generation: A survey, ACM Comput. Surv. 56(9), April 2024a. ISSN 0360- 0300. doi: 10. 10.1 145/364944,
Huang, W., Xia, F., Xiao. T., Chan, H., Liang. J., Florence, P. Zeng, A., Tompson, J., Mordatch, I., Chebotar, Y., et al. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint arXiv:2207.05608, 2022b.
URL https: /doi.
org/10. 11 1011114/335555 3644449.
Li, X., Zhou, Y., and Dou, Z. Unigen: A unified generative framework for retrieval and question answering with large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 8688-8696, 2024b.
ichter, b., Brohan, A., Chebotar, Y., Finn, C., Hausman, K., Herzog, A., Ho, D., Ibarz, J., Irpan, A., Jang, E., Julian, R., Kalashnikov, D., Levine, S., Lu, Y., Parada, C., Rao, K., Sermanet, P., Toshev, A. T., Vanhoucke, V., Xia, F., Xiao, T., Xu, P., Yan, M., Brown, N., Ahn, M., Cortes, O., Sievers, N., Tan, C., Xu, S., Reyes, D., Rettinghouse, J., Quiambao, J., Pastor, P., Luu, L., Lee, K.-H., Kuang, Y., Jesmonth, S., Joshi, N. J., Jeffrey, K., Ruano, R. J., Hsu, J., Gopalakrishnan, K., David, B., Zeng, A., and Fu, C. K. Do as i can, not as say: Ground- ing language in robotic affordances. In Liu, K., Kulic, D., and Ichnowski, J. (eds.), Proceedings The 6th Con- ference on Robot Learning, volume 205 of Proceedings of Machine Learning Research, pp. 287-318. PMLR, 14- 18 Dec 2023. URL www.t:ttphhhoc /tattp:::/prrreeeeee /ttp////oeoeeeeeeeee/ocooocooeeo ngs. mlr.. press /v205/225//// http:/ihhherra..... html..
Littman, M. I.. Algorithms for sequential decision-making. Brown University, 1996.
Liu, Z., Hu, H., Zhang, S., Guo, H., Ke, S., Liu, B., and Wang, Z. Reason for future, act for now: A principled framework for autonomous Ilm agents with provable sam- ple efficiency. arXiv preprint arXiv:2309113882 2023.
McCarthy, J. et al. Situations, actions, and causal laws. Comtex Scientific, 1963.
Meta. Meta Ilama 3. https://llame hwwp:l/aaammmm.....mmmmmm com/ 1 1lama3/, 2024a.
Meta. Meta llama 3.1. https://att meta. com/ blog/ seg/meta-ll 11 3-11--1//, 2024b.
Jain, A., Wojcik, B., Joachims, T., and Saxena, A. Learning trajectory preferences for manipulators via iterative im- provement. Advances in neural information processing systems, 26, 2013.
Mialon, G., Dessi, R., Lomeli, M.. Nalmpantis, C., Pa- sunuru, R., Raileanu, R., Roziere, B., Schick, T., Dwivedi- Yu, J., Celikyilmaz, A., Grave, E., LeCun, Y., and Scialom, T. Augmented language models: a survey, Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL https: openreview. nett Forum?id= 7ittjjhhwH22KK Survey Certification.
Jiang, A. Q.. Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. I., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. Mistral 7b. arXiv preprint arXiv:2310.0682. 2023.
Paglieri, D., Cupial, B., Coward, S., Piterbarg, U., Wolezyk, M., Khan, A., Pignatelli, E.,, Kucinski, f.., Pinto, L., Fergus, R., Foerster, N., Parker-Holder J., and Rock- täschel, T. Benchmarking agentic Ilm and vlm reasoning on games. arXiv preprint arXiv:2411. 13543, 2024.
Jin, H., Zhang, Y., Meng, D., Wang, J., and Tan, J. A comprehensive survey on process-orienter automatic text summarization with exploration of Ilm-based methods. arXiv preprint 0032222403.0290,, 2024.
Jordan, M. I. et al. Why the logistic function? a tutorial discussion on probabilities and neural networks, 1995.
Peng, X. B., Kumar, A., Zhang, G., and Levine, Advantage-weighue regression: Simple and scalable

---

LLM-based Actor-Critic
off-policy reinforcement learning. arXiv:1910.00177, 2019.
arXiv preprint
Tan, W., Zhang, W. Liu, S., Zheng, L., Wang, X., and An, B. True knowledge comes from practice: Aligning large language models with embodied environments via reinforcement learning. In ICLR, 2024.
Peters, J. and Schaal, S. Reinforcement learning by reward- weighted regression for operational space control. In Proceedings of the 24th international conference on Ma- chine learning, pp. 745-750, 2007.
Team, G., Mesnard, T., Hardin, C., Dadashi, R., Bhupatiraju, S., Pathak, S., Sifre, I.., Rivière, M., Kale, M. S., Love, J., et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.0822 008.0. 2024.
Prystawski, B., Li, M., and Goodman, N. Why think step by step? reasoning emerges from the locality of experience. Advances in Neural Information Processing Systems, 36, 2024.
Touvron, H., Martin, L.,, Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint rrXiv:2307.09288, 2023.
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. Er- mon, S., and Finn, C. Direct preference optimization: Your language model secretly reward model. Ad- vances in Neural Information Processing Systems, 36, 2024.
Watkins, C. J. and Dayan, P. Q-learning Machine learning, 8:279-292 1992.
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E. Le, Q. V., Zhou, D., et al. Chain-of- of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824 24837, 2022.
Roziere, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I.,
Tan, X. E., Adi, Y., Liu, J., Remez, T., Rapin, J., et al. Code llama: Open foundation models for code. arXiv preprint ar arXiv:2308. /2950, 2023.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.0634 2017.
Yao, S., Chen, H., Yang, J., and Narasimhan, K. Web- shop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Informa- tion Processing Systems, 35:20744 20757, 2022.
Shinn, N., Cassano, F., Gopinath, A., Narasimhan, K., and Yao, S. Reflexion: Language agents with verbal rein- forcement learning. Advances Neural Information Processing Systems, 36, 2024.
Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., and Cao, Y. React: Synergizing reasoning and act- ing in language models. In International Conference on Learning Representations (ICLR), 2023.
Shridhar, M., Thomason, J., Gordon, D., Bisk, Y., Han, W., Mottaghi, R., Zettlemoyer, L., and Fox, D. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In Proceedings of the IEEF/CVF con- ference on computer vision and paltern recognition, pp. 10740-10749, 2020.
Yao, W., Heinecke, S., Niebles, J. C., Liu, Z., Feng, Y., Xue, L., Rithesh, R., Chen, Z., Zhang, J., Arpit, D., et al. Retroformer: Retrospective large language agents with policy gradient optimization. In The Twelfth International Conference on Learning Representations.
Shridhar, M., Yuan, X., Côté, M.-A., Bisk, Y., Trischler, A., and Hausknecht, M. Alfworld: Aligning text and embodied environments for interactive learning, 2021.
Zhang. B., Mao, H., Ruan, J., Wen, Y., Li, Y., Zhang, S.. Xu. Z.. Li, D.,. Li, Z., Zhao, R., al. Con- trolling large language model-based agents for large- scale decision-making An actor-critic approach. arXiv preprint arXiv: 2.. 2023.
Singh, I., Blukis, V., Mousavian, A., Goyal, A., Xu, D., Tremblay, J., Fox, D., Thomason, J., and Garg, A. Prog- prompt: Generating situated robot task plans using large language models. In 2023 IEEE International Conference Robotics and Automation (ICRA), pp. 11523-11530. IEEE, 2023.
Zhang, D., Chen, I.., Zhang, S., Xu, H., Zhao 7.,, and Yu, K. Large language models are semi-parametric reinforce- ment learning agents. Advances in Neural Information Processing Systems, 36, 2024.
Szot, A., Schwarzer, M., Agrawal, H., Mazoure, B., Metcalf, R., Talbott, W., Mackraz, N., Hjelm, R. D., and Toshev, T. Large language models as generalizable policies for embodied tasks. In The Twelfth International Conference on Learning Representations, 2023.
Zhou, A., Yan, K., Shlapentokh- Rothhommn M., Wang, H., and Wang, Y.-X. Language agent tree search unifies reasoning, acting, and planning in language models. In Forty-first International Conference on Machine Learn- ing, 2024a.
12

---

LLM-based Actor-Critic
Zhou, Y., Zanette, A., Pan, J., Levine, S., and Kumar, A. Archer: Training language model agents via hierarchical multi-turn rl. In Forty-first International Conference Machine Learning, 2024b.
13

---

LLM-based Actor-Critic
A. Extra results
A.1. Results of other tasks Baby A--ext
For a complete comparison, we show the performance of LAC and baselines in other tasks from BabyAl-Text in Figure 6. Our LAC outperforms all other baselines, which further validates the effectiveness of LAC.
A.2. Results of using other definition of QLLM
In LAC we define critic QLLM as QL he, he, = log P( (t........)) P(w (a There are also other definitions, for example, the simplest variant is (g, he: af) log P(y lg, he, a.). In this subsection, we provide performance comparison between them in Figure 7. LAC outperforms the variant in most situations across tasks and models. We speculate that this is because LAC uses more information, ie., both P(ywly lg, he, at) and P(yllg Ig, he, at), than the variant, and the evaluation might be more accurate and more stable. There might be other definitions of QLLM and among them, our QLLM is simple and effective.
A.3. Results of directly prompting the LLMs to output action evaluation
To further demonstrate the strengths of our critic QIIM- We conducted additional experiments comparing the performance of LAC and LAC w/ direct evaluation on the WebShop benchmark. LAC generates value-based evaluations by extracting LLMs' internal beliefs success and failure as described in Equation (2). For LAC w/ direct evaluation, we prompt the LLMs to directly output the probability of success P(yw). while keeping all other components unchanged. The Q-value is then calculated as log P(y.)
I-P(y-)
The results, presented in Table 3 and Table 4, show that our LAC method outperforms LAC w/ direct evaluation in terms of success rate and reward across most base models. Analysis of the resulting trajectories revealed that LAC w/ direct evaluation often produces non-informative success probability (e.g., P(yw) = 0.5), leading to ineffective improvements in policy.
Table 3. Success rate of LAC and LAC w/ direct evaluation in WebShop
CodeLlama-7B
Gemma-7B
Llama-3-8B
Mistral-7B
LAC
32%
46% %
39%
32%
LAC w/ direct evaluation
27%
29%
33%
24%
Table 4. Rewards of LAC and LAC w/ direct evaluation in WebShop
CodeLlama-7B
Gemma-7B
Llama-3-8B
Mistral-7B
LAC
0.5840
0.7237
0.6733
0.6299
LAC w/ direct evaluation
0.5636
0.6975
0.6453
0.6333
A.4. Comparison of our reflection component and chain-of-thought
In this subsection, we empirically compare the reflection method used in LAC and the Chain-of- -hhiinoffThhough (CoT) methods (Wei et al., 2022; Kojima et al., 2022).
Though the reflection draws inspiration from CoT, is not trivial to determine what the reflection content should be. Our method generates judgments based on previous actions and their outcomes, whereas CoT uses arbitrary thought generation without structured focus past mistakes. The reflection is specially designed for decision-making tasks and owns two advantages over naive CoT:
Strength 1. Our reflection can improve the policy directly by avoiding previous mistakes when the policy is conditioned on its generation.
To substantiate this claim, we conducted experiments comparing the performance of prior policy w/ reflection and piror policy w/ CoT on the WebShop benchmark. Here are the details of these two variants:
14

---

LLM-based Actor-
go
to after pick
0.30
0.25
0.7
C.6C
0.20
0.5
0.45
Slucoui Natee
0.45
Rato
0.15
0:14
0.3
0.10
11.0
0.06
0.05
0.1
5.02
0.20
2.03
Codel Lama-78
Gemma-771 Different
Uama-3388 Models
Mistrai-78
Codellama-70
Gemma-78 Different
Uamma-344
Mistral-71
pick up
pick up then go to
0.99
0.5
A.D
6.43
0.28
Succasee Ram
0.29
0.3
5.76
0.20
0.00
0.20
0.20
0.20
0.15
0.14
116
0.166
5.19016
0.12
0.10
0.00
0.1
6.66
00
0.05
004
2.02
0.00
CodeLiama
Gemma. 78
Mistral-78
CodeLiama
Gemma
Uama
Mistrai-78
Language
Different large Language
open door
put next to
0.04
Succes Rate
0.06
0.03
0.04
0314
0104
1UD44
0.00
6:22
0.0
OLE0R.200
0.00
0.000
0.00
0.00
lama-78
Laam--3888 Large Language Models
Codel lama-78
Llama-3-88
Different Large Language Moduls
Figure 6. Performance of our LAC compared with various baselines in all tasks from Baby Ae-Text.
15

---

LLM-based Actor-Critic
AlrWorld
LAC (ours)
LAC (o log 11)
BabyAl-text (task: go to)
0.8
0.80
0.70
0.7
2:46
0.66
0.7
0.0
0:34
0.5
Succ Rate
0.46
0.5
0.4
0.3
0.3
0.2
CodeLlama-78
Gomma-78 large language
Uama-3388
Mistrai-78
CodeUlama-78
Large lnguuge Models
Figure 7. Performance of LAC when using different definition of critic QuL
(I) prior policy w/ reflection: We remove all components of LAC, leaving the policy and the reflection unchanged. Specifically, at each step, after observing the action results, the reflection first generates some judgments on previous actions, and then the policy selects the next action based on the judgments. (2) prior policy w/ CoT: We remove all components of LAC, except the policy, Additionally, we equip the policy with CoT by adding "Let's think step by step" to the prompt. Specifically, at each step, before choosing the next action, the CoT prompting component first outputs arbitrary thoughts that may help solve the task.
As shown in Table and Table 6, prior policy w/ reflection consistently surpasses prior policy w/ CoT across most base models in terms of both success rate and reward. By analyzing the results, we found that prior policy w/ CoT may make the same mistake multiple times and get stuck at this mistake, while prior policy w/ reflection can largely avoid seen mistakes.
Table Success rate of prior policy w/ reflection and prior policy w/ CoT WehShop
Codel lama-78
Gemma-7B
Llama-3-8B
Mistral-7B
prior policy w/ reflection
21 %
46%
39%
31%
prior policy w/ CoT
21% 11%
20%
31%
20%
Table 6. Rewards of prior policy w/ reflection and prior policy w/ CoT in WebShop
CodeLLama-77
Gemma-7B
Llama-3-8B
Mistral-7B
prior policy w/ reflection 0.5739
0.6564
0.6556
0.6288
prior policy w/ CoT
0.5520
0.5347
0.6379
0)4661
Strength 2. Our reflection can be seamlessly integrated with critic QIIM. which helps the critic to generate more accurate value-based evaluations.
To further evaluate this integration, compare the performance of LAC and LAC w/ CoT on the WebShop benchmark. The details of the two methods are as follows:
(1) LAC: Our original method. (2) LAC w/ CoT: We replace the reflection component of LAC with CoT and keep other components unchanged.
We show the results in Table and Table Our method LAC consistently outperforms LAC CoT regarding success rate
16

---

LLM-based Actor-Critic
and reward across all evaluated base models. This is because, without reflection's judgment on previous steps, the critic may output inaccurate value-based estimations, hindering the poliey improvement phase.
Table 7. Success rate of LAC and LAC w/ CoT in WehShop
Gemma-7B
Llama-3-8B
Mistral-7B
LAC
32%
46%
39%
32%
LAC w/ CoT
27%
29%
33%
24%
Table 8. Rewards of LAC and LAC w/ CoT in WebShop
CodeLlama-77 Gemma-7B Llama-3-8B
Mistral-7B
LAC
0.5840
0.7237
0.6733
0.6299
LAC w/ CoT
0.5734
0.5270
0.6515
0.5101
In summary, the reflection's focused evaluations of past actions provide critical advantages over the simple reflections of CoT, leading to improved performance in decision-man decision-making tasks.
A.5. Comparison of LAC with more baselines ALFWorld
Parc WC polc
Carcer Soldl MOMET CISIICE
SPH- MAN
GPhii Reect
200
CAL
URNEN P-AU
Figure Performance of our and LAC's variants compared with various baselines in benchmark AI. A.rFo....
In this subsection, compare LAC with more baselines including some traditional RL methods implemented using LLMs on ALFWorld (Shridhar et al., 2021). The comparison is shown in Figure 8.
A. FINETUNE POLICY
While LAC we fine-tune the model for generating reflections using a few trajectories, it is also possible to fine-tune the policy to generate actions using those trajectories. Therefore, a potential baseline could be fine-tuning the policy in ReAct. demonstrate the improvement brought by fine-tuning the policy, we fine-tune the policy in ReAct (Yao et al., 2023) and show the results in Figure 8. We also show the results of LAC w/ fine-tuned policy in Figure In brief, ReAct w/ fine-tuned policy is strong baseline compared with other baselines, but still inferior to our method LAC and LAC w/ fine-tuned policy. Compared to LAC, the underperformance of LAC w/ fine-tuned policy arises from its tendency to overfit the training trajectories. This overfitting causes the policy to favor actions that are more frequent in the dataset, potentially leading to suboptimal action selection.
For example, in the ALFWorld training dataset, the action "take an apple from X" occurs frequently. After fine-tuning, the policy may disproportionately generate this action, even when it irrelevant the current goal. One case that the current
17

---

LLM-based Actor-Critic
goal is to "heat some egg and put it in the garbage can". When the agent sees an "apple 2" in "fridge 1", it generates and selects irrelevant action "take apple from fridge 1", which does not align with the task.
This tendency towards overfitting arises because the complexity of the policy function, which maps states to actions a. often exceeds that of the critic. The policy often has to capture a wide variety of potential actions for each state, particularly in complex environments. However, the quite limited training dataset in our setting restricts its ability to generalize effectively, resulting in memorization of specific actions rather than flexible decision-making. In contrast, the model for reflection generation and rollouts focuses on capturing more predictable dynamics of the environment and simpler evaluation criteria. This typically requires simpler mappings than those needed for the policy, thus avoiding overfitting.
A.5.2.
RL VARIANTS
We also include some LLM-based RL variants as baselines to show the superiority of LAC over conventional RL. algorithms. We design three LLM-based RL variants that are built upon pre-trained LLMs and directly extract actions/values information from LLMs without adding action/value heads, namely Conventional Policy Gradient, Conventional Policy Gradient w/ dense rewards and Conventional Actor-Critic in Figure 8.
For the implementation of the Conventional Policy Gradient, we need the probability of actions and the returns. To obtain the probability of actions, we directly use LLM to compute the conditional probability of each token in action a, [w] W2," W a. given the goal g. history h and then calculate their product:
(arlg. h2) | PLLM (wylg. g, hes wes))
in which PLEar (wyll lg, he, weez ) is the probability of token wy given goal g. history he and previous tokens Wey computed by LLM. Then we regard the cumulative future rewards as the return Gs, which is +1 for successful trajectories and -1 for failed trajectories in the tasks we considered. Finally, the gradient of policy is E,E, Vlogrr(a (g. ht G.]. Conventional Policy Gradient w/ dense rewards is similar to Conventional Policy Gradient except that we manually add intermediate rewards for each step, and then use the cumulative future rewards as the return G..
For the implementation of the Conventional Actor-Critic, we additionally need critic estimate action values. As it is possible to train a new value head using only 18 trajectories, we instead approximate the action value similar to QLIM in our method LAC, i.e.
QLLM(g, he, oz, uz) = log PLLAS (yw gg, hi, at, We)
PLLM (yilg, hes az, Ue)
in which PLEMI (H(n,yyl ae, ud) is the output probability of special positive/negative tokens like GOOD or BAD that indicate positive/negative results as LLM'' belief on success/failure. Finally, the gradient of policy is E, VlogT he) Qg he, az, 4:)].
In summary, Conventional Policy Gradient exhibits almost all zero performance, which is due to the extremely sparse reward problems, compared with Conventional Policy Gradient w/ dense rewards. Conventional Actor-Critic demonstrates non-zero performance only on some stronger LLMs like Gemma-7B (Team et al., 2024), Llama-3-8B (Meta, 2024a) and Llama-3.1-8 (Meta, 2024b), which may be because the optimization method of conventional actor-critic is not suitable in insufficient data settings.
A.5.3. DECISION TRANSFORMER
In addition to the aforementioned LLse RL variant, Decision Transformer (Chen et al., 2021) is also a potential solution in combining RL and transformer-basee LLMs. We fine-tune pretrained LLMs in a similar way as conventional decision transformers. We construct a dataset using deeision-transformmes'' trajectory representation: T [Ri, , #1, R2, 82; 02 ...] in which R2 is return-to-go, i.e., +1 for successful trajectories and -1 for failed trajectories in our extremely sparse reward settings. Then we fine-tune LLMs with next-token prediction loss on these trajectories. During execution, we insert +1 before state to specify the desired outcome. The results are shown in Figure 8 as Conventional Decision Transformer. In short, Conventional Decision Transformer exhibits similar performance to ReAct, which may be because the 18 trajectories are insufficient for fine-tuning decision transformers.
Our method LAC is better than all considered baselines because of its ability to handle extremely sparse reward problems using L.M''s prior knowledge and fully utilize insufficient data.
18

---

LLM-based Actor-Critic
A.6. Computational cost analysis of LAC with more baselines in ALFWorld
IAC loum)
Reaet finetuned Ricktt
RATA RAP
Consentional Gradient st dense Convention ActorCriic
Conventionall Decidoon Tandormer
LAC Trncturdd od policy
Convertional Palic Grodient
Success Rate
Steps per lask
Time Used per lask
Token Cost per lask
3.5
0.8
X2.05
1.18
35-
3.0
2,5
0.5
75-
0.4
16.201
455.5
3.30
1.5
1.0
500000
0.1
0.5
0.
Dffieent Nuthods
Offirent Muthoss
Difierent Metheds
0.0
Diferrnt Nethodls
Figure 9. Computational cost analysis of our LAC compared with various baselines in benchmarks ALFWorld. Though LAC may have a higher computational cost per step due to the extra inference procedure of crities and the forward model, the total cost of LAC is still lower than most LLM-based baselines because LAC requires fewer steps to finish each task.
In this subsection, we demonstrate the computation cost of LAC and other baselines in Figure 9, We show the success rate, steps per task, time used per task, and token cost per task respectively. Specifically, though LAC has higher computational cost per step due the extra inference procedure of crities and the forward model, the total cost is still lower than most LLMM-based baselines because LAC has higher success rate and requires fewer steps to finish each task.
A.7. Illustration of ALFWorld and BabyAI-Text
We show the illustrative example of ALFWorld and BabyAl-Text in Figure 10 and Figure 11, respectively.
Figure 10 presents a concrete scenario where the agent's goal is to "put a saltshaker in the drawer". At a critical decision step (Step 4), we observe the following intuitive distinction between components: (1) LL.M-based prior policy alone mistakenly suggests "go to drawer 1" because the base LLM model overlooks that the agent has already found the correct object ("saltshaker 1"") in cabinet This error exemplifies the common hallucination problem LLMs, which occurs when the model disregards previous states and incorrectly recommends irrelevant actions. (2) In contrast, the critic suggests "take saltshaker from cabinet 2" because it evaluates potential actions by predicting future trajectories and determines that this action will successfully pick up the correct object. (3) Our method leverages these distinct insights by optimizing the prior policy's action distribution based on the critic's 's evaluation. It effectively corrects the errors introduced by the prior policy, balancing the strengths of prior policy (flexible but sometimes inaccurate) and critic evaluations (accurate but computationally intensive).
This analysis shows that: (1)When both models agree, the confidence of the improved policy is highest; (2) When they disagree, the improved policy tends to trust the more confident model; (3) In low-confidence cases, the policy remains conservative.
These results suggest our method implicitly aggregates decision knowledge from both sources by weighting based on model confidence, enabling effective policy improvement without explicit gradients.
A.8. Results of different critic improvement methods
Empirically, we found that the critic can be improved via fine-tuning on few trajectories. Please refer to Appendix B.2 for more fine-tune tuning details. To show the effectiveness of fine-tuning, we present the performance of LAC and other variants when we just add these examples into the prompt, i.e., in-context learning, on task "go to" and "pick up" from Baby Text in Table 9 and Table 10 respectively. We also show the the performance improvement if we do fine-tuning in the parentheses. This result indicates that tun-tt can incorporate extra knowledge into LLMSs better than in-context learning in our
19

---

LLM-based Actor-Critie
Prompt
[nntrructio]]: Interact with a household to solve a task. Here are two examples. (...)
[Task]: You are in the middle of a room. Looking quickly around you, you see a cabinet 6, cabinet 5, cabinet 4, cabinet 3, a cabinet 2, cabinet 1, coffeemachine countertop 1, drawer 3, drawer 2, drawer 1, fridge 1, garbagecan 1, microwave 1, shelf 3, shelf 2, shelf toaster 1.
countertop 3, countertop 2,
sinkbasin
stoveburner
stoveburner stoveburner stoveburner and
Your task is put saltshaker in drawer.
Inputs & Outputs of LAC (Ours)
Action Probs. (Partial)
(...)
prior policy
LAC LAC LAC reflection
[Aet 1]: go to cabinet 1
critic
critic
(Ours)
[Obs 2]: On the cabinet 1, you see glassbottle 1. [LAC Inference
[Aet 2]: go to cabinet 2 /
[Obs 3]: The cabinet 2 is closed. [LAC Inference 3] [Act3]: open cabinet 2 /
[Obs 4]: You the cabinet 2. The cabinet 2 is open. In it, see plate saltshaker 1, and soapbottle 1.
[LAC [nferenee
[Act take saltshaker from cabinet 2 V
[Obs You pick up the saltshaker from the cabinet 2. (...)
[Obs 7]: You open the drawer 1. The drawer 1 is open. In it, you see fork 1, and peppershaker 1.
[LAC Inference
X
[Aet 7]: saltshaker in/on drawer
[Obs 8]: You put the saltshaker in/on the drawer
Bectonnnte 706 #hwe73
Cooe Shael'
Figure illustrative explanation our method LAC in ALFWorld. The histogram on the right shows the action probabilities different methods. While LLM's prior policy (TTx) and critic, well as LAC w/o reflection, make mistakes at different time steps. LAC (ours) can select the correct action by optimizing policy given action evaluation. LAC inference step detailed in Figure
20

---

LLM-based Actor-Critie
Prompt
[Instruction]: Assume that you are an agent in a Grid World. Given goal, your task is to execute a sequence of actions to achieve the goal. [Possible actions]: turn left, turn right, go forward, pick up, drop, toggle. [Goal]: go to the green ball
Inputs & Outputs of LAC (Ours)
Action Probs. (Partial)
[Obs I]: You see a wall steps forward, You see wall steps right, You see green ball 3 steps left and 4 steps forward, You see a grey key steps left and steps forward, You see blue key 1 step left and 3 steps forward, You see a green key step left and steps forward, You see a blue ball 1 step right and step forward [LAC Inference 1] [Act I]: go forward /
prior policy
critic
LAC w/o reflection
LAC w/o critic
LAC (Ours)
X
go forward
turn left
turn right
[Obs 2]: (...) You see green ball steps left and steps forward, (...) You see a blue ball 1 step right [LAC Inference 2]
X
[Act2]: go forward /
(...)
[Obs 5]:
wall step forward, (...) You
X
X
green ball steps left [LAC 5] []c turn left /
[Obs 6]: (...) You red ball 4 steps forward, You green ball 3 steps forward
X
[LAC Inference 6] [Act 6]: go forward V
Figure 11. An illustrative explanation of our method LAC in BabyAl-Text. The histogram on the right shows the action probabilities of different methods. While LLM's prior policy (Mass) and critic, well LAC w/o reflection, make mistakes at different time steps, LAC (ours) can select the correct action by optimizing the policy given action evaluation. Please refer to Table 30 for the full trajectory.
21

---

LLM-based Actor-Critic
case and improve the quality of action evaluation, yielding better performance. It is worth mentioning that our LAC still outperforms baselines without fine-tuning.
Table 9, Performance of two critic improvement methods: in-context learning or fine-tuning.
CodeLlama-7B 7B
Gemma- 7B
Llama-3-8B
Mistral-7B
LAC
0.30 (t 0.16)
0.62 (t 0.14)
0.32 (t 0.34)
0.24 (1 0.46)
LAC w/o reflection
0.30 (f 0.02)
0.58 (f 0.02)
0.38 (f 0.10)
0.26 (1 0.28)
LAC w/o critic
0.28 (t 0.06)
0.48 (1 0.04)
0.42 (1 0.10)
0.10 (1 0.28)
critic-only
0.42 (t 0.04)
0.40 (1 0.24)
0.34 (f 0.22)
0.38 (1 0.24)
Table 10. Performance of two critic improvement methods: in-context learning or fine-tuning.
CodeL
Gemma-7B
Llama-3-8B
Mistral-7B
LAC
0.20 (t 0.06)
0.22 (f 0.20)
0.34 (f 0.08)
0.20 (t 0.16)
LAC w/o reflection
0.16 (t 0.08)
0.32 (1 0.04)
0.32 (1 0.04)
0.22 (1 0.06)
LAC w/o critic
0.12 (f 0.14)
0.36 (f 0.20)
0.28 (f 0.06)
0.26 (f 0.04)
critic-only
0.22 (t 0.04)
0.24 (1 0.26)
0.16 (f 0.16)
0.16 (1 0.26)
A.9. Analysis of the fine-tuning process in LAC
In order to improve the quality of the reflections generated by LLM, we finetune the LLM that generates the reflections. In this section, we analyze the finetuning process, showing the impact of finetuning, well the impact of different data amounts and positive and negative sample ratios on task success rates. The comparison can be seen in Figure 12.
In Figure 12 (a), we show the influence of fine-tuning data size. We use 9, 18, 27 and 36 trajectories to fine-tune LLMs, and show the final success rate on 134 evaluation tasks. In summary, larger data sizes (27 or 36 trajectories) generally bring higher success rate, while small data sizes (18 and even trajectories some cases) are already enough for LAC to achieve outperformance.
Figure 12 (b) shows the influence of different positive/negative sample ratio (positive:negative 0:1, 1:3, 1:1,3:1 and 1:0) on final performance. We keep the total number of samples the same and just change positive/negative ratio. In short, LAC is robust to reasonable positive/negative ratios (e-g- 1:3, 1:1, 3:1), while LAC based on 2023) and Gemma-7B (Team et al., 2024) even perform better when given all positive samples (1:0).
(Roziere et al.,
Figure 12 (c) shows the learning curves of the fine-tuning process. We plot the next prediction loss and positive/neeaatiee tokens prediction accuracy for Codel 7B (Roziere et al., 2023). In short, as the next token prediction loss decreases during fine-tuning, the accuracy of predicting the special tokens (GOOD BAD) increases, which exhibits the effect of the fine-tuning process.
A.10. Ablation studies on the impact of finetuning
The finetuning process is important in our method, but can also be replaced by in-context learning, where several examples inserted in each input. Here we compare our method with ReAct regarding the effect of finetuning in ALFWorld. The results, presented in Table 11, show that our method outperforms the baseline both with and without finetuning, and the performance can be further improved with finetuning. With finetuing, our method provides more consistent performance with different LLMs.
A.11. Results of different hyper-parameter a
The hyper- paremmerreeterr a in Equation (4) controls the deviation from the original policy TLLM- In this subsection, we grid- search this hyper-parameter over {1/2, 1,2,5, 5, 10} in task "go to" of BlbyAl-Text, then we fix a for other tasks: a = 1 for model CodeLlama- 7B, a = 2 for model Gemma-7B: a = 2 for model Llama-3-88 3-BB and a = 10 for model Mistral-7B.
As for benchmark ALFWorld, we fixed a = 1 in all experiments.
22

---

LLM-based Actor-
becney
UA poenee
Finetuiing Analysic
1.0
Acoun
0.5
0,7
2.4
92
Cokrbbe 24
fineturing Staps
(b)
(c)
Figure 12. Analysis regarding the fine-tuning process of our LAC. (a) Influence of the fine-tuning data size. Larger data sizes (27, 36 trajectories) generally bring higher performance, but small data sizes (18 and even 9 trajectories) are already enough for method achieve outperformance. (b) Influence of the positive/negative data ratio. LAC is robust to reasonable positive/negative ratios (1:3, 1:1, 3:1) while CodeLlama-78 and Commma-7B-basee LAC even perform better given all positive data (1:0). (c) Learning curves of next-token prediction loss and positive/negative tokens prediction accuracy for CodeLlama-7B and ALFWorld.
Table 11. Ablation studies on the impact of finetuning.
CodeL.amm-77B
Gemma-7B
1lama-3-8B
Mistral-7B
LAC (w/o finetuning)
0.39
0.59
0.71
0.57
ReAct (w/o finetuning)
0.20
0.54
0.31
0.34
LAC (w/ finetuning)
0.79
0.84
0.78
0.79
ReAct (w/ finetuning)
0.38
0.70
0.73
0.65
A.12. Performance comparison in other benchmarks
To further show the strength of our method, we conduct preliminary experiments on Crafter (Hafner, 2022), using the implementation from BALROG (Paglieri et al., 2024). Crafter is a 2D survival game specifically designed to test long-horizon reasoning and sequential decision-making, with tasks involving resource gathering, crafting, and combat. It represents a significantly more complex setting than ALFWorld and WebShop.
We evaluated our method this benchmark using Llama-3.1 1-1--B, following BALROG's official evaluation protocol. We compare LAC with several representative baselines from BALROG's GitHub repository. The preliminary results are summarized in Table 13. Our method achieves higher performance than other available baselines under identical evaluation settings. These preliminary results provide further evidence of the robustness, effectiveness, and adaptability of our proposed actor-critic approach, particularly in significantly more challenging and complex becision-making environments. is worth noting that CoT performs worse than the baseline Naive on Crafter benchmark. We hypothesize that due to the model's inconsistency within their chain-of-thor -fttohhtttt Two contiguous chains of thoughts might lead model to take actions which push towards different goals, which not ideal. The authors also note that this is problem especially with smaller, weaker models.
A.13. Preliminary experiments with reasoning LLMs
We have conducted preliminary experiments with reasoning LLMs like DeepScek-RI RI-Distill---eennn (DeepSeck-AI, 2025). However, observed that they often tend overthink rather than output direct environmental actions in both our method and the baseline ReAct. For instance, even when we explicitly prompt the reasoning LLMs to output actions (e.g., "Please make sure your response is in the following format/In> {The action you choose)") the models still generated detailed explanations but avoided selecting the next action. A typical response might be: "I need to find a key to open the safe or locate the pencil in the drawers. Since I can't (...), ) I'm unable (...), must (...)". This issue has also been noted in prior work (Cuadron al., 2025). We believe that using reasoning L.MMs for decision-making tasks requires deeper
23

---

LLM-based Actor-Critic
Table 12, Results of different hyper-parameter
CodeLlama-7B
Gemma- -B
Llama-3-8B
Mistral-7B
LAC (a = 1/2)
0.46
0.54
0.62
0.68
LAC (a 1)
0.46
0.62
0.64
0.58
LAC (a 2)
0.44
0.76
0.66
0.64
LAC (a 5)
0.46
0.72
0.62
0.64
LAC (a 10)
0.40
0.58
0.60
0.70
Table 13. Performance comparison of LAC (Ours) and other baselines in Crafter (from BALROG)
-lama-3.1-8B-It
LAC (Ours)
25.91 % # 1.93%
Naive (direct action generation)
20.45% + 4.26%
Robust Naive (formatted actions)
4.55% # 1.57%
CoT (ReAct, reason then act)
18.64% # 3.24%
Robust COT (reason + formmated actions)
15.46% # 3.59%
Few-Shot (in-context examples)
12.73% = 1.25%
exploration to balance internal reasoning with effective environmental interaction.
B. Method details
B.1. Deriving the solution of the KL-constrained maximization objective
In this subsection, we will derive Equation (5). We optimize the following objective:
max E (otxx(((
[OLIM(S he, of ut)] De [m(a[lg h mollmum h.)]
We now have:
max E af (a;1g.t)))
[uuL he, ai uf)]
DEL [ he) (aj he)
a
1
Ttallg he) log
-max E (a
CuIM(g, he, ahs ui)
mLIM( 1,, he)
log T(ailg, he)
-min E (a
QLiM (g, he, ars ut)
(a h4)
T I(a (g, h2)) Zighh.) mLM exp (a Crimiy le, at, ut))
min E (af
log
log Z(g, h4)
where we have the partition function:
7(g,h4) = Σ MLIM(afla h2) exp (aQuu hy ar ut))
Since the partition function is function of only g, he and the original policy FLLMs but does not depend on the optimized policy n, we define
1
n" (aaglg,
Tum(allg he) exp (a QLLM(( he, of, ut)).
h(g,h.))
24

---

LLM-based Actor-Critic
This definition of policy if valid probability distribution as x² x" g, h4) for all oj and Eaj x ("(agg, hy) 1. As 2(g,1u) is not a function of af, we can then re-organize the objective as:
π { g. he) B(g. TLLM (ailg, h4) exp (a QLLM he at, u!))
min E off
log
log Z(g, he)
log < g. h4)
min E (a
log Z h4)
(=(at g, hr)
min DAL [aallg he)ll=" ht]]
llo Z(g, h4).
Then since (g, h2) does not depend on n, we can only care about the KL-divergence, which is minimized at 0 if and only the two distributions are identical. Therefore, the optimal solution is
mim he) exp (a bey oi, u)) T(all,, he) - n" (a'lg, (iilg, he) Zig. h4)
(7)
which completes the derivation.
B.2. Critic improvement of LAC
The reflections critic QLLM. and forward model fLLM we used can be easily implemented by prompting LLMs via providing instructions few-shot examples from similar tasks like prior work (Yao et al., 2023; Liu et al., 2023). However, empirically, we found that they can be further improved via fine-tuning LLMs with simple next-token prediction loss on several samples collected from training tasks. In this work, we consider fine-tuning for ALFWorld and BI-TAT-Text to show the impact of fine-tuning. We use 18 trajectories for each benchmark for fine-tuning, including two successful trajectories and one failed trajectory for each task-type. Though 18 trajectories are significantly fewer than what required for conventional reinforcement learning algorithms, they are generally enough for our method. Each trajectory has the following format: (g. og. 01, 01, C1, ................................ aH 0H, cH). where is the episode length and c is reflection of action ag. Each e includes an explanation about the action a, (e.g., "I have found object-X. This step is "I should take object-X instead of object-Y Y first. This step is ") and special token that indicates positive/negative judgment (e.g., "GOOD" "BAD").
Practically, we just fine-tune the LLM once and use it to generate all the reflections, construct critic QLIM. and forward model fULM. thanks to the fine-tuning with the above data format. Specifically, when minimizing the loss of predicting future trajectories, the forward model fLLM is improved. When minimizing the loss of generating reflections, the reflections, critic QLeM are both improved. The latter is because reflections also contain special tokens that indicate positive/negative judgments, whose generated probabilities are used calculate QLIM in Equation (3). We analyze this fine-tuning process in Appendix A.9. Some examples of the labeled trajectories in ALFWorld and Baby Al-Text are shown in Table 27 and Table 28 respectively.
C. Experiment details
C.1. Benchmark details
C.I.I. ALFWORLD: BENCHMARK WITH HIGH-LEVEL ACTIONS
We choose ALFWorld (Shridhar et al., 2021), a text-based household environment, to demonstrate the effectiveness of LAC on high-level planning. ALFWorld a synthetic text-based game aligned with ALFRED (Shridhar et al., 2020) benchmark. There are types of tasks in this environment, which require the agent to achieve a high-level goal through sequence of high-level actions, e.g. "go to place-X", "take object-Y from place-X", etc. The details about the 6 task types in ALFWorld are shown in Table 14.
A challenge built into ALFWorld is that the agent needs to explore the environment to find a target object. The commonsense knowledge in LLMs about the likely locations for common household items makes this environment suitable for LLMs to solve. The reward is 1 only when the agent reaches the goal. Following ReAct, we evaluate 134 unseen evaluation games in task-specific setup.
25

---

LLM-based Actor-Critic
Table 14. All the task types and the corresponding goals for ALFWorld
Type
Description
Pick & Place
The agent needs to pat a turget object to a lurget place, c.g. put some spraybottle on toilets, lind some apple and put it in sidetable, etc. The agent needs to find a target object, clean and pat turget place, e.g. clean apple and put sidetable, put clean lettuce in diningtable, etc. agent noeds find target objoct, heat and pat target place, e.g. heat some egg and pot it in diningtable, put bot apple in fridge, etc.
Clean Place
Heat Place
Cool Place
The agent needs find target objoct, cool and put target place, e.g, cool some pan and pot in stoveburner, put cool mug in shelf, etc.
Examine & Place
The agent needs to find a lurget object, and examine il with desklump, e.g. look at bowl under the desklamp, examine the pen with the desklamp,
Pick Two & Place
The agent needs to put two target objects to target place, e.g. pur two saltshaker in drawer, find two pen and pat them in dresser, etc.
C. BABYAI-TEXT: BENCHMARK WITH LOW-LEVEL ACTIONS
For decision decision-making making tasks with low-level planning, adopt BabyAI-Text (Carta et al., 2023b) as test-bed. BabyAI- Text is text-only version environment extended from the BabyAI platform (Chevalier-Be) Bebvvelier-Boiivert et al., 2018). BabyAl-Text is a Grid World environment, in which the agent and objects are placed in room of 8 x 8 tiles. The agent has 6 primitive actions: turn left, turn right, go forward, pick up, drop, toggle, to solve a task described in natural language (e.g. Pick up the red box). The agent has access to a 7 x 7 partial view, which means it can only observe the objects belonging to the 7 x 7 grid in front of In addition to objects relevant to completing a given task, there are also other distractors in the room. All the task types in BabyAl- Text are shown in Table 15.
Table 15. All the task types and the corresponding goals for BabyAl-Tea
Type
Description
go
The agent needs to find target object and go it, e.g. go the green key, go the red ball, etc.
pick up
The agent needs to find target object, go to and pick up it, e.g. pick up the blue key, pick up the purple ball, etc.
go after pick up pick up then to put next
blue key you pick up green key etc. The agent needs to find and pick up one object. then to another ubject, e.g. pick up the green box, then go to the purple box etc. agent needs to find and pick up one object, then go to another object and put the first object next to it, r.g. put the grey key next to the yellow ball etc.
The agent needs to find and pick up one object, then go another object, e.g. go
open door
The agent needs to know which key to pick up, then find and pick up it to open the door, e.g. open the door, open the blue door, etc.
Unlike ALFWorld, an agent interacting with Baby Ae-text needs to find out the suitable low-level action to execute at each step. We evaluate on the test environment in BabyAl-Text. The objects in a room are randomly chosen, and their position, as well as the agent's 's position, are also random. Considering the time and computational resource constraints, we evaluate on 50 tasks for each task type, yielding 300 tasks total.
1.3.3. WEBSHOP: BENCHMARK WITH POTENTIALLY INFINITE ACTION SPACE
In some scenarios, the space of possible actions can be infinite. We adopt a simplified environment, WebShop (Yao et al., 2022), with potentially infinite action space to demonstrate the effectiveness of our method. WebShop requires an agent to purchase a desired product based on textual instructions through the interaction with a simulated web environment. The available actions include search something using textual descriptions and choose button to click like "fitem ID]", "back to search", "buy now", etc. Due to the nature of natural language, one can actually search anything, leading to an unbounded set of potential actions. We evaluate 100 tasks in WebShop for our method and all the baselines.
One thing that should be mentioned about this benchmark is that the final reward is value between 0 and I, depending on the degree to which the final purchased product meets the requirements. Thus in this work, we report the success rate and final reward simultaneously, where success rate indicates the proportion of products that the agent finally buys that fully meet the requirements, and final reward indicates the average final reward on all evaluated tasks.
C.2. Implementation details of baselines
C.2.1. REACT
ReAct (Yao et al., 2023) uses LLM as policy to generate action directly. They combine reasoning and acting during the interaction with the environment to improve action selection.
Table 16 shows an example of trajectory interleaving between actions and observations in ALFWorld. To leverage the
26

---

LLM-based Actor-Critic
reasoning ability of LLMs, ReAct adds an extra thinking action the original action set in ALFWorld, allowing LLM agent engage in reasoning about task decomposition and goal achievement during task execution. An annotated trajectory for ALFWorld given by ReAct is shown in Table 17. Text labeled in light green is the annotated thought to help LLM complete a task better.
Table 16. An example of successful trajectory for ALFWorld
You are in the middle of a room. Looking quickly around you, you see cabinct 4, cabinet 3, cabinet 2, a cabinet 1, a countertop 1, garbagecan 1, a handtowe owndtowelholde 2, handtoow thol 1, sinkbasin 2, a sinkbasin 1, toilet 1, Loiletpapechangge L, and Lowelholder 1. Your task is to: put seme spraybottle on soiles.
go cabinet
On the cabinet 1, you see a eloth 2, soapbar 1, a spapbot:le 1.
go cabinet
The cobinet 2 is closed.
open cabinet
You open the cabinet 2. The cabinet is open. In it, you candle and spreybottle 2.
> take spraybottle 2 from cabinet 2
pick up the spraybottle
fzom the cabinet
> to teilet 1
on the toilst 1, you see a soapbottle
> put spraybottie in/on to' let 1
You put the spraybottle in/on the toilet 1.
Table 17, An example of annotated successful trajectory for ALFWorld ReAct
You are in the middle of a room. Leoking quickly around you, you see a cabinet 4, cabinet 3, a cabinet cabinet 1, countertop 1, garbagecan
handtowelholder 2, landtooelhooddrr 1, sinkbasin 2, a sinkbasin 1, a toilet 1, toiletpapechanger 1, and a towelaolder 1.
Your task to: put seme spraybottle on soilet.
> think: To solve the task, nead to find and take a aporybottle, then put it on toilet.
OK.
think: Pirst I need to find spraybottle. A spraybettle is more likely to appear in cabinet (1 4), countertop (1), toilet (1), sinkbasin (1-2), garbagecan (1). 1 cen check one by cne, starting with
cabinet
OK.
go cabinet
On the cabinet 1, you see cloth 1, soaphar 1, a soapbottle 1,
go to cabinet 2 The cabinet is closed.
open cabinet
cabinet open. In it, you see candle spreyboctie > think: Now 1 find spraybottle (2). Next, i need to take it.
You open the cabinet
take spraybattle
from cabinet
You pick up the spraysottle 2 from the cabinet 2.
think: Now I sake spraybottle (2). Next, I need to put it in/oa toilet 1.
OK.
> go to toilet
On the tot 1, you see soapbottle 2.
> put spraybottle 2 in/an toilet 1
You put the lprayhottla in/on the toilet 1.
During the interaction with the environment, two successful trajectories like in Table 17 are provided LLMs leverage the in-context learning ability of LLMs. The LLMs will generate action to be executed in the environment or conduct some reasoning about how to achieve the final goal. The chosen action and new observation are appended to the prompt for LLMs to form sequential decision-making process.
Similar to ALFWorld, we annotate some successful trajectories in Baby BlbyAl-Text to teach a ReAct agent how to complete a task in BabyAl-Text. Examples of successful trajectories without thoughts and with thoughts are shown in Table 18 and Table 19 respectively.
As to WebShop, adopt the implementation of ReAct and use one example for in-context learning. The successful example without thoughts and with thoughts are shown in Table 20 and Table 21 respectively.
C.2.2. RAP
RAP(Hao et al., 2023) utilizes LLMs as policy and world model, and adopts tree-search planning methods to evaluate each possible action candidate. Trajectories interleaving between actions and observations like in Table 16, Table 18 and Table 20 are provided LLMMs to learn how to choose action as an policy and the dynamics of the environment as a world model.
27

---

LLM-based Actor-Critic
Table 18. An example of successful trajectory for BlbyAl-Tex
Goal of the agent: go to the green key Coservat you see wall steps right, You see a wall steps forward, You aee a grey box 3 ateps left, You grey key 3 steps left and 1 step forward, You blue box steps left and steps forward
Action:Luzn left
Coservation: see well 3 steps right, You see a blue key 3 steps left and 2 steps Forward, You see step forward, see a green ball step left and steps forward, see grey box 3 steps forward, You see groy key 1 step right and 3 steps forward, You see a blue hox 2 steps right and steps forward Actionige forward
green key steps left
Coservation:Yon see wall 3 ateps right, You see a blue key 3 sceps left and 1 step forward, You aee green
green key 2 steps left, You forward, You see grey key 1 step right and 2 steps forward, You see a blue box 2 steps right and 2 steps forward
step and sleps Corward, You
grey steps
Action:turn left
Ooservation: You:Yoo see wall 3 steps left, You see green. box 2 sieps left and 3 steps forward, You see green box steps left and step forward, You green key 2 steps forward, You blue key atep right and 3 ateps Iorward, You see qreen ball 2 stepa right and 1 step forward, You aee a grey steps right
Actionige forward
ooservation see green steps aee green key 1 step forward, You see blue key step right and steps forward, You see green ball 2 steps sigh:
staps forward, You see green steps laft, You
Table 19. An example of annotated successful trajectory for BabyAl-Text in ReAct
Goal of the agent: go to the green key Doservat You see walll steps right, You see a wall steps forward, You aee a grey box 3 ateps left, You see grey key 3 ateps left and 1 step Lorward, You blue box steps left and ateps forward
Actionsthink: To complete the task, need find green key and go to it.
Coservation:00
Actionithink:
have not seen green key. sirstly, need find green key.
Oaservation:0x
Action:turn left
well 3 steps right, You blue key 3 steps left and 2 steps forward, You a green key staps left and step forward, you see a green ball step left and 3 steps forward, You grey hax steps forward, You gray key step right and steps forward, You blue hox 2 steps right and steps forward
Coservation:
Action:think: have seen green key 2 steps left and 1 step forward. Next, nead to go to the green
key.
Ooservation:08
Actionige forward
coservation:Yon see a well 3 steps right, You see a blue key 3 steps left and 1 step forward, You sea z green key steps left, You green ball step laft and steps forward, You gray box steps forward, You see grey key 1 step right and 2 steps forward, You see a blue box 2 steps right and 2 steps forward
Action:turn left
Ooservat ion: see well 3 steps left, You see a green. box 2 steps left and 3 steps forward, You see green box steps left and step forward, stop right and 3 steps forward, You see a green hall 2 stops right and stop forward, You see a groy steps right
green key 2 steps forward, You blue key
Action:ge forward ioservation see green hox steps laft steps forward, You see green box steps laft, You aee green key 1 step Iorward, You see blue key step right and steps forward, You see green ball 2 steps righ:
28

---

LLM-based Actor-Critie
Table 20, An example of successful trajectory for WebShop
Webshop
Instruction:
would like a ounce bottle bright citrus daadorant for sensitive akin, and price lower than 50.00
lars
[Search]
Action: search ounce bright citrus deodorant sensitive skin] Ooservation:
[Back to Search]
Page (Total results: 56)
>]
[BO786NBC1J]
Bright citrua Deodorant by Barth Mama Natural and Safe for Senaitive 8kin, Fregnancy and freding
Contains Organic Calendula 3 Cunce
$10.99
[#078CΙкVXY]
Ginger Fresh beodorant by Rarth Mana Natural and Safe for Sensitive skin, Pregnancy and Breast feedf ng, Organic Calendula Ounce
$10.99
[В08K3РЛАKN]
Barrel and Oak Aluminur, Pree Desdorent, Decdorant for Ken, Essential 0i1 Based Scent, 24-Hour Odos
Protection, Cedar #aschouli Blend, Gentle on Sensitive skin (Mountain Sage, 2.7 oz, 2-Pack) $15.95
Action: c1TC((1370
ooservation:
[Back to Search]
[< Frev]
scent (assorted scenta) [bright citrus) [calming lavendar] [ginger fresh) (simply non-scents] (4-pack)
[travel
(pack 1) [3-ounce (2-paok)]
Bright Citrus Deodoran: by Barth Mama Natural and Safe for Senaitive Skia, Fregnancy and Breast Conteins Organic
3-Ounce
price: $10.99
Rating: N.A.
[Descriptien] [Features]
[Reviewa] [Buy Now]
Action: (bright citrus]
Coservation: have eliexed bright estrus.
Action: click(3 ource (pack of 11] Ouservation: elicked
(pack
Action: click[ewy
Doservation: Your scare (nin 0.0, max 1.0): 1.0
29

---

LLM-based Actor-Critic
Table 21. An example of successful trajectory for WebShop in ReAct
Wehahop
Instruction ion:
like ounca bright citrus daodorent sensitive skin, price than
darlars
[9earch]
Action: search ounce bright citrus decdorant sensitive skin) Doservations
[Back Lo Bearch]
Page (Tota) results: 50)
[Next >]
[B078CNRC1J]
gright Litrus Deadorans Sarth Mama Natural and Safe for Sensitive Skin, Fregnancy and Breastfeeding,
Containg Organic calendula 3-Cunce
$10.99
[B0786IΚVXY]
Ginger Fresh Deodorant by Eazth Mana Natural and Safe for Sensitive skin, Pregnancy and Breastffeed.ng Conteins Organic Calendula 3 Ounce
$10.99
[*0екзчJ4XN]
Barrel ard Oak Alumf Aluminur-Preet Fur-Pree heodorant, Deodorant for Men, Rasential 61 -Rased Scent, 24-Hour Odor
Protection, Cedar Pacchouli Blend, Gentle Sensilive (Mountain Sage, 2.7 oz, Pack) $15.95
Action: think [B07ECRRC1J] and [3076БлкVXу] bright citrus deodczant less then 50 dollars. = check [#с7ввиис1]] first.] Oogervation: GX.
Action: c1TCX(1078
ooservation:
[Back to Search]
[< Frev]
scent (assorted scenta) [bright citrus) [calming lavendar] [ginger fresh) (simply non-scents] [4-pack)
[travel
(pack 1) [3-ounce (2-pack)]
Bright Citrus Deodoran: by Barth Mama Natural and Safe for Senaitive Skia, Fregnancy and Breain.e Conteins Organic
3-Ounce
price: $10.99
Rating: N.A.
[Description]
[Features]
[Reviewa]
[Buy Now]
Action: [For 3 ounce hottle of bright citrus deodorant for sensitiva skin, the item has options [oright citrus] [3 ounce (pack 11] seems good to buy.]
Coservation: Cx.
Action: (bright citrus] Coservation: You have elicked bright citrus.
Action: c3 ource (pack of 1;] Coservation: You have cliexed ounce (pack of 11.
Action: click[Buy Mow. Ooservation: Your (min max 1.0): 1.0
30

---

LLM-based Actor-Critic
The assessment of each step is performed by a reward function, which can be the log probability of the action self- evaluation given by LLMs, based some task-specific heuristics. In our implementation, actions given by LLMs the reward. For simplicity of implementation, we adopted greedy approach to expand the tree, generating only one action at time. More specifically, at each step. LLMs will sample some action candidates. For each action candidate, LLMs will generate a rollout trajectory until a maximum step or terminal state. The summation of log probabilities of all the actions on the rollout accessed by LLMs are used as Q value for each action candidate. The candidate with the highest Q value chosen be executed in the environment.
adopt the log probability of
C.2.3. ICPI
Table 22. An example provided to critic in ICPI for ALFWorld
Looking quickly around you, cabinet 2, a cabinet 1, countertop 1, a garbagecen L, handtowelholder 2, hand: 1, toilet 1, toiletpaperhanger L, and towellolder
You are in the middle of
cabinet 4, cabinet
sinkbasir. 2, sinkbasin Your task 1a to: put some spraybott on bot let. go cabinet
On the cabinet 1, you see cloth 3, soaphar 1, a soapbottle 1, Reward:0
cabinet
The cabinet is closed.
Roward:0
> open cabinet
open the cabinet 2. eabinet open. In it, you see candle sprayboctic
Reward:0
Lake spraybottle 2 from cabinet 2
You pick up the spraybottle 2 from the cabinet 2,
Reward:0
tot
On the toilet 1, you see aoapbottle 2.
Reward:0
put apraybottle in/on toilet
You put the spraybottle in/on the toilet 1.
Keward:l
Table 23. An example provided to critic ICPI for BabyAl-Text
Goal agent: go to the green key
Coservation:You wall 2 steps right, You see wall ateps forward, You see grey box 3 steps left, You sae a grey key 3 steps left and 1 step forward, You see blue box stops left and steps
Action:Lura left observation
well steps right, see a key steps and steps forward, You see green key staps laft and step forward, You see a grean ball step left and 3 stops forward, You see grey box steps forward, You see gray key step right and steps forward, see a blue box steps right and steps Tozward Reward:0
Action:go forward
Ooservation: You see well 3 steps right, You ses a blue key 3 sceps left and 1 step forward, You see green key steps left, You green step and steps forward, You grey steps forward, You see gray key 1 step right and 2 stops forward, You see a blue box 2 stops right and 2 ateps forward
Reward:0
Action:turn left Ooservation
see well steps left, You green. steps and steps Lorward, You green box 2 steps left and step forward, You see a green key 2 steps forward, You see a blus key 1 steps forward, You see green ball steps right and step forward, You see grey box 2. steps right
step right
Reward:0
Action:ge forward
Uoservation:Yon see a green bex steps left and staps forward, You see green box staps left, You see a green key 1 step forward, You blue key step zight steps forward, see green ball 2 steps righ:
ICPI (Brooks et al., 2024) proposes to implement policy iteration using LLMs through in-context learning. At each step, the policy in ICPI will sample some action candidates and the critic will compute the Q values for each action candidates. The action candidates with the highest Q values is chosen to executed.
The policy is implemented using L.Mss, and successful trajectories like in Table 16 and Table are provided to it.
31

---

LLM-based Actor-Critic
As to the critic, ICPI prompts LLMS to give the numerical reward for each step directly. Given the current history and an action candidate, the critic in ICPI will rollout trajectory starting from the action candidate. Apart from predicting the observations, the critic will give the numerical reward for each step on the rollout trajectory. The (discounted) return on the rollout is treated as the Q value for the action candidate. For both ALFWorld and AI-Txt, we define the reward as I when the agent reaches the goal. All other steps will have a reward O. The examples provided to the critic are like in Table 22 for ALFWorld and Table 23 for BlbyAl-Text.
C.2.4. RAFA
Table 24. An example provided to critic in RAFA for ALFWorld
You are in middle of room. Leoking quickly around you, you see a cabinet 4, cabinet a cabinet 2, a cabinet 1, a countertep 1, garbagecen i, handtowelholder 2, a hsndrowelhhider 1, sinkbasin 2, a sinkbasin 1, a toilet 1, toiletpapezhanger 1, and a towelaolder 1.
Your task is to: put some apraybott on toillet.
> critie: My taak requirea two sub-goals in order: take a apraybottle and put the spraybottle on the
toilet. Xy current state satisfies zero of the two sub goals. The value is 0/2-0.
> OK.
On the cabinet 1, you
aloth 4, soaphar 1, a spaphottle 1.
> OK.
The cabinet is closed.
> OR.
You open the cabinet 2. The cabinet 2 is open. In it, you see a candle 1, and sprayboltle 2, OK.
You pick up the spraybottle 2 from the cabinet 2,
eritia: Now take spraybott My current state satisfies the first of the sub-goala: take spraybottle. the value is 1/2=0.5.
OK.
On the tollet 1, you see aoapbottle 2.
> OK.
You put the spraybottle 2 in/on the toilet 1. > critic: NoM put the apraybottle on the toilet. My current state aatiafies all the two aub-geals. The value 2/2=1.
Table 25. An example provided to critic in RAFA for BabyAl-Text
Goal of the agent: go to the green key
You see wall steps right, You see wall 3 ateps forward, grey key 3 steps left and 1 step ferward, You see a. blue box 3 steps left and 2 steps Forward seritie: My task requires two sub-goals in order: find the green key, and go to the green key, My current state satisfies sero of the two sub goals. The value is 0/2-0. SUx.
see a grey box sceps left, You
You see wall steps right, You
blue key steps left and steps forward, You ateps left and atep forward, You see a greer ball step left and 3 steps forward, You see a grey box sLeps Iorward, You see a grey key 1 step zight and 3 steps Iorward, You see a blue box steps right and 3 steps forward
grean key
peritie: Now I find the green. key, My current state satiafies the first of the two aub aub-goales Cind the green key, The value is 1/2-0.5.
sux.
You see wall steps right, You
blue key steps left and step forward, You a green key 2 ateps left, You see a green ball 1 step left and 2 steps forward, You see a grey box 2 stepa forward, You see a grey key step right and steps ferward, You see bive box steps zigh: and 2 steps forward
Sox.
You see wall steps left, You
green box steps left and step forward, You see a greer kay 2 steps forward, You see a blue key 1 step right and stepa right and 1 step forward, You grey box steps right
green box steps left and steps forward, You
steps forward, You see green >OX.
You see a green box steps left and steps forward, You see a green box steps left, see a green key 1 step forward, You sea a blue key i step right and 2 steps forward, you see a green ball 2 steps right
seritic: Now go the green key. My current state satisfies all the two sub-goals. the value is 2/2=1.
The framework of RAFA (Liu et al., 2023) is also like RAP or ICPI. The main difference is how the action evaluation is conducted.
RAFA implements tree-search using LLM to evaluate each action candidate. Different from ICPI, RAFA uses the task completion progress as the value for each step. They have the LLMs decompose a goal into sub-goals, and use the completion status of the sub-goals after each step as the value for the step. RAFA evaluates the completion status of sub-goals based on the predicted observations. Examples provided in RAFA are like in Table 24 for FL. and Table 25 for BabyAl- Al-Text.
32

---

LLM-based Actor-Critic
C.2.5. LATS
LATS (Zhou al., 2024a) combines the reasoning, acting, and planning capabilities of LLMs with MCTS (Kocsis & Szepesvári, 2006) and external feedback mechanisms enhance maaiin,,, achieving competitive results in web navigation. We adopt the official implementation to evaluate its performance WebShop.
LATS cannot be directly applied to the other benchmarks we used for two main reasons. (I) LATS requires the ability to revert the agent to earlier states in the environment, which ALFWorld and BabyAl-Text do not support. LATS relies on model-free MCTS, using environment simulator as a world model, reverting simulators to earlier states during tree search. This limitation also noted in their original paper (Page 9). (2) While might be possible to modify these environments make them reversible, it would create an unfair comparison. Our method and other baselines do not rely on simulators during reasoning in ALFWorld and BabyAl- Aexte,, whereas, LATS would gain advantage from this modification.
Nevertheless, we still attempted to adapt LATS for ALFWorld by using LLMs as world models, similar to our method, for a fair comparison. The results. presented in the Table 26, show that LATS fails in almost all tasks. This is because its tree search severely depends on the environment simulator for precise state transitions. With only LL.M-based world models, the state transitions often deviate from the actual environments, due to LLMs' inherent hallucinations and partial observability of ALFWorld.
Table 26. Performance comparison of LAC (Ours) and LATS in ALFWorld.
CodeLlama-7B
Gemma-7B
Llama-3-8B
Mistral-7B
LAC (Ours)
0.79
0.84
0.78
0.79
LATS
0.00
0.00
0.03
0.00
C.3. Implementation details of LAC, ablation studies, and variants
Table 27. An example of annotated successful trajectory for ALFWorld in LAC
You are in the middle of a room. Leoking quickly around you, you see a cabinet 4, cabinet 3, a cabinet 2, a cabinet 1, a countertop 1, garbagscan
handtowelholder 2, handcowelholder 1, sinkhasir. 2, a sinkbasin 1, a toilet 1, toi toslepppperharger 1, and a towelholder 1,
Your task to: put some spraybott on coilec.
think: solve the task, need find and take aparybottle, then put it toilet. OK.
crities The thought is reasonable. This step is GOOD.
> think: Pirst I need to find a spraybottle. A spraybcttle is mere likely to appear in cabines (1-4), countertop (1), toilet (1), sinkbasin 11-2), garbagecan (1), cen check one by ene, starting with cabinet 1.
Ox.
critie: I naed to find sproynottle firstly. The thought is reasonable, Thts step is co0D.
> qo to cabinet 1
On the cabinet 1, you see eloth soapbar 1, a soapbotzle 1.
me> critie: 1 have not seen spraybottle in this place. This step is UNKNONE.
go to cabinet
The cabinet 18 closed.
eritie: The cabinet 2 is closed. This step is UNENOHN.
open cablnet
You open the cabinet 2. The cabinet 2 is open. In it, you see a candle 1, and sprayboltie 2. this place. This step is GOOD. think: Now I find spraybottle (2). Next, 1 need to take it. ok.
critie: 1 have seen spreyboztie
oritie: I can take the spraybeltle after rinding it. The thought is reasonable, This step is
GOOD.
take spraybottle 2 from cabinet 2
You pick up the spraybottle 2 from the cabinet 2,
critie: 1 have taken spsaybottle in this place. this step is GOLD. think: Now I take spraybottle (2). Naxt, I need to put it in/on toilat 1. ox.
eritie: can put the apraybollle in/on toilet after taking it. The thought is ressonable. This step is GOOD.
tcilet
On the toilet 1, you see a soapbottle 2.
critie: should go to toilet
want to put the aprayhottle on it. This step GOOD.
put spraybottle 2 in/on let 1
You put the spraybottle in/on the toilet 1.
critie: I have pus the spraybettle in/on the toilet 1. This step is GOOD.
33

---

LLM-based Actor-Critic
Table 28. An example of annotated successful trajectory for AI- in IAC
Goal of the agent: go to the green key
Doservation: see wall 2 steps right, You see a wall 3 ateps forward, You aee a critic box 3 stepa left, You see a critic key 3 steps left and 1 step forward, You see a blue bex 3 ateps left and 2 steps forward
Aotion:think: To complete the Lask, I need to Find a green key and go to it.
doservation:0K
critictihe though: is reasonable. This step is g0OD.
Action:think:
have not seen a green key. Firstly, need to find green key.
0o
critic:I need to find a green key firstly. The though: is reasonable. This step is G0OD.
Actioniturn left
Cnservation:Yon see a wall 3 ateps right, You sea a blue key 3 steps left and 2 ateps forward, You see gzeen key 2 steps left and step forward, Yeu see a green ball 1 step left and steps forwerd, You see critic box 3 steps forward, You see a critic kay 1 step right and 3 steps forward, You sae a blue hox 2 steps right and 3 stepa forward
critie:I have Lurned left and seen green key 2 sleps left and 1 step Lorward. This alep is GOOD. have seen a green key 2 sceps left and 1 step forward. Next, 1 need to go to the green
Action:think:
key,
Coservations
critic:i can go
green key after seeing it.
thought is reasonable. This step GOOD.
Action:ge forward
Coservation You see wall 3 steps right, You see a blue key 3 steps left and 1 step forward, You see green key steps left, You
green ball 1 step left and steps forward, You critie box 2 steps forward, You see a critic key i step right and 2 steps forwerd, You see a blue box steps right and steps forward
critic:T have gone forward. The green key la 2 steps left. This step is UNKNOWN. Actioniturn left
Ooservat ievatiionsie You see wall 3 steps left, You see greer box 2 steps left and 3 steps forward, You see green box steps left and siep Iorward, You see a green key 2 steps forward, You see a blue key 1 green ball 2 steps right and step forward, You eritic
step right and steps forward, You box 2 steps right
tr'ticsT turned left. The green key is 2 steps forward. This step is UNKNONN
Action:go [orward
Oogervation see green box steps left and staps forward, You see green box steps laft, You aee green key 1 step Iorward, You see blue key 1 step right and 2 steps forward, You see green ball 2 steps sigh:
critie:d have gene forward. The green key is step forward. This step is GooD.
We build our method based on ReAct (Yao al., 2023). Specifically, we adopt the in-context examples from ReAct and annotate reflections for those trajectories. For each task type in ALFWorld, we annotate two trajectories from ReAct as successful examples to teach LLMs how to complete this type of tasks. For BabyAl-Text, we annotate two trajectories with thoughts like in ReAct, as well as reflections, for each task type. As to WebShop, we annotate the example from ReAct with reflections, and we remove the thoughts annotated in ReAct because we found better results can be achieved without thoughts. The annotated trajectories for ALFWorld, Baby Ae-Text and WebShop are shown in Table 27, Table 28, and Table 29 respectively. The reflections we annotated are labeled in light orange. We also use ReAct algorithm collect some failed trajectories from the training set for each task type of ALFWorld and BabyAl-Text. The collected failed trajectories also annotated with reflections, aiming to teach LLMs to distinguish between steps that are beneficial, harmful, or indeterminate for achieving goal.
We annotate the reflections using the following criteria:
(1) If an action on a trajectory is necessary for achieving the final goal, the following reflections will end with "This step is GOOD." to indicate the completion of an essential step, such action "take spraybottle from cabinet 2" for the goal "put some spraybottle on toilet".
(2) If action is crucial for obtaining essential information to complete goal, the subsequent reflections will end with "This step is GOOD.", such action "open cabinet 2" in Table 27 which finds the target object.
(3) If an action is definitely useless incorrect for achieving goal, the following reflections will end with "This step is BAD." to indicate this action is undesirable, such as action "take cloth from cabinet 1" for the task in Table 27.
(4) If an action can not be evaluated as good or bad from the history, the following reflections will end with "This step is UNKNOWN.", such as action "go to cabinet 1" or "go to cabinet 2" in Table 27.
All the annotated successful and failed trajectories are used to construct fine-tuning examples to fine-tune LLMs to generate better reflections.
34

---

LLM-based Actor-Critie
Table 29, An example of successful trajectory for WebShop in LAC
Kebshop
Instruction:
would 1ike ounce lettle bright cuu deodorant for sensitive akin, and price Tower than 50.00
doilars
[Search]
Action: search ouace bright citrus decdorant sensitive skin] observation:
[Rack to Search)
Page (Total results: 58)
[Next >]
[B078C3RC1J]]
Bright Citrus Deodoran: by
Natural
Sensitive Skin, Fregnancy Breast.eedingg
Contains Organic Celendula 3-Dunce
$10.99
[R078CTRVXY]
Gingez Fresh Deodozant by Bazth Mana Natuzal and Safe for Sensitive Skin, Pregnancy and BreastCeedLag, Contains Organic Calendula 3-Dunce
$10.99
[ВOеКЗРЛ1XX]
sarrel and Oak Aluminum-Fres Deadorant, Decdorant for Ken, Essential cil-Based scent, 24-Hour Odor Protection, Cedar Paschouli Blend, Gentle Sensitive Skin (Mountain Sage, 2.7 oz, 2-Pack)
$15.95
Critic: The item zequized the instruction ion is "bright ciccus decdorant for sensitive skinf. I have searched '3 cunce bright citrus decdorant sensitive skin', which is similar the required iten. also see that [B07EGHRC13] is bright citzus devdorant. This step is GOOD.
Action: c1ick(=07cc
Doservation:
[Back to Search]
[< Frev]
(assorted scents) (bright citrus] [calming lavendar] [ginger fresh) (simply non-scents] size [travel set (4 (1-pack) [3 cunce (pack of [3 oonzee (2 (2-pack)) sright Litrus pesdoranz by Conteing Organic calendula 3-Cunce
Natural
Sensitive Skin, Fregnancy Breastfeeding,
Price: $10.99
Rating: N.A.
[Deccription]
[Features]
[Reviews]
[Buy Now]
CH tittc: have elkked [5078CRRC13] and found that this item is <bright eitrua deodorants and it can
satisty the attributes in Lastruction, which is (3 ounce! and "price lower than 50.00 dollars'. This is GOOD.
Action: (bright citrua]
Ooservation: You have cliexed bright citrus. critic: elickad [bright cirtrus] choose option that sotiafies the inatruction, step is GOOD.
Action: click[3
(pack of 1)]
coservation: You have clicked ounce (pack of 11.
olkad
lpack
choose option that satlaties irstruction, This
aLep is GOOD.
Action: click[Buy New! Coservation: Your score (min 0.0, max 1.0): 1.0 Critic: have bought the item that satisfies the inscruction. This step is GOUD.
35

---

LLM-based Actor-Critic
Considering the computational cost of fully finetuning of LLMs, we use LoRA (Hu al., 2021) finetune our models. In ALFWorld, with two successful trajectories and one failed trajectory for each task type, we have 485 (input,output) pairs in total of six task types to finetune models. In BabyAl-Text, the number of finetuning (input,output) pairs is 418. We finetune models for 1,000 steps with learning rate 2.5e-5 and batch size 2. We use A100 GPU with 80GB memory to fine-tune our model. With just about 400-500 (input,output) pairs and 1,000 fine-tuning steps, we can complete the fine-tuning process within one and a half hours.
During testing, the fine-tuned models are used to generate reflections after executing an action in the environment, as well as to forecast the potential outcomes of each action candidate.
The number of candidate actions n. is hyperparameter, set to 5 in our experiments, that is, we evaluate the top 5 candidate actions sampled from #LLM per state. It is worth noting that sampling top candidate actions introduces nonzero probability of missing the true argmax action, especially when distinctions among candidate actions are subtle. However, this choice is primarily driven by computational practicality: explicitly computing or evaluating the full action distribution for large or open-ended action spaces common in LLM-based recision-making is typically intractable. Empirically, we find that generating small subset of candidate actions from a strong LLM prior is often sufficient to include promising actions, thus making the trade-off between computational efficiency and accuracy acceptable.
After sampling action candidates, we use the fine-tuned model to predict future outcomes for each action candidate. The model needs to predict the possible observation and generate reflections for each predicted step. We set the maximum prediction step as 4, the model will continue the prediction until it generates reflection ending with "This step is GOOD." or "This step is BAD", or when it reaches the maximum prediction step.
For the optimization of TLLMs we solve an optimization problem in Equation (4) with a hyper- hyper--arameeter a, which balances the generating probabilities of FLLM and the values given by QIIM- For ALFWorld, we set a as 1, which yields superb performance over baselines. For BabyAI-Text. we conduct a grid-search over (1/2,1.2.5,10) for a, finding that different LLMs will have best performance with different a. The results can be seen in Table 12. For WebShop, we also conduct a grid search over (1)) to find the best a. Equation (4) is weighted combination of the original policy and the action evaluation values. It updates the distribution by adjusting the probabilities of the top candidate actions, while leaving the probabilities of other actions unchanged.
We set the maximum horizon length to 40 for ALFWorld, 30 for BIby and 15 for WebShop. If the agent has not reached the final goal after the maximum steps, this episode will be marked as failure.
We use A000 GPU with 80GB memory to evaluate method. For LAC, the execution time for ALFWorld is about 10 hours for 134 tasks using single A100 GPU. And for BabyAl-Text, the execution time can be varied for different task types, ranging from 4 to 10 hours for 50 tasks using one A100 GPU. For WebShop, 100 tasks will be completed within or 4 hours. The GPU memory usage may range from 15GB to over 70GB during the interaction according to the length of inputs to LLMs.
We compare our method with all the aforementioned baselines, demonstrating the effectiveness of our method on decision- making tasks with both high-level actions and lewelevel actions, even with potentially infinite actions. To demonstrate the effectiveness of each component in our method, we conduct ablation studies on each component. We removes the reflections from LAC as well as the integration during pre-action-geeneraton phase. This variant is called LAC w/o reflection. We also evaluate the role of QLIM by removing it from LAC as well as the integration during post-action-generatii phase. This variant is called LAC w/o critic. We also demonstrate the role of the action prior given by LLM policy by using only critic QLLM for decision-making. We call this variant as critic-only. The execution time of those variants during evaluation can be varied according to its performance because method having poor performance typically will cost more time to execute. On ALFWorld, it may be 10-20 hours. The comparisons between those variants are shown in Figure 4.
We found that each component in LAC is crucial for the superb performance. Removing some components may lead to wrong choice of action candidates. Such an example is shown in Table 30. LAC can complete this task successfully, while eliminating some components in LAC will lead to failure. The comparison is shown in Figure 10.
36

---

LLM-based Actor-Critic
Table 30. An example of successful trajectory in test environment in Baby given by LAC
Coal the agontigo to tha green ball
Coservation You see wall 5 steps forward, You see wall 2 steps left, You see yellow ball 1 step left and steps forward, You see blue ball step left and 3 ateps forward, You see a red ball 1 step left and step ferward, You steps right and 1 step forward, You see a blue kay stops right and stap forward
grey key 2 steps right and 2 steps ferward, You green key 2 Action:thinkee comp lete the task, ouservation:02
find green ball and go it.
ce tieiThig thought is reasonable. This step is 6000. Aetion:think:I have seen green ball atep left and 4 stepa Corward. Next, I need to go to the green ball.
Uoservation:ux
triticiThig thought is reasonable. I car go to the green hall after seeing it. This step is GOoD.
Action:turn right
Coservation: see well 5 steps forward, You see well 2 steps right, You see a green ball 3 steps left and steps forward, You see grey key 2 steps left and 2 ateps forward, You see a blue key 1 step left and steps forward, You
green key step left and 2 ateps Ierward, You see blue ball 1
step right and step forward
Critic:l have turned right. The green ball a steps left and 4 steps forward, This step is UNKNOWS. Action:go forward
Ouservation: see wall steps [orward, You see green ball steps left and steps Torward, see grey key 2 steps left and step forward, You see a blue key 1 step left and steps forward, You see a green key 1 step left and step forward, You see a blue ball 1 step right Critic:l have gone forward. The green ball 3 steps left and 3 steps forward, This step is UNKNOWN. Action:ge forward ooservation
wall steps forward, You left and 2 steps forward, You see a grey key 2 steps left, You see a blue key 1 step left and 1 atep forward, You
wall 2 atops right, You
green ball steps
green key step left
critierd have gone forward. the green ball is 3 steps left and 2 steps forward. This step is UnKNOws. Actionigo forward
Ooservation:Yon
well 2 steps forward, You left and step forward, You see a blue key 1 step left
wall 2 steps right, You green ball 3 steps
cr tiitt have gona forward. The green nall 1a steps left and step forward. This step UNKNOWN,
Action:ge forward
coservation:You see wall 1 step forward, You see wall 2 steps right, you see a green ball steps left
Criticiest have gone forward. The green ball is steps left. This step is UNKNUWN.
Action:turn left
Coservation
grey key steps left and steps forward, You see green key 2 steps left and 1 step forward, You see blue key 1 step left and 1 step forward, You see red ball steps forward, You see green ball 3 steps forward
wall step right, You
criticiBst hava turned laft. The green. ball is 3 stops forward. This stop in UNKNOKN. on:gon forward
coservation:Yon
wall 1 step right, you green key 2 ateps left, You see blue key step left, You see a red hall 3 stepa forward, You see green ball stepa [orward
gray key steps left and step forward, You see
Critic:But I have gons forward. The green ball is 2 steps forward. This step is UNKNOWN. Action:ge forward
Onservationn You see wall 1 step right, You see gray key steps left, You see a red ball 2 steps forward, You see a green ball 1 step forward CriticiBst have gona forward. The green hall in step forward. This stop coop.
37


Reviews:
[{"review_id": "nqKGmNDRpr", "reviewer": "Reviewer_Zqc7", "summary": "To enable planning with a next-token generation autoregressive model, this paper proposes to evaluate each action by a critic.\nRather than directly self-judging actions, the critic ranks them based on the output logits, associated with the likelihood of predicting actions being good or bad. The policy is then updated by selecting the action with the highest logit value.\nThis approach is tested on several tasks, including AFAWorld, BabyAI, and Webshop. Across all models, it consistently demonstrates dominant performance over other baselines.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your comments and valuable suggestions. Here we provide detailed explanations and experimental results to address your questions.\n\n> `Q1: Questions regarding gradient-free policy updates: (1) How many actions are evaluated per state? (2) Is the policy updated after each action, or in batch updates? (3) How are policy parameters updated, particularly in the case of an infinite action space?`\n\n**A:** (1) The number of candidate actions is a hyperparameter, set to 5 in our experiments, that is, we evaluate the top 5 candidate actions per state. Empirically, evaluating 3-5 top actions is generally sufficient for the benchmarks used, as these actions are sampled by leveraging LLMs' prior knowledge. In most cases, effective actions are included within this set. \n\n(2) The policy is improved by reweighting with Eq.5 after evaluating all sampled candidate actions for the current state. \n\n(3) We do not update policy _parameters_ directly. Instead, we derive a new policy from the original one using a closed-form solution (Eq.5-6) of the policy improvement objective (Eq.4). For infinite action spaces, we evaluate the policy by assessing a set of top candidate actions, similar to how we handle finite action spaces.\n\n> `Q2: Report performance without finetuning for comparison and show in a separate figure that the performance can be further enhanced.`\n\n**A:** Thank you for the suggestion. We will include comparisons without finetuning in the revised version of the paper. The results, presented in the table below, show that our method outperforms the baseline both with and without finetuning, and the performance can be further improved with finetuning. With finetuing, our method provides more consistent performance with different LLMs.\n\n**Table 1:** Ablation studies on the impact of finetuning.\n\n| Methods / Models       | CodeLlama-7B | Gemma-7B | Llama-3-8B | Mistral-7B |\n| ---------------------- | :----------: | :------: | :--------: | :--------: |\n| LAC (w/o finetuning)   |     0.39     |   0.59   |    0.71    |    0.57    |\n| ReAct (w/o finetuning) |     0.20     |   0.54   |    0.31    |    0.34    |\n| LAC  (w/ finetuning)   |     0.79     |   0.84   |    0.78    |    0.79    |\n| ReAct (w/ finetuning)  |     0.38     |   0.70   |    0.73    |    0.65    |\n\n> `Q3: How can a proportional policy be updated without gradients?`\n\n**A:** We update the distribution by adjusting the probabilities of the top candidate actions, while leaving the probabilities of other actions unchanged. Specifically, we formulate the policy improvement objective as a KL-constrained optimization problem (Eq.4). We then derive a closed-form solution (Eq.5) to this problem, which is a weighted combination of the original policy and the action evaluation values.\n\n> `Q4: What does maximizing Q mean if it does not represent cumulative rewards? Why optimize Q-values instead of maximizing the prediction probability?`\n\n**A:** In our method, maximizing the Q-function corresponds to maximizing the success probability for the current task. The benchmarks simulate realistic scenarios where there is no immediate reward during execution, and the environment only provides feedback on task completion (success or failure) at the end of each episode. To model this, we design the Q-function to be positively correlated with success probability using a Sigmoid function (Eq.1), ensuring that maximizing Q effectively maximizes the success probability.\n\nWhile directly maximizing the success probability is also valid and could lead to a different Q-function formulation, we compared our approach against it in Appendix A.2. Our method outperformed it in most tasks and models. We speculate this is because our method leverages more internal information from LLMs, utilizing both success and failure probabilities (as derived in Eq.2), resulting in more accurate and stable action evaluations. While there could be other formulations that use additional internal information, our approach remains both simple and effective.\n\n> `Q5: In Equation 4, what is the distribution of the history h?`\n\n**A:** The history $h_t$ in Eq.4 is not a distribution, but rather a given context for the policy and critic.\n\n> `Q6: What task is depicted in Figure 5? Why does LAC require fewer steps?`\n\n**A:** The task depicted in Figure 5 is from ALFWorld. The metric \"Steps per task\" is calculated as an average over both successful and failed tasks. LAC requires fewer steps due to its higher success rate, enabling it to complete tasks within the maximum step limit, while other baselines often reach this limit without completing the tasks. If we only consider successful tasks, the step cost is similar across methods: Ours: 15.32 steps, ReAct: 17.75 steps, and RAP: 16.36 steps.\n\nThanks again for your efforts and insightful comments! We hope our clarification addresses your concerns. Any further feedback and discussions are much appreciated.", "claims_and_evidence": "As claimed, the paper presents a method to leverage LLMs' prior knowledge for action evaluation by using prediction logits to classify actions as good or bad.\n\nLong-term planning is achieved by annotating actions as good or bad based on their effects on the final goal, as described in Appendix C.3.\n\nHowever, the process of gradient-free policy updates remains unclear. In Equation 5, the new policy is obtained by multiplying exponentials of Q-values, but several questions arise: (1) How many actions are evaluated per state?\n(2) Is the policy updated after each action, or in batch updates?\n(3) How are policy parameters updated, particularly in the case of an infinite action space?", "methods_and_evaluation_criteria": "The proposed method effectively enables long-term planning by evaluating actions and updating policies with fewer steps while achieving enhanced performance.\n\nEvaluation is conducted on several control tasks, measuring success rate, rewards, and computational requirements across various base models. Additionally, ablation studies are also thoroughly performed.", "theoretical_claims": "No, I did not check the proof.", "experimental_designs_or_analyses": "For experimental results shown in Figure 2, the proposed method, LAC, is finetuned, while other baselines are not. As shown in Figure 9, the performance of ReAct can indeed be improved with further finetuning. The comparison is unfair. Is it possible to report performance without finetuning for comparison and show in a separate figure that the performance can be further enhanced?", "relation_to_broader_scientific_literature": "The paper is well fitted into the literature with enough baselines chosen.", "essential_references_not_discussed": "Not to my knowledge.", "other_strengths_and_weaknesses": "See other sections.", "other_comments_or_suggestions": "See other sections.", "questions_for_authors": "1. How can a proportional policy be updated without gradients?\n\n2. What does maximizing Q mean if it does not represent cumulative rewards? Why optimize Q-values instead of maximizing the prediction probability?\n\n3. In Equation 4, what is the distribution of the history h?\n\n4. What task is depicted in Figure 5? Why does LAC require fewer steps?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dTQeGeB1ml", "reviewer": "Reviewer_e1C1", "summary": "The paper proposes LAC, a framework for improving decision-making capabilities of LLMs by integrating base LLM with action evaluations derived from token logits and trajectory rollouts. The authors conduct experiments across ALFWorld, BabyAI-Text, and WebShop benchmarks and show the superiority of LAC over baselines.\n\n## update after rebuttal\n\nThanks for running the additional evaluation on Crafter. Based on the new results, it seems there is overlap between the Naive baseline and LAC. Overall, I find the idea interesting, but my main concern remains about LAC's effectiveness on more complex tasks. Therefore, I will keep my current score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your comments and valuable suggestions. Here we provide detailed explanations to address your questions.\n\n> `Q1: Lack of discussion on failure cases. It would be nice to have a small section on the failure cases of using Q_llm (are there any specific kinds of tasks/trajectories where Q_llm doesn't generalize?)`\n\n**A:** We have provided illustrative examples from ALFWorld and BabyAI-Text in Figures 10 and 11, respectively. In these cases, the critic ($Q_{LLM}$) alone may occasionally fail to select the correct actions. This is because the estimation of Q-values relies on the LLM's ability to predict future trajectories. When the LLM experiences significant hallucinations, the Q-values can deviate, leading to suboptimal action selection. \n\nFor example, in Figure 10, the agent needs to find and take a \"saltshaker\". In step 2, the critic suggests \"go to drawer 1\", because it incorrectly predicts that the saltshaker is there, resulting in a suboptimal action. In contrast, the prior policy suggests a more systematic search, starting with \"cabinet 1\" and then moving to \"cabinet 2\", where the target saltshaker is actually located.\n\nSimilarly, there are failure cases where incorrect action evaluations of critic lead to unsuccessful task completion. For instance, in the task \"find two soapbars and put them in the cabinet\", the critic mistakenly identifies a \"soapbottle\" as a \"soapbar\", causing the agent to take the wrong object and fail the task.\n\nIn summary, the critic may struggle to generalize in cases where the LLM suffers from significant hallucinations, such as mis-predicting future trajectories due to partial observability or incorrectly identifying target objects due to the base LLM's inherent hallucinations. Thank you for the suggestion, and we will expand the discussion on failure cases in the revised version.\n\n> `Q2: In related works, it would be nice to cite works like: Tree search for language model agents. arXiv preprint arXiv:2407.01476.`\n\n**A:** Thank you for pointing out this relevant work. We will include this work in the revised version of our paper.\n\n> `Q3: [Minor] The approach mostly combines existing ideas rather than introducing fundamentally new concepts. This is a minor concern of mine.`\n\n**A:** While our method draws inspiration from the actor-critic algorithm in classic RL, it is distinguished by two key contributions that address significant challenges in enhancing LLMs' decision-making capabilities: (1) We propose a novel Q-function estimation approach (Sec. 4.1) to extract action evaluation information from LLMs that leverages LLMs' internal belief about success or failure of the current task; (2) We formulate the policy improvement problem as a KL-divergence constrained optimization and derive a closed-form solution (Sec. 4.2), allowing us to optimize the policy in a gradient-free manner using the action evaluation.\n\n\n> `Q4: How does the method perform with reasoning-LLMs like DeepSeek-R1 Distill Llama or Qwen?`\n\n**A:** We have conducted preliminary experiments with reasoning LLMs like DeepSeek-R1-Distill-Qwen-7B. However, we observed that they often tend to overthink rather than output direct environmental actions in both our method and the baseline ReAct. For instance, even when we explicitly prompt the reasoning LLMs to output actions (e.g., \"Please make sure your response is in the following format:\\n> {The action you choose}\"), the models still generated detailed explanations but avoided selecting the next action. A typical response might be: \"I need to find a key to open the safe or locate the pencil in the drawers. Since I can’t (…), I’m unable (…), I must (…)\". This issue has also been noted in prior work [1]. We believe that using reasoning LLMs for decision-making tasks requires deeper exploration to balance internal reasoning with effective environmental interaction.\n\n\n> `Q5: Can LAC be applied to more challenging environments (WebArena or BALROG), and multimodal decision-making tasks (VisualWebArena)?`\n\n**A:** Thank you for the suggestion. Due to the limited time and computing resources available during the rebuttal period, we were unable to conduct experiments on these benchmarks, which generally require larger base LLMs for better performance. Given that our method has already demonstrated effectiveness in various complex sequential decision-making tasks (e.g., ALFWorld, BabyAI-Text and Webshop), we believe it can be extended to these benchmarks as well. We will explore this direction in future work.\n\nThanks again for your efforts and insightful comments! We hope our clarification addresses your concerns. Any further feedback and discussions are much appreciated.\n\n---\n\n[1] Cuadron, Alejandro, et al. \"The Danger of Overthinking: Examining the Reasoning-Action Dilemma in Agentic Tasks.\" arXiv preprint arXiv:2502.08235 (2025).", "claims_and_evidence": "The paper's claims about LAC improving LLM decision-making are generally supported by the experimental results across ALFWorld, BabyAI-Text, and WebShop benchmarks. The experiments show consistent performance improvements over baselines. The ablation studies adequately demonstrate that each component contributes to the overall performance. However, the significance of these improvements is somewhat limited by the simplicity of the chosen benchmarks, which may not sufficiently challenge newer LLMs.", "methods_and_evaluation_criteria": "- The evaluation criteria and benchmarks (ALFWorld, BabyAI, WebShop) are now too simple for latest LLMs. Testing on more challenging environments like WebArena [1] or BALROG [2] would provide a more compelling evaluation. \n- Not a concern but it would be interesting to see how the method performs with reasoning-LLMs like DeepSeek-R1 Distill Llama or Qwen which might also be suitable for the policy network (π_LLM)\n\n\n[1]: Zhou, S., Xu, F. F., Zhu, H., Zhou, X., Lo, R., Sridhar, A., ... & Neubig, G. WebArena: A Realistic Web Environment for Building Autonomous Agents. In The Twelfth International Conference on Learning Representations.\n[2]: Paglieri, D., Cupiał, B., Coward, S., Piterbarg, U., Wolczyk, M., Khan, A., ... & Rocktäschel, T. (2024). Balrog: Benchmarking agentic llm and vlm reasoning on games. arXiv preprint arXiv:2411.13543.", "theoretical_claims": "yes - appendix B.1", "experimental_designs_or_analyses": "- The experiments are generally sound with appropriate comparisons to relevant baselines and comprehensive ablation studies.\n- The computational analysis is also useful", "relation_to_broader_scientific_literature": "Improving decision-making capabilities of LLMs is a very relevant problem statement in today's times and this paper aims to improve the capabilities of LLMs in order to become better agents", "essential_references_not_discussed": "In related works, it would be nice to cite works like: Koh, J. Y., McAleer, S., Fried, D., & Salakhutdinov, R. (2024). Tree search for language model agents. arXiv preprint arXiv:2407.01476.", "other_strengths_and_weaknesses": "## Strengths:\n- Well-written and easy to follow with clear explanations\n- Comprehensive experiments and ablation studies that isolate the impact of each component\n\n## Weaknesses:\n\n- [Minor] The approach mostly combines existing ideas rather than introducing fundamentally new concepts. Although I personally feel a lot of agentic LLM applications are more about implementation and orchestration and less about research novelty - so this is a minor concern of mine.\n- Evaluation on relatively simple benchmarks that don't sufficiently challenge newer LLMs - as I said above, I would like to see the results on either WebArena or BALROG or another challenging benchmark\n- Lack of discussion on failure cases - Since agentic LLMs are already seeing rapid adoption in applied use-cases, it would be nice to have small section on the failure cases of using Q_llm (are there any specific kind of tasks/trajectories where Q_llm doesn't generalize?)", "other_comments_or_suggestions": "Line 135 column 2: typo: typo: \"few-show\" -> \"few-shot\"", "questions_for_authors": "1. see \"Other Strengths And Weaknesses\"\n2. Curious if LAC can also be extended to multimodal LLMs for decision-making in multimodal benchmarks like VisualWebArena [1]?\n\n[1]: Koh, J. Y., Lo, R., Jang, L., Duvvur, V., Lim, M. C., Huang, P. Y., ... & Fried, D. VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks. In ICLR 2024 Workshop on Large Language Model (LLM) Agents.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "te7CGLUqEb", "reviewer": "Reviewer_WfPB", "summary": "This paper proposes a gradient-free Actor-Critic framework (LAC) to enhance the decision-making capabilities of LLMs. LAC integrates a value-based critic that offers quantitative feedback to guide policy improvement and employs a gradient-free optimization approach to update actor. Experimental results demonstrate that LAC outperforms GPT-4 with ReAct in Alfworld and BabyAI-test tasks.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> `Q1: Lack of intuitive explanations and in-depth analysis.`\n\n**A:** We appreciate the reviewer's emphasis on intuitive explanations and analyses. In Fig.10 and 11 (Appx.A.7), we have provided detailed illustrative analyses using representative examples from ALFWorld and BabyAI-Text, which demonstrates into how each component of our actor-critic framework (LAC) influences action selection.\n\nFig.10 presents a concrete scenario where the agent's goal is to \"put a saltshaker in the drawer.\" At a critical decision step (Step 4), we observe the following intuitive distinction between components. \n\n- LLM-based prior policy alone mistakenly suggests \"go to drawer 1\" because the base LLM model overlooks that the agent has already found the correct object (\"saltshaker 1\") in cabinet 2. This error exemplifies the common hallucination problem in LLMs, which occurs when the model disregards previous states and incorrectly recommends irrelevant actions.\n- In contrast, the critic suggests \"take saltshaker 1 from cabinet 2\" because it evaluates potential actions by predicting future trajectories and determines that this action will successfully pick up the correct object.\n- Our method leverages these distinct insights by optimizing the prior policy's action distribution based on the critic's evaluation (see Lines 4-8 in Algo.1). It effectively corrects the errors introduced by the prior policy, balancing the strengths of prior policy (flexible but sometimes inaccurate) and critic evaluations (accurate but computationally intensive).\n\nThis illustrative example explicitly reveals why integrating actor (prior policy) and critic leads to substantial performance improvement. \n\n> `Q2: The difference of the actor-critic framework between the decision-making scenario and the general LLM reasoning scenario.`\n\n**A:** We respectfully disagree with this claim. \n\nCompared to classic RL actor-critic algorithm, our method is distinguished by two key contributions that specifically address significant challenges in enhancing LLMs' decision-making capabilities: (1) To effectively extract action evaluation information from LLMs, we propose a novel Q-function estimation approach (Sec.4.1) that leverages LLMs' internal belief about success or failure of the current task; (2) To efficiently utilize these action evaluation insights for policy optimization, we formulate the policy improvement problem as a KL-divergence constrained optimization and derive a closed-form solution (Sec.4.2), allowing us to optimize the policy in a gradient-free way.\n\nCompared to the general LLM reasoning scenario, our method introduces two novel features that are specially designed for sequential decision-making problems: (1) To support long-term planning, our critic module first predicts possible future trajectories for action evaluation, which significantly improves the accuracy of action-value estimation. (2) To improve LLM's decision-making ability, our method introduces a deliberate and principled integration of LLMs as both prior policy (actor) and action evaluation (critic). Rather than simply prioritizing one over the other as in prior work, our approach synergistically combines the two, achieving a balanced framework optimized explicitly for sequential decision-making scenarios.\n\n> `Q3: LATS's performance in other benchmarks.`\n\n**A:** LATS cannot be directly applied to the other benchmarks we used for two main reasons. (1) LATS requires the ability to revert the agent to earlier states in the environment, which ALFWorld and BabyAI-Text do not support. LATS relies on _model-free_ MCTS, using environment simulator as a world model, reverting simulators to earlier states during tree search. This limitation is also noted in their original paper (Page 9). (2) While it might be possible to modify these environments to make them reversible, it would create an unfair comparison. Our method and other baselines do not rely on simulators during reasoning in ALFWorld and BabyAI-Text, whereas LATS would gain an advantage from this modification.\n\nNevertheless, we still attempted to adapt LATS for ALFWorld by using LLMs as world models, similar to our method, for a fair comparison. The results, presented in the table below, show that LATS fails in almost all tasks. This is because its tree search severely depends on the environment simulator for precise state transitions. With only LLM-based world models, the state transitions often deviate from the actual environments, due to LLMs' inherent hallucinations and partial observability of ALFWorld. \n\n**Table 1:** Performance comparison of LAC (Ours) and LATS in ALFWorld.\n\n|Methods / Models|CodeLlama-7B|Gemma-7B|Llama-3-8B|Mistral-7B|\n|-|:-:|:-:|:-:|:-:|\n|LAC (Ours)|0.00|0.00|0.03|0.00|\n|LATS|0.79|0.84|0.78|0.79|\n\nThanks for your efforts and insightful comments! We hope our clarification addresses your concerns and sincerely appreciate it if you could re-evaluate our work. Any further feedback is much appreciated.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Yes.", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "The proposed framework demonstrate the potential of 'actor-critic' for LLM-based decision-making scenario.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths:\n1. This paper proposed a token-level action evaluation method, and utilized a gradient-free method to update policy LLM. The idea is totally makes sense and Concise.\n2. The experiment setting and provided implementation details are comprehensive.\n\nWeakness:\n1. Lack of In-depth analysis and explanation. This work constructs a tailored actor-critic framework in decision-making scenario, and the performance improvement results is promising. Nevertheless, in the experimental results part (i.e., sections 5.2, 5.3, 5.4), we didn't see a very illustrative explanation to reveal the potential reason behind this improvement. This limits the impact of this paper as basically, the actor-critic framework is somehow similar with the general domain (e.g., reasoning).\n2. From this paper, i cannot see the technical difference of the actor-critic framework between decision-making scenario and the general LLM reasoning scenario. It seems that this paper just simply adapts the general actor-critic framework into decision-making task without domain-specific justification or design. This limits the overal novelty of this paper.\n3. The overall paper writing is barely satisfactory; some parts lack intuitive explanations, making it difficult for readers to understand.", "other_comments_or_suggestions": "1. More in-depth analysis of the experiment results.", "questions_for_authors": "1. LATS is also one of the baselines, which uses external feedback and the MCTS method to improve performance. Why did the author only report its' performance in the WebShop benchmark? Can it be adapted to another benchmark?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: In this work, the authors propose LLM-based Actor-Critic (LAC), a framework designed to improve LLM-based agents' sequential decision making capabilities by leveraging LLM's own belief to the world as a critic. There are two core components: a Q function estimation method, leveraging the the LLM's internal belief rather than relying on hand-crafted reward design; and a gradient-free policy optimization method via KL-constrained policy optimization. 

The reviewers' concerns were mainly regarding two aspects:
1. clarity of the methodology, especially in terms of its connection and difference with traditional actor-critic in RL literature. 
2. simplicity of the datasets/environments used to demonstrate LAC, insufficient evidence how LAC could generalize to more realistic tasks. 

The authors made efforts on addressing both aspect:
1. There are extensive discussion between the authors and reviewers on the first point, especially the conversation between reviewers and Reviewer WfPB. In my opinion the authors did a great job, the additional statistical analysis trying to understand how each key component (Q-function estimation and gradient-free policy optimization) really works under the hood really strengthens the work, providing stronger evidence in addition to the experimental results and analyses in the paper. Since there are multiple reviewers mentioned the qualitative analysis, I suggest the authors to improve Appendix A.7 (currently only a single sentence) and somehow move it to the main paper. 
2. For the second point, the authors conduct new experiments on the Crafter environment (part of BALROG, as suggested by Reviewer e1C1). I fully acknowledge that given the short period of time, the authors have already done a good job following the reviewers' suggestions. However, given the fact that 1) Crafter is arguably another simple environment (2d grid world), it is even simpler than ALFWorld in many ways; 2) the new results suggest an overlap between the performance of the Naive baseline and LAC (considering variance), I am less convinced that this particular additional results fully addressed concern 2. 

Overall the work has merits, the authors did a good job addressing most of the reviewers' concerns.
Decision: Accept (poster)