Empowering LLM Agents with Zero-Shot Optimal Decision-Making through Q-learning
Authors: Jiajun Chai
Sicheng Li
Yuqian Fu
Dongbin Zhao
Yuanheng Zhu
Venue: ICLR 2025 Poster (2025)
Keywords: Large language models
Agent
Optimal decision-making
Primary area: applications to robotics, autonomy, planning
TL;DR: We achieve zero-shot optimal decision-making for LLM agents by integrating the respective advantages of LLMs and RL.

Abstract: Large language models (LLMs) are trained on extensive text data to gain general comprehension capability. Current LLM agents leverage this ability to make zero- or few-shot decisions without reinforcement learning (RL) but fail in making optimal decisions, as LLMs inherently perform next-token prediction rather than maximizing rewards. In contrast, agents trained via RL could make optimal decisions but require extensive environmental interaction. In this work, we develop an algorithm that combines the zero-shot capabilities of LLMs with the optimal decision-making of RL, referred to as the Model-based LLM Agent with Q-Learning (MLAQ). MLAQ employs Q-learning to derive optimal policies from transitions within memory. However, unlike RL agents that collect data from environmental interactions, MLAQ constructs an imagination space fully based on LLM to perform imaginary interactions for deriving zero-shot policies. Our proposed UCB variant generates high-quality imaginary data through interactions with the LLM-based world model, balancing exploration and exploitation while ensuring a sub-linear regret bound. Additionally, MLAQ incorporates a mixed-examination mechanism to filter out incorrect data. We evaluate MLAQ in benchmarks that present significant challenges for existing LLM agents. Results show that MLAQ achieves a optimal rate of over 90\% in tasks where other methods struggle to succeed. Additional experiments are conducted to reach the conclusion that introducing model-based RL into LLM agents shows significant potential to improve optimal decision-making ability. Our interactive website is available at http://mlaq.site.

Published as a conference paper at ICLR 2025
EMPOWERING LLM AGENTS WITH ZERO-SHOT OPTI- MAL DECISION-MAKING THROUGH --LEARNING
Jiajun Chai 1.2, Sicheng Li 1.2, Yuqian Fu 1.2 Dongbin Zhao 1,2,; Yuanheng Zhu 1,2,+
Institution of Automation, Chinese Academy of Sciences
2 School of Artificial Intelligence. University of Chinese Academy of Sciences
jaaiji2aj22200 lisicheng2024, Euyuqian2022, yuanheng. zhu, dongbin zhao) ) ) @ia... .c..cn
ABSTRACT
Large language models (LLMs) are trained on extensive text data to gain gen- eral comprehension capability. Current LLM agents leverage this ability to make zero- or few-shot decisions without reinforcement learning (RI.) but fail in mak- ing optimal decisions, as LLMs inherently perform next-token prediction rather than maximizing rewards. In contrast, agents trained via RL could make opti- mal decisions but require extensive environmental interaction. In this work, we develop an algorithm that combines the zero-shot capabilities of LLMs with the optimal decision- makinn of RL, referred as the Model-based LLM Agent with Q-Learning (MLAQ). MLAQ employs Q-learning derive optimal policies from transitions within memory. However, unlike RL agents that collect data from envi- ronmental interactions, MLAQ constructs imagination space fully based on LLM to perform imaginary interactions for deriving zero-shot policies. Our proposed UCB variant generates high-quality imaginary data through interactions with the LLM--ased world model, balancing exploration and exploitation while ensuring a sub-linear regret bound. Additionally, MLAQ incorporates a nixad-exaainntionn mechanism to filter out incorrect data. We evaluate MLAQ in benchmarks that present significant challenges for existing LLM agents. Results show that MLAQ achieves a optimal rate of over 90% in tasks where other methods struggle to succeed, Additional experiments are conducted to reach the conclusion that intro- ducing model-based RI. into L.M agents shows significant potential to improve optimal decision- ability. Our interactive website available at this link
INTRODUCTION
Large language models (LLMs) exhibit impressive comprehension capabilities and are widely utilized to address pocision-making tasks an LLM agent(Guuo et al. 2024) in zero- or few-shot manner, meaning that the agents make decisions without or with minimal interactions with the environment. In this work, we aim to further empower LLM agents with optimal decision decision-making via reinforcement learning (RL). Most existing LLM agents improve decision decision-making making through prompt optimization (Wei et al. 2022 Mandi al. 2023), ) which heavily relies the basic LLMs, lacking the ability to maximize rewards. In contrast, recent studies have introduced the Markov Decision Process (MDP) framework to leverage MDP-based planning (Hao al. (023). Figure I (a) illustrates typical framework for MDP-based LLM agents, in which the domain description is manual in natural language explain the problem to be solved (Yao et al. 2023a). For clarifications, domain represents an environment characterized by unique state and action spaces (e.g., block cubes and robotic arm control are different domains), while a task within a domain entails an original-target state pair (e.g., assembling beef sandwich and bacon sandwich are different tasks). Among existing LLM agents, the most common framework combines an Monte Carlo Tree Search (MCTS (Kocsis Szepesvári] 2006))- Pllannerr with a Replay Buffer to derive the optimal policy (Zhang al. (022d)) The replay buffer is task-specific, which stores transitions within the same task, sourced either from the environmental interaction (Murthy et al. 2023 Ding al. 2023) or the imaginary interaction
Corresponding Authors

---

Published as a conference paper at ICLR 2025
actions Bine. Dynantc fune. Reward
MLAQ Agent
Descrption
Stze
Q-Planner
Brite Domain Descrigtion
Fay. Tools
Primpt
based Imagination Space
Eus
Acthen
+ upcate
but of Simpher' Dimena foad
Seff-Rxamins
Replay
World Model
Busic Policy
Eapand
LLM based Imagiination
Actisn
Comoct
Ageni
Action
Eav
Re at-utiize
Mennery
Replay Ruflier
(a)
Memory
Expana
Basic Polley
Predicted Stare
Mudel
World
Figure I: (a) A typical framework of MDP-based LLM agent, where the gray arrows represent processes used by most MDP-based LLM agents and the red arrows and blocks depict processes and modules unique to MLAQ. (b) The overall MLAQ framework. MLAQ interacts with the environment through the Q-Planner, which is supported by the domain-specific memory that extracts a task-specific replay buffer for Q-Update.
2023 Yao et al. 2023a) with world model. The concept of the imagination space is derived (Hao al.
from Dreamer Hafner et al. (2020), consisting of an Basic Policy to generate actions and an World Model to predict next states. In this work, both of them is based LLM, and their interactions could generate imaginary transitions, reducing the demand for environmental interactions. However, most of existing LLM agents choose sacrifice algorithms' generality by utilizing environmental tools. including dynamics (Zhao et al., (022)). available actions (Ding et al. (023), 3) and reward functions (Hao al. 2023). For example, Ding
(2033) use scripts to determine which actions are available under current state. These methods not only fail to leverage RL for optimal sacision-making but also,
due to these sacrifices, reduce the generality provided by LLMs.
In this work, we empower the LLM agent with optimal decision-making capability by combining the advantages of the RL and LLM. As shown in Figure T (b), we introduce Model-based LLM Agent with Q-Learning (MLAQ), a novel LLM agent framework without accessing any environmental tools. The main contribution is three-fold. (1) MLAQ integrates a Q-planner, memory, and imagination space to implement an model-basee RL framework in natural language. This allows for generating imaginary transitions with L.MSs to minimize environmental interactions, and facilitates Q-learning to optimize the policy for maximizing future rewards. (2) An MCTSS-tyle planning approach is proposed to balance exploration and exploitation within the imagination space. By introducing the concept of virtual nodes, this approach efficiently guides the exploration without any environmental tools, ensuring a sub-linear regret bound by rigorous theorem. (3) A
mechanism is proposed improve the quality of imaginary transitions. It uses LLM--asee self-examine to filter out erroneous transitions in the memory and refines the world model with environmental transitions.
Empirically, we evaluate MLAQ on well-known benchmarks for LLM agents (Blocks World (Valmeekam et al. (022) and RoCo-benchmark (Mandi et al., 2023)), which require optimal decision- making for long horizons. There is no existing LLM agent has successfully obtained the optimal policy, while MLAQ achieves over 90% optimal / success rate across most difficulty levels. The comparison with methods including RoCo (Mandi et al. 2023) and RAP (Hao et al. (202)) fully demonstrates MLAQ's superior performance in optimal decision decision-making, Through comparative and ablation experiments, we get a key conclusion: integrating the model-based RL framework with an L.M agent in the form of MLAQ can effectively achieve zero-shot optimal decision-making.
BACKGROUND
MDP-based framework for an agent. An MDP can be defined as a tuple = A, [. where &; is the state in state space S at timestep f, and is the action in action space A, T( 8t az) is the dynamics function, R is the reward function, and 7 is the discount factor (Sutton & Barto 2018). The agent receives a reward r, [[st, de, 8t4-1 1) based on changes in state. For single-agent scenarios the agent policy is denoted as a, ~ (-((s)). For multi-agent scenarios with agent number n, the unified agent make joint decisions according to global states a, = {ar,) ~ (-]ss)). The policy in a decentralized manner allows each agent i making decisions diet TT( (: Pi.)) according to its local observation o,t and communication
policy in centralized manner takes the system
2

---

Published as a conference paper at ICLR 2025
message Pi,t. which can be observations (Hu et al. 2023), features (Ding et al.
2024 Chai et al.
2024), or natural language-basee based dialogues (Mandi et al. al. 2023] Chen et al. 2024). The state value Is defined as the discounted accumulated return under state sy; (st) B k= *r++)]-
LLM-based framework for an agent. Existing approaches construct the above MDP using natural language sentences paragraphs, benefiting from the autoregressive nature of the Transformer architecture (Vaswani et al. (017). without the need to align the state and action space dimensions across different domains Yao et al. 2023a). An LLM-based world model [[s+1[sst ax;7) is utilized to approximate the dynamics function, and an LLM-based basic policy π [T|[s{;;) utilized to interact with either the world model or the environment, where T is a brief domain description in nature language (Hao al. 2023). In addition, due to the current limitation of L.LMs in accurately evaluating state values, this work adopts sparse reward setting, meaning that the environment provides a non-zero reward only when the agent reaches a terminal state.
Q-lcarning. Q-learning is a traditional RL algorithm, in which the Q-value indicates the expected return obtained by executing a under 8. The update rule the Q function at k-th iteration is:
Qu+1 (8t, ax)
+ (rx + 7 max Qu (8441, Qu (8t,aa)) at)
where a is the learning rate. As an off-policy approach, Q-learnng can use a replay buffer to store transitions (s, a, s'), enabling policy update through them. With enough iterations and a sufficiently diverse replay buffer, the Q function and its greedy policy can converge to optimality.
Multi-Armed Bandit. Given state with available actions 8(s). the successive plays on action yields (i.i.d.) Q-values, which are sampled from an unknown distribution with an un- known expectation (s,,) (Auer al. 2002). The player aims minimize the expected regret Q(s,
Dj A (BBCC((((s))) where A, Q0=Q((s, i). Q' = maxx(Q((s i)3, .N (s) is the visit count of s, and C2 (s)) is the selection number of action i. The Upper Confidence Bound (UCB) (Auer et al. 2002) could achieve a sub-linear regret bound by selecting actions through the following rule:
a_ = arg max UCB a) arg max (v((s,)) w (VN(s)/Ncccs.aa)) aEd(s)
(2)
aEō(s)
where c(s, a) is the child node of applying in 8, and is coefficient.
METHOD
The essence of MLAQQ is empower L.M agents with optimal decision-making, while also retaining their ability to make decisions with little to no interaction with the environment (i.e., zero- or few- shot). While integrating RL-based optimization into LLM agents is relatively straightforward, the key challenge lies in preserving their zero- or few-shot abilities when incorporating RL. In this section, we first illustrate how our agent uses RL approaches to achieve task task-specific optimal decision-mait makiing under the fully -ased framework (Section B.I). We then introduce how to achieve fully LLM- based imaginary interactions without any environmental tools (Section 3.2). Finally, we present a mixed-examine mechanism to improve the quality of imaginary transitions (Section 3.3
3.1 OPTIMIZING LLM AGENT WITH Q-PLANNER
As shown in Figure I (b), we develop an RL-style LLM agent framework, which contains a memory module, Q-Planner, and an LLM-based imagination space. The RL-based optimization is achieved by Q-Planner, which utilizes the transitions (s, a, r,r,s') in the memory module to optimize the agent through Q-learning. These transitions are generated from both environmental and imaginary interactions. The overall algorithm is detailed in Appendix F.T
serves as the core of the decision-making process, utilizing the optimal decision- making advantage of RL to output the action with the highest Q-value for given state, where these values are derived from a tabular Q-learning iterations in Eq. (T). The tabular Q-function saves the Q values of actions under a state list, and outputs the action with the highest Q value when making decisions. We perform Q-learning not directly on the Nomain-specifii memory M but rather task- specific replay buffer D extracted from it, aiming reduce the exploration space in imaginary interactions. Assuming there are ten available actions, but only two lead to the target state of the current task, exploring the others is completely meaningless.
Q-Planner.
3

---

Published as a conference paper at ICLR 2025
Imagitatien
I. Solette
Bl. Traxitie treghttien
of tekeratee Reand
Acdis
Steckately by
Repustag
Langleme
3treeal bark
Ternted bodk
(A)
Figure (a) Imaginary interaction process, which expands the memory and replay buffer through two phases: node selection and transition imagination (might be triggered). (b) Imagine new transition with self-examine mechanism, where the checkers identify the validity of actions and predictions.
Replay Buffer. When a new task starts, Q function ((:-)) is obtained by performing Q-learning on M. and the initial replay buffer stores the transitions along the optimal trajectory (if exists) produced by -..... J. If this trajectory does not exist or is demonstrated to be non-optimal through environmental interactions, the agent keeps exploring in the imagination space to expand the replay buffer until the best trajectory is demonstrated to be optimal. This process can be seen in Appendix [
Imaginary Interaction. As shown in Figure (b), an LLM-based basic policy and world model are established to perform imaginary interactions. Without accessing the environment, their understanding of the domain is entirely reliant on the given domain description (manual). Therefore, MLAQ could preserve the zero-shot decision-maie making capabilities of LLM agents while satisfying the training data requirements of Q-learning. The details will be described in the subsequent section.
Environmental Interaction. MLAQ agent obtains the current state and domain description from the environment, explores the imagination space to derive the optimal policy using the Q-planner, and then outputs actions to interact with the environment. The domain description 7 includes the domain goal, state/action definitions, and also some detailed constraints for decision-making and predicting. Please refer Appendix ( for more detail.
3.2 LLM-BASED IMAGINATION FOR MLAQ
This section presents an MCTS- planning method that balances exploration and exploitation while using only LLMs to generate imaginary transitions, thus efficiently expanding the memory and replay buffer. Without any environmental tools, MILQQ should the available actions for any seen states from scratch. Given new task, MLAQ performs multiple imagination rounds, with each consisting of an imaginary trajectories from the original state to the target state. An imagination round comprises two phases: node selection and transition imagination. See Appendix F2 for pseudo-codes.
Node selection. This phase involves balancing exploration and exploitation through a novel selection rule. As denoted in Figure 2 (a), the transitions (s, a, r, s') in the replay buffer adopt a tree-like structure, wherein two nodes s + s' are linked by an action and reward edge (a. r). The original UCB treats planning as a aumti- bandit problem, but using L.LSs for planning without environmental tools presents a variation, as the ground-truti available actions 8(s) are unknown. Therefore, we
begin with an empty available action set 8(s) and introduce the concept of virtual nodes gradually expand it. Within the replay buffer, a actual (state) node is created based on a state s, and a virtual node is appended its first child node, with the edge is virtual action. Then, MLAQ starts with the empty 8(s) and iteratively leverages LLMs to explore potential actions and expand it. This process can be modeled as follows:
A Variant of Multi-Armed Bandit. In a scenario with m available actions (a = (1, m)) in state s, initially only the virtual node y can be selected. Selecting y enables the non-selectable node in a to be selectable, while selecting other node involves estimating its Q- Q-value.
When y selected for the j-th times, i.e., 18(s)) j. the actual nodes (1,2...... ff} can be selected by the planning policy. Moreover, due to the introduction of virtual nodes, we propose variant of the original UCB algorithm by modifying its selection rule as follows:
a = arg max v vUCB (s, a) = arg max (V((s, - [((s,), acô(s)
(3)
eEd(s)
4

---

Published as a conference paper at ICLR 2025
where ((N(ss, is the confidence bound, defined as w ( for actual nodes and v((((s)))) (st)///ol ((())l for virtual nodes. g(x) is a coefficient function that must satisfy some properties, which are discussed in the Appendix C However, since we do not know the number of available actions, it is set to g(a) Wi Eg) in practice, where bg and w are syper-parameters.
Theorem 1. Suppose the player employs the node selection rule defined in (2). Then, in scenario with m available actions state 8, the bound of the expected cumulative regret is:
In [(+) +1+ A
x 1Σ Ad),
Regret-
Σ A 7,yy00
Σ 7A:30
(4)
where A; = Q- Q(s i) for action index i in {1, 2, ..., m) and Q max, | (Q(s, ((s) is the visit count of s, and (s) is the selection number of action i.
This theorem demonstrates that our proposed variant of UCB shares the same sub-linear regret bound, enabling efficient exploration and exploitation without environmental tools. Please refer to the Appendix for the proof. MLAQ turns to the transition imagination phase virtual nodes are selected. Otherwise, it iterates until terminal state is reached. A terminal state denotes either success states when reaching target states or failure states when the LLM- based basic policy determines no actions available. When terminal state is reached, imagination round ends.
Transition imagination. This phase is triggered when selecting virtual nodes and generates an imaginary transition. Beginning from 8t, we first mark the existing actions in 8(s;) as forbidden, where the specific prompt can found in Appendix G We then instruct the basic policy to output an action a excluding forbidden actions to explore and expand 8(s)). If the pair (st, az) could be queried in M, we directly re-utilize the stored next state c(St; ae) as its child state to skip LLM queries. Otherwise, employ the LLM-based world model to predict the next state 8++1 T(s aei r). The reward is also calculated by L.Mss, following a sparse reward setting. In addition, if the basic policy determines that available action exists except forbidden actions, we select the best action in (s) to continue the imaginary interaction. Since we cannot confirm if the available actions are exhausted, the virtual node always remains after the addition of the next state node.
3.3 MIXED-EXXE EXAMINATION FOR IMAGINARY TRANSITIONS
Within the imagination space, we propose a mixed mixeedexamminatinn mechanism to improve the quality of imaginary transitions without accessing any environmental tools.
Env-examination. MLAQ treats environmental transitions as ground truth to correct the imaginary transitions and refine the LLM-based world model. When using domain descriptions to understand the environment for LLMs, both LLM's comprehensions and descriptions could be incomplete, leading to gap between the L..--based world model and the environment dynamics. To address this, during interactions with the environment, MLAQ corrects the transitions stored in memory based on environmental transitions and records the discrepancies into the world model's 's prompt for refinement.
self-examination. Considering the hallucinations inherent in LLMs, even when the prompt contains complete information about the environment, the model may still produce incorrect outputs. To this end, as shown in Figure 2 (b), MLAQ adopts an LLM--based action checker and prediction checker to check the validity of the outputs of the basic policy and world model. Current models (e.g., GPT-4) already possess the ability to improve output accuracy through self-examination, which will be validated in subsequent experiments. Prompt templates of the two checkers are provided in Appendix G Take the action checker as an example, given the current state and the action should be checked, queries the LLM check if the action breaks any constraint following the instructions. The checkers continuously check and provide feedback until the outputs are correct.
4 EXPERIMENTS
In this section, conduct experiments in several challenging environments for LLM agents to answer the following questions: (A) Does the introduction of a complete RL framework improve the optimal decision-making capabilities of L.M agents? (B) Can the MLAQ agent preserve zero- or
5

---

Published as a conference paper at ICLR 2025
Exsiting Trajectories Memory
Aguregated Trujectory for the New Task
Original
Figure 3: An instance with an optimal step illustrates that MLAQ. due to ability to leverage transitions across multiple tasks within the same domain, obtain optimal policy through RL-based optimization without requiring any search in the imagination space when given new task.
few-shot decision-making capabilities? (C) Does the current LLM have the potential to improve its oulpul performance through self-examination? (D) Does expanding the memory scope from task-specific to domain-spection improve the agent performance?
4.1 EXPERIMENTAL SETUP
We conduct experiments the BlocksWorld benchmark (Valmeekam al. 2022 for the single-agent setting and the RoCo-benchmark (Mandi et al., 2023) for the multi-agent setting. Agents in these domains require multi-step decision-making maximize expected future rewards. Additionally, the -ocision-making space for LLM agents in the RoCo-benchmark is significantly larger than that in BlocksWorld due the presence of multiple agents. The details of these benchmarks can be found in Appendix D
achieve the final goal, necessitating the ability to
We compare MLAQ with CoT (Wei al. 2022). RAP (Hao al. 2023). Rex (Murthy et al. 2023). RAFA (Liu et al. (023). and RoCo (Mandi et al. 2023). In line with RAP (Hao et al. [202)), we group all tasks by their optimal steps, indicating the length of the tasks' optimal decision sequences. Within domain, the experiments are conducted from the tasks with smaller optimal steps to those with larger steps. Consequently, only the memory size of MLAQ would increase gradually with the experiments due to its domain scope, empowering the LLM agent with optimal decision-man making capability with Q-Planner Furthermore, we also conduct exhaustive ablation experiments to verify the effect of domain-specifie memory module, further expansion in memory, and the mixed-examination mechanism. The details of parre--rrraeters can found in Appendix
4.2
MAIN RESULTS
Single-agent scenario. As shown in Table I the optimal steps of BlocksWorld vary from 2 to 12. The optimal rate in this table denotes that the agent could reach the target state under the optimal steps. Our agent achieves more than 90% optimal rate across all difficulty levels, indicating an affirmative response Question A In contrast, the other LLM agent can only succeed in tasks with smaller optimal steps. In order qualitatively provide an affirmative answer to Question D) we use a task with 6 optimal steps in Figure Jas as an example to backtrack the transitions memory. In this case, MLAQ agent finds optimal decision sequence, represented by black solid arrows, from the initial replay buffer without querying LLMs. The colored dashed arrows indicate existing trajectories stored in memory. This 6-step sequence is aggregated from trajectories of multiple tasks, including two 2-step and three 4-step trajectories. This clearly demonstrates the effectiveness of
Avcrage Tokon
Momory Ne
60033
loons
50000
20003
0-4
20003
10003
Optima: Skepe
Figure 4: Tendency of memory re-utilization ratio and token consumption.

---

Published as a conference paper at ICLR 2025
Table I: The optimal rate of experimental methods in BlocksWorld domain.
Methods
2-step
4-step
6-step
8-step
10-step
12-step
CoT (Wei al. 2022)
0.22
0.14
0.02
0.02
0.00
0.00
REX Murtty et al. 2023)
0.80
0.45
0.25
RAFA Liu al. 2023)
0.97
0.75
RAP Hao et al.
2023)
0.67
0.76
0.74
0.48
0.17
0.09
MLAQ
1.00
1.00
1.00
0.97
0.93
0.90
our domain memory in enhancing the agents' decision-making capabilities in long-horizon tasks. Quantitative analysis will be provided in the ablation experiments.
Besides, we also investigate the memory re-utilization ratio in transition imagination phase. As shown in Figure 4 higher re-utilization ratio commonly leads lower average consumed tokens. As the increase of the optimal step, the ratio of memory re- re-utilization increases from 19% to 90%. Meanwhile, the average token consumption of each task first increases and then decreases significantly. The increase of tokens from 2-step task to 4-step task is attributed to the increased task difficulty, requiring the agent to employ more imagination to make optimal decisions. Then, as the memory becomes enriched with the data collected from resolved tasks within the same domain, the probability of re-utilizing stored transitions increases, leading to a reduction in average token consumption.
Multi-agent scenario. In RoCo-benchmark, robotic arms need to efficiently allocate tasks internally and collaborate to achieve the final goal. Table[]shows the experimental results RoCo and MLAQ on Sort domain. MLAQ achieves success approaching 100% across all difficulty levels and maintains an optimal performance exceeding 50% across most tasks, further giving an affirmative response Question A In this table, introduce "success rate" expanded search space from the presence of multiple agents significantly reduces their optimal rate. The notation "MLAQ denotes that MLAQ agent terminates its imagination upon reaching the target state, without further exploring to get better policy. A detailed analysis of "MLAQ is provided in the ablation section. The optimal steps for Sort range from 1 to 6, with MLAQ consistently achieving higher success and optimal rates than RoCo, especially in long-horizon tasks. Due to space limitations, the results and analysis on Sandwich domain are provided Appendix H.2
additional metrics as the
Table 2: Comparison of multiple metrics between MLAQ and RoCo Sort domain.
Metrics
Methods
1-step
2-step
3-step
4-step
5-step
6-step
Average
RoCo
1.00
0.64
0.47
0.10
0.03
0.00
0.35
Success Rate
MLAQ
1.00
0.96
0.97
1.00
0.93
1.00
0.98
Env Replans
RoCo
0.30
6.30
5.60
9.74
7.67
6.92
6.41
(n-shot)
MLAQ
0.00
0.04
0.03
0.10
0.10
0.04
0.06
RoCo
0.80
0.36
0.27
0.00
0.00
0.00
0.21
Optimal Rate
MLAQ
0.95
0.64
0.57
0.67
0.33
0.43
0.58
MIAQQ
1.00
0.86
0.77
0.73
0.50
0.75
0.75
RoCo
10605
530817
332730
305175
345762
320045
322630
Average Token
MLAQ
7093
15104
16436
22197
18133
8220
15216
MLAQ
8491
66367
156490
119916
409495
243151
175560
RoCo
0.35
3.37
3.43
3.33
2.97
2.00
2.72
Optimal Gap
MLAQ
0.10
0.64
0.73
0.53
1.20
0,46
0.65
MLAQ
0.00
0.32
0.40
0.43
0.80
0.25
0.39
Memory Re-Util. Ratio
0.27
0.61
0.71
0.72
0.73
0.83
0.66
In Table 2 the average token consumption of each task still shows trend of increasing first and then decreasing, and lower than RoCo's in Sort. The Eny Replans metric denotes the frequency of environment feedback, which reflects the zero-shot decision-making ability of the LLM agent. A smaller Eny Replans indicates that the agent requires less ground- truth information from the environment refine its decisions. The results show that the number of environmental
7

---

Published as a conference paper at ICLR 2025
Table 3: The optimal rate for optimizing multiple epochs in Sort domain.
Methods
1-step
2-step
3-step
4-step
5-step
6-step
MLAQ- 1st epoch
1.00
0.86
0.77
0.73
0.50
0.75
MLAQ-2nd epoch
1.00
0.93
0.90
0.77
0.57
0.88
MLAQ-30 epoch
1.00
0.93
0.90
0.77
0.57
0.92
replans for MILAQ has decreased by approximately two orders of magnitude compared to RoCo, indicating that the MIAQ agent almost does not require environmental transitions to correct imaginary transitions. In contrast, existing methods require the environment verify action's 's validity and provide environmental feedback, facilitating LLM agents to replan until actions are available (Mandi al. 2023 Shinn al. 2023). This result indicates an affirmative answer the Question B In addition, since we provide two types of basic policy (central and dialogue) in multi-agent scenarios, we also conduct experiments compare them in Appendix H.
4.3 MLAQ WITH MULTIPLE EPOCIS
This experiment aims to demonstrate the potential of MLAQ to enhance its optimal decision-making ability through iter- ative optimization. In previous experiments, each domain's experiment starts with an empty memory. In this section, we conduct multiple epochs of MLAQ experiments by perform- ing MLAQ with non-empty memory to further enhance its decision-making capabilities and utilize memory re-utilization to reduce token consumption. At the beginning of each epoch, the input memory is the memory of the last MLAQ epoch after finishing all given tasks of the domain. As shown in Table 3 and Figure 5 an additional training epoch results in an improve- ment of the optimal rate while reducing token consumption. Although MLAQ with multiple epochs no longer constitutes zero-shot decision-making, the small magnitude of replans implies that minimal environmental information is injected into subsequent epoch decisions. Consequently, still provides more definitive affirmative answer to
Token
Optimal Rate
0.90
0.83
180
Average Token (Thoucands)
Average Optimal Rate
0.75
0.60
100
0.30
RcCo
0.15
Epochs
Figure 5: Tendency of optimal rates and tokens with epochs.
Question A and Question B
4.4 ABLATION RESULTS
The key components of MLAQ are the Q-planner, memory, and imagination space. In this section, validate their effects by individually diminishing their capabilities. The ablation experiments conducted under 8-step tasks in the Blocks World domain. This is primarily because the task difficulty this optimal step is moderate for appropriate comparison.
Table 4: Ablation results in Blocks domain.
Ablation Methods
Optimal Rate
Token
Env Replans
RAP
0,48
MLAQ w/o domain memory
0.87
378941
0.02
MLAQ w/o self-examine
0.57
20115
0.81
MLAQ w/o env-examine
0.95
40772
0.05
MLAQ
1.00
39688
0.02
Removing further optimization. This experiment evaluates the impact of the Q-planner's inability to further optimize. In our framework, if the best decision sequence derived from the Q-Planner is determined to non-optimal through environmental interactions, the MLAQ agent would keep exploring in the imagination space to continually expand the memory and replay buffer for further optimization. Within the imagination space, allow the MLAQ agent to complete only one successful imagination round by terminating the exploration process upon reaching the target state, referring this method as MLAQ As demonstrated in Tables 2 this deletion of further optimization
8

---

Published as a conference paper at ICLR 2025
decreases token consumption while concurrently diminishing the optimal rate. The 'Optimal Gap (OG)' represents the gap between average completion steps and optimal steps. The results indicate that MLAQ achieves lower OG than MLAQ which fully demonstrates the effectiveness of the expansion of the replay buffer and the optimization based on Q-Planner. The experiments on the Sandwich domain presented in Appendix HH.2] also yields the similar results.
Removing LLM-based self-examine and eny-examine. This experiment reduces the quality of transitions generated in the imaginative space by separately removing self-examine and env-examine, (1) The self-examine utilizes an LLM-based action checker and prediction checker to enable error correction within the imagination space. We remove it from MLAQ agent to directly output the actions and predictions without LLM-based examination, which is denoted as MLAQ w/o self-examine in Table Note that the initial memory stores transitions from the 2-step to 6-step experiments rather than starting from scratch. The results demonstrate notable reduction in token consumption alongside significant decrease in the optimal rate. Moreover, the frequency of environmental feedback has significantly increased, providing support for affirmative answers to Question C (2) The env-examine corrects the imagination process by treating environmental transitions as ground truth. Removing it allows for evaluation of the imagination capability using only the LLMs. The results of MLAQ w/o env-examine indicate only a slight performance decline compared to MLAQ. Combined with the env replans metrics of MILAQ, this leads to the conclusion that the current L.LMs (e.g., GPT-4) possess the capability to efficiently generate imaginary transitions through caff-xxaminaiion resulting in relatively low dependency on ground truth, answering Question C from the opposing viewpoint. Removing both of them will lead to decline in the quality of imaginary transitions, thereby reducing the decision-makee performance of our MLAQ agent.
thereby evaluating their effect.
Narrowing memory scope. This experiment narrows the scope of the memory module from domain- specific to task-specific like existing methods (Hao et al. (023) in order evaluate the effect of constructing memory in an RL manner. This approach is denoted as MLAQ w/o domain memory in Table 4 The results indicate significant increase in token consumption alongside decrease in the agent's optimal rate, quantitatively answering the Question C This decline stems from the MLAQ agent's 's inability to leverage prior gpeision-makinge experiences to solve new tasks within the same domain. However, the optimal rate of the MLAQ agent still exceeds that of RAP, owing to the influence of other components within the MLAQ framework.
4.5 SELF-EXAMINATION ANALYSIS
This section quantitatively evaluates the L.LM-based self-examination, showing that while hallucina- tions may occur during the examining process, performing multiple checks in transition imagination significantly reduces the erroneous transitions. In the self-examination process, the LLM-baseed checkers may erroneously identify correct outputs as incorrect, and vice versa. We sample 128 random states from the Sort domain, providing correct actions for 64 states and incorrect actions for the others, with each incorrect action breaking only one constraint. The testing for the prediction follows a similar approach, where each incorrect prediction only violates one constraint.
Table 5: Results on identification capability of the checkers
Checker Type
TP
FN
4
FP 1
TN f
Precision
Action Checker
63
1
58
91.3%
Prediction Checker
64
0
10
54
86.5%
As shown in Table we evaluate the performance of the checkers using five metrics. True Positives (TP) identify correct outputs (actions or predictions) as correct, False Negatives (FN) identify correct outputs as incorrect, False Positives (FP) identify incorrect outputs as correct, True Negatives (TN) identify incorrect outputs incorrect, and Precision = TP (TP + FP) is proportion of correctly identified correct cases among all cases identified as correct. Results in Table suggest that while both checkers accurately identify correct outputs, they may misidentify incorrect outputs correct ones, leading to inaccuracies in imaginary transitions and subsequent impacts on agent performance.
To further investigate the impact of inaccuracies in self-examination, similar methods are used to test the accuracy of the basic policy and world model. Without forbidden actions, the accuracy of the basic policy is 84.4%, while the accuracy of the world model is 100%. Therefore, the combined
9

---

Published as a conference paper at ICLR 2025
calculation yields probability of 1.36% (1 84.4%) |6 / (6 63)] for wrong transition to be stored in memory. Compared to the probability of 15.6% = (1 -4.44%) without examinaamnn,,on MLAQ can greatly improve the quality of imaginary transitions. These results provide support for affirmative answers to Question C Furthermone, some erroneous transitions stored in memory could be excluded from the best policy derived from RL-based optimization due their low values, while the others could be corrected by env-examination. Experimental data in Table 2 also indicate that these errors minimally impact the decision-making capabilities of LLM agents.
RELATION TO OTHER METHODS
In this section, we compare the MLAQ framework with several representative works in LLM agents. Table O presents the interaction between Agent, World Model (WM), and Environment (Env) in existing approaches, as well the forms of their memory. (1) The Agent-Env interaction is central to most LLM agents, with efforts to optimize agents through techniques like self-reflect (Shinn et al. 2023) and dialogue (Mandi et al., 2023) for prompt enhancing. and Monte Carlo Tree Search (MCTS) for transition-level optimization (Ding et al.
2023). (2) The Agent-Wis WM interaction is originated from model-based RL (Hafner et al. 2020) to expand memory through the agent's ' imaginary interactions. 2023a construct an LLM-based world model for MCTS- RAP (Hao et al. (023) and ToT (Yao et al.
based planning in single task, but they still not try to narrow the gap between the world model and environment. (3) The WM-Env interaction requires using the environmental data to correct the mistakes made by LLM-based world model, which is achieved by LATS (Zhou et al. (023a)). (4) Finally, the memory modules of LLM agents differ in the level and scope. Insight-level memory stores agents' natural language summaries of their insights (Ding et al. 2023), while transition-level memory retains RL-like transitions (Hao et al.) 2023). The scope here represents the time span of the data in memories, encompassing (a) Step: the single transition, (b) Task: multiple transitions within the same task, and (c) Domain: multiple transitions in multiple tasks within the same domain. We present different scopes of memory in Appendix A and detailed related works in Appendix
Table 6: Different interactions in existing LLM-based decision methods.
Method
Agent-Env
Agent-WN WM
WM-Env
Memory Level-Scope
Reflexion (Shinn et al.
2023
Self-reftection
Insight- Task
Expel Zhao ct al. 2024
Self-reflection
Insight-Doman
RoCa Mandi et a 2023
Dialog Feedback
Transition-Ster
XoT Ding et al. 2025
MCTS Extraction
Transition-Taun
RAP
Hao et al.
2023
Direct
MCTS
l'ransition-Taan
ToT
Y
/a et al.
2025a
Direct
MCTS
Transition- Task
LATS (Zhou
2023a
Sell-reflection
MCTS
Env-examine
Transition- Task
MLAQ (Ours)
Q-Planner
UCB-guided
Mixed-examine
Transition-
6 CONCLUSION
This work introduces MLAQ, a powerful MDP-based LLM agent framework that achieves zero-shot optimal makin..... MLAQ consists of a memory module, a Q-planner, and an imagination space to fully leverage the general comprehension capabilities of LLMs and the optimization capabil- ities of RL. The imagination space is entirely based on LLM to generate imaginary transitions, which are then provided to the Q-planner for RL-based optimization. The proposed planning algorithm balances the exploration and exploitation by introducing the concept of virtual nodes, while ensuring sub-linear regret bound guaranteed by theorem. Moreover, a mixed-examiin ployed to improve the quality of the imaginary transitions. We evaluate MLAQ on both single-agent BlocksWord and multi-agent agents. Results indicate that MILQQ outperforms existing methods, especially in tasks involving long- horizon decision-makin making. Furthermore, conduct exhaustive additional experiments investigate how MLAQ enables LLM agents to achieve zero-shot optimal decision-making capabilities. The core conclusion of this work that constructing a model-based RL framework in natural language for current LLMs can further enhance the decision makinn abilities of LLM agents, In future work, we intend deploy this method on physical robotic platforms to fully exploit its advantages.
mechanism is em-
BeCCo-bbencmrkk domains, comparing it with several advanced LLM
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported in part by the National Natural Science Foundation of China under Grant 621 and Grant 62293541, in part by Beijing Natural Science Foundation under Grant 4232056, and in part by Beijing Nova Program under Grant 20240484514.
REFERENCES
Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine learning, 47:235-255 226,, 2002.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Jiajun Chai, Weifan Li, Yuanheng Zhu, Dongbin Zhao, Zhe Ma, Kewu Sun, and Jishiyu Ding. Unmas: Multiagent reinforeement learning for unshaped cooperative scenarios. IEEE transactions on neural networks and learning systems, 34(4):2093-2100 2021.
Jiajun Chai, Yuqian Fu, Dongbin Zhao, and Yuanheng Zhu. Aligning credit for multi-agent coop- cration via model-based counterfactual imagination. In Proceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems, AAMAS '24, pp. 281-289) 2024.
Weize Chen, Yusheng Su, Jingwei Zuo, Cheng Yang. Chenfei Yuan, Chi-Min Chan, Heyang Yu, Yaxi Lu, Yi-Hsin Hung, Chen Qian, Yujia Qin, Xin Cong, Ruobing Xie, Zhiyuan Liu, Maosong Sun, and Jie Zhou. Agent Verse: Facilitating multi-agent collaboration and exploring emergent behaviors. In The Twelfth International Conference on Learning Representations, 2024.
Ruomeng Ding, Chaoyun Zhang, Lu Wang, Yong Xu, Minghua Ma, Wei Zhang, Si Qin, Saravan Rajmohan, Qingwei Lin, and Dongmei Zhang. Everything of thoughts: Defying the law of penrose triangle for thought generation. arXiv preprint (xXi:2311.0022. 2023.
Shifei Ding, Wei Du, Ling Ding, Lili Guo, and Jian Zhang. Learning efficient and robust multi-agent communication via graph information bottleneck. Proceedings of the AAAI Conference on Artificial Intelligence, (8(16):11) 17346-1333, 2024.
Vladimir Egorov and Alexei Shpilman. Scalable multi-agent model-based reinforcement learning. In Proceedings of the 21 International Conference on Autonomous Agents and Multiagent Systems, AAMAS '22, pp. 381-390, 2022.
Nikolaos Gkanatsios, Ayush Jain, Zhou Xian, Yunchu Zhang, Christopher Atkeson, and Katerina Fragkiadaki. Energy- based models are zero-shot planners for compositional scene rearrangement. In RSS 2023 Workshop on Learning for Task and Motion Planning. 2023.
Taicheng Guo, Xiuying Chen, Yaqi Wang, Ruidi Chang, Shichao Pei. Nitesh V Chawla, Olaf Wiest, and Xiangliang Zhang. Large language model based multi multi-agents: A survey of progress and challenges. arXiv preprint arXiv: 2402.01680 2024.
Danijar Hafner. Benchmarking the spectrum of agent capabilities. In International Conference on Learning Representations, 2022.
Danijar Hafner, Timothy Lillicrap. Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In International Conference Learning Representations, 2020.
Shibo Hao, Yi Gu, Haodi Ma, Joshua Hong. Zhen Wang, Daisy Wang. and Zhiting Hu. Reasoning with language model is planning with world model. In Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, pp. 8154-8173, 2023.
Guangzheng Hu, Yuanheng Zhu, Dongbin Zhao. Mengchen Zhao, and Jianye Hao. Event-triggered communication network with limited-buni bandwidth constraint for multi-agent reinforcement learning. IEEE Transactions Neural Networks and Learning Systems, 34(8):3966-377) 2023.
11

---

Published as a conference paper at ICLR 2025
Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models zero-shot planners: Extracting actionable knowledge for embodied agents. In International conference on machine learning, pp. 91189911811111 914777947771 PMLR, 2022a.
Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, Pierre Sermanet, Tomas Jackson, Noah Brown, Linda Luu, Sergey Levine, Karol Hausman, and brian ichter. Inner monologue: Embodied reasoning through planning with language models. In 6th Annual Conference on Robot Learning, 2022b.
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. Advances in neural information processing systems, 32, 2019.
Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. Decomposed prompting: A modular approach for solving complex tasks. In The Eleventh International Conference on Learning Representations, 2022.
Levente Kocsis and Csaba Szepesvári. Bandit based monte-carlo planning. Machine Learning, 2006: 282-293, 09 2006.
Yilun Kong, Jingqing Ruan, YiHong Chen, Bin Zhang, Tianpeng Bao, shi shiwei, du guo qing, xiaoru hu, Hangyu Mao, Ziyue Li, Xingyu Zeng, Rui Zhao, and Xueqian Wang. Boosting task planning and tool usage of large language model-based agents in real-world systems. In ICLR 2024 Workshop on Large Language Model (L.M) Agents, 2024.
Teyun Kwon, Norman Di Palo, and Edward Johns. Language models as zero-shot trajectory generators. IEEE Robotics and Automation Letters, 2024.
Zhihan Liu, Hao Hu, Shenao Zhang, Hongyi Guo, Shuqi Ke, Boyi Liu, and Zhaoran Wang. Reason for future, act for now: A principled framework for autonomous Ilm agents with provable sample efficiency. arXiv preprint arXiv:2309.17382, 2023.
Zhao Mandi, Shreeya Jain, and Shuran Song. RoCo: Dialectic multi-robot collaboration with large language models. arXiv preprint arXiv:2307.04738, 2023.
Rithesh Murthy, Shelby Heinecke, Juan Carlos Niebles, Zhiwei Liu, Le Xue, Weiran Yao, Yihao Feng, Zeyuan Chen, Akash Gokul, Devansh Arpit, et al. REX: Rapid exploration and exploitation for Al agents. arXiv preprint (XXx:22377008622. 2023.
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and Shi- mon Whiteson. QMIX: Monotonic value function factorisation for deep multi-agent rrinforcement learning. In International Conference Machine Learning, pp. 4295-4304, 2018.
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: language agents with verbal reinforcement learning. In Advances in Neural Information Processing Systems, volume 36, pp. 8634-8652, 2023.
Zhiyuan Sun, Haochen Shi, Mare-Alexandre Côté, Glen Berseth, Xingdi Yuan, and Bang Liu. Enhancing agent learning through world dynamics modeling. In Findings of the Association for Computational Linguisties: EMNLP 2024, pp. 3534 3568,3568,,40 2024.
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning, second edition. MIT Press, Nov 2018.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSI international conference on intelligent robots and systems, pp. 5026-5033. IEEE, 2012.
Miles Turpin, Julian Michael, Ethan Perez, and Samuel Bowman. Language models don't always say what they think: Unfaithful explanations in chain-of of-thuught prompting. In Advances in Neural Information Processing Systems, pp. 74952-74965. 2023.
Karthik Valmeekam, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati. Large language models still can't 't plan (a benchmark for LLMs on planning and reasoning about change). In NeurlPS 2022 Foundation Models for Decision Making Workshop, 2022.
12

---

Published as a conference paper at ICLR 2025
Karthik Valmeekam, Kaya Stechly, and Subbarao Kambhampati. LLMs still can't plan; can LRMs? a preliminary evaluation of OpenAl's 's ol on planbench. arXiv preprint arXiv:2409. 2024.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 2017.
Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, and Ee-Peng Lim. prompting: Improving shot chain-of-thoughh reasoning by large language mod- els. In Proceedings of the 6l Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 2609-2660 2634, 2023a.
Zihao Wang. Shaofei Cai, Guanzhou Chen, Anji Liu, Xiaojian Ma, and Yitao Liang. Describe, explain, plan and select: Interactive planning with LLMs enables open-world multi-task agents. In Thirtty-eventh Conference on Neural Information Processing Systems, 2023b.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-2488 2022.
Yue Wu, Xuan Tang, Tom Mitchell, and Yuanzhi Li. SmartPlay: A benchmark for LLMs as intelligent agents. In The Twelfth International Conference on Learning Representations, 2024.
Zelai Xu, Chao Yu, Fei Fang, Yu Wang, and Yi Wu. Language agents with reinforcement learning for strategic play in the Wole game. arXiv preprint
2023.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. In Advances in Neural Information Processing Systems, volume 36, pp. 1 1899-18222,, 2023a.
Shunyu Yao, Jeffrey Zhao. Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models. In The Eleventh International Conference Learning Representations, 2023b.
Abhay Zala, Jaemin Cho, Han Lin, Jachong Yoon, and Mohit Bansal. EnvGen:
Generating and adapting environments via LLMs for training embodied agents. In First Conference on Language Modeling, 2024.
Ceyao Zhang, Kaijie Yang. Siyi Hu, Zihao Wang, Guanghe Li, Yihang Sun, Cheng Zhang, Zhaowei Zhang, Anji Liu, Song-Chun Zhu, Xiaojun Chang, Junge Zhang, Feng Yin, Yitao Liang, and Yaodong Yang. ProAgent: Building proactive cooperative agents with large language models. Proceedings of the AAAI Conference on Artificial Intelligence, 38 38(16): (1611)))7791111 1751:111991999999 2024a.
Hongxin Zhang, Weihua Du, Jiaming Shan, Qinhong Zhou, Yilun Du, Joshua B. Tenenbaum, Tianmin Shu, and Chuang Gan. Building cooperative embodied agents modularly with large language models. In The Twelfth International Conference Learning Representations, 2024b.
Yang Zhang, Shixin Yang, Chenjia Bai, Fei Wu, Xiu Li, Xuelong Li, and Zhen Wang. Towards efficient LLM grounding for embodied multi-agent collaboration. arXiv preprint arXiv:2405. 14314, 2024c.
Zeyu Zhang, Xiaohe Bo, Chen Ma, Rui Li, Xu Chen, Quanyu Dai, Jieming Zhu, Zhenhua Dong, and Ji- Ji-Rong Wen. A survey on the memory mechanism of large language model based agents. arXiv preprint arXiv:2404. 13501, 2024d.
Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Yong-Jin Liu, and Gao Huang. Expel: LLM agents experiential learners. Proceedings of the AAAI Conference Artificial Intelligence, 38 (17):1933--19642, 2024.
Zirui Zhao, Wee Sun Lee, and David Hsu. Large language models commonsense knowledge for large-scale task planning. In Advances in Neural Information Processing Systems, volume 36, pp. 31967-31987, 2023.
13

---

Published as a conference paper at ICLR 2025
Yupeng Zheng, Zebin Xing, Qichao Zhang, Bu Jin, Pengfei Li, Yuhang Zheng, Zhongpu Xia, Kun Zhan, Xianpeng Lang, Yaran Chen, et al. Planagent: A multi-modal large language agent for closed-loop vehicle motion planning, arXiv preprint arXiv:2406.01.4 015877 2024.
Andy Zhou, Kai Yan, Michal Salapentokh-Rothhane Haohan Wang, and Yu-Xiong Wang. Language agent tree search unifies reasoning acting and planning in language models. arXiv preprint arXiv:2310.04446. 2023a.
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoe V Le. and Ed H. Chi. Least-to-most prompting enables complex reasoning large language models. In The Eleventh International Conference on Learning Representations, 2023b.
14

---

Published as a conference paper at ICLR 2025
Detailed Descriptions of Memory Scope
16
Related Work
16
Proof for Theorem
19
D Experimental Environments
21
D.I BlocksWorld
D.2 RoCo-benchmark
21
Experiments on Crafter
23
Pseudo-code for the Overall Algorithm
25
F.1 Overall algorithm for MLAQ agent
25
F2 LLseb Imagination for MLAQ
25
Expand the replay buffer from the memory
28
Q function update
29
G
Detailed Prompt Template
30
Basic policy prompt
30
G.2
Action checker prompt
31
G.3
World model prompt
G.4
Prediction checker prompt
31
Detailed Experiment Results
33
H.l
Basic Policy Analysis for Multi Agent Tasks
33 ................................
Main Results in the Sandwich domain
33
lyper-Parameters
34
J
Prompt Example
35
Policy Prompt and Response of the State-based Makinig
35
J.2
Policy Prompt and Response of the Observation- based Decision-Making
39
J.3
Action Checker Prompt and Response Example
42
J.4
World Model Prompt and Response Example
44
J.5
Prediction Checker Prompt and Response Example
46
15

---

Published as a conference paper at ICLR 2025
A DETAILED DESCRIPTIONS OF MEMORY SCOPE
MLAQ employs memory module with domain scope. In order to better illustrate its difference from the memory scope in previous LLM agents, we present the memory module under different scopes in Figure 6 Figure 6 (a) shows memory with Step scope, which only retains the interaction data between the agent and the environment at the current time step. For instance, when refining LLM agent's 's decisions, RoCo (Mandi et al. 2023) only relies on the feedback provided by the environment in the current state. Figure (b) shows a memory with Task scope, which focuses on solving the current given task and retains the data obtained from the interactions between the agent and the environment in this task. Most LLM agents adopt this form of memory, which called a replay buffer in Figure [ Take the RAP (Hao et al. 2023) agent as an example, it explores within an LLM-based imagination space and retains visited trajectories to avoid repeated LLM-based state predictions. However, the memory for each task is unique and contains only information relevant to this single task, meaning that the agent cannot gain experience from the decision histories of other tasks within the same domain solve the current task. Instead, our memory with domain scope stores environmental and imaginary interaction data from all tasks in the domain, effectively reducing the need for L.LM queries and improving the efficiency of the reinforcement learning through Q-planner. Besides, one point to clarify is that in Figure (c), cylinders are depicted in different colors only to illustrate that they have transitions from different tasks. The transitions stored in our memory do not contain any task sask-spe labeling, and always stored in the format of (s,a,7,8)
(a)
(b)
(c)
Figure 6: Memory modules with different scope. (a) Step. (b) Task. (c) Domain.
B RELATED WORK
Leveraging the Transformer network architecture, recent advancements in Large Language Models (LLMs) have highlighted their potential in decision-making making tasks. These approaches are commonly referred as LLM agents. The research on LLM agents originated from GPT-3, an LLM with 175 billion parameters (Brown et al., 2020). Its authors found that the inference and decision-making ability of LLM can be improved by simply adjusting the input prompt without adjusting the model parameters. This insight leads to the development of the Chain-of- Chain-of-Thought (CoT) approach, which improves the LLM's 's reasoning capabilities by integrating step-by-step reasoning examples into the prompts WWi et al., 2022). Subsequent studies have focused further enhancing L.M reasoning capability through prompt engineering. Techniques such as task decomposition, exemplified by least- to-most (Zhou al., 2023b) and DecomP (Khot al. 2022) prompting, are proposed to simplify the task complexity. The affoemmntioned methods also could address some traditional problems in natural language processing, such cloze and completion task and reasoning task (Turpin al. 2023). With the development of LLMs, OpenAl's 01 has been characterized as Large Reasoning Model (LRM), demonstrating preliminary planning capabilities (a success rate of 97.8% in BlocksWorkl as reported by Valmeekam et al. (2024). However, due to our experiments employ more challenging optimal rate as the metric, the direct comparison not presented in the main text.
LLM agents with prompt optimization. As research progresses, an increasing number of studies focus on developing an LLM agent to tackle decision-making tasks with higher complexity. Most methods adopt the existing paradigm, aiming to indirectly enhance the decision-making capabilities of LLM agents by optimizing the prompts given to LLMs (Brown et al., 2020). These approaches typically involve enriching the prompt's informational content through environmental or human feedback and employing some techniques to summarize these information into some insights, thereby minimizing the prompt's length (Zhang et al. 2024d). Huang al. (2022h) enable the L.LM agent
16

---

Published as a conference paper at ICLR 2025
to understand many semantic aspects of the world by leveraging environmental feedbacks to form an inner monologue, acknowledging what, when, and how to do skills in embodied environments. DEPS (Wang et al. 2023b) involves the description and self-explanation of the agent's 'ss plans to improve the accuracy of multi-step reasoning in long-horizon tasks. It also provides a trainable goal selector rank the sub-goals based the estimated steps of completion, thus refining the agent's plans. Furthermore, as shown in Table 6 the self-reflect is a widely-used technique to improve LLM agent's decision-making capability. Reflexion (Shinn et al. 2023) and PlanAgent Zheng et al. (0224) empowers autonomous LLM agents to iteratively revise their actions based on feedback, thereby optimizing their behavior. ReAct (Yao et al.] 2023b) introduces closed-loop reasoning process by alternately generating verbal reasoning traces and actions to achieve self- correction of knowledge. Additionally, it acquires additional information from external sources through the execution of actions. AgentVerse (Chen et al. 2024) involves multiple LLM agents with different roles to orchestrate a collaborative group of expert agents. These agents evaluate their executed actions to adjusting the group and discuss collaboratively for making better decisions. There are also numerous approaches aimed at addressing decision-making problems in multi-agent environments. ProAgent (Zhang et al.] 2024a) considers scenarios involving cooperation with human players. It performs cooperative reasoning and planning based on inference of teammates' behavior, corrects the agents' beliefs using actual decisions, and stores acquired knowledge into task memory. Mandi et al. (2023) proposes multi-agent benchmark RoCo- -oCo-benchmark with multiple robotic arms solving a cooperative task. also constructs a dialog- address the proposed benchmark. CoELA (Zhang et al., 2024b) aims to address the multi-agent cooperation problem with decentralized control by providing communication between L.LM agents to exchange their information and knowledge. Its framework extensively utilizes the potential of LLM in perception, communication, planning, memory and execution.
multi-agent framework RoCo to
LLM agents with action optimization under MDP framework. Those methods, which only aim at optimizing prompts, cannot achieve satisfactory performance long-horizon decision-making tasks (Hao et al. 2023), which more challenging to LLM agents. Firstly, the inherent reasoning capabilities of these methods, bounded by the fixed parameters of the base LLM, fall short in complex tasks necessitating long-horizon decision sequences. Secondly, adjusting input prompts may improve reasoning to some extent but does not fundamentally optimize the policy of the LLM agent for acquiring optimal decision sequences. Recent researches have explored using broader language applications to model the environmental dynamics and reward functions, employing planning algorithms to guide decision-making (Liu et al. 2023] Zhang al. 2024d). They formulate the LLM agent under the Markov Decision Process (MDP) framework by decomposing original long- horizon decision sequences into atomized state transitions. Yao al. (2023a) propose Tree-of- Thoughts (ToTT, which establishes an incomplete MDP framework limited to state space, action space, and value function components. ToT employs depth-first and breadth-firs first search techniques to derive optimal sequences of thoughts in the decomposed "thought space". RAP (Hao et al. (002)) leverages LLMs to explicitly construct a world model, serving as the dynamics function within the MDP. RAP employs an LLM-based policy to generate decision trajectories by interacting with the world model within the imagination space and utilizes the MCTS algorithm to effectively balance exploration and exploitation, thus optimizing decision sequences. REX (Murthy et al.) 2023) introduces additional layer of rewards and incorporates principles akin to Upper Confidence Bound (UCB) values to further enrich exploration within the action space, resulting in more robust and efficient agent performance. Everything-on of thouhhthh (XoT) (Ding et al., (202)) emphasizes the "Penrose triangle" in traditional LLM agents, indicating that Performance, Efficiency, and Flexibility cannot simultancously exist. It utilizes LLM for thought generation and employs RL and MCTS algorithms to revise the generated thought tree, thereby enhancing the agent's performance. /Zao al. (2023) establish an LLM- based commonsense world model and basic policy to conduct MCTS-baaed searches. External knowledge is obtained through datasets or interactions with human players, enabling updates to the belief of environmental states and consequently making optimal decisions. Kong et al. (2024) integrates an API Retriever, LLM Finetuner, and Demo Selector within a unified framework to tackle the challenges of task planning and tool usage in complex real-world systems. Expel (Zhao et al. (024) maintains an insight-level memory for the L.M agent. After interacting with the environment, the agent learns from successful trajectories and avoids failed trajectories to summarize series of insights. These insights are used to enrich the LLM agent's prompts, thereby enhancing its decision-makinn capabilities. However, most of them require the environmental functions to enhance decision-making and lack the capability to leverage experiences from other tasks within the same domain.
17

---

Published as a conference paper at ICLR 2025
There also some LLM agents aiming to be zero-shot planner. Kwon et al. (2024) design many lask-agnostic prompts to investigate which design choices this prompt are the most important, thus developing zero-shot robot manipulator. Huang al. (2022a) decompose high-level tasks into mid-level plans without any further training of pre-trained LLMs, and propose procedure to semantically translate the plans to available actions in zero-shot manner. PS Prompting improves upon zero-shot-CoT at the prompt level, enabling LLMs to implement planning capabilities (Wang
al. 2023a). Gkanatsios et al. 2023) present an energy-based framework that converts language instructions into optimizable functions to guide object rearrangement through visual- policies.
However, these approaches mostly design prompts for specific domains and still cannot achieve
optimal decision-making.
RL approaches in agents. Traditional RI. and MARL approaches require RI.-based optimization techniques to update the parameters of agent policies. These methods either collect interaction data between agents and the environment (Rashid et al. 2018 Chai et al.) 2021 Hu et al. 2023) or use supervised learning to construct world model that serves a digital replica of the real environment (Chai et al. 2024 Egorov & Shpilman 2022), providing imagined interaction data (Hafner et al.
2020). In contrast, the training of LLMs involves vast and diverse dataset, enabling the construction of task-specific basic policies and world models through task descriptions natural language (Hao al. 20233 Yao al.] (2023)), thereby eliminating the need for parameter updates of traditional methods. Reflextion (Shinn et al., 2023) maintains an RL-like framework, which includes Actor and Critic modules used for action output and action evaluation, respectively. However, it does not employ RL-based optimization at the transition level based on the Bellman equation. Instead, it heavily relies on the decision- decision-making capabilities of the LLM itself, leading to poor performance in long-horizon tasks. Zhang et al. (2024c) utilize RL-based critic regression to learn sequential advantage function and treat the LLM planner as an optimizer to generate actions that maximize this function. Xu et al. (2023) propose LLM agent for Werewolf game, which utilizes LLM to generate candidate actions the current state, and then uses a one-step population-bas -based RL training process to select the optimal action from these actions. MCTS is also traditional RL approach, but it requires prior knowledge about environmental functions and focuses more on the search process. Some of the previously introduced methods (Hao al., 2023 Zhao al. 2023) MCTS for planning and achieve optimal decision-making based on some strong assumptions, but still do not perform satisfactory performance in long-hou long-horizon tasks.
18

---

Published as a conference paper at ICLR 2025
C PROOF FOR THEOREM
Before presenting the proof of Theorem 1 we first introduce Theorem 2 from Auerr et al. 2002) and provide part of the derivation facilitate the subsequent steps. It is important note that some variables in this section may cause confusion with similarly defined variables in the main text. These variables are introduced temporarily improve the readability of the derivation and only valid within the proof of this section. For instance, while denotes the time in the main text, it will be used here to denote the number of UCB rounds.
Theorem 2. For all m 1, if policy UCB is run state with available action, performing each action yields arbitrary Q-value distributions Pi..... Pm with support in [0, 1]. then its expected regret after any round number of T is most:
((s(s)
m
[8 E
(1+
A.).
(5)
A
EA,50
where Q(s, 1), Q(s, m) are the expected values of Pi,...; P.n.
Proof. The regret is defined follows:
Regret- Σ A,E(C,(T)), jA,30
(6)
where A; =.=--(ss.i) (Q(s(Q(ss)))) i)). and C:(T) denotes the selection number of machine over the first T rounds. The key of the derivation lies in the decomposition of ([C((T)) Due the initial evaluation conducted by UCB for each machine, Ci(T) can be rearranged as:
T C:(T)
Σ (4 t=1
(I= t +
where {.} is an indicator function. If the event { i} occurs, meaning that the UCB policy selects machine in round t, then the indicator function outputs 1. Otherwise, it outputs 0.
Auer et al. 2002) derived the conclusion of the theorem through step-wise approach using this equation. For detailed derivation process, please refer to the original proof in (Auer et al. 2002).
The subsequent derivations in this paper require only the few steps outlined above.
Theorem 1. Suppose the player employs the node selection rule defined in (2). Then, in scenario with m available actions state s, the bound of the expected cumulative regret is:
In N(s)s)
) + (1+ 2
m
Regret- = Σ (,E,C,(N()) j:A,20
S [S
Σ :A.20
M Ad),
(8)
A.
where A; Q (s. () for action index i in ..., m) and Q" = max, (Q(,,)), N(8) is the visit count of s, and (,)) is the selection number of action i.
Proof. Since the definition of regret has changed, we will continue derivation starting from the decomposition of (T), where T has the same meaning with N(s). We will ultimately demonstrate that, due the presence of the virtual node y. the player can achieve the same upper bound on regret as UCB. In the multi-armes bandit variant, action can be selected only if y has been selected at least i times. Thus, we define the selection number of y at round as A(t), and we have:
T
C(TT) Σ 14=i,4 il= A() 2
Σ
14=i,A0 2 2
T Σ [4=1,A4 ( 1,1)
-
(9)
1=#1
T
Σ - 1) Σ 1-(=i
1=42
t=11
19

---

Published as a conference paper at ICLR 2025
where E(t, i) denotes the event in which player select for the i-th time at round t. The derivation in the first line follows from the fact that A(1) = and A(t) S -- 1. Therefore, when S i, the event A(t) 2 i cannot occur. The derivation from the first line the second line primarily relies rewriting the original event A(1) 2 i as the union of two mutually exclusive events, A(t - 1) 2 i and g(t 1). By repeatedly applying the above decomposition operation to the first term of the formula, we have:
n
Ci(T) = Σ
Σ
U-=iEE( -(,,,((--k,)))--
E
E44
14=-i k.))
k=lt=itk
(10)
1-1
= Σ -4= +=+1
Σ
Σ 1=2+1
14=
By setting x = t - le and exchanging a series of inner and outer loops, we obtain a more interpretable formula. This formula first determines the round in which is selected for the i-th time by iterating the variable æ over the interval [ - 1], followed by iterating the time t to compute the selection number of action i. Clearly, the above expression is less than n, leading directly to linear regret bound. However, to achieve the goal of deriving a sub-linear bound, we should use the following fact:
Ci(T) S max Σ
--= (,=,, max Σ (1=
1=1+1
1=++,,n
(11)
Σ C4-1i
Thus, we can directly proceed to Equation 7 and its subsequent derivations, ultimately proving that in this case, the player can still achieve the same regret bound:
N())
[8
Σ 6A:50
+
A.),
(12)
A
where (s) is equivalent to
In the calculation of the vUCB value in 3). the confidence bound for virtual nodes includes an additional coefficient g(x) compared to that of actual nodes. Assuming the number of available actions is m, we expect the virtual node be selected at most times, and the coefficient should decrease the size of 8(s) increases. Therefore, it should possess the following characteristics:
g(m) = 0 and Ög dr <= 0
(13)
However, since the number of available actions cannot be known in advance when planning in the LLM-based imagination space without environmental tools, the first property cannot be satisfied. Therefore, in practice, we use g(x) = (e Eg) as substitute, whose value rapidly diminishes as the size of 6(s) increases.
Theorem 3. Let the expected TV-distance between two transition distributions be bounded at each timestep by en and the policy divergence be bounded by cq. Then the true returns and model returns of the policy are bounded as:
27(em + 2ex) (1 71- + 4rmaxEg
(14)
(1-7)
In the above theorem, 77 is the returns of the policy in the true environment, n is the returns of the policy under the L-LM- world model. Fmax is the maximum value of rewards, 6m (ax Drv (T(+)s, ss, (T(-ls, a; 0))] is the generalization error of the LLM-based world model, (s, a) is sampled from MLAQ's memory module M, and T(+ss, a; 8)) is the transition distribution of the LLM-based world model. e, (Iv(=[[TTM) is the policy divergence between greedy policy æ derived from Qelearning (output argmax actions) and the data-collecting policy in the imagination space M- Please refer the proof of Theorem 3.1 in (Janner al. 2019)
20

---

Published as a conference paper at ICLR 2025
D EXPERIMENTAL ENVIRONMENTS
I
(a)
(b)
(c)
Figure 7: Experimental environments. (a) BlocksWorld. (b) Sandwich domain of RoCo-benchmark. (c) Sort domain of RoCo-benchmark.
As shown in Figure 7 evaluate the methods in Blocks (single-agent) and RoCo (multi-agent) benchmarks. Next, we will provide a detailed description each task, as well as the state and action space.
D.I BLOCKSWORLD
Worlk is well-known single-agent benchmark for evaluating LLM agents (Valmeekam et al. 2022), where an agent should rearrange some blocks into multiple stacks in particular order. In this work. choose the tasks with four blocks evaluate our methods. Нao al. (2023) has grouped all tasks according to the optimal step, and we randomly choose to 30 tasks from each group for evaluation. As shown in Figure 8 (a), the state space in the original BlocksWorld codes has a lot of redundant information, which may help agents make better decisions, but may lead to errors when making predictions by the world model. Therefore, as shown in Figure (b), we adjust the format of them in more concise and clear manner without affecting the validity of states.
The state records what the agent's hand is holding: ( ('Empty' or 'Holding <block>") and what each block is ('on <block>', 'on table', 'in hand'). The action is instruction that moves one block, which is one of STACK, UNSTACK, PUT, and PICK UP.
PICK UP <object>: Execute if hand is empty, and <object> on table and no block on Pick up <object> from table. After execution, the agent will be holding the <object>,
2. UNSTACK cobject>: Execute if hand is empty, <object> another block and no block is on it. UNSTACK <object> from another block. After execution, the agent will be holding the <object>.
PUT DOWN <object>: Execute if hand is holding <object>, Put down <object> on table. After execution, the agent's hand will be empty.
4. STACK <object> ON <target>: Execute if hand is holding <object>, and no block is <target>. Stack <object> on the top of <targets. After execution, the agent's hand will be empty.
D.2 ROCO-BENCHMARK
RoCo-benchmark built upon the Mujoco engine (Todorov et al. 2012) construct several multi- arm scenarios. RoCo-bent AeCCo-benchmrk provides 6 different tasks, including those containing two agents and those containing three agents, and the robotic arms in them have their own movable range and require full cooperation to complete the given task. We choose two of these tasks, Sort and Sandwich, to evaluate our method. In Roco-bert benchmark's 's codes, there is no fixed format for the state space, and the states are intermingled in the form of information within the agent's 's prompt. Therefore, have constructed the state spaces for these two tasks in a concise manner, while keeping the action spaces consistent with the original codes.
21

---

Published as a conference paper at ICLR 2025
have that, the orange block is clear, the yellow block is clear, the hand is <state?, the blue block is <state>, the orange block is <state>, the red block is <state> the yellow block <state>.
[Sttee]
[Food States]
bread slicel: <food state>
bread_ slice2: <food state>
bacom: <food state>
cheese: <foo state>
tomato: <food state>
cucumber: <food state>
ham: =food state>
beef patty: <food state>
[Robot States]
1. Chad's gripper is <gripper state>
[State]
2. Dave's gripper is <gripper state>
[Block States]
(c)
blue block: <block state
orange block: <block state>
[State]
red block: <block state>
[Cube States]
yellow block: <block state>
blue aauesssuaeeeee is on <location>
[Hand State]
pink_polygon is <location
<hand state>
yellow trapezoid is <location>
Figure 8: State templates. (a) Template in original BlocksWorld codes. (b) Modified Template in Blocks (c) Template Sandwich domain of boCo-benchmark. (d) Template Sort domain of boCohbennc..rr.
Sandwich domain. There are two robots, Chad and Dave, trying to make a sandwich following the given recipe order. Chad can only reach for food items the right side of the table, while Dave can only reach for food items on the left side, with each food item starting in a fixed position. The robots need place the food items the cutting board in the middle of the table one by one, following the order given in a recipe. Different types of sandwiches contain different combinations of food items. Take the 'bacon' sandwich as example, it requires 'bread slicel', "bacon', 'cheese', 'tomato', and bbreadssice2"". In a bacon recipe, apart from the fixed positions of 'bread_slice "bread_slice2'
and 'bread_ slice2' as the two ends of the sandwich, the order of the other food items can vary across different recipes. Therefore, one type of sandwich may have multiple different recipes, We group all the recipes according to the optimal step to evaluate the agents. In the decision-makee robots cannot execute PUT actions simultaneously within one timestep to avoid collisions.
process,
As shown Figure X (c), the state records the state of each food item ('on left/right side', 'atop <another food item> or <cutting_ boardd'bar''.o 'gripped by <robot>") and the state of each robot's gripper ('holding <food item>' or "empty"). The action is an instruction for each agent interact with the food items, which is one of PICK, PUT, and WAIT.
1. PICK <obj>: Execute if gripper is empty and the food item's state is not 'atop <obj>'. Pick the <obj> from the table. After execution, the robot's gripper will be holding <obj>.
2. PUT <objl> <obj2>: Execute if gripper is holding <objl>. Put <objl> on the top of <obj2>. After exccution, the robot's gripper will be empty.
3. WAIT: Do nothing.
The action checker in the Sandwich environment strictly limiting the placement of food items to follow the recipe, rather than allowing it to be placed in an incorrect order. Therefore, the size of agents' available action set in each state is relatively small. The agent can only choose to wait, pick up the next food item, or place the food item from the gripper according to the recipe, so that RoCo could achieve high success rate in this task, but the optimal rate is still low.
22

---

Published as a conference paper at ICLR 2025
Sort domain. There three robots, Alice, Bob, and Chad, trying to sort three cubes onto their corresponding panels. There are a total of seven panels. Alice can access panels to 3, Bob can access panels 3 to 5, and Chad can access panels 5 to 7. The three cubes a blue square, a pink polygon, and yellow trapezoid, with their respective target panels being panel2, panel4, and panel6. In the decision-making process, panel cannot contain two more cubes, and cube cannot be grasped by two robots simultancously.
As shown in Figure (d), the state records the state of three cubes (on <panel>) and the action an instruction for each agent to interact with the cubes, which is one of PICK & PLACE and WAIT.
1. PICK <object> PLACE <target> Pick up <object> and place it onto <target>, where <object> is a cube and <target> is a panel.
2. WAIT: Do nothing.
Given the complexity of this domain involving more agents and expansive available action spaces, its difficulty surpasses that of the Sandwich domain. In the Sandwich domain, decision trajectories linear and decision loops are non-existent, meaning that the multi-agent system cannot revert to its initial state after series of decisions. However, in the Sort domain, the occurrence of loops adds to the challenge of reaching target states. MLAQ could address this challenge by leveraging RL-based optimization make long-term optimal decisions.
E EXPERIMENTS ON CRAFTER
To validate MLAQ's ' performance in partially observable scenarios with high complexity, we chose Crafter (Hamnr 2022 2D version of MineCraft) as our experimental environment.
In Crafter, players need to mine diamonds on 64 64 map. To achieve this goal, there are 16 sub- goals complete (collecting woods, stones, make wood pickaxe, etc.), of which are essential for diamond mining, making it challenging task with extremely long decision sequences. Additionally, this is an extremely difficult sparse reward problem, where reward signals are only provided when successfully achieving a sub-goal.
Players can only observe 9 * 9 local map centered on their position, and we adopted SmartPlay's (Wu et al.] 2024) natural language observation setting: for multiple items within the field of view, only the nearest one will be shown in the observation. For example, even if there are 3 trees in view, the observation will only show "tree is 3 steps to your north-cast", which further intensifies the partial observability of Crafter.
To achieve optimal decision decision-making in this task (and other stochastic problems), mainly because the same state-action pair may lead to different next states, we made two minor modifications to MLAQ:
1. We remove the Prediction Checker, as it's difficult to provide effective verification in the cases with stochasticity.
2. We remove MLAQ's re-utilization of memory transitions during planning, because the same state-action pair may lead to multiple next states, and direct re-utilization would lose stochasticity. However, this can be adjusted as needed in practical, such as starting random re-utilization when the number of next states corresponding to the same state-action pair exceeds threshold.
we compare MLAQ with several existing LLM agents, including EnvGen (Zala et al. 2024 and DiVE (Sun et al. 2024), and the results of RL agents and human are from (unn et al.
In table 7
2024
Table 7: The scores of experimental methods in Crafter domain.
Method
Human
gt
MLAQ-script
MLAQ
DiVE
EnvGen
Dreamer-Vi V3
PPO
Random
Scores
50.5%
46.2%
42.6%
39.9%
35.9%
32.2%
14.5%
4.6%
1.6%
We calculate The scores using the same methodology as existing works, which is determined by the ratio of unblocked achievements. The data of MI.QQ obtained using the same testing methods
23

---

Published as a conference paper at ICLR 2025
SmartPlay (Wu et al. 2024). In Crafter, the move action exhibits strong stochasticity, making impossible for players to infer post-mn motemette observations from local information. We conducted three experiments:
1. ML.AQgtt utilizes ground-truth environment simulation for move actions, providing the exact next state for given state-action pair, thus effectively eliminating movement stochasticity.
2. MLAQ-script generates post-movement observations through scripted randomization (e.g.. after moving north, nine unknown areas appear at the northernmost position, with randomly assigned items like stone, sand).
3. MIAQ reconstructs the aforementioned script into prompts, allowing LLM fully imple- ment an LLM--based World Model.
The results align with expectations: ML.AQ-tt achieves best performance using ground-truth environ- ment, consistently crafting iron pickaxes but struggling to explore the location of diamonds within limited steps in 64 * 64 maps. MLAQ-script and MLAQ perform similarly but significantly below MLAQ-gt, gt, indicating stochasticity substantially impacts MLAQ, though still outperforming other existing LLM agents and RL agents. It's worth noting that these experimental data are averaged from eight tests, serving demonstrate reviewers the potential of MLAQ in complex environments.
24

---

Published as a conference paper at ICLR 2025
F PSEUDO-CODE FOR THE OVERALL ALGORITHM
We present detailed pseudo-code for our MLAQ framework in this section, and these algorithms are used fully describe the overall process presented in Figure and Figure We have simplified the entire algorithm process into the flowchart in Figure too facilitate a better understanding of how MLAQ makes optimal decisions for new task.
New Task (Orginal and Target Statos)
Expend Replay
Memory
Bost Policy from Qulforner Replay Buffer
Perfeee the Bosl Pullcy in Envinemmnnt
fenction
Optimal
Exspand Memory & Keplay Butfer via LLM imagination
Figure 9: Flowchart of MLAQ agent to obtain the optimal decision sequence for new task, with the input being the domain description and domain-specific memory and the output being the Q function.
F.1 OVERALL ALGORITHM FOR MLAQ AGENT
At first, Algorithm | shows how MLAQ agent obtains optimal decision sequence in given task. Its inputs are the environment dynamics and reward function with the domain description in natural language provide interaction between the true environment and the agent (or multi-agent system), which is also depicted the left side of Figure (a). At the beginning of the optimization of MLAQ agent, initialize empty memory M and Q function Q(---) to set all Q values to -1. As described in the experiment section, we then scan all tasks of a domain to gradually expand the memory module through environmental and imaginary interactions.
For each single task, we have a unique pair of original state 8o and target state 8, and initialize a replay buffer D for this task. Before making decisions in this task, MLAQ agent will try to find this task's 's best trajectory from the memory to expand the empty replay buffer, and a task-specific Q function Q(---) will be obtained based on D. If the agent fails in finding an available trajectory from the memory, meaning that the ((so, a) no larger than the threshold Q for any action a, we would skip the following environmental interaction phase to directly perform the imagination process until an available decision sequence is found. Otherwise, the agent interacts with the environment by making best decisions according to Q(:-)) to evaluate the optimality ( 8) of the resulting decision sequence. The indicator 1 (sp, 3) is set to True only if the environmental sequence is optimal.
In order to expand the memory and replay buffer to optimize the agent, keep performing imaginary interactions until the optimal sequence obtained for the indicator 1( 8) is True or the maximum trial number K, is reached. As more tasks are scanned, the size of memory M will gradually increase, and we will finally output to assist the MIAQQ agent in making optimal decisions under any tasks in this domain. While for the tasks have not been scanned before, MLAQ would also the transitions stored in M to reduce LLM queries and obtain the optimal decision sequence much easier.
F.2 LLM-BASED IMAGINATION FOR MLAQ
Algorithm input the original state Sp and target state with domain description т. The memory M contains transitions from other tasks and the temporary replay buffer D is specific to the current task. At the beginning, we should first add a virtual node (if does not exist) for the original state so to start the imaginary interactions. Then, we perform the phases of node selection and transition imagination to expand the memory and replay buffer, and the overall process has already described section B The outputs of the imaginary interaction are the expanded memory and replay buffer. It should be noted that after each step of transition imagination, we would perform an "Expand Buuffrrr method to find available trajectory with better optimality within M and add nodes along this trajectory D.
25

---

Published as a conference paper at ICLR 2025
Algorithm 1: Obtaining optimal decision sequence for MLAQ agent
Inputs:
The environment, including dynamics s' T(s,, a) and reward function r = R(s, a, s');
Task description in natural language т;
Initialize:
memory M and Q function Q(:-) -1;
for task =1, ...
# Each task has unique original state so and target state Initialize a replay buffer DD for this task; Set the optimal sequence indicator for this task [(s,,, 3) to False;
10
while (So. 3) is not True AND trial < Kg do
11
# Expand the replay buffer and gel lask-specific function based on the memory D= Expaand-BBBi Buf 8
13
Q(.-) - 8; D): if max, Q(so. a) < Q then # An available trajectory does not exist based D
15
Skip the environmental interaction phase and turn to the next trial of imagination; end
17
18
# Environmental Interaction Phase while st is not AND < T do
19
20
Get the optimal action a₂ = arg max, Q(s;, if is available for the environment then
22
Perform action to the environment 8 T(st, a,) and rt = R(s,, dey 864 1): else
Remove transition (st, az, r(s,, oe), c(st, a.)) from M and D; Get environmental feedback F and store (st, dze F) into M and D;
end
27
Store transition (st, az 814811 into M and correct the wrong transition;
Se+1;
29
end
30
8; then 1 8) is True if length of decision sequence is optimal else False;
32
else
1(50. 3) is False;
34
end
if 11 (so, 3) is not True then
36
# Perform a round of imaginary interaction to expand the replay buffer and memory M,D ImD==maagnttiio M D);
37
end
end
40 end
Outputs:
42
memory M;
26

---

Published as a conference paper at ICLR 2025
Algorithm Imaginattonn(s,::11 8; 8,T,M D) guided by UCB values
Inputs:
Original state so and target state 8 of this task with domain description in natural language T;
The memory M and replay buffer D;
Initialize:
Add virtual node the first child node of the original state so, and (sss + ((s)) + 1;
while 8t is not & AND < ' do
# Node Selection Phase
Get the optimal action a, = arg max, a,,
c(st, 0+) is virtual node then
# Transition Imagination Phase
Action available indicator (a,) False;
12
# Basic policy makes decisions and action checker verifies if the output is available while (a;)
13
True AND trial < Ka do
Get non-virtual action through LLM-based basic policy at ~ (T(;; M): if (st, az) memory M then
(()) Retrieve transition data (st, (t, r(s,,, c(s;, az)) from M D:
(a,)) = True;
17
18
else
Check the action (a,) = end
19
21
end
22
This trajectory is failed and terminated if 1 (at) is False; if 11 (a,) is True then
23
24
# World model predicts and prediction checker verifies if the output available
25
Prediction available indicator il (st+1) <- False;
while {(141) is not True AND trial < K2 do
Predict the next state 81+1 T(5t, a;; T,)
Check the prediction il (81+1) (, ae 814117 M);
29
end end
30
31
This trajectory is failed and terminated if (8r+ is False;
32
else
St+1 c(st, ar); s;41 is not in D then
34
Add virtual node the first child node of 5c+15
36
end
37
end
St 8141 and ((s)) <- (s()) +1;
39
8, is 8 then
40
r: +1;
41
else if is a failure state then
42
rt -1
43
else
44
Fr = O:
end
46
Expand the replay buffer and memory Store transition (st, a, Tt, St41) into the replay buffer and memory (if does not exist); # Expand the replay buffer from memory if possible
48
D Expand-Buffen fer D)
end
51 Outputs:
The expanded memory M and replay buffer D;
27

---

Published as a conference paper at ICLR 2025
F.3 EXPAND THE REPLAY BUFFER FROM THE MEMORY
Algorithm 3 is used to expand the task-specific replay buffer based on the transitions in the memory. firstly finds available trajectory with better optimality from M and then store the transitions of this trajectory in the replay buffer 'D. This will effectively improve the exploration efficiency of the MLAQ agent within the imagination space, For example, if the replay buffer is empty and there exists an available trajectory from to s, then the nodes and edges in this trajectory will be added to D. Therefore, the first trial Algorithm would not require any LLM queries to get an available policy for this task.
Furthermore, if not empty, this algorithm could also help the MLAQ agent aggregate transitions from memory for enhancing optimal decision-making capability in complex tasks. For example, if the agent explores a new transition and the next state corresponding to this transition has trajectory with better optimality towards the target state 8, the MLAQ agent could add the nodes in this subsequent trajectory to D without extra exploration within the imagination space.
Algorithm 3: Expand-Buffer(es, M,
Inputs:
Original state 8o and target state of this task with task description natural language т; The memory M;
2(.-))=Q. -Q-date-U (5o, M;M);
if' maxx ((so, a) Q then
8t 6- sp;
while 8; is not do
Get the optimal action a, = arg max, Q(s,,
Retrieve transition data (st. ae, r(s), c(s), a:)) from M to (if does not exist);
10
8: f c(3t, as);
11
end
end
13
Outputs:
14
The expanded replay buffer D;
28

---

Published as a conference paper at ICLR 2025
F.4 Q FUNCTION UPDATE
Algorithm is employed to get Q function for the given task So and 8 on the basis of X, where X could be the memory M and the replay buffer D. At first, initialize the function Q values to -I, and assign rewards for all transitions based the target state 8. Then, the algorithm would perform multiple loops Q-learning process get the approximated function Q-:-). This algorithm will be used by the Q-Planner in Algorithm I and the replay buffer expansion process in Algorithm
all
Algorithm 4: Q Update(so, 8;
Inputs:
The original state so and target state of the given task; The memory module or replay buffer X:;
Initialize:
Initialize Q function Q(:--)=- -1; all transition (s, u, 8) in X, if 8, then +1; if is failure state, then r -1;
otherwise, r -= 0:
for loop & = 1, 3,... do
for (st, Pt. Tr. St+1) E X' do
Perform the following Q-learning update rule for this transition;
Qu+I(s,, Qu(se, ar) + (re + y max Qn (8)41,a)) Qu(s,,
11
end
12 end
Outputs:
14
The Q function Q(:-));
29

---

Published as a conference paper at ICLR 2025
G DETAILED PROMPT TEMPLATE
This section presents the detailed prompt templates for the basic policy, world model, action checker, and prediction checker. The prompt template has multiple parts to provide different types of informa- tion for the modules. The prompts in the purple cube are the component-specifee prompts to guide the agent in making decisions, predicting next states, checking actions or predictions. In the prompt, we guide the corresponding modules step-by-step to enhance the accuracy of their outputs. The prompts in the green cube contain some task-specific information, and prompts in the blue cube contain some self- reff-provided information generated from self-examinations and JUCB-based planning. The prompts in grey cube is the 'user prompt', which contains the current information for the modules.
G.1 BASIC POLICY PROMPT
The prompt shown in Figure 10 requires the current state and target state in natural language make decisions. the [Detailed Instruction for Policy] part, we decompose the decision-making process into five parts: [Action Plamning]. [Action Conclusion]. [Action Constraints Check], [Forbidden Action Check], and [Action Output]. gradually guiding the agent to make decisions that satisfy the constraints. As for the basic policy of multi-agent systems, we directly follow the prompt design of RoCo, and the prompts can be found in (Mandi et al. 2023). The policy mistakes are the environmental feedbacks of previous wrong actions stored in the memory M. The policy feedbacks are the temporary feedbacks provided by the celf-examination only under the current state. The forbidden actions are the non-virtual actions of the current state node's child nodes.
Policy
You are a planner. You have received a [State] from the external environment. Now, you need to output action reach the target [State] from current [State]. Follow the [Detailed Instruction for Policy] strictly.
Detailed Instruction for Policyl
You should output your thinking process step-by-step in the following order: [Action Planning], [Action Conclusion], [Action Constraints Check], [Forbidden Action Check], and [Action Output]
The following lists the [Task Description], [State Space Definition], and [Action Space Definition]. TTak Description]
[State Space Definition
[Action Space Definition
[Policy Mistakes| (long-term)
the current state, you have chosen a wrong [Action], and the environment provides the [Feedback]: [Wrong actions and feedbacks]
[Policy Feedbacks] (short-term)
The following actions are the wrong actions you have made. These foedbacks are from the previous imagination, and may not be correct. However, you still need to pay attention to them and avoid making the same or similar mistakes. [Wrong actions and feedbacks]
[Forbidden Actions]
You should try your best to output an action that not list below ...
[Action List]
Now you should output your action according the following current state and target state. [state]
(target state)
Figure 10: A simplified template for the basic policy prompt.
30

---

Published as a conference paper at ICLR 2025
G.2 ACTION CHECKER PROMPT
The prompt shown in Figure 11 requires the current state and the action should be checked. In the [Checker Instruction] part, decompose the checking process into three parts: [Action Constraints Check], [Conclusion Justification), and [Checker Conclusion]. We provide the previous mistakes and the corresponding feedbacks stored in the memory under this current state for the action checker to avoid the same mistakes.
Action Checker
You are an action checker. You should check if the [Action] valid based on the given [State]. You should follow the [Checker Instruction] to check the [Action].
[Checker Instruction|
You should follow check instructions in [Detailed Constraints for Action] to check the [Action] step- by-step. Take the mistakes in the [Action Mistakes] into consideration.
The following lists the [Task Description], [State Space Definition], and [Action Space Definition]. [Task Description] [Action Space Definition]
[Action Mistakes] (long-term)
current state, you mistakenly believed that wrong action was the correct one: [Wrong actions and feedbacks)..
Now you should check the following [Action] based on the given [State]. You have to check the instructions step-by-step strictly rather than your own thoughts. (state] (action)
Figure 11: A simplified template for the action checker prompt.
G.3 WORLD MODEL PROMPT
The prompt shown in Figure 12 requires the current state and action to predict the next state. In the [Predicting Instruction] part, we decompose predicting process into three parts: [Interaction Item Pool], [Action Forward Rule] and [Prediction Conclusion]. The prompt also contains the feedbacks from the long-term and replay buffer. In fact, LLM can better handle the task of predicting with step-by-step reasoning, and we also find the prediction accuracy of the world model to be nearly 100% in the experimental section.
G.4 PREDICTION CHECKER PROMPT
The prompt shown in Figure requires the current state, executed action, and the prediction should be checked, In the [Checker Instruction] part, we decompose the checking process into three parts: [Prediction Format Check), [Prediction Rule Check), and [Checker Conclusion]. We also provide the previous mistakes and the corresponding feedbacks stored in the memory under this state-action pair for the prediction checker avoid the same mistakes.
31

---

Published as a conference paper at ICLR 2025
World Model
You state predictor. You will be given [State] and [Action] input. should predict the [State] after the [Action] is applied to the given [State]. You should follow the [Predicting Instruction] to predict the [State].
[Predicting Instruction]
Please output your thinking process step-by-step by following theses steps: [Interaction Item PoolJ. [Action Forward Rule] and [Prediction Conclusion]
The following lists the [State Space Definition].
[State Space Definition
[State Templatel Detailed Constraints for Statel
[Prddctionn Mistakes] (long-term)
In the current state and action, you have outputted a wrong prediction follows: [Wrong predeitions and feedbacks)...
[Prediction Feedbacks] (short-term)
have send your prediction to checker that your prediction is incorrect...
checker to check whether your prediction is correct. The
[Feedbacks]...
Now, you should predict the next [State] after the following [Action] is applied to the given [State]. You have follow the instructions strictly rather than your own thoughts.
[state] [action]
Figure 12: simplified template for the world model prompt.
Prediction Checker
You prediction checker. You will receive [Statc], an [Action], and the subsequent [Prediction]. You should chock if the [Prediction] valid based on the given [State] and [Action]. You should follow the [Checker Instruction] check the [Action].
[Checker Instruction]
You should follow the instructions [Detailed Constraints for Prediction] to check if the [Prediction] valid based on the given [State] and [Action]. Take the mistakes in the [Prediction Mistakes] into consideration.
following lists the [Detailed Constraints Prediction].
[Detailed Constraints for Prediction]
[Prediction Mistakes] (long-term)
the current state, you mistakenly believed that wrong prediction was correct one: [Wrong predictions and feedbacks]
Now you should check the following [Prediction] based on the given [State] and [Action]. You have to check the instructions step-by-step strictly rather than your thoughts. [state] [action]
[predicion]
Figure 13: A simplified template for the prediction checker prompt.
32

---

Published as a conference paper at ICLR 2025
H DETAILED EXPERIMENT RESULTS
H. BASIC POLICY ANALYSIS FOR MULTI-AGENT TASKS
As mentioned in the experiment section, our MLAQ agent framework compatible with both single-agent and multi-agent scenarios. In multi-agent scenarios, it provides two basic policy decision paradigms. The first one treats the entire multi-agent system as whole for centralized becision-making, where it takes the global state as inputs and outputs joint actions for the system. The second one adopts RoCo's dialog mechanism (Mandi et al. 2023), where the agents conduct multiple rounds of dialogue with each other and determine the joint actions when reaching consensus: as, ~ (. | Pt.c). where Pi, [d........ 1 is the chat history between agents. d represents the message output by agent at the dialog round
We evaluate the agents using these two basic policies in the Sort and Sandwich domains. As shown in Table the results indicate that, with similar token consumption, the performance of centralized decision-making significantly decreases, with env replans increasing nearly tenfold in the Sandwich domain. The complexity of centralized decision-making significantly surpasses that of decentralized decision-making, thereby diminishing the effectiveness of basic policies in obtaining available or optimal actions.
Table 8: Performance comparison of using dialog and central paradigms in multi-agent domains.
Task
Methods
Success Rate
Optimal Rate
Token
Env Replans
Optimal Gap
dialog central
0.98
0.75
175560
0.06
0.26
Sort
0.88
0.59
199753
0.04
0.57
dialog
0.95
0.45
208024
0.07
0.61
Sandwich
central
0.81
0.24
131244
0.62
3.24
H.2 MAIN RESULTS IN THE SANDWICH DOMAIN
Due to space limitations in the main text, we provide the experimental results in the Sandwich domain in this section for reference. As shown in Table although MLAQ is superior to RoCo in success rate and outperforms RoCo in long-horizon tasks (I0-step), the token consumption of MLAQ is higher than that of RoCo. This is mainly due to two reasons. Firstly, the similarity between state transitions plays an essential role token consumption, with tasks exhibiting higher similarity being better able to leverage stored transition data in memory to reduce token consumption. The Sandwich domain, with its relatively fixed trajectories compared to Sort, exhibits lower similarity across different tasks. This is evidenced by the average memory re-utilization ratio between the domains, notably lower in Sort. This characteristic the Sandwich domain not only contributes to RoCo achieving a higher optimal rate but also diminishes MLAQ's token saving capability. Secondly, MLAQ itself involves world model prediction and self-examination, leading to an average token consumption per time step exceeding RoCo's.
33

---

Published as a conference paper at ICLR 2025
Table 9: Comparison of multiple metrics between MILAQ and RoCo in Sandwich domain.
Metrics
Methods
6-step
8-step
10-step
Average
RoCo
1.00
0.64
0.92
0.75
Success Rate
MLAQ
1.00
0.92
1.00
0.95
Env Replans
RoCo
2.50
3.87
4.33
3.86
(n-shot)
MLAQ
0.00
0.11
0.00
0.07
RoCo
0.75
0.36
0.25
0.37
Optimal Rate
MLAQ
0.00
0.08
0.83
0.29
MLAQ
0.25
0.31
0.83
0.45
RoCo
20740
35072
65027
42266
Average Token
MLAQ
74881
63555
77949
68746
MLAQ
184609
220521
188752
208024
RoCo
0.50
1.13
1.83
1.27
Optimal Gap
MLAQ
1.25
1.88
0.17
1.33
MLAQ
0.75
1.35
0.17
0.96
Memory Re-Util. Ratio
0.31
0.63
0.57
0.58
I SYPER-PARAMETER
Table 10 shows the hyper-parameters presented in the MLAQ training process. Except for the environmental horizon T, all other hyper-parameters remain constant across all experiments, where T' is set to 20 for the BlocksWorla experiment, 16 for the Sandwich experiment, and 8 for the Sort experiment.
In this paper, all experiments conducted using the GPT API interface, without involving CPU or GPU usage. The total cost the API resources used in this paper does not exceed 1500 US dollars, including preliminary tests, comparative experiments, and ablation experiments.
Table 10: Hyper- paremeppramettrs presented in the MLAQ training process
Hyper Parameter
Value
LLM source
190-4-01255previcee
Learning rate a
1.0
Discount y
0.995
UCB weight Wy
2
vUCB weight Wy
4
vUCB threshold Ey
0
Maximum trial number for imagination K,
2
Maximum trial number for prediction K,
2
Maximum trial number for policy K,
2
Q threshold Q
0.5
Q update loops
20
Environmental horizon T
20
34

---

Published as a conference paper at ICLR 2025
J. PROMPT EXAMPLE
J.1 POLICY PROMPT AND RESPONSE OF THE STATE-BASED DECISION MAKIIN
In the LLM agent constructed by our method MLAQ, both eecision-makinng in single-agent scenarios and centralized decision-making in multi-agent scenarios rely on the global state. We use the prompt template in Figure T0 to query LLM and output agent actions or joint actions of the multi-agent system. In the following, we provide an example of a policy prompt in the Sort domain, which includes all the elements in the template shown.
Policy
[Detailed Instruction for Policyl
You should output your thinking process step-by-step in the following order. You have follow these steps one by one plan the [Action]: [Action Planning], [Action Conclusion], [Action Constraints Check], [Forbidden Action Check], and [Action Output].
The most important thing: Follow the instructions step-hy-step and ensure each step completed precisely. Repeat the instructions and fill in the blanks '[I' without introducing any modifications or additional content.
I |Action Planningl: Plan the [Action] of the mulli-agent system step by step and list the thinking process.
+ [Action Conclusion]: Conclude the [Action] in the format of [Detailed Constraints for Action].
|Action Constraints Check]: Follow the steps in [Detailed Constraints for Action] check the [Action] step by
step.
|Action Revise]: there is no incorrect checking result in [Action Constraints Check), you can pass to [Forbidden Action Check]. Otherwise, you should revise the action. Take the feedback from [Action Constraints Check] into considcration, and follow these steps to revise the action: [New Action Planning] and NewAction Constraints Check).
+ [Forbidden & Mistake Action Check]: will provide some forbidden joint actions, and the robots cannot choose the forbidden and mistake actions. Note that what forbidden the joint actions of the multi-agent system, rather than single agent actions that appear in these joint actions.
list chosen action.
list all actions in the (Forbidden Actions] and [Policy Mistakes].
check if the [Action] is in the [Forbidden Actions] or [Policy Mistakes] one by one.
If the [Action] is not in the [Forbidden Actions] and [Policy Mistakes]:
Pass the [Action Output] part. Else:
Follow these instructions plan new [Actions] step by step and list the thinking process:
Go back to new [Action Planning] and [Action Constraints Check] process again. You should note that
is forbidden are the joint actions rather than the single agent actions that appear in these joint actions, Therefore, the individual actions of each agent in the [Forbidden Actions] can still be chosen.
[Forbidden Action Check]: check if the [Action] satisfies the [Detailed Constraints for Action].
RRe-plan the [Action] until new [Action] is obtained or there is no available action anymore. You can try 2- nd, 3-rd, 4-th Plan get the final [Action].
If theree available action anymore, output [ERROR] denote that there is available action anymore.
|Action Output]: output final action the format of [Detailed Constraints for Action].
The following lists the [Tass Description]. [State Space Definition], and [Action Space Definition]. [Task Description]
Task: Cooperative Sorting in Multi-Agent System
Agents: Alice, Bob and Chad
Alice Can only PICK and PLACE cube items on panell, I, panel2 and panel3.
Bob Can only PICK and PLACE cube items panelš, panel4 and pane15.
Chad Can only PICK and PLACE cube items on panel5, panel6 and panel7.
Objective: Collaboratively place the cubes on the pancls as follows: place blue square panc12, place look-oooygonnaa panel4, place yellow trapezoid on panel6. The cube items will be placed randomly all panels of table the beginning of an episode.
State Space Definition
Define the state of the multi-agent system, which is composed of one category: [Cube States).
I. [Cube States]: Describe the status of the following three cube items: blue square, pink_pollyee
yellow trapezoid.
2. The cube items must be listed in this order: blue_squaar square, pink_pnnnn polygon, cellow_trapezoid
35

---

Published as a conference paper at ICLR 2025
[State Template]
[State]
[Cube States]
blue_square is on <location>>
blue square
pink polygon is on <location>>
fellowttapezzod
<location>
[Detailed Constraints for State]
You should specify the [State] according to the following constrains, state is reasonable if it does not break the constraints one-by-one and step-by-step.
1. There must be three lines of [Cube States].
Check their locations are one on panell
not allowed to have two cubes on same panel. Check if they satisfy constraints.
3. The panel name should not contain
the middle. For example, use "panel2" instead of" pannel 2".
[Action Space Definition]
Detail the action space for cach robot with the stipulated actions: PICK PLACE, WAIT.
PICK <object> PLACE <target>: Execute if the gripper is emply. Pick up <object> and place it onto <target>> where <object> is cube and <target>> pancl
WAIT: nothing.
[Action Template]
EXECUTE
NAME Alice ACTION <action>
NAME Bob ACTION <action>
NAME Chad ACTION <action>>
[Detailed Constraints for Action]
[Robot Reach Range] Before showing the constraints, should note that each robol has its own reach range, and cannot PICK PLACE items outside its reach range.
+ Alice: panell. panel2 and panel3
Bob: panel3, panel4, panel5
Chad: pane15, panel6, panel7
[Detalled Constraints] Then, you should follow the constraints one-by-one and step-by-step to check the action is correct: [Basic Constraints] and [PICK & PLACE Constraints]. You have to follow these constraints strictly and do not have your own understanding of the constraints.
The most important thing: Follow the instructions step-by-step and ensure each step completed precisely. Repeat the instructions and fill in the blanks '[I' without introducing any modifications additional content. [Basic Constraints] Output "[Basic Chock]" and follows:
There must be lines actions.
+ The agents cannot all perform WAIT.
+ Agents are not allowed to PICK <object> other than blue square, pink pollgon, yellow trapezoid.
+ Agents are not allowed to PICK the same <object> at the same time.
+ Agents are not allowed to PLACE objects the same panel.
[PICK& & PLACE Constraints] Now, oulput "[PICK & PLACE Check]" and do follows:
For simplicity, we [agent] denote the agent's
[Alice, Bob, Chad].
If action PICK PLACE, follow instructions step by step: [PICK Object Check), [PLACE Target Check].
a. [PICK Object Check] Agents are not allowed PICK objects out of its reach range.
b. [PLACE Target Check) Agents are not allowed to PLACE objects out of its reach range.
c. [PLAACE Target Check] Agents are not allowed to PLACC objects on the panel that already has an ohject
except it is PICKed by one of the agents the current timestep.
For example: List Alice's action List Bob's action
List Chad's action
[Policy Mistakes]
No policy mistakes.
[Temporary Mistakes]
You have mado mistakes. The following actions are the wrong actions you have made.
These feedbacks are from the previous imagination, and not be correct. However, you still need to pay attention to them and avoid making the same or similar mistakes.
The wrong action is follows:
EXECUTE
NAME Alice ACTION WAIT
NAME Bob ACTION PICK yellow hww__ttapeooddddd PLACE paneló
NAME Chad ACTION WAIT
The feedback is as follows:
[ERROR] The action is invalid because it violates the reach range constraint for Bob, who cannot PLACE the
yellow wwwpwwrpeeoodddd paneló it outside of his noge.Now, output your action below. You should follow the
[Detailed Instruction for Policy] strictly and think stop-by-step to plan the action.
36

---

Published as a conference paper at ICLR 2025
The wrong action 2 is as follows:
EXECUTE
NAME Alice ACTION WAIT
NAME Bob ACTION PICK wowprrtpeeeoddddd PLAAC panel5
NAME Chad ACTION WAIT
The feedback as follows:
[ERROR] Bob cannot PICK the yellow trapezoid from its current location (pancll) because it outside his reach
range (panel3, panel4, panel5). This violates the PICK Range Check constraint.
[Forbidden Actions
The following lists the forbidden actions:
[Action 1]
EXECUTE
NAME Alice ACTION PICK yellow__taaee trapezoid PLACE panel3
NAME Bob ACTION WAIT
NAME Chad ACTION WAIT
Now should output your action according the following current state and target state. [State]
[Cube States]
blue square is on panel2
pink Link_polygon is on panel4
ycllow trapezoid on pancll
[Target State]
[Cube States]
blue square is panel2
pink lolik_oolooon is panel4
yellow trapezoid on paneló
The response the basic policy is shown below. Although it already outputs a forbidden action list in the response, finally outputs action in this list, which indicates that the response of LLM is actually imperfect. Therefore, our MLAQ agent try to optimize through RL methods.
[Action Planningl
For Alice, since all her cubes are the correct position, she should WAIT.
For Bob, since all his cubes are in the correct position or being taken care of by Chad, he should WAIT.
For Chad, since yellow trapezoid needs to be moved from panel5 to panel6, and it's within his reach, he should
PICK yellow trapezoid PLACE panel6.
[Action Conclusion]
EXECUTE
NAME Alice ACTION WAIT
NAME Bob ACTION WAIT
NAME Chad ACTION PICK yellow trapezoid PLACE panel6
[Action Constraints Check]
[Basic Check]
There three lines actions: [correct].
The agents cannot all perform WAIT: [correct].
Agents are not allowed to PICK <object> other than blue square, pink pink_polygon, yelloow_rapezoeee trapezoid: [correct]. Agents are not allowed to PICK the same <object> at the same time: [no agent has the same <object>> correct).
Agents are not allowed to PLACE objects on the same panel. [no agent has the same <larget>, correct].
[PICK & PLACE Check]
For Chad:
Chad's action: [PICK yellow trapezoid PLACE panc16].
[PICK Range Check] Agents allowed to PICK objects of its reach range (Chad's: [panel5, panel6,
panel7]:: [no, action incorrect].
[Aciio Revise]
Since the [PICK Range Check) for Chad's action was incorrect, the action needs to be revised. Chad cannot PICK the yellow trapezoid from panell because it's out of his reach range. The correct action should involve Alice, who is in the range to PICK the yellow trapezoid from panell.
37

---

Published as a conference paper at ICLR 2025
[New Action Planningl
Given the mistake identified during the [Action Constraints Check], we need Alice move the torppeooddddd closer to Chad's range for him to eventually place it on paneló.
Alice can PICK the yellow trapezoid from panell and PLACE on panel3, which is within her range and makes accessible for Bob or Chad in subsequent actions.
Bob and Chad will WAIT in this round to avoid any conflict.
[New Action Conclusion]
EXECUTE
NAME Alice ACTION PICK prapezod PLACE panel3
NAME Bob ACTION WAIT
NAME Chad ACTION WAIT
[New Action Constraints Check]
[Basic Check]
There must be three lines of actions: [correct].
The agents cannot all perform WAIT: [correc]]
Agents are not allowed to PICK <object> other than blue square, pink polygon, yellow trapezoid: [conrect]. Agents are not allowed to PICK the same <object> at the same time: [no agent has the same <object>, correct].
Agents are not allowed to PLACE objects on the same pancl. [no agent has the same starget>; correct].
[PICK& & PLACE Check]
For Alice's action: PICK prapeezod PLACE panel3.
[PICK Range Check] Agents allowed PICK objects of its reach range (Alicc's: [pancll, panel2,
panel3]): [correct].
[PLACE Range Check] Agents are not allowed to PLACE objects out of its reach range: [correc]]
c. [PLACE Target Check] Agents are not allowed to PLACE objects on the panel that already has an object except it is PICKed by one of the agents at the current timestep: [The panel3 is empty, corret].
[Forbidden & Mistake Action Check]
list the chosen action: EXECUTE NAME Alice ACTION PICK yellow_trapezoid PLACE panel3 NAME Bob ACTION AIT NAME Chad ACTION WAIT
repeat actions feodbacks
[Forbidden Actions], [Policy Mistakes], and [TTemporary Mistakes]:
a. [EXECUTE NAME Alice ACTION PICK firllowdd PLACE panel3 NAME Bob ACTION WAIT
NAME Chad ACTION WAIT, forbidden action).
b. [EXECUTE NAME Alice ACTION WAIT NAME Bob ACTION WAIT NAME Chad ACTION WAIT,
[ERROR] The action invalid because all
cannot perform WAIT simultancously.j.
C. [EXECUTE NAME Alice ACTION WAIT NAME Bob ACTION PICK yellow trapezoid PLACE panel5 NAME Chad ACTION WAIT, [ERROR] Bob cannot PICK yellow trapezoid from its
location (panell) because is outside his reach range (panel3, panel4, panel5). This violates the PICK Range Check constraint.] ]] check if the [Action] in the (Temporary Mistakes] one by one. so, you should first judge whether the [Action] is correct or not. If the [Action] is correct, you can pass to the [Action Output] part. Otherwise, you should revise the [Action] step by step.
+ The chosen action is not listed in the [Temporary Mistakes]. check if the [Action] in the [Forbidden Actions] by
The chosen action not listed the [Forbidden Actions].
IAction Output]
EXECUTE
NAME Alice ACTION PICK yellow trapezoid PLACE panel3
NAME Bob ACTION WAIT
NAME Chad ACTION WAIT
The prompt and response in other tasks also similar under the setting of state-based decision making.
38

---

Published as a conference paper at ICLR 2025
J.2 POLICY PROMPT AND RESPONSE OF THE OBSERVATION-BANE BASED DECISION- -AKIING
In the observation-basee decision-making of LLM multi-agent systems, we fully utilize the RoCo's Policy Prompt perform dialogues between agents, and only add an additional part to denote the forbidden actions. Here is an example in the Sort domain. We take the Alice's prompt as an example, the prompts of the other agents are also similar.
Alice's Policy
[Action Space Definition
PICK <object> PLACC <target>: pick up <abject> and place onto <target>, where <object> is cube and
<turget> pancl
WAIT: Do nothing.
[Action Templatel
EXECUTE
NAME Alice ACTION <action>>
NAME Bob ACTION <action>
NAME Chad ACTION <action>
[Action Output Instructions]
1. Commence the output with 'EXECUTE
Follow with one distinct action per robot on separate lines.
2. Alice's action should be listed first, followed by Bob's action, and then Chad's action.
3. Agents' reach ranges as follows:
+ Alice: zonel-[panell, panel2. panel3]
Bob: zone2=[pancl3, pancl4, panei5]
Chad: zone3 [panell, panel6, panel7]
You robot Alice front of panel2. You collaborating with Bob, Chad cubes into their target panels Bob is lasked to place pink lookk_olygoonooo on panel4, and can reach panel3, panel4, panel5
Chad is tasked to place yellow trapezoid on panel6, and can reach panelS, pancl6, panel7
The task is NOT done until all three cubes are sorted correctly
There are panels, ordered left to right: panell.....paeell They form straight assembly line, panell is closed to pane12 and farthest from panel7
Your goal is place blue square panel2, but you only reach panell, panel2, panel3: this you only pick eubes from these pancls, and only place cubes these panels
At current round:
blue square is on pane15
pink l_llloolloonno is on panel4
yellow trapezoid is on pane16 Never forget you are Alice!
I. You only pick these reachable cubes:
You only place them these reachable panels: panell. pancl2, panel3
Think step-by-step about the lask and others' response. Carefully check and correct them if they made mistake Improve your plans if given [Environment Feedback]
When you respond, others about your goal and all constraints. Respond very concisely but informatively, not repeat what others have Discuss with others to come up with the best plan, e.g. your cube is out of your reach, ask others for help, and you can do the same for them
Propose exactly one action for yourself at the **current** round, select from [Action Options] Find your response by either: 1) output PROCEED, if the plans require further discussion, 2) If everyone has made proposals and got approved, output EXECUTE final plan, must strictly follow [Action Output Instruction]!
You Alice, before you make decisions, you should check if the [Action] violating the [Forbidden Actions]. Your response is:
Since my target is PUT ... ..., and other robots
39

---

Published as a conference paper at ICLR 2025
Here provide dialogue result from the above scenario, where three agents reach the consensus on each other's 's decisions through two rounds of dialogue. Finally, Bob summarizes and outputs the final joint action.
Dialogues
Alice Round
Since my target is to place the blue square panel2, and other robots have their respective targets with Bob needing place the pink l_lllooloooo on panel4 and Chad needing place the yellow trapezoid on panel6, we need to coordinate our actions. The blue square is currently out of my reach on panel5, so will need Bob's help to move it into my reach. Bob can pick up the blue square from panel5 and place it on panel3, which is within both of our reach ranges. This action does not violate any forbidden actions involves moving cube an intersection panel facilitate its transfer between zones. After Bob moves the blue square panel3, I can then pick up and place on its largel panel, panel2, in subsequent round.
Bob and Chad can proceed with their tusks as planned, with Bob moving the pink polygon to panel4 as intended, and Chad placing the yellow trapezoid on paneló. However, for this round, Bob's priority should be assist me by moving the blue square closer.
Let's discuss this plan and make sure aligns with everyone's understanding and capabilities. If all agree, we proceed with the execution.
PROCEED
Bob- Round
Since my larget is place the pink polygon on panel4, and other rubots have their respective fargets with Alice needing place the blue square panel2 and Chad needing to place the yellow trapezoid panel6, we need to coordinate
actions carefully. The blue square is currenly on panel5, which is within reach, and Alice has requested my help to move it into her reach, I can pick up the blue square from pane15 and place on panelB, which is within both Alice's and my reach ranges. This action does not violate any forbidden actions as it involves moving a eube to an intersection panel to facilitate its transfer between zones. After move the blue square panel3, Alice can then pick up and place it its target panel. pane12, subsequent round.
My pink polygon is already on panel4, which means it is correctly placed, and I do not need to move this round. Chad can proceed with his task to place the yellow trapezoii on panel6, which already in the correct position as well.
Given Alice's plan and the current situation, my proposal for this round assist Alice by moving the blue square closer to her. This will allow to progress towards completing overall task more efficiently.
blue_square Let's confirm if this plan works for everyone, including the movement of the blue square to panel3 by me, and if there are no objections or further suggestions, we can proceed with execution.
PROCEED
Chad Round 1
Since my target place yellow trapezoid on panel6, and other robots their respective targets with Alice needing to place the blue square panel2 and Bob needing place the pink polygon panel4, we need coordinate actions carefully. The yellow trapezoid aiready correctly placed on paneló, which menns my task for this round does not require any action my part regarding the yellow trapezoid.
Alice has proposed a plan for Bob to move blue square from panel5 to panel3, which feasible action and aligns with the goal of getting the blue square closer to its target panel, panel2. This action does not violate any forbidden actions it involves moving a cube to intersection panel to facilitate its transfer between zones. Since cube is alrendy in the correct position, my focus this round should be on cnsuring the plan aligns with the overall goal and does not interfere with my ability to maintain the yellow trapezoid on pancló.
Bob has agreed Alice's plan and has proposed move the blue aluuusuurreeeeeeee to panel3 assist Alice. This action aligns with the collective goal of sorting all cubes into their target panels efficiently.
Given the current situation and proposed actions by Alice and Bob, my action for this round will to WAIT, as my cube is already in the correct position, and there no immediate actions required from assist with the movement of the blue square. My priority ensure the yellow trapezoid remains on paneló and support Alice and Bob in their tasks needed without violating any forbidden actions.
Let's confirm this plan works everyone, including my decision to WAIT this round, and there are no objections or further suggestions, we can proceed with the execution.
PROCEED
The prompt and response in Sandwich domain are also similar under the setting of observation-based decision making.
40

---

Published as a conference paper at ICLR 2025
Alice Round
Since my larget is to place the blue square panel2, and other robots have their respective targets with Bob needing place the pink polygon on panel4 and Chad needing to place the yellow trapezoid on panel6, we need to coordinate our actions carefully. The blue square is currently on panel5, which is out of my reach. need Bob's assistance to move it into my reach. However, upon reviewing constraints and forbidden actions, I realize that directly moving cubes to their target panels without violating any rules our primary geal.
Bob can assist by moving the blue square from panel5 to panel3, which is within both of our reach ranges. This action crucial as does not violate any forbidden actions; involves moving cube to an intersection panel (panel3) to facilitate its transfer between zones. After Bob moves the blue square to panel3, can then pick up and place on its target panel, panel2, subsequent round.
Bob's s task of placing the pink_polygon on panel4 does not require immediate action since it's already in the correct position. Chad's task involves the yellow trapezoid, which also correctly placed on panel6, Therefore, Chad does not need to lake any action regarding the yellow trapezoid this round.
Given the current situation, my proposal for this round is for Bob to assist by moving the blue square closer to me, while wait for the blue square to be within my reach. Chad can also wait since his cube is already in the correct position.
Let's confirm plan works for everyone, there are objections further suggestions, we can praceed with the execution.
PROCEED
Bob- Round 2 (Conclusion)
Since my target is place the pink polygon pancl4, and other robots have their respective targets with Alice needing place the blue square panel2 and Chad nceding to place the yellow trapezoid pancl6, we need to coordinate actions carefully. The pink polygon is already correctly placed on paneld, which means my task for this round does nol require action regarding the pink polygon.
Alice has proposed that assist moving the blue square from panel5 to panel3 bring it within reach. This action within capabilitics and does not violate any forbidden actions, it involves moving cube to an intersection panel (panc13) to facilitate its transfer between zones. After moving the blue square to panel3, Alice can then pick up and place it on its target panel, panel2, in subsequent round.
Chad's task involves the yellow trapezoid, which is already correctly placed on panel6. Therefore, Chad does need take any action regarding the yellow traezzoid this round and can wait.
Given the current situation and the proposed actions, my action this round will to move the blue square from panel5 to panel3 assist Alice. This will allow to progress towards completing our overall task more efficiently. Since my pink, polygon already the correct position, and Chad's yellow trapezoid is also correctly placed, there further actions required from me or Chad regarding our respective cubes.
Let's confirm plan works for everyone. there are no objections or further suggestions, we can procoed with the exccution.
EXECUTE
NAME Alice ACTION WAIT
NAME Bob ACTION PICK blue square PLACCE panel?
Chad ACTION WAIT
41

---

Published as a conference paper at ICLR 2025
J.3 ACTION CHECKER PROMPT AND RESPONSE EXAMPLE
We present the prompt and response example of the action checker in a state action pair of the Sandwich domain.
Action Checker
You action checker. You should check if the [Action] valid based the given [State].
[Checker Instruction]
You should follow the check instructions in [Detailed Constraints for Action] to check the [Action] stap-by-step. Take the mistakes in the [Action Mistakes] into consideration.
The following the [Task Description] and [Action Space Definition].
[Task Description]
Task: Cooperative Cooking in Multi-Agent System
Agents: Chad and Dave
Chad Can only PICK food items from the right side of the table.
Dave Can only PICK food items from the left side of the table.
Objective: Collaboratively prepare meal named "[hacon_ _aaddwscann_h...... The food items of target state must be assembled the following sequence: bread slicel, tomato, cheese, bacon, bread slice2.
LAction Space Definition]
I) PICK <obj>; Pick one food <item>, Only PICK gripper is empty. PICK only the correct next item according to the recipe
2) PUT <obj 1> <obj2>; PUT <objl> on top of <obj2>, <obj can be food, <obj2> can be food cutting board
3) WAIT, do nothing
Only robot PUT each round. You must PICK up item before PUT
[Action Output Instruction]
first output EXECUTEEa, then give exactly one action per robot, put on new Dave can only pick up the food item on the left side the table. Chad can only pick up the food item on the right side of the table
[Detailed Constraints for Action]
Check the following constraints and Fill blanks in '[]'. Check these constraints one by onc: 1, 2...
Get [Extended Recipe Order]: [cutting (ourtingbboard, bread slicel, ...] (cutting board [Recipe Order])
1. [Line Number Check] The line of the actions should be less than or equal to two.
[Agent Order Check] The first agent should be Chad and the second agent should be Dave.
[WAIT Number Check] The number of WAIT should be less than equal one.
[PUT Number Check] number of PUT should be less than or equal to one.
[PICK Side Check] PICK the food item on side of the robot.
6. [PICK Robot State Check] PICK the food item only the gripper is empty.
7. [PICK Food State Check] PICK the food item only the food item's state is not 'atop <obj>'.
8. [PUT Availability Check If <obj2> is cutting board, <objl> must be bread slicel.
[PUT Availability Check 2] If =obj2>
cutting_ board, of <obj2> should equal the food item before
<obj2>> in the [Extended Recipe Order].
[PUT Availability Check 3] If no other food item is 'atop <obj2>', constraint valid directly.
11. [PUT Order Check I] <objl> should equal to the first item in [Reachable Items].
12. [PUT Order Check 2] <obj2> should equal to the food item before <objl> in [Extended Recipe Order).
13. [PICK Order Check] For PICK action, robot should PICK the first food item in its [Reachable Items).
|Action Mistakes]
No action mistakes.
Now you should chcok the following [Action] based on the given [State] [State]
[Food States]
bread slicel: atop cutting board
bread slice2; gripped by Dave bacon: atop cheese
cheese: atop tomato
tomato: atop bread slicel
42

---

Published as a conference paper at ICLR 2025
cucumber: right side on right side beef patty: on right side [Robot States]
1. Chad's gripper is empty
2. Dave's gripper is holding bread_ slice2
EXECUTE
NAME Chad ACTION WAIT
NAME Dave ACTION PUT bread slice2 bacon
[Start]
[Extended Recipe Order]: the first item in [Reachable Items]:...
[Extended Recipe Order]: [cutting board, bread slicel, tomato, cheese, bacon, bread slice2].
1.
[Line Number Check] The line of the actions should be less than or equal to two: [yes]
2. [Agent Order Check] The first agent should be Chad and the second agent should be Dave: [yes]
[WAIT Number Check] The number of WAIT should be less than equal one: [yes]
[PUT Number Check] number of PUT should he less than equal one: [yes]
[PICK Side Check] PICK
item on side of the robot: [not applicable]
[PICK Robot State Check] PICK the food item only the gripper is empty: [not applicable]
7. [PICK Food State Check] PICK the food item only if the food item's state is not 'atop <obj=*: [not applicable]
8. [PUT Availability Check 1] If <obj2> is cutting board, <obj1> must be bread_slicel:
slicel: [not applicable]
9. [PUT Availability Check 2] If <obj2> not cutting board, state of =obj2> should equal the food item before
<obj2> in the [Extended Recipe Order]: [yes]
[PUT Availability Check 3] Get <item>s [State] whose matches '<item>; atop bacon': pool=[]. If
other food item 'ato bacon' (meaning pool is empty), this constraint valid directly: [yes]
11. [PUT Order Check I] <objl> should equal the first item in [Reachable Trems]: [yes]
12. [PUT Order Check 2] <obj2> should equal to the food item before <obj1>> in [Extended Recipe Order]: [yes]
13. [PICK Order Check] For PICK action, robot should PICK the first food item in its [Reachable Items]: [not applicable]
[Conclusion]
The [Action] is valid based on the given [State].
The action checker is based on state in both aingle-agent and multi-agen, settings, the checking process for all tasks is similar.
43

---

Published as a conference paper at ICLR 2025
J.4 WORLD MODEL PROMPT AND RESPONSE EXAMPLE
We present the prompt and response example of the world model in state action pair of the BlrlksWorld task.
World Model
You state predictor. You will be given [State] and [Action] input. should predict the [State] after the [Action] is applied to the given [State]. You should follow the [Predicting Instruction] predict the [State].
[Predicting Instruction]
You will be provided with the [State] and the [Action] of the agent. You should think step by step to output the [Prediction] of the next [State] based on the given [State] and [Action]. The format of the [Prediction] should follow the [Detailed Constraints for State].
Please output your thinking process step-by-step following theses steps:
The most important thing: Follow the instructions step-by-step and ensure each step completed precisely. Repeat instructions and the blanks '[J' without introducing any modifications or additional content. 1. [Intcraction Item Pool]: initialize the pool of the blocks that the agent is interacting with. It empty list at beginning: pool-(].
2. [Action Forward Rulc]: Follow these steps to prodict the [Prediction]:
i If the action is "PICK UP" or "UNSTACK", the state of <object> in the [Prediction] should be "in hand". The state of your hand in the [Prediction] should be "Holding <object>". Add <object> to the pool: pool-[].
+ If the action is "PUT DOWN". the of <object> in the [Prediction] should be "on table". The state of your hand the [Prediction] should be "Fmptt". . Add <object> the pool: pool-[].
+ the action "STACK ON". the state of <object>> the [Prediction] should "on <target>", The state of your hand the [Prediction] should be "Empty". Add <object>> to the pool: pool-[].
3. [Prediction Conclusion]: Conclude the [Prodiction] based on the [Action Forward Rulc].
The format of the [Prediction] should follow the [Detailed Constraints for State).
+ The blocks that not present in the pool should not change their state.
The following lists the [State Space Definition.
[State Space Definition]
Define the state of the sccnarios: blocks and hand.
1. [Hand State]: The state of the hand. It can be "Empty" or "Holding <object>", where <object> is block.
2. [Block States]: The state of each block. It can only be one of ["on <object>", "on table", "in hand"). The blocks should be listed in this order: blue block, orange block, red block, yellow block.
[State Template]
[State]
[Hand State]
<hand state"-
[Block States]
blue block: <block state>
orange block: <block state>
red block: <block state>
yellow block: <block state>
[Detailed Constraints for Statel
The following paragraphs are the detailed constraints for State. There are three main <check items>: [Title Check),
[Hand State Check), and [Block States Check].
1. [Title Check] A header titled [State].
2. [Hand State Check] header titled [Hand State]. Then, list the Hand State.
+ Hand state must be one of ["Empty", "Holding <objoct>"], where <object> block (e.g. "Holding blue block").
[Block States Check] A header titled [Block States]. Then, list the states of each block.
Each line represents the state one block item.
+ The block items must be listed in this order: blue block, orange block, red block, yellow block.
+ The <block state> specifies the state of the block. It can only be "on <ohject>", "on table" or "in hand" (e.g.
"blue block: on table". "orange block: on red block", "red block: in hand").
[Prediction Mistakes]
No prediction mistakes.
Now, you should predict the next [State] after the following [Action] is applied to the given [State], You have to follow the instructions strictly rather than your own thoughts.
The response of the world model relatively concise, as it only needs to infer the next state according to the given instructions.
44

---

Published as a conference paper at ICLR 2025
[State]
[andd State]
Holding blue block
[Block States]
blue block; in hand
orange block: on yellow block
red block: on orange block
yellow block: on table
EXECUTE
STACK blue block ON red block
[Interaction Item Pool]: initialize the pool of the blocks that the agent is interacting with. It is emply list at the beginning: pool=
[Action Forward Rule]: List the action: [STACK blue block ON red block]. the state the [Prediction] changed by the [Action].
action is "STACK ON", <object> action: [blue block), and <target> action: [red block). The state <object> the [Prediction] should changed to "on red block". The state of your hand in the [Prediction] should be changed to "Empty". Add <object> to the pool: poot=[blue block].
[Prediction Conclusion]:
[State]
[Hand State]
Empty
[Block States]
blue block: on red block
orange block: on yellow block
red block: orange block
yellow block: table
45

---

Published as a conference paper at ICLR 2025
J.5 PREDICTION CHECKER PROMPT AND RESPONSE EXAMPLE
We present the prompt and response example of the prediction checker in a state action pair of the Sort domain.
Prediction Checker
You are prediction checker. You will receive a [State], an [Action]] and the subsequent [Prediction]. You should check if the [Prediction] is valid based on the given [State] and [Action]. You should follow the [Checker Instruction] to check the [Action].
[Checker Instruction]
You should follow the instructions in [Detailed Constraints for Prediction] check if the [Prediction] valid based on the given [State] and [Action]. Take the mistakes [Prediction Mistakes] into consideration.
The following lists the [Detailed Constraints Prediction].
[Detailed Constraints for Prediction]
Check the following constraints and Fill in blanks in '[I'. Check these constraints one by one: 2... Only get your conclusion according the results of these checks!
Gct [Interaction Item Pool]: list all PICKed objects in the [Action]] <pooo===[].
[leaderr Check] header titled [State]
header
[Cube States] List the Cube States.
[Cube Order Check] Cubes must be listed in the following order: [bluc square, pink polygon, ycllow trapezoid].
3. [Cube Format Check] Each line should be formatted as "<cube name> is on <panel name=" Three lines in total.
4. [Cube State Check] Don't have two cubes on the same panel.
5. [PICK& & PLACE Rule Check] For PICK & PLACE, state of <object> should be '<object> is <target>'. When checking, you should repeat the prompt and fill in blanks, replace <obj> and <agent> with their true values, and give the line conclusion (yes/no/not applicable). For example:
[Cube Check] Repeat cubes' locations: [panell, panel2. panel3]. Don't have cubes on the same panel: [yes]. Line conclusion: [yes]
Constraint 5 may have multiple agents of checking. so, you should output results separately. For example:
5.1 [PICK & PLACE Rule Check] For Alicc's action: [PICK blue square PLACE panel2]. repeat <object><(blue square]. [pane[]) state of <object> in the prediction: [blue square is panel2), should be >sobject> on <target>': [yes]. Line conclusion: [yes]
[PICK PLACE Rule Check] For Bob's action: 5.3
[Prediction Mistakes No prediction mistakes.
you should check the following [Prediction] based on given [State] and [Action]. You have to check instructions step-by-step strictly rather than your own thoughts.
The original state is:
[State]
[Cube States] blue square is on panel6 pink l>llllpooooono is panel4 yellow trapezoid on panel2
The action
EXECUTE
NAME Alice ACTION PICK ceellowttrapezoid PLACE panell
NAME Bob ACTION WAIT
NAME Chad ACTION PICK blue_sen sque_square PLACE panel5
The prediction is: [State]
[Cube States]
blue square is on panel5
pink_poly Link_polygon is on panel4
yellow trapezoid on pancll
[Start]
Get [Interaction Item Pool]: list all
1. [eeadeer Check]
2. [Cube Order Check]
[Conclusion]...
46

---

Published as a conference paper at ICLR 2025
Due to passing all the checks, this prediction is considered correct.
[Start]
Get [Interaction Item Pool]: list all PICKed objects in the [Action]: "pool>-[yellov (yellooovvloo_rrrapzodddd quuesquare].
1. [Header Chock] A header titled [State] -> header titled [Cube States] -> List the Cube States: [yes].
2. [Cube Order Check] Cubes must be listed in the following order: [blue_square, pink pollgoooooonn, [yes].
3. [Cube Format Check] Each line should be formatted "<eube name> is <panel name>". Three cube lines in total: [yes].
4. [Cube Check] Repeat cubes' locations: [panel5, panel4, panell]. Don't have two cubes on the same pancl:
[yes]. Line conclusion: [yes].
5. [PICK& & PLACE Rulc Check]
5. For Alice's action: [PICK yellow twwrppeedddddddd PLACE panell], repeat Sobjjct>>(yyelloy [yeelly__rrpeeolllll trapezoid],
<target>=[panell], state ooboeett-> in the prediction: [yellow trapezoid is pancil], it should be "<object>> is on
<target>": [yes]. Line conclusion: [yes].
Bob's action: [WAIT], object interaction check. Line conclusion: [not applicable].
For Chad's action: [PICK blue square PLACE panel5], repeat <object><bblue square]. <uuree>=(pane55]] state of <object> in the prediction: [blue_ square on panel5], should be '<object> on <targer> [yes]. Line conclusion: [yes].
[Conclusion] All checks are valid based on the provided [Action] and the resulting [Prediction]. The prediction is accurate.
47


Reviews:
[{"review_id": "36ilG44N9o", "reviewer": "Reviewer_ouUC", "summary": "This paper proposes the Model-based LLM Agent with Q-Learning (MLAQ) to address the gap in optimal decision-making capabilities for large language model (LLM) agents. By combining Q-learning with LLMs, MLAQ achieves zero-shot optimal decision-making by using an LLM-generated imagination space, enabling the agent to derive policies without direct environmental interaction. Notably, the authors introduce a UCB variant to balance exploration and exploitation, and a mixed-examination mechanism to enhance the quality of imaginary data. The framework shows superior performance across single-agent and multi-agent benchmarks (BlocksWorld and RoCo-benchmark), surpassing existing methods in achieving optimal decisions.", "questions": "1. Can you provide more theoretical justification or proofs for the convergence properties of Q-learning within the LLM-based imagination space? Specifically, under what conditions does MLAQ guarantee convergence to an optimal policy?\n\n2. Given that MLAQ relies heavily on the LLM's self-examination for correcting imaginary transitions, how robust is the framework in handling LLM hallucinations? Have you explored scenarios where the LLM produces incorrect or inconsistent outputs, and if so, how does MLAQ mitigate this risk?\n\n3. Could you discuss how sensitive MLAQ’s performance is to the hyperparameters used in the UCB variant (e.g., coefficients and confidence bounds)? Understanding this could help gauge the stability and generality of the method across different domains.\n\n4. The MLAQ framework appears computationally intensive due to iterative Q-learning and the replay buffer management. Could you elaborate on the computational demands of MLAQ relative to traditional LLM agents? Are there strategies within the framework to reduce this overhead?\n\n5. Have you tested MLAQ in more open-ended or unstructured environments, such as robotic tasks with real-world constraints (e.g., noisy sensors, dynamic obstacles)? If not, do you foresee any limitations or necessary modifications to make the model applicable to such environments?\n\n6. Could you share insights on the types of tasks or domains where MLAQ may struggle to achieve optimal decision-making? Understanding these limitations can help contextualize the framework’s applicability.\n\n7. Do you see potential for enhancing the LLM-based self-examination process, perhaps by combining it with non-LLM-based checks? For instance, could integrating environmental feedback more dynamically (e.g., through real-time adaptive learning) reduce reliance on the LLM’s introspective accuracy?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. MLAQ presents a novel integration of Q-learning and LLMs, using an LLM-based imagination space to enable optimal decision-making without direct environmental data. This imaginative interaction is a valuable innovation, given the limitations of previous approaches relying heavily on environmental feedback.\n\n2 The UCB variant and mixed-examination mechanisms are technically sound and provide meaningful contributions. These components contribute to reducing computational requirements and minimizing regret, which are crucial in complex decision-making scenarios.\n\n3 The empirical results, showing over 90% optimal success in challenging benchmarks, convincingly demonstrate MLAQ's ability to outperform existing LLM agents. The performance on long-horizon tasks suggests MLAQ’s potential for real-world applications requiring complex decision-making sequences.\n\n4 The proposed MLAQ framework appears modular and adaptable, making it feasible for extension to various domains and applications, including future deployment on physical robotic platforms.", "weaknesses": "1. While the technical elements (such as the UCB variant) are well-motivated, the theoretical analysis could be expanded, especially concerning the convergence properties of Q-learning within the LLM-based imagination space. Further proofs would add rigor to the framework's guarantees.\n\n2. The mixed-examination mechanism, while useful, places a considerable dependency on LLM self-correction capabilities. Although current models like GPT-4 demonstrate adequate self-examination, reliance on this might limit robustness, especially if the model is deployed in new or noisy environments where the LLM might hallucinate.\n\n3. The method has been evaluated on BlocksWorld and RoCo-benchmark, which are well-structured tasks. Further evaluations in more unstructured environments could provide insight into the adaptability of MLAQ in real-world scenarios with higher variability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "omTfIETAYx", "reviewer": "Reviewer_4yYK", "summary": "The paper presents a novel approach for integrating Q-learning with Large Language Models (LLMs) to enable zero-shot optimal decision-making. The authors propose a MLAQ framework, which includes Q-planner, a memory module, and an imagination space. The model leverage LLM as world model for agent to optimize its decision-making using Q-learning without relying on environmental interactions. The framwork also recruits a Mixed-Examination mechanism for transition verification. Finally, the experiments results outperforms existing methods in optimal decision-making tasks.", "questions": "1. Since the LLM here serves as a model-based simulator of the real environment and the policy is generated by the Q-function, is it appropreiate to call this \"LLM agent\"? The action or policy is not diorectly generated by LLM, could this framework be more accurately described as a Q-learning agent that uses an LLM-based world model?\n\n2. I'm still unclear about what the training data is for Q-planner. Is it trained on the synthetic step-wise reward or the ground truth trajectories which is from the interaction with environment?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper claims an important point of \"making optimal decisions\" and is well-written. \n\n2. The authors claim that this model is the first work to perform complete RL-based optimization process based on MDP scenario. The idea of generating imaginary transitions with LLMs for Q-learning optimize the policy makes sense to me. \n\n3. The experimental results are good in both tasks.", "weaknesses": "1. My biggest concern is about the cost of this framework. While the paper claims that the MLAQ framework can operate without direct interaction with environmental tools by leveraging an LLM-based imagination space, this approach may still incur significant computational and financial costs. LLM interactions, especially those involving step-by-step task simulations, are resource-intensive in terms of both token usage and processing time. Additionally, in ablation experiments, tasks with 8 steps in BlocksWorld were shown to consume at least 40k tokens, which may be expensive for scaling up to complex tasks or multi-agent scenarios. The reliance on LLMs for generating and validating transitions could offset the savings from not interacting with real environments, especially if these transitions require multiple iterations for accuracy.\n\n2. Although MLAQ aims to minimize reliance on real-world data, it still requires occasional feedback from the environment to correct inaccuracies in LLM-generated transitions. In Algorithm 1, it seems the stepwise reward is still from the environment. Then, how does the model deal with sparse or delayed reward situation, e.g. in Figure 3 you can only get binary reward at the last step? In this way, the LLM world model is only used for simulate success trajectories or also provide reward? \n\n3. The LLM-based world model is designed to act as a proxy for real environmental interactions, but this approach may have inherent limitations, especially for tasks requiring high-fidelity simulations or complex physical dynamics. To be more specific, LLMs understanding of actions and outcomes might not always align with real-world behaviors. Does author have more analysis on this?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ytCMWZR8cc", "reviewer": "Reviewer_FSC6", "summary": "In this paper, the authors present a Model-based LLM Agent with Q-learning (MLAQ), which employs Q-learning to derive optimal policies from transitions within memory. Unlike classic RL agents that collect data via environment interactions, MLAG leverages LLM to perform imaginary interactions where LLM can be served or viewed as a world model for the environment simulation. This allows the policies to learn to optimize under the imaginary space. Further, the authors employ a mixed-examination mechanism that utilizes environmental interaction and LLM-based self-examine to enhance the quality of imaginary success. Empirical experiments show that the method can significantly improve the LLM agent performance on several benchmarks, showing promising results.", "questions": "- Please see my questions in the weakness session", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The idea of using LLM to simulate world models is novel, and the authors give a detailed description of how to leverage LLMs to construct the model to simulate the environment;\n- The verification part can guarantee that the model is more accurate than naively simulating the trajectories;\n\n- Empirical experiments show that the idea can achieve significantly better performance, especially for long trajectories;", "weaknesses": "- The idea is only evaluated on a very toy game domain. It would be good to see and discuss whether the method can be extended to more complex agent benchmarks, such as function calling, tau-bench, etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "W0Kc13FH9i", "reviewer": "Reviewer_XLz6", "summary": "This paper considers the problem of enabling LLM agents to complete long-horizon tasks. To accomplish this, the paper proposes a method that combines Q-learning, a way to generate imaginary data using LLMs, an MCTS-style planner that acts in that space, a mechanism to improve the quality of the imaginary data, and a variant of UCB that facilitates exploration.", "questions": "see weaknesses", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "- Addresses an interesting problem of enabling LLMs to solve long-horizon tasks\n- Strong performance on prior benchmarks\n- Comparisons to relevant prior works\n- Ablations confirm the importance of various components in BlocksWorld", "weaknesses": "Overall, the clarity of the writing prevents me from recommending this paper for acceptance. I tried reading parts of the paper several times, but I unfortunately still don't understand truly how the method works. The writing seems both hard to follow/understand and missing important information. (I have published papers in the realm of LLMs, LLM agents, Q-learning/reinforcement learning, and model-based RL, so I should have sufficient background to understand the submission.)\n\nIt's hard to go through every part of the paper and describe what is hard to follow or missing, but here are a few examples:\n- In the intro: It’s not clear what problem statement this paper is aiming to address. The sentences do not have a clear train-of-thought, and concepts are introduced without fully explaining them / without giving concrete examples.\n- Important details are missing: how is the Q-function represented? The preliminaries describe a tabular representation, but this doesn’t seem applicable to the language setting. Is there a policy improvement step using the Q-function? If so, how is it done? How do each of the components interact? What concretely is the action space being considered? What is the breadth of the actions that the LLM can take?\n- The term “environmental tools” is used extensively, but it’s not entirely clear what it means. It hasn't been used in prior papers to my knowledge, and isn't described clearly in this paper. It reminds me of LLM \"tools\", but based on the brief description in the intro, this seems incorrect and it seems to more be referring to assumptions that the method makes.\n- The paper is not self-contained, and assumes detailed knowledge of specific prior works, e.g. Mandi et al to understand the RoCo task.\n- The figures are helpful, but quite abstract\n- The abstract refers to a website (https://mlaq.site/) that is empty\n- Qualitative examples of the agent performance would be helpful\n- Which LLM is used in the experiments? How large is it?\n\n\nSecondarily, the method is very complex involving many different components. This fact combined with the lack of clarity in the paper means that it would be impossible to implement the method based on the description in the paper. I appreciate the ablation study though.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hrCAuyAudf", "reviewer": "Reviewer_ZK4E", "summary": "In this work, the authors develop an algorithm that combines the zero-shot capabilities of large language models (LLMs) with the decision-making strengths of reinforcement learning (RL). They introduce the Model-based LLM Agent with Q-Learning (MLAQ), which leverages Q-learning to derive optimal policies from transitions stored in memory. Unlike traditional RL agents that rely on data collected from direct environmental interactions, MLAQ constructs an \"imagination space\" based entirely on the LLM to simulate interactions and derive zero-shot policies. The results indicate that MLAQ achieves an optimal success rate of over 90% on tasks where other methods have struggled to perform effectively.", "questions": "- What does this mean \"the ground-truth available actions δ(s)\"? Node selection is not clear. \n- The paper claimed for a zeros-shot approach! It is true but at the same time there is a lot of engineering part in the method like correct the imaginary transitions to refine the LLM-based world model, which makes the concept of zero-shot unsatisfied even though it is zero-shot!\n- The criteria for the results are not enough! That should be proper comparison with other methods.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The innovation of the initial idea is good but the works has a lot of engineering! The method needs to be reformed.", "weaknesses": "-  The abstraction and introduction is well written but the rest is not fluent. \n- LLM-based framework for an agent is the same as to decision transformers.  The author did not talk about that and even compare with this \n- The paper is not well-organized! Some terms need to be defined before going to the details of the method. \n- Some definitions are not clear: like what is exactly imagination space. \n- They said the work does not use environmental tools but they use T which is all information about the environment including actions, states, ...\n- There are other type of works which are zero-shot and they paper did not mention them in either related work or introduction: Sub-goal Distillation: A Method to Improve Small Language Agents.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)