Empowering Large Language Model Agents through Action Learning
Authors: Haiteng Zhao
Chang Ma
Guoyin Wang
Jing Su
Lingpeng Kong
Jingjing Xu
Zhi-Hong Deng
Hongxia Yang
Venue: COLM (2024)
Keywords: Large Language Model Agent, Agent, Large Language Model
TL;DR: We introduce a framework LearnAct with an iterative learning strategy to improve learnable action space.

Abstract: Large Language Model (LLM) Agents have recently garnered increasing interest yet they are limited in their ability to learn from trial and error, a key element of intelligent behavior. In this work, we argue that the capacity to learn new actions from experience is fundamental to the advancement of learning in LLM agents. While humans naturally expand their action spaces and develop skills through experiential learning, LLM agents typically operate within fixed action spaces, limiting their potential for growth. To address these challenges, our study explores open-action learning for language agents. We introduce a framework LearnAct with an iterative learning strategy to create and improve actions in the form of Python functions. In each iteration, LLM revises and updates the currently available actions based on the errors identified in unsuccessful training tasks, thereby enhancing action effectiveness. Our experimental evaluations across Robotic Planning and Alfworld environments reveal that after learning on a few training task instances, our approach to open-action learning markedly improves agent performance for the type of task (by 32 percent in AlfWorld compared to ReAct+Reflexion, for instance) highlighting the importance of experiential action learning in the development of more intelligent LLM agents.

Published as a conference paper at COLM 2024
Empowering Large Language Model Agents through Action Learning
Haiteng Zhao!
Jing Su? Zhi-Hong Dengt Hongxia Yang²
Chang Ma² Guoyin Wang²
Lingpeng Kong²
Jingjing Xu²
Peking University ByteDance 3 The University of Hong Kong
{zhaohai teng, zhdeng/@pku looiku.edd.com edu.cn cn {cma, 1pk) hku. hk
guoyinwang. duke@gmail. com {sujing. 29, xujingjing, h.. waa.yann@yytddanne..oommm
Abstract
Large Language Model (LLM) Agents have recently garnered increasing interest yet they are limited in their ability to learn from trial and error, a key element of intelligent behavior. In this work, we argue that the capacity learn new actions from experience fundamental to the ad- vancement of learning in LLM agents. While humans naturally expand their action spaces and develop skills through experiential learning, LLM agents typically operate within fixed action spaces, limiting their potential for growth. To address these challenges, our study explores open-action learning for language agents. We introduce a framework LearnAct with an iterative learning strategy create and improve actions in the form of Python functions. In each iteration, LLM revises and updates the currently available actions based on the errors identified in unsuccessful training tasks, thereby enhancing action effectiveness. Our experimental evalua- tions across Robotic Planning and Alfworld environments reveal that after learning on few training task instances, our approach to open-action learning markedly improves agent performance for the type of task task-by 32% in AlfWorld compared to ReAct+Reflexion, for instance- highlighting the importance of experiential action learning in the development of more intelligent LLM agents.
1 Introduction
Language agents, which employ the large language model (LLM) as the policy model to control agents to iteratively take actions and interact with environments, have recently garnered increasing interest (Yao et al., 2023a; Brohan et al., 2023; Wang et al., 2023a; Xie et al., 2023; Song et al., 2023; Xu et al., 2023). The underlying reason is that LLMs offer fresh angle to address the commonsenuse issue that is difficult to tackle in the reinforcement learning paradigm which learns the agent policy solely from trial and error. Reasoning and planning of agents are often derived from prior knowledge in the particular environment, precisely where LLMs excel.
Researchers increasingly recognize that, while leveraging LLMs for agent control shows promise, it is far from perfect due to their limited ability to learn from experience (Yao et al., 2023b; Shinn et al., 2023; Huang et al., 2023). The substantial scale of LLMs makes direct policy model finetuning impractical. Instead, LLMs depend on incorporating historical interactions into prompts to leverage past experiences for future action planning (Yao al., 2023a; Shinn et al., 2023; Sun et al., 2023; Madaan et al., 2023), However, these approaches are often constrained their capacity to learn from long-term experiences and typically draw experience single-instance (Wang et al., 2023c; Ma et al., 2024; Wang et al., 2023a).
While humans naturally expand their action spaces and develop skills through experiential learning, LLM agents typically operate within predetermined action spaces (Qin et al., 2023; Schick et al., 2023), limiting their potential for growth. In this work, we propose novel learning paradigm for LLM agents that focuses on learning to expand and iteratively refine

---

Published as a conference paper at COLM 2024
Train Stage: Create/Revise Learnable Actions
Test Stage: Using Learned Actions Act Prompting
with
Corry. corry corry_tel E2 enrry. Mnfryl ane greers
Aa
Pickup: pickl
[Instruction]: You can use the following action: pick chop, carry, Io, /wwo to.
Move: movel Sart Fiall
escept:
past grigoed)
User
(Goal: You should perform actions accomplish the goak: bailt momb.. bullt0 roods
Drop: drop . fal
Action Learner
moseljooon (isoed)) ropootiioon22rrppp
[Iistory)] Init obs: Ballt is boll. Ball) at moma, Ballto is ball. Balllo is as rooma.... Leit grippe: Loft &s frve. Right
REPORT
gripper: Right is frre Robbby is at roomb, rome a room.
roomb is room.
enen You aheady hove a hard connet pick
Error Feedback
Augmented Action Space
Policy Model
Adapted
Action move trommb/, 'rooma))
Action carry to/"alll" Troomaa Hightt
ball. Ball
Robby
Ruilt is hall. Balll roomb. .o...
cerry.vv.R..
rovigg ?
Act Agent
Action: carry_ /rommma /tigmm
carry
Obs: Bulll balll Ballt is roonls Ball2 bull.
Assess Agent on Training Instances
ratrroom........
Figure 1: Illustration of the training and test stage of LearnAct: Left: During the training stage, LearnAct expands the action space by first creating actions and then optimizing them based on the execution feedback. Right: The test stage uses learned action space to facilitate sequential decision-making. The prompting format follows the Act (Yao et al., 2023a) agent.
the action space, thereby aligning tasks more closely with the agents' planning abilities. By adapting the action space to fit the LLM's planning, we address the limitations imposed by fixed action spaces, such as the misalignment between commonsense mnowledge-guided planning and actions, and the prevalence of action errors due to unmet prerequisites or ineffective strategies(Gu et al., 2022; Ma al., 2024; Ahn al., 2022). This approach not only mitigates bottlenecks in language agent performance but also allows transferring experience across different tasks.
For more illustrative view, we could look at the example where an LLM agent is asked to make cocktails. With a learnable and open action space, LLMs may naturally instruct the agent to gather all the components in a specific order and mix them, while the closed action space may only involve basic handling, such as moving to different places and grabbing bottles, which greatly increases the difficulty of the task for LLM agent. On the other hand, it is necessary to update existing actions to accommodate changing factors in the environment. Still, in the cocktail example, when making an Old Fashioned cocktail, muddling sugar is often the standard practice. However, if only syrup is available, the agent should adapt its pre-set actions to prevent repeated failures despite being competent to accomplish the task.
To solve these issues, we conduct an extensive exploration of action learning in LLM agents. We introduce a framework LearnAct, designed to generate and optimize new action types as APIs dynamically. The newly generated action types are in the form of Python functions, leveraging the LLM's extensive prior knowledge and code generation ability devise diverse and representative action space. Furthermore, LearnAct stands out due to its iterative learning strategy, which continuously refines actions through a feedback loop. In each cycle, the LLM evaluates the effectiveness of current actions using training examples, identifying and rectifying errors in failed instances. The learning strategy progressively deepens task understanding and improves learnable actions.
Our experimental results demonstrate that this method of iterative refinement not only creates complex and user-friendly action types but also achieves more effective and efficient learning compared to previous sfate-of-the-art methods such as Reflexion (Shinn et al., 2023). By learning on a few problem instances, LearnAct can generalize to general type of task with strong performance. In summary, our contributions are as follows.
We propose the action learning framework for interactive decision making language model agents by generating and updating learnable actions, enabling learning customized action space that better fits the LLM's planning capacity.
To implement the action learning framework, our method LearnAct employs Python functions to generate new actions, enabling flexible definitions of action types. Our iterative learning strategy incorporates LLM to autonomously refine
2

---

Published as a conference paper at COLM 2024
and updates the currently available actions based on errors in failed tasks.
Our experimental results demonstrate that LearnAct effectively learns action spaces within a few trials, acquiring transferable capabilities in diverse environments. Through action learning, LearnAct outperforms SOTA agents by a significant mar- gin, showcasing the potential of action learning for LLM agents.
2 Related Work
LLM Agent Learning Recent research has advanced the use of large language models (LLMs) in embodied agents (Duan al., 2022; Huang al., 2022a; Ahn al., 2022; Huang et al., 2022b; Yao et al., 2023a; Park et al., 2023; Wu et al., 2023; Sun et al., 2023; Ma et al., 2024; Yuan et al., 2023; Qiao et al., 2023; Stengel-Eskin et al., 2024). These methods differ (Wei et al., 2022; Chen
from previous reasoning-eentered centered methods like Shain-of-Thouge et al., 2022; Liang et al., 2023; Wang et al., 2023b) as they iteratively incorporate environment feedback to modify subsequent plans. This closed-loop process is centered on the ability of LLM agents to learn from environment.
The majority of previous work multi-turn reflection concentrates on learning from environment feedback within the same problem (Lai et al., 2023; Le et al., 2022; Chen et al., 2023; Liu & Abbeel, 2023; Singh et al., 2023). ReAct (Yao et al., 2023a) propose basic framework for using thought to enforce the LLM reflect on previous behaviors within the same trial. Reflexion (Shinn et al., 2023; Park et al., 2023), on the other hand, uses a multi-trial approach to perform reflection. By summarizing historical experiences and identifying reasons for failure, LLM is prompted to generate insights for more effective agent instruction. Similarly, ExpeL (Zhao et al., 2023) facilitates general learning system that extracts insights and past experiences with retrieval. Retroformer (Yao et al., 2023b) propels introduce prompt-based training to enhance LLM reflections.
Our method differs from this line of work two points (1) LearnAct performs direct learning in the action space, which substantially improves the reliability and utility of generated actions, (2) LearnAct does not target a single problem instance but learns experience from few training instances and is tested on general type of task. A more detailed comparison is provided in Table 1.
Hierarchical Reinforcement Learning LearnAct generates new action types based atomic actions to enable more informative and applicable actions, similar to the approach in hierarchical reinforcement learning (Erol et al., 1994; Kulkarni et al., 2016; Bacon et al., 2017), where a high-level executor plan with high-level action types, and then executes the seed actions according to the high-level plan (Yao et al., 2023a; Sharma et al., 2022; Wang et al., 2023a; Sun et al., 2023). Our approach also differs from hierarchical reinforcement learning by not only using code-based action space, providing stronger capacity due to the code statements such as conditions and loops (Cai et al., 2023; Qian et al., 2023), but also avoiding the typical imposition of strict two-level structure, using flexible mixed-level actions instead.
Closed Loop
Action Type
Learnable Transferable Learnable Module
ReAct
Natural language
ReAct+Reflexion
Natural language
x
Policy
CodeAsPolicy
Code
Voyager
Code
Action
LearnAct
Code
Action
Table 1: The comparison of our method and other open action agents. of Voyager used curriculum learning pipeline tailored for Minedojo to learn new actions.
3 Problem Statement
The task of a language agent can be succinetly modeled as a Partially Observable Markov Decision Process (POMDP), which is defined by a tuple (,,O,, R), with S representing
3

---

Published as a conference paper at COLM 2024
Algorithm 1 Training Process of LearnAct
Input: Training Set Dirain = ( EM}, Language Agent G, Learner LLM F.
Input: Original Actions Ap, Basic Task Instruction 700
Assess action space
function Nolveerobleem(,,,.44
for = 1 to maxsteps do
A: ktt
Execute at, add observation or fail info to ht, end for
return I
end function
# Training Procedure
A1, A, = ActionCreation(To/
for i = 1 to maxiter do
results = SolveProblem( + T.A E Em A; G), to M
S1:T, #1.7, = SelectErrorCase (resullss
Ai A111, ATH1 S
end for Return Ajast Agast
the set all possible states, O being the observation space through which the agent perceives the state, A denoting the action space, T : x A + S being the state transition function, and R:
A + {0, 1} being the reward function.
At step t, the language agent G takes an action step based policy (policy prompt), problem & (problem instructions), and previous observation-action trajectory h; (0,, [0, ................................ , 01-1,
= G(n,E,
(1)
r,, E, and ht; correspond to the instruction, goal, and history in Figure (right), respectively. Action a is successfully grounded (executable) only if it is within the action space, i.e. E A. Even successfully grounded action may be invalid due to unmet conditions or has no effect in the current state. The agent aims to maximize the final reward r. this work, we use an outcome-based reward mechanism (ORM) (Uesato et al., 2022) that assigns binary indicator as a reward, depending on whether the task has been successfully completed.
In this work, try to answer the question How to learn from experience and apply them to other decision-making problems? We focus on the wraining-esstigg setting, where for each task, LLM agent optimizes its action space and corresponding policy the training set Dtrain = {E,..., EM) before trying to accomplish problems in the test set Diest ,EM+L...,EMM
These problems share the same original action spaces and rules, though requiring varied policies to solve them as they have different scenarios and goals. This setting poses a challenge for the agent to develop general capacity through learning to accumulate experience and accomplish tasks of this general type.
To address this problem, we introduce an action-learning framework designed for language agents, enabling them to autonomously enhance their skills by learning from interactions within their environment, as shown in Figure The framework considers the action space, denoted as A, to be an open set capable of learning and expanding. More specifically, we define the original action space as Ag and augment with newly learned action type (API) A', represented as Ao U A'. Also, instruction for using these new actions πT,'' is updated to the original policy instructions Ti, iee. the new policy is denoted as 7l + T( Consequently, subsequent actions by this LLM agent are characterized
a; G( TL h; a é Ag U A', as depicted in Figure 1 (right), where the agent is guided to generate both origin and learned actions.

---

Published as a conference paper at COLM 2024
4 Method
The overall pipeline of LearnAct is illustrated in Figure and Algorithm 1. The training stage of LearnAct involves first creating new actions and then refine them based on the error feedback on training samples. After learning the action space and policy instructions in the training stage, the refined agent perform actions from the augmented action space and try to accomplish the problem step by step. The training stage is specified in $4.1, and test stage detailed in $4.2.
4.1 Training Stage
1. Expanding Action Space by Action Creation. Our method first creates custom action types, akin to API, to enable LLMs to interact with the environment more seamlessly, denoted as ActionCreation. These newly crafted action types are implemented as code functions, thereby unlocking the potential for more intricate logical expressions that leverage the foundational APIs through constructs such as conditional statements (if-else), loops, and assertions. ActionCreation involves two steps:
(1) Generating Action Function A': Upon receiving detailed instructions and problem specifics, the LLM, denoted as F, is prompted to summarize high-level actions to complete this task in the form of Python functions for agent use. The detailed prompt is the Appendix. These functions can call multiple basic or defined actions to complete subtasks. After generation, functions are parsed and added to the action space. See detailed prompt in Appendix.
(2) Generating Policy Instruction N for Using New Action: After generating new action functions, we then update the agent with information about their potential applications. As the basic instruction contains action descriptions and usage examples, the updated policy guidance includes both a description of each function and a usage example.
First, the LLM F is prompted to provide comprehensive description of each new function. The purpose of this description is to offer the agent an overview of the anticipated outcomes and necessary conditions, along with the input-output format of the new action. Second, the language model generates illustrative usage examples for the new functions. This is crucial for guiding LLM to use the new action in the appropriate scenario, indicated by previous work (Schick et al., 2023). Prompts are in Appendix.
We denote this process by the ActionCreation:
AI, Ar = ActionCreation (To, F),
(2)
2. Learning Actions Based Error Feedback After creating the initial set of actions, there's a possibility that these actions may have errors, misunderstand the task, overlook certain task scenarios, or possibility for misuse. To address this, we devise the training phase that allows the language model to learn through trial and error, as shown in Figure 2. In this phase, the agent is executed in the training instances, identifies action failures, and then iteratively updates the action set, until successfully solving all the instances with no action errors or exceeding maximum optimization steps.
During the learning process, the agent first tries to solve problems in the training set with the current available action space A and policy instruction A, denoted as the SolveProblem process. Then failed problem with action error is sampled with SelectErrorCase operator and then the ActionLeary either fixes it by revising the used action or writing note as an annotation. We present each of them below:
(1) SolveProblem: The assessment of current policy and action space is done by trying to solve problems in the training set. We follow the simple Act(Yao et al., 2023a) agent and prompt the agent to solve the task by interacting with the environment and generating actions based on history.
(2) SelectErrorCase:
This step identifies error-inducing steps in action sequences based on the environment's 's feedback, such invalid actions due to unmet preconditions, ineffective actions, or errors in action names or parameters. For learnable actions composed multiple atomic steps, it evaluates errors in each atomic step execution.
5

---

Published as a conference paper at COLM 2024
Assess Revised Action Space
eet your shat in seileef inpeediest
lonzzz
Pro take follwing inttrs
Action de
Revised Samples of Action Space and Policy
dian chatird repaddent,
live, chave harth loutt lowel))
hedd lnve The active the
Shcker ingedd
Environment
LLM- Agent
Ei Llea to oegtoereeee deeiet diht bet speneet
coacian cowciaeiiggeeeeeo ingendewzs
Action Learner
Obsernation
Function Updating
Notes for Functions
Sample Best A. A
det shat, dletss Ingroderr, Lnvel
Instuction:
Resilts
pau ahiuis clan shakertibr.
par. lnvert thlcoatie
ingrediente araae
Find Failed Case
Trajectory Fesebock M1.
Action Space
A
deam uwwtiintt inpooteet handt. hind2
pear shet csed alost
ixnd to oour ingeudi skan
iogddiee Heie our arloncic
During the learning stage, action usage by agent and action optimization are repeatedly executed. The improved action is evaluated on the training instances, identifying the failed case for the subsequent learning step. Actions are improved through either updat- ing the functions or writing notes. Multiple samples are produced during the learning, and upon evaluation against training instances, the optimal one selected for the next iteration. Right: Case example of action updating and note writing. The action update addresses previous shortcomings by refining functions for improved issue resolution. Conversely, note writing advises agents on proper action usage. LLMs have the freedom to select from two learning options.
Figure 2: Left:
(3) ActionLearn: We address errors by implementing either function updates writing notes. Function updates refines the Python function to correct the misunderstanding and overlooking of tasks, whereas writing notes entails enhancing the function's 's description to guide the agent toward more accurate use. The two options are both available and LLM free to choose one of them, shown in Figure 2. When function updates are made, the corresponding action instructions TA, are also updated later to adapt to the new actions. Although function updating triggered for a specific action function, it is not restricted to modifying just this single action. By accessing all action functions via prompt, LLM could choose by itself the actions revise and could also incorporate new actions. Consequently, ActionLearn can freely update the entire action space in each iteration. The detailed prompt in the Appendix.
3. Augment Action Learning with Sampling In practice, the action learner could generate low-quality actions that can greatly affect the efficiency of the overall optimization process. Also, some action revisions may be spurious. Thus to enhance the stability of the learning process and improve the quality of each step, we sample times with ActionLearn, yielding K revised results (A), - I.=I...... During the learning iteration, each action-policy pair is evaluated on the training set, and the best one is chosen, as shown in Figure 2 (left) and the
detailed algorithm in Appendix.
We select the best action based on the results obtained with the agent. The score μ for sample AK, Tik is follows:
p Psucc + Pstepace,
Here, Psuce denotes the success rate of the agent in training instances, and Pstepace refers to the ratio of successfully executed actions to the total number of steps taken by the agent, indicating the practicality of generated actions. Based on these selection criteria, we identify the most beneficial and feasible action sample A,, TT.
4.2 Testing Stage
After completing the learning phase, the agent possesses updated action space and refined policy instructions. In the testing phase, the agent attempts to solve problems in the same procedure the SolveProblen process used during training. Notably, whereas

---

Published as a conference paper at COLM 2024
most prior research (Sun et al., 2023; Shinn et al., 2023) has concentrated on the learning loop within single problem instance, LearnAct is designed to facilitate the transfer to this general type.
Experiment
5.1
Tasks
Robotic Planning (Ma et al., 2024) includes four challenging Robotic Planning tasks, namely Gripper, Blockworld, Barman, and Tyreworld. We followed the environment implemen- tation of AgentBoard (Ma et al., 2024) and LLM+P (Liu et al., 2023). These tasks involve long-horizon robot planning problems, such as creating cocktails based on customer orders using available ingredients and containers, moving objects between different rooms using grippers, or rearranging piles of blocks to achieve specified target configuration. These tasks put a significant demand on the agent's s long-term planning capabilities.
AlfWorld (Shridhar et al., 2020) simulates six types of objectives within a household. For instance, agents are required locate an apple within the house, heat it, and then place it in target area. These tasks push the agent to explore the house systematically, given that the complete state of the environment is unknown to the agent.
5.2 Baselines
Act ReAct (Yao al., 2023a): The basic agent, Act, is prompted iteratively take actions and obtain observations in the environment. Based on Act, the ReAct agent employs "Think" as an additional action.
Reflexion (Shinn et al., 2023): Reflexion is learning method for language agents that utilize language as learned policy. The original Reflexion is designed for learning individual policies in each instance. In our study, we adapt Reflexion for the training- soaning-testing setting, where policies are designed to be not instance-specific, facilitating transferability. Details are in the Appendix.
AsPoicyy (Liang et al., 2023): The method utilizes code to generate the entire solution for the task in open-loop way. As the method does not interact with environments, only report the results of CodeAsPolicy on Robotic policy tasks, because the ALfWorld task necessitates exploration in the environment to obtain information, and CodeAsPolicy cannot generate code solution without access to the complete environmental information.
Voyager (Wang al., 2023a): Voyager proposes functions as actions solve tasks in closed-loop agent manner. The original Voyager is specifically designed for Minedojo (Fan et al., 2022), emphasizing skill acquisition through a structured hierarchical task curriculum. For comparison, we reimplement it as Wong et al. (2023), which creates skills via code with basic verification.
5.3 Setting
We test on both GPT-4 and GPT-3.5 Turbo models as the LLM during testing. We employ GPT-4 throughout our learning for action creation and improvement, as well as the learning of Reflexion and Voyager. We set the temperature as 0.0 for consistency and reproducibility in our experiments. The sampling number in our method is set to 4. For each task, we randomly select 3 instances for the training set, with the remaining instances used for testing. We report test set results for our LearnAct and all baseline models. The primary evaluation metric was the task success rate. Each experiment is conducted three times per task, and we present the average results. We ensure fair comparison by providing a single in-context example to LearnAct and all baselines. CodeAsPolicy and Voyager's 's action creation utilize the identical code example ours.
7

---

Published as a conference paper at COLM 2024
GPT-3.5
GPT-4 barman
blockworld
gripper
barman
tyreworld
Arg.
blockworld
gripper
lyreworld
Avg.
Act
0.0
5.9
0.0
10.0
4.0
57.1
38.7
64.7
62.1
60.7
ReAct
0.0
5.9
9.8
0.0
3.9
71,4
45.1
70.5
66.4
63.3
Act=Reflexion
11.8
62.1
38.8
70.6
ReAcctReeflxion
4.3
9.9
11.8
4.3
7.6
72.2
57.1
74.4
63.1
66.7
CodcAsPolicy
Voyager
18.6
2.0
6.0
19.3
11.5
66.4
76.5
80.5
65.0
72.1
LearnAct
10.0
34.8
15.0
32,9
73.9
82.3
87.4
87.6
Table 2: Performance of LearnAct and baselines Robotic Planning tasks.
GPT-3.5
GPT-4
Clean
Heat
Avg.
Put
Clean
Heat
Look
Put 2
Avg.
Act
34.7
15.1
13.0
14.2
48.1
23,6
24.8
77.7
50.5
13.0
42.8
40,8
66.6
48.6
ReAct
25.4
8.3
3.2
13.1
11.1
36.1
16.2
73.9
61.9
30.5
46.4
59.7
49.8
53.7
Act+Reflexion
44.5
22.6
24.9
26.2
33.6
11.8
27.3
714
47.5
6.8
35.1
80.2
524
48.9
ReActt-Refexion
41.4
11.9
3.2
7.1
8.9
39.3
18.6
61.8
54.0
39,9
62.9
57.6
52.3
51.7
CodeAsPolicy
Voyager
31.5
27.4
11.7
22.6
9.6
18.3
84.1
60.8
27.9
49.7
68.6
57.1
58.0
LeamAct
65.1
36.1
30.3
42.0
24.1
16.9
35.8
85.6
76.2
53.2
64.5
81.5
72.2
Table 3: Performance of LearnAct and baselines on Alfworld tasks.
Action Calling Rate
Initial Learned
0.8
0.4
barman
blockworld
gripper
tyreworld
clean
cool
éxamine
heat
Put
Puttwo
Stepe
(doi: got
Action Success Rate
Initial
Learned
0.6
0.4
0.2
0
barman
blockworld
gripper
byreworld
clean
cool
examine
heat
put
puttwo
Heration Sugpa
Stape
Figure 3: Left: The frequency of use and accuracy of learned actions before and after learn- ing. Post-learning, there is marked increase in the action usage frequency and accuracy, indicating their enhanced reliability and utility for the agent. Right: The performance with different maximum learning iteration steps. The performance varies with different maximum learning iteration steps. LearnAct's performance notably improves with the application of learning, particularly at step two. Although ReAct+Reflexion also shows improvement, its progress is less significant and stable.
5.4 Main Results
We present a comparison of LearnAct with baseline models in Tables 2 and 3. LearnAct outperforms the baselines in various tasks, utilizing both GPT-3.5 Turbo and GPT-4 as backbones. We further analyze the performance and discuss the conclusions drawn from the comparison.
A well-designed action space enhances the language agent's ability to plan and solve tasks. LearnAct markedly surpasses the Act baseline in performance, highlighting the crucial role of the action space. It is noteworthy that, after its learning phase, LearnAct is also an Act type agent except with a developed action space. This supports our hypothesis that the limitation of language agents lies in the effective integration of the planning process with the action space, and an adaptable action space can substantially unlock the latent planning capabilities of LLM. Additionally, LearnAct surpasses ReAct in both Robotic
8

---

Published as a conference paper at COLM 2024
Planning and AlfWorld, demonstrating that while "Think" improves planning, it suffers from alignment with the action space.
Action learning outperforms the verbal policy learning. In the realms of Robotic Planning and AlfWorld, LearnAct markedly outperforms both Act+Reflexion and ReAct+Reflexion. Reflexion writes policy prompts during the learning process to assist the agent in task- solving. This approach does yield improvements over the Act and ReAct methods; however, these enhancements are considerably less pronounced than those achieved by LearnAct, which focuses on learning within the action space. While learned verbal policies and prompts can aid in enhancing the agent's planning capabilities, possessing proficient skills is essential for the agent to fully realize its potential.
Action learning is important for the action quality. LearnAct outperforms coding baselines such as CodeAsPolie Poliyy and Voyager. CodeAsPolicy directly generates code in open loop without interacting with the environment, leading to inferior performance compared to closed-loop agent baselines. Notably, while Voyager generates actions using method similar to LearnAct, it lacks the component of iterative action learning. This highlights the significance of action learning. Without this learning phase, the created actions are constrained by insufficient understanding of the task.
5.5 Analysis of the Learning Process
Learning enhances performance through action reliability and utility. We further analyze action utilization to explain why action learning enhances agent performance. We assess the frequency of use and accuracy at both the initial stage of learning and the post-learning. These results are presented in Figure 3 (left). It is evident that, compared to the initial action values, the actions post-learning demonstrate significantly higher correctness, indicating their enhanced reliability and utility for the agent. This is a direct consequence of our learn- ing method, which focuses on correcting errors encountered during usage. Additionally, it is observable that the rate of action usage also improves after learning. It is possible because the agent is more inclined to employ these actions upon recognizing their usefulness, i.e., observing valid effects after calling the action.
Robotic Planning Alfworld
Robotic Planning AlfWorld
Function Updating 0.86
0.92
Initial After Learning
3.75
3.44
Note Writing
0.14
0.08
3.83
3.50
Table 4: The ratio of two learning choices. Table 5: Average number of learned actions.
LearnAct learns action by updating functions, writing notes, and generating new actions. We analyze the learning choices and learned action numbers. The strategies for learning encompass function updates and writing notes. We find that the model prefers to update functions. Table 4 shows function updating occurs for about 90 % in the learning. This may be due to the model's ability to circumvent most invalid executions through code adjustments. Additionally, the learning process can lead to the creation of new functions. As shown in Table 5, the average number of learned actions ranges from 3 to 4 for both Robotic Planning and AlfWorld, and the number of actions tends to increase after learning, showing that new actions could be innovated while updating existing ones.
The iteration number influences the performance of the model. We illustrate the impact of learning iterations on LearnAct and CeAct=Reflexion that in both AlfWorld and Robotic Planning tasks, LearnAct's learning markedly enhances performance, achieving optimal results within just two iterations. The enhancement from our learning approach is notably more substantial than that observed with Reflexion. While ReAct+ Reflexioo also shows performance gains, its impact is less consistent and displays more negative effects.
Figure 3 (right). It is evident
Why does prolonged learning with LearnAct detrimentally affect performance? We observe that it cannot be avoided for agents to make mistakes, excessive optimization for these mistakes can lead to overfitting to specific training cases and misunderstanding the task rule. See failed cases in Appendix.

---

Published as a conference paper at COLM 2024
5.6 Ablation Study
We conducted ablation experiments to demonstrate the significance of various components in our method. First, we evaluated the action format our method, including descriptions and usage examples of actions. The results of employing GPT-4 the agent in Robotic Planning and Alfworld are presented in Table 6. It was observed that both the description and usage examples contribute to the performance, with the usage examples having more pronounced impact compared to the descriptions. Interestingly, we discovered that the usage examples often incorrect. Despite this, they still encourage the agent to utilize the learned actions. The descriptive aspect also plays a crucial role in Robotic Planning tasks. This could be attributed to that the actions have many pre-conditions, so the descriptions of the learned actions aid the agent in their correct usage.
Robotic Planning Altworld
LearnAct sampling only
82.8
72.2
Robotic Planning Alfworld
73.4
60.2
LearnAct 82.8
72.2
w.o. updating function
75.3
62.0
w.o. description 78.5
72.4
w.o. writing notes 80.8
71.5
W.O. usage example 77.4
65.4
w.o. sampling
73.2
62.4
Table 6: The ablation study on the form of ac- Table 7: The ablation study of the learning tions, specifically the action description and method, including the options during learn- ing and the sampling method.
usage examples of actions.
Next, examine the components of our learning algorithm, which includes two types of learning: updating function and writing notes, as well as sampling during the learning process. The results are presented in Table 7. Function updating plays crucial role in action learning, and writing notes for actions further enhances performance. We observed that function updating occurs approximately 90% times during learning, suggesting that the learner predominantly opts to modify functions to improve actions rather than merely adjusting notations for more accurate action use.
Moreover, sampling is vital for the effectiveness of the learning process. In the absence of sampling, the learner generates only single action proposal for the next iteration, which is highly prone to errors and can negatively impact overall performance. However, note that sampling alone (w.o. learning) ineffective in learning actions as LearnAct does, highlighting the importance of our learning algorithm.
6 Conclusion
In conclusion, research advances LLM agents by equipping them with the ability to learn and refine actions through direct interaction with the environment. Our proposed framework LearnAct demonstrates significant improvement in agent performance by enabling open-action learning, which aligns closely with how humans acquire and enhance skills. The empirical success of our methods in Robotic Planning and Alfworld environments underscores the potential of action learning in developing more intelligent and capable LLM agents.
References
Ahn, M., Brohan, A., Brown, N., Chebotar, Y., Cortes, O., David, B., Finn, C., Fu, C., Gopalakrishnan, K., Hausman, K., al. Do as i can, not as say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.016691, 2022.
Bacon, P.-L,, Harb, J., and Precup, D. The option-critic architecture. In Proceedings of the AAAI conference artificial intelligence, volume 31, 2017.
Brohan, A., Chebotar, Y., Finn, C., Hausman, K., Herzog, A., Ho, D., Ibarz, J., Irpan, A., Jang, E., Julian, R., et al. Do as can, not as i say: Grounding language in robotic affordances. In Conference on Robot Learning, pp. 287-318 PMLR, 2023.
10

---

Published as a conference paper at COLM 2024
Cai, T., Wang, X., Ma, T., Chen, X., and Zhou, D. Large language models tool makers. ar Xiv preprint ar (ix::2305.17 2023.
Chen, W., Ma, X., Wang, X., and Cohen, W. W. Program of thoughts prompting: Disentan- gling computation from reasoning for numerical reasoning tasks, 2022.
Chen, X., Lin, M., Schärli, N., and Zhou, D. Teaching large language models to self-debug. arxiv preprint ar XrX::3304..5518, 2023.
Duan, J., Yu, S., Tan, H. L., Zhu, H., and Tan, C. A survey of embodied ai: From simulators to research tasks. IEEE Transactions on Emerging Topics in Computational Intelligence, 6(2): 230-244, 2022.
Erol, K., Hendler, J., and Nau, D.S. Htn planning: complexity and expressivity, In Proceedings of the Twelfth AAAI National Conference on Artificial Intelligence, pp. 1123-1128, 1994.
Fan, L.,, Wang, G., Jiang, Y., Mandlekar, A., Yang, Y., Zhu, H., Tang, A., Huang, D.-A., Zhu, Y., and Anandkumar, A. Minedojo: Building open-ended embodied agents with internet- scale knowledge. Advances in Neural Information Processing Systems, 35:18343-18362, 2022.
Gu, Y., Deng, X., and Su, Y. Don't generate, discriminate: A proposal for grounding language models to real-world environments. ArXiv preprint, abs 2b1...22.09736 2022. URL https://arxi wwt..orr//bbb//2.../9///
Huang, J., Chen, X., Mishra, S., Zheng, H. S., Yu, A. W., Song, X., and Zhou, D. Large language models cannot self-correct reasoning yet. arXiv preprint ar Xi::23100007998 202.3
Huang, W., Abbeel, P., Pathak, D., and Mordatch, 1. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In International Conference on Machine Learning, pp. 9118-911 91477 PMLR, 2022a.
Huang, W., Xia, F., Xiao, T., Chan, H., Liang, J., Florence, P., Zeng, A., Tompson, J., Mordatch, I., Chebotar, Y., Sermanet, P., Brown, N., Jackson, T., Luu, L., Levine, S., Hausman, K., and Ichter, B. Inner monologue: Embodied reasoning through planning with language models, 2022b.
Kulkarni, T. D., Narasimhan, K., Saeedi, A., and Tenenbaum,) J. Hierarchical deep reinforce- ment learning: Integrating temporal abstraction and intrinsic motivation. Advances in neural information processing systems,, 29, 2016.
Lai, Y., Li, C., Wang, Y., Zhang, T., Zhong, R., Zettlemoyer, L., Yih, W.-t,, Fried, D., Wang, S., and Yu, T. Ds-1000: A natural and reliable benchmark for data science code generation. In International Conference on Machine Learning, pp. 18319-18345, PMLR, 2023.
Le, H., Wang, Y., Gotmare, D., Savarese, S., and Hoi, S. H. Coderl: Mastering code generation through pretrained models and deep reinforcement learning. Advances in Neural Information Processing Systems, 35:21314-21328, 2022.
Liang, J., Huang, W., Xia, F., Xu, P., Hausman, K., Ichter, B., Florence, P. and Zeng, A. Code as policies: Language model programs for embodied control. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 9493-9500. IEFE, 2023.
Liu, B., Jiang, Y., Zhang, X., Liu, Q., Zhang, S., Biswas, J., and Stone, P. Llm+ p: Empowering large language models with optimal planning proficiency. arviv preprint arXiv:2304.11477, 2023.
Liu, H. and Abbeel, P. Emergent agentic transformer from chain of hindsight experience. ar Xiv preprint ar arXiv:2305. 16554, 2023.
Ma, C., Zhang. J., Zhu, Z., Yang. C., Yang, Y., Jin, Y., Lan, Z., Kong, I.,, and He, Agentboard: An analytical evaluation board of multi-turn 11m agents, 2024.
Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., et al. Self-refine: Iterative refinement with self-feedback. arxiv preprint arXit:2303. 17651, 2023.
11

---

Published as a conference paper at COLM 2024
Park, J. S., O'Brien, J., Cai, J. Morris, M. R., Liang, P., and Bernstein, M. S. Generative agents: Interactive simulacra of human behavior. Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, pp. 1 1-22, 2023.
Qian, C., Han, C., Fung, Y. R., Qin, Y., Liu, Z., and Ji, H. Creator: Disentangling abstract and concrete reasonings of large language models through tool creation, 2023.
Qiao, B., Li, L.,, Zhang, X., He, S., Kang, Y., Zhang, C., Yang, F., Dong, H., Zhang, J., Wang, L., et al. Taskweaver: A code-first agent framework. arXiv preprint ar Xrii:2311.17541, 2023.
Qin, Y., Liang, S., Ye, Y., Zhu, K. Yan, L., Lu, Y., Lin, Y., Cong, X., Tang, X., Qian, B., Zhao, S., Tian, R., Xie, R., Zhou, J., Gerstein, M., Li, D., Liu, Z., and Sun, M. Toolllm: Facilitating large language models to master 16000+ real-world apis, 2023.
Schick, T., Dwivedi-Yu, J., Dessi, R. Raileanu, R., Lomeli, M., Zettlemoyer, L., Cancedda, N., and Scialom, T. Toolformer: Language models can teach themselves to use tools, 2023.
Sharma, P., Torralba, A., and Andreas, J. Skill induction and planning with latent language.
In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1713-1726, 2022.
Shinn, N., Cassano, F., Gopinath, A., Narasimhan, K. R., and Yao, S. Reflexion: Language agents with verbal reinforcement learning. In Thirty-seventhr Conference on Neural Informa- fion Processing Systems, 2023.
Shridhar, M., Yuan, X., Côté, M.-A,, Bisk, Y., Trischler, A., and Hausknecht, M. Alf- world: Aligning text and embodied environments for interactive learning. arxio preprint ar arXie:2010.03768, 2020.
Singh, I., Blukis, V., Mousavian, A., Goyal, A., Xu, D., Tremblay, J., Fox, D., Thomason, J., and Garg, A. Progprompt: Generating situated robot task plans using large language models. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp- 11523-11530. IEEE, 2023.
Song, C. H., Wu, J., Washington, C., Sadler, B. M., Chao, W.-L., and Su, Y. LIm-planner: Few- shot grounded planning for embodied agents with large language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2998-3009, 2023.
Stengel-Fskin E., Prasad, A., and Bansal, M. Regal: Refactoring programs to discover generalizable abstractions. arXiv preprint ar arXiv:2401.16467 2024.
Sun, H., Zhuang, Y., Kong, L., Dai, B., and Zhang, C. Adaplanner: Adaptive planning from feedback with language models, 2023.
Uesato, J., Kushman, N., Kumar, R., Song, F., Siegel, N., Wang, L., Creswell, A., Irving, G., and Higgins, 1. Solving math word problems with process-and outcome-based feedback. ar Xiv preprint ar XrXi::2211.14275 2022.
Wang, G., Xie, Y., Jiang, Y., Mandlekar, A., Xiao, C., Zhu, Y., Fan, L., and Anandkumar, A. Voyager: An open-ended embodied agent with large language models. arviv preprint arXit:2305.16291, 2023a.
Wang, H., Pumrarig, G., Sharma, Y., and Choudhury, S. Demo2code: From summarizing demonstrations to synthesizing code via extended chain-of-thought arviv preprint xxie:2205.16744, 2023b.
Wang, X., Wang, Z., Liu, J., Chen, Y., Yuan, L., Peng, H., and Ji, H. Mint: multi-turn interaction with tools and language feedback. ArXiv preprint, abs/ 1bs/2209.00691,, 2023c. URL https: //arxiv//axvvv... org/ htgg/bb/2330 2309. 10099 18691.
Evaluating llms in
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F. Chi, E., Le, Q. V., Zhou, D., et al. Srain-of-thoughee prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 05:24824-24883 2022.
12

---

Published as a conference paper at COLM 2024
Wong, L., Mao, J., Sharma, P., Siegel, Z. S., Feng, J,, Korneev, N., Tenenbaum, J. B., and Andreas, J. Learning adaptive planning representations with natural language guidance. arxiv preprint ar X11::2211..08566, 2023.
Wu, Y., Min, S. Y., Prabhumoye, S., Bisk, Y., Salakhutdinov, R., Azaria, A., Mitchell, T., and Li, Y. Spring: Gpt-4 out-performs rl algorithms by studying papers and reasoning. ar Xiv preprint X51::2305.15486, 2023.
Xie, T,, Zhou, F., Cheng, Z., Shi, P., Weng, L., Liu, Y., Hua, T. J.J., Zhao, J., Liu, Q., Liu, C., et al. Openagents: An open platform for language agents in the wild. arv preprint ar Xiv::2330. 106,44,,,, 2023.
Xu, Y., Su, H., Xing, C., Mi, B., Liu, Q., Shi, W., Hui, B., Zhou, F, Liu, Y., Xie, T., et al. Lemur: Harmonizing natural language and code for language agents. arXiv preprint ar Xri::2310.06830, 2023.
Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., and Cao, Y. React: Synergizing reasoning and acting language models, 2023a.
Yao, W., Heinecke, S., Niebles, J. C., Liu, Z., Feng, Y., Xue, L.,, Murthy, R., Chen, Z., Zhang, J., Arpit, D., et al. Retroformer: Retrospective large language agents with policy gradient optimization. arv preprint ar arXiv:2308.0213: 022151, 2023b.
Yuan, L.,, Chen, Y., Wang, X., Fung, Y. R., Peng, H., and Ji, H. Craft: Customizing Ilms by creating and retrieving from specialized toolsets. arXiv preprint ar XrX::2309917748, 2023.
Zhao, A., Huang, D., Xu, Q Lin, M., Liu, Y-J., and Huang, G. Expel: Lim agents are experiential learners. Xiv preprint ar Aii:22308.1014 2023.
13

---

Published as a conference paper at COLM 2024
A Appendix
A.1 The Action Learning with Sampling
Algorithm illustrates the general framework for the learning iteration. In practice, we include multi-sampling in each learning iteration, to avoid the misdirection of low-quality solutions the whole learning. Here we give detailed algorithm of our learning method with sampling in Algorithm 2.
Algorithm 2 Training Process of LearnAct with Sampling
Input: Training set Drainn = (E1,..,, E LLM G.
Input: Original Actions Ag, Basic Task Instruction go
Input: Sampling Number K
Training Procedure
At A = ActionCreation (onteerreeiionoo)).kooo ro), t
for = 1 to maxiter do results SolveProblem(ne T + 2 Em, An U G), M,K t
# Compute score and select the best sample nº Pluce + Pssepaccc k 1 to K
kbest = arg max; μd
results, A, Ai results Abett Afeer
A
$1,T A/T11 7 end for Return Alastr π Agst
seleectErorCaseeresulle
A1t1 ActionLear(AA,, (1 to
A.2 Complexity Analysis
The testing stage of LearnAct follows the standard language agent setting, except that the action space is augmented with manufactury actions. The overall time complexity is thus the same as the standard Act method. As for the learning stage, the time complexity is OMMKI), where M is the training instance number, K is the sampling number, and I denotes the max iteration number.
A.3 A Bayesian View of Open-Action Learning
In this subsection, we present Bayesian perspective on our learning approach. We express the transition function T and reward function R in probabilistic terms p(Sy=1|sp, a;) and p(r(s1.T). Denote the observation function of state o; o(s:), and the history at time is denoted as h(s1 41# 1) (...)).).............. o(s; 1),a, a, 1, (s:)], i.e., the function of $ a1 The distribution of Partially Observable Markov Decision Process trace is as follows:
T-1
PART (SIT, p(s))
I
41. 1)) (4)
[=1
The reward conditioned on action space A and policy π follows the distribution:
p(r)A, AA =E EA (((ssr)
(5)
The objective of POMDP is to maximize the reward, which can be seen as the posterior of action space A and policy A given the maximum reward:
P(A, AA = -mmax)= p(A, π AT PANA A (T = Fmax (51:7)
(6)
Zr
14

---

Published as a conference paper at COLM 2024
where Zymax E (r.r (Ar Tmax((1:7) is the normalization factor. In con- ventional reinforcement learning, A is predetermined and remains constant, while the model only learns A through maximum likelihood rather than posterior estimation. This is attributed to the complexity of computing the posterior, which is due to the necessity of domain knowledge for the prior p(A, A and the intractable calculation of the posterior.
However, leveraging the advanced capabilities of language models, we propose employing them estimate the posterior of the action space and policy toward the open action space learning. The prior p(A, AA) can be defined as the operator ActionCreation, and the posterior updating can be inferred as ActionLearn.
A.4 Detailed Prompt
We present the prompt used in experiment here, due space limitations of the main text.
We first present the prompts for action learning. The ActionCreation process entails two phases: initially, function generation is guided by Prompt 4, followed by the crafting of function descriptions and usage instructions, as directed by Prompt 5 and 6, respectively.
During the action learning iterations, ActionLearn leverages Prompt to derive results. If the LLM suggests updates action functions, descriptions and usage instructions are regenerated using the Prompt 5 and 6.
Prompt for Action Creation
[basic task instruction]
Please propose several high-level steps for this task.
Each high-level step should be Python function encompassing multiple (at least two) basic actions. All the values used in the function should be given as input rather than fixed in the function.
The provided actions are Python functions and can be executed directly, for example, python [basic action example)
No additional interfaces besides the provided actions are available. All the code should be wrapped by" pytthon
Here examples: (created action example]
Now please write your solution:
Figure 4: Prompt for action creation.
Prompt for Action Description Generation
[basic task instruction]
Now here are some Python function encompassing multiple basic actions serve as high-level interface in this task. Please write interface instruction for the given Python function.
15

---

Published as a conference paper at COLM 2024
Here are examples:
{action description examples)
Now please write interface instruction for this high-level step {func name):
Function:
python
(function)
Instruction:
Figure Prompt for action description generation.
Prompt for Action Usage Example Generation
{basic task instruction]
Now, here are some Python functions encompassing multiple basic actions to serve as high-level interfaces in this task. Please complete the task with the interface following the format of the examples. (format instruction]
Here examples:
{action usage examples)
Now, please complete the task using these interfaces:
Function:
python
[function]
Example:
Figure 6: Prompt for action usage example generation.
Prompt for Action Learning
(basic task instruction] (task goal]
The actions provided are Python functions and can be executed directly, for example, pythhnn
{action example]
No additional actions besides the provided actions are available. All the code should be wrapped by" python
Now here are some high-level steps to complete this task. Each high-level step is general Python function encompassing multiple (at least two) basic actions. All the values used in the function should be given as input rather than fixed in the function.
16

---

Published as a conference paper at COLM 2024
The high-level steps are executed but failed. Please analyze why the execution failed, and give one of the following improvement: Update, Plan. Please respond in the following format:
Failed reason: <>
Improve: <Update or Plan: [The target function]>
Content: <>
Test case: <> (This is only for Update case, not for Plan)
Here are examples:
{in context example]
Now please analyze this case:
python
{actions)
The agent performs this task, and the high-level action {function name) is executed last:
{agent trajectory)
But an error is observed in the last call (ferror info)). The detailed subprocess of this step is: {error subprocess]
Failed reason:
Figure 7: Prompt for action learning.
We then show the prompts during the testing stage, i.e. the prompt of our agent. It is the same as the basic Act agent, as shown the Prompt 8 for the Blockworld task.
Prompt Details for Blockworld
System Prompt
You are a master in planning.
Instruction
The domain assumes a world where there are a of blocks that can be stacked on top of each other, an arm that can hold one block at a time, and table where blocks can be placed.
The actions defined in this domain include:
Pickup(block): allows the arm pick up a block from the table if it is clear and the arm is empty. After the pickup action, the arm will be holding the block, and the block will no longer be on the table or clear.
Putdown(block). allows the arm put down block on the table if it is holding block. After the putdown action, the arm will be empty, and the block will be on the table and clear.
Stack(block1 bacckbllokk.bblockk)) allows the arm to stack block on top of another block if the arm is holding the top block and the bottom block is clear. After the stack action, the arm will be empty, the top block will be on top of the bottom block, and the bottom block will no longer be clear.
allows the arm to unstack a block from on top of another block if the arm is empty and the top block is clear. After the unstack action, the arm will be holding the top block, the top block will no longer be on top of the bottom block, and the bottom block will be clear.
(Bottcck(bocck,,block//
17

---

Published as a conference paper at COLM 2024
[learned action instructions]
Here are examples:
Goal:
The goal is to satisfy the following conditions: b1 is b2., b2 is on b3.
Observation: b2 is on b3. b3 is on b1. bl is on the table. Robot arm is empty. The b2
is clear.
Action: Unstack(b22/bb3))
Observation: b3 is on bl. b1 is on the table. The b3 is clear. You are holding b2. Action: Putdown("b2)
Observation: b3 on b1. b1 is on the table. b2 is on the table. Robot arm is empty. The b2 is clear.
Action: Ub3ta)k(b33,b11))
Observation: b1 is on the table. b2 is on the table. The bl is clear. The b2 is clear. Robot arm is empty. You are holding b3.
Action: Putdown('b3)
Observation: b1 is on the table. b2 is on the table. b3 is on the table. Robot arm is empty. The b1 is clear. The b2 clear. The b3 is clear.
Action: fickup('b2")
Observation: b1 is on the table. b2 is the table. The bl is clear. The b3 is clear. You are holding b2.
Action: Stack((b2'/bb33)
Observation: b1 is on the table. b2 is on b3. b3 is on the table. Robot arm is empty. The b1 is clear. The b2 is clear.
Action: Pickup("b3))
Observation: The action not valid and therefore takes no effect. Please remember to satisfy the restriction of actions.
Action: Pickup('b1)
Observation: b2 is on b3. b3 is on the table. The b2 is clear. You are holding b1. Action: Stack((b1,'b22)
Observation: b1 is on b2. b2 is on b3. b3 is the table. Robot arm is empty. The bl is clear. The goal is satisfied.
|usage example for learned actions}
{goal of testing task)
(history trajectory)
Action:
Figure 8: Prompt details of agent testing for task Blockworld.
A.5 Learned Result Case
We illustrate the learned actions and corresponding instructions in Figure 9. The (learned action instructions) } and (usage example for learned actions] are used in Prompt 8 to inform the agent model about how to use the learned action function.
Learned Result Case for Blockworld
learned action function
def camantle_staccee www.nniilln (block_list. block_ tarkett)
for top_block bottom_ block in (block_list, block licckllst(Il. [1:]): if top_block block_target: break
Unstack ( (top_ block, bottom_block)
Putdown (rfoown(ta (top_block
18

---

Published as a conference paper at COLM 2024
def construct_ censsrruc_sstckbbllee (backk)llo..list)
10
for top_block, bottom_block in reversed block_list(1:))])
11
12
Stack (topbbbok,,, bottom_block)
learned action instructions
block_target): Allows the arm to dismantle stack of blocks one by one, stopping when it reaches specific target block. 'dismantle_stee leeaantlessack_uue fatackkunillblockke. from the block list starting from the top and places it on the table until it reaches the target block. The blocks in the list must be clear and stacked consecutively. If the top block from the list matches the target block, the function ends, leaving the the table. For example, if blocks b3, b2,
block.target) sequentially unstacks each block
arm empty and the blocks dismantled and b1 are clear with b3 on top of b2 and b2 on top of b1, and the arm is empty, bb1l 'b2'y will unstack b3 from b2 and put b3 on the
"dismantle biismantlesstak...e stack_unniil(('br table without touching b2 or b1.
construct stack(block list): This function allows the arm construct stack of blocks given list of blocks if the arm is empty and all the blocks are clear and on the table. It achieves this by iterating from the end of the list, picking each block starting from the penultimate block and stacking it on the block next to it (assuming the list arranged from bottom block to top block). After the execution of construct_stack top of each other in the order they were
all the blocks in the list will be stacked arranged in the block list, the arm will be empty and only the top block will be clear. For example, if block1, block2, and block3 are all clear and the table and the arm is empty, calling peastructsstac((ble
block2, block3]) will result in block3 being stacked on block2 and block2 on block1, and the arm will be empty.
usage example for learned actions
The goal is to satisfy the following conditions: b1 is on b2, b2 is on b3. Observation: b3 is on b2, b2 is on b1, bl is on the table. Robot arm is empty. The b3 is clear.
Action: bismantle_stack.. (lacc__util(('b3"" /b2', /b''], 'bl')
Observation: b1 is on the table. b2 is on the table. b3 is on the table. Robot arm is empty. The bl clear. The b2 clear. The b3 is clear.
Action: cennsruct_stacc([[" (b'' "/222///1]]]
Observation: bl is on b2. b2 is on b3, b3 is on the table. Robot arm is empty. The bl is clear. The goal is satisfied.
Figure 9: Prompt details of agent testing for task Blockworld.
A.6
Experiment Details
A.6.1 Baselines
We imply baseline Reflexion for our learning-testing setting, where the testing instances are unseen during learning. Original Reflexion learns policy on single instance, which can generate very detailed hints for the next turn, like "In this trial, was able to find the desklamp on desk 1, but I did not find the bowl under the desklamp. In the next trial, I will go to desk 1, turn on the desklamp, then look for the bowl under the desklamp on desk 1 or desk 2. If still cannot find the bowl, I will check the shelves and drawers." The detailed hint helps the agent perform in the same task instance, but can not transfer to different instances. We imply Reflexion to generate transferable policies based the failed trial, such as "To accomplish the task, first locate the object, then navigate towards the cleaning location, clean the object, and finally deposit it at the specified receptacle. Follow the action
19

---

Published as a conference paper at COLM 2024
def find and take
observation qoto (secep)
objjcct name parse, ob jact, naie (ob), observation)
Find the doject should explore di fffreet receptacles.
object
Hones
observation
take teb jeas nane, zмсно]
aucuzn observation.
Erro::
The action is not valid. The cabinet clased.
Failed Trail. the trial, saw a closed cabinet but not open it. Reflexion:
Update:
def find and Bidd__ta take tob1, zecep):
osservation (zecee)
12 (Crece,
flocationt
looation ["ahel Ft, 'ccuntentog! 1
Defure atiempling Lo interact with Lhe receptacle, ensure that it open oz accessible in tirst place. Volidate object's exiaterce and sccesstbl itty In the said location.
obsorvation open tzezop1
obsarvatian:
objjcct caee garse object observalion!
ceteect_naeeeeeee name tohe
if object is not Nones
narte recteg))
zeturn ocservation
Figure 10: The failed cases of LearnAct (left) and Reflexion (right). The learning overfits the current task instance with misunderstanding. Left: The agent causes error when using action Find_and take because the action does not consider the case that the cabinet closed. However, the learner updates action Find_and. take by adding conditional based on the receptacle name, which is not the correct condition to open the receptacle. Right: The agent failed in trial in which it saw closed cabinet but did not open it. The Reflexion learner advises the misleading policy "before attempting to interact with the receptacle, ensure that is open or accessible in the first place".
sequence: locate navigate clean store. Ensure recheck your inventory confirm the success of your actions.". The experiment shows that general policy can improve agent performance, as shown in Table 2 and 3.
A.6.2 Setting
We use language models GPT-4 and GPT-3.5 Turbo through interface ope- maiCCaatComlleton.croe
via Azure platform. For all the learnable models, including LearnAct, Reflexion, and Voyager, GPT-4 serves as the learner for both GPT-4 and GPT-3.5 Turbo agents. This choice is driven by the necessity for substantial model capacity to facilitate agent learning, aligning with original papers of baselines where learning is implied by strong language models.
A.6.3 Case Study for Failed Learning
Figure 3 demonstrates that excessive iterations degrade performance, likely due to overfitting and misinterpretations of the learning task. Failed cases involving LearnAct and Reflexion shown in Figure 10, in which the learner inaccurately diagnoses failure causes, resulting in misguided actions or policies that further impair performance.
20


Reviews:
[{"review_id": "HsEkqwo7TA", "reviewer": "Reviewer_X3uD", "summary": "This paper proposes a training framework that instructs LLMs to learn new actions. During the training stage, the action is created in the form of executable code function, and is refined through execution feedback. In the inference time, the LLM can utilize the created action to tackle more novel scenarios and perform more complex tasks.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the question! In our work, we have performed a comparative analysis with Voyager in both related work and experiment. We will add CRAFT in the related work in the next version. \n\nAlthough function creation has attracted significant interest as a general technique recently, prior studies such as CRAFT and Voyager primarily emphasize coding-based task resolution, acquiring tools from the task prior and optimizing them via code refinement. These approaches are not applicable to interactive decision-making language agents. Our study differs from these works by targeting the action space learning of interactive decision-making language agents. LearnAct's learning strategy consistently refines the entire action space according to the task-solving trajectories, introducing an innovative and general method for action learning in interactive decision-making language agents.\n\nIn one-turn coding tasks (LATM, CREATOR, and CRAFT), the function abstracted from one example can be easily used to tackle similar examples, Consequently, their learning approach is devised to optimize the performance of the single tool's code in the training instances through a code-refine iteration. \n\nVoyager suggests a curriculum for MineDojo that advances from elementary to complex levels within the hierarchical task framework. The solution for low-level tasks can naturally serve as a tool for higher-level tasks. The learning method also optimizes the code solution for the current task in a code-refine iteration, as Voyager resolves each task via open-loop code writing.\n\nNone of these studies can be directly applied to general interactive decision-making agents. For such tasks, there is no predefined skill structure. In LearnAct, skills are generated through task decomposition based on the language model's commonsense. Moreover, our learning-testing framework addresses the iterative refinement of action space based on historical trajectories. This poses a considerable challenge, as the learner must determine why specific actions within the trajectory are suboptimal and improve them to support the agent's accomplishment of the overarching goal.\n\nWe delineate the differences in a table:\n\n|  | Task | Tool Prior | Tool Learning |\n|---|---|---|---|\n| CRAFT,LATM | Coding | Given sample | Code revision |\n| Voyager | Coding | Given sample and Curriculum | Code revision |\n| LearnAct (Ours) | Interactive decision-making | None (Proposed by LM) | Trajactory based skill reflecion |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The idea is interesting, and can potentially solve the problem of constrained action space. \n2. The experimental results demonstrate the effectiveness of the proposed framework.", "reasons_to_reject": "Overall, the idea is intuitive and makes sense to me. But the biggest concern is that I do not find any fundamental difference and novel parts that distinguish this work from some previous publications, including but not limited to CRAFT, and Voyager. Please correct me if I have any misunderstanding. Basically, I think the fundamental part is creating some tools or actions defined in this work during the training time that are reusable, which can help LLMs to address some special test cases. Which part is novel in this work?", "questions_to_authors": "None", "extra_scores": {}}, {"review_id": "pLeIccbi3z", "reviewer": "Reviewer_vBTB", "summary": "This paper is about improving the performance of language agents by refining the action space during the learning phase. Specifically, the authors propose LearnAct, an iterative system that uses LLMs to create new (or edit existing) actions based on past experiences and to act as a policy. In this work, the actions are expressed as Python code which enables the use of conditional statements, loops, and function calls (i.e., composition of actions). Experiments were conducted on robotic planning tasks and ALFWorld (a text-based game version of ALFRED, a 3D embodied environment). Empirically, the authors show that LearnAct achieves better performance on average compared to some relevant baselines (ReAct, Reflexion, CodeAsPolicy, Voyager) on those environments.\n\nEdit: I have increased my score to 7.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your questions and suggestions!\n\nRelated work: In the next version, we'll include TaskWeaver, ReGAL, and other recent works.\n\nLimitations: Other potential factors include the learner's capacity, the constraint on the number of samples, and so forth. We'll add a section on limitations in the next version.\n\n Closed-loop Definition: We use \"closed-loop\" to differentiate CodeAsPolicy from alternatives, as it generates the entire code without agent feedback processing. Our method focuses on multi-turn agents capable of accessing environmental feedback after executing a action (not mean the action to process feedback). Indeed, Voyager's original method does not constitute an Act-type agent, as it generates a one-time solution to the task while iterating on feedback to refine the code outcome. We will modify the term for clarity.\n\nSampling Temperature: We conducted experiments with different sampling temperatures and found that a temperature of 0 still produces varied skill propositions due to GPT's randomness. The ablation study of sampling supports this. We present the temperature's impact as follows:\n\n|  | Robotic Planning | AlfWorld |\n|---|---|---|\n| t=0, K=4 | 82.8(5.2) | 72.2(4.4) |\n| t=0.7, K=4 | 78.9(7.8) | 72.7(7.4) |\n| t=0, K=8 | 82.6(5.5) | 73.5(4.1) |\n| t=0.7, K=8 | 83.1(5.8) | 73.9(6.5) |\n\nK is sampling number. The number in () represents the std. For K=4, higher temperatures don't significantly improve performance but increase variance. To capitalize on sampling variance, a larger sampling number is more advantageous. The results indicate that a greater sampling temperature necessitates an increased sampling size to attain improved and consistent performance.\n\nDomain Generalization: This is an interesting idea! Such a setting would be more common in areas like OS operation and Sheet Operation, where various tasks share a common fundamental interface.\n\nOpen-source Models: We perform experiments using Mistral-7b and DeepSeek-67b as backbone models. The results for Robotic Planning are as follows:\n\n|  | Mistral-7b | DeepSeek-67b | GPT-3.5 | GPT-4 |\n|---|---|---|---|---|\n| ReAct+Reflection | 3.5 | 8.7 | 7.6 | 66.7 |\n| LearnAct | 5.1 | 20.8 | 23.2 | 82.8 |\n\nFor powerful open-source LLMs like DeepSeek-67b, LearnAct can enhance performance. For weaker models, task completion remains challenging. Note that action learning iteration for Mistral-7b is difficult as the trajectory offers limited information for the learner to update actions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Letting the the action space evolve based on past experiences is a nice idea. This is a good way to get actions that operated at different level of abstraction, i.e. hierarchical planning.\n- Having higher-level actions (expressed as code) help reducing the compute cost during inference. That is, you don't have to call the LLMs at each timestep.\n- Transferability of the learned actions to new tasks from a same domain. I think this is a very important aspect of the work, as many prior works focuses on single-problem solving.", "reasons_to_reject": "- The research on Language Agents is evolving very fast and there is probably several prior works missing from the related work section. On top of my head, two came in mind: [TaskWeaver: A Code-First Agent Framework](https://arxiv.org/abs/2311.17541) and [ReGAL: Refactoring Programs to Discover Generalizable Abstractions](https://arxiv.org/abs/2401.16467). I think it would be beneficial to the reader to have a more comprehensive related work section to better position their work.\n\n- This is a smaller concern. I noticed a small paragraph in A.6.3 about failure mode of the proposed technique regarding excessive iterations (which according to Fig.3 anything above 2 iterations looks worst!). Is this that the only failure? I doubt it, e.g., limit of the context length in ActionLearn to update the full action space at each iteration. I think having a proper discussion of the limitation as part of the main text would strengthen the paper.", "questions_to_authors": "- What is the definition of closed-loop in the context of this work? Is it the action learning process that is closed-loop or the actions (i.e., Python functions) themselves? If you are referring to the actions, have you checked if they are really closed-loop? In my experience Voyager tends to generate actions that are open-loop, i.e., they ignore the state of the environment as they send command to the environment. Granted this might be less problematic in your environments compared to Minecraft where other entities are moving in parallel.\n- In section 5.3, it says the **temperature** is 0, but it also mentions sampling number is 4. I'd expect to get very similar samples (if not all the same). Can the authors clarify this?\n- Do the authors have any insights on the generalization of the learned actions to unseen domains but that are sharing the same predefined actions?\n- I'm curious to know if the authors have considered any open-source models during their experimentation?", "extra_scores": {}}, {"review_id": "mfcxtKD2eA", "reviewer": "Reviewer_ADgy", "summary": "The authors present alba approach for LLM agents to learn new actions and thereby improve the ability to successfully perform tasks in a simulation. The presented idea to learn new actions formalized as python code and how to integrate this new ablities into the policy of the agent.\nThe approach is overall presented clearly, however some details could be explained in more detail.\nThe approach is very well evaluated on two different tasks (Robotic Planning and AlFWorld). Thereby, the authors compare the method to several related work and the presented results are promising.\nThe authors missed a detailed analyze of the influence of the LLM. The LLM is only treated as a black box. The influence of the quality of the LLM is only analyzed very briefly and the authors do not evaluate any open source LLM to perform a more detailed analyze", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your questions and suggestions!\n\nInfluence of LLM: We perform experiments using Mistral-7b and DeepSeek-67b as backbone models. The results for Robotic Planning are as follows:\n\n|  | Mistral-7b | DeepSeek-67b | GPT-3.5 | GPT-4 |\n|---|---|---|---|---|\n| ReAct+Reflection | 3.5 | 8.7 | 7.6 | 66.7 |\n| LearnAct | 5.1 | 20.8 | 23.2 | 82.8 |\n\nFor powerful open-source LLMs like DeepSeek-67b, LearnAct can enhance performance. For weaker models, task completion remains challenging. Note that action learning iteration for Mistral-7b is difficult as the trajectory offers limited information for the learner to update actions.\n\n\nBefore learning in Table 5: Our apologies for any ambiguity. The terms \"init\" and \"after learning\" refer to the scenario in which actions and policies are initially generated through the ActionCreation process and subsequently undergo iterative refinement via ActionLearn. To enhance clarity, we will modify the column names to \"init\" and \"after updating.\" Prior to the update, the actions function in an identical manner to their post-update counterparts, as ActionLearn merely serves to optimize the action and instruction. As depicted in Figure 1 (right) and Figure 8 in the Appendix, the instructions for the generated actions are provided to the Act agent similarly to predefined actions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Good motivated approach to improve the performance of LLM Agents\nDetailed analysis of the presented approach\nPromising results", "reasons_to_reject": "The influence of the LLM on the overall performance is only investigated very briefly.", "questions_to_authors": "Tabel 5: How can the model use learn actions before learning?", "extra_scores": {}}, {"review_id": "IpNDLJi3yj", "reviewer": "Reviewer_n7eb", "summary": "This research paper introduces an innovative framework named LearnAct, designed to enhance the capabilities of LLM agents through action learning. The study tackles the limitations of fixed action spaces in LLM agents by allowing these agents to learn and refine their actions based on trial and error, a process akin to human experiential learning. This is achieved through an iterative learning strategy where actions are continuously updated based on the outcomes of previous tasks. The authors demonstrate the effectiveness of this approach in the Robotic Planning and Alfworld environments, showing significant improvements in task performance.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your questions and suggestions!\n\n**Comparative Analysis**: In Experiment, CodeAsPolicy and Voyager are baselines without iterative learning strategies. The Ablation Study also shows LearnAct's performance without iterative learning.\n\n\n**Baseline Selection**: Thanks for the baseline suggestion! FireAct and KnowAgent are based on fine-tuning, distinguishing them from the prompting-based learning approach in this study, which may lead to better performance. We'll revise our statement and include these works in the related section.\n\n\n**Flexibility**: This is a good question! As outlined in the introduction, our hypothesis is that for most multi-turn agent tasks, abstracting higher-level actions is reasonable, consistent with hierarchical reinforcement learning's core assumption. Theoretically, frequent patterns in trajectory can be abstracted as higher-level skills, emphasizing their semantic meaning, especially for complex tasks with extended action sequences.\n\nTechnically, besides semantic judgment, we can implement action learning and evaluate performance to determine if action space learning is necessary. Improved performance indicates the need for action space expansion; otherwise, the existing action space may be adequate, which we believe is rare.\n\n\n**Generalizability**: Our framework targets general multi-turn decision-making agents, with the agent's form Act being versatile for various tasks. The learning method is generalizable to many tasks like OS operation, Sheet operation, and tool usage scenarios where actions can be defined as code.\n\n\n**Scalability Concerns**: The Appendix shows LearnAct's testing phase time complexity adheres to the standard language agent setting, dependent on prompt size O(N), where N is the number of learned actions. The learning phase time complexity is O(MKI), with M being the number of training instances, K the sampling number, and I the maximum iteration step. \n\nLearnAct suits most current language agent task scales. For extremely complex tasks, a retriever-based skill selector might be needed due to the vast array of potential skills. We will add this discussion in the Appendix.\n\n\n**Overfitting Measures**: We mitigate overfitting by requiring action generation and updating instructions \"All the values used in the function should be given as input rather than fixed in the function.\", preventing overfitting to training instances. We also limit training iteration to avoid overfitting.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. **Innovative Approach:** The concept of open-action learning to dynamically expand and refine actions for LLM agents is novel and addresses a critical limitation in the current use of LLM agents in various applications.\n2. **Thorough Experimentation:** The authors provide extensive experimental results to validate their claims. The use of multiple environments for testing and the iterative approach to learning and refining actions are well-executed and robustly documented. Although the baseline selection is not sound enough. \n3. **Potential for Broader Implications:** The study’s findings could have significant implications for the development of more autonomous and efficient LLM agents across a wide range of applications, potentially leading to broader advancements in artificial intelligence.", "reasons_to_reject": "1. **Lack of Comparative Analysis:** While the paper shows that LearnAct outperforms certain existing methods, there is a lack of comprehensive comparative analysis with a wider range of existing solutions (especially SOTA and more up-to-date methods), which could provide a better understanding of LearnAct’s positioning in the field.\n2. **Complexity and Replicability:** The complexity of the LearnAct framework and its dependency on specific environments and configurations may limit its replicability and broader application. The paper could benefit from a clearer explanation of how LearnAct can be adapted or generalized for other contexts.\n3. **Potential Overfitting:** There is a concern about the potential for overfitting in the iterative learning process, especially given the significant improvements reported. The paper could be strengthened by including a discussion on measures taken to avoid overfitting and ensure the generalizability of the learned actions.", "questions_to_authors": "1. **Comparative Analysis:** Could the authors provide a more detailed comparative analysis with other methods in action learning for LLM agents? Specifically, how does LearnAct perform compared to methods that do not use iterative learning strategies? \n2. **Baseline Selection:** The authors claimed that LearnAct achieves more effective and efficient learning compared to state-of-the-art methods, but the baselines are not state-of-the-art and up-to-date enough. In fact LearnAct performs worse than some up-to-date models, such as FireAct [6] and KnowAgent [7]. \n3. **Flexibility:** Is it necessary to “Expanding Action Space by Action Creation” first? If not, how can you adaptively decide to expand the action space or not? If so, could you explain why action space expansion is a necessity, as the current action space is sufficient enough in many cases. \n4. **Generalizability:** Can the authors elaborate on how the LearnAct framework can be adapted for use in environments or tasks significantly different from those tested (e.g., non-robotic applications)?\n5. **Scalability Concerns:** How does the LearnAct framework scale with increasingly complex tasks or larger environments? Are there computational constraints that could limit its practical applicability?\n6. **Overfitting Measures:** What measures have been implemented in the LearnAct framework to prevent overfitting during the iterative learning process? How do you ensure that the actions learned are not overly tailored to the specific training examples used?", "extra_scores": {"ethics_concerns_details": "The newly-generated actions may require ethics check"}}]

Author rebuttal:

Meta-review: This paper describes a structured prompting method for using LMs to solve interactive tasks by writing (and iteratively refining) code fragments implementing high-level actions. It outperforms a variety of other prompting methods on two simulated interactive tasks. As with  ReAct, Voyager, and other work in this space, it's a little unclear whether there's a generalizable scientific finding here, or just a well-designed collection of prompts and some code gluing them together. But empirical results are good, the engaged reviewers appear enthusiastic, and so the paper seems likely to be a useful contribution in this sub-area.

[At least one review was discounted during the decision process due to quality]
Decision: Accept