Empowering Large Language Model Agents through Action Learning

Authors: haiteng  zhao
chang  ma
guoyin  wang
jing  su
lingpeng  kong
jingjing  xu
zhi-hong  deng
hongxia  yang
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-21 12:39:41

Abstract: Large Language Model (LLM) Agents have recently garnered increasing interest yet they are limited in their ability to learn from trial and error, a key element of intelligent behavior. In this work, we argue that the capacity to learn new actions from experience is fundamental to the advancement of learning in LLM agents. While humans naturally expand their action spaces and develop skills through experiential learning, LLM agents typically operate within fixed action spaces, limiting their potential for growth. To address these challenges, our study explores open-action learning for language agents. We introduce a framework LearnAct with an iterative learning strategy to create and improve actions in the form of Python functions. In each iteration, LLM revises and updates the currently available actions based on the errors identified in unsuccessful training tasks, thereby enhancing action effectiveness. Our experimental evaluations across Robotic Planning and Alfworld environments reveal that after learning on a few training task instances, our approach to open-action learning markedly improves agent performance for the type of task (by 32 percent in AlfWorld compared to ReAct+Reflexion, for instance) highlighting the importance of experiential action learning in the development of more intelligent LLM agents.

Main paper: 
 introduction: 
 Language agents, which employ the large language model (LLM) as the policy model to control agents to iteratively take actions and interact with environments, have recently garnered increasing interest (Yao et al., 2023a; Brohan et al., 2023; Wang et al., 2023a; Xie et al., 2023; Song et al., 2023; Xu et al., 2023) . The underlying reason is that LLMs offer a fresh angle to address the commonsense issue that is difficult to tackle in the reinforcement learning paradigm which learns the agent policy solely from trial and error. Reasoning and planning of agents are often derived from prior knowledge in the particular environment, precisely where LLMs excel.Researchers increasingly recognize that, while leveraging LLMs for agent control shows promise, it is far from perfect due to their limited ability to learn from experience (Yao et al., 2023b; Shinn et al., 2023; Huang et al., 2023) . The substantial scale of LLMs makes direct policy model finetuning impractical. Instead, LLMs depend on incorporating historical interactions into prompts to leverage past experiences for future action planning (Yao et al., 2023a; Shinn et al., 2023; Sun et al., 2023; Madaan et al., 2023) . However, these approaches are often constrained in their capacity to learn from long-term experiences and typically draw experience on single-instance (Wang et al., 2023c; Ma et al., 2024; Wang et al., 2023a) .While humans naturally expand their action spaces and develop skills through experiential learning, LLM agents typically operate within predetermined action spaces (Qin et al., 2023; Schick et al., 2023) , limiting their potential for growth. In this work, we propose a novel learning paradigm for LLM agents that focuses on learning to expand and iteratively refine the action space, thereby aligning tasks more closely with the agents' planning abilities. By adapting the action space to fit the LLM's planning, we address the limitations imposed by fixed action spaces, such as the misalignment between commonsense knowledge-guided planning and actions, and the prevalence of action errors due to unmet prerequisites or ineffective strategies (Gu et al., 2022; Ma et al., 2024; Ahn et al., 2022) . This approach not only mitigates bottlenecks in language agent performance but also allows transferring experience across different tasks.For a more illustrative view, we could look at the example where an LLM agent is asked to make cocktails. With a learnable and open action space, LLMs may naturally instruct the agent to gather all the components in a specific order and mix them, while the closed action space may only involve basic handling, such as moving to different places and grabbing bottles, which greatly increases the difficulty of the task for LLM agent. On the other hand, it is necessary to update existing actions to accommodate changing factors in the environment. Still, in the cocktail example, when making an Old Fashioned cocktail, muddling sugar is often the standard practice. However, if only syrup is available, the agent should adapt its pre-set actions to prevent repeated failures despite being competent to accomplish the task.To solve these issues, we conduct an extensive exploration of action learning in LLM agents. We introduce a framework LearnAct, designed to generate and optimize new action types as APIs dynamically. The newly generated action types are in the form of Python functions, leveraging the LLM's extensive prior knowledge and code generation ability to devise a diverse and representative action space. Furthermore, LearnAct stands out due to its iterative learning strategy, which continuously refines actions through a feedback loop. In each cycle, the LLM evaluates the effectiveness of current actions using training examples, identifying and rectifying errors in failed instances. The learning strategy progressively deepens task understanding and improves learnable actions.Our experimental results demonstrate that this method of iterative refinement not only creates complex and user-friendly action types but also achieves more effective and efficient learning compared to previous state-of-the-art methods such as Reflexion (Shinn et al., 2023) . By learning on a few problem instances, LearnAct can generalize to a general type of task with strong performance. In summary, our contributions are as follows.• We propose the action learning framework for interactive decision making language model agents by generating and updating learnable actions, enabling learning customized action space that better fits the LLM's planning capacity.and updates the currently available actions based on errors in failed tasks.• Our experimental results demonstrate that LearnAct effectively learns action spaces within a few trials, acquiring transferable capabilities in diverse environments. Through action learning, LearnAct outperforms SOTA agents by a significant margin, showcasing the potential of action learning for LLM agents. 
related work: 
 LLM Agent Learning Recent research has advanced the use of large language models (LLMs) in embodied agents (Duan et al., 2022; Huang et al., 2022a; Ahn et al., 2022; Huang et al., 2022b; Yao et al., 2023a; Park et al., 2023; Wu et al., 2023; Sun et al., 2023; Ma et al., 2024; Yuan et al., 2023; Qiao et al., 2023; Stengel-Eskin et al., 2024) . These methods differ from previous reasoning-centered methods like Chain-of-Thought (Wei et al., 2022; Chen et al., 2022; Liang et al., 2023; Wang et al., 2023b) as they iteratively incorporate environment feedback to modify subsequent plans. This closed-loop process is centered on the ability of LLM agents to learn from environment.The majority of previous work on multi-turn reflection concentrates on learning from environment feedback within the same problem (Lai et al., 2023; Le et al., 2022; Chen et al., 2023; Liu & Abbeel, 2023; Singh et al., 2023) . ReAct (Yao et al., 2023a) propose a basic framework for using thought to enforce the LLM reflect on previous behaviors within the same trial. Reflexion (Shinn et al., 2023; Park et al., 2023) , on the other hand, uses a multi-trial approach to perform reflection. By summarizing historical experiences and identifying reasons for failure, LLM is prompted to generate insights for more effective agent instruction. Similarly, ExpeL (Zhao et al., 2023) facilitates a general learning system that extracts insights and past experiences with retrieval. Retroformer (Yao et al., 2023b) propels introduce prompt-based training to enhance LLM reflections.Our method differs from this line of work in two points (1) LearnAct performs direct learning in the action space, which substantially improves the reliability and utility of generated actions.(2) LearnAct does not target a single problem instance but learns experience from a few training instances and is tested on a general type of task. A more detailed comparison is provided in Table 1 .LearnAct generates new action types based on atomic actions to enable more informative and applicable actions, similar to the approach in hierarchical reinforcement learning (Erol et al., 1994; Kulkarni et al., 2016; Bacon et al., 2017) , where a high-level executor plan with high-level action types, and then executes the seed actions according to the high-level plan (Yao et al., 2023a; Sharma et al., 2022; Wang et al., 2023a; Sun et al., 2023) . Our approach also differs from hierarchical reinforcement learning by not only using code-based action space, providing stronger capacity due to the code statements such as conditions and loops (Cai et al., 2023; Qian et al., 2023) , but also avoiding the typical imposition of a strict two-level structure, using flexible mixed-level actions instead.  
problem statement: 
 The task of a language agent can be succinctly modeled as a Partially Observable Markov Decision Process (POMDP), which is defined by a tuple A 1 , π A 1 = ActionCreation(π 0 , F) for i = 1 to maxiter do results = SolveProblem(π 0 + π A i , E m , A 0 ∪ A i , G), m = 1 to M s 1:T , a 1:T , r = SelectErrorCase(results) A i+1 , π A i+1 = ActionLearn(A i , s 1:T , a 1:T , r, F) end for Return A last , π A lastthe set of all possible states, O being the observation space through which the agent perceives the state, A denoting the action space, T : S × A → S being the state transition function, and R : S × A → {0, 1} being the reward function.At step t, the language agent G takes an action step a t based on policy π (policy prompt), problem E (problem instructions), and previous observation-action trajectoryh t = [o 1 , a 1 , . . . , o t-1 , a t-1 , o t ], a t = G(π, E , h t ) (1)π, E , and h t correspond to the instruction, goal, and history in Figure 1 (right), respectively. Action a t is successfully grounded (executable) only if it is within the action space, i.e. a t ∈ A. Even successfully grounded action may be invalid due to unmet conditions or has no effect in the current state. The agent aims to maximize the final reward r. In this work, we use an outcome-based reward mechanism (ORM) (Uesato et al., 2022) that assigns a binary indicator as a reward, depending on whether the task has been successfully completed.In this work, we try to answer the question How to learn from experience and apply them to other decision-making problems? We focus on the training-testing setting, where for each task, our LLM agent optimizes its action space and corresponding policy on the training set D train = {E 1 , . . . , E M }, before trying to accomplish problems in the test set D test = {E M+1 , . . . , E M+N }. These problems share the same original action spaces and rules, though requiring varied policies to solve them as they have different scenarios and goals. This setting poses a challenge for the agent to develop a general capacity through learning to accumulate experience and accomplish tasks of this general type.To address this problem, we introduce an action-learning framework designed for language agents, enabling them to autonomously enhance their skills by learning from interactions within their environment, as shown in Figure 1 . The framework considers the action space, denoted as A, to be an open set capable of learning and expanding. More specifically, we define the original action space as A 0 and augment it with newly learned action type (API) A ′ , represented as A 0 ∪ A ′ . Also, instruction for using these new actions π A ′ is updated to the original policy instructions π 0 , i.e. the new policy is denoted as π 0 + π A ′ . Consequently, subsequent actions by this LLM agent are characterized asa t = G(π 0 + π A ′ , E , h t ), a t ∈ A 0 ∪ A ′ , as depicted in Figure 1 (right), where the agent is guided to generate both origin and learned actions.In practice, the action learner could generate low-quality actions that can greatly affect the efficiency of the overall optimization process. Also, some action revisions may be spurious. Thus to enhance the stability of the learning process and improve the quality of each step, we sample K times with ActionLearn, yielding K revised results {A k i , π k A i } k=1,...,K . During the learning iteration, each action-policy pair is evaluated on the training set, and the best one is chosen, as shown in Figure 2 (left) and the detailed algorithm in Appendix.We select the best action based on the results obtained with the agent. The score µ for a sampleA k i , π k A iis as follows:EQUATIONHere, p succ denotes the success rate of the agent in training instances, and p stepacc refers to the ratio of successfully executed actions to the total number of steps taken by the agent, indicating the practicality of generated actions. Based on these selection criteria, we identify the most beneficial and feasible action sample A i , π A i . 
method: 
 The overall pipeline of LearnAct is illustrated in Figure 1 and Algorithm 1. The training stage of LearnAct involves first creating new actions and then refine them based on the error feedback on training samples. After learning the action space and policy instructions in the training stage, the refined agent perform actions from the augmented action space and try to accomplish the problem step by step. The training stage is specified in §4.1, and test stage is detailed in §4.2.1. Expanding Action Space by Action Creation. Our method first creates custom action types, akin to API, to enable LLMs to interact with the environment more seamlessly, denoted as ActionCreation. These newly crafted action types are implemented as code functions, thereby unlocking the potential for more intricate logical expressions that leverage the foundational APIs through constructs such as conditional statements (if-else), loops, and assertions. ActionCreation involves two steps:(1) Generating Action Function A ′ : Upon receiving detailed instructions and problem specifics, the LLM, denoted as F, is prompted to summarize high-level actions to complete this task in the form of Python functions for agent use. The detailed prompt is in the Appendix. These functions can call multiple basic or defined actions to complete subtasks. After generation, functions are parsed and added to the action space. See detailed prompt in Appendix.(2) Generating Policy Instruction π A ′ for Using New Action: After generating new action functions, we then update the agent with information about their potential applications. As the basic instruction contains action descriptions and usage examples, the updated policy guidance includes both a description of each function and a usage example.First, the LLM F is prompted to provide a comprehensive description of each new function.The purpose of this description is to offer the agent an overview of the anticipated outcomes and necessary conditions, along with the input-output format of the new action. Second, the language model generates illustrative usage examples for the new functions. This is crucial for guiding LLM to use the new action in the appropriate scenario, as indicated by previous work (Schick et al., 2023) . Prompts are in Appendix.We denote this process by the ActionCreation:EQUATION2. Learning Actions Based on Error Feedback After creating the initial set of actions, there's a possibility that these actions may have errors, misunderstand the task, overlook certain task scenarios, or possibility for misuse. To address this, we devise the training phase that allows the language model to learn through trial and error, as shown in Figure 2 . In this phase, the agent is executed in the training instances, identifies action failures, and then iteratively updates the action set, until successfully solving all the instances with no action errors or exceeding maximum optimization steps.During the learning process, the agent first tries to solve problems in the training set with the current available action space A and policy instruction π A , denoted as the SolveProblem process. Then a failed problem with action error is sampled with SelectErrorCase operator and then the ActionLearn either fixes it by revising the used action or writing note as an annotation. We present each of them below:(1) SolveProblem: The assessment of current policy and action space is done by trying to solve problems in the training set. We follow the simple Act (Yao et al., 2023a) agent and prompt the agent to solve the task by interacting with the environment and generating actions based on history.(2) SelectErrorCase: This step identifies error-inducing steps in action sequences based on the environment's feedback, such as invalid actions due to unmet preconditions, ineffective actions, or errors in action names or parameters. For learnable actions composed of multiple atomic steps, it evaluates errors in each atomic step execution.Find Failed Case (3) ActionLearn: We address errors by implementing either function updates or writing notes. Function updates refines the Python function to correct the misunderstanding and overlooking of tasks, whereas writing notes entails enhancing the function's description to guide the agent toward more accurate use. The two options are both available and LLM F is free to choose one of them, as shown in Figure 2 . When function updates are made, the corresponding action instructions π A i are also updated later to adapt to the new actions.Although function updating is triggered for a specific action function, it is not restricted to modifying just this single action. By accessing all action functions via prompt, LLM could choose by itself the actions to revise and could also incorporate new actions. Consequently, ActionLearn can freely update the entire action space in each iteration. The detailed prompt is in the Appendix.After completing the learning phase, the agent possesses an updated action space and refined policy instructions. In the testing phase, the agent attempts to solve problems in the same procedure as the SolveProblem process used during training. Notably, whereas most prior research (Sun et al., 2023; Shinn et al., 2023) has concentrated on the learning loop within a single problem instance, LearnAct is designed to facilitate the transfer to this general type. 
tasks: 
 Robotic Planning (Ma et al., 2024) includes four challenging Robotic Planning tasks, namely Gripper, Blockworld, Barman, and Tyreworld. We followed the environment implementation of AgentBoard (Ma et al., 2024) and LLM+P (Liu et al., 2023) . These tasks involve long-horizon robot planning problems, such as creating cocktails based on customer orders using available ingredients and containers, moving objects between different rooms using grippers, or rearranging piles of blocks to achieve a specified target configuration. These tasks put a significant demand on the agent's long-term planning capabilities.AlfWorld (Shridhar et al., 2020) simulates six types of objectives within a household. For instance, agents are required to locate an apple within the house, heat it, and then place it in a target area. These tasks push the agent to explore the house systematically, given that the complete state of the environment is unknown to the agent.Act & ReAct (Yao et al., 2023a) : The basic agent, Act, is prompted to iteratively take actions and obtain observations in the environment. Based on Act, the ReAct agent employs "Think" as an additional action.Reflexion (Shinn et al., 2023) : Reflexion is a learning method for language agents that utilize language as a learned policy. The original Reflexion is designed for learning individual policies in each instance. In our study, we adapt Reflexion for the training-testing setting, where policies are designed to be not instance-specific, facilitating transferability. Details are in the Appendix.CodeAsPolicy (Liang et al., 2023) : The method utilizes code to generate the entire solution for the task in an open-loop way. As the method does not interact with environments, we only report the results of CodeAsPolicy on Robotic policy tasks, because the ALfWorld task necessitates exploration in the environment to obtain information, and CodeAsPolicy cannot generate a code solution without access to the complete environmental information.Voyager (Wang et al., 2023a) : Voyager proposes functions as actions to solve tasks in a closed-loop agent manner. The original Voyager is specifically designed for Minedojo (Fan et al., 2022) , emphasizing skill acquisition through a structured hierarchical task curriculum.For comparison, we reimplement it as Wong et al. (2023) , which creates skills via code with basic verification.We test on both GPT-4 and GPT-3.5 Turbo models as the LLM during testing. We employ GPT-4 throughout our learning for action creation and improvement, as well as the learning of Reflexion and Voyager. We set the temperature as 0.0 for consistency and reproducibility in our experiments. The sampling number in our method is set to 4. For each task, we randomly select 3 instances for the training set, with the remaining instances used for testing. We report test set results for our LearnAct and all baseline models. The primary evaluation metric was the task success rate. Each experiment is conducted three times per task, and we present the average results. We ensure a fair comparison by providing a single in-context example to LearnAct and all baselines. CodeAsPolicy and Voyager's action creation utilize the identical code example as ours. The frequency of use and accuracy of learned actions before and after learning. Post-learning, there is a marked increase in the action usage frequency and accuracy, indicating their enhanced reliability and utility for the agent. Right: The performance with different maximum learning iteration steps. The performance varies with different maximum learning iteration steps. LearnAct's performance notably improves with the application of learning, particularly at step two. Although ReAct+Reflexion also shows improvement, its progress is less significant and stable.We present a comparison of LearnAct with baseline models in Tables 2 and 3 . LearnAct outperforms the baselines in various tasks, utilizing both GPT-3.5 Turbo and GPT-4 as backbones. We further analyze the performance and discuss the conclusions drawn from the comparison.A well-designed action space enhances the language agent's ability to plan and solve tasks. LearnAct markedly surpasses the Act baseline in performance, highlighting the crucial role of the action space. It is noteworthy that, after its learning phase, LearnAct is also an Act type agent except with a developed action space. This supports our hypothesis that the limitation of language agents lies in the effective integration of the planning process with the action space, and an adaptable action space can substantially unlock the latent planning capabilities of LLM. Additionally, LearnAct surpasses ReAct in both Robotic Planning and AlfWorld, demonstrating that while "Think" improves planning, it suffers from ill alignment with the action space.Action learning outperforms the verbal policy learning. In the realms of Robotic Planning and AlfWorld, LearnAct markedly outperforms both Act+Reflexion and ReAct+Reflexion. Reflexion writes policy prompts during the learning process to assist the agent in tasksolving. This approach does yield improvements over the Act and ReAct methods; however, these enhancements are considerably less pronounced than those achieved by LearnAct, which focuses on learning within the action space. While learned verbal policies and prompts can aid in enhancing the agent's planning capabilities, possessing proficient skills is essential for the agent to fully realize its potential.Action learning is important for the action quality. LearnAct outperforms coding baselines such as CodeAsPolicy and Voyager. CodeAsPolicy directly generates code in open loop without interacting with the environment, leading to inferior performance compared to closed-loop agent baselines. Notably, while Voyager generates actions using a method similar to LearnAct, it lacks the component of iterative action learning. This highlights the significance of action learning. Without this learning phase, the created actions are constrained by insufficient understanding of the task.Learning enhances performance through action reliability and utility. We further analyze action utilization to explain why action learning enhances agent performance. We assess the frequency of use and accuracy at both the initial stage of learning and the post-learning. These results are presented in Figure 3 (left). It is evident that, compared to the initial action values, the actions post-learning demonstrate significantly higher correctness, indicating their enhanced reliability and utility for the agent. This is a direct consequence of our learning method, which focuses on correcting errors encountered during usage. Additionally, it is observable that the rate of action usage also improves after learning. It is possible because the agent is more inclined to employ these actions upon recognizing their usefulness, i.e., observing valid effects after calling the action. We analyze the learning choices and learned action numbers. The strategies for learning encompass function updates and writing notes. We find that the model prefers to update functions. Table 4 shows function updating occurs for about 90 % in the learning. This may be due to the model's ability to circumvent most invalid executions through code adjustments. Additionally, the learning process can lead to the creation of new functions. As shown in Table 5 , the average number of learned actions ranges from 3 to 4 for both Robotic Planning and AlfWorld, and the number of actions tends to increase after learning, showing that new actions could be innovated while updating existing ones.The iteration number influences the performance of the model. We illustrate the impact of learning iterations on LearnAct and ReAct+Reflexion in Figure 3 (right). It is evident that in both AlfWorld and Robotic Planning tasks, LearnAct's learning markedly enhances performance, achieving optimal results within just two iterations. The enhancement from our learning approach is notably more substantial than that observed with Reflexion. While ReAct+Reflexion also shows performance gains, its impact is less consistent and displays more negative effects.Why does prolonged learning with LearnAct detrimentally affect performance? We observe that as it cannot be avoided for agents to make mistakes, excessive optimization for these mistakes can lead to overfitting to specific training cases and misunderstanding the task rule. See failed cases in Appendix.We conducted ablation experiments to demonstrate the significance of various components in our method. First, we evaluated the action format of our method, including descriptions and usage examples of actions. The results of employing GPT-4 as the agent in Robotic Planning and Alfworld are presented in Table 6 . It was observed that both the description and usage examples contribute to the performance, with the usage examples having a more pronounced impact compared to the descriptions. Interestingly, we discovered that the usage examples are often incorrect. Despite this, they still encourage the agent to utilize the learned actions. The descriptive aspect also plays a crucial role in Robotic Planning tasks. This could be attributed to that the actions have many pre-conditions, so the descriptions of the learned actions aid the agent in their correct usage. Table 7 : The ablation study of the learning method, including the options during learning and the sampling method.Next, we examine the components of our learning algorithm, which includes two types of learning: updating function and writing notes, as well as sampling during the learning process. The results are presented in Table 7 . Function updating plays a crucial role in action learning, and writing notes for actions further enhances performance. We observed that function updating occurs approximately 90% times during learning, suggesting that the learner predominantly opts to modify functions to improve actions rather than merely adjusting notations for more accurate action use.Moreover, sampling is vital for the effectiveness of the learning process. In the absence of sampling, the learner generates only a single action proposal for the next iteration, which is highly prone to errors and can negatively impact overall performance. However, note that sampling alone (w.o. learning) is ineffective in learning actions as LearnAct does, highlighting the importance of our learning algorithm. 
conclusion: 
 In conclusion, our research advances LLM agents by equipping them with the ability to learn and refine actions through direct interaction with the environment. Our proposed framework LearnAct demonstrates a significant improvement in agent performance by enabling open-action learning, which aligns closely with how humans acquire and enhance skills. The empirical success of our methods in Robotic Planning and Alfworld environments underscores the potential of action learning in developing more intelligent and capable LLM agents.where Z r max = E p(A,π A ) E p A,π A (s 1:T ,a 1:T ) p(r = r max |s 1:T ) is the normalization factor. In conventional reinforcement learning, A is predetermined and remains constant, while the model only learns π A through maximum likelihood rather than posterior estimation. This is attributed to the complexity of computing the posterior, which is due to the necessity of domain knowledge for the prior p(A, π A ) and the intractable calculation of the posterior.However, leveraging the advanced capabilities of language models, we propose employing them to estimate the posterior of the action space and policy toward the open action space learning. The prior p(A, π A ) can be defined as the operator ActionCreation, and the posterior updating can be inferred as ActionLearn. 
 
 Appendix: We present the prompt used in our experiment here, due to space limitations of the main text.We first present the prompts for action learning. The ActionCreation process entails two phases: initially, function generation is guided by Prompt 4, followed by the crafting of function descriptions and usage instructions, as directed by Prompt 5 and 6, respectively.During the action learning iterations, ActionLearn leverages Prompt 7 to derive results.If the LLM suggests updates to action functions, descriptions and usage instructions are regenerated using the Prompt 5 and 6.Please propose several high-level steps for this task.Each high-level step should be a Python function encompassing multiple (at least two) basic actions. All the values used in the function should be given as input rather than fixed in the function. We illustrate the learned actions and corresponding instructions in Figure 9 . The {learned action instructions} and {usage example for learned actions} are used in Prompt 8 to inform the agent model about how to use the learned action function. learned action instructions dismantle stack until(block list, block target): Allows the arm to dismantle a stack of blocks one by one, stopping when it reaches a specific target block. 'dismantle stack until(block list, block target)' sequentially unstacks each block from the block list starting from the top and places it on the table until it reaches the target block. The blocks in the list must be clear and stacked consecutively. If the top block from the list matches the target block, the function ends, leaving the arm empty and the blocks dismantled on the table. For example, if blocks b3, b2, and b1 are clear with b3 on top of b2 and b2 on top of b1, and the arm is empty, 'dismantle stack until (['b3','b2','b1'] , 'b2')' will unstack b3 from b2 and put b3 on the table without touching b2 or b1. construct stack(block list): This function allows the arm to construct a stack of blocks given a list of blocks if the arm is empty and all the blocks are clear and on the table. It achieves this by iterating from the end of the list, picking each block starting from the penultimate block and stacking it on the block next to it (assuming the list is arranged from bottom block to top block). After the execution of construct stack, all the blocks in the list will be stacked on top of each other in the order they were arranged in the block list, the arm will be empty and only the top block will be clear. For example, if block1, block2, and block3 are all clear and on the table and the arm is empty, calling construct stack ([block1, block2, block3] ) will result in block3 being stacked on block2 and block2 on block1, and the arm will be empty.The goal is to satisfy the following conditions: b1 is on b2, b2 is on b3. Observation: b3 is on b2, b2 is on b1, b1 is on the table. Robot arm is empty. The b3 is clear. Action: dismantle stack until (['b3','b2','b1'] , 'b1') Observation: b1 is on the table. b2 is on the table. b3 is on the table. Robot arm is empty. The b1 is clear. The b2 is clear. The b3 is clear. Action: construct stack (['b1','b2','b3'] ) Observation: b1 is on b2. b2 is on b3, b3 is on the table. Robot arm is empty. The b1 is clear. The goal is satisfied. We imply baseline Reflexion for our learning-testing setting, where the testing instances are unseen during learning. Original Reflexion learns policy on a single instance, which can generate very detailed hints for the next turn, like "In this trial, I was able to find the desklamp on desk 1, but I did not find the bowl under the desklamp. In the next trial, I will go to desk 1, turn on the desklamp, then look for the bowl under the desklamp on desk 1 or desk 2. If I still cannot find the bowl, I will check the shelves and drawers." The detailed hint helps the agent perform in the same task instance, but can not transfer to different instances. We imply Reflexion to generate transferable policies based on the failed trial, such as "To accomplish the task, first locate the object, then navigate towards the cleaning location, clean the object, and finally deposit it at the specified receptacle. Follow the action 


Decision: This paper describes a structured prompting method for using LMs to solve interactive tasks by writing (and iteratively refining) code fragments implementing high-level actions. It outperforms a variety of other prompting methods on two simulated interactive tasks. As with  ReAct, Voyager, and other work in this space, it's a little unclear whether there's a generalizable scientific finding here, or just a well-designed collection of prompts and some code gluing them together. But empirical results are good, the engaged reviewers appear enthusiastic, and so the paper seems likely to be a useful contribution in this sub-area.

[At least one review was discounted during the decision process due to quality]

Reviews: review: paper_summary: paper_summary: This paper proposes a training framework that instructs LLMs to learn new actions. During the training stage, the action is created in the form of executable code function, and is refined through execution feedback. In the inference time, the LLM can utilize the created action to tackle more novel scenarios and perform more complex tasks.
strength_weakness: strength_weakness: Overall, the idea is intuitive and makes sense to me. But the biggest concern is that I do not find any fundamental difference and novel parts that distinguish this work from some previous publications, including but not limited to CRAFT, and Voyager. Please correct me if I have any misunderstanding. Basically, I think the fundamental part is creating some tools or actions defined in this work during the training time that are reusable, which can help LLMs to address some special test cases. Which part is novel in this work?
questions: questions: None
paper_summary: paper_summary: This paper is about improving the performance of language agents by refining the action space during the learning phase. Specifically, the authors propose LearnAct, an iterative system that uses LLMs to create new (or edit existing) actions based on past experiences and to act as a policy. In this work, the actions are expressed as Python code which enables the use of conditional statements, loops, and function calls (i.e., composition of actions). Experiments were conducted on robotic planning tasks and ALFWorld (a text-based game version of ALFRED, a 3D embodied environment). Empirically, the authors show that LearnAct achieves better performance on average compared to some relevant baselines (ReAct, Reflexion, CodeAsPolicy, Voyager) on those environments.

Edit: I have increased my score to 7.
strength_weakness: strength_weakness: - The research on Language Agents is evolving very fast and there is probably several prior works missing from the related work section. On top of my head, two came in mind: [TaskWeaver: A Code-First Agent Framework](https://arxiv.org/abs/2311.17541) and [ReGAL: Refactoring Programs to Discover Generalizable Abstractions](https://arxiv.org/abs/2401.16467). I think it would be beneficial to the reader to have a more comprehensive related work section to better position their work.

- This is a smaller concern. I noticed a small paragraph in A.6.3 about failure mode of the proposed technique regarding excessive iterations (which according to Fig.3 anything above 2 iterations looks worst!). Is this that the only failure? I doubt it, e.g., limit of the context length in ActionLearn to update the full action space at each iteration. I think having a proper discussion of the limitation as part of the main text would strengthen the paper.
questions: questions: - What is the definition of closed-loop in the context of this work? Is it the action learning process that is closed-loop or the actions (i.e., Python functions) themselves? If you are referring to the actions, have you checked if they are really closed-loop? In my experience Voyager tends to generate actions that are open-loop, i.e., they ignore the state of the environment as they send command to the environment. Granted this might be less problematic in your environments compared to Minecraft where other entities are moving in parallel.
- In section 5.3, it says the **temperature** is 0, but it also mentions sampling number is 4. I'd expect to get very similar samples (if not all the same). Can the authors clarify this?
- Do the authors have any insights on the generalization of the learned actions to unseen domains but that are sharing the same predefined actions?
- I'm curious to know if the authors have considered any open-source models during their experimentation?
paper_summary: paper_summary: The authors present alba approach for LLM agents to learn new actions and thereby improve the ability to successfully perform tasks in a simulation. The presented idea to learn new actions formalized as python code and how to integrate this new ablities into the policy of the agent.
The approach is overall presented clearly, however some details could be explained in more detail.
The approach is very well evaluated on two different tasks (Robotic Planning and AlFWorld). Thereby, the authors compare the method to several related work and the presented results are promising.
The authors missed a detailed analyze of the influence of the LLM. The LLM is only treated as a black box. The influence of the quality of the LLM is only analyzed very briefly and the authors do not evaluate any open source LLM to perform a more detailed analyze
strength_weakness: strength_weakness: The influence of the LLM on the overall performance is only investigated very briefly.
questions: questions: Tabel 5: How can the model use learn actions before learning?
paper_summary: paper_summary: This research paper introduces an innovative framework named LearnAct, designed to enhance the capabilities of LLM agents through action learning. The study tackles the limitations of fixed action spaces in LLM agents by allowing these agents to learn and refine their actions based on trial and error, a process akin to human experiential learning. This is achieved through an iterative learning strategy where actions are continuously updated based on the outcomes of previous tasks. The authors demonstrate the effectiveness of this approach in the Robotic Planning and Alfworld environments, showing significant improvements in task performance.
strength_weakness: strength_weakness: 1. **Lack of Comparative Analysis:** While the paper shows that LearnAct outperforms certain existing methods, there is a lack of comprehensive comparative analysis with a wider range of existing solutions (especially SOTA and more up-to-date methods), which could provide a better understanding of LearnAct’s positioning in the field.
2. **Complexity and Replicability:** The complexity of the LearnAct framework and its dependency on specific environments and configurations may limit its replicability and broader application. The paper could benefit from a clearer explanation of how LearnAct can be adapted or generalized for other contexts.
3. **Potential Overfitting:** There is a concern about the potential for overfitting in the iterative learning process, especially given the significant improvements reported. The paper could be strengthened by including a discussion on measures taken to avoid overfitting and ensure the generalizability of the learned actions.
questions: questions: 1. **Comparative Analysis:** Could the authors provide a more detailed comparative analysis with other methods in action learning for LLM agents? Specifically, how does LearnAct perform compared to methods that do not use iterative learning strategies? 
2. **Baseline Selection:** The authors claimed that LearnAct achieves more effective and efficient learning compared to state-of-the-art methods, but the baselines are not state-of-the-art and up-to-date enough. In fact LearnAct performs worse than some up-to-date models, such as FireAct [6] and KnowAgent [7]. 
3. **Flexibility:** Is it necessary to “Expanding Action Space by Action Creation” first? If not, how can you adaptively decide to expand the action space or not? If so, could you explain why action space expansion is a necessity, as the current action space is sufficient enough in many cases. 
4. **Generalizability:** Can the authors elaborate on how the LearnAct framework can be adapted for use in environments or tasks significantly different from those tested (e.g., non-robotic applications)?
5. **Scalability Concerns:** How does the LearnAct framework scale with increasingly complex tasks or larger environments? Are there computational constraints that could limit its practical applicability?
6. **Overfitting Measures:** What measures have been implemented in the LearnAct framework to prevent overfitting during the iterative learning process? How do you ensure that the actions learned are not overly tailored to the specific training examples used?

Comments: title: Discussion period
comment: Thank you for your valuable feedback! We are delighted to have addressed your questions and remain available to address any further questions you may have.
Thanks for the additional experiments with other LLMs. This information is very helpful. And also the question is answered well.
We sincerely hope that our response addresses your questions. We remain available to address any further questions you may have.
We sincerely hope that our response addresses your questions. We remain available to address any further questions you may have.
We sincerely appreciate your feedback! We remain available to address any further questions you may have.
Dear Authors,

Thank you very much for the clarification! Most of them have addressed my concerns. 

I appreciate the response, to include those in the revised draft will strengthen the paper. 

Overall, I acknowledge the idea of this work, but still have reservations about viewpoints of writing quality, and decide to increase my original scores 5 to 6. 

Best Regards,

Reviewer n7eb
We sincerely appreciate your valuable feedback! We remain available to address any further questions you may have.
Thank you for the insightful rebuttal. All my questions have been answered and the authors acknowledged that my *minor* concerns will be addressed in the final version. As far as I know evolving the action space based on gather experiences as proposed is novel (at least novel as it can be in this rapidly evolving research area). I will raise my score to 7 to help with potential tiebreak.
Hi reviewers! This is a friendly reminder that the authors have posted responses to the initial set of reviews. I'd appreciate it if you could look through the authors' comments soon and let us know if your concerns have been addressed / if there are any outstanding issues soon so there's time to discuss.

With appreciation,
Your AC