Towards Autonomous Agents: Adaptive-planning, Reasoning, and Acting in Language Models

## Abstract

Abstract We propose a novel in-context learning algorithm for building autonomous decision-making language agents. The language agent continuously attempts to solve the same task by reasoning, acting, observing and then self-correcting each time the task fails. Our selected language agent demonstrates the ability to solve tasks in a text-based game environment. Our results show that the gemma-2-9b-it language model, using our proposed method, can successfully complete two of six tasks that failed in the first attempt. This highlights the effectiveness of our approach in enhancing the problem-solving capabilities of a single language model through self-correction, paving the way for more advanced autonomous agents. The code is publicly available at https://github.com/YenCheHsiao/AutonomousLLMAgentwithAdaptingPlanning.git .

## 1 Introduction

Large language models (LLMs) are large statistical models that predict the next word, phrase, sentence, or paragraph based on a given input []. The quality of the output from a language model can be heavily influenced by the input prompt it receives []. One of the capabilities of LLMs is in-context learning, where they learn a new task from a small set of exemplars provided in the prompt during inference []. Prompt engineering is the process of designing and refining input prompts to elicit desired responses from LLMs [].

In Chain-of-Thought (CoT) [], given a prompt with exemplars that include an input part and an output part, a chain of thought consists of a series of intermediate natural language reasoning steps added between the input and output parts in each exemplar to produce the final output. However, the CoT prompting doesn’t have the ability to update its knowledge from the external world. ReAct [] prompting addresses the problem by providing the language model with a prior language description to guide its reasoning about solving diverse language reasoning and decision making tasks and adapting this reasoning by acting on and receiving the feedback from the external world. In Reflexion [], they proposed autonomous decision-making LLM agents by adding a reflection step to the CoT or ReAct prompt to adjust the reasoning, facilitating language agents’ learning from prior failings through verbal reinforcement. VOYAGER [] is a LLM based agent designed to explore an open-ended world and attain diverse skills through the integration of automatic curriculum, skill library management, and an iterative prompting mechanism incorporating environmental feedback, execution errors, and self-verification to enhance program performance. In Motif [], an agent is trained through reinforcement learning to maximize rewards from a parameterized model, which is trained based on preferences selected by a language model over pairs of actions, aimed at achieving a specific goal in a given environment.

In this study, we propose Self-Adaptive Language Agent (SALA), which is an adaptive decision-making language agent that self-adjusts the reasoning process of ReAct with a correction mechanism from Reflexion. Here, we let the language model agent adapt its own policy by correcting its previous failure using its internal knowledge. The proposed SALA differs from Reflexion [], which uses two LLMs: one for action generation and another for reflection whereas SALA employs a single LLM that can self-adapt its reasoning and acting behavior making it an autonomous language agent. Our experimental results show that in twelve different decision making tasks from the ALFWorld [] environment, the proposed SALA achieves a success rate of approximately 83%, which is higher than the ReAct-based agent, which achieved a success rate of 67%. In addition, the SALA could solve two tasks that couldn’t be completed in the first trail, demonstrating the effectiveness of our approach.

## 2 Methods

Let a text game be denoted as a function $f$ that maps the state $s\in\mathbb{V}$ and the action $a\in\mathbb{V}$ to an observation $o\in\mathbb{V}$, where $\mathbb{V}$ is a set of vocabulary. Let $\pi_{\theta}$ be an LLM agent over a pre-trained set of parameters $\theta$. Let $s_{0}$ be the initial state of the environment $f$, we aim to produce a sequence of actions $(a_{0},a_{1},a_{2},\dots)$, where $a_{i}\in\mathbb{V}$ for $i\in\mathbb{Z}$, to change the state to a terminal state that indicates the game is cleared.

In ReAct prompting [], they propose to use an LLM to produce the thought $(t_{0},t_{1},t_{2},\dots)$, where $t_{i}\in\mathbb{V}$ for $i\in\mathbb{Z}$, by $t_{i}\sim\pi_{\theta}(t_{i}|s_{i})$, where $s_{i}=\{s_{i-1},t_{i-1},a_{i-1},o_{i}\}$ for $i\in\mathbb{Z}^{+}$, $a_{i}\sim\pi_{\theta}(a_{i}|s_{i},t_{i})$ for $i\in\mathbb{Z}$, and $o_{i+1}=f(s_{i},a_{i})$ for $i\in\mathbb{Z}$.

The limitation of ReAct prompting [] is that complex tasks with a large action space require more demonstrations to learn effectively. The LLM may produce incorrect actions that do not lead to task completion. Reflexion [] addresses this problem by using an additional LLM to iteratively provide reflection text that will be added to the ReAct prompt for improvement. More specifically, for each trial $ep\in\mathbb{Z}^{+}$, if $ep\geq N$, where $N$ is a maximum number for each trail, a self-reflection $r^{ep}$ is generated and a new state $s^{ep+1}_{0}=\{s^{ep+1}_{N},r^{ep}\}$ is formed to be used as the initial state in the next trial $ep+1$. However, the method in Reflexion [] necessitates two LLMs, where one LLM is used to generate the thought or action, and another LLM is used to generate the reflection. We will modify this by using a single LLM to generate thought, action, and self-adaptation, which is the correction from the previous failed trial.

The architecture of the main idea of our work is shown in Figure 1. A desire is provided to an agent to motivate it to solve a specific task in a given environment. The agent can perform an action to interact with the environment, causing the state of the environment to change. The agent then receives an observation that describes the status of the environment and a reward signal. The action may be proposed from three different processes: the reasoning process determines the next action based on the current progress; the planning process proposes a series of actions that can be used to solve a specific task; and the adaptation process summarizes previous progress to provide a better plan towards maximizing the reward.

Figure: Figure 1: An architecture towards autonomous agent. Created with BioRender.com.
Refer to caption: extracted/5977606/Adaptive_planning_ReAct.png

We present a novel algorithm in Algorithm 1. Initially, we have the initial state $s_{0}$ which provides instructions, presents exemplars, and describes the environment and the goal for a specific task. $\pi_{\theta}$ is an LLM agent with a set of parameters $\theta$. $\tau=\{s_{0},t_{0},a_{0},o_{1},\dots\}$ is a sequence of the concatenation of state, thought, action, and observation, where $s_{k}$, $t_{k}$, $a_{k}$, and $o_{k}$ are sequences of tokens representing the $k$-th state, thought, action, and observation for $k\in\mathbb{Z}$, respectively. The return $R(\tau)$ is a string indicating whether the task is completed or not. $ep$ is a variable indicating the number of trials. The environment is reinitialized at each trial.

At the first time step $k=0$, the thought is sampled from

$$ $t^{1}_{0}\sim\pi_{\theta}(t^{1}_{0}|s^{1}_{0}),$ (1) $$

where the subscript 0 0 indicates the first time step, and the superscript $1$ indicates the first trail. $t^{1}_{0}$ represents the first thought in the first trial, and $s^{1}_{0}$ represents the first state in the first trial, with both being sequences of tokens. The action is then sampled from

$$ $a^{1}_{0}\sim\pi_{\theta}(a^{1}_{0}|s^{1}_{0},t^{1}_{0}),$ (2) $$

where $a^{1}_{0}$ represents the first action in the first trial, and the action is a sequence of tokens. The second observation in the first trial, $o^{1}_{1}$, is a sequence of tokens obtained by executing the action $a^{1}_{0}$ in the environment $f$ at state $s^{1}_{0}$ as

$$ $o^{1}_{1}=f(s^{1}_{0},a^{1}_{0}).$ (3) $$

A new state $s^{1}_{1}$ is formed by concatenating the thought $t^{1}_{0}$, action $a^{1}_{0}$, and observation $o^{1}_{1}$ after state $s^{1}_{0}$ as

$$ $s^{1}_{1}=\{s^{1}_{0},t^{1}_{0},a^{1}_{0},o^{1}_{1}\}.$ (4) $$

If a maximum time step is reached, the task fails and the return $R(\tau)$ is concatenated with "New plan: ". They are concatenated after the current state of the environment $s^{1}_{50}$ to form the initial state in the next trial $s^{2}_{0}$ as

$$ $s^{2}_{0}=\{s^{1}_{50},R(\tau)\},$ (5) $$

where $\tau=\{s_{0},t_{0},a_{0},o_{1},t_{1},a_{1},o_{2},t_{2},\dots o_{50}\}$. In the next trial, the first thought in the second trial, $t^{2}_{0}$, is sampled from the LLM by

$$ $t^{2}_{0}\sim\pi_{\theta}(t^{2}_{0}|s^{2}_{0}),$ (6) $$

We call the initial thought $t^{ep}_{0}$ at the $ep$-th trial for $ep>1$ as the adaptation from the ($ep-1$)-th trail and $t^{ep}_{0}$ indicates the correction of the ($ep-1$)-th failed trail to improve the next trail. In the next step, we propose to replace the initial state in the second trial with the initial state in the first trail to reduce the context length. We call this step compression. By performing compression, the first action in the second trail will only be conditioned on the initial state in the first trail $s^{1}_{0}$ and the adaptation from the first trail $t^{2}_{0}$ as

$$ $a^{2}_{0}\sim\pi_{\theta}(a^{2}_{0}|s^{2}_{0},t^{2}_{0}).$ (7) $$

The overview of the SALA architecture and interaction process are shown in Figure 2.

Figure: Figure 2: Overview of the SALA architecture and interaction process. (Left) The SALA consists of an LLM backbone, context, and decision-making processes. These elements can be customized to create specialized language agents that solve a wide variety of decision-making tasks. (Right) An example of how the SALA interacts with the environment to solve a decision making task, showing the flow of actions, progress, updates, and replanning as part of the task-solving process. Created with BioRender.com.
Refer to caption: extracted/5977606/SALA_intro_2.png

Figure: Algorithm 1 Self-Adaptive Language Agent

## 3 The ALFWorld environment

There are six types of tasks in the ALFWorld environment []: Pick and Place, Examine in Light, Clean and Place, Heat and Place, Cool and Place, and Pick Two and Place. For each task, a description of available receptacles is given in the first part of the instruction as follows: You are in the middle of a room. Looking quickly around you, you see a {recep1 id}, a {recep2 id}, …, and a {recepN id}, where recepN refers to the Nth receptacles like drawers or cabinet and id$\in\mathbb{Z}^{+}$. An example is shown in the text in Figure 3 with a green background.

After the description of the available receptacles, the goal instructions are provided based on the six different task types. For a Pick and Place task, the instruction will be either "put a obj in recap" or "put some obj on recap". For an Examine in Light task, it will be either "look at obj under the lamp" or "examine the obj with the lamp". For a Clean and Place task, it will be either "put a clean obj in recap" or "clean some obj and put it in recap". For a Heat and Place task, it will be either "put a hot obj in recap" or "heat some obj and put it in recap". For a Cool and Place task, it will be either "put a cool obj in recap" or "cool some obj and put it in recap". For a Pick Two and Place task, it will be either "put two obj in recap" or "find two obj and put them in recap". Inside the curly brackets, {obj}, {recep} and {lamp} refer to object, receptacle, and lamp classes, respectively. An example is shown in the text in Figure 3 with a red background.

After the goal instruction of the task, the agent or the user can interact with the game environment using the following nine different text actions: go to recap id, open recap id, clean obj id with recap id, take obj id from recap id, close recap id, close recap id, heat obj id with recap id, put obj id in/on recap id, and use recap id. Given the action, the game environment will return text observations accordingly. An example is shown in the text in Figure 3 with a magenta background.

Figure: Figure 3: A trajectory in the ALFWorld environment []. The text in the black box is composed of the description of available receptacles, the description of the goal instruction, and a sequence of actions and observations. (1) The description of available receptacles is listed in the first part of the text with a green background. (2) The description of the goal instruction with a red background shows that the task is a Cool and Place task, and the goal is to cool some pan and put it in countertop. (3) The sequence of actions and observations with a magenta background shows the actions performed and the corresponding observations from the environment. The actions are in boldface after the greater than symbol, and the observations are in regular text below each action.
Refer to caption: extracted/5977606/ex_traj_2.png

## 4 Experimental designs and results

### 4.1 Experiment on ReAct prompting in the Alfworld environment

In ReAct [], they randomly annotate the trajectories for each task type. Each trajectory includes sparse thoughts that decompose the goal, track subgoal completion, determine the next subgoal, and use commonsense reasoning to find an object and determine what to do with it. An exemplar used in ReAct [] is shown in Figure 4. The instruction text, “Interact with a household to solve a task. Here are two examples.”, is added above the exemplars to indicate that the general goal is to complete a household task, and the texts below it include two exemplars. Following the examples, the text ’Here is the task.’ is concatenated, followed by the description of available receptacles and the description of the goal instruction from the ALFWorld environment []. Finally, the greater-than symbol (’$>$’) is added to indicate that the language model should generate text after it, starting with actions or thoughts.

Figure: Figure 4: An exemplar in ReAct [] for a Heat and Place task in the ALFWorld environment []. The text in the black box is composed of the description from the ALFWorld environment [] and the thoughts annotated by ReAct []. The thought that decomposes the goal is shown with a magenta background. The thought that uses commonsense reasoning to find an object and determine what to do with it is shown with a red background. The thoughts that track subgoal completion are shown with a cyan background. The thoughts that determine the next subgoal are shown with a green background.
Refer to caption: extracted/5977606/ReAct_ex_2.png

ReAct [] evaluated their method on 134 tasks in the ALFWorld environment [], achieving a success rate of 70.9% using PaLM-540B [] and 78.4% using the GPT-3 text-davinci-002 model [], where each output token is selected with the highest probability. However, PaLM-540B [] is not publicly available, and the text-davinci-002 model [] was shut down by OpenAI on January 4th, 2024. Due to the difficulty in reproducing the results from [], we tested various open-source LLMs and selected the one with the highest success rate to develop our method. The results in Table 1 show that the gemma-2-9b-it model outperforms other models with a success rate of 62% in a 12-hour run. We will use gemma-2-9b-it for the subsequent experiments.

**Table 1: Success rate of different open-source language models with ReAct [] applied to the ALFWorld environment [] in a 12-hour run for each model**
| Developer | Name of the language model | Success<br>rate (%) | Number of<br>success tasks | number of<br>tasks |
| --- | --- | --- | --- | --- |
| Google | gemma-2-9b | 40 | 25 | 62 |
| Google | gemma-2-9b-it | 62 | 37 | 59 |
| Mistral AI | Mistral-7B-v0.3 | 28 | 16 | 57 |
| Mistral AI | Mistral-7B-Instruct-v0.3 | 25 | 14 | 54 |
| LLaMA | Llama-2-7b-hf | 8 | 4 | 50 |
| Microsoft | Phi-3-medium-128k-instruct | 41 | 12 | 29 |
| DeepSeek | deepseek-llm-7b-base | 14 | 15 | 103 |
| Hugging Face H4 | zephyr-7b-alpha | 15 | 8 | 52 |

We observed that gemma-2-9b-it model outperforms other models (gemma-2-9b, Mistral-7B-v0.3, Mistral-7B-Instruct-v0.3, Llama-2-7b-hf, Phi-3-medium-128k-instruct, deepseek-llm-7b-base, and zephyr-7b-alpha) with a success rate of 40% in solving various tasks in the ALFWorld environment [] using ReAct prompting []. We identified three common issues with these LLMs. First, the LLMs may attempt to retrieve an object from a location where the object does not exist, repeatedly performing the same action until reaching the maximum number of steps, $50$. Second, the LLMs may select an incorrect item. For instance, when the task is to "put a clean cloth in countertop", the gemma-2-9b model may pick up a "handtowel" instead. Subsequently, the LLM cannot complete the task by cleaning the "handtowel", resulting in repeated attempts to clean and place the "handtowel" in the "countertop" until the maximum step limit is reached. Third, some LLMs will misinterpret the order of sub-goals. For example, when the goal is to "examine the cd with the desklamp", the gemma-2-9b-it model erroneously attempts to retrieve a "desklamp" first. After failing to obtain the "desklamp", the LLM searches for the "cd" but only revisits previously searched locations instead of exploring new ones.

### 4.2 Experimental Result of SALA

In Reflexion [], a status text indicating whether a task is successful or not, along with a reflection text that guides the next trial toward success, are concatenated after the ReAct exemplars. Reflexion [] used two exemplars, as shown in Figure 5 in Appendix A.1, to guide the LLM in generating the reflection.

Unlike Reflexion [], which uses one LLM to generate thoughts and actions and another LLM to generate reflections, SALA uses a single LLM to generate thoughts, actions, and self-adaptations by concatenating the two Reflexion exemplars [] after the two ReAct exemplars [] for each task. This approach allows the adaptation to be generated after the LLM receives the string "STATUS: FAIL". We experiment with this single-LLM setup using the exemplars from ReAct [] and Reflexion [] as depicted in Figure 6 in Appendix A.1.

Initially, the input to the LLM is a prompt with the instruction, exemplars, and the description of the environment and the task to be completed. The output of the LLM will be the input prompt concatenated with a series of thoughts, actions, and observations. We extract the first thought or action from the output by extracting the line immediately following the input prompt content. The line immediately after the greater than symbol ($>$) is used as input to the environment. If the input to the environment contains the string "think:", we overwrite the output with "OK.", following the process described in [, ].

If the task is completed, the environment will return a variable with a value of $1$, and the process is finished. Otherwise, we concatenate the extracted thought or action with the observation and append this new text to the input prompt to form the next input prompt. After $50$ steps, if the task remains unfinished, we reinitialize the ALFWorld environment [] and append "STATUS: FAIL" and "New plan:" after the last observation. This updated text becomes the next input prompt to the LLM. The output of the LLM will include the adaptation text following "New plan:", which is extracted and concatenated above the initial prompt, $s^{ep}_{0}$. This new prompt is then used as the next input to the LLM to obtain corrected thoughts and actions. We set a maximum of $9$ trails; if $ep\geq 10$, we terminate the trial and proceed to the next task. The results of this experiment, conducted for 14 different tasks using gemma-2-9b-it, are shown in Table 2.

**Table 2: Number of steps for the ReAct-based [] agent and the proposed SALA to complete tasks in the ALFWorld environment []**
| Task number | 1 | 2 | 3 | 4 | 5 | 6 | 7 |
| --- | --- | --- | --- | --- | --- | --- | --- |
| ReAct [] | fail | fail | 14 | 10 | 12 | 19 | fail |
| SALA | 13 | 12 | fail | 10 | 17 | 87 | fail |
| Task number | 8 | 9 | 10 | 11 | 12 | 13 | 14 |
| ReAct [] | 23 | fail | 10 | 19 | fail | 16 | 15 |
| SALA | 19 | fail | 10 | 17 | 67 | fail | 10 |

Among the fourteen tasks, ten were completed successfully, while four were not finished after reaching 10 adaptation steps. Of the ten successful tasks, eight were completed without reaching the adaptation step. Failures in tasks 7 and 9 for both methods were due to issues within the ALFWorld environment []. Although both agents successfully finished the tasks for tasks 7 and 9, the environment failed to indicate the task were completed. Consequently, tasks 7 and 9 are excluded from the evaluation of different methods. Excluding these tasks, the proposed SALA achieves a success rate of approximately 83%, outperforming the ReAct-based agent, which achieved a success rate of 67%.

In one of the failed tasks (task 3) from SALA, the only adaptation obtained was: "I was stuck in a loop in which I continually looked for a lettuce in the fridge. I should have looked for a lettuce in the fridge, then taken it, then put it in the countertop. I will try to execute a different action if I am stuck in a loop again." for every trial. However, the SALA agent did not attempt different actions in subsequent trials and continued to examine the fridge. The actual issue in task 3 is that the fridge does not contain lettuce, but the SALA agent incorrectly assumes it is present and persistently attempts to retrieve it. A similar issue is also found in task number 13 which has the adaptation: "I was stuck in a loop in which I continually examined fridge 1 instead of using the fridge to cool the lettuce. I should have taken the lettuce from the fridge and then put it on the countertop. I will try to execute a different action if I am stuck in a loop again." The SALA agent did not attempt different actions in subsequent trials and continued to examine the fridge.

In task 6, the task could not be completed in the first trial but was successfully completed in the second trial with 38 steps. The goal of this task is to "find two pillow and put them in sofa". In the first trial, the agent failed to complete the task because it attempted to pick up ’pillow 2’ from ’sofa 1’; however, ’sofa 1’ only contained ’pillow 1’. Afterward, the agent continued trying to put ’pillow 2’ on ’sofa 1,’ but it failed because it did not have the pillow. Subsequently, the LLM did not output any text until the end of the trial. After this, the following adaptation was generated and appended to the input prompt for the second trial: "I was stuck in a loop in which I continually looked for the second pillow in sofa 1. I should have looked for the second pillow in armchair 1, sidetable 1, and cabinet 1-4. I will try to execute a different action if I am stuck in a loop again.". In the second trial, the agent found that ’pillow 2’ was on ’armchair 1,’ picked it up, placed it on the sofa, and completed the task. The trajectories of the two trials in task number 6 are shown in Figure 7 in Appendix A.1.

In task 12, the task could not be completed in the first trial but was successfully completed in the second trial within 18 steps. The goal of this task is to "put a cool tomato in microwave". In the first trial, the agent failed to complete the task because it attempted to take a ’tomato’ from ’fridge 1’; however, ’fridge 1’ doesn’t contained any tomato. Afterward, the agent continued trying to take a ’tomato’ from ’fridge 1’ until the end of the trial. After this, the following adaptation was generated and appended to the input prompt for the second trial: "I was stuck in a loop in which I continually looked for a tomato in the fridge. I should have looked for a tomato in a different environment. I will try to look for a tomato in a different environment in the next trial.". In the second trial, the agent found that ’tomato 1’ was on ’countertop 1,’ picked it up, cooled it with the fridge, placed it in the microwave, and completed the task. The trajectories of the two trials in task number 12 are shown in Figure 8 in Appendix A.1.

## 5 Conclusion

We present a novel in-context learning algorithm designed for a single language model to complete tasks in a text-based game by correcting its previous failures. This approach, by introducing self-adaptation reduces the number of models used in previous work [] from two LLMs to one LLM, gaining in autonomy. Our findings indicate that the gemma-2-9b-it model achieves the highest success rate of 62% for completing tasks in the ALFWorld environment [] using ReAct prompting [], compared to other selected open-source language models. In the twelve selected tasks from the ALFWorld environment [], the proposed SALA, utilizing the gemma-2-9b-it model, achieved a success rate of 83%, outperforming the ReAct-based agent [] with the same model, which attained a 67% success rate. In addition, we show that using SALA with gemma-2-9b-it, two of the six tasks that could not be completed in one trial can be completed in the second attempt by appending the adaptation from the previous trial. Future work will involve further experimentation with different in-context learning algorithms to complete decision-making tasks using a single language model without Reflexion exemplars for reducing the number of tokens in the input prompt.

## Acknowledgments

The computational work for this project was conducted using resources provided by the Storrs High-Performance Computing (HPC) cluster. We extend our gratitude to the UConn Storrs HPC and its team for their resources and support, which aided in achieving these results.

## Appendix A Appendix / supplemental material

### A.1 Figures

Figure: Figure 5: Two exemplars in Reflexion [] for the ALFWorld environment []. The text in each black box comprises one exemplar from Reflexion [] designed to guide an LLM in generating the correct action to complete a task in the ALFWorld environment []. In each black box, the text preceding the yellow-background text represents a ReAct trajectory, as shown in Fig. 4. The text next to "STATUS: " indicates whether the task is completed. If the task is completed, the yellow-background text will read "STATUS: OK". If the task is not completed, it will read "STATUS: FAIL". The reflection text is highlighted with a cyan background.
Refer to caption: extracted/5977606/Reflexion_exs.png

Figure: Figure 6: The process of using a single LLM to generate cycles of thoughts, actions, and self-adaptations by fusing ReAct’s [] and Reflexion’s exemplars [] for solving tasks in the ALFWorld environment [].
Refer to caption: extracted/5977606/Append_Reflexion_Exemplars.png

In Figure 7, the text in each black box represents one trajectory in the ALFWorld environment [], as detailed in the experiment in Section 4.2. In each black box, the text at the top includes the description of available receptacles and the goal instruction, as mentioned in Fig. 3. Below the goal instruction, the text following the greater-than symbol (’$>$’) represents the thoughts or actions generated by the gemma-2-9b-it LLM. Text beginning with ’$>$ think:’ indicates a thought, while all other text beginning with without ’think’ represents actions. The text beneath each thought or action is the observation from the ALFWorld environment []. The trajectory of the first trial in task number 6 is shown in the left black box. The LLM agent failed to complete the task because it couldn’t find the second pillow, incorrectly believing it had finished the task, and then continued generating empty strings. The adaptation the LLM agent generated is highlighted with a cyan background. In the second trial, shown in the right black box, the adaptation is appended above the description of the available receptacles. The agent successfully found the second pillow in the second trial and completed the task.

Figure: Figure 7: Two trajectories for task 6 generated by SALA. The left trajectory represents a failure, where the agent incorrectly assumes task completion and ceases actions. The right trajectory includes an adaptation (highlighted in cyan), leading to successful task completion.
Refer to caption: extracted/5977606/Reflexion_out_6.png

In Figure 8, the text in each black box represents one trajectory in the ALFWorld environment [], as detailed in the experiment in Section 4.2. In each black box, the text at the top includes the description of available receptacles and the goal instruction, as mentioned in Fig. 3. Below the goal instruction, the text following the greater-than symbol (’$>$’) represents the thoughts or actions generated by the gemma-2-9b-it LLM. Text beginning with ’$>$ think:’ indicates a thought, while all other text beginning with without ’think’ represents actions. The text beneath each thought or action is the observation from the ALFWorld environment []. The trajectory of the first trial in task number 6 is shown in the left black box. The LLM agent failed to complete the task because it couldn’t find a tomato, and then continued checking the same receptacle. The adaptation the LLM agent generated is highlighted with a cyan background. In the second trial, shown in the right black box, the adaptation is appended above the description of the available receptacles. The agent successfully found a tomato in the second trial and completed the task.

Figure: Figure 8: Two trajectories for task 12 generated by SALA. The left trajectory shows a failure, where the agent repeats the same action with no progress until the maximum step is reached. The right trajectory includes an adaptation (in cyan), leading to successful task completion.
Refer to caption: extracted/5977606/Reflexion_out_12.png

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: 1. The experimental results lack detailed statistical significance testing. Can you provide additional quantitative analysis, such as confidence intervals or statistical tests, to verify the robustness of the reported success rates for SALA compared to other baselines?
In your experimental setup, hyperparameter details and training configurations are not clearly outlined. Could you specify these parameters, including learning rates, training duration, and the criteria used to choose these values, to ensure reproducibility?

2. The comparison with baselines is somewhat limited. Have you considered evaluating SALA against more recent, high-performing language model agents beyond those mentioned?

3. The introduction touches briefly on the limitations of existing methods but lacks specificity. Could you elaborate more explicitly on the shortcomings of prior methods and explain how each of these motivates the design choices in SALA? This would help clarify the intended contributions and situate them within the broader research context.
soundness: 3
strengths: 1. The proposed SALA method effectively combines reasoning, acting, and adaptation within a single language model, reducing model complexity while achieving autonomy.

2. The integration of a correction mechanism enhances decision-making capabilities, demonstrating significant improvements over prior methods, such as Reflexion. Besides, experimental results show an 83% success rate, outperforming ReAct by a substantial margin, thus demonstrating the practical significance of SALA in task completion.

3. The use of self-adaptation allows for reducing resource requirements compared to Reflexion, which relies on two LLMs, showing efficient use of model capabilities.
summary: The paper introduces a novel in-context learning algorithm for building autonomous decision-making agents using a single language model (SALA). The approach integrates reasoning, acting, and self-adaptation mechanisms to enhance problem-solving capabilities in text-based environments, achieving higher success rates compared to existing methods like ReAct and Reflexion.
weaknesses: 1. The introduction lacks a clear articulation of the specific limitations of existing methods that the proposed SALA intends to address, making it difficult to understand the motivation behind the approach.

2. The experimental results are poorly presented, with insufficient statistical analysis to demonstrate the robustness of the reported success rates, leaving room for questioning the significance of the results.

3. Lack adequate details on the hyperparameter settings or the training configurations, which makes it challenging to reproduce the experiments or assess the validity of the claims. The comparison with baselines is superficial, lacking a comprehensive evaluation against more recent methods that have demonstrated competitive results in similar environments.

4. The paper fails to justify the use of a single LLM for both action generation and self-adaptation in terms of scalability or performance, especially when dealing with more complex decision-making tasks.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 2
questions: Please take a look at the section on weaknesses.
soundness: 2
strengths: The paper is clearly written.
summary: The paper tackles the problem of planning and reasoning in a text-world environment (ALFWorld). The paper proposes to use a single large language model to output and correct its plans guided by mistakes in its generation (failures in task completion in the environment) by iteratively prompting the same model to adaptively re-plan. The paper highlights that iterative correction with an LLM allows their Agent to complete more tasks than a single round of interaction.
weaknesses: - The sole difference between Reflexion (Shinn et al., 2023), and the proposed approach seems to be the use of a single vs two models to provide feedback. With both the reflection and action model being the same pre-trained model, it is unclear whether this distinction is even a significant one--both the methods require the same number of inference steps with the same model. If there are other differences, please indicate them in the paper and additionally include a quantitative comparison to Reflexion as a baseline.
- Missing implementation and dataset details: There is little clarity on the metrics being reported here. 
  - How is the accuracy computed? 
  - Does Table 2 report the average number of steps across data points in each task type? 
  - What is the size of the test set? 
  - Missing details around the prompt provided to the model for feedback? 
- Why are the standard metrics for testing task performance on ALFWorld / Alfred not reported? 
- What is the task success rate of the methods across each of the task types?
- The paper states that it proposes a "new in-context learning algorithm". The description needs to be more measured. There isn't a new in-context learning algorithm, but a modified prompting strategy.

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 1
questions: This paper would require quite significant modifications to be considered for publication IMHO.
soundness: 1
strengths: Overall, I don’t think this is paper is ready to be considered at ICLR yet for the reasons listed in the weakness section below.
summary: This paper describes a new in-context learning algorithm for LLM to repeatedly attempt to solve tasks.
weaknesses: * The writing of this paper is unclear. There is no proper discussion on related work. Some paragraphs in the results section, around line 370 and line 380, are completely repetitive without providing much insight.
* The proposed method seems to be minor incremental contribution on top of ReAct, with the main difference being using one model instead of two models. Large portions of the method section is repeating existing method or discussing lengthy but trivial procedures of applying LLM for solving tasks but no insight is provided about the proposed method.
* The evaluation is minimal. It is only evaluated on ALFWorld.  Additionally, it only compares against ReAct but does not consider any ablations or other baselines.
* The paper lacks content. Among its limited evaluation, large portion is devoted to evaluating ReAct, which is not proposed by this work, on different open-sourced models. This type of evaluation is useful but should not be considered as experimental contribution for ICLR.

---

confidence: 4
contribution: 1
initial_rating: 1
presentation: 2
questions: **Questions**
- Reflexion uses three models: an Actor to generate text and actions, an Evaluator to score the outputs from the Actor and a Self-Reflection model to generate verbal reinforcement cues for self-improvement. Why the paper states that Reflexion uses only two models? The paper needs more clarifications on this discrepancy.

- What is the main motivations for modifying the Reflexion to use a single LLM to generate thought, action, and self-adaptation? Why this minor modification would differentiate this work from Reflexion? Authors should clarify the motivations.

**Suggestions**
- Authors should add a column of average success rate and average steps in the table for easier comparison (line 327).
soundness: 1
strengths: **Originality**: The paper modifies the Reflexion that uses two large language models (LLMs) to use only one LLM to generate thoughts, actions, and self-adaptations, which may use less memory. However, there are problems in the originality, which will be further discussed in the Weakness Section.

**Quality**: (Experiments are not sufficient, which will be detailed in Weaknesses Section)

**Clarity**: The paper is easy to follow. Illustrative figures are helpful for understanding the paper.

**Significance**: (The proposed method SALA lacks significance, which will be detailed in Weaknesses Section)
summary: The paper proposes an architecture towards autonomous agent (SALA), which use one large language model (LLM) to generate thoughts, actions, and self-adaptations by concatenating two Reflexion (Shinn et al., 2023) exemplars and two ReAct (Yao et al., 2023) exemplars for each task. They show that this approach is more effective than ReAct on the twelve selected tasks from ALFWorld environment and it achieves a success rate of 83%. They also find that the self-adaptation is particularly important for decision making in the next trial.
weaknesses: **Lack of Novelty** 
- The proposed method SALA use one model for action generation and self-adaptation, but this lacks novelty in the methodology because Reflexion pipeline is sequential and one can use one LLM for different tasks with different prompts. 
- Minor modification from the original method Reflexion is not considered to be novel if there's no superior results to the original method. The paper does not compare the method SALA to Reflexion on either accuracy or efficiency.

**Lack of Experiments**
- The experiments only compare the proposed method SALA with ReAct, which is less capable than Reflexion (the method augmenting the ReAct with self-reflection). Since SALA applies minor modification from the Reflexion, authors should add experiments with the Reflexion.
- There are only text-based game ALFWorld in the experiments, which is not enough to demonstrate the advantages of the proposed method. Authors should consider adding other datasets used in Reflexion like HotpotQA [1] for reasoning, and MBPP [2], HumanEval [2], LeetcodeHardGym [3] for programming.
- The paper only have results on the open-sourced language model gemma-2-9b-it, which are not enough. Authors should evaluate at least 3 open-sourced models with different model size (e.g. 30B, 65B). Additionally, it is recommend to add API models like GPT-4o since they are generally more capable than those open-sourced models which may lead to different model behaviors with different methods. At the end, authors should aggregate the results and also average them to provide a more comprehensive evaluation.

*References*

[1] Yang, Zhilin, et al. "HotpotQA: A dataset for diverse, explainable multi-hop question answering." arXiv preprint arXiv:1809.09600 (2018).

[2] Austin, Jacob, et al. "Program synthesis with large language models." arXiv preprint arXiv:2108.07732 (2021).

[3] Chen, Mark, et al. "Evaluating large language models trained on code." arXiv preprint arXiv:2107.03374 (2021).

[4] Shinn, Noah, et al. "Reflexion: Language agents with verbal reinforcement learning." Advances in Neural Information Processing Systems 36 (2024).