Retroformer: Retrospective Large Language Agents with Policy Gradient Optimization
Authors: Weiran Yao
Shelby Heinecke
Juan Carlos Niebles
Zhiwei Liu
Yihao Feng
Le Xue
Rithesh R N
Zeyuan Chen
Jianguo Zhang
Devansh Arpit
Ran Xu
Phil L Mui
Huan Wang
Caiming Xiong
Silvio Savarese
Venue: ICLR 2024 spotlight (2024)
Keywords: Language Agent
AI Agent
Reinforcement Learning
Primary area: applications to robotics, autonomy, planning
TL;DR: This paper introduces a principled framework for reinforcing large language agents by learning a retrospective model, which automatically tunes the language agent prompts from environment feedback through policy gradient.

Abstract: Recent months have seen the emergence of a powerful new trend in which large language models (LLMs) are augmented to become autonomous language agents capable of performing objective oriented multi-step tasks on their own, rather than merely responding to queries from human users. Most existing language agents, however, are not optimized using environment-specific rewards. Although some agents enable iterative refinement through verbal feedback, they do not reason and plan in ways that are compatible with gradient-based learning from rewards. This paper introduces a principled framework for reinforcing large language agents by learning a retrospective model, which automatically tunes the language agent prompts from environment feedback through policy gradient. Specifically, our proposed agent architecture learns from rewards across multiple environments and tasks, for fine-tuning a pre-trained language model which refines the language agent prompt by summarizing the root cause of prior failed attempts and proposing action plans. Experimental results on various tasks demonstrate that the language agents improve over time and that our approach considerably outperforms baselines that do not properly leverage gradients from the environment.

Published as a conference paper at ICLR 2024
RETROFORMER: RETROSPECTIVE LARGE LANGUAGE AGENTS WITH POLICY GRADIENT OPTIMIZATION
Weiran Yao", Shelby Heinecke', Juan Carlos Niebles Zhiwei Liu', Yihao Feng', Le Xue Rithesh Murthy Zeyuan Chen', Jianguo Zhang Devansh Arpiti, Ran Xu², Phil Mui!, Huan Wang Caiming Xiong Silvio Savarese
Salesforce Al Research
ABSTRACT
Recent months have seen the emergence of powerful new trend in which large language models (L.LMs) are augmented to become autonomous language agents capable of performing objective oriented multi-step tasks on their own, rather than merely responding to queries from human users. Most existing language agents, however, not optimized using environment--ppecffic rewards. Although some agents enable iterative refinement through verbal feedback, they do not reason and plan in ways that compatible with gradient-basee learning from rewards. This paper introduces a principled framework for reinforcing large language agents by learning retrospective model, which automatically tunes the language agent prompts from environment feedback through policy gradient. Specifically, our proposed agent architecture learns from rewards across multiple environments and tasks, for fine-tuning a pre-trained language model which refines the language agent prompt by summarizing the root cause of prior failed attempts and propos- ing action plans. Experimental results on various tasks demonstrate that the lan- guage agents improve over time and that our approach considerably outperforms baselines that do not properly leverage gradients from the environment.
INTRODUCTION
Recently, we have seen the emergence of powerful new trend in which large language models (LLMs) are augmented to become autonomous language action agents capable of performing tasks on their own, ultimately in the service of goal, rather than responding to queries from human users. Prominent studies, including ReAct (Yao et al., 2023), Toolformer (Schick et al., 2023), Hugging- GPT (Shen et al., 2023), Generative Agents (Park et al., 2023), WebGPT (Nakano et al., 2021), AutoGPT (Gravitas, 2023), BabyAGI (Nakajima, 2023), and Langchain (Chase, 2023), have cessfully showcased the viability of creating autonomous decision- mannin agents by leveraging the capabilities of L.LMs. These approaches use I.LMs generate text-based outputs and actions that can be further employed for making API calls and executing operations within a given environment.
Given the immense scale of LLMs with an extensive parameter count, the behaviors of most existing language agents, however, are not optimized or aligned with environment reward functions. An exception is very recent language agent architecture, namely Reflexion (Shinn et al., 2023), and several other related work, e.g., Self-Refine (Madaan et al., 2023b) and Generative Agents (Park al., 2023), which use verbal feedback, namely self-reflection, to help agents learn from prior failure. These reflective agents convert binary or scalar reward from the environment into verbal feedback in the form of textual summary, which then added as additional context to the prompt for the language agent. The self-reflectio feedback acts as semantic signal by providing the agent with a concrete direction to improve upon, helping it learn from prior mistakes and prevent repetitive errors to perform better in the next attempt.
Although the self-reflection operation enables iterative refinement, generating useful reflective feed- back from pre-trained, frozen LLM is challenging, as showcased in Fig. 1, since it requires the
Corresponding Authors
WWebsite for Ret & demos: https: pettps://Retrooe /Rtttroooooorr..../ro gi thub. io/ kesearcch/etrrooormee
*Code: https: / thub, http:////gttut..c.//aaaaoe

---

Published as a conference paper at ICLR 2024
LLM to have good understanding of where the agent made mistakes in a specific environment, i.e., the credit assignment problem (Sutton & Barto, 2018), as well as the ability to generate a summary containing actionable insights for improvement. The verbal reinforcement cannot be optimal, if the frozen language model has not been properly fine-tuned to specialize in credit assignment problems for the tasks in given environments. Furthermore, the existing language agents do not reason and plan in ways that are compatible with differentiable, gradient-based learning from rewards by ex- ploiting the existing abundant reinforcement learning techniques. To address these limitations, this paper introduces Retrofo oomer. a principled framework for reinforcing language agents by learn- ing plug-in retrospective model, which automatically refines the language agent prompts from environment feedback through policy optimization. Specifically, our proposed agent architecture can learn from arbitrary reward information across multiple environments and tasks, for iteratively fine-tuning a pre-trained language model, which refines the language agent prompts by reflecting on failed attempts and assigning credits of actions taken by the agent on future rewards.
1. Task instruction
Lollipop Chainsaw featured Juliet Starling, who was voiced by Canadian-Amerrcan actress who has done voice roles for what Teen Titans spinoff series?
2. Action sequences in prior trial
4. Action sequences in next trial
Action I: Search[Juliet Starling]
Action 1: Search[Lollipop Chainsaw]
Action 2: Search(L Lollipop Chainsaw]
Action 2: Search[Tare Tarr Strong]
Action 3: Search Tara Strong]
Action 3: Finish[Teen Titans, Teen Titans Go!]
Action 4:
Finish[Teen Titans and Teen Titans Go!]
add to agent prompt
3. Verbal feedback (self-refiection)
should have searched for Lollipop Chainsaw first and looked up the Cunadian- Camadian-American actress who voiced Julict Starling ufterwards. also should have looked up Tara Strong's filmography and searched for any voice roles she did specifically for Teen Titans or Teen Titans Go!
Figure 1: An example of uninformative self-reflections from a frozen LLM. The root cause of failure in prior trial is that the agent should have only submitted the spinoff series "Teen Titans Go" and not "Teen Titans" in the answer. The agent forgot its goal during a chain of lengthy interactions. The verbal feedback from frozen LLM, however, only rephrases the prior failed actions sequences as the proposed plan, resulting repetitive, incorrect actions in the next trial.
We conduct experiments on a number of real-world tasks including HotPotQA (Yang et al., 2018), which involves search-based question answering tasks, AlfWorld (Shridhar al., 2021), in which the agent solves embodied robotics tasks through low-level text actions, and WebShop (Yao et al., 2022), a browser environment for web shopping. We observe Retroformer agents are faster learners compared with Reflexion, which does not use gradient for reasoning and planning, and are better decision-maker and reasoners. More concretely, Retroformer agents improve the success rate in HotPotQA by 18% with 4 retries, 36% in AlfWorld with 3 retries and 4% in WebShop, which demonstrate the effectiveness of gradient- learning for LLM action agents.
To summarize, contributions are the following:
The paper introduces Retroformer, which iteratively refines the prompts given to large lan- guage agents based on environmental feedback to improve learning speed and task completion. We take a policy gradient approach with the Actor LLM being part of the environment, allowing learning from wide range of reward signals for diverse tasks.
The proposed method focuses on fine-tuning the retrospective model in the language agent sys- tem architecture, without accessing the Actor LLM parameters needing to propagate gradients through it. The agnostic nature of Retroformer makes a flexible plug-in in module for various types of cloud- based L.LMs, such OpenAl GPT or Google Bard.
2

---

Published as a conference paper at ICLR 2024
2 RELATED WORK
Autonomous Language Agents We summarize in Table I the recent language agent litcrature related to our work from five perspectives and differentiate our method from them. The completion of a complex task typically involves numerous stages, An AI agent must possess knowledge of these stages and plan accordingly. Ohain-of-Thouuhts or CoT (Wei al., 2022) is the pioneering work that prompts the agent to decompose challenging reasoning tasks into smaller, more manageable steps. ReAct (Yao et al., 2023), on the other hand, proposes the exploitation of this reasoning and acting proficiency within LLM to encourage interaction with the environment (e.g. using the Wikipedia search API) by mapping observations to the generation of reasoning and action traces or API calls in natural language. This agent architecture has spawned various applications, such as HuggingGPT (Shen al., 2023), Generative Agents (Park al., 2023), WebGPT (Nakano et al., 2021), AutoGPT (Gravitas, 2023), and BabyAGl (Nakajima, 2023).
Table 1: Related work on large language agents.
Approach
Gradlent learning
Arbitrary reward
Iterative refinement
Hidden constraints
Decision making
Memory
CoT (Wei et al., 2022)
X
x
x
x
ReAct (Yao et al., 2023)
X
x
x
Self-refine (Madaan et al., 2023b)
X
x
/
x
x
RAP (Hao et al., 2023)
X
x
/
Reflexion (Shinn et al., 2023)
X
X
Retroformer (our method)
However, these approaches fail to learn from valuable feedback, such as environment rewards, to en- hance the agent's behaviors, resulting in performances that are solely dependent the quality of the pre-trained LLM. Self-refine (Madaan et al., 2023a) addresses this limitation by employing a single LLM generator, refiner, and provider of feedback, allowing for iterative refinement of outputs. However, it is not specifically tailored for real-world task-based interaction with the environment. On the other hand, RAP (Hao al., 2023) repurposes the LLM function both world model and a reasoning agent. It incorporates Monte Carlo Tree Search for strategic exploration within the extensive realm of reasoning with environment rewards. This approach enables effective naviga- tion and decision- decision-making making in complex domains. Recently, Shinn et al. (2023) presents Reflexion, framework that equips agents with dynamic memory and self-reflection capabilities, enhancing their reasoning skills. Self-reflection plays pivotal role, allowing autonomous agents to iteratively refine past actions, make improvements, and prevent repetitive errors.
Transformer Reinforcement Learning Reinforcement learning with provided reward function or a reward-labeled dataset, commonly referred to as RLHF, has become a standard practice within the LLM fine-tuning pipeline. These endeavors have convincingly demonstrated the efficacy of RL as a means to guide language models towards desired behaviors that align with predefined reward functions encompassing various domains, including machine translation, summarization, and gen- erating favorable reviews. Among the prevalent transformer RL methods are online RL algorithms such Proximal Policy Optimization such as Implicit Language Q-Learning or IIQ (Snell et al., 2022) and Direct Preference Optimiza- tion or DPO (Rafailov al., 2023). These methods have been implemented in TRL/TRLX (von Werra et al., 2020; Max et al., 2023) distributed training framework.
PPO (Schulman et al., 2017), and offline RI. techniques
3 NOTATION AND FORMULATION
In this work, we denote a large language model (LLM) based action agent as a function Mg, X A, where A' is the space of prompts, which may include the actual prompts z: provided by the users, as well as some contextual information c C. Here C is the space of context as a representation of the current state S returned by the environment Ω. AA is the space of actions. Note the actions taken by most language model based agents are sampled auto-repressively. so A a random function. The subscript & denotes the re- re-paramterriee random variables involved in the sampling process. Another note is, the LLM-bbsed agent itself is stateless. All the states and possible memorization characterized text in the agent prompt x.
3

---

Published as a conference paper at ICLR 2024
The environment is defined tuple ( R). Tg. 8 A - S is the state transition function, where is the space states and A the action space. Here we assume the states and actions are represented using text. Again used &o represent the randommess involved in the state transition. For each state E S, a reward function is defined as R S R. At each step of the play, the state described using natural language, and integrated into the context c. In the context, previous states may also be described and embedded to help LLMs making good guess on the next action to take. As in all the reinforcement learning setting, the final goal is maximize the cumulative rewards, or episode returns Goum Et= R(st). In many situations, the rewards are sparse, i.e., R(st) are mostly except very few states, such in the terminal state for indicating task success failure.
The retrospective model takes the all the previous states $1,-- actions ai,....t rewards f1,". and the user prompt x as imput, and massage them into new prompt z to be consumed by the LLM:
T [S; R, X
X,
(1)
where & stands for the randomness involved the retrospective model, and the set of learnable parameters in the retrospective model. The goal of the RL optimization is
T
Σ R(se)
arg max 6
EE
s.L.
8t+1 = Tc. (st, Lei E' (1& (1, Ti a )))i=))
(2)
Note that the only learnable parameters are in the retrospective model M,- Since LLM action agent frozen, it can be considered as part of the environment. Specifically, we construct another environment with the transition function T T(S, C X' S, and the same reward function R, then Eq. (2) is just regular RL optimization so all the popular RL algorithms apply.
4 OUR APPROACH: REINFORCING RETROSPECTIVE LANGUAGE AGENT
As illustrated Fig. 2, our proposed framework Retroformer is comprised of two language model components: actor LLM, denoted Ma, which generates reasoning thoughts and actions, and a retrospective LLM, denoted as My. which generates verbal reinforcement cues to assist the actor in self-improvement by refining the actor prompt with reflection responses.
Reflection prompt
Retrospective LM
Reflection response
Trial
Relurs
response
Gu.
Trajectory (s.. Oj,
5)
Prompt
Actor LM
Env 2 Returss
Reflection response
Env 2 Reiums
1,2.3...
Action
Roturn
Reflection response
Raturns
(a,,),..))
Environment
Environment
Rating for reflection response
Environment K
AG Guits
Gu.
[a] Retrospective agent
(b) Ratings for reflection responses
Figure 2: Framework overview. (a) The retrospective agent system (Sec. 4.1) contains two LLMs communicating to refine agent prompts with environment feedback. (b) The retrospective LM is fine-tuned with response ratings using proximal policy optimization (Sec. 4.2).
We assume in this paper that the actor model is frozen LLM whose model parameters are inac- cessable (e.g., OpenAl GPT) and the retrospective model is a smaller, local language model that can be fine-tuned under low-resource settings (e.g.. Llama-7b). In addition, Retroformer has an iterative policy gradient optimization step which is specifically designed to reinforce the ret- rospective model with gradient-based approach. We provide in this section detailed description
4

---

Published as a conference paper at ICLR 2024
of each of these modules and subsequently elucidate their collaborative functioning within the Retroformer framework. The implementation details are presented in Appendix C.
4.1 RETROSPECTIVE AGENT ARCHITECTURE
As illustrated in Fig. 2(a), for the actor and retrospective models, we apply a standard communi- cation protocol modified from the Relexion agent architecture (Shinn et al., 2023), in which the retrospective model refines the actor prompt by appending verbal feedback to the prompt.
Actor Model The actor model is LLM hosted in the cloud, whose model parameters are hidden and frozen all the time. The actor LM is instructed to generate actions with required textual content, taking into account the observed states. Similar reinforcement learning, we select an action or generation, denoted as at, from the current policy To at time step and receive an observation, represented by St, from the environment. We use ReAct (Yao al., 2023) as our actor prompt.
[t- 0kii, Ma ([SK,, Okd,, (
Retrospective Model The retrospective model M, is instantiated as a local LM. Its primary func- tion is to produce self-reflections, offering valuable feedback for diagnosing a possible reason for prior failure and devising new, concise, high-level plan that aims to mitigate same failure. Operat- ing under sparse reward signal, such binary success status (success/failure), the model detects the root cause of failure by considering the current trajectory alongside its persistent memory.
yh.r = Mr (M.(s,,,, Tki GR
44)
Reflection prompt Wi,
This self-reffection feedback yk is appended to the actor prompt to prevent repetitive errors in a specific environment in future attempts. Consider multi-step task, wherein the agent failed in the prior trial. In such a scenario, the retrospective model can detect that a particular action, denoted as ag, led to subsequent erroneous actions and final failure. In future trials, the actor LM can use these self-reflections, which are appended to the prompt, to adapt its reasoning and action steps at time t, opting for the alternative action a'. This iterative process empowers the agent exploit past experiences within a specific environment and task, thereby avoiding repetitive errors.
Memory Module The actor model generates thoughts and actions, by conditioning on its recent interactions (short-term memory) and reflection responses (long-term memory) in the text prompt.
Short-term memory. The trajectory history t; of the current episode serves as the short-term memory for decision making and reasoning.
Long-term memory. The self-reflection responses that summarize prior failed attempts are ap- pended the actor prompt as the long-term memory.
To facilitate policy optimization in Section 4.2, store the instructions and responses of the ret- rospective model of each trial, together with the episode returns in a local dataset, which we call replay buffer. We sample from the replay buffer to fine-tune the retrospective model. The long and short-term memory components provide context that is specific to given task over several failed trials and the replay buffer provides demonstrations of good and bad reflections across the tasks and environments, so that our Retroformer agent not only exploits lessons learned over failed trials in the current task, but also explores by learning from success in other related tasks.
Replay buffer. The memory DRI. which stores the triplets (KK,i Wa,ss GR.a) of the reflection in- struction prompt Ta.i, reflection response yk.s and episode return GK of trial i and task k.
Reward Shaping Instead of exactly matching the ground truth to produce a binary reward, soft matching (e.g., fl score) whenever possible to evaluate the alignment the generated output with the expected answer product as the reward function. The details are in Appendix C.3.
5

---

Published as a conference paper at ICLR 2024
4.2 POLICY GRADIENT OPTIMIZATION
The actor model M. is regarded as an frozen LLM, such as GPT, with inaccessible model parame- ters. In this scenario, the most direct approach to enhancing actor performance in a given environ- ment is by refining the actor LM's prompt. Consequently, the retrospective model M,, a smaller local language model. paraphrases the actor's prompt by incorporating concise summary of errors and valuable insights from failed attempts. We therefore aim to optimize the M, model using en- vironment reward. The desired behavior of M, is improve the actor model Ma in next attempt. Hence, the difference in episode returns between two consecutive trials naturally serves as a reward signal for fine-tuning the retrospective model M, with reinforcement learning.
Rellection prompt
PPO trainer
Reflection response V
Rotings
Diagnoue
stuck where
search for Fnglish
(npue) Answer INCORRECT. Reflection:
intigatte
Retrospective
Villiams: Furcabulosal, bun the search tern was too gencrcl.
schr appeared both Peontes from lleaven Kernath
r- 0.92
should lave broken idown searching for the English actor who appeared both TV sertes.
Inputi trajectory
drocclly looked for the neet toam he couchel after WSL, Previuus triil Qucstion: What the capital of france? Thouuhtt 1: roed smarch Trance and loot for the capital Action
0.31
Figure 3: Policy gradient optimization of retrospective LM using RLHF training pipeline.
Instruction and Response Generation The retrospective model generates a pair of instruction and response at the end of each episode i in the environment k. In the episode i, the actor produces a trajectory т; by interacting with the environment. The reward function then produces a score ri- At the end of the episode, to produce verbal feedback for refining the actor prompt, M, takes the set of [T,, ri} as the instruction Zi, and is prompted to produce a reflection response yk,á. All these instruction ressoosse- pairs (Zk,5, yk,i) across tasks and trials stored to a local dataset DRL, which we call "replay buffer", for fine- teee-tunig the M,.
Response Rating As illustrated in Fig. 2(b), let assume reflection prompt Fi. and the cor- responding episode return Gk.s, and the retrospective model M, generates the response yk.i that summarizes the mistakes in i, which results in the return GR.i+1 in the next attempt + Because the actor is frozen LM and the temperature is low as default (Yao et al., 2023), the injected ran- domness that leads to differences returns AGE, = GR.H+1 Gk. are mostly from the reflection responses Yk,,, in which positive AGr,; indicates better responses that help the actor learn from prior errors, and hence should be rated with higher scores; negative or zero AGs, indicates worse responses that needs to be avoided and hence should be rated with lower scores. Therefore, we approximate the rating score of a reflection instruction reruucttoo--esposeee pair (#2,5, Bk.,) as:
r( Vk..) & Gk,s+1 Gk.5-
(5)
Proximal Policy Optimization The optimization step of Retroformer is visualized in Fig. 3. We use the differences of episode returns as the ratings of the generated reflection responses. The retrospective language model is fine-tuned with the response ratings following the RLHF training procedures (although we do not have human in the loop) with proximal policy optimization (PPO):
LLM (y|z) [y|x)
Lyro = Ezn Ey-LLMy-
ro(x.)) - Blog
L.M
Ref (y/z)
where ()) are sampled from the replay buffer (note there is only step in the Retrospective model's 's trajactory), (I y) the defined reward model, and the second term in this objective the KL divergence make sure that the fine-tuned model LLM does not stray too far from the frozen reference model Re
For offline training, we collected the dataset DRL by rolling out base policy, i.c., the frozen actor LM and the initialized retrospective LM, in the tasks in the training sets for N trials and compute

---

Published as a conference paper at ICLR 2024
the ratings. We apply the standard RLHF pipeline to fine-tune the retrospective model offline before evaluating the agent in the validation tasks. In online execution, we use best-of-n sampler, with the scores evaluated by the learned reward model from RLHF pipeline (Ouyang et al., 2022), for generating better retrospective responses in each trial.
EXPERIMENTS
Extensive experiments are conducted evaluate our method, including comparisons with ReAct and Reflexion performances, and visualization and discussion of agent's generated text and actions.
5.1
EXPERIMENT SETUP
5.1.1 ENVIRONMENT
We use open-source environments: HotPotQA (Yang et al., 2018), WebShop (Yao et al., 2022) and AlfWorld (Shridhar et al., 2021), which evaluates the agent's s reasoning and tool usage abilities for question answering reasoning, multi-step decision making, and web browsing.
HotPotQA
The agent asked to solve question answering task by searching in Wikipedia pages. At each time step, the agent is asked choose from three action types or API calls:
1. SEARCH[ENTITY] which searches the exact entity Wikipedia and returns the first paragraph it exists. If not, will return some similar entities to search.
2. LOOKUP(KEYWORDJ, which returns the next sentence containing keyword in the last passage successfully found by Search.
3. FINISH(ANSWER), which returns the answer and finishes the task.
The agent is asked to perform six different tasks, including finding hidden objects (e.g., finding a spatula a drawer), moving objects (e.g.. moving a knife to the cutting board), and manip- ulating objects with other objects (e.g., chilling tomato in the fridge) by planning with the following action APIs, including GOTO[LOCATION,, TAKE[OBJ], OPEN[OBJ], CLOSE[OBI], TOGGLE[OBJ], CLEAN[OBJ], HEAT[OBJ], and COoL[OBJ], etc.
AlfWorid
WebShop The agent is asked solve a shopping task by browsing websites with detailed prod- uct descriptions and specifications. The action APIs include searching in the search bar, i.e.. SEARCH[QUERY] and clicking buttons in the web pages, i.e., CHOOSE[BUTTON], The clickable buttons include, product titles, options, buy, back to search, prev/next page, etc.
5.2 EXPERIMENT SETTINGS
We GPT-3 (model: text-davinci-0033 and GPT-4 the frozen actor model. For the retrospective model, we fine-tune it from LongChat (model: longchat-7b-16k) The implementation details, which include data collection and model training in Appendix C.
Evaluation Metrics We report the success rate over validation tasks in environment. The agent is evaluated on 100 validation tasks from the distractor dev split of open-source HotPoiQA dataset, 134 tasks AlfWorld and 100 tasks in WebShop, as in (Shinn al., 2023).
Baselines We experiment with two language agent baselines: 1) ReAct (Yao al., 2023). This is the tate-of-the-art frozen language agent architecture, which does not learn from the environ- ment rewards at all, thus serving as baseline for showing how the agent performs without using environment feedback. 2) Reffexion (Shinn et al., 2023). This is the state-of-the-art language agent architecture that the authors identify from literature so far. This agent enhances from verbal feedback of the environment, but does not gradient signals explicitly. It can serve baseline for show- ing the effectiveness gradient- based learning. 3) SAC. Furthermore, we include one online RL (Haarnoja al., 2018) (0, SAC as baseline model for comparison.
algorithm, i.e., Soft

---

Published as a conference paper at ICLR 2024
5.3 RESULTS
We present the experiment results in Table 2 and discuss the details below.
Table 2: Results with Retroformer in the HotPoiQA, AlfWorld and Webshop environments. We
report the average success rate for the language agents over tasks the environment. "#Params" denotes the learnable parameters of each approach. "#Retries" denotes the number of retry attempts.
"LoRA r" denotes the rank of low-rank adaptation matrices for fine-tuning.
Method
#Params
#Retries
HotPotQA
AlfWorld
WebShop
SAC
N=1
27%
58.95%
30%
2.25M
N=4
27%
59.7%
30%
Actor LLM
GPT-3
GPT-4
GPT-3
GPT-4
GPT-3
GPT-4
ReAct
0
34%
40%
62.69%
77.61%
33%
42%
Reflexion
0
N=1
42%
46%
76.87%
81.34%
35%
42%
N=4
50%
52%
84.33%
85.07%
35%
44%
Retroformer 0.53M
45%
48%
93.28%
95.62%
36%
43%
(w/ I.oRA r=1)
53%
53%
100%
100%
36%
45%
Retroformer 2.25M (w/ Lo/AAoA
N=1
48%
51 %
97.76%
97.76%
34%
43%
N=4
54%
54%
100%
100%
36%
46%
Question Answering HotPotQA We visualize the performances of Retroformer against
the baselines in Fig. 4. As shown in Table 2, we observe that our method con- sistently improve the agent performances over trials and the effects of fine-tuned retrospective model (Retroformer) are mostly significant in the first few trials.
Furthermore, as shown in Fig. 4. our agent outperforms the two strong baselines. Specif- ically, the results indicate that our reinforced model provides the language agents with bet-
HotPotQA (100 distractor tasks)
Success (%)
ter reflection responses in early trials, which enables the agents to learn faster, while also
fidrdomma
achieving better performances in the end. Our
Berrof
9773)
Bertedon (2P731
Retroformer agent achieves 54% success rate in 4 trials, which is better than the state- of-the-art 50% success rate reported in (Jang, 2023) that model, i.c., GPT-3 (model: text-davinci-003) as the reflection component. The results show the effectiveness of our policy gradient approach for fine-tuning the agent with offline samples.
Aedevann (0P7-4)
(ofF3)
(om14
0.0
1.5
2.0
3.0
Episode ID
much larger frozen language
Figure 4: Retroffrrer shows faster and con- sistent performance improvement of success rate.
We then examine how the retrospective model is improved with policy optimization by comparing the generated responses from the frozen L.M and the ones from the fine-tuned, reinforced L.M. an example, Fig. 5 illustrates how the uninformative self-reflections from frozen I.IM, which we propose Fig. 1, are tackled by RL. The agent failed in the last attempt because submitted "Teen Tians" and "Teen Titans Go" as the answer to the Teen Titans spin-off series, while the correct answer includes only "Teen Titans Go": The agent forgot its original goal during a chain of lengthy interactions. The self-reflection from the frozen model reiterated the prior action sequences that led to failure as the steps that the agent should have done, which prompts the agent to repeat these steps in the next attempt, resulting in infinite loop. On the contrary, the agent to focus on spinoff series and asks the agent to find the answer in the previous actions and observations with the search results of "Tara Strong". Retroformer presents better credit assignment and root cause analysis abilities, and has the power to generate actionable insights.
reinforced response prompts
Decision Making AlfWorld As showcased in Fig. 6(a), the performance improvement by Retroformer over the frozen baselines is significant and method solves the environment
8

---

Published as a conference paper at ICLR 2024
Question
Prior falled trial
Reflection (Green: Retroformer: Brown: Frezen LM)
T Thought 1: need to search Juilet Staring and find the
CanudianAmme American actress who voiced her, then find the Tean spinolf series she done
lbilldd to Gad the Toon Trans spinolt series that Tura Strong has done
Action I: SeachhJJliitt Stullnng]
for, Insiead, found Lollipop Chaissow, video game that Juliet Strling appeared in. should searched Tara
Observation 1: Could not lind (Paket Staring)... ...
Strong and Teen Tiams Teen Tinns Go! specifically my previous action to find the cornect snewer.
Thought Jubet Starling is character from Lollipp
Chuineow, Action 2: Senech[Lollipo Chainese]
search Lallipep Chainan
Lollipup Chainsew festured Julict
Studing, who was veuecd a Cataalisa-Amerien octress who done roles for what Teen spinoff series?
Observation 2: Lollipop Claissow is
I houghht Jubet Siarling is voiced Tara Strong, so Titass spinoff
My thought process was strong but my exceution was flaneed. 1 should have seaeched for Lollipop Chainere first and looked the voical Juliet Stuling
need
Tara Strong find
series she has done voice roles for.
Canadian-
Action 3: Search]Trea Strong] Observation 3: Tam Lyu Strong...
also should lave looked up Strnng's filnagraphy and scarched for mny voice toles she did specifieally for Teen Tituss Teen Tiuas
Gol Qual:Qutstion The novel was adapted inso a dilm by director known for work another iconic 1980s franchise. What on?Thought
Strong voiced characters Tians and Teen Tians Got, so the answer is Teen Tinas
Thought
novel
and Tecn Titms Gol.
Action 4: Finsti[Teen Thans and Teen Thans Go!]
Figure 5: Response refinement from the reinforced retrospective model. Note that the lengthy ob- servation step in the prior failed trial column is abbreviated for better presentation purposes.
Alfworld (134 tasks)
WebShop (100 tasks)
Success rate 196)
44-
40-
doukiie hgvobrra rosk (FT.3)
70-
36
12754)
Suki (OPT4)
0.0
0.5
10
1.5
2.0
2.5
3.0
35
4.0
Episode ID
Episode ID
(a) AlfWorld environment
(b) WebShop environment
Figure 6: Comparisons of Retroformer against baselines in (a) AlfWorld and (b) WebShop environments under different base Actor LLM and LoRA rank 1,
within retries. Similar patterns are observed that the agent performs slightly better with more learnable parameters (r = 4) and that the improvements are mostly from early retries. We find that the reinforced retrospective model behaves like a summarization model of the prior failed plans and finds the differences of the prior plan with the task descriptions. With the permissible actions seen in the task instructions, this behavior effectively prevents repetitive failures and reduces search spaces.
Web Browsing - WebShop As in Fig. 6(b), the performance improvement by Retroformer over the frozen baselines is observed but the improvements may be limited, when compared with HotPotQA and AlfWorld, with 4% improvement in success rate with 4 retries. This limitation was also observed in (Shinn et al., 2023) web browsing requires a significant amount of exploration with more precise search queries, if compared with HotPotQA. The results probably indicate that the verbal feedback approach (Reflexion, Retroformer) not an optimal method for this environment, but our fine-tuning method still proves effective.
CONCLUSION
In this study, we present Retroformer, an elegant framework for iteratively improving large language agents by leaming plug- in retrospective model. This model, through the process of policy optimization, automatically refines the prompts provided to the language agent with environmental feedback. Through extensive evaluations on real-world datasets, the method has been proven to effectively improve the performances of large language agents over time both in terms of learning speed and final task completion.
By considering the LLM action agent as a component of the environment, our policy gradient ap- proach allows learning from arbitrary reward signals from diverse environments and tasks. This facilitates the iterative refinement of a specific component within the language agent architecture the retrospective model, in our case, while cireumventing the need to access the Actor LLM parame- ters or propagate gradients through it. This agnostic characteristic renders Retroformer concise

---

Published as a conference paper at ICLR 2024
and adaptable plug-in module for different types of cloud-hosted LLMs, such OpenAl GPT and Bard. Furthermore, our approach is not limited to enhancing the retrospective model alone; can be applied to fine-tune other components within the agent system architecture, such the memory and summarization module, or the actor prompt. By selectively focusing on the component to be fine- tuned while keeping the remainder fixed, our proposed policy gradient approach allows for iterative improvements of the component with reward signals obtained from the environment.
REFERENCES
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, et al. Do as i can, not as say: Grounding language in robotic affordances. arXiv preprint 00110222011 2022.
Harrison Chase. Langchain. http:p///tthh..ooo//////aaaaaaaa 2023.
Significant Gravitas. Auto- 2023.
Autogpt.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- ence on machine learning. pp. 1861-1870. PMLR, 2018
Shibo Hao, Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhiting Hu. Reasoning with language model is planning with world model. arXiv preprint arXiv: /4992, 2023.
Eric Jang. Can llms critique and iterate their own outputs? evjang.com, Mar 2023. URL https: /tss://evjjn../ caeaaannccoo//oommmamaaaaaaa cwm/c023333330/00 03/26//222/// 6/ sel f-re ferefflect flectiiosn html.
Aman Madaan, Alexander Shypula, Uri Alon, Milad Hashemi, Parthasarathy Ranganathan, Yiming Yang. Graham Neubig, and Amir Yazdanbakhsh. Learning performance-improving code edits.
arXiv preprint arXiv:2302.07867, 2023a.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with selfbeee arXiv preprint 17315 2023b.
Max, Jonathan Tow, Leandro von Werra, Shahbuland Matiana, Alex Havrilla, cat state, Louis Castri- cato, Alan, Duy V. Phung, Ayush Thakur, Alexey Bukhtiyarov, aaronrmm, alexandremuzio, Fab- rizio Milo, Mikael Johansson, Qing Wang, Chen9154, Chengxi Guo, Daniel, Daniel King, Dong Shin, Ethan Kim, Gabriel Simmons, Jiahao Li, Justin Wei, Manuel Romero, Nicky Pochinkov, Omar Sanseviero, and Reshinth Adithyan. CarperAl/rlx: v0.7.0: NeMO PPO, PEFT Migration, and Fixes, June 2023. URL https: //t:::/ddooo htt.//ii...oggg//////// 5281/ zenodo.... 8076391.
Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. CORR, hbs/1660.0078.. 2016.
Yohei Nakajima. Babyagi. https:://// / http:/g.tt......ooooooooooe www./oooeinaaeeeooooooooooooooee mima bbayyagabbaaa.aao 2023.
Reiichiro Nakano, Jacob Hilton, Suchir Balaji. Jeff Wu, Long Ouyang, Christina Kim, Christo- pher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint
2021.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730-27744, 2022.
Joon Sung Park. Joseph C O'Brien, Carrie Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra human behavior. arXiv preprint arXiv:2304.0344 03442, 2023.
10

---

Published as a conference paper at ICLR 2024
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. arXiv preprint arXiv:2335.18829 2023.
Timo Schick, Jane Dwivedi-Yie Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. arXiv preprint arXiv:2302.04761, 2023.
John Schulman, Sergey Levine, Philiip Moritz, Michael I. Jordan, and Pieter Abbeel. Trust region policy optimization. CoRR, abs/1 0024150220577,, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, 0an//777.00647,, 2017.
Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugginggpt: Solving ai tasks with chatgpt and friends in huggingface. arXiv preprint arXiv:2303. 2023.
Noah Shinn, Federico Cassano, Beck Labash, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. arXiv preprint arXiv:2303.11366, 2023.
Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. ALFWorld: Aligning Text and Embodied Environments for Interactive Learning. In Proceedings of the International Conference Learning Representations (ICLR), 2021, URL https: // atw.vv.vvvvv..////v//.aa./aa.ava org/ abs/ 2010. 03768.
Charlie Snell, Ilya Kostrikov, Yi Su, Mengjiao Yang, and Sergey Levine. Offline rl for natural language generation with implicit language q learning. arXiv preprint arXiv:2206. / 1871, 2022.
R. Sutton, D. Mcallester, Singh, and Y. Mansour. Policy gradient methods for reinforcement learning with function approximation. In Advances in Neural Information Processing Systems 12, volume 12, pp. 1057-1063. MIT Press, 2000.
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. The MIT Press, second edition, 2018 URL http:: http:://iccommeeeee.eee deeeeedddeee.... www.:nettttttteeeeeoeebeee.e. 2nd. html.
Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, and Nathan Lambert. Trl: Transformer reinforcement learning. btttps://githi wow.://///o///wee//tttlllllll 2020.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903 2022.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. Conference on Empirical Methods in Natural Language Processing (EMNLP), 2018.
Shunyu Yao. Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Pro- 2022.
cessing Systems,
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), 2023.
Xingdi Yuan, Marc-Alexandre Côté, Alessandro Sordoni, Romain Laroche, Remi Tachet des Combes, Matthew Hausknecht, and Adam Trischler. Counting to explore and generalize in text- based games. arXiv preprint arXiv: 11525, 2018
11

---

Published as a conference paper at ICLR 2024
Appendix for
"Retroformer: Retrospective Large Language Agents with Policy Gradi- ent Optimization"
A CHALLENGES
Although LLMs are not designed to handle tool use or take actions, it has been observed (Gravitas, 2023; Nakajima, 2023; Chase, 2023) that empirically for text-rich environment, especially when the actions and states are accurately described using natural languages, LLMs work surprisingly well. However there are still plenty of challenges applying LLM--ased agents. Here we list several below.
Spurious Actions LLMs are not pre-trained designed with an action-agent application in mind. Even some restrictions are explicitly specified in the prompt, the LLM model may still generate spurious actions that are the action space A.
Limited Prompt Length LLM itself is stateless. However, in applications it is preferred to em- power agents with states or memories for better performance. It has been observed that LLM based agents are easy to into infinite loops if the states are not handled nicely. Many LLM agents concatenate all the previous state descriptions and actions into the prompt so that LLM as a way to bestow "state" the LLM. Inevitably this methodology runs into the prompt length issues. As the trajectory grows longer. the prompt runs out of spaces.
Heuristic Prompt Engineering Even though a lot of paradigms have been proposed to improve LLM agents' performance (Yao et al., 2023; Ahn al., 2022), there lack of systematic method- ologies for consistent model refinement. In fact, manual prompt tuning is still widely used in lot of the application scenarios.
Prohibitive Training Most of the well perfoornno LLMs are too large be fit in just one two GPUs. It is technically challenging to optimize the LLMs directly as is done in the the classical reinforcement learning setting. In particular, OpenAl has not provided any solution for RL based finetuming. Most of the issues are caused by the fact that LLMs are not pre-trained or designed with an action-agent application in mind.
B INTUITION
Compared to the LLM-based action agents, classical RL agents, though not able to handle text-based environments nicely in the shot setting, are able to keep improving based the feedback and rewards provided by the environment. Popular RL algorithms include Policy Gradient (Sutton al., 2000), Proximal Policy Optimization Algorithm (PPO) (Schulman et al., 2017), Trust Region Policy Optimization (TRPO) (Schulman et al., 2015), and Advantage Actor Critic methods (Mnih al., 2016).
In this draft we are proposing a simple but powerful novel framework to tackle the challenges men- tioned above. On one hand, we would like leverage the classical RL based optimization algorithms such as policy gradient to improve the model performance. On the other hand, our framework avoids finetuning on the LLM directly. The key is, instead of training the LLM directly. we train retro- spective LM. The retrospective LM takes users' prompt, rewards and feedback from the environment as input. Its output will prompt for the actual L.MM to be consumed. RI. algorithms are employed to optimize the weights in the retrospective LM model instcad of directly on the LLM. In our frame- work the weights the actual LLM is assumed to be fixed (untrainable), which aligns well with the application scenario when the LLM is either too large tune prohibited from any tuning.
Another perspective viewing our framework is, we train a retrospective LM to apply automatic prompt tuning for the LLM agents. In this case, the RL algorithms such policy gradients employed to optimize the prompts. Idcally the retrospective LM can help summarize the past "ex- perience",, the users' prompt, the environments' feedback into condensed text with length limit
12

---

Published as a conference paper at ICLR 2024
so that it is easier for the LLM digest. To some extent, in setting the original LLM can be considered as part of the environment since its parameters are all fixed.
C IMPLEMENTATION DETAILS
C.I RETROFORMER
Model We use GPT-3 (model: text-davinci-003) as the frozen actor model. For the retrospective model, we instantiate it from LongChat (model: longchat-7b-16k) 16k), which is LM with 16k context length by fine-tuning Ilama-7b on instruction-folloing samples from ShareGPT. In experiments, set the temperature of actor LM zero, i.e., T=0 and top =1 to isolate the randomness of LM from the effects of reflections. We acknowledge that setting a higher temperature value can encourage exploration but it can obscure the impact of the proposed approaches, making difficult to compare against existing baselines with T=0 (Yao et al., 2023; Shinn et al., 2023).
Setup Our proposed learning framework is developed by using multiple open-source tools fol- lows. We use the OpenAl connectors from langchain to build our actor models Ma. During in- ference of the retrospective model, host API server using FastChat and integrates it with langchain agents. The tool can host longchat-7bb 7a-tt-7b-666 with concurrent requests to speed up RL pol- icy rollouts. For fine-tuning the retrospective model, we develop our training pipeline with trl, which supports transformer reinforcement learning with PPO trainer.
We present the details of the specific prompts used and the full agent demonstrations and exam- ples for each environment in Appendix E.
Data Collection For HotPotQA environment, We collected 3,383 reflection samples by running the base rollout policy for trials (N = 3) for 3,000 tasks the training set, which 1,084 Instruction-respons. pairs have positive ratings. For AlfWorld, we collected 523 reflection samples and for WebShop, we collected 267 reflection samples.
Training We fine-tune the retrospective model M, with 4-bit quantized LoRA adapters (r1 or r=4) on the offline RL datasets with epochs=4; batch size=8; Ir=1.4e-5. The number of trainable parameters is 0.53M (0.015% of Ilama-7b) 2.25M. Since longchat-166 is based Llama, used the default llama recipes for finetuning. Specifically, we first run supervised fine-tuning trainer on the samples with positive ratings for 2 epochs and then the RLHF pipeline, including reward modeling, and RL fine-tuning with PPO, on the whole offline rating dataset using the default settings for Ilama-7b model. We list the key hyperparameters here:
Supervised Finetuning: learning rate=le-5, batch size=32, steps=5,000
Reward Modeling: learning rate=2.5e-5,, batch size=32, max steps=20,000
Policy Gradient Finetuning: learning rate=1.4e-$ max steps=20,000, output max length=128, batch size=64, gradient accumulation steps=8. ppo epochs=4
Reproducibility All experiments are done in Google Cloud Platform (GCP) GKE environment with A100 40GB GPUs. The code can be found in https:/ poos://anonymm /aannyyoouu 4open. scissceie..... z/Retroformer: F107. We plan to open source the code repository after the review period.
Algorithm The offline PPO algorithm we used for finetuning the Retrospective component in this data collection, reward model learning, and policy gradient finetuning. We use the offline ratings data to train a reward model first, and plug in the reward model for PPO finetuning.
paper is presented below in Algorithm 1. It contains three steps: offline
13

---

13. end for Published as a conference paper at ICLR 2024
Algorithm Retroformer with Policy Gradient Optimization
Initialize TEXT- TEXT-DAVINCI-O02 as the Retrospective model with LONGCHAT- 16K. Set the maxi- mum trials for rollouts as N 3. The temperature used for sampling i, 0.9. Step 1: Offiin Data Collection. Collect multiple rollouts for each environments k (k: 1, K) for the tasks in the training sets and save as DRI-
for episode f I.... N do for source domain k == I..... K do
Receive trajectory [8k,i.T 0k,i,, T'k,i, and episodic returns Gk, for task for unsuccessful tasks do
Randomly sample pair of reflection responses (yk... yk.j/ with Retrospective LM tem- perature set t,s with the same instruction prompt defined in Eq. (4).
(2)
Roll out the next episode with Yk.j, and receive the episodic returns (G G(2) k.++)).
Compute reflection response rating by r(JRk, Ws,,s) - GR.L+I GR.i in Eq. (5).
10:
Label the response with higher ratings as the accepted response while the lower response labeled as the rejected response.
end for
12: end for
14. Step Reward Model Learning. Use the REWARDTRAINER TRL train a model for
classifying accepted and rejected responses given instructions.
15: Step 3: Policy Gradient Finetuning. Plug-in the trained reward model and use PPOTRA ANNER in TRL to finetune the Retrospective model for generating reflection responses with higher ratings.
C.2 BASELINE: SOFT-ACTOR CRITIC AGENT
Traditional reinforcement learning methods have been recognized to perform well within the same framework of cooraattoonfeeddaac-leeee
We include one online RL algorithm, i.c. Soft Actor- Critic (Haarnoja et al., 2018), or SAC as baseline model for comparison. Given that the three environments are text-based games, inspired by (Yuan al., 2018), we do mean-pooling for the embeddings of the generated text outputs, such as "Search/It Takes Familyl" as the agent actions. Therefore, the action space is continuous and of 768 dimension. We apply LoRA adapters with 4 on the agent Action model instantiated from longchat- and use SAC to do the online updates, with discount factor gamma=0.99 interpolation factor polyak=0.995, learning rate=0.01, entropy regularzation alpha=0.2, and batch size=8.
C.3 REWARD FUNCTION
HotPotQA Fl reward is used in the HotPorQA environment for comparing the matching of a generated answer question against the ground truth answer. After removing the stopwords in both answers, we calculate the number of common tokens in two answers. Then Precision is # of common tokens divided by generated answer tokens and the Recall # common tokens divided by # ground truth answer tokens. We can then compute fl from precision and recall.
AlfWorld The binary success (1) and failure of the tasks the end of episode is used as the reward.
WebShop In each episode, the agent receives reward r R(ST, a) in the end at timestep T, where choose [buy], is the product chosen by the agent in the final state sT, and Y, and Yop are its corresponding attributes and options. The reward is defined as:
Unu n Yaul + Uupt n Yopl - 1 [3/ricc S Uprice
Ptype
|nnn + |Uppl +1
where the type reward Faype = TextMatchhf (") is based on text matching heuristics to assign low reward when and y have similar attributes and options but obviously different types of products. For example, "butter" and "plant- based meat" differ in types but may both contain attributes "cruelty-free", "non-GMO", and option "size: pack of 2".
14

---

Published as a conference paper at ICLR 2024
D ADDITIONAL EXPERIMENTS
Frozen model response
Reinforced model response
think the issue might have been with search. was not able find paragraph, which why not able find James Heary Miller's wife. should have scarched for "Miller's wife" instead of "James Honry Miller's wife" to avoid this confusion.
I got stuck a loop where I kept searching for the same person.
the "Miller"
Henry Miller, instead Beatrice Sylvas Wickems, It's pussible
may mishcard misunderstood the question. To avoid
this, should have written down the question and made understood it correctly before beginning the scarch. Additionally, should searching "James Henry Miller's wife" instead "James Henry Miller".
New Plan:
To avoid confusion, I will search for "Miller's wife" instead of "James Henry Miller's wife" and directly look the nationality of the wife.
Figure 7: Response refinement from the reinforced retrospective model with structured format.
Structured Reflections and Action Plans. We discover emergent behavior of the reinforced model that it can automatically paraphrase the original responses into two separate structured sec- tions, namely Reflection section and New plan: section, although not being explicitly trained or prompted for. One such example is shown in Fig. 7. The paraphrased response retrospects in the first paragraph and provides actionable insights next, while the response from the frozen LM terleaved both parts in one paragraph, making it hard to comprehend. We can also observer from Fig. 5 that the reinforced response removes the messy, irrelevant "Next trial:" content the end for cleaner format, which may very likely result from LLM hallucination.
E FULL EXAMPLES
E. ACTOR PROMPT ENGINEERING
An example of the HotPotQA actor language model prompt is shown below.
15

---

Published as a conference paper at ICLR 2024
Sulve question answering Lask with interlesving Thought, Action, Observaticn stepe. Thought can season about the cursent situation, and Action can be chree types:
(1) dearch(entityy) which searches the exact entity Mikipedia and returns the firat paragraph
if it existe. If not, it will return some similar entiries to search.
which returns the next sentence containing the keyword in the last passage successtully found by Search.
(2) Lookup[kkywoork
(3) Finish which
Finishes the cask.
You may take as many steps as necessary.
You have attempted to answer the following question before and failed. The rollowing reflection (e) give a plan to avoid fatling to ansver the question in the same way you did previcusly. then improve your strategy of correctly answering given question.
Reflections:
I should have searched for the title of the book instead of the author. I also didn't 't cteck for waa not question.
the proverb
should
searched for "It
Family Rick Santorun" instead of just *It Takes
Family" to get Uhe vorrect
My seazch was not concise and I did not specify what I was looking for. I also did not cteck similar results, In this case, should have searched for 'It Takes Pamily by Rick Santorun' to correct answer.
question: I: Takes a Family is response to this 1996 book that was published by who Thought 1: 1 need to search for the title of the book, "It Takes a Family", and find who published is.
Action Searchi"Is Takes Family"]
Observation 1: Richard John Santorum san-ToR-am; born May 10, 1958) is American politician, atcorney, author, and political comentator who represented Pennsylvanta in the Urited States Senate from 1995 to 2007, wаa the Senate's shird-ranking Republican during the final six years Lenuze.
Presiden. of United States Republican primaries, fimishing second to Mit: Ronney, Santorun elected to the United States Senate from Pennsylvania in 1994. Re served two terms until losing his 2006 reelection bid. A Roman Catholic, Santorum is social conservative who opposes abortion and same-sex marrlage and embracod a cultural Amendment, auchored
unsuccessfully
2012
image during his Senate tenure. While serving as a serator, Sancorum would
teaching intelligent design. was leading aponsor of the 2003 federal law known as the Par:icl-Birth Abortion San Int. the years following his departure from the Senate, Santorum has worked as consultant, private practice lawyer, and nexs contributor. Be ran for the Republican nomination the 2012 U.S.
pronoted
presidential election. Before suspending expectations by winning 11 primaries and caucuses and receiving nearly four million votes, making hin the zunner-up to eventual nomines Nitt Remney. Sanzorum ran for president again in 2016, but ended his campaign in rebruary 2016 after poor showing in the Iowa caucuses, In Jamuary 2017, sanior political commentator. However, was cerminated from contract. with cMt May 2021 deemed "dismissive".
campaign April 10, 2012,
exceeded
hncame
commenta made about Nativa Americans few weeks prior which
Thought 2: The paragraph mencions Rick Santorun, 3o I need to search "It Takes Ponily by Rick Sancorun",
Action 2: Search["Ic Takes Family by Santorun"]
Takes Family is 2005 book by then Pennsylvenia Senator Rick Santorum, The ti:le is response za the 1996 book It Takes Village by then-Firat Lady Hillary Clinton. In the book, Santorum states that the family structure is necessary, de argues that liberal social policies have devastated the family suructure and
Observation
prevailing socially liberal attitudes sysrematioslly devalued families across the board. argues that the government should take proactive role in promoting strong families.
Thought 3: Tt Takes a Family wаа publ faeed by Rick Sanzorum, so the answer ia Rick Santorum.
Action 3: Cinish[Rick Santorum]
Observation Answer INCORRECT
16

---

Published as a conference paper at ICLR 2024
An example of the AlfWorld actor language model prompt is shown below.
You are in the middle of room, Looking quickly acound you, you see cabinet 16, cabinet 9, cabinat 8, cabinet 7, a cobinet 6, a cabinet 5, a cabinet 4, a cabinet 3, a cabinet 2, cabinet 1, coffeemachine 1, a countertop a councertop 1, diningtable 2, diningcable drawer drawer draver eLoveburner 4, slovebucner eLoveburner sLoveburaes and Loasler 1.
fridge garbagecan microwave ainkbaein 1,
[Few-Shor Demonstrations]
menory
tack
Trial w1l1 pry find different taak comp' wf11 1ook other objects and Ley to complete Lasks related those objects, For example, could Lry Lo find book to read or glass to drink from. will alse try to use the objects I have found in the roon in di ifferent waya to see if that helps me complete teak.
Your tack is heat
and put coffeenschine.
think: To solve the task, need to find and take mug, then heat with stoveburner, then pul it in coffeenschine. ok.
Th middle room. Looking quickly acound you, you see aothing.
think: Pirst need find mug, mug likely appear in countertep (1-2), ciningsable (1-2), draver (1-3), fridge (11, sinkbasin (1), stoveburnes (1-4), toaster (1). I car check one by ene, starting with countertop 1.
ok.
the middle
Looking quickly around you, you nothing.
> to countertop 1
On the councertop 1, you see butterknife 1, cup i, fork 2, and mug 1.
>
think: Now I find a mag (1). Next, need to take it.
mug
countertop
pick the zug from the countertop
think: Now I take mag (1). Next, I need to heat it with stoveburner (1-4).
ok.
> go to stoveburner
atovaburner you see pan
examine stoveburner
On the atoveburner 1, you see pan examine stoveburner 1
On the atovaburner 1, you see a pon atovaburner
you pan examine stoveburner
On the atoveburner you see pan 2.
STATUS: PAIS
1oop which continually examined stoveburner inetead heating with sieveburne: I should have Laken aug 1 frem counterlop 1, Uhea healed with atoveburner 5, then put it in coffoemachine 1. rt did not help to execute two identical actioons in a row, I will try to execute a different action 1f I an stuck in 1oop again.
plan:
17

---

Published as a conference paper at ICLR 2024
An example of the WebShop actor language model prompt is shown below.
Wobshop
(Tww-Shot Damonstrations]
Instruction:
would like ounce bottle of bright aitrus deodorant for sensitive akin, and price lower than 50.00 dellara
[Search]
Your memory for the task below:
trial 0: in this attempt, wea unsuccessful. accidentally bought product that was more than ny budget 950. initial resulita were not good.
Action: search
bright citrus deodorant sensitive skin]
observation:
[Rack t0 Search]
Page (Total resulte: 50)
>]
[B0785MBC1J]
Bright Citrus Deodorant by Earth Mama Natural and Safe for Sensitive Skin, Fregnancy and DrednnGeeding Contains Organic Calenduls 3-Ounce
810.99
Ginger Fresh beodorant Earth Moma Natural and Safe Sensitive Skin, Prognaney and Rreastfeeding, Contains Organto calendula 3-Ounde
$10.99
Rarrel: and nun-Free Deodorant, Reodorant Men, Essential 3il-Rased Scant, 24-Hour Protection, Cedar Fatchouli Blend, Gentle Senaitive skin (Mountain Sage, oz, 2-Pack1
$15.95
B07вCтEVXY bright check B078CNRC1J first.]
Observation: OX.
Action: click[s078ccRcH)
Observation:
Search]
Prev]
scent :assorted scents] (bright citrus] [calning laverder] [ginger fresh] (simply nen-scents] size [iravel set [4-pack] ] [3 ounce (pack of 1)] (3-onnce (2-pack)] Dright Citrus Deodurant by Earth Mama I Natural and Safe for Sensitive Skin, Pregnancy and Breassfeeding, Contains Organic Calendula 3-Ounce
Frice: $10.99
Rating: N.A.
[eescription] [Features]
[Reviews]
[Ruy Now]
Action: think 3 ounce bottle of bright citrus deodorant for sensitive skin, the item bas options 'bright citrus' and '3 ounce (pack of 1) and seems good to buy.] Observation:
Action:
click (bright citrus]
observation: You have clicked bright citrus.
Action: click[3 ounce (pack 1)] Observation: You olicked (pack
Action: click (Buy Now]
E.2 RETROSPECTIVE PROMPT ENGINEERING
An example of the HotPotQA retrospective instruction prompt is shown below.
18

---

Published as a conference paper at ICLR 2024
You advanced ressoning agent that improve based self reflevtion. You will be given previous ressoning trial in which you were giver access to an Recatore APT ervironment and a question to answer, You were unsucceseful in answering the question either because you queszed wrong answer with Ennihh[Sanawep)) stepa, In few sentences, Diagnoae poaatble reason for fatture and devi new, conet high level plar that aims to mitigate the same failure, Use complete sentences.
or you used up your set number ressoning
examples.
Sewt Demonstrations]
Previous trial: Question: The 2016 Weshington State Oougare were led by coach previcusly helmed which other team?
Thought: I need to search Kashington State Cougars and find the coach of the 2016 team, then the coach previ oualy helmed.
find whichh
Action: Search (Bsshingtnn State Cougars]
Observation: The Rashington State Cougars (known informmlly as the Cougsl are the athletic teana that represent Washingten State Eniveraity. Located in Putlman, Kashington, RSU is nember the Pac-12 Conference In NCAA Division The sthletic program comprises women's sports and six nea's intercollegiate sports, and also effers vacious Intranural sports.
Thought: The parsgraph does not mention the soach of the 2016 team. need to look up "cosch"
find coach of 2016 teat. Action: seokup[coach,
Observation: (Result 2/5) During the 2001-03 sessuns, the Cougar: football Leams were distinguithed by three ten-win seasons, three top ten poll zankings, and appearances in the Sun, Rose,
Roliday Bowls. The Cougara shared the Pac-10 title KSU's 31at head cosch, in four seasont. Mike Leach coached the team Erom 2012 to 2019, guiding them to six boul gamea, leach left to cosch Miasissippi State in 2020 and was replaced by Bawali head coach Rolovich. Rolovitch
Atunnua Paul Wuler, fired late November 2011, after compiling 3-40 (.194) record
fired in October 2021 for not complying with the state's COVID-19 vacoine mardate. Defensive coord' nator Jake nickert was named the interim head coach.
Thought: The 2016 Rashington State Cougazs were led by Mike Leach. I need to search Mike Leach and rind which other tean previously yy helmed.
Action: Search (Mike leach]
Observation: Michsel Charles Lesch (March 9, 1961 December 12, 2022) was an American college rootball coach who primartly cosched at the NCAA Division FRS level. He was a two-tine year and mesternind behind the bead cosch Texas Tech University from 2000 to 2009, where he became the winningest coach in school history. After Texas Tech, he coached at Reshington State Universicy from 2012 to 2019, where he recorded the third-most wins of any coach in school history. then cosched death in 2022, Leach was known for directing offenaes using lots of psssing aeveral
national coach of year, three-tine conference coech
the RCAA record-setting siz raid effenre. ll
Mississippi State from 2020 until his
receiveré, in a spreed system known as the sic raid, which Leach developed with Bal Mumme when Numne waa head coach and Tesch waa offenaive coordirato- at Towa Nesleyan, Valdosta State, and Kentucky in the 1990s, Leach's offenses with Numee and later a head coach himeelf, have broken numeroas school and NCAA recorde. On eighteen occesions, Leach led his unrenked team to victory over team ranked in the AP poll, which 19 the most in the AP erg.
Thought: Mike Lesch previously helned Texaa Tech University and Kentucky, Action: Einish(Texas Tech University and Kentucky] Observation: Answer is INCORSECT
Tech University and Kentucky, So answer
Reflection:
19

---

Published as a conference paper at ICLR 2024
An example of the AlfWorld retrospective instruction prompt is shown below.
You will be given the history of a past experience in which you were placed in an envt ronment and given a taak to complete. You were unsuccesstul in completing the task, Do not surnarise your enviconment,
complete the action. that accounts for your mistake with reference to specific actions that you should have taker, sor example, if you tried a and B but forgot c, then devise plan to achieve C with anvironeett-apecifi actions. You will need this later when you are solving the same task, Give your plan after an"a"", Here two examples:
task, Devize cencise, think
slrstegy and path you
atlempt
PeewSShot Demonstrations]
Your task is to: hest some mug and put it in coffeemachine. take mug, heat staveburner,
the task, naed put coffeemachine.
think:
look
You are in the middle of a room. tooking quickly around you, you see nothing.
think: First I need find a nug, A mag more likely to appear in countertop (1-2), (1-3), fridge (1), check by starting with countertop Ok.
dininglable (1-2),
111, eLuveburner: (1-4),
can
look
You are the middle room. Looking quickly around you, you see nothing,
go countertop 1
On the couteerepp 1, you bullerknice
1, lock 2, and 1.
think: Now I find mug (1). Next, I need to take it.
Ox.
take mug
countertop
You up mag from countertop
think: Now I take (1) Next, need to heat it with stoveburner (1-4).
Ok.
go to stovaburner 1
On stoveborner 1, you see pan
examine atoveburner
On the stoveburnes you
> examine atoveburner 1
On the steveburner 1, you see a pan
examine stovaburner
you pan examine atoveburner
On the stoveburaes you see pan 2.
FAIL
New plan: was stuck in 1oop 'n whichh cont continuslly examined atoveburner inatead of heating
mug with etoveburner: 1, I should have taken mug 1 fzon countertop 1, then heaced it with atoveburner 1, then put it in coffeemsehire 1. It did not help to execute two identiorl actions row. try to execute different action if stock Toop again.
20

---

Published as a conference paper at ICLR 2024
An example of the WebShop retrospective instruction prompt is shown below.
placod cavironnenz
to corplete. You were unsuccesstol completing the tast, not curmacize your envi environment but retaer think about the strategy and path took to atterpt to complete the taok. Devise concize, plan of wotioon that
accounts foz your mistake zeference specifle actiona that you ahould caken. There Ewo exarples aclow.
(Pev-Shot Denonatrations)
Instruction:
looking cairy and [Search]
chipe, and than 30.00 dollare
Action: seazch]jsiey free and apple variety pack chips]
Observation:
[Back Search)
Page (Tota] resultx: 50)
[8ext
[80786Е]N28]
Lite Foods Sott Baked Breektast Bazs, Not Free Bars, Soy Free, Dairy Free, Glaten
Vegen, Veriety Pack, foxes (20) Beer (otal))
$100.0
[B018MHV3BQ]
Snjoy Lite Soft Baked Chenyy Bare, Variety Pack, Not. Fiee Baze, Soy Free, Dairy Frec, Glaten Free, Boxes (30
$21.48
[600802X8C6]
Bnjoy Lite Sentil Chipo Variety Pack, Dairy Free Chips, Soy Pros, Rut Feco, Bor 380, Vegan, Glutcn Frec, 24 19.8
$100.0
Action: think[The first txo itene aze ment. ot chige Nant.)
dairy apple voriety pack
Action: c11ck(B008D2X804
Observation:
[Back to Search]
Prev]
flavoz nane [di11 and seus crean) [qarlic parnasan] (light sea salt] (marchorita pizza] [thal chil1 1ime] [vaxiaty pack]
size 10.8 ounce spack ot 24) ounce ipack 12).
$100.0
Bating: N.A.
[Description]
[Peatures]
[Reviese]
[buy Nox]
Action: think[7he options 'varicty pack" and *0.8 cunce (pack ot seens good bey. Observation: OK.
Action: click[variety pack) Observation: You clicked
lons click ounce (peck of 24)]
Observation:
ounze (pack of 24).
Autions click[Buy
STATUE: FAIL
Nexts plans this atleegh, 1 monumessfll. socidentally bought product (hat was $100, which ay budget of Fither way, the search resulta vare not good. Nex: time, kill soarch[vvareety
scarch l'variety pack of chipo"] and then check it the resuits neet the dairy cree and the 930 budget. can procucte.
will continue
ny
21


Reviews:
[{"review_id": "UoFfFj8nHb", "reviewer": "Reviewer_96C3", "summary": "This paper introduces \"Retroformer\", a framework designed to enhance LLM-assited agents. The Retroformer system comprises two language models, the actor and the retrospective model. The actor model executes tasks while the retrospective model provides feedback to the actor model, allowing it to self-improve. Retroformer employs both short-term and long-term memory to shape the rewards. Short-term memory is represented by the actor model's action history while long-term memory is created by appending summaries of previous failures to the task prompt. Experimental results demonstrates the effectiveness of Retroformer in various tasks.", "questions": "- The feedback obtained from the interaction between the agent and the environment is usually sparse, and the computed rating $r=G_{k,i+1}-G_{k,i}$ represents the difference in return between two consecutive episodes. This means that the data used for finetuning retrospect models are not significantly different within a certain period. How does Retroformer address the overfitting issue caused by fine-tuning on a large amount of repetitive data?\n- Are there any differences in the prompts used by Retroformer and baselines methods, and are there experimental results to analyze this part through ablation analysis?\n- What are the possible reasons for limited performance improvements in HotPotQA and WebShop?\n- How much efficiency is gained by using a retrospective model to tune prompts, rather than directly tuning the LLMs?\n- Are there any details about the hyperparameters of PPO?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The overall framework designed by Retroformer is interesting and alleviates some of the shortcomings in previous works.\n- The paper is well-structured with clear explanations of terminology and content, aiding in readability.", "weaknesses": "- The improvements brought by Retroformer are limited. There are no significant improvements in HotPotQA and WebShop, only meaningful improvement is observed in AlfWorld.\n- The experiments are not solid enough. It lacks comparisons with RL methods that have been recognized to perform well within the same framework of interaction-feedback-learning. Additionally, there is no comparison with the currently acknowledged GPT-4 model, which has impressive decision-making capabilities, making it insufficient to demonstrate the contribution of this work.\n- Only the prompt used in the HotPotQA environment is displayed, and it is difficult to determine whether special prompt engineering is needed in different environments. Therefore, it is insufficient to verify the claim of 'automatically tunes the language agent prompts'.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GaVvBZn0iG", "reviewer": "Reviewer_SkxB", "summary": "Summary: the paper introduces Retroformer, an algorithm for training a LLM to optimize a retrospective model which provides feedback on another \"agent\" model's trajectory, incorporating it into a prompt which the agent LLM can condition on for its next trial at the task.\n\nThey do this by rolling out an agent LLM in the environment (where observations and actions are all text), prompting the retrospective model on the trajectory and the final reward (which is computed heuristically), then prompting the retrospective model to output text which reflects on what went wrong and what the agent can do better next time. The actor model then conditions on this text.\n\nThey create many rollouts using this procedure, score them, and finetune the retrospective agent using PPO.\n\nOn HotPotQA, Alfworld, and Webshop, Retroformer shows modest success improvements over Reflexion, the previous SOTA.", "questions": "- I'm confused how the \"f1 score\" reward works.\n- I'd like to see the following additional curves added to Fig 4 and 6 (possibly in an appendix), which might make it clearer how the finetuning contributes:\n  - Retroformer, rolling out the base policy (before RL finetuning).  This would make it easier to see how much of the improvement above Reflexion is due to finetuning vs other algorithmic details.\n  - Retroformer, rolling out the base policy, but with GPT-3 as the reflection model. This would answer the question of whether it's typically a better strategy to finetune a small model or just prompt a more capable cloud model.\n- I'd recommend writing out the exact algorithm you used for PPO finetuning, especially since when PPO is used with language models slight details from the original algo are changed. Also, it's typically an online algorithm, so it would be helpful to detail any changes you made to make it work well in the offline setting.\n- Equation 6 seems to be optimizing for each (x, y) pair individually (i.e. not treating a sequence of (x1, y1), (x2, y2) (x3, y3)... as a trajectory. Is this correct? If so, I'd recommend making this clearer in the text.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper is easy to read and positions itself clearly with respect to related work. \n- It addresses a challenge which has not been solved yet - the problem of how to do RL finetuning with language agents.\n- The proposed algorithm can be used in conjunction with a hosted actor LLM (e.g. GPT4) which cannot be finetuned. This seems important for making this algorithm useful for users.\n- The results show consistent improvements over Reflexion.", "weaknesses": "- The paper's examples of bad/good prompts do not obviously show that the finetuned reflection model produces better prompts. For instance, in Figure 5, the Reflexion plan is better formatted but still doesn't point out that it was incorrect to return two series rather than one. It would be useful to see an analysis of how often the plan correctly identifies an improved plan (e.g. have a human hand-label 100 prompts for Reflexion and 100 from the frozen LM and record their success rates.)\n-\n- See my questions/suggestions below", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pDA7ZPw5zd", "reviewer": "Reviewer_Hrx6", "summary": "The paper propose a RLHF way to tune the prompt for the LLM agent.", "questions": "See the weakness section", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The authors design a RLHF framework to tuning the prompt for the agents and have better scores on multiple tasks.", "weaknesses": "I think the main weakness is that this kind of prompt tuning may not be necessary for a well trained agent. The agent should be well tuned to understand all kinds of different prompts. So the problem itself may not be very significant. Instead of fixing the agents and tuning the prompts, tuning the agents from RLHF feedback may have a more significant effect for the LLM. But this is just my opinion. The authors can discuss whether it's correct or not.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)