Reflect-RL: Two-Player Online RL Fine-Tuning for LMs (2024)

Abstract: As language models (LMs) demonstrate their capabilities in various fields, their application to tasks requiring multi-round interactions has become increasingly popular. These tasks usually have complex dynamics, so supervised fine-tuning (SFT) on a limited offline dataset does not yield good performance. However, only a few works attempted to directly train the LMs within interactive decision-making environments. We aim to create an effective approach to finetune LMs with online reinforcement learning (RL) in these environments. We propose Reflect-RL, a two-player system to fine-tune an LM using SFT and online RL, where a frozen reflection model (player) assists the policy model (player). To generate data for the warm-up SFT stage, we use negative example generation to enhance the error-correction ability of the reflection model. Furthermore, we designed single-prompt action enumeration and applied curriculum learning to allow the policy model to learn more efficiently. Empirically, we verify that Reflect-RL outperforms SFT and online RL without reflection. Testing results indicate GPT-2 XL 1.56B fine-tuned with Reflect-RL outperforms larger open-source LMs, such as Mistral 7B. The benchmarks, dataset, and code involved in this work are publicly available.1

heading: 1 Introduction
text: Large language models (LLMs) have shown promising results in problem-solving, coding, and document retrieval (Mialon et al., 2023). While performing these
˚University of Washington. Part of this work done when Runlong was an intern at Microsoft Research, Redmond. Email: vectorzh@cs.washington.edu
:University of Washington. Email: ssdu@cs.washington.edu
;Microsoft Research, Redmond. Email: beibin.li@microsoft.com
1https://github.com/zhourunlong/Reflect-RL
tasks, LLMs exhibit considerable reasoning, planning, and reflection skills, enabled by prompting techniques like ReAct (Yao et al., 2022), Reflexion (Shinn et al., 2023), Chain of Thought (CoT, Wei et al. (2023)), Tree of Thoughts (ToT, Yao et al. (2023a)), and reasoning via planning (Hao et al., 2023). Some recent studies (Magister et al., 2023; Mukherjee et al., 2023; Mitra et al., 2023) also try to improve reasoning capabilities of smaller models to match those of advanced LLMs.
The reasoning and reflection skills enable LLMs to act as agents and interact with real-world environments (Durante et al., 2024; Cheng et al., 2023), including code interpreters, embodied robotics (Shridhar et al., 2021; Ahn et al., 2022; Tan et al., 2024), games (Park et al., 2023), and many other spaces (Vezhnevets et al., 2023). This interaction ability is closely tied to reinforcement learning (RL), where agents can learn optimal behaviors through trial and error within an environment.

heading: 1.1 Motivations
text: This research is motivated by three distinct application domains within the same system, which include: document querying (Izacard et al., 2022), database searching (Floratou et al., 2024), and coding (Chen et al., 2021). In these applications, a chatbot needs to navigate in a file system to read documents, modify files, and execute code to answer users’ questions. Central to these tasks is the chatbot’s ability to autonomously explore within a repository using system commands, such as, ls, cd src/, cat main.py, similar to the paradigm in Yang et al. (2023).
Interactive chatbot for file systems (NVIDIA, 2024), multi-agent frameworks (Wu et al., 2023), tool selection (Karpas et al., 2022; Patil et al., 2023), and many other industrial applications require interactive decision-making capabilities. Even if LLMs can perform these tasks, they are usually trained heavily with offline supervised learning rather than with online training within complex environments. Moreover, some recent studies have found that LLMs might not be able to correct themselves without external feedback during interactions (Huang et al., 2023). On the other hand, online RL training could enable LMs to dynamically adapt and make informed decisions beyond static
995
datasets. Some recent studies have incorporated RL to align LMs with human preference and to prompt LM for problem-solving (see Table 1 for details). Szot et al. (2023) and Tan et al. (2024) have started contemporary explorations to integrate LMs within interactive RL environments, but these pioneering studies have not fully utilized the LMs’ reasoning capabilities. Motivated by the strength of RL and expansiveness of LLMs, our work aims to fine-tune smaller, faster, and more secure locally-operated LMs that are capable of decisionmaking and adaptation through reflection, which are essential for domain-specific interactive tasks.

heading: 1.2 Contributions
text: In this work, we introduce Reflect-RL, a novel approach to dynamically improve LMs with online RL (Figure 1), applied with Markov decision processes (MDPs) for multi-step decision making. Most of the previous RL-LM works can be categorized into three classes (Table 1): ① treating token-generation as RL, rather than considering embodied tasks, games, or interactive decision making within environments; ② using LMs as agents to augment policy generation with additional textual information, without directly learning from the environment (gradient-free); ③ engaging primarily with single-step bandits rather than multistep MDPs. Our method seeks to improve multi-step decision making in textual environments by integrating techniques from RL and LMs, enabling LMs to adapt more efficiently to complex environments. We summa-
rize our key techniques below.

heading: Key Techniques:
text: ‚ Reflection (Section 4.1.3). We distill reflection abilities for our domain-specific environment from GPT4 (OpenAI, 2023) through supervised learning. The distilled small LM is frozen and deployed as a reflection model (player) to assist the trainable policy model (player) in decision-making. Reflection accelerates training convergence and improves test performance. ‚ Negative example generation (Section 4.2). The reflection data gathered from GPT-4 is unbalanced, with the majority consisting of positive (near-optimal) decisions. To balance the dataset, we generate negative examples by perturbing the GPT-4 trajectories and optimal trajectories. Negative examples enhance the quality of reflection, ultimately leading to better success rates in the benchmarks. ‚ Single-prompt action enumeration (Section 4.3). We incorporate all possible valid actions into a single prompt, allowing the LM to select the appropriate option using only one token. This approach improves upon the normalization techniques in previous works to generate valid actions and also reduces time complexity. ‚ Task-specific curriculum learning (Section 4.4). The core challenges of RL include planning for a long horizon and sparse reward signals. Vanilla policy optimization methods often fail to obtain sufficient useful trajectories efficiently. We incorporate the idea of cur-
2
riculum learning into our pipeline, designing a specific curriculum to guide training by giving extra rewards or scheduling the data order.
New Benchmark for Online RL Fine-Tuning. Additionally, we introduce AutoExplore, a benchmark inspired by industrial applications, along with other benchmarks adapted from previous works. These benchmarks are suitable for both research and application purposes. They can be integrated with either local LMs for training or remote LLMs for in-context inference. Our demonstrations show positive results of LLMs on industrial applications. Both RL training and data generation are made easy by their use.
Paper Overview. This paper begins by discussing LLMs in Section 2 and RL preliminaries in Section 3. Then, we introduce our proposed Reflect-RL in Section 4 and benchmarks in Section 5. The results are presented in Section 6. Finally, we discuss the findings and future directions in Section 7.

heading: 2 Related Works
text: Language models (LMs). LMs play a pivotal role in tasks such as sentiment analysis (Zhong et al., 2023; Wang et al., 2023b), machine translation (Gulcehre et al., 2017; Lample and Conneau, 2019), and automated text generation (Chen et al., 2020; Dathathri et al., 2020), showcasing their versatility and capability in handling complex linguistic structures.
LM agents and multi-agent collaborations. Autonomous LM agents (Bran et al., 2023; Park et al., 2023; Wu et al., 2023; Wang et al., 2023a) underscore LMs’ capabilities of autonomous and collaborative problem-solving. Such agent collaboration can achieve a level of sophistication and efficiency that is difficult to obtain through solo efforts.
Fine-tuning of LMs. Supervised fine-tuning (SFT, Howard and Ruder (2018); Radford et al. (2019)) and reinforcement learning from human feedback (RLHF) are the most commonly used alignment methods for adapting pre-trained LMs to specific tasks. Additionally, LoRA (Hu et al., 2021), QLoRA (Dettmers et al., 2023), and other parameter-efficient fine-tuning (PEFT) algorithms can facilitate this process.
LMs for interactive decision-making. As summarized in Table 1 and discussed in Section 1.2, only a few studies have applied online RL to LMs for making multi-step decisions. Szot et al. (2023) and Tan et al. (2024) are the two most relevant studies.

heading: 3 Preliminaries
text: Notations. For any set X , we use ∆pX q to denote the probability simplex over X . Let the tokenizer be
fixed throughout the paper. For a string s, we use |s| to denote the number of tokens in s after using this fixed tokenizer.
Markov decision processes (MDPs). Reinforcement learning (RL, Sutton and Barto (1998)) problems are usually formulated as MDPs. They enable agents to learn optimal behaviors through interacting with the environment, without human intervention or labeling. A (finite-horizon) MDP can be described as M “ pH, S, A, µ, T , rq, where H is the planning horizon, S is the state space, and A is the action space. µ P ∆pSq is the initial state distribution, which can represent a distribution over tasks. We study deterministic environments in this work as the tasks in our motivations are deterministic. The transition function maps a state-action pair to a state T : S ˆ A Ñ S, and the reward function immediately yields a reward r : S ˆ A Ñ r´1, 1s. Given a (Markovian) policy π : S Ñ ∆pAq, we define its value function and Qfunction as
V πh psq :“ Eπ « Hÿ
t“h rt
ˇ̌ ˇ̌ ˇ sh “ s ff ,
Qπhps, aq :“ Eπ « Hÿ
t“h rt
ˇ̌ ˇ̌ ˇ psh, ahq “ ps, aq ff .
The expected return of a policy π is Jπ :“ Es1„µrV π1 ps1qs, and the goal of RL is to find the optimal policy maximizing Jπ .
When modeling an application as an MDP, we may encounter the fact that each state s has a separate “valid” action space Apsq. Though we can define A “ YsPSApsq, the union could be intractably large. A viable workaround is to define a mapping function fs at each state, such that Apsq Ď tfspaq | a P Au. This formulation works smoothly with our approach named “single-prompt action enumeration” (Section 4.3) where A consists of choices such as 0, 1, 2, . . . , and fspaq maps them to detailed actions.
Policy optimization for MDPs. Policy optimization is an approach to solve MDPs using parameterized policies. Policy optimization techniques for MDPs surround the class of policy gradient (PG, or REINFORCE algorithm, Sutton et al. (1999)) methods, which directly adjust the parameters of the policy in a way that maximizes Jπ . Let πθ be a policy parameterized by θ, then the policy gradient is computed as
∇θJπθ “ Hÿ
h“1 Es,a„dπθh rQ πθ h ps, aq∇θ lnπθpa|sqs .
Here dπθh is the distribution of ps, aq pairs at step h under policy πθ. An update step using policy gradient is θt`1 “ θt ` η∇θJπθt .
3
Proximal Policy Optimization (PPO, Schulman et al. (2017)) is another exemplary method applied in this field, whose details are deferred to Appendix A.
4 Methodology 4.1 Reflect-RL Here, we propose Reflect-RL, an online reinforcement learning fine-tuning method for LMs in MDPs.

heading: 4.1.1 LM as an RL policy
text: We use a language model as an RL policy πθpa|sq where s “ ps1, s2, . . . , sLq P S is the current state (represented by tokens) and a “ pa1, a2, . . . , aKq P Apsq is the generated token sequence (also represented by tokens). Let a:k denote the subsequence pa1, a2, . . . , akq. We apply policy model to multi-step RL tasks, where the language model reads s in the input prompt, and then generate a in the completion.
In environments where states are not represented in natural languages, we need a function ppsq to convert the original state s to make it a legal input for an LM. For instance, p can be a ViT (Dosovitskiy et al., 2020) for images, as used in LLaVA (Liu et al., 2023)); or, p can be a text representation for simple graphs. Naturally, for s1 ‰ s2, we require pps1q ‰ pps2q. With a little bit abuse of notations, prompt ppsq and state s are equivalent throughout our paper.
4.1.2 Training stages of Reflect-RL We propose a two-stage training pipeline for the abovementioned language model policy. An illustration is shown in Figure 1.
Stage 1. Supervised fine-tuning (SFT). The tasks included in this work all require the instructionfollowing capability to a certain degree: for any valid state s, the generated action a should follow an instructed format. For example, the model should output a paragraph reflecting on previous decisions before making the next action, with two parts separated by a special token. For these tasks, we fine-tune LMs with a dataset D comprised of strings which follow the instruction. This process only calculates losses on the completion part.
Stage 2. Reinforcement learning fine-tuning (RLFT). We use reinforcement learning to fine-tune a pretrained language model πθ0 , which can either be a publicly available LM or the one after SFT. This stage proceeds in T update steps. In step t P t0, 1, . . . , T ´ 1u, we use πθt to sample a batch of B trajectories from the environment, estimate Q-functions for each step, then perform updates using the policy optimization algorithm.

heading: 4.1.3 Training details
text: Reflection-aided decision-making. As demonstrated in previous works (Yao et al., 2022, 2023b; Shinn et al., 2023), generating reflection is helpful for improving the decision-making performance, which inspires us to incorporate reflection in RL. We combine the idea of reflection with both SFT and RLFT. Specifically, we first assume access to an independent reflection model R to generate reflections before the policy model πθ makes decisions. Upon observing state s, R generates the reflection R “ Rpsq
4
which possibly includes analyses of current situation and plans of future steps. Then, the policy model generates the action after taking both s and R as inputs. The reflection model R is independent of πθ: it can be either a local, pretrained language model, or a publicly-hosted LLM such as GPT-4 or Gemini (Gemini Team, 2023). One illustration can be found in Appendix F.1.
In our work, we train local LMs in SFT stage using data collected from Azure OpenAI GPT-4 (details in Appendix C.3) to serve as the reflection model pRϕ (Line 21 of Algorithm 1). pRϕ is frozen (denoted as pR) throughout the RLFT stage. The policy model is SFTed using data containing the reflection (Line 22 of Algorithm 1). Formally, let D “ tpsi, Ri, αi, aiq : 1 ď i ď Nu be the dataset (Line 20 of Algorithm 1), with |Ri| “ Li and |ai| “ Ki, then the loss functions are
Lreflectpϕq “ 1 N
Nÿ
i“1
Liÿ j“1 ´ log pRϕpRi,j |si, Ri,:j´1q,
Lpolicypθq “ 1 N
Nÿ
i“1
Kiÿ j“1 ´ log πθpai,j |si, Ri, αi, ai,:j´1q.
Here αi “ αpApsiqq and α is the action enumeration function defined in Section 4.3.
In RLFT stage, we first query pR for the reflection, then incorporate this reflection into the policy model’s input (Lines 28 and 29 of Algorithm 1). The probability of the action is
πθtpa|sq “ Kź
j“1 πθtpaj |s, pR,α, a:j´1q.
Two-player design simplifies the training process. Splitting responsibilities to two players (reflection and policy) can simplify the RLFT stage because the gradients of the policy model do not affect the reflection model. We experimented using the same model for reflection and policy, while computing gradients only on the policy part. Observations (in Appendix F.2) show that such implementation greatly degraded the reflection ability. An alternative single-player approach is to perform RL and SFT concurrently so that the reflection ability can be retained, but this strategy would complicate the training process.

heading: 4.2 Generating Reflection for Training
text: Two components are essential in reflection generation: ‚ Logical consistency. We want a trajectory to be logically consistent, in that the action ah at step h logically follows the reflection Rh at step h. This requirement is critical for the policy model πθ to derive the correct action from the reflection.
Algorithm 1 Training with Reflect-RL 1: Input and initialize: Environment E, batch size B,
prompting function p, action enumeration function α, SFT data size N , pretrained LM M, LLM to generate reflection data MR, number of updates T .
2: Dreflect Ð ∅, Dnegative Ð ∅. 3: for n “ 1, 2, . . . , N do 4: E.reset(), hÐ 1 5: while ␣E.done do 6: sh Ð E.observation() 7: Rh ÐMRpppshq, preflectq 8: ah ÐMRpppshq, Rh, αpApshqqq 9: Dreflect Ð Dreflect Y tpsh, Rh, αpApshqq, ahqu
10: a1h „ UniformpApshqzahq // random action 11: E, E1 Ð E.steppahq, E.steppa1hq 12: hÐ h` 1 13: // Look ahead: reflect after the “wrong” action 14: s1h Ð E1.observation() 15: R1h ÐMRppps1hq, pnegativeq 16: a1h ÐMRpppshq, Rh, αpAps1hqqq 17: Dnegative Ð Dnegative Y tps1h, R1h, αpAps1hqq, a1hqu 18: end while 19: end for 20: D Ð Dreflect YDnegative 21: pRÐ SFTpM, tpR | ppsqq P Duq 22: πθ0 Ð SFTpM, tpa | ppsq, R, αpApsqqq P Duq 23: for t “ 0, 1, . . . , T ´ 1 do 24: for b “ 1, 2, . . . , B do 25: E.reset(), hÐ 1 26: while ␣E.done do 27: sh Ð E.observation() 28: Rh „ pRpppshqq 29: ah „ πθtpppshq, Rh,Apshqq 30: E Ð E.steppahq, hÐ h` 1 31: end while 32: τb Ð ps1, R1,Aps1q, a1; . . .q 33: end for 34: θt`1 ÐPolicy Gradientpθt, tτ1, . . . , τBuq 35: end for
‚ Negative examples. Using optimal or oracle actions to train policy models is a well-established strategy in RL. However, employing this strategy to generate training data with LLM may introduce a bias towards producing predominantly affirmative reflections on previous actions. If such data are exclusively used for training, the reflection model might merely flatter the decisions made by the policy model, without providing substantive self-reflections. Consequently, the model’s ability to generalize to new or sub-optimal actions could be significantly limited. To mitigate this, incorporating negative examples (sub-optimal actions) can help balance the dataset and enhance the errorcorrecting capabilities of the reflection model.
Accordingly, we use two methods to generate the SFT dataset, with two types of special prompts preflect and pnegative.
At step h, we get the state sh from the environment and send psh, preflectq to GPT-4. Here preflect tells GPT4 to first analyze current situation, plan for the next
5
steps, then generate the action. GPT-4 will generate a response, from which we can easily extract out reflection RGPTh and action ah because of GPT-4’s highlevel instruction-following capability. Next we send ah to the environment and increment h until termination. The above procedure generates a logically consistent trajectory τ . The illustration can be found between Lines 7 and 9 of Algorithm 1 and Figure 1.
To get negative examples, we start from τ or an optimal trajectory τ‹ by perturbing each step. For any step h, we first restore the environment to state sh´1, then we randomly pick an action a1h´1 from the set Apsh´1qztah´1u. This perturbed action will lead us into another state s1h. We send ps1h, pnegativeq to GPT-4, where pnegative tells GPT-4 that the last action a1h´1 is sub-optimal, and lets it to find out the reason of suboptimality, plan for the next steps to correct the mistake, then generate the action a1h. The reflection generated at this step is pRGPTh q1. We halt at this step, using only ps1h, pRGPTh q1, a1hq as a negative example.

heading: 4.3 Single-Prompt Action Enumeration
text: The action spaces in the benchmarks are extremely large and state-dependent. Moreover, a valid action spans over several tokens, and has constraints on the token combination. For instance, in ALFWorld, the action spaces can differ across tasks or locations, due to variations in the objects that can be interacted with. A typical valid action is “go to cabinet 10” which contains 4 tokens, while “take cabinet 10” is invalid. However, this valid action may become invalid when presented in another task where “cabinet 10” does not exist. As stated in various works (Ahn et al., 2022; Tan et al., 2024), it is highly possible for the language model to generate a long token sequence that does not meet the constraints.
The remedies proposed by these works share the same spirit. SayCan (Ahn et al., 2022) and Action prompt normalization (Tan et al., 2024) are similar approaches enumerating all the valid actions a P Apsq, calculating the probability πθpa|sq, and normalizing over Apsq. Calculating πθpa|sq using a Transformer model takes Θpp|s| ` |a|q2q time. This approach takes ΘpřaPApsqp|s|`|a|q2q “ Θp|Apsq| |s|2`ř
aPApsq |a|2q time, which is intractable when |Apsq| is large. Here we assume |s| " |a| as in almost all of the benchmarks. For two benchmarks (AutoExplore and ALFWorld) considered in our work, we have |Apsq| « 20, |s| « 500, and |a| « 5 for almost all the states. As a result, action prompt normalization cannot be applied to our benchmarks.
We propose single-prompt action enumeration which shares spirit with many language classification tasks (Zellers et al., 2018; Bisk et al., 2019; Hendrycks et al., 2021) to reduce time complexity while enforc-
ing valid actions. This method works on two sides. On the environment side, we introduce an extra component: the action enumeration function α. Suppose a1, a2, . . . is an order of actions in Apsq, then we compose αpApsqq “ p1, a1; 2, a2; . . .q by explicitly writing down the choice letter i and action ai. α is sent to the policy model as additional input, together with state s and reflection R. On the model side, we restrict the policy model to output exactly one token, representing the choice in α. We also mask out rows of lm head (neurons of the final output layer) that does not decode into a choice letter. With these combined, we are ensured that the generated action is valid. As a comparison with action prompt normalization, the running time of our approach is Θpp|s|`řaPApsq |a|q2q “ Θp|s|2 ` řaPApsq |a|2q, which is strictly better. Here reflection R is considered as part of s without loss of generality.

heading: 4.4 Curriculum Learning
text: Curriculum learning (Elman, 1993; Bengio et al., 2009) is a paradigm in machine learning using a topological ordering of tasks to help with training. Starting with easy tasks, the model can have a faster convergence on hard tasks compared with directly training on them. In this work, we experiment on a curriculum design called “extra reward signal”. For tasks with long horizons and sparse rewards, it is nearly impossible for a policy to sample a trajectory with a meaningful reward signal, thus policy gradient methods will make slow progress. We design the curriculum by manually adding rewards to some “milestones”. In experiments of DangerousTaxi (see Section 5), which requires to first pick up then drop off a passenger while only giving reward after a successful dropoff, we design the curriculum to give a reward after a successful pickup.

heading: 5 Benchmarks
text: Motivated by the LLF-Bench (Cheng et al., 2023), we have created a natural language environment base class (NatLangEnv) that is compatible with the OpenAI Gym framework, characterized by its unique approach of utilizing textual representations for both observations and actions. This adjustment allows us to effectively train and test language models.
AutoExplore. To verify our methodology of Reflect-RL on the exploration example mentioned in Section 1.1, we built a complete benchmark for autonomous exploration. This benchmark contains three components: a AutoExploreSandbox for file protection, a multi-agent system AutoExploreCopilot for interactive decision-making, and a labeled dataset for performance assessment. The AutoExplore environment enables LMs to interact with the file
6
system safely, with the ultimate goal of answering a natural language question specified by users. The labeled dataset is composed of several real-world and synthesized repositories, with over 2500 trajectories. See Appendix C for more details.
This exploration task draws inspiration from Retrieval Augmented Generation (RAG) (Lewis et al., 2020) and InterCode (Yang et al., 2023). RAG’s performance is linearly dependent on the amount of content (e.g., number of files) in the search space, presenting scalability challenges. In contrast, InterCode utilizes a tree-structured search methodology, requiring merely logarithmic space and time. This approach is notably beneficial for expansive search spaces or environments prone to frequent updates (e.g., Docker environments, customized systems). By integrating online RL training into InterCode, our proof-of-concept environment aims to create code interpreter designed for large code repositories.
During interaction with AutoExploreCopilot, each step the agent receives ´1 reward as the cost of time. After 15 steps or the agent explicitly terminates, if the correct file is identified, a `15 reward is given; otherwise a ´15 reward is given. DangerousTaxi. We extended the OpenAI Gym’s Taxi environment to introduce a higher level of challenge, thereby creating the “DangerousTaxi ” environment. This game concludes prematurely if the
player commits any invalid action, such as colliding with a wall, or incorrectly picking up or dropping off passengers at unauthorized locations. This modification crucially elevates the task’s difficulty by eliminating the opportunity for the model to correct its mistakes after a wrong decision—a common allowance in the standard environment.
We applied curriculum learning to DangerousTaxi. In the designed pickup curriculum, we assign a positive reward 20 and terminate the environment after the driver successfully pickup the passenger. In the dropoff stage, the pickup reward is retained, but the driver needs to further dropoff the passenger at destination to receive the full reward.
ALFWorld. Our study leverages ALFWorld (Côté et al., 2019; Shridhar et al., 2020), a multi-turn platform tailored for simulating household tasks by converting the graphical representation of a house into descriptive language. A robot in is required to complete certain tasks based on the descriptions. This benchmark has gained recognition to evaluate LLM agents, with studies like Arabzadeh et al. (2024) demonstrating its efficacy. Our focus on the tomato picking task stems from its optimal mix of simplicity and representativeness.
7

heading: 6 Experimental Results
text: To verify our approach, we apply Reflect-RL on GPT-2 XL (Radford et al., 2019). Table 2 presents a comprehensive evaluation of various models’ performance across different environments. LMs still face challenges in multi-step decision-making in interactive environments, and Reflect-RL has significantly improved their decision-making capabilities in complex environments. This method not only utilizes the inherent strengths of LMs in reflection but also closely aligns with the multi-step decision-making process intrinsic to RL. Our findings highlight the potential of merging advanced prompting techniques with LMs to address complex RL tasks, establishing a new benchmark for future research in this field.
Open source models and commercial GPT models. We evaluated three open-source 7B models with necessary prompt engineering such as ReAct and memory mechanism included. These models all perform poorly on the three tasks, except for Mistral 7B on AutoExplore depth 1. We also examined GPT-3.5turbo and GPT-4 (version 1106) through Azure OpenAI API. GPT-4 can achieve a success rate of 71% in AutoExplore depth 1, 81% in depth 2, and 84% in ALFWorld; meanwhile, GPT-3.5-turbo achieves a success rate of 31% in AutoExplore depth 1, 8% in depth 2, and 6% in ALFWorld. During the evaluation, we noticed potential data contamination of these two models: GPT-4 can sometimes identify near-optimal actions without extensive exploration of the space. In the DangerousTaxi environment, the success rates of the dropoff curriculum for GPT-4 and GPT-3.5-turbo are both 0%. Even though GPT-4 has 70% chance executing a valid action in each step, it is prone to failure upon committing minor errors along the long navigation path during multi-turn interactions. These observations suggest that even powerful LLMs may still need online RL training for multi-turn interactions.
SFT is not enough. Supervised fine-tuning (SFT) has been widely used offline to improve LMs’ performance on specific tasks. However, our results (Table 2) indicate that SFT alone is not sufficient for complex RL tasks requiring multi-step decision-making. While SFT enhances task-specific knowledge, it fails to solve problems requiring deep reasoning, planning, and reflection.
Reflection helps learning. Incorporating reflective processes into LLMs significantly enhances decisionmaking and learning from past actions. Our comparative analysis between models with and without reflection capabilities highlights the importance of reflection for advanced understanding and adaptability in RL tasks. As shown in Figure 2, the curves representing online RL without reflection are constantly below the curve of Reflect-RL. Figure 4 shows a similar result.
Reflecting from mistakes is beneficial. The philosophy of “learning from mistakes” plays a meaningful role in Reflect-RL. Without negative reflection samples, the model’s performance would be worse (in absolute difference) than the model trained with both positive and negative data. For AutoExplore, the test accuracies without negative examples are 33% and 12% for each curriculum, compared with 36% and 17% with negative examples. As shown in Figure 3, the solid curve represents the integration of negative examples into the SFT dataset, and we observed a faster convergence during RLFT.
Curriculum learning (CL) accelerates learning. As shown in the top two curves in Figure 4, CL accelerates the learning curve for complex RL tasks by structuring the training process with challenging tasks. To ensure a fair evaluation, both learning approaches (Reflect-RL with and without CL) are pre-trained with the same reflection dataset during the SFT phase. The curriculum learning approach begins with an initial RL training phase focused on the pickup curriculum, followed by the dropoff curriculum. Without cur-
8
riculum learning, the model is trained directly using the dropoff curriculum, resulting in slightly inferior performance.
Sensitivity of the policy model with respect to the reflection model. For DangerousTaxi pickup subtask, using the same policy model after Reflect-RL, we switch the reflection model to GPT-2 Small 0.12B and Mistral 7B SFTed with the reflection data. The results for GPT-2 Small 0.12B, GPT-2 XL 1.56B, and Mistral 7B are 55%, 58% (as in Table 2), and 64%. This phenomenon indicates that the policy model is not extremely sensitive to the robustness/accuracy of the reflection model as the policy model can easily adapt. Additionally, using a more capable reflection model can improve the performance.

heading: 7 Discussion and Conclusion
text: Risk, impact, and responsible AI. In this study, we adhere to principles of Responsible AI by ensuring transparency, efficiency, and security in both the training and evaluation stages. An exemplar of our commitment is the development of AutoExploreSandbox, designed to reduce the risk of security issues in the file system. Recognizing the importance of ethical considerations and the social impact of our work, we pledge to engage in continuous evaluation of LMs’s performance in multi-step environment.
Limitations. Our study, while comprehensive, acknowledges certain limitations. Although ALFWorld benchmark is multimodal, this study primarily focued on the text representation, leaving the examination of multimodal models and cross-attention encoding of other modalities (such as images and audio) for future work. Comparisons with commercial models is discussed in Section 6, but the proprietary nature and potential biases (e.g., unknown training data) limit a fair comparison with open-source models. Standard-
ized benchmarks in the field are needed for further evaluation. Lastly, the reflection data utilized in our study is generated by GPT-4, which may not fully capture the distribution of real human data. This indicates the importance of integrating more authentic humangenerated data in future evaluations.
Future direction. The primary goal of this study is to create an efficient online RL pipeline for LMs to perform multi-step problem solving. Building on this foundation, future research directions may explore the scalability of Reflect-RL to develop larger foundation models, enabling them to adapt to previously unseen environments with out-of-domain generalization capabilities. The two-player design in our framework may naturally be extended to other multi-agent settings where language models can show their strengths. Another future direction is to train the reflection model in RLFT stage as we freeze it because of the interference with the policy model (Appendix F.2), which will improve the reasoning ability of language models for decision-making tasks.

heading: Acknowledgement
text: We thank Adith Swaminathan, Julia Kiseleva, Ishai Menache, Chi Wang, Yin Tat Lee, and Yi Zhang from Microsoft Research for useful discussions and support.
We also want to thank Jieyu Zhang from University of Washington for their valuable feedback.

heading: Appendix

heading: Table of Contents

heading: A Discussion on PPO 13
text: B Illustrations of Pipeline 13

heading: C Autonomous Exploration Details 15

heading: D Other Benchmark Details 16

heading: E Experiment Details 17

heading: F Qualitative Observations 17

heading: A Discussion on PPO
text: Proximal policy optimization (PPO) is an advanced policy gradient method, which aims to take the largest possible improvement step on a policy while ensuring the deviation from the previous policy is reasonably small. The update step is
θt`1 “ argmax θ
Es,a „ min " πθpa|sq πθtpa|sq Aπθt ps, aq,
clip ˆ πθpa|sq πθtpa|sq , 1 ´ ϵ, 1 ` ϵ ˙ Aπθt ps, aq * ȷ ,
where ϵ usually takes small values such as 0.1 or 0.2. In practice, we found that PPO did not work well. For tasks with a large state space, action space, and a long horizon, the training processes were constantly unstable, with sudden drops of the expected total reward. Such tasks pose high difficulty for the value function estimator to learn the value functions when the policy network changes. Most importantly, inherent randomness (including dropout, padding length in different batches, top p and top k) of LMs results in high sensitivity for the terms of πθpa|sqπθt pa|sq . Though we want the policy to sample actions with small possibilities (e.g., πθtpa|sq ă ε) to encourage exploration, high sensitivity will result in such values becoming to 0 in almost all the future re-evaluations.
B Illustrations of Pipeline
In this section we present detailed versions of Figure 1. Figure 5 is the illustration for data generation. Figure 6 is the illustration for SFT stage. Figure 7 is the illustration for RLFT stage.
13
14

heading: C Autonomous Exploration Details
text: Autonomous exploration in a well-organized repository can reduce the number of reads of files to a large extent. Ideally, if the repository has n files and is organized as a k-ary tree, the best language model only takes Opk ` logk nq (compared to Opnq using exhaustive enumeration) commands to identify the correct file, then proceed with the specific needs of reading, editing, and executing. This serves as an motivation of autonomous exploration benchmark.

heading: C.1 Autonomous Exploration Sandbox
text: AutoExploreSandbox is a sandbox protecting the original repository from modification. An instance of AutoExploreSandbox could be initialized with the path to the original repository, then this instance will create a temporary directory in a specified location (could possibly be a ram disk) and make a duplication of the original repository. AutoExploreSandbox supports two main functions:
1. Executing system commands: For the purpose of our work, commands such as “cd”, “ls”, “cat”, “head”, “tail”, “echo”, “python” and “pip” are supported to enable document retrieval and coding.
2. Tracking changed files: The user of AutoExploreSandbox could call a function to get the list of the changed files and their contents compared to the original status when creating the sandbox.

heading: C.2 Autonomous Exploration Copilot
text: AutoExploreCopilot is an agent mediating between language models, humans, and AutoExploreSandbox. An instance of AutoExploreCopilot could be initialized with a natural language question and the corresponding repository to work in. The main function of AutoExploreCopilot is to give natural language descriptions of the current autonomous exploration task for either human or language models to make decisions. The interaction proceeds in loops (k starts from 0):
• Step 3k`1: Prompting. AutoExploreCopilot compiles a prompt pk given the current status of AutoExploreSandbox, which includes the question, current working directory (cwd) in the repository, files and folders under cwd (optional, can be used to replace ls and reduce interaction), historical commands c0, c1, . . . , ck´1 from the human or language model, and execution result of the last command ck´1.
• Step 3k ` 2: Querying. AutoExploreCopilot sends the prompt pk to human or LM and gets the response. This response may contain excessive information such as analysis of the current situation (which is a typical behavior of GPT-4), so AutoExploreCopilot needs to extract system command ck from the response.
• Step 3k ` 3: Executing. AutoExploreCopilot sends the system command ck to AutoExploreSandbox and gets the execution results. The results contain standard output and standard error, such as the file content after “cat” and runtime error of “python”.
The interaction ends when the response in step 3k ` 2 contains an exit signal stipulated in the prompt. AutoExploreCopilot is capable of prompting GPT-4 to do the entire task, while for the smaller models in this work we only set the goal to be a subtask (file identification).

heading: C.3 Labeled Dataset
text: The licenses are bounded by each open-source repository used in this dataset.
Using GPT-4 from Azure OpenAI service, we constructed a synthetic repository called “Coffee Company”, which contains documents (in .md format), codes (in various programming languages), and database files (in .csv format). This repository contains around 12 million tokens. In addition, we downloaded 12 open-source repositories containing codes and documentations from GitHub.
After collecting the repositories, we built a labeled dataset regarding autonomous exploration. Each datum in the dataset contains the following fields: the name of the repository n, a natural language question q, an answer to this question a, the related file f , and the shortest system command path to reach this file c‹ “ pc‹0, c‹1, . . . , c‹L´1q. As a start point, this work focus on an important step in autonomous exploration: find the correct file f given the natural language question q, so the answer a is for future work.
This dataset is generated in a “reverse question generation” manner. We first enumerate the pair pn, fq, then send the content of f to GPT-4 to let it generate several pairs of pq, aq. We prompt GPT-4 to ask questions on the functionality of the file by requiring it to analyze the file’s role in the whole repository.
This dataset contains 1764 training data (292 user queries), 505 validation data (86 user queries), and 252 test data (44 user queries).
Here is the prompt template for AutoExplore label generation: Below is a text file {NAME} from a
repository. This repository is deployed
as a backend service, providing users
15
with certain services. Users want to use
specific functionality or ask questions
about the services, such as "tell me the
business philosophy of this company" or
"what is the high-level architecture of
the proposed model". These inquiries are
guaranteed to be answered by reading some
text files.
Your task is to first analyze its
content. Then, come up with some user
queries which involves this text file,
along with the answers to them. Use the
following format:
# ANALYSIS
...
- QUERY1: ...
- ANSWER1: ...
- QUERY2: ...
- ANSWER2: ...
----- Text ----- {CONTENT}

heading: C.4 Reflection Generation
text: Here is the system message for AutoExplore reflection generation: You are a helpful assistant to explore
a file system. Given a natural language
task, you need to generate a sequence
of system commands to identify the
correct file. During interaction, you
can only output a single choice number
as response, which comes from a list
of commands given to you. For example,
the possible commands are: ["A. cat
test.py", "c. cd progs", "9. cd .."].
Your answer should be "A", "c", or "9",
not the entire command.
A special command ‘id X‘ is introduced
to this task, which means to identify
the file X as the final answer. Once
you are sure X is the answer, use ‘id‘
to explicitly identify it, then the
interaction terminates. Remember, simply
‘cat‘ a file does not identify it.
Here is preflect: Now analyze the current situation and plan what to do next using
50 words. Don’t give the choice yet. If
you have identified the correct file in
previous steps, you should exit at this
step.
Here is pnegative: The last command opens a wrong folder or file, which is a
suboptimal move. Give reasons why the
folder or file is incorrect and plan what
to do next using 50 words. Don’t give
the choice yet.

heading: D Other Benchmark Details

heading: D.1 Dangerous Taxi
text: OpenAI Gym uses MIT License. Here is the system message for DangerousTaxi reflection generation: Given a problem state, the actions you
have taken, and the observations you
have.
You need to give reflection on your
actions, such as:
- What is the consequence of your
previous action?
- How is your previous action? Good or
bad? Why?
- What is the next action you want to
take if possible? Why?
I might give you some spoiler information
and optimal action for cheating, but you
should not mention that you have seen any
spoilers, optimal actions, or any other
information that you should not know.
Pretend you are smart and just know these
information.
Don’t use any words related to "optimal"
in your reflection.
Keep your reflection concise within 100
words.
For instance,
Because ..., so I ...
The task is to,... I
I found ... So...
etc.
Here is pnegative: The previous actions might contain some mistakes. pnegative is directly appended to the observation prompt ppsq.

heading: D.2 ALFWorld
text: ALFWorld and TextWorld use MIT License, Fast Downward uses GNU General Public License (GPL) v3.0. ALFWorld shares the same system message and pnegative with DangerousTaxi.
16

heading: E Experiment Details
text: We use NVIDIA RTX 4090, RTX A6000, and Tesla A40 for the training and evaluation of our proposed Reflect-RL method. Python, PyTorch, HuggingFace PEFT, and AutoGen are used throughout the project.
All the experiments share the set of hyperparameters in Table 3.
For AutoExplore, it takes around 30 sec per iteration under the setting of horizon H “ 15. For DangerousTaxi, it takes 30 sec per iteration under the setting of horizon H “ 15 in pickup, and 60 sec per iteration under the the setting of horizon H “ 30 in dropoff. For ALFWorld, it takes 90 sec per iteration under the setting of horizon H “ 15.

heading: F Qualitative Observations
text: F.1 Illustration of Reflection-Aided Decision-Making
Figures 9 to 14 show an entire trajectory of our reflection-aided decision-making method. Irrelevant observations and actions are omitted by “...”.

heading: F.2 Same Model for Reflection and Policy
text: Figures 15 and 16 are two log scripts in an experiment. In this experiment, the reflection model pR and policy model πθt are the same. Thus, the gradients computed on the action part will be influence the reflection part. The reflect quality degraded drastically after 1194 iterations of updates.
17

heading: Observation:

heading: Observation:

heading: Observation:
text: 18

heading: Observation:

heading: Observation:
text: 19

heading: Observation:
text: 20
21

heading: Future Work
text: As a start point, this work focus on an important step in autonomous exploration: find the correct file f given the natural language question q, so the answer a is for future work. This dataset is generated in a “reverse question generation” manner. We first enumerate the pair pn, fq, then send the content of f to GPT-4 to let it generate several pairs of pq, aq. We prompt GPT-4 to ask questions on the functionality of the file by requiring it to analyze the file’s role in the whole repository. This dataset contains 1764 training data (292 user queries), 505 validation data (86 user queries), and 252 test data (44 user queries). Here is the prompt template for AutoExplore label generation: Below is a text file {NAME} from a
repository. This repository is deployed
as a backend service, providing users
15
with certain services. Users want to use
specific functionality or ask questions
about the services, such as "tell me the
business philosophy of this company" or
"what is the high-level architecture of
the proposed model". These inquiries are
guaranteed to be answered by reading some
text files. Your task is to first analyze its
content. Then, come up with some user
queries which involves this text file,
along with the answers to them. Use the
following format:
# ANALYSIS. - QUERY1:. - ANSWER1:. - QUERY2:. - ANSWER2:. ----- Text ----- {CONTENT}