Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learning
Authors: Yuexiang Zhai
Hao Bai
Zipeng Lin
Jiayi Pan
Shengbang Tong
Yifei Zhou
Alane Suhr
Saining Xie
Yann LeCun
Yi Ma
Sergey Levine
Venue: NeurIPS 2024 poster (2024)
Keywords: large vision language model
reinforcement learning
Primary area: natural_language_processing
TL;DR: We directly use reinforcement learning to fine-tune large vision-language model using task-specific reward function

Abstract: Large vision-language models (VLMs) fine-tuned on specialized visual instruction-following data have exhibited impressive language reasoning capabilities across various scenarios. However, this fine-tuning paradigm may not be able to efficiently learn optimal decision-making agents in multi-step goal-directed tasks from interactive environments. To address this challenge, we propose an algorithmic framework that fine-tunes VLMs with reinforcement learning (RL). Specifically, our framework provides a task description and then prompts the VLM to generate chain-of-thought (CoT) reasoning, enabling the VLM to efficiently explore intermediate reasoning steps that lead to the final text-based action. Next, the open-ended text output is parsed into an executable action to interact with the environment to obtain goal-directed task rewards. Finally, our framework uses these task rewards to fine-tune the entire VLM with RL. Empirically, we demonstrate that our proposed framework enhances the decision-making capabilities of VLM agents across various tasks, enabling 7b models to outperform commercial models such as GPT4-V or Gemini. Furthermore, we find that CoT reasoning is a crucial component for performance improvement, as removing the CoT reasoning results in a significant decrease in the overall performance of our method.

Fine-Tuning Large Vision-Language Models as
Decision-Making Agents via Reinforcement Learning
Yuexiang Zhai
Hao Bai
Zipeng Lin
Jiayi Pan Shengbang Tong
Yifei Zhou !1
Alane Suhr
Saining Xie
Yann LeCun
Yi Ma²
Sergey Levine
UC Berkeley
2UIUC
NYU
Batch Observations & Rewards
Environments
You are blackjack player. You are observing the current game state, you can choose between ['stand; hit].
Dealer
Please first describe the current state, then output the action.
teget
Player
Pretrained
RL
Vision Language Model
Fine-Tuning
"thoughts": "I have points in total. dealer least points. My total points are large, should stand." "action": "stand"
Actions
Figure 1: Method overview. We propose framework for training large Vision-L Language Models (VL.M) with Reinforcement learning (RI.. At each time step, the VI.M takes the current observation and predesigned prompt as input and outputs an utterance containing chain thought reasoning and a text action. The text action is parsed into the environment for generating task rewards. Finally, we apply RL with the task reward to fine-tune the entire VLM.
Abstract
Large vision- lasguae models (VLMs) fine-tuned on specialized visual instruction- following data have exhibited impressive language reasoning capabilities across various scenarios. However, this fine-tuning paradigm may not be able to effi- ciently learn optimal decision-making agents multi-step goal-directed tasks from interactive environments. To address this challenge, we propose an algorithmic framework that fine-tunes VLMs with reinforcement learning (RL). Specifically, our framework provides task description and then prompts the VLM to gener- ate of-touh (CoT) reasoning, enabling the VLM efficiently explore intermediate reasoning steps that lead to the final text-based action. Next, the open-ended text output is parsed into an executable action interact with the envi- ronment to obtain goal-directed task rewards. Finally, our framework uses these task rewards fine-tune the entire VLM with RL. Empirically, we demonstrate that our proposed framework enhances the decision-making capabilities of VL.M
Project Lead, email: simonzhai @ berkeley.cdu. Project page: https: //rl:v1///1.1....m Sww...m.mttui.. github. Equal contribution, listed in alphabetical order, Appendix for list of contributions.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

agents across various tasks, enabling 7b models to outperform commercial models such as GPT4-V or Gemini. Furthermore, we find that CoT reasoning is crucial component for performance improvement, as removing the CoT reasoning results in a significant decrease in the overall performance of our method.
Introduction
Large vision-languase models (VLMs) [7, 44, 18] demonstrate remarkable capabilities as general- purpose agents in solving various tasks through language reasoning. particular, fine-tuning VLMs with specialized visual instruction following data appears to be key technique for improving the capabilities of VLMs [34, 84, 33, 30]. However, visual instruction tuning may not be optimal for training decision-making agents in multi-step interactive environments requiring visual recognition and language understanding, as visual instruction tuning mainly performs supervised learning on pre- collected datasets without interacting with the environments [22]. Consequently, if the pre-collected datasets lack sufficient diversity to cover a wide range of decision maaiinnnnmaiigg scenarios, visual instruction tuning may fail to improve the VLM agent's 's maiiing capabilities.
To unleash the learning capabilities of VLM agents in multi-step goal-directed decision-making environments, reinforcement learning (RL), method that has proven effective in training multi-step interactive agents [41, 59, 6, 69], naturally offers a paradigm that supports this purpose. However, while RL has been widely adopted for training purely text-based tasks for large language models (LLMs) [60, 50, I, 83), end-to-end VLM fine-tuning with RL for goal-directed multi-step tasks has not yet been studied, to the best of our knowledge.
Our main contribution in this paper is an algorithmic framework that directly fine-tunes VLMs with RL for multi-step goal-directed decision meriiiinn-maiingggg tasks requiring vision- langua-angguag understanding. In our framework, the VLM first receives task description prompt. which guides it to generate task-specific chain-of-thought (CoT) reasoning [75, 73] (blue parts in Figure 1), followed by a text-based action (red parts in Figure 1). The CoT reasoning is designed for efficient explorations by prompting the VLMs to generate intermediate reasoning that leads to the final text-based action. Our framework then parses the text-based actions into executable actions for the environment, which generates potentially goal-directed rewards and the next state for RL training.
To evaluate the effectiveness of method in enhancing a VLM's preiiion-making capabilities, we adopt a 7b model [35] as the backbone VLM and apply our method to five decision-making tasks. These tasks come from two domains: an original domain, which evaluates the VLM's s decision- making capabilities requiring fine-graine visual recognition and language reasoning, and an embodied Al domain [58] focusing on testing tasks demanding visual semantic reasoning capabilities. Empirical results show that our method enhances the cerision-making capabilities of VLMs in both domains, enabling 7b models to surpass the performance of commercial models such as GPT4-V [44] and Gemini [18]. Moreover, our experiments reveal that CoT reasoning is crucial for performance improvement in our RL training. Specifically, we test our method on the same tasks without the CoT reasoning and observe a significant drop in overall performance in both domains.
2 Related Work
Training LLMs or VLMs with RL. RL has been widely adopted for training LLMs and VLMs
[85, 61, 70, 45, 10, 50, 9, 43, 18, 62. 60, 1, 20, 83]. Some studies [85, 61, 45, 10, 43, 18, 62] focus on applying RL from human feedback (RLHF), which involves learning reward models from human feedback before deploying RL. Other research [50, 9, 60. 1. 20, 83] focuses on deploying RL with task-specific reward functions without using human preference data. Our paper is similar to the latter [50, 9, 60, 20, 83] which applies RL to train LLMs on customized reward functions from different environments. There are two major differences between our paper and prior works [50, 60, 1, 20, 83]. Firstly, our method incorporates visual inputs, broadening its applicability to a wider range of tasks that require vision- language understanding or multimodal reasoning [29, 38). Secondly, while previous works do not explore how CoT reasoning affects RL training on large models in general, we identify COT reasoning as crucial component for enhancing RL training. We empirically observe that incorporating CoT reasoning significantly improves the overall performance of RI. training on all tested domains.
2

---

Adopting LLMs and VLMs as decision-makinn agents. Many prior works have studied various methods of using frozen LLMs and VLMs for becision-making. One line of work studies the prompting techniques [75, 14, 79, 78, 74, 31, 76, 47, 71, 48, 24] for enhancing the decision- makinn capabilities of large foundation models, see Dong et al. [14], Yang et al. [77] for detailed survey for other prompting based methods. Our work differs from all prompting-based methods since we directly use RL to fine-tune the entire VLM as recision-making agents. Other studies [42, 64, 4, 52, 11] integrate frozen VLMs ot LLMs into their training pipeline for processing task descriptions or feature extraction, without using text-based actions. focuses on integrating different components from VLMs for downstream RL training. For example, some studies use the VLMs CLIP vision encoder [46, 42, 64] as reward models for training, which differs from our method since we adopt rewards from the environments. Other studies [42, 64, 11] integrate frozen VLMs/LLMs into their training pipeline for processing task descriptions [42, 64, 46] or feature extraction [11] 1]] without using text-hased actions. Our paper differs from these works [42, 64, 11] in two major aspects. From technical perspective, we focus on a more challenging paradigm by directly fine-tuning the entire VLM with RL., whereas previous methods [42, 64, 11] only train additional MLP or transformer layers to connect the frozen LLM/VLM with the action space. More importantly, our method directly interacts with the environments using open-ended text, enabling to utilize the CoT reasoning capability of VLMs for more efficient explorations for decision-making.
Evaluating VLMs as decision masssinn-maiing agents. Previous studies have thoroughly examined the fundamental evaluations of VLMs in non-interactive tasks [3, 37, 80, 32, 65, 81, 16]. Our focus, however, is on evaluating a VLM's decision -aaiingaaiiinnii capabilities in interactive environments that require both visual recognition and language reasoning. Representative interactive environments include purely text- bated environments [13, 28, 72] or embodied AI environments [40, 58, 56, 15]. We adopt the ALFWorld [58] embodied environment for evaluating our method's ' ability to improve VLM's s visual semantic reasoning capabilities. In addition to the ALFWorld embodied AI environment, we also design an original "gym-like" [8] environment to test VLM's decision-make capabilities in tasks that require fine-grained visual recognition and language reasoning.
CoT prompting. Recent studies in prompting for LLMS have demonstrated the crucial role of CoT in enhancing complex reasoning capabilities [75, 26, 17, 73, 82, 79]. Wei al. [75] show that CoT reasoning can significantly boost LLMs' performance across different reasoning tasks by showing that adding simple exemplar-based prompts, leading to better performance on benchmarks such as the GSM8K [12]. A follow-up study [73] proposes novel self-consistency decoding strategy that explores multiple reasoning paths, demonstrating substantial gains arithmetic and commonsense reasoning tasks. Other works [26, 82, 17] have shown that adding prompts to break complex tasks into subtasks and solve them step-by-step significantly improves LLM's 's reasoning capability. Our work differs from these CoT prompting studies as we aim to provide an algorithmic framework that can train VLMS with RI, where the CoT prompting appears as a key component of the framework. In contrast, prior works focus on improving the reasoning capabilities of LLMs with increasingly sophisticated prompting of frozen models.
3 Preliminaries
Standard RL terminologies. We follow the standard notations from classic RL literature [63, 2]. Specifically, we use MM = (S, 7 to denote an MDP, where S denotes the state space, A denotes the action space, P denotes the transition dynamics, r: S x A R denotes the reward function and > E [0, 1] denotes the discount factor. Our goal is to learn a policy π :: S + A that maximizes the overall discounted return max Ex ITo "'r(s,, a:)] where T is the maximum number of steps per episode. Without loss of generality, we use (Taas) E [0, 1] to denote probability of π choosing a at 8.
Adapting the RL formalism to VLMs. We use V to denote the discrete and finite vocabulary (token) space, and we use ymu T to represent the input and output text space, where m and n represent the maximum token length of the input and output sequence. We adapt the RL formalism to VL.Ms by treating the combination of the vision and language inputs to VL.Ss as the state space: S O V.h where O is the space of all RGB images. We view each utterance [1, 83] of the
3

---

language outputs from VLMs as the action space V", Therefore, the input and output of VLM policy with parameter 0 can be written as To O Vm V For example, the Blackjack task shown in Figure 1, each state consists of an RGB image with the cards of the dealer and the player, as well as an input prompt w with maximum token length m, and the text output u = To(o, en)) (with maximum token n. will later be parsed as action to interact with the environment. Similar to the standard RL setting, we use mo (v Jo, v")) E [0, 1] to denote the probability of a VLM policy Ty outputting you with input image and prompt vla.
4 Training VLMs with RL
Compared to classic MLP-based policy networks [53-55, 19], a natural advantage of VLM policies is that they can leverage CoT reasoning for efficient exploration, by performing intermediate reasoning steps that lead to the final decision. However, training a VLM policy Te with RL presents additional challenges. First, the VLM policy (o, v²") directly generates open-ended text rather than vectorized actions in classic policy gradient-based RL methods [53-55, 19], complicating direct interaction with the environment. Even with a parsing mechanism f V" A that maps open-ended text o to a legal action for interaction with the environment, it remains unclear how to estimate the action probability no o, " from the text generation process.
Figure 2 presents an overview of our framework, leveraging the CoT reasoning and addressing the two aforementioned challenges. We design task-specific prompt vi that requires the VLM to generate formatted output you, including the CoT reasoning. Next, we adopt post-processing function f to parse open-ended text into legal action a, that can directly interact with the environment. To compute To (alo, )²), we develop a method to estimate its value based on the probability of each output token in "
St Ot
PPO
Replay buffer
ne ( (u.lo.17
Task description
concat
Legal action space
compute
Desired output:
VLM
log-likellood of oq
r( @z),
9c-1
CoT reasoning
Text action
env.saap
1/2 "thought" T.. "action" "look"
legal action
environment
A diagram of the proposed RL fine-tuning framework. At time step I, the state sı contains an input prompt of"' and visual observation or. The VLM takes s; [or, v!'] as imput and outputs open-ended text vi containing the CoT reasoning, keywords "action": "a,", and log-likelihood ot We first apply a post-processing function f on ot to obtain a legal action o, which can interact with the environment. Then, we input to the environment for obtaining reward r(s,, oz) and the next observation O Afterward, we devise a method to compute numerical value of mo(ae lo,, uf"). Finally. we use r(s,, a.) and nh (ar lo,, w/") for the RL training.
Figure 2:
The remaining Section is structured as follows. First, we describe the format of our input prompt e and the desired output oi (Section 4.1). Next, we present the post-processing function (Section 4.2). Then, we introduce a method to compute numerical value of тe lot, w!") (Section 4.3). Finally, we conclude our framework Algorithm (Section 4.4).
4.1 Prompt Design for Domain-Specifie Outputs
For each task M, our input prompt oi contains a description of the task, the legal action space the current observation, and the desired output format (including the CoT reasoning). Our desired output out contains a CoT reasoning followed by the keywords "action" "a," for post-processing. Figure 3 provides particular, we define a fuunction h which constructs w from the current observation o;: v.t h((o), to accommodate for tasks that may contain observation-dependenn information." We provide additional examples of in and out in Appendix B.
example of our input prompt oi" and the desired formatted output w In
E.gg. the alfworld environment (to be introduced in Section 5.2) contains an Abservation-dependee admissible action space.

---

CoT prompt Di for task M You are trying to solve task M. (Description of the task), You are ohserving the current status of the task. The action space of M is (text version of all legal actions é A]. Your response should be valid json file in the following format:
"thoughts": "(first describe the current status of the task, then think carefully about which action to choose]".
"action": [Choose action "a AA))
Formatted text output vi
"thoughts": "I am solving task T, given the current status of the task, 1 should choose oz".
"action": "at"
Figure 3: A template of our input prompt and output text. The blue part represents the CoT reasoning and the red part is the text-hased action. Note that the CoT reasoning may contain other task-specific descriptions, see Appendix B more details.
4.2 Post-Processing Open-Ended Text for Legal Actions
Our post-prose sett-poceesing mechanism involves both oi and f. In the input prompt vf", we directly ask the VLM to output text-based action in the format of "action" "a," (see Figure I and Figure 2 for examples). After obtaining puu our post-processing function directly searches for the text-based keywords "action": "a," from oh and maps it to legal action a,, either in symbolic or in text depending on the task of interest. For the case shown in Figure I, will map o the symbolic operator that represents the action "stand" in the Blackj ack task (to be introduced in Section 5. as the Blackjack task takes symbolic actions as input. For the alfworld [58] environment shown in Figure 2, f will map v.tt to the text "look", because the alfworld environment takes text-based actions as inputs.
However, VLMs are not always guaranteed to generate u that contains the keywords "action" "a," ", even when we explicitly request a formatted output from v To continue the RL training when vith does not contain any legal action, we perform random exploration by selecting legal action a, E uniformly random. Mathematically, defined as follows:
f
O,
"action": "a" ouu
(4.1)
(n) otherwise.
4.3 Estimating Action Probabilities of VLM Policies
To estimate the action probability log (a lo., wi)) (or equivalently log Te (arlor, of")) for pol- icy gradient-based methods [55], naïve calculation is directly using log TT (ot" w(t) as log (aaaa,o v."), by summing the ligliiilihood of all tokens in uith. This is because
Ploc, vi U?
log no(w" out los, v. = log
P(or, o!")
(4.2)
P(o, of (12)) P(o,, of" "iai) P(or, oi U|=- P(o,, wi Mi1)) P(or, vfr)
Plor, vi, Upon))
P(o of v(s))
= =log
= Σ log
P(o,, vi u[s-n)
In the equation above, we use to denote the output token woe for simplicity, and we use [ij) to denote the first tokens in whth and we slightly abuse notion by using P(o,, )ia)) to denote P(or, vit) in the log summation. Hence, a natural way compute a numerical value for log Te(aa or, 0 Σ log
P(a.,)))"))) Plar,e,
However, the naive calculation log (a; or, v!t) E.= log P(o. "f ..4) P(o,, - may be ideal for computing my (ag oe, vf()) since our formatted output eqte also contains CoT reason- ing. This is because in oou =[ot v. the CoT reasoning tokens wf are generally much
5

---

longer than the action tokens o (see the blue and red parts in Figure for examples, and see Table I for relative scaling of their sum log- log-likelihood). Hence the naive computa- tion log Ta(a (t: v < log No ( (0t, xt log To (T( (0t, ty a(th) will make log To (ot, w largely determined by the CoT tokens log ( Ot; vf",, which is practically undesirable because our post-processing function f only relies on 2t for decision-making.
As shown in Table log (o( los, e typically has a much larger magnitude than log P( Or, vi w(") across all tasks we have tested (in terms of absolute value). Hence, to mitigate the effect of the CoT to- kens, we adopt scaling factor À [0, 1] to scale down log To (o( (oz, v/")) for obtaining regularized version of log my (az Ot, o("),, which results
log t0
NL
BJ
EZP
P24
ALF
-3.4
-2.2
-9.0
-37.6
-20.3
0.0
0.0
-0.4
lo,,
Table 1: The absolute values of sum log probability of o! is much larger than uf". Each number averaged among 1000 samples on our evaluation tasks to be introduced in Section 5.
log To (arloo,, v²")
(4.3)
A log ( Jos, v!") log to (v. (os; vi'; v!
Empirically, we observe the scaling factor À could largely affect the final performance. As we will show in Section 6.2, choosing extreme À value (close to I or 0) will degrade overall performance. All of our experiments adopt À E 0.5).
4.4 Formal Implementation
Putting the prompt construction function h (Section 4.1), the post-processing function f (Section 4.2), and the computation T oc, w(t) (Section 4.3) together, we conclude our method Algorithm 1.
Algorithm Training VLM with RL
Input: An environment env, initial VLM with parameters 0g-
2: Input: A post-processing function f. CoT reasoning scaling factor A.
3: Input: Replay buffer size B, maximum episode length T.
4: for k = O...., K
do
5:
= 0
b Reset RL time step Reset the initial state Generate wi from or, h defined in Section 4.
6: O: env.reset()
7:
etth h(()))
8:
B. 0
b Initialize an on-policy replay buffer
9:
while [Bx < B do 0t TB, (or, v
10:
Generate text output
11:
0, = f(v(")
> Obtain a legal action from with f is defined in Equation 4. (vtt (br)) + log ne, (1.2 loe, lo,, wi afth))
12:
log no, (0ilor, Dit) Alog no,
Equation 4.2
13:
Tt. 01+1 = env.step(a,
14:
B₁ B3 U ((o, ae, re, 2tt log ne, (aclor, ofr))}
> Add data the buffer By
15:
I 1= +++
16:
if / T then
17:
1 -= 0
D Reset RL time step if the maximum step reached
00 env.reset()
Reset environment
19:
end if T :((o))
Prepare the next o
21:
end while
22:
Run PPO [55] with data B₁ to obtain 0x+1
23:
end for
Output: 0k.
Evaluation Tasks
How does our method improve a VLM's decision-making capabilities tasks that require fine- grained lanionnl reasoning semantic understanding? To study this question, we adopt two different domains: gym_carde and alfworld [58]. Our original cards domain is a "gym-like" environment [8] containing four tasks designed to test the decision-making capabilities of VLMs.

---

These tasks require fine- tne-grained visual-language reasoning, specifically focusing on recognizing numbers for arithmetic reasoning. In addition, we also adopt alfworld [58], which assesses the decision-mmking capabilities of VLMs in an embodied Al setting that demands visual semantic understanding. We present some examples of the visual observations of each task in Figure 4. We do not include standard image-based Atari benchmarks [5, 39] due to limited computation resources."
Target: 5
Player
Current: 1
11 10 51
(a) NumberLine
(b) EZPoints
(c) Points24
(d) Blackjack
(e) alfworld
Figure 4: Examples of observation of our evaluation tasks. (d) are from our original gym_cards domain. (a)-(c are deterministic tasks with increasing difficulties; (d) is stochastic task.
5.1 Gym Cards
Our gym_ cym_cards domain designed to evaluate VLM's decision-making capabilities requiring fine-grained vision recognition and language reasoning. More precisely, tasks in the gym_ cards domain require the VLM to recognize the numbers (potentially from cards) and utilize the numbers for language reasoning. As depicted in Figure 4, the first three tasks-NunberLa, Points24-arr deterministic, and developed to assess the VLMs ability identify and process num- bers or mathematical operators each time step. These tasks increase in complexity: NumberLine requires recognition of two numbers in an image, EZPoints involves identifying numbers from two cards, and Points24 extends to recognizing four cards. The Blackj ack task challenges the VLM further by requiring the agent to reason based on visual information and adapt to stochastic outcomes. This subsection outlines the goals of each task, and leave the detailed descriptions of their state spaces, action spaces, and reward functions to Appendix B.
EZPoints, and
NumberLine. In this task, the goal is to move number to the target on synthetic number line. At each state $t, the visual observation o contains two lines of text: "Target: x" and "Current: yt". The agent needs to move the current number Ye to the target number æ, by outputting text uoat that interacts with the discrete action space { ("+". "_"). Mapping the uf to "4" or will increase decrease the current number by 1, respectively.
EZPoints. In this task, the goal is to output a formula using the numbers the cards that evaluates to 12. At each state st, the agent observes image of two cards and text version of (potentially incomplete) "formula" below the cards. The goal is to use all numbers the cards (only once) to compute 12. The action space contains natural numbers in [1, 10), well operator in {"+", "*", " "="}. At each state 8t, only operators and numbers that appear in the cards are legal actions, and "J", "Q", or "K" treated "10". In particular, if the output text anth is mapped to a legal action a, at state 8t, the text version of a, will be appended to the "formula" in the current image 9t resulting St+1, otherwise S will remain the same St.
Points24. In this task, the goal to output formula using the numbers in the cards that evaluates to 24. The Points24 task is harder version of EZPoints it contains 4 cards, hence requiring the VLMS to generate a longer formula. The rules of Points24 are similar to EZPoints, despite two minor differences: the Points24 task requires the VLM to compute target number of 24, and its action space contains more operators: {"+",
"g "=").
Blackjack. In this task, the goal is to win the current blackjack game. At each state st, the visual observation o, consists two cards (one face-down) from the dealer and all cards from the player.
'Image- based Atari tasks generally take at least 2 million environment steps to reach a reasonable perfor- mance [23]. Our method needs roughly 30 hours to run 15k environment steps due to the model size of the backbone VLMs, which requires roughly half' year to run 2 million environment steps.

---

The agent's goal in this task is to win the current game, by outputting text vt that can be mapped to ("stand", "hit"). J. The agent will receive one more card if vt is mapped to "hit", and the game will terminate if oeee is mapped to "stand".
5.2 ALFWorld
While the gym_cards domain designed to assess the VLM's arithmetic reasoning requiring fine- grained visual recognition, the alfworld environment aims at testing VLM's decision-making tasks requiring visual semantic understanding.
ALFWorld. The ALFWorld embodied environment [58] is combines a text-based interactive environment [13] with a large vision-languaa instruction following dataset [57]. It contains different types of conliconditioned tasks ("Pick & Place", "Examine in Light", "Clean & Place", "Heat & Place", "Cool & Place", and "Pick Two & Place" ") and the agent's goal is to navigate in the environment via text-based actions (e.g., "go shelf 1", "examine sidetable 1"). Unlike our original cards environment, where all states share the same action space, the aloworld actions are only available
environment contains state-dependent admissible action space certain states. For example, if the agent's 's goal is to "put some pillows on armchair", then the agent can only put pillow after picking up a pillow. Hence, to incorporate the state-dependent admissible action set, our prompt of alfworld asks the VLM to choose among an admissible action. See Figure 2 for an example of the visual observation of alfworld. We leave the detailed descriptions of the alfworld (state space, action space, reward functions, and the CoT prompt) to Appendix B.2.
6 Experimental Results
The first part of our experiment examines how our method improves the decision decision-making capabilities of VLMs (Section 1. The second part investigates the role of CoT reasoning in our method (Section 6.2). Details of our experimental setup are provided in Appendix C.
6.1 Improving VLM Decision-Making Capabilities
Does our method improve the decision-makinn capabilities of VLM agents across various domains? To investigate this, we assess how our method improves arithmetic tasks requiring fine-grained visual recognition in the gym_cards domain and visual semantic reasoning in the alfworld domain. The car_s experiments include deterministic tasks (NumberLine, EZPoints, and Points24, cach with increasing difficulty) and a stochastic task (Blackjact) ack). In the alfworld domain, we evaluate overall performance and detailed task task-specific performance as discussed in Section 5.2. We instantiate our method on top of the llava-vl. dova-vl..-mmstrall 16-misstrll-7b [35] model and compare against commercial models (GPT4-V V and Gemini), a supervised fine-tuned version of the llava-v1 1lava-v1.6-mistral 66-mistral-77b model (LLa VA- s and a vanilla RL implementation using a CNN-based policy network (CNN+RL). The final results and learning curves are presented in Table 2 and Figure 5. Details of the experimental setup are provided Appendix C.
Enhancing decision-making capabilities of VLM agents across various tasks. As illustrated in Table 2 and Figure 5, our method demonstrates consistent improvement across various tasks, including deterministic (NumberLine and EZPoints)' or stochastic (Blackj ack) arithmetic tasks and visual semantic reasoning task (alfworld). Specifically, our method improves the average performance from the initial VA-sft model by 27.1 % on arithmetic tasks (18.4% - 45.5%) and
"To ensure the RL training starts from a model with reasonable instruction following capabilities [45], our RL training for VLM starts from the VA-VA-sft checkpoint of cach task, we leave the detailed training pipeline of our method to Appendix C.I.
'The CNN-based method adopts the same CLIP vision encoder as LLa VA- Additionally, for tasks that require text inputs (e.g., alfworid), we adopt the RoBERTa-base [36] model to encode the text feature and concatenate the text and CLIP visual features for downstream RI. training. Details of our CNN-based model are provided Appendix C.2.
Althooghh Points24 shares similar rules with EZPoints, it requires the VLM to recognize all four cards and generate much longer equations. Most failure cases in Points24 are caused by either inaccurate visual perception flawed language reasoning. We provide some examples of these failures in Appendix C.5.
8

---

gym_cards
alfzorld
NL
EZP
P24
BJ
Avg.
Exp. Data
Piek
Look
Clean
Heat
Cool
Pick2
Avg.
BUTLER, BUTLER
17.0
70.0
76.0
12.0
22.0
V
46.0
22.0
39.0
74.0
100.0
24.0
37.0
CNN-RL
87.1
0
0
38.8
31.5
X
0
0
GPT4-V
65.5
10.5
0
25.5
25.4
K
38.2
12.1
18.8
6.7
17.8
14.6
19.4
Gemini
12.0
LLaVA sft
24.8
23.0
2.6
23.
18.4
X
39.2
0
14.4
11.1
0
28.6
17.7
18.0
Table 2: Average episode success rates (%) of different methods on gym_ cards and alfworld. For all RL-based methods (CNN+RL and our method), we present the peak numbers (first 15k environment steps for the gym_ and 5k environment steps for al alfworld) from each training curve from Figure 5. We average the performance of all 4 tasks on gym_carde with equal weight. Due to the nature of the alfworld environment, where each subtask does not appear with equal probability. the average performance on alfworld is weighted average among all types tasks. We mark the BUT,ER, and BUTLER agent [58] in gray since they require expert data, while the remaining methods do not require expert data. As discussed by Shridhar et al. [58]. the performance discrepancy between BUTLER, and BUTLER happens due to different decoding strategies in evaluation strategies: BUTLER, uses greedy decoding, which may repeat failed actions, whereas BUTLER employs beam search during evaluation.
gym Numbrrnuu
gym_cards/E2PPe EZooiis
gym Banccsaaaaaakkkkk
50
40
50
CNN-RL
Gume
301
20
40
10
Sk:
10k
16k
0
5c.
10k
15k
5k
10k
1k
24.
3k
5k
6nv stops
onv stops
6nV stops
DIV stops
Figure 5: Episode success rates (%) of different methods on gym_carda and alfworld during training. Left to right: gym_cards cards/Numbens /uunberliee,,eeeeeeeeeeeaarm gards Soardd//EPPintt... gym_ gym_cards/ /Blackjackk, and alfworld (all). The curves of Pointa24 not included because none of the tested methods achieve reasonable performance.
4.0% % on visual semantic decision-making task (17.7% + 21.7%) In addition, our method also achieves the best performance among all comparative methods, surpassing the second-best method by 14.0% (CNN+RL) on gym_cards and 2.3% (GPT4-V) on alfworld.
6.2 Understanding the Role of the CoT Reasoning
In Section 6.1, we have demonstrated that our method improves the arithmetic and visual semantic reasoning capabilities of VLM agents. Conceptually, our method can be viewed as an augmented version of the standard CNN-based RL, where the text output [o _=d (from Figure 3) serve as the text action aett augmented by CoT reasoning syll This raises an important question: How does the CoT reasoning tht influence the overall performance of our method? To assess the impact of CoT reasoning our method's performance, we conduct two sets of ablation experiments. The first set (presented in Table and Figure 6) evaluates our method without the CoT reasoning, and the second part (shown in Figure 7) examines various scaling hyperparameters À for the ligeliied of CoT tokens, as defined Equation 4.3.
gyn_carde
alfworld
CoT
NL.
P24
Avg.
Pick
Examine
Clean
Heat
Cool
Pick
Avg.
89.4
50.0
2.3
40.2
45.5
47.4
14.7
10.4
14.4
18.8
18.0
21.7
26.9
40.4
2.8
14.4
17.7
16.3
Diff. (V- X)
+62.5
+20.1
+2.3
0.2
+21.2
+6.9
+2.7
+7.6
+5.9
+4.4
+0.3
+5.4
Table 3:
Episode success rates (%) ) of our method with and without CoT reasoning. We report the best results from Figure 6 (first 15k environment steps for the gym_carde and 5k environment steps for alfworld).

---

@ym__ardmmeee Numbernne
gym cards
gym Baadkkaa
ALFWordd
50
25
40
00
W/ Gol
40
We GoT
30
15
5k.
-0k
15k
20
5k
13k
15k
5k.
10k
15k
1K
24
54.
44
5k
env steps
emv sleps
env sleps
BmW slops
Figure 6: Training curves of our method without and without the CoT reasoning. Left to right: kym_cards, Sards/Numberline gym_a Sarda//ZPPintss sym__cardsBBlacjeeee and alfworld (all). The curves of Points24 are not included because none the tested methods achieve reasonable performance.
The erucial role of CoT reasoning in performance improvement. As presented in Table 3 and Figure 6, the performance of our method significantly decreases without the CoT reasoning. Besides the improvement in the final performance, CoT reasoning appears be crucial component for deterministic arithmetic tasks (NumberLine and EZPoints). as our method fails to improve these two tasks without the CoT reasoning.
The importance of moderate scaling factors A. As dis- cussed in Section 4.3, integrating CoT reasoning into our frame- work involves tuning an additional hyperparameter, À E [0. 1] (proposed in Equation 4.3). To identify an optimal range for X, we conduct experiments assessing the impact of various X. Our results in Figure 7 indicate that a moderate À (between 0.3 and 0.5) enables effective training the NumberLine task. Conversely, our method fails when À is set too large (2 0.7) or too small (< 0.1), and we empirically find that an optimal typically falls within 0.2 to 0.5. This is because a large À results in an estimate of log Te (aglor, v²t)) being overly influ- enced by log ( (o lo,, ) while a small À value causes to be predominantly affected by log ( jo:, v/", ofth) thereby reducing the effect of the CoT reasoning in RL training.
0.1
0.3
60
0.7
40
0.9
20
2k
6k env steps
10k
Figure 7: Episode success rates ( (%) of our method under different À on NumberLine.
7 Conclusions, Limitations, and Future Directions
In this paper, we introduce an algorithmic framework that directly fine-tunes VLMs using RL, with the help of the VLM's CoT reasoning capability. Empirical results demonstrate that method can enhance the decision-making abilities of VLMs across diverse domains that require fine-grained visual recognition visual semantic understanding. In addition, we demonstrate that CoT reasoning is crucial component for enabling RL. training, allowing 7b VLMs to outperform established commercial models such GPT-4V and Gemini on most tasks. While our results suggest that CoT reasoning is crucial to the performance improvement of VLM training with RL, we have not extensively explored the effects of different prompting techniques in this work, which will be interesting future direction. The performance gain of our method is also limited by the size of the action space and the difficulties of the task. For example alfworld does not enjoy as much performance gain as gym_ cards, since al fworld is multi-task environment and it has much larger action space than gyr_s.
8 Acknowledgement
We would like to thank William Chen, Kuan Fang, Aviral Kumar, Qiyang Li, Fangchen Liu, Oier Mees, Seohong Park, Karl Pertsch, Haozhi Qi, Chun- Yeh, and Andrea Zanette for the early discussions and suggestions on the project. A.S. is partly supported by A12 Young Investigator Grant, and a Gemma Academic Program Award. S.X. is partly supported by an Amazon research award and the Google TRC program. This research was supported by NSF RI IIS- TIS-2246811 1, AFOSR PA9550-22-11 22-1-02 the joint Simons Foundation-Ne DMS grant #2031899 the ONR grant N00014- 22-1-2102, Tsinghua Berkeley Shenzhen Institute (TBSI) Research Fund, and the Hong Kong Center for Construction Robotics Limited (HKCRC) Award 052245. We would also like thank Hyperbolic Labs for the computing support.
'Eceept for the Blackjack ack task, where the peak performance without CoT is slightly better (+0.2%)
10

---

References
[1] Marwa Abdulhai, Isadora White, Charlic Snell, Charles Sun, Joey Hong, Yuexiang Zhai, Kelvin Xu, and Sergey Levine. Lmrl gym: Benchmarks for multi-turn reinforcement learning with language models. arXiv preprint arXiv:231 182332 2023.
[2] Alekh Agarwal, Nan Jiang, Sham M Kakade, and Wen Sun. Reinforcement learning: Theory and algorithms. CS Dept., UW Seattle, Seattle, WA, USA, Tech. Rep, 32, 2019.
[3] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitmick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pages 2425 2433,243,222 2015.
[4] Kate Baumli, Satinder Baveja, Feryal Behbahani, Harris Chan, Gheorghe Comanici, Sebastian Flennerhag, Maxime Gazeau, Kristian Holsheimer, Dan Horgan, Michael Laskin, al. Vision- language models as a source of rewards. arXiv preprint (2Xiv:23/2.09)
2023.
[5] M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279,
jun 2013.
[6] Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Dębiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota 2 with large scale deep reinforcement learning. arXiv preprint urXiv:1912 2.06000, 2019.
[7] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.00 072558 2021.
[8] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint 1wXiv:1606.01500 2016.
[9] Thomas Carta, Clément Romac, Thomas Wolf, Sylvain Lamprier, Olivier Sigaud, and Pierre- Yves Oudeyer. Grounding large language models in interactive environments with online reinforcement learning. In International Conference on Machine Learning, pages 3676-3713
PMLR, 2023.
[10] Louis Castricato, Alex Havrilla, Shahbuland Matiana, Duy V. Phung, Aman Tiwari, Jonathan Tow, and Maksym Zhuravinsky. trlX: A scalable framework for RLHF, June 2023. URL
https://gi thuubb.. com/ Sonn/CrrerrAA/tl..
[11] William Chen, Oier Mees, Aviral Kumar, and Sergey Levine. Vision-language models provide promptable representations for reinforcement learning. arXiv preprint arXiv:2402.02633 02651,, 2024.
[12] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Hcewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichhro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110. 14.666 2021.
[13] Marc-Alexandre Côté, Akos Kádár, Xingdi Yuan, Ben Kybartas, Tavian Barnes, Emery Fine, James Moore, Matthew Hausknecht, Layla EI Asri, Mahmoud Adada, et al. Textworld: A learning environment for text-based games. In Computer Games: 7th Workshop, CGW 2018, Held in Conjunction with the 27th International Conference on Artificial Intelligence, IJCAI 2018, Stockholm, Sweden, July 13, 2018, Revised Selected Papers 7, pages 41-75. Springer, 2019.
[14] Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. A survey for in co-tentext learning. arXiv preprint arXiv:2301 002334, 2022.
[15] Linxi Fan, Guanzhi Wang, Yunfan Jiang, Ajay Mandlekar, Yuncong Yang, Haoyi Zhu, Andrew Tang, De-An Huang, Yuke Zhu, and Anima Anandkumar. Minedojo: Building open-ended embodied agents with internet-scale knowledge. Advances in Neural Information Processing Systems, 15:11333-18862 2022.
[16] Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, et al. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306. 13394, 2023.
[17] Yao Fu, Hao Peng, Ashish Sabharwal, Peter Clark, and Tushar Khot. Complexity-based based prompting for multi-step reasoning. The Eleventh International Conference on Learning Representations, 2023. URL attpp::/oppnrre
11

---

[18] DeepMind Google. Introducing gemini: our largest and most capable ai model, 2023. URL gereee-eeiiniaa
https://blon httpg///eeeeeoooggaaaaaaa/eaaaee wtwpo/////aggggoggggogooooo
[19] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference machine learning, pages 1861-1870. PMLR, 2018.
[20] Joey Hong, Sergey Levine, and Anca Dragan. Zero-shot goal-directed dialogue via rl on imagined conversations. arXiv preprint arXiv:2311.05588 2023.
[21] Edward Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https://oppnreeiiee
net/foruun:ddnZooWeerr
[22] Jiaxing Huang. Jingyi Zhang. Kai Jiang, Han Qiu, and Shijian Lu. Visual instruction tuning towards general-purpose multimodal model: A survey. arXiv preprint arXiv:2312.1660) 2023.
[23] Shengyi Huang, Quentin Gallouédec, Florian Felten, Antonin Raffin, Rousslan Fernand Julien Dossa, Yanxiao Zhao, Ryan Sullivan, Viktor Makoviychuk, Denys Makoviichu, Mohamad H Danesh, et al. Open rl benchmark: Comprehensive tracked experiments for reinforcement learning. Xrviv preprint arXiv:2402.033065 2024.
[24] Wenlong Huang, Chen Wang. Ruohan Zhang, Yunzhu Li, Jiajun Wu, and Li Fei-Fei. Vox- poser: Composable 3d value maps for robotic manipulation with language models. In 7th Annual Conference on Robot Learning, 2023. URL wwtttt:t//pppeeeeeeeeeeeeee http://pprreveee.eetffrorraaaaa net/forum?id 9_8LF30mOC.
[25] Albert Q Jiang. Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023.
[26] Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199-2222 2211192221,,,,-111 22,13,22 2022.
[27] Ilya Kostrikov. Pytorch implementations of reinforcement learning algorithms. https:// github. com/ hwtm//kkstrikoo/pyteeooo hatpchh-acpppp-cccccppp patcc-ppppcccct--eeeee
www.aaakttgiiiii gail, 2018.
[28] Heinrich Küttler, Nantas Nardelli, Alexander Miller, Roberta Raileanu, Marco Selvatici, Edward Grefenstette, and Tim Rocktäschel. The nethack learning environment. Advances in Neural Information Processing Systems, 33:7671 1-7644, 2020.
[29] Chunyuan Li. Large multimodal models: Notes on evpr 2023 tutorial. arXiv preprint arXiv:2306. 14895, 2023.
[30] Chumyuan Li, Zhe Gan, Zhengyuan Yang. Jianwei Yang, Linjie Li, Lijuan Wang, and Jianfeng Gao. Multimodal foundation models: From specialists to general-purpose assistants. arXiv preprint arXiv:2309. /0020, 1(2):2, 2023.
[31] Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, llya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint arXiv:2305.20050, 2023.
[32] Fuxiao Liu, Tianrui Guan, Zongxia Li, Lichang Chen, Yaser Yacoob, Dinesh Manocha, and Tianyi Zhou. Hallusionbench: You see what you think? or you think what you see? an image- context reasoning benchmark challenging for gpt-4v (ision), 1lava-1.5, and other multi-modality models. arXiv preprint arXiv:2310. 1..00.14455 14566, 2023.
[33] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jac Lee. Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03744 2023.
[34] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https:
//openevieww wtwp:/neeeeeeeeeeeee hett/forum?:ddx002GGGHH
[35] Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge. 2024.
[36] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: robustly optimized bert pretraining approach. arXiv preprint 197.:11977.11922 2019.
12

---

[37] Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang. Conghui He, Ziwei Liu, al. Mmbench: Is your multi-modal model an all-around player? arXiv preprint wxXiv22377.06288 2023.
[38] Chaochao Lu, Chen Qian, Guodong Zheng, Hongxing Fan, Hongzhi Gao, Jie Zhang. Jing Shao, Jingyi Deng, Jinlan Fu, Kexin Huang, et al. From gpt-4 to gemini and beyond: Assessing the landscape of milms on gencralizability, trustworthiness and causality through four modalities. arXiv preprint arXiv:2401. 15071, 2024.
[39] Marlos C. Machado, Mare G. Bellemare, Erik Talvitie, Joel Veness, Matthew Hausknecht, and Michael Bowling. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. Journal of Artificial Intelligence Research, 61:523-562, 2018.
[40] Manolis Savva*, Abhishek Kadian Oleksandr Maksymets Yili Zhao, Brik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, Devi Parikh, and Dhruv Batra. Habitat: A Platform for Embodied Al Research. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019.
[41] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland. Georg Ostrovski, al. Human-level control through deep reinforcement learning. nature, 518(7540):552-5333 2015.
[42] Yao Mu, Qinglong Zhang. Mengkang Hu, Wenhai Wang, Mingyu Ding, Jun Jin, Bin Wang, Jifeng Dai, Yu Qiao, and Ping Luo. Embodiedgpt: Vision-language pre-training via embodied chain of thought. arXiv preprint arXiv:2305.15001, 2023.
[43] OpenAl. Gpt-4, 2023. URULtttpps///ppnaa.cmm/eeaaaaaae gpt-4.
[44] OpenAL. Gpt-4v, 2023. URL atttpp://ppenaa..cm/reeeeeee
www.vvvssseeeeeeee parstem-card...
[45] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744, 2022.
[46] Jiayi Pan, Yichi Zhang, Nicholas Tomlin, Yifei Zhou, Sergey Levine, and Alane Suhr. Au- tonomous evaluation and refinement of digital agents. arXiv preprint arXiv:2404.066 04444444 2024.
[47] Liangming Pan, Michael Saxon, Wenda Xu, Deepak Nathani, Xinyi Wang, and William Yang Wang. Automatically correcting large language models: Surveying the landscape of diverse self-correction strategies. arXiv preprint arXiv:2308.00 03 088, 2023.
[48] Joon Sung Park, Joseph O' O'Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behavior. In Proceed- ings of the 36th Annual ACM Symposium on User Interface Software and Technology, pages 1-22, 2023.
[49] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pages 8748-8763. PMLR, 2021.
[50] Rajkumar Ramamurthy, Prithviraj Ammanabrolu, Kianté Brantley, Jack Hessel, Rafet Sifa, Christian Bauckhage, Hannaneh Hajishirzi, and Yejin Choi. Is reinforcement learning (not) for natural language processing: Benchmarks, baselines, and building blocks for natural language policy optimization. In The Eleventh International Conference on Learning Representations, 2023. URL https: http////ppe//eeeeeeeeeeee/e wttp:eetteetttttttataeteaaa dd-aimiddiddaaaa2uuuuuyya
[51] Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & 2020.
Data Mining, pages
[52] Juan Rocamonde, Victoriano Montesinos, Elvis Nava, Ethan Perez, and David Lindner. Vision- language models are zero-shot reward models for reinforcement learning. In The Twelfth International Conference on Learning Representutions, 2024. URI. https: htap:::///peereviee
13

---

[53] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pages 1889-1897.
PMLR, 2015.
[54] John Schulman, Philipp Moritz, Sergey Levine. Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. In Proceedings of the International Conference Learning Representations (ICLR), 2016.
[55] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 177.006777 2017.
[56] Bokui Shen, Fei Xia, Chengshu Li, Roberto Martin, Linxi Fan, Guanzhi Wang, Claudia Pérez-D' Arpino, Shyamal Buch, Sanjana Srivastava, Lyne Tchapmi, al. igibson 1.0: A simulation environment for interactive tasks in large realistic scenes. In 2021 IEEE/RSJ International Conference on Intelligent Robols and Systems (IROS), pages 7520-7527. IEEE, 2021.
[57] Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk. Winson Han, Roozbeh Mot- taghi. Luke Zettlemoyer, and Dieter Fox. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pages 10740-10749, 2020.
[58] Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Cote, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. (ALFW) oold:: Aligning text and embodied environments for interactive learning. In International Conference on Learning Representations, 2021. URL https: / //openeeview
[59] David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre,, George Van Den Driess- che, Julian Schrittwieser, loannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mas- tering the game of go with deep neural networks and tree search. nature, 529(7587):484 489, 2016.
[60] Charlie Victor Snell, Ilya Kostrikov, Yi Su, Sherry Yang, and Sergey Levine. Offline RL for natural language generation with implicit language q learning. In The Eleventh International Conference on Learning Representations, 2023. URL
IddaBBHDDdEEve
[61] Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-3021, 2020.
[62] Zhiqing Sun, Sheng Shen, Shengeao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang- Yan Gui, Yu- Yu-Xiong Wang, Yiming Yang, et al. Aligning large multimodal models with factually augmented rlhf. arXiv preprint arXiv:2309.1452 14525, 2023.
[63] Richard Sutton and Andrew Barto. Reinforcement learning: An introduction. MIT press, 2018.
[64] Andrew Szot, Max Schwarzer, Harsh Agrawal, Bogdan Mazoure, Walter Talbott, Katherine Metcalf, Natalic Mackraz, Devon Hjelm, and Alexander Toshev. Large language models as generalizable policies for embodied tasks. arXiv preprint arXiv:2310.17722 2023.
[65] Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms. arXiv preprint arXiv:2401.06209 2024.
[66] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Ann Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 2023.
[67] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.099. 37..00988.8.... 2023.
[68] Mark Towers, Jordan K. Terry, Ariel Kwiatkowski, John U. Balis, Gianluca de Cola, Tristan Deleu, Manuel Goulão, Andreas Kallinteris, Arjun KG, Markus Krimmel, Rodrigo Perez- Vicente, Andrea Pierré, Sander Schulhoff, Jun Jet Tai, Andrew Tan Jin Shen, and Omar G. Younis. Gymnasium, March 2023. URL https: // zenodo. org www.reeooo///////
14

---

[69] Oriol Vinyals, Igor Babuschkin, Wojeiech M Czarnekki, Michaël Mathieu, Andrew Dudzik, Jun- young Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782)3350 354, 2019.
[70] Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, and Shengyi Huang. Trl: Transformer reinforcement learning. https:/ gg thub.
com/huugingface//tte
2020.
[71] Guanzhi Wang. Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv: 2022.
[72] Ruoyao Wang. Peter Alexander Jansen, Marc-Alexander Côté, and Prithviraj Ammanabrolu. Sci- enceworld: Is your agent smarter than 5th grader? In Conference on Empirical Methods in Nat- ural Language Processing, 2022. URL https: //api. semanticscholaa org/ rer/rrupssID
247451124.
[73] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistenee improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, 2023. URL
http::///ppeneviee.. wwwpnreiiew..ee/ooorrreeeeeeeeee wwt:/fouumm 1d=1PLINIMMr..
[74] Zihao Wang, Shaofei Cai, Guanzhou Chen, Anji Liu, Xiaojian Ma, and Yitao Liang. Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents. arXiv preprini arXiv:2302.01500 2023.
[75] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain- thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 0122822--24877 2022.
[76] Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, et al. The rise and potential of large language model 2023.
based agents: A survey. arXiv preprint arXiv:2309.07885
[77] Sherry Yang. Ofir Nachum, Yilun Du, Jason Wei, Pieter Abbeel, and Dale Schuurmans. Foun- dation models for decision making: Problems, methods, and opportunities. arXiv preprint arXiv:2303.00 029,0, 2023.
[78] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas I. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. arXiv preprint arXiv:2305. 100.0 2023.
[79] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, 2023. URL
[80] Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang. and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.022 02490,, 2023.
[81] Yuexiang Zhai, Shenghang Tong, Xiao Li, Mu Cai, Qing Qu, Yong Jae Lee, and Yi Ma. Investigating the catastrophic forgetting in multimodal large language model fine-tuning. In Conference on Parsimony and Learning, pages 202-227. PMLR, 2024.
[82] Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schu- urmans, Claire Cui, Olivier Bousquet, Quoc V Le, and Ed H. Chi. Least-to-most prompting enables complex reasoning in large language models. In The Eleventh International Con- ference on Learning Representations, 2023. URL https::/ http:p/:///ppeeeiee...ee net/f wwt.:/rumffffoofe--ooooo WZH7099tgfM.
[83] Yifei Zhou, Andrea Zanette, Jiayi Pan, Sergey Levine, and Aviral Kumar. Archer: Training language model agents via hierarchical multi-turn rl. arXiv preprint arXiv:2402. 19446, 2024.
[84] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing
understanding with advanced large language models. arXiv preprint ar0 /0592, 2023.
[85] Daniel M Ziegler, Nisan Stiennon, Jeffrey WI, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint 1XXiv:19090085000 2019.
15

---

A
Contributions
YXZ:
proposed, led, and managed the project; integrated all code bases; ran all ablations for method development; babysat all experiments; implemented the post-processing function f; proposed and implemented the scaling factor for action tokens; beautified the gym_cards environment; maintained all codebases; wrote the major part of the paper.
HB: set up the infrastructure and initial experiments for supervised fine-tuning before RL
training; maintained codebases; partially wrote the paper.
ZL: set up the alfworld environment; set up major infrastructures for data collection; maintained all codebases; partially wrote the paper.
JP: proposed the CoT idea for end-to-end RL training; optimized the RL training framework with quantization and enabled distributed training; implemented the initial version of the cards environment; partially wrote the paper.
ST: maintained the usage of LLaVA repo [34, 33, 35]; implemented the queries for GPT4-V and Gemini; partially wrote the paper.
YFZ: implemented the initial version RL training on LLaVA; partially wrote the paper.
AS, SX, YI., YM, SL. provided suggestions for the project. AS, SX, SL also provided feedbacks writing. YM, SL inspired YXZ to initiate the entire project.
B Additional Details of the Evaluation Tasks
B.1
Gym Cards
B.1.1
NumberLine
State and action space. In the NumberLine task, the visual observation at each state s, contains two lines of text: "Target: x² and "Current: yr", where Vt are both integers such that x, 3r E [0, Rmmax where max is an environment input variable that controls the maximum position of the numbers. The goal is to move the current number yt to the target number x, by sequentially choosing actions from the discrete action space ("+", "-"). We set nama = 5 for all experiments this work, but Mmax be set to any positive integers. Choosing "+" or " "_ will increase decrease the current number y, by respectively, and the agent will stay at the boundary it takes an action that attempts cross the boundary (e.g., taking de "4" when y/2 Tmma or a, when I; 0). See an example of the state action transition in Figure 8.
Target: 3
Target: 3
"action": "4"
Current:
Current:
Figure 8: An example of the transition in NumberLine.
Reward functionn and the CoT prompts. An episode in NumberLine ends when the current number equals the target number the maximum step T 2n is reached. The agent receives a terminal reward of r(st, 4z) = when Ut+1
The agent also receives reward penalty of upon taking an incorrect action that does result in closer position to the target (lr - yel 2 I Dt+1l), otherwise the agent receives reward r(st, a,) 0. In the example provided above (Figure 8), the agent receives a reward = 0, since it moves closer to the target, but not reaching the target yet. For the NumberLine task, we adopt the following CoT prompt in Figure 9, and for the case without CoT reasoning (discussed Section 6.2), we use the same prompt without the blue COT reasoning parts.
r(st, ae)
16

---

CoT prompt ohth for task NumberLine
You are playing game called number line. You will see target number and a current number in the image. And your goal is to the current number closer to the target by choosing either adding or subtracting one to current number. Your response should valid json file the following format:
"current number"; "x",
"target number": "x",
"thoughts": [first read out current and larget number, then think carefully about which action to choose].
"action": "-" or "+"
Figure 9: Task-specific CoT prompt input ot"" for NumberLine. The blue part represents the CoT reasoning and the red part is the text-based action.
B.1.2 EZPoints
State and action space. the EZPoints task, the agent will observe an image of cards and text version of "formula" below the cards, at each state The goal is to use the cards in the image to compute a target number 12 and view ["J", "q". "K" as "10", The action space of EZPoints is |"1", "2" "10", "+", "*", "="} and each number in the cards can only be used once. Any action attempting to either select number not shown the cards or use a card more than once are illegal. At if legal action is taken, the action will be appended to the text "formula" 8 and becomes the next state 8t+1. On the other hand, when an illegal action is taken, 8t+1 will remain the same as 8, All images generated from the EZPoints environment are guaranteed to have a viable solution for computing 12.
"action": "4"
Formula:
Formula:
Figure 10; An example of the transition in EZPoints.
Reward functions and the CoT prompts. An episode terminates when "=" is taken or the maximum step T is reached. The agent receives reward of -1 upon taking an illegal action, and r = O while taking legal action. When "=" is taken, the agent will receive positive if formula equals 12, and
otherwise. For the EZPoints task, we adopt the following CoT prompt in Figure 11, and for the case without CoT reasoning (discussed in Section 6.2), we use the same prompt but without the blue CoT reasoning parts and the brown part in Figure 11 the text version of the current formula directly extracted from the current state St-
reward
17

---

CoT prompt o for EZPoints
You are expert card game player. You are observing two cards in the image. You are observing the
current formula: '5', You choose between [' '2', '3', '4', '5', '6', '7', '8', '9', '10', '+', rea, '.'J. J. number or operator you choose will be appended to the current formula. Note that 'J', 'Q', and 'K' count as '10', Your goal to output formula that evaluates to 12, and each number can only be used once. Your response should be valid json file in the following format:
"cards": [x.y]. yl.
"current formula": '5',
"thoughts": [First check whether the current formula 'z' is complete. If the current formula 'z' is complete, output '=', Otherwise consider which number or operator should be appended to the current formula make equal 12.) "action": "I "(number]" or "[operator]"
Figure 11: Task-specific CoT prompt input o for EZPoints given the observation in Figure 10. The blue part represents CoT reasoning, the red part is text- action, the brown part the dete-dependent text from formula in the image.
B.1.3 Points24
State and action space. Similar to EZPoints, the goal of Points24 is also to generate a formula compute the target number of 24, using all four cards. Points24 has slightly larger action space: ("1", "2" "2"..... "10", "4", "-", "*", "/", "(", ")", "="] and two more cards. Each number in the cards can only be used once. Similar to EZPoints, any action attempting either select a number not shown in the cards or use card more than once are illegal. At st, if legal action is taken, the action will be appended to the text "formula" in 8 and becomes the next state 81+1- When an Different from EZPoints where all images
illegal action is taken, $1+1 will remain the same are guaranteed to have viable solution for computing 12, the images generated by Points24 do not always have viable solution to 24.
"action": "4"
Formula:
Formala:
2
Figure 12: An example of the transition in Points24.
Reward functions and the CoT prompts. The reward functions and termination conditions of Points24 are the same as those in EZPoints. An episode terminates when "=" is taken or the maximum step T 20 is reached. The agent receives reward of -1 upon taking an illegal action, and
while taking legal actions. When "=" is taken, the agent will receive positive reward 10 when the formula equals 24, and -1 otherwise. For the Points24 task, we adopt the following CoT prompt in Figure 13, and for the case without CoT reasoning (discussed without
blue CoT reasoning parts and the brown part in Figure 13 is the text version of the current formula directly extracted from the current state 8t- We also provide an additional feature that allows us to view ["J", "Q", "K") as ( (111", "12", "13") instead of [.
Section 6.2), we use the same prompt
18

---

CoT prompt uhth for Points24
You are expert 24 points card game player. You observing these four cards in the image. You observing the current formula: '(2.. You choose between ['l', '2', '3', '4', '5', '6', '7', '8', '9', '10', '+', ',', 1 1/ '('. ')', '='J. The number operator you choose will appended to the current formula. Note that 'J', 'Q', and 'K' count as '1',, Your goal is to output formula that evaluates to 24, and cach number can only be used once. Your response should be a valid json file in the following format:
"cards": [x. y, 2, w].
"current formula": '(2'
"thoughts": (First check whether the current formula equals 24, If the current formula equals 24, output '=', Otherwise consider which number operator should be appended to the current formula to make equal 24.}
"action": "[number]" or "[operator""
Figure 13: Task-specific CoT prompt input dh for Points24 given observation in Figure The blue part represents CoT reasoning and the red part is text-based action, brown part is the state-dependent text. that directly obtained from the formula in the image.
B.1.4 Blackjack
State and action space. For the Blackjack task, the visual observation at state s; consists two cards (one face-down) from the dealer and all cards from the player. The agent's goal in this task is to win the current game, by choosing actions ("stand", "hit"). The agent will receive a new card upon choosing "hit". See Figure 14 for an example transition.
Dealer
Dealer
Player
"action": "hit"
Player
Figure 14: An example of the transition in Blackjack.
Reward functions and the CoT prompts. The game terminates when the player chooses "stand" or busts (total points exceed 21). We adopt the same reward function as the Blackjack-vi task in Gymnasiym [68]. where r(s,, a.) = 1, 0, 1 upon win, draw, and loss, respectively. We also provide a similar feature as Gymnasium [68], where the "blackjack" winning (the agent win with an "A" and "10", "J", "Q" or "K") reward of the player will become 1.5. In the example provided in Figure 14, the game not terminated after taking the action "hit", ", hence the agent will not receive any rewards, CoT prompt in Figure 15, and for the case without CoT reasoning (discussed Section 6.2), we use the same prompt but without the blue CoT reasoning parts.
though it has total points of 21. For the Blackj ack task, adopt the following
CoT prompt w for Bl Blackjack
You are blackjack player. You are observing the current game state, you can choose between ('stand', 1 "hit']. Your response should he a valid json file in the following format:
"thoughts": "[first describe your total points and the dealer's total points then think about which action choose]" "action": "stand" or "hit"
Figure 15: Task-specific CoT prompt input of for Blackjack. The blue part represents the CoT reasoning and the red part is the text- based action.
19

---

B.2 ALFWorld
State and action space. Inherited from Text World [13], at each state 8, of alfworld, the agent will observe an RGB image and text-based description. The action space of alfworld can be summarized these following format [58]: (1) goto {recep); (2) take {obj} from {recep): (3) put {obj} in/on {recep): (4) open {recep); (5) close {recep): (6) toggle {obj] Krecept; (7) clean {obj} with (recep); (8) heat {obj} with {recep); (9) cool {obj} with {recep), where {obj} and {recep} stands for objects and receptacles, See Figure 16
for an example of the state action transition in the alfworld environment.
"action": "go to cabinet 2"
You arrive at loc 0. The cabinet is open.
You arrive at loe 2. The cabinet 2 open. the cabinet you see houseplant pot I, a bread I, kettle 1, bowl soapbottle 1, and knile 2.
you see pan I, kettle I, a winebottle I, apple 1. stoveknob I, stoveknob stoveknob 3, staveknob 4, knile a saltshaker and a bread
Figure 16: An example of the transition alfworld.
Reward functions and the CoT prompts. Each state s E S of alfworld has a set of admissible actions (sm a final goal Stask and subgoals Joub Since the goal of alfworld is to complete the sanguage-based based goal-conding cond-iiood tasks, we reward the agent upon reaching subgoals and completing the task, while penalizing the agent upon taking inadmissible actions. To summarize, we define the reward function of alfworld as r(st, dz. S141 (++1l)raek) = 50 11 { Itaek + { goub 1 (a, & Aada For the alfworld task, we adopt the following CoT prompt Figure 17, and for the case without CoT reasoning (discussed in Section 6.2), we use the same prompt but without the blue CoT reasoning parts and the brown part Figure is the text description of the task directly extracted from the current state st.
20

---

CoT prompt o for alfworld
Your are an expert in the ALFRED Embodied Environment. You are also given the following text description of the current scene: [* You arrive loc The cabinet is open. On the cabinet 1. you see a pan 1. kettle 1. winebottle a apple 1, stoveknob a stoveknob a stoveknob stoveknob 4, a knife 1, saltshaker 1, and a bread I. Your task to put a cool mug in cabinet. Your admissible actions of the current situation are: go to countertop 1', "go to cabinet 'go to countertop 2', go stoveburner 1', go to drawer 1', go drawer 2', "go to drawer 31, 'go to stoveburner 2', go to stoveburner 3 "go to stoveburner 4', 'go to drawer 4', 'go cabinet go to cabinet 4', go to microwave 1', go to cabinet 5', "o to cabinet 6', 'go cabinet 7', 'go to sink 1', go to sinkbasin 1', 'go to fridge I', "go to toaster 1', "go to coffeemachine 1', 'go to cabinet 8',
to drawer 5', 'go to drawer 6', 'go to drawer 7', 'go to drawer 8', "o to shelf 1', "go shelf 2', "go countertop 3', go shelf 3 go drawer 9', "o garbagecan 1', 'open cabinet 1', 'close cabinet 1', 'take pan from cabinet
'take kettle from cabinet 1', 'take winebottle from cabinet 1', 'take apple from cabinet 1', 'take stoveknob from cabinet 1', "take stoveknob from cabinet 1', "take stoveknob 3 from cabinet 1,, "take stoveknob 4 from cabinet 1', take knife from cabinet 1', "take saltshaker from cabinet 1', 'take bread from cabinet 1', 'inventooy'.. 'look', examine cabinet I'J. Your response should be valid json file in following format:
"thoughts": "first describe what do you see in the image using the text description, then carefully think about which action complete the task.
"action": "an admissible action"
Figure 17: Task-specific CoT prompt input o for alfworld given the observation in Figure 16. The blue part represents the CoT reasoning and the part is text-based action, brown part the state-dependent text that directly obtained from the text description and the admissible actions of the current state.
C Additional Details on the Experiments
We provide additional detailed of the experimental results in Section 6 here. Details of our experimen- tal pipeline is provided in Section C.I. including preparing the initial SFT checkpoints and the RL training. Section C.2 contains details setup of all comparative methods. We list task-specific training details in Section C.3. We provide additional experimental results in Section C.4. Section C.5 lists several failure examples of the Points24 tasks.
C.1 Experimental Pipeline
Our experiments adopt similar pipeline as RLHF [45], where we first apply supervised fine-tuning (SFT) to the backbone lava-vl..-mistral-77 model, before RL training. As outlined by Ouyang et al. [45], the RLHF training procedure consists of three distinct stages: SFT, learning reward models from human preference data, and applying RL with the learned reward models. Our pipeline analogous to RL.F but without requiring the collection of human preference data for learning reward models, as we can directly collect rewards from the environment." Consequently, our experimental pipeline only contains two stages: SFT and RL, which we will explain below.
Supervised fine-tuning. For the original gym_ cam_sards environment, we manually construct instruction-followinng data for all tasks following the format specified in Figure of Section 4.1. As for al alfworld, we use GPT4-V [44] to collect instruction following data for SFT. For all tasks, prepare two versions of the instruction--ollowe leave the details of the CoT prompts for each task, and the details of each fine-tuning dataset in Appendix D. After constructing the instruction-folloiin data (with and without CoT), we fine- tune 1lava-vl. 1--.s.tt..mmit-- for epoch on the collected data for each task and report the results for VA-sft.
data, one with CoT and one without. We
RL training. For each task, we start our RI. training from the VLa-AAsft checkpoint. The LLaVA model [34] consists of three jointly trainable components, CLIP vision encoder [49], an LLM
We adopt the same pipeline for the evaluation without CoT reasoning (discussed in Section 6.2) while changing the data for SFT well n (see more details SET data and o in Appendix D)
21

---

backbone [66, 67, 251, and an MLP projector that connects visual features and the word embeddings, and we directly apply PPO [55] train all three components. Due to computation resource limitations. we instantiate our experiments via LoRA [21], with the LoRA configuration of = 128, or = 256, dropout = 0.05, for all trainable components. For the CoT coefficient X, we set À = 0.5 in the gym__crrd domain and À 0.2 in alfworld.
C.2 Experimental Setup for Comparative Methods
GPT4-V and Gemini. All of our experimental results on GPT4-V [44] and Gemini [18] are tested on March 15, 2024, using the same prompt for our RI. training (see detailed prompts in Appendix D). For gym_cards, the numbers from both GPT4-V and Gemini are averaged among the same number of episodes: 200 episodes for deterministic tasks (NumberLine, EZPoints and Points24); 1000 episodes for stochastic task (Blackjack). As for alfworld, we report the performance of GPT4-V on all 1000 episodes we collected, see Appendix D.5 for our data collection on alfworld using G.T4-V. Due to the financial budget, we report the results Gemini using 100 episodes.
LLaVA-sft. For each number of VLaVA-sft, first collect the instruction-follont following dataset for each task and then fine-tune LLaVA-I.6-7B for 1 epoch on the collected data using the official LLaVA VA fine-tuning script. 10 Details of our data collection process is provided in Appendix D. We also use the same LLaVA-sft checkpoint as initializations for the downstream RL training.
CNN-based RL. Since the LLaVA-7b model adopts a CLIP ViT-L/14 vision encoder which is more powerful than vanilla CNN embeddings, we instantiate our CNN-based method using the feature from the same CLII ViT-L/144 for fair comparison. For tasks (EZPoints, Points24, and alfworld, see our detailed prompt in Appendix D) that require text inputs, we adopt the RoBERTa-base [36] model encode the text feature and concatenate the text and CLIP visual features for downstream RL training. After obtaining the CLIP (potentially concatenated with text) features, we adopt 2 MLP layers followed by a fully connected layer map the clip features into the action space. We adopt the PPO [55] implementation from Kostrikov [27] as the backbone RL algorithm. In addition, we adopt DosineennealinglA learning rate scheduler, with the initial learning rate of 3e 4, the final learning rate of le 8, and the maximum learning rate step of 25. The remaining task specific hyperparameters are the same as the VLM case in Section C.3.
C.3 General Setup for End-to-End RL Training
All experiments are conducted on an 8 A A100s DGX machine (80G), while the maximum VRAM requirement is 40G. Each curve from Figure 5 and takes at most 36 hours to finish. We adopt DeepSpeed zero2 [51] for multi-gPi gpu training. During our training for the VLM, we directly train all trainable components (vision encoder, LLM, and the MLP projector). We adopt an open-source implementation [27] for the PPO. Inspired by von Werra et al. [70]. Castricato et al. [10], we apply a 3-layer MIP as the value head, on top of the output hidden states layer before the output tokens, to estimate the value function V After obtaining the value estimate Vg. we adopt the generalized
advantage estimator (GAB) [54] to estimate the return function f((s) and the advantage function AT of my. In addition, we adopt rate of 1e- 5, the final learning rate of 1e 9, and the maximum learning rate step of 25. For all experiments in the cards and alfworld environment, we set the scaling hyperparameter 0.5, 0, 2, respectively. The learning rate decay happens after every PPO update, which consists of 4 epochs of gradient updates with PPO. The number of data for on-policy training and batch size deskddepn,,nt, we list them below.
Anneaeenn learning rate scheduler, with the initial learning
Numberline and Blackjack. For NumberLine and Blackjack, our VL.M training curves in Figure 5 use 4 GPUs. Our implementation naturally enables different random seeds on different GPUs, hence our VLM curves are averaged among 4 seeds. For one PPO update on each GPU, we collect 512 transitions, with a batch size of 128 per GPU (batch size 512 in total). The episode return and success rate are averaged with NunberLine, Blackjack are averaged among 200 and 1000 episodes, respectively. We averaged the return of Blackjack on more episodes because
10 "https://gitte wtt.//oooiicaeeeeeee from the Ilava-vI w.ra-vl..-misralll 6-mistrrll- instead of the v1.5 checkpoint in the script.
wttpan/sscrpps////eeeeeee waapstss/vlv///eeee wtt.n//ninn.....eemeemeeeeee.e.e sh. We start
22

---

B1 ackkackk contains stochastic while NumberLine is deterministic task. We adopt the same number of transitions and batch size for the on-policy training the CNN-based method on both tasks. The CNN-based methods are averaged among 4 random seeds as well.
EZPoints and Points24. For EZPoints and Points24, our VLM training curves in Figure use 4 GPUs. Our implementation naturally enables different random seeds different GPUs, hence our VLM curves are averaged among 4 seeds. For one PPO update on each GPU, we collect 1024 transitions, with batch size of 128 per GPU (batch size 512 in total). We use 1024 transitions because the episodes of EZPoints and Points24 usually have longer horizons than NumberLine and Blackjack. The episode return and success rate are averaged with EZPoints and Points22 are averaged among 200. We adopt the same number of transitions and batch size for the on policy training in the CNN-based method on both tasks. The CNN-base methods averaged among random seeds as well.
ALFWorld. For the alfworld environment, each of VLM training curves in Figure and Figure 19 are conducted on one GPU, and each curve is averaged among 4 seeds. We do not conduct multi-GPU training for alfworld because the on-policy sampling time has huge variance on different GPUs, which will largely increase the synchronization time across different GPUs. For each PPO update, collect 1024 transitions, and with batch size 256. The episode success rates are averaged among 200 episodes. We adopt the same number of transitions and batch size for the on-policy training the CNN- based method on both tasks. The CNN-based methods are averaged among 4 random seeds as well.
C.4 Additional Experimental Results
We provide some additional experimental results on the episode returns on the gym_cards and the task-specific training curves for alfworld here.
Episode Success Rate (%)
Episode Return
NL
EZP
BJ
EZP
P24
BJ
CNN+RL
87.1
0
38.8
0.79
-1.02
-1.12
-0.17
GPT4-V
65.5
10.5
25.5
-0.59
-1.30
-4.39
-0.44
Gemini
82.5
2.0
30.0
0.74
-2.57
-2.68
-0.35
LLaVA-sft
24.8
23.0
2.6
23.1
-2.30
-0.50
-13.52
-0.50
Ours
89.4
50.0
2.3
40.2
0.87
4.46
-11.84
-0.13
Table Average episode success rates and returns different methods on gym_cards For all RL-based methods (CNN and our method), we report the best results in each training curve from Figure 18.
gym cardsi/umberrine
gym__crdmsseee cards/EZPoints
gym_ Baadds/Blackjakk
1.0
4.0
-0.1
0.0
-0.2
Ours
-10
0.0
-0.3
GPTG-V
-2.0
Genini
-0.4
LLAVA
4.0
-0.5
5k
10k
15k
5k
10k
15k
0
5k
10k
15k
steps
steps
steps
Figure 18: Episode returns of different methods on gym_cards. An extended version of Figure S containing episode success rates and returns.
23

---

Pick
Look
Clean
15
20
20
1k
2k
3k
eTV steps
env steps
emv steps
Cool
Heat
Pick2
100
50
40
80
CN
40
30
80
GPT4-V
30
Gemini
40
20
10
1k
2k
3k
4K
5k
1k
2k
3k
4k
5k
1k
2k
3k
4k
5k
Cnv steps
steps
steps
Figure 19: Episode success rates our method on different types of tasks from alfworld. The curves come from the same experiments from the alfworld figure of Figure 5.
C.5 Failure Examples of Points24
Ground Truth Cards: [2, 8, 5, J]
Error Type: Fail to recognize all numbers
"cards": [10, 1.,
"formula": "10+".
"thoughts": "'10+" is an incomplete formula, since 110+10-5+9=4', shouul append '10' to the current formula", "action": "10"
Figure 20: Failure example of Points24 due to incorrect perception. In this example, the VLM fails to recognize the four numbers from the cards correctly it only recognizes numbers correctly. The reasoning part correct, as indeed generates a formula that equals 24.
Ground Truth Cards: [J. K, 5, 6]
Error Type: Fail to recognize all numbers
"cards": [10, 5,, 7).
"formula": "10+10+5+".
"thoughts": "=10+1++5' is an incomplete formula, since 110+10+5+=-24', I should append '7' to the current formula",
"action": "7"
Figure 21: Failure example of Points24 due to incorrect perception and reasoning, In this example, the VI.M fails to recognize the four numbers from the cards correctly only recognizes numbers correctly. The reasoning part also incorrect, 10+10+5+7 does not equal 24.
24

---

D Details on Supervised Fine-tuning Data
In this section, we provide detailed descriptions of the supervised fine-tuning data for all tasks.
D.1 Numberl.ine
In the NumberLine task, we collect 20k instruction-following expert data samples using a task solver. See examples with and without CoT in Figure 22 and 23, respectively.
Inputs:
You are playing a game called number line. You will see a target number and current number in the image. And your goal is move the current number closer to the target by choosing either adding or subtracting one the current number. Your response should be a valid json file in the following format:
Target: 3
Current:
"current number": "x".
"target number": "x"
"thoughts": (first read out the current and target number, then think carefully about which action to choose), "action": "_" or "+"
Responses:
"current number": "0",
"target number": "3".
"thoughts": "The current number is O, which is smaller than the target number is 3. To move the current number closer to the target, should increase the current number by choosing +.",
"action": "+"
Figure 22: An example of the supervised fine-tuning data for NumberLine with CoT.
Inputs:
You are playing a game called number line. You will see a target number and current number in the image. And your goal is to move the current number closer to the target by choosing either adding or subtracting one to the current number. Your response should be a valid json file in the following format:
Target: 3
Current: 0
"action": "_" or "+"
Responses:
"action": "4"
Figure 23: An example of the supervised fine tuning data for NumberLine without CoT.
25

---

D.2 Blackjack
In the plackjack task, we collect 50k instruction instruction-folloiing expert data samples following an open-soureed solver." !! See examples with and without CoT in Figure 24 and 25, respectively.
Inputs:
You are blackjack player. You are observing the current game state, you can choose between Ittaand', 'hit'] Your response should be a valid json file the following format:
Dealer
"thoughts": "(first describe your total points and the dealer's total points then think about which action to choose)", "action": "stand" or "hit"
Responses:
"thoughts": "I have points and the dealer has points. think should hit.", "action": "hit"
Figure 24: An example of the supervised fine-tuning data for Blackjack with CoT.
Inputs:
You are blackjack player. You are observing the current game state, you can choose between I'stand', 'hit') Your response should be a valid json file the following format:
Denler
"action": "stand" or "hit"
Responses:
"action":
"hit"
Figure 25: An example of the supervised fine-tuning data for Blackjack without CoT.
"httpss///ww.co cmu hauu.edu wwtp://hhhgweeebblleeeeee. http://laaaaaaakaaaaaaaaaaaa html
26

---

D.3 EZPoints
In the EZPoints task, we directly collect 50k instruction-following expert data samples using task solver. See examples with and without CoT in Figure 26 and 27, respectively.
Inputs:
You are an expert card game player. You are observing two cards in the image. You are observing the current formula: '5'. You can choose between [' '2', '3', '4', '5', '6', '7', '8', '91, '10', '+', ** '=']. The number operator you choose will be appended to the current formula. Note that 'J', 'Q', and K count as '10'. Your goal to output a formula that evaluates to 12, and each number can only be used once. Your response should be valid json file the following format:
Tormula: 5
"cards": [x, yl.
"current formula": '5'
"thoughts": (First check whether the current formula 'z' is complete. If the current formula 'z' is complete, output '='. Otherwise consider which number operator should be appended to the current formula to make it equal 12.) "action": "(number)" "(operator)"
Responses:
"cards": [7, 5],
"formula": "5".
"thoughts": "'5' 1. is incomplete formula, since '5+7=12', I should append '+' to the current formula", "action": "4"
Figure 26: An example of the supervised fine-tuning data for EZPoints with CoT.
Inputs:
You are expert card game player. You are observing two cards in the image. You are observing the current formula: '5'. You can choose between ['1', '2', '3', '4', '5', '6', '7', '8', 19', '10', '+', '=']. The number or operator you choose will be appended to the current formula. Note that 'J', 'Q', and K count output a formula that evaluates to 12, and each number can only be used once. Your response should be valid json file in the following format:
'10'. Your goal is to
Tormule:
"action": "(number)" "(operator)"
Responses:
"action": "4"
Figure 27: An example of the supervised fine-tuning data for Blackjack without CoT.
27

---

D.4 Points24
In the Points24 task, directly collect 50k instruction-following expert data samples using task solver. See examples with and without CoT in Figure 28 and 29, respectively. Note that the supervised fine-tuning data has slightly different distribution the states in the Points24 environment, since the expert data always have a viable solution to 24, while not all states from Points24 has a viable solution to 24.
Inputs:
You are expert card game player. You observing two cards in the image. You are observing the current formula: '5'. You can choose between ['1', '2', '3', '4', '5', '',, '7', '8', '9', '10', '+', ** '=']. The number or operator you choose will be appended to the current formula. Note that 'J', 'Q', and K count output a formula that evaluates to 12, and each number can only be used once. Your response should be valid json file in the following format:
'10'. Your goal is to
Form
"cards": [x, yl.
"current formula": '(2'.
"thoughts": (First check whether the current formula 'z' is complete. If the current formula 'z' is complete, output '='. Otherwise consider which number or operator should be appended to the current formula make equal 12.) "action": "(number)" or "(operator)"
Responses:
"cards": [1, 2, 10, 1].
"formula": "(2".
"thoughts": "'(2' is an incomplete formula, since (1+)0)) should append '+' to the current formula", "action": "+"
Figure 28: An example of the supervised fine-tuning data for Points24 with CoT.
Inputs:
You are expert card game player. You observing two cards in the image. You are observing the current formula: '5'. You can choose between [' '2', '3', '4', '5', '6', '7', '8', '9', '10', '+', **1 '=']. The number or operator you choose will be appended to the current formula. Note that 'J', 'Q', and K count as '10'. Your goal output a formula that evaluates to 12, and each number can only be used once. Your response should be valid json file in the following format:
Formulo:
(2
"action": "(number)" "(operator)"
Responses:
"action": "4"
Figure 29: An example of the supervised fine-tuning data for Points24 without CoT.
28

---

D.5 ALFWorld
The data collection procedure of the alfworld embodied environment is slightly different than the hym_cards, we do not have solvers to generate the instruction- instruction-following data, even with the expert text action. Therefore, we take a step back and directly use our prompt as presented Figure (or Figure 30) to collect Ik episodes (around 45k) instruction-following data from GPT4-V [44] with the CoT reasoning for the Instruction-follovinn fine-tuning with CoT. for the case without CoT, we directly clean the collected CoT by removing the CoT reasonings. See examples with and without CoT in Figure 30 and 31, respectively.
Inputs:
Your are an expert in the ALFRED Embodied Environment. You are also given the following text description of the current scene: [' You arrive at loc 0. The cabinet 1 is open. On the cabinet I, you see pan stoveknob
stoveknob stoveknob 3, a stoveknob 4, a knife 1, saltshaker 1, and bread 1. 'J. Your task is put a cool mug in cabinet. Your admissible actions of the current situation are: [' go to countertop 1', 'go to cabinet 2', 'go to countertop 2', go stoveburner 1', 'go to drawer 1', 'o to drawer 2', 'go to drawer 3', go to stoveburner 2', 'go stoveburner 3', 'go stoveburner 4', 'go to drawer 4', 'go to cabinet 3', 'go to cabinet 4', 'go to microwave 1', 'go to cabinet 5' 'go to cabinet 6', 'go to cabinet 7', "oo sink 1', "go to sinkbasin 1', 'go to fridge I', go to toaster 1', go to coffcemachine 1', 'go to cabinet 8 'go to drawer 5', 'go to drawer 6', 'go to drawer 7', go to drawer 8', go to shelf I', go to shelf 2', 'go to countertop 3', 'go to shelf 3', "o to drawer 9', "go to garbagecan
kettle 1, a winebottle I, apple
'open cabinet 1', "close cabinet 1', "take pan 1 from cabinet 1', 'take kettle 1 from cabinet 1 'take winebottle from cabinet 1', 'take apple I from cabinet 1', take stoveknob from cabinet 1', 'take stoveknob from cabinet 1', 'take stoveknob from cabinet 1', 'take stoveknob from cabinet 1 I', 'take knife I from cabinet 1', 'take saltshaker from cabinet 1 ', take bread from cabinet 1', 'inventory', 'look', 'examine cabinet 'J. Your response should be valid json file in the following format:
"thoughts": "first describe what do you see in the image using the text description, then carefully think about which action to complete the task. ", "action": "an admissible action"
Responses:
"thoughts": "Based the text description, I am at loc 0 with open cabinet 1 in front of me. Items such as a pan, kettle, wine bottle, apple, several stoveknobs, a knife, saltshaker, and bread are visible on the cabinet 1. To find cool mug to put in the cabinet, may need first locate mug, which might be in one of the cabinets or drawers. Since the task does not involve any items currently in cabinet and the mug is not mentioned in the description, my best course of action would be to start by searching the other cabinets or drawers for the mug.". "action": "go cabinet 2"
Figure 30: An example of the supervised fine-tuning data for alfworld with CoT.
29

---

Inputs:
Your are an expert in the ALFRED Embodied Environment. You are also given the following text description of the current scene: [' You arrive at loc O. The cabinet 1 is open. On the cabinet I, you see pan 1, kettle 1, a winebottle 1, apple 1, stoveknob 1, stoveknob 2, saltshaker and bread 1. I. Your task is to put a cool mug in cabinet. Your admissible actions of the current situation are: I'go to countertop I', 'go to cabinet 2', 'go to countertop 2', "o to stoveburner 1', go to drawer 1', go to drawer 2', 'go to drawer 3', go to stoveburner 2 go to stoveburner 3', 'go stoveburner 4', go to drawer 4', "go cabinet 3', 'go to cabinet 4', 'go to microwave 1', 'go to cabinet 5', 'go to cabinet 6', 'go to cabinet 7' 'go to sink I', go to sinkbasin 1', 'go to fridge 1', 'go to toaster 1', "go coffeemachine I', 'go cabinet 8 'go drawer 5', 'go drawer 6', 'go to drawer 7', go to drawer 8', go to shelf 1', 'oo to shelf 2', 'go to countertop 3'. 'go to shelf 3', 3', go drawer 9', go to garbagecan 1', 'open cabinet 1 'close cabinet 1', take pan 1 from cabinet 1', 'take kettle from cabinet 1', take winebottle from cabinet 1', 'take apple I from cabinet 1', 'take stoveknob from cabinet 1', 'take stoveknob from cabinet 1 'take stoveknob from cabinet 1', 'take stoveknob from cabinet 1', 'take knife from cabinet 1', take saltshaker 1 from cabinet 1 'take bread from cabinet 1 'inventory'! 'look', 'examine cabinet J.. Your response should be valid json file in the following format:
stoveknob 3, stoveknob 4, a knife
"action": "an admissible action"
Responses:
"action":
"go to cabinet 2"
Figure 31: An example the supervised fine-tuning data alfuorld without CoT.
30

---

E
NeurIPS Checklist
NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: In the paper, we propose an end-to-end RL training framework for improving a VLM agent's decision-making capabilities. We claim that our method can improve VLM's decision-making capabilities and we show the CoT reasoning appears as a crucial component for performance improvement. All claims are justified in the experiment section (Section 6).
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: We have provided a paragraph to discuss the conclusions, limitations, and future directions of our paper (Section 7). particular, we have pointed out that we have not extensively tried different prompts for the CoT reasoning, and our method currently can only train the VLM on one task a time. And would like to leave them for future studies.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they seale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
31

---

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: This is an empirical paper, we not have any theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We have listed all details for reproducing our experiments in Appendix C. We have also provided detailed readme md file in the supplementary for reproducing our results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
32

---

(c) If the contribution is a new model (e.g.. a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Our supplementary materials contain all our codes, and we have provided a detailed readme.. md file in the supplementary for reproducing our experiments.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co
Begili//guides//oeeSubbbbee ott.eeeuuummsssoooooooooooooeoeo sissnonicyy)) for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ /nips. cc/public/guieee gww.d/ge/////eeeec/ooocco /tcegguiidd//odeeubmiieeeee /CodeSSumissionnii wtibbbiisiinoooiiiy....... for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
We have provided general descriptions of the tested tasks in Section 5. We also have provided the general discussion of our experiments in Section 6. The details are provided in Appendix C and D.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
33

---

Answer: [Yes]
All of our experiments contain standard deviation the training curves, all standard devi- ations calculated among 4 different random seeds, as specified in Appendix C.3. See Figure 5, 6, 7, 18, and 19.
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We have reported such results for computation costs in Appendix C.3,
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't itdn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: :ctps://neurips. httpppice/tticccciiddep.p..pppea
Answer: [Yes]
Justification: We have read through the guidelines and we believe they are all satisfied to the best of our understanding.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consid- eration due to laws or regulations in their jurisdiction).
34

---

10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: We proposed a method for end-to-end RL training on VLMs on vision based decision- making tasks. To the best of our knowledge, our method does not lead to potential negative societal impacts.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We majorly proposed a method for applying end-to-end RL training on VLMs, which does not release any large models.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
35

---

Justification: We adopt the LLaVA-1.6-77 [35] as our initial model for RL training, we have cited LLaVA properly throughout the paper and our code (in the supplementary).
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We have prepared our own data for the supervised fine-tuning phase. And we have anonymized the dataset for reproduction in the supplementary well.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve any crowdsourcings and human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
36

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve any crowdsourcings and human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
37


Reviews:
[{"review_id": "RClkykAVbA", "reviewer": "Reviewer_MKrq", "summary": "The paper proposes an algorithmic framework for fine-tuning large vision-language models (VLMs) using reinforcement learning (RL) for multi-step decision-making tasks. The framework enhances VLMs' reasoning and decision-making capabilities by incorporating chain-of-thought (CoT) reasoning. The empirical results demonstrate that the proposed method outperforms existing models like GPT-4V and Gemini in various decision-making tasks, highlighting the significance of CoT reasoning.", "questions": "1. In the appendix, it is observed that CoT not only provides thought processes but also manually encoded key states, such as the “current number” and “target number” in the NumberLine task (Figure 9). This crucial information is hidden in the main text (See Figure 3). The authors need to clearly differentiate the source of CoT's effectiveness in improving reasoning—whether it stems from manually labeled key states or the textual thought process itself. This distinction should be experimentally validated and presented in the main text rather than being relegated to the appendix.\n\n2. Moreover, regarding the CoT issue, Figures 11 and 13 show that CoT encodes not only the key states used for decision-making but also information \"directly obtained from the formula in the image.\" There seems to be no reasonable justification for this, as theoretically, this information can be derivable from the images. An effective RL method should be capable of optimizing to identify this critical information through rewards. The inclusion of this manually encoded information makes the experiment tricky because it allows any difficult image understanding problem to be reduced to a manual extraction process. Authors should not manually encode such information to boost the final performance of their method.\n\n3. Did your baselines (GPT4-V and Gemini) include CoT information?\n\n4. To validate the effectiveness of the proposed RL algorithm, it would be beneficial to compare it with other RL methods, such as \"OFFLINE RL FOR NATURAL LANGUAGE GENERATION WITH IMPLICIT LANGUAGE Q LEARNING\" and \"ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RL.\" If using the authors' SFT model with your CoT, can we replace the proposed RL algorithm with these RL methods to achieve optimization?", "limitations": "NA", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The topic is interesting and valuable to the community.\n2. The presentation is clear, and the paper is easy to follow.\n3. The methodology is general sound.", "weaknesses": "My primary concerns lie with the experimental results and methodology. Please refer to the questions below for detailed points.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer MKrq,\n\nThank you very much for your valuable review and your questions! We will definitely integrate them into the updated paper to address your concerns. \n\n---\n### General response\n\nWe sincerely thank you for your appreciation of our work and your insightful suggestions to improve our paper. The primary contribution of our work is to **provide the first integrated system that enables end-to-end RL fine-tuning on VLMs, incorporating customized prompts as inputs**. Hence, we directly utilized PPO, which has been well studied in RLHF, rather than comparing the effects of different prompts or RL algorithms within our framework, as these topics are not the primary focus of our paper. However, we highly value your suggestions regarding the effects of different prompts and comparisons of various RL algorithms. These are indeed important topics for future studies within our framework, and we will incorporate more discussions on these subjects as future work in the updated paper draft. Once again, we greatly appreciate your attention to detail and your thoughtful feedback.\n\n---\nTo answer your questions:\n\n\n## Q.1:\n\n### Regarding the presentation in Figure 3.\n> In the appendix, it is observed that CoT not only provides thought processes but also manually encodes key states, such as the “current number” and “target number” in the NumberLine task (Figure 9). This crucial information is hidden in the main text (See Figure 3).\n\nA: Thank you for bringing to our attention the presentation issue in Figure 3. We appreciate your careful review. The main purpose of Figure 3 is to provide a template for our expected input and output, for facilitating the post-processing described in Section 4.2, rather than detailing the design of each task-specific prompt. We apologize for any confusion this may have caused. To improve clarity, we will update the figure to include an additional key, \"{optional task-specific prompts},\" before \"thoughts\" to elaborate on the difference in our task-specific prompts carefully.\n\n\n> The authors need to clearly differentiate the source of CoT's effectiveness in improving reasoning—whether it stems from manually labeled key states or the textual thought process itself. This distinction should be experimentally validated and presented in the main text rather than being relegated to the appendix.\n\n\nA: Once again, we apologize for any confusion in our presentation and thank the reviewer for highlighting this point. Our intention is not to compare the performance of different prompts, but to **present an end-to-end RL fine-tuning paradigm that effectively utilizes customized CoT prompting for VLM**. However, we agree that studying the effects of different customized prompts using intermediate reasoning steps [1,2] could potentially further improve the performance, and we would like to leave that for future research.\n\n## Q.2: \n> Moreover, regarding the CoT issue, Figures 11 and 13 show that CoT encodes not only the key states used for decision-making but also information \"directly obtained from the formula in the image.\" There seems to be no reasonable justification for this, as theoretically, this information can be derivable from the images. An effective RL method should be capable of optimizing and identifying this critical information through rewards. \n\nA: We sincerely thank you for your valuable suggestions to further improve the presentation of our work. We agree that the formula can be directly obtained from the image in principle; however, the backbone VLM (Llava-1.6b) struggles to recognize the formula from the images, which causes the entire pipeline to fail.\n\n\nTo demonstrate this, we provide additional evaluation results on the visual recognition capabilities of EZPoints, and Points24 in the Table below. In the table below, we have conducted additional experiments to evaluate the accuracy of the visual recognition for cards and equations in EZPoints and Points24, on the original llava-1.6-7b checkpoint (0-shot) and the supervised fine-tuned (SFT) checkpoint. \n\nThe results in the table below are tested with the same prompt in the paper and averaged among 1000 trials. For recognizing numbers, each trial is considered correct, if *all* numbers (or cards) are correctly recognized. For example, in the EZpoint and Points24, a trial is considered correct if the VLM recognizes the numbers in *all* cards. For recognizing equations, each trial is considered correct if the VLM successfully recognizes the equation in the image.\n\n\n| Recognition Accuracy | 0-shot          |               | SFT            |               |\n|----------------------|-----------------|---------------|----------------|---------------|\n|                         | Numbers         | Equation      | Numbers        | Equation      |\n| EZP                 | 0.10            | 0.07          | 0.79           | 0.12          |\n| P24                 | 0.01            | 0.03          | 0.48           | 0.03          |\n\n\n\nAs shown in the table above, while the VLM can reasonably recognize the cards or numbers from the images, it encounters significant difficulties in recognizing the equations – even for EZPoint, the accuracy of recognizing equations is roughly 12% after supervised fine-tuning. Hence, if we ask the VLM to recognize both the numbers and the equation at the same time, the probability of the agent being able to recognize both the equation and the cards throughout the entire trajectory becomes very low – e.g., in the EZPoint environment, the probability of the agent that can successfully recognize the equations for at least 3 steps, is less than $0.12^3\\approx0.0017$. \n\n**Page 1/ 2, to be continued.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EAwb1azZVG", "reviewer": "Reviewer_RYx1", "summary": "- This paper studies the training of vision-language models for decision-making tasks via reinforcement learning.\n- The authors train a 7B parameter Llava model and a baseline CNN model with proximal policy optimization (PPO) on the alfworld and `gym_cards` environments.\n- Additionally, the authors study the impact of also generating and reinforcing CoT rationales produced by the VLM.\n- They find that RL is often able to substantially improve the performance of a fine-tuned VLM, and that producing CoT tokens significantly helps during exploration.", "questions": "- I would have liked to see more of an explanation of why RL does not outperform SFT across the board on Alfworld.", "limitations": "Not applicable.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This is a useful empirical paper that serves as a sanity check for using RL on multimodal tasks.\n- I was surprised to see the impact that reinforcing CoT tokens had. I didn't expect this. This could be a useful signal for the community.\n- The experimental setup is sound and well-designed. I especially liked the carefully prepared Llava-sft baseline.\n- The paper makes a convincing case that RL adds something beyond SFT, at least for `gym_cards`.", "weaknesses": "- Only one VLM. I would like to have seen whether the conclusions change (even on a single environment) with a different VLM or different sizes of VLM.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer RYx1,\n\nThank you very much for your high appreciation of our work. We are delighted to hear that you found our results in CoT and SFT insightful.\n\n---\n### General response\n\nIn addition to your appreciation, we plan to incorporate the following results and discussions into the updated paper based on your valuable suggestions and questions:\n1. **We have conducted additional experiments on the recently released [Cambrian-1](https://github.com/cambrian-mllm/cambrian) model, which has significantly better vision recognition capabilities.** Detailed results are provided below.\n2. **We will include a more comprehensive explanation of the discussion on RL and SFT in Alfworld**, with further details also provided below.\n\n---\n### Experiments on additional VLM\nThank you for your suggestion. We conducted additional experiments using the recently released [Cambrian-1](https://github.com/cambrian-mllm/cambrian), a VLM with enhanced visual capabilities, on the EZPoints and Points24 tasks. We provide all experiments using the same setup in the paper but with Cambrian-1-8b as our backbone VLM.\n| Base VLM                        | EZP | P24   |\n|---------------------------------|-----|-------|\n| Cambrian-1-8b                   | 54.0% | 9.1%   |\n| Llava-1.6-7b (same in paper)    | 50.0% | 2.3%  |\n\n\nAs shown in the table above, the final performance improves when the backbone VLM has better visual capabilities (`EZP: 50.0% -> 54.0%`, `P24: 2.3% -> 9.1%`), which further demonstrates that (1) our framework can adapt to other VLMs as well; (2) VLMs with better visual capabilities will enjoy better final performance. **Note that we have not extensively swept the hyperparameters on Cambrian-1 due to the limited time, we believe the performance of Cambrian-1 can be further improved after more parameter sweeping.**\n\nTo see how Cambrian-1 is better than Llava-1.6 quantitatively, we ran additional experiments to compare the number recognition accuracy in `EZP` and `P24`. More specifically, we evaluate the accuracy of the original llava-1.6-7b / Cambrian-1-8b checkpoints (0-shot) and the supervised fine-tuned (SFT) checkpoint. The results below are tested with the same prompt in the paper and averaged among 1000 trials. Each trial is considered correct, if *all* numbers (or cards) are correctly recognized. For example, in the EZpoint and Points24, a trial is considered correct if the VLM recognizes the numbers in *all* cards.\n\n| Recognition Accuracy | Llava 0-shot | Llava SFT | Cambrian 0-shot | Cambrian SFT |\n|----------------------|--------------|-----------|-----------------|--------------|\n| EZP                  | 0.10         | 0.79      | 0.92         |     0.98     |\n| P24                  | 0.01         | 0.48      |  0.70        |     0.73      |\n\n\n**In conclusion, we showed that (1) our framework can still improve the performance of another VLM (Cambrian-1-8b); (2) and using Cambrian-1-8b, a mode with better visual capability, as the backbone will achieve much better performance compared to Llava.**\n\n---\n### Explanations on why RL does not improve across all the SFT tasks in ALFworld\n\nThank you very much for raising this insightful question. One potential explanation for this phenomenon is that the `ALFWorld` is a multi-task environment, where each task in `gym_cards` is trained in a single-task environment, which makes `ALFWorld` substantially harder than tasks in `gym_cards`. Following this line of thought, it would also be interesting to explore how RL can improve the multi-task learning capability of VLMs using the proposed framework. We sincerely thank the reviewer for raising this intriguing question, and we will include it in the discussion section in our updated draft.\n\n---\n### Concluding remarks\nWe highly appreciate your suggestion for additional experiments on another VLM and discussions on the performance of ALFworld. We will definitely include them in the discussion section in the future update. **If you think our additional experiments and discussion address your concerns, would you mind kindly further improving your rating?** Thank you again for your appreciation and your valuable suggestions for improving our paper!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SgdzEqklow", "reviewer": "Reviewer_QEzb", "summary": "This paper provides a framework that fine-tunes a large vision-language model (VLM) with reinforcement learning (RL) for decision-making tasks requiring vision and language understanding.\n\nIn the framework, the VLM takes as input a state s_t that contains a visual observation o_t and input prompt v_t^{in}. The input prompt v_t^{in} contains a task description, legal action space, and desired output. Then, the VLM generates text v_t^{out} that contains CoT reasoning and an open-ended text action. The framework applies a post-processing function f on the open-ended text action, and obtain a legal action a_t. Then, the framework inputs the legal action a_t to the environment and obtains reward r(s_t, a_t) and the next visual observation o_{t+1}. Based on the reward r(s_t, a_t), the framework fine-tunes the VLM with RL (i.e., PPO).\n\nThe proposed framework is evaluated on two environments: gym_cards and ALFWorld. The gym_cards environment contains four tasks like NumberLine, EZPoints, Points24, and Blackjack, and these tasks require an ability to recognize numbers in figures for arithmetic reasoning. This paper empirically demonstrates that the LLaVA-7B model fine-tuned by the proposed framework can outperform close-source LLMs such as GPT4-V and Gemini on these environments.", "questions": "Q1. Regarding the weaknesses W1 above, is there any reason the performance gain does not be significant on ALFWorld?", "limitations": "L1. The authors provide limitations in the Section 7 (i.e., Conclusions, Limitations, and Future Directions).", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "S1. This paper provides a method that can fine-tune VLMs with RL for decision-making tasks. The overall problem setting seems novel and interesting.\n\nS2. Technically, the method consists of (1) prompt design for domain-specific outputs, (2) post-processing open-ended text for legal actions, and (3) estimating action probabilities of VLM policies. Among them, the third technique is very interesting. To generate a better text action, the VLM generates CoT reasoning text before it. To estimate action probabilities of VLM policies, this paper proposes to split the probability of generating CoT reasoning and the probability of generating a text action.\n\nS3. This paper empirically demonstrates that LLaVA-7B model fine-tuned with RL can outperform GPT-4V and Gemini on the gym-card environment that requires vision and language understanding.", "weaknesses": "W1. Even though this paper demonstrates that LLaVA-7B fine-tuned with RL outperforms GPT4-V on the gym-card environment, according to Table 2, on ALFWorld, the performance gain does not seem significant (GPT4-V 19.4 vs Ours 21.7).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer QEzb,\n\nThank you very much for your high appreciation of our work, we are glad to hear that you appreciated the novelty, technicality, and performance of our paper!\n\n---\n\n### Regarding the explanation of the limited performance on ALFWorld\n\nBesides your appreciation, we would like to provide some potential explanations for the limited performance of the ALFWorld environment as you have mentioned in the weakness and question sections. The ALFworld environment is generally harder than the gym_card environment in these two following aspects:\n\n1. Each game in the `gym_card` is a single-task environment, while the `ALFworld` is a **multi-task environment**, which substantially increases the difficulty for RL training in `ALFworld`, as the agent needs to learn more diverse actions based on the types of tasks. In fact, showing improvements in the `ALFworld` environment also sheds light on the multi-task learning capabilities of our framework.\n2. In addition, the action space of `ALFworld` is generally much larger than the action space in gym_cards, which makes the learning process of `ALFworld` more difficult. More specifically, see Figure 17 on page 19 for example: (1) the agent is required to choose from more than 40 admissible actions at every step; (2) and at each step the admissible action set may be different. \n\n---\n### Concluding remarks\n\nWe highly appreciate your suggestion for the additional discussion on the limited performance of the `ALFWorld`, and we will definitely include them in the discussion section. **If you think our response addresses your concerns or our additional experiments improve the quality of this work, would you mind kindly further improving your rating? If not, please let us know as well, and we would like to further engage and improve our work based on your future suggestions!**  Thank you again for your appreciation and your insightful suggestions for improving our paper!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JWhi9himM0", "reviewer": "Reviewer_gYxm", "summary": "The paper introduces a algorithmic framework that fine-tunes VLM using RL  to enhance their performance in multi-step, goal-directed decision-making tasks. The authors highlight the limitations of traditional visual instruction tuning, which relies on pre-collected datasets and may not effectively train VLMs for interactive decision-making scenarios. Their proposed framework addresses this by providing task descriptions to the VLMs, prompting them to generate chain-of-thought (CoT) reasoning, and translating these into executable actions within an environment. Empirical results show that this method significantly improves the decision-making capabilities of VLMs, enabling them to outperform commercial models like GPT4-V and Gemini.", "questions": "It would be interesting to explore how visual prompting could enhance the framework and improve performance.", "limitations": "yes.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written and clear, making it accessible and easy to follow.\n2. The proposal of an alternative to current visual instruction tuning for decision-making is intriguing, with well-justified design choices that demonstrate a thoughtful approach.\n3. The execution is particularly noteworthy and well-designed. For instance, the method of balancing the influence of chain-of-thought (CoT) output and action output is elegantly handled and demonstrates innovative thinking.", "weaknesses": "The paper is good overall. There are a few areas that could benefit from improvement:\n\n1. The current implementation requires fine-tuning for each task individually. However, large Vision-Language Models (VLMs) are generally capable of handling multiple tasks such as Visual Question Answering (VQA) and Optical Character Recognition (OCR).\n2. The trade-off between the CoT output and action output is manually set. This means that the balance must be adjusted for different tasks depending on their nature. It is unclear how this parameter can be set for training a multi-task policy.\n3. Despite the use of Low-Rank Adaptation (LoRA) for fine-tuning, the computational cost remains high.\n4. The framework shows promise, but the evaluation tasks do not fully convey the benefits of using a multi-modal large language model (MLLM). Specifically, the model's ability to reason or the potential to explicitly correct its own behavior in scenarios where this is crucial is not thoroughly demonstrated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer gYxm, \n\nThank you very much for your positive feedback and insightful suggestions on our paper! We are delighted to hear that you appreciated the presentation of our work and our innovative method!\n\n--- \n### On the computational cost of LoRA\n\nWe’d like to note that the computational costs of our LoRA approach are comparable to standard supervised LoRA fine-tuning. A computationally similar scenario, RLHF, has been extensively studied and proven to work well, even in resource-constrained academic labs. Additionally, by switching between different LoRAs with the same base model, we only maintain one set of VLM weights, reducing VRAM consumption to approximately one-third of standard PPO training. Looking ahead, we are excited about future algorithmic improvements to PPO that will further reduce training costs.\n\n---\n### On multi-task training\n\nThank you for pointing this out. Our current framework only allows training in a single environment but is not limited to a single task. For example, experiments in ALFWorld suggest that our framework can perform well in a multi-task environment containing different types, potentially shedding light on its multi-task learning capabilities (in a single environment) as well.\n\n\n### Regarding other weaknesses and questions\n\n1. **On the hyperparameter of the CoT tokens.** We agree that manually setting a hyperparameter to adjust the log probabilities of the CoT tokens is not algorithmically elegant. This could be improved by providing an auto-tuning mechanism to handle the additional hyperparameter. Due to limited development time, we are unable to conduct such additional experiments during the rebuttal period, and we would like to leave this for future studies.\n2. **On exploiting the capability of MLLM.** We agree that we have only provided a working version of the end-to-end RL training framework, without extensively exploiting the capability of the MLLM. Using our framework, one can extensively try other methods (e.g., different prompting techniques) to exploit the capabilities of the MLLM.\n3. **On the visual prompting.** Thanks for the suggestion! While our current framework does not naturally allow visual prompting (VP), it is indeed interesting to see the performance of incorporating VP into our framework, we would like to leave this for future study.\n\n---\n### Concluding remarks \nWe highly appreciate your appreciation of our work, suggestions for improvements, and potential future studies! Please be assured that we will include an additional paragraph to discuss the computational cost compared to RLHF, and the multi-task training capabilities in the updated version. Although we are only able to address 2 out of your 4 concerns, **if you think addressing these 2 concerns or our additional experiments further improve the quality of our work, would you mind kindly improving the rating of our work?** However, if you feel this is not sufficient, we understand, as the other suggestions you made are already very important future directions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear reviewers,

We would like to express our sincere gratitude for your overall positive recommendations and insightful suggestions regarding our work. Based on the overall feedback, **we have conducted two additional experiments**, which we believe further enhance the quality of our work. Specifically, we have added:

1. **An evaluation of the visual recognition capabilities of the backbone VLM (Llava-1.6-7b) on EZPoints, and Points24**, illustrating that the visual recognition capabilities of the backbone VLM are a major bottleneck for further performance improvement.
2. **Additional experiments using the recently released [Cambrian-1](https://github.com/cambrian-mllm/cambrian) model**, which has better visual capabilities, to demonstrate that (1) our framework can work with another VLM (as suggested by reviewer RYx1), and (2) with improved visual capabilities, task performance can be further improved. 

---
### Additional evaluation results on the visual capabilities of the backbone VLM (Llava-1.6)
In the table below, we present the results of additional experiments evaluating the accuracy of visual recognition for cards and equations in EZPoints and Points24, using the original Llava-1.6-7b checkpoint (0-shot) and the supervised fine-tuned (SFT) checkpoint. The results are based on 1000 trials and tested with the same prompt used in the paper. For number recognition, each trial is considered correct if all numbers (or cards) are correctly recognized. in EZPoints and Points24, a trial is correct if the VLM recognizes the numbers on all cards. For equation recognition, each trial is correct if the VLM successfully recognizes the equation in the image.


| Recognition Accuracy | 0-shot          |               | SFT            |               |
|----------------------|-----------------|---------------|----------------|---------------|
|                         | Numbers         | Equation      | Numbers        | Equation      |
| EZP                 | 0.10            | 0.07          | 0.79           | 0.12          |
| P24                 | 0.01            | 0.03          | 0.48           | 0.03          |


As shown in the table above, the number recognition accuracy of our current backbone VLM (Llava-1.6-7b) decreases (`0.10 -> 0.01`) when the task difficulty increases (`EZP -> P24`). While the supervised fine-tuning can improve the accuracy of all tasks: `EZP` (`0.10 -> 0.79`), and `P24` (`0.01 -> 0.48`), the backbone VLM still suffers from recognizing the equations from the images, even after supervised fine-tuning.

**In conclusion, these evaluation results suggest that the backbone VLM suffers from poor visual recognition**, which can be slightly improved by supervised fine-tuning, but SFT does not always enhance visual capabilities across all tasks.

---
### Additional experiments using another backbone VLM (Cambrian-1)
To evaluate the adaptability of our framework with another VLM, **we conducted additional experiments using the recently released [Cambrian-1](https://github.com/cambrian-mllm/cambrian), a VLM with enhanced visual capabilities, on the EZPoints, and Points24 tasks as the backbone VLM for our framework**. The experiments were conducted using the same setup as in the paper but with Cambrian-1-8b as the backbone VLM.
| Base VLM                        | EZP | P24   |
|---------------------------------|-----|-------|
| Cambrian-1-8b                   | 54.0% | 9.1%  |
| Llava-1.6-7b (same in paper)    | 50.0% | 2.3%  |

As shown in the table above, the final performance improves when the backbone VLM has better visual capabilities (`EZP: 50% -> 54%`, `P24: 2.3% -> 9.1%`), which further demonstrates that (1) our framework can adapt to other VLMs as well; (2) VLMs with better visual capabilities will enjoy better final performance. **Note that we have not extensively swept the hyperparameters on Cambrian-1 due to the limited time, we believe the performance of Cambrian-1 can be further improved after more parameter sweeping.**
To quantify how Cambrian-1 compares to Llava-1.6, we ran additional experiments to evaluate number recognition accuracy in `EZP` and `P24`. Specifically, we evaluated the accuracy of the original Llava-1.6-7b / Cambrian-1-8b checkpoints (0-shot) and the supervised fine-tuned (SFT) checkpoint. The results are based on 1000 trials, using the same prompt as in the paper. Each trial is considered correct if all numbers (or cards) are correctly recognized.

| Recognition Accuracy | Llava 0-shot | Llava SFT | Cambrian 0-shot | Cambrian SFT |
|----------------------|--------------|-----------|-----------------|--------------|
| EZP                  | 0.10         | 0.79      | 0.92         |     0.98     |
| P24                  | 0.01         | 0.48      |  0.70        |     0.73      |



**In conclusion, we showed that (1) our framework can improve the performance of another VLM (Cambrian-1-8b), and (2) using Cambrian-1-8b, a model with better visual capability, as the backbone achieves much better performance compared to Llava.**

Meta-review: This work introduces a novel approach using reinforcement learning to enhance the decision-making capabilities of vision-language models (VLMs). Three reviewers have expressed positive feedback, while one reviewer has raised some concerns. The Area Chair finds the work interesting and recommends it for publication at NeurIPS 2024. The reviewers have highlighted some valuable concerns that should be addressed in the final camera-ready version. The authors are encouraged to incorporate the necessary revisions in the final submission.
Decision: Accept (poster)