Towards Efficient Online Tuning of VLM Agents via Counterfactual Soft Reinforcement Learning
Authors: Lang Feng
Weihao Tan
Zhiyi Lyu
Longtao Zheng
Haiyang Xu
Ming Yan
Fei Huang
Bo An
Venue: ICML 2025 poster (2025)
Keywords: vision-language model
agent
reinforcement learning
online fine-tuning
counterfactual
Primary area: deep_learning->foundation_models

Abstract: Online fine-tuning vision-language model (VLM) agents with reinforcement learning (RL) has shown promise for equipping agents with multi-step, goal-oriented capabilities in dynamic environments. However, their open-ended textual action space and non-end-to-end nature of action generation present significant challenges to effective online exploration in RL, e.g., explosion of the exploration space. We propose a novel online fine-tuning method, Counterfactual Soft Reinforcement Learning (CoSo), better suited to the textual output space of VLM agents. Compared to prior methods that assign uniform uncertainty to all tokens, CoSo leverages counterfactual reasoning to dynamically assess the causal influence of individual tokens on post-processed actions. By prioritizing the exploration of action-critical tokens while reducing the impact of semantically redundant or low-impact tokens, CoSo enables a more targeted and efficient online rollout process. We provide theoretical analysis proving CoSo's convergence and policy improvement guarantees, and extensive empirical evaluations supporting CoSo's effectiveness. Our results across a diverse set of agent tasks, including Android device control, card gaming, and embodied AI, highlight its remarkable ability to enhance exploration efficiency and deliver consistent performance gains. The code is available at https://github.com/langfengQ/CoSo.

Towards Efficient Online Tuning of VLM Agents via Counterfactual Soft Reinforcement Learning
Lang Feng'
Weihao Tan
Zhiyi Lyu'
Longtao Zheng'
Haiyang Xu² Ming Yan
Fei Huang? Bo An
Abstract
decision-making agents for autonomous tasks in both physi- cal or virtual environments, with applications in device con- trol (Wang et al., 2024b; Hu et al., 2024; Furuta et al., 2024), gaming (Wang et al., 2024a; Tan et al., 2024b), robotic control (Zhao et al., 2023; Zitkovich et al., 2023), and au- tonomous driving (Tian et al., 2024).
Online fine-tuning vision-language model (VLM) agents with reinforcement learning (RL) has shown promise for equipping agents with multi- step, goal-oriented capabilities in dynamic envi- ronments. However, their open-ended textual ac- tion space and non-end-to-ent nature of action generation present significant challenges to effec- tive online exploration in RL, e.g., explosion of the exploration space. We propose novel on- line fine-tuning method, Counterfactual Soft Re- inforcement Learning (CoSo), better suited to the textual output space of VLM agents. Compared to prior methods that assign uniform uncertainty to all tokens, CoSo leverages counterfactual rea- soning to dynamically assess the causal influence of individual tokens on post-pree post-processed actions. By prioritizing the exploration of action-critical tokens while reducing the impact of semantically redundant or low-impact tokens, CoSo enables a more targeted and efficient online rollout process. We provide theoretical analysis proving CoSo's convergence and policy improvement guarantees, and extensive empirical evaluations supporting CoSo's effectiveness. Our results across di- verse set of agent tasks, including Android de- vice control, card gaming, and embodied AI, highlight its remarkable ability to enhance ex- ploration efficiency and deliver consistent perfor- mance gains. The code is available at https: //github. www.o///aangg///////cooooo
Despite these advantages, adapting VLM agents dynamic environments remains challenge due to the limited task- specific capabilities of the base models (Zeng et al., 2023; Chen et al., 2023). Recent advances (Bai et al., 2024; Zhai et al., 2024; Wen et al., 2024) leverage reinforcement learn- ing (RL) (Sutton & Barto, 2018) for online fine-tuning, which has shown particular promise. RL enables VLM agents iteratively interact with online environments and optimize for multi-step, goal-directed objectives. This aligns naturally with the requirements of autonomous systems. For instance, in Android device controls, agents manage multi- step workflows like launching the Amazon app. locating specific items, and completing purchases.
However, the RL fine-tuning of VLM agents presents unique challenges for the online exploration. Two primary factors contribute to these difficulties. First, VL.M agents operate in a text-based action space, where the open-ended text of- ten includes thoughts, planning, and actions (Zhang et al., 2023; Niu et al., 2024). Unlike traditional RL, which en- ables vectorized action sampling via softmax distributions, the sampling space for VLM agents' textual outputs scales exponentially with the vocabulary size and sequence length. This immense policy space significantly complicates effec- tive exploration. Second, action generation in VLM agents is not end-to-end. Generated textual outputs (utterances) need to be post-processed into executable actions via pars- ing functions that match the environment's API calls (Zhai et al., 2024; Bai al., 2024). introduces misaligmment between exploring textual utterances and exploring final parsed actions. Many tokens in the utterances, particularly those with fixed formats or semantically redundant elements, do not contribute meaningful changes to the post-processed actions. As result, naive RL exploration becomes ineffi- cient and struggles to drive meaningful progress.
1. Introduction
Large Vision-Language Models (VLMMs) like Qwen-VI (Bai al., 2025b), Gemini (Team al., 2023), and GPT-
4o (Achiam et al., 2023), increasingly employed
'Nayyang Technological University FAlibaba Group. Corre- spondence to: Bo An <boan @ edu.c.u.sg>
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
In this study, we propose a novel RL fine-tuning method, better suited to the online exploration of VLM agents' tex-

---

Towards Efficient Online Tuning of VLM Agents via Counterfactual Soft Reinforcement Learning
tual action space. The core insight of our approach is that the influence of tokens on the parsed action varies, with small subset of action-critical tokens decisively shaping the final outcome. In contrast to naive RL training that treats all tokens as equally important, our proposal, Coun- terfactual Soft Reinforcement Learning (CoSo), analyzes the causal contributions of VL.M-generated tokens to the actions. By leveraging counterfactual reasoning com- pute causal weights for tokens, CoSo introduces causal- weighted entropy-regularized optimization built upon soft RI. (Haarnoja et al., 2017). This mechanism performs more targeted and efficient exploration, focusing on the causally significant tokens that induce meaningful variations in ac- tions rather than random trial-and-error. As such, CoSo significantly reduces the redundant exploration space and enables the exploration of open-ended utterances align more closely with the exploration of final parsed actions.
ized tasks.
Reinforcement learning for LLM/VLM training. RL has become cornerstone for improving the capabiliitess of LLMs and VLMs. A prominent application is Reinforce- ment Learning with Human Feedback (RLHF) (Ouyang et al., 2022; Rafailov et al., 2024), which aligns models with human preferences and ethical guidelines by iteratively refining their outputs. Also, RL has been applied en- hance reasoning capabilities like DeepSeek-RI (Liu et al., 2024a; Guo et al., 2025), enabling models to perform com- plex multi-step reasoning and logical deductions. In the LLM/VLM agent domain, works like (Wen et al., 2024; Zhai et al., 2024; Bai et al., 2024; 2025a; Wang et al., 2025) have employed RI. for online fine-tuning, creating agents that can adapt and learn from dynamic environments. Yet, they rely on classic tndal-and-rrror for online exploration, which can be inefficient in open-ended text action spaces.
We provide a theoretical justification of CoSo's 's convergence and effectiveness. In our analysis, CoSo establishes guaran- tees for both policy evaluation and policy improvement. As CoSo provides a highly general objective, it can be flexibly adapted to the more specific RL objective for agents' fine- tuning. In our empirical evaluations, we extend CoSo two RL fine-tuning objectives: an AWR-based approach (Bai et al., 2024) and a PPO-based approach (Zhai et al., 2024), subjecting VLM agents across various autonomous tasks, including Android device control (Rawles et al., 2024), card gaming (Brockman, 2016), and embodied AI (Shrid- har et al., 2021). Our results demonstrate its impressive ability enhance the efficiency and effectiveness of online fine-tuning, Over existing RL frameworks, CoSo-reinforeed agents achieve consistent performance improvements, with average gains of 12.3% in Android device control. 9.3% in card gaming, and 16.7% in embodied Al tasks.
Exploration in RL. Effective exploration remains a key challenge in RL, particularly in complex environments. In- trinsic motivation methods (Bellemare et al., 2016; Pathak et al., 2017; Burda et al., 2018) address this challenge by introducing auxiliary reward signals that drive agents to ex- plore novel states. On the other hand, entropy- -bsed meth- ods (Ziebart, 2010; Haarnoja et al., 2017; 2018; Ji et al., 2024) encourage diverse policy behaviors by increasing de- cision uncertainty. While effective in balancing exploration and exploitation for RL agents operating in vectorized ac- tion spaces, such uncertainty on textual action could inadver- tently complicate exploration for VLM agents. Our method reduces unnecessary complexity by prioritizing impactful to- kens, enabling efficient online fine-tuning for VLM agents.
3. Preliminaries
Problem Settings and MDP. The task-completion process of a VLM agent involves completing a series of actions in response to a user instruction in natural language to achieve specified goal. The entire process of agent-environment interactions can be modeled as finite-horizon Markov De- cision Process (MDP) (Sutton & Barto, 2018), defined by the tuple (S, P, r., 7). The state space S combines visual and textual modalities, defined as S = O x V'T, where O denotes the RGB image space, V denotes the vocabulary (token) space, and m denotes the maximum token length of the input sequence. A is the action space, consisting of executable actions translated from the VI.M-generated utterances via deterministic parsing (post- prost-processing) func- tion franse y + AA with n denoting the maximum out- put token length. The transition probability function P S x A S - [o. 1] defines the likelihood of transitioning between states given an action. r S x A [rmin: Tmaa is the reward function and y is the discounted factor. At each time step t, the VLM agent occupies state 8; = (o,, 2z),
2. Related Work
LLM/VLM as agents. Recently, there has been surge of interest in using large language models (LLMs) or VLMs as goal-driven agents in various domains like code (Zhang et al., 2024b), device control (Gur et al., 2024), gam- ing (Wang et al., 2024a), and robotic control (Zitkovich et al., 2023). One line of works focuses on test-time scaling by incorporating sophisticated modules (Wang et al., 2024b;
Tan et al., 2024b), prompting techniques (Yao et al., 2023;
Shinn et al., 2024), external tools (Schick et al., 2023;
Xie et al., 2024; Zhang et al., 2024a) with frozen models like GPT-4 (Achiam et al., 2023). Another line explores improving the task adaptability of agents by fine-tuning the base models with Supervised Fine-Tuning (SFT) (Zhang & Zhang, 2024) or RL (Carta et al., 2023; Tan et al., 2024a). These fine-tuned models often exhibit improved generaliza- tion and robustness, making them better suited for special-
2

---

Towards Efficient Online Tuning of VLM Agents via Counterfactual Soft Reinforcement Learning
where or E O and æ Vmm, and then generates an action a f (y:) with the utterance Yr V., This action induces transition to the next state 81+11 = P(s:, ar), and the agent receives a reward r(s a.).
vocabulary size: |V|* 32, 100100 (assuming a vocabulary size of 100 and a sequence length of 100 tokens). No- tably, this space includes many tokens with fixed formats or semantic redundancies, which contribute little to meaningful action exploration. This immense combinatorial complex- ity significantly impedes exploration efficiency during the online fine-tuning of VLM agents, posing a fundamental challenge for RL online optimization.
RL fine-tuning for VLM agents. VLM with parameter 0 is used as the policy of agent no (a(aa)).. i.e., given a envi- ronment state s, output an action a. To use RL to online fine-tune the VLM agent, the process typically follows two key steps: (1) Online exploration: the VLM agent interacts with the environment to sample rollout data based on its current policy. (2) Exploitation: the collected rollout data is then used to optimize the VLM policy using RL algorithms like PPO (Schulman et al., 2017). The general objective of RL. fine-tuning is to maximize the expected cumulative re- ward: max, >e
4. Training VLM Agents with CoSo
While VLM policies offer the advantage of executing in- termediate reasoning and planning steps, enabling efficient cognitive steps that lead to final decisions, their text-based outputs cause the online exploration highly inefficient when using RL for fine-tuning. This problem grows more pro- nounced as the sequence length increases. To address this, we introduce our Counterfactual Soft Reinforcement Learn- ing (CoSo) in this section. CoSo innovatively introduces a counterfactual structure to analyze the causal relationships between the VLM -LM-generated tokens = (y',...,') v²) and the post-processed action a. Built upon this, CoSo intro- duces a causal-weighted entropy-regularized optimization based on soft RI. (Haarnoja et al., 2017). This process en- hances agents' exploration of action-critical tokens (< 10% in our experiments) while minimizing perturbations to low- impact tokens (>> 80%). Considering our previous example, CoSo can effectively narrow the sampling space to approxi- mately 10 key tokens out of 100 total tokens, significantly improving the efficiency of exploration by 32, 10000-fold
[ az)], where p" is the state-action visitation distribution under policy æ. We omit the subscript of Ty for convenience.
Online exploration with entropy regularization. Effi- cient data sampling during the online exploration phase is crucial for optimizing the VLM agent's performance. To this end, RL often incorporates an entropy regularization term encourage exploration, yielding the soft RL optimization, the objective of which can be expressed as:
max Σ Blor (r((((s) a.) + [(()(++s)))),, (1)
where 2 0 is weighting coefficient, and H(((sss))
denotes the entropy of the policy.
RL agents, the policy of the VLM agent is not an end-to-end function but rather a multi- step, segmented process: (1) The VI. first autoregres- sively generates an output utterance y = (y'.... (y'....,y") ,"") via [["") 1))= s). where y typically includes elements such as tf (CoT) (Wei et al., 2022) and textual actions. (2) The sequence y is then transformed into an executable action a through a deterministic parsing function a = frunee (y). In this context, the entropy term ((()(-1s) over action a in Equation (1) is decomposed into the sum of the conditional entropies over the tokens y¹ .y:
Note that, unlike traditional
In the following sections, will delve into the process of deriving token-to-action causality through counterfactual reasoning, outline the causal cuusal-weighted entropy-regularize objective, provide a theoretical analysis of CoSo's effective- ness, and finally present its practical implementation.
4.1. Token-to-Action Causality
In V-.--ooeeeed agents, the relationship between the gen- erated utterance y and the resulting parsed action a can be viewed as a structured causal process. Each token in the sequence y contributes to the action a via frane, yet not all tokens y² y.,..., y exert the same level of influence on a. Some tokens are pivotal in determining the action, while others have negligible or no causal impact. To formalize this relationship. we explicitly encode the token-to-action causal structure into the parsing function as follows:
H(T(-1s)) = E" H(y"ly 1
where ااااا denotes the conditional entropy of y² given 213-11 We provide the proof in Appendix A. This feature introduces a significant challenge for RL in train- ing VLM agents: explosion of the exploration space. For instance, consider VLM agent applied to Android mobile control tasks. Typically, there are six actions avail- able: Type, Click, Slide, Home, Back, and Enter. er. In traditional RL, the action sampling space at each state is [A| = 6. However, in the case of VLM agents, the explo- ration space scales exponentially with sequence length and
a = frarse (By-ra ② y),
33)
where @ denotes the element- wise product, By E R is vector that represents the graph structure encoding both causal directions and causal effects. Specifically, By = O indicates the absence of a causal edge between token y and action a, while B = c (where e > 0) implies that y
3

---

Towards Efficient Online Tuning of VLM Agents via Counterfactual Soft Reinforcement Learning
Environment
Observation &Roward
Unterance y
Action Plan: (DUAL POINT,
"[0,1941, 0.6172) "typed text":
RL Fine-T Fine-Tuning
Google
VLM Agent
Counterfactual
CoSo
Utterance
Causal Weights Action Plan: "99,194, 0.6112)
By-ra
Data
POINT
Parsing via fpase
"oyped text
Replay Buffer
Action
Causal-Weightte 2thh (=(-1))
Entropy
Exploration & Rollout
CoSo
Figure 1. Overview of the CoSo framework for online fine-tuning VI.M agents. The process includes two modules: (1) Exploration and rollout, where the VLM agent interacts with the environment generate online data; (2) CoSo, which fine-tunes the VLM for multi-step. goal-directed objectives while promoting efficient exploration of criticlli tokens. Darker red indicates higher causal importance.
has causal influence on with effect magnitude c.
This probabilistic formulation is straightforward to imple- ment, for instance, by using surrogate model with a softmax output layer to instantiate the SCM. Building on this, we define the distance metric D(.-) as the likeli- hood difference and thereby the causal weight By for each token y is given by
4.2. Counterfactual Reasoning
To determine the causal weights By- we leverage counter- factual reasoning, which facilitates the analysis of "whattiff" scenarios by considering hypothetical interventions. To for- malize this, we begin by introducing the basic assumption for conducting counterfactual analysis.
B y-a = P(aly. ca) - P(aayyy " U yimll ca]l-
(6)
This approach enables a more sensitive measurement of the causal influence each token y' exerts on the action a, thereby facilitating a nuanced analysis of the interactions between VLM-generated utterances and the post-proeessed actions under the SCM framework.
Assumption 4.1 (Structural Causal Model). We assume that the relationship between the tokens y and the action can be captured using structural causal model (SCM) defined by a fSCM (y. Ca). where ca represents an exogenous independent and identically distributed (i.i.d.) noise term.
Then, we compute the causal weight of each token y' by measuring the sensitivity of the parsed action to coun- terfactual interventions on y'. Specifically, for each token yy, we compare the action resulting from the original sequence y with the action resulting from a counterfactual sequence in which y' is replaced with a nullified value v'null while keeping all other tokens unchanged. Formally, this process is defined as:
4.3. Counterfactual Soft RL
Based on the causal weights By-ra. we next propose our counterfactual soft RI., novel approach that explicitly lever- ages the identified per-token causal importance to guide the maximization entropy policy optimization of VLM agents. The central idea of CoSo is to prioritize exploration around cetion-critical tokens in the generated utterance y instead of aggregating uniform uncertainty across all tokens. Specifi- cally, we introduce causal- weighted entropy term into the soft RL objective (Equation (1)), yielding:
By-na = EE [D (FSCM(B) (y. ca). FSCM(g fSCM - yuun [
(4)
where -). is a distance metric that quantifies the change in the action a resulting from the intervention. y denotes the sequence excluding token y²
max Σ,
E Eo [at (r(st, ai) + 20H (=((1s:)))
where (T(((11()) = =E Byya H(y' ly 1 (7)
In practice, the effect of intervening a single token y on PSCM can be subtle, making it challenging to observe an absolute shift on parsed action a. To capture the fine- grainer effects, we analyze changes in the output likelihood of the action rather than absolute shifts in outcomes. Specifically, we define JSCM (y, <a) using probabilistic model followed by an argmax operator, which can be expressed as:
Intuitively, if token y is causally important (B) high), we prioritize exploring its possibilities since their changes can effectively lead to the changes of the final parsed action. Conversely, tokens that do not lead to meaningful changes in a are assigned lower entropy weight, reducing the incentive to explore them. By doing so, CoSo achieves a more effi- cient exploration process, targeting the space with a higher likelihood of meaningful impact on action.
|SCM (y, ea) = arg max ((a y, sa).
(5)
4

---

Towards Efficient Online Tuning of VLM Agents via Counterfactual Soft Reinforcement Learning
Algorithm 1 Online fine-tuning VLM agents with CoSo
4.4. Implementation
1: Require: VLM with parameters 0, SCM with parame- ters o. parsing function frane, and environment env.
In our implementation, we follow prior works (Ouyang et al., 2022; Bai al., 2024) and adopt a two-stage offline-to- online fine-tuning process to train VLM agents with CoSo.
2: for k 0 K 1 do
3: Initialize the environment 8D env. resete
4: Initialize a replay buffer 21 =
Offline phase. Before RL, we initialize the VLM agent using SFT on task-specific datasets. This step is critical for equipping the agent with prior knowledge of the task, such as the expected input-output formats and valid utterances y interpretable by the parsing function fearse, By grounding the model in task-relevant knowledge, the offiin phase ensures the VLM starts with good initialization, preparing the agent with foundational skills and structures necessary to interact effectively with the environment later.
5: // /Roolouut phase
6: for t = 0, 2...... do
7: Sample output from VLM: y: ~ mo()18))
8:
Parse action: a = f (yr)
9:
r,, 8: env. ste
u u= (,, T,,
11:
end for
12:
I/ Counterfactual reasoning phase
13:
Generate nullified sequences: {(y
{yy- U yimn) N=
14:
Compute causal weights By-a via Equation (6)
15:
Compute the causal-weigg weighted entropy 70 (To(-1a))
Online phase. In this step, the VLM agent interacts with the environment and leverages CoSo to refine its policy, Unlike the offline phase, where the model's learning is con- strained by static datasets, the online phase allows the agent to dynamically adapt to the environment through real-time feedback. The whole online process is outlined in Figure I and Algorithm 1. Specifically, we instantiate SCM using a lightweight BERT-based (Devlin et al., 2019) network & with 0.01 parameters, which is used to approximate the probabilistic function I (aly,, Ea). Each iteration starts with a rollout phase to sample online data, where the agent interacts with the environment to explore diverse states and transitions. Next, the causal weights By-a are computed via counterfactual reasoning, a critical step in guiding the agent's 's exploration. This enables the agent to perform more targeted and efficient exploration, focusing on causally sig- nificant regions of the policy space rather than random trial- and-error. Finally, we update SCM using cross-entropy loss to improve its approximation of the causal relationships be- tween y and a and update VLM using CoSo objective to enhance the agent's decision-making and exploration.
16:
Model updates phase
17:
Update & via ropll,, co())
Update 0 via Equation (7)
19:
end for
While introducing causal-aware adjustments H (T)-ss) to guide exploration, CoSo retains the fundamental proper- ties of standard soft RL. (Haarnoja et al., 201 8), including convergence (Lemma 4.2, Proposition 4.4) and monotonic improvement (Lemma 4.3, Proposition 4.4). Below, we present key theoretical results.
Lemma 4.2 (Policy Evaluation). By Equation (7), the Bell-
man backup operator of CoSo is T (t. a²) r(s,, a.) +
-R - 8HH (T(-11((())))11)
Ba,, (Q( ac+1))
Let Q²: S A + R be an arbitrary initialization (with
|A| < co), and define the iterative process Q+++1 = TB
Then Q³ converges to the fixed Q-value as k + co.
Lemma 4.3 (Policy Improvement). Let E be any policy and ग be the solution to the maximization of Equation (7). Then, Qa(s,, a) 2 Q* (s, a) holds for V(s, a) S A and |A| < co.
5. Experiments
In this section, we present the empirical evaluations of the proposed CoSo across range of VLM agent tasks. We choose two existing RL fine-tuning methods, AWR-based DigiRL (Bai et al., 2024) and PPO-based RL4VLM (Zhai et al., 2024), and adapt CoSo directly. Our experiments are designed to demonstrate (1) the ability of CoSo to boost the performance of VLM agents over existing RL fine-tuning methods: (2) its ability to identify action-critical tokens and achieve more efficient exploration; (3) comprehensive ablation study to understand the contribution of CoSo's components.
Proposition 4.4 (Policy Iteration). Repeatedly applying the steps of Lemma 4.2 and Lemma 4.3 starting from any II obtains sequence of policies T1, ..,... which converges to πt, such that, QT" (8, a) 2 Q" a) for ((,8a) E S x A and [A| < cc.
All proofs are provided in Appendix A. Therefore, by lever- aging causal importance, CoSo can achieve a more purpose- ful exploration strategy, leading to faster and more efficient policy optimization for VLM agents, yet it does so without compromising the stability or robustness of the underlying RL algorithm. Moreover, since Equation (7) defines a highly general mntropy-based optimization objective, CoSo can be casily adapted to more specific RL objectives (c.g., PPO), making it as flexible as classical maximum entropy RI.
5.1. Performance Boosting
In this part, we evaluate CoSo's ability to boost the perfor- mance of VLM agents over existing RL fine-tuning methods
5

---

Towards Efficient Online Tuning of VLM Agents via Counterfactual Soft Reinforcement Learning
Table 1. Average success rates (%) on AitW General and AitW WebShopping. Results averaged over random seeds.
AitW General
AitW WebShopping
Train
Test
Train
Test
Avg.
Gemini 1.5 Pro
32.3
16.7
6.3
11.5
16.7
GPT-4V
5.2
13.5
3.1
8.3
7.5
Prompting
Gemini 1.5 Pro + AppAgent
14.6
16.7
5.2
8.3
11.2
GPT-4V + AppAgent
13.5
17.7
12.5
8.3
13.0
CogAgent
25.0
25.0
31.3
38.5
30.0
AutoUI+ + SFT
38.8
37.6
40.3
41.5
39.6
Learning
Online Filtered BC
53.2 + 7.6
50.9 8.2
59.5 + 7.7
52.8 + 6.8
54.1
DigiRL
64.6 + 7.8
62.7 5.3
68.1 +
64.2 6.1
64.9
CoSo (Ours)
72.9 + 5.9
71.3 6.5
77.0 + 7.8
70.5 + 8.2
72.9
search 1903", and ielecttle entry
You Madqadk playe That")
expen
interactions with Android emulators. This suggests the im- portance of online adjustment in enhancing VLM agents' ability to adapt dynamic environments. However, their ex- ploration of all tokens of utterance remains highly inefficient. In contrast, CoSo excels in online fine-tuning: achieving both higher exploration efficiency and performance gains over prior methods. Notably, CoSo achieves an average success rate of 72.9%, an improvement of 12.3% over the previous state-of-the-art.
Ait
Gym Cards
ALFWorld
Figure2 2. Examples of tasks in different environments: (1) AitW: mobile control tasks that involve interacting with GUTs to com- plete multi-step instructions. (2) Gym Cards: virtual card-based reasoning tasks requiring numerical understanding, arithmetic op- erations, and decision- makinn.. (3) ALFWorld: embodied Al tasks that combine visual understanding and textual reasoning to per- form complex, goal-directed actions in interactive environments.
Gym Cards. Next, we implement our approach on RI4VIM (Zhai al., 2024) on top of the LLaVA-1.6 (Liu et al., 2024b), and evaluate it within the Gym Cards en- vironment (Brockman, 2016), which requires numerical reasoning and arithmetic operations. It includes four tasks: Numberline, involving aligning a value with a specified tar- get using arithmetic operations, EZPoints and Points24 that require constructing arithmetic expressions to achieve target values, and Blackjack that introduces uncertainty by requir- ing decisions based game states to maximize success. The results are given in Table 2.
in both physical and virtual environments.
Android-in-ther Wild. We first implement CoSo based on DigiRL. (Bai et al., 2024) on top of the AutoUT (Zhang & Zhang, 2024), which is used for fine-tuning mobile agents. We evaluate CoSo on the General and WebShopping tasks from the Android-in-the-Wirn (AitW) (Rawles al., 2024), which features rich variety of real-world mobile control tasks that requires multi-step reasoning and interaction with graphical user interfaces (GUIs) on Android emulators, as illustrated in Figure 2.
Prompting-basee methods like Gemini-1.5 Pro and GPT-4V perform comparatively better in Gym Cards than in AitW. This likely due to the proficiency of VLMs in reasoning tasks related to code and mathematics. Nevertheless, their performance remains inferior to learning-based algorithms like LLa + SFT (Liu al., 2024b), CNN RL. and RLAVIM (Zhai et al., 2024), suggesting a limited scope without fine-tuning. With more purposeful online explo- ration, CoSo exhibits a greater ability to adapt efficiently to numerical reasoning and arithmetic operation and demon- strates 9.3% improvement over RL4VLM.
can seen in Table that prompting-ben bbssed methods, such as Gemini 1.5 Pro (Team et al., 2023), GPT-4V (Achiam al., 2023), and AppAgent (Zhang et al., 2023), face sig- nificant challenges in completing user instructions they rely pre-trained knowledge, which constrains their ability to adapt to task-specific requirements. Online fine-tuning approaches, like Online Filtered BC and DigiRL. (Bai et al., 2024), outperform offline learning techniques such as CogA- gent (Hong et al., 2024) and AutoUT SFT by facilitating
ForWorld We next test CoSo the ALFWorld (Shridhar et al., 2021). an embodied environment for testing decision- making in scenarios requiring visual-semantic understand- ing. The ALFWorld domain includes tasks like Pick& Place, ExaminelnLight, Clean&Place Heat& Place, Cool& Place.
6

---

Towards Efficient Online Tuning of VLM Agents via Counterfactual Soft Reinforcement Learning
Table 2. Average success rates (%) on Gym Cards and A..Wooldd For Gym Cards, the average performance is computed across all tasks, with each task weighted equally. For ALFWorld, the average success rate is calculated as a weighted mean to account for the varying occurrence probabilities subtasks in the environment. Results are averaged over random seeds.
Gym Cards
ALFWorld
NL
EZP
P24
BJ
Avg.
Pick
Look
Clean
Heat
Cool
Pick2
Avg.
Gemini-1.5 Pro
82.5
2.0
0.0
30.0
28.6
34.6
16.7
0.0
0.0
0.0
12.0
13.5
GPT-4V
65.5
10.5
0.0
25.5
25.4
38.2
12.1
18.8
6.7
17.8
14.6
19.4
Prompting
LLaVA-1.6
14.3
2.2
0.0
21.5
9.5
16.7
0.0
0.0
0.0
0.0
0.0
5.0
CNN + RL
87.1
0.0
0.0
38.8
31.5
0.0
0.0
0.0
0.0
0.0
0.0
0.0
LLaVA-1.6+ SFT
24.8
23.0
2.6
23.1
18.4
39.2
0.0
14.4
11.1
0.0
28.6
17.7
Learning
RLAVLM
88.4
50.0
2.5
39.3
45.1
44.7
13.8
18.3
15.8
14.3
16.8
22.7
CoSo (Ours)
100.0
50.0
5.8
41.5
49.3
42.5
16.7
21.4
11.9
22.1
25.9
26.5
and & Place, which involve reasoning over high- level goals and executing step-by-step plans in visual envi- ronments. As depicted in Figure 2, tasks such as Pick&Place require finding an object (e.g., a kettle) and placing it in a specified location (e.g., a countertop), involving navigation and object manipulation.
fective recovery in this situation requires corrective actions, such as pressing the "Back" or "Home" button. In this sce- nario, we compare CoSo's action exploration performance against other algorithms, including naive RL and RL with entropy regularization. Each algorithm samples multiple utterances to assess whether the agent can produce actions that enable recovery from its own mistake.
The results in Table 2 demonstrate that online fine-tuming in RL4VLM plays crucial role in task completion, allowing the agent to dynamically adjust its strategies during training. It achieves the sfte-offtthe-rr art result (22.7%), surpassing prompting-based methods like Gemini-1.5 Pro and GPT- 4V. Consistent with the results observed in AitW and Gym Cards, CoSo leverages causel-aware exploration to attain the best success rate of 26.5%, an improvement of 16.7%. These findings highlight the effecttivnness of CoSo fine- tuning VLM agents across various task domains.
As demonstrated in Figure (below), naive RL, without antropy-based exploration, tends to repeatedly select the "DUAL POINT" (i.e., click on the "Search Walmart") even when this button is unclickable in the given situation, results in repetitive and ineffective exploration. Indeed, naive RL can sample different utterances with varying planning re- sults and coordinates. However, as the exploration space of complete utterances is vast, most token changes cannot re- sult in meaningful variations in actions. Introducing entropy regularization into RL encourages more diverse action sam- pling and facilitates the generation of varied action plans. However, since it uniformly assigns uncertainty to all tokens during optimization, the agent often explores irrelevant to- kens or produces outputs in incorrect formats, as highlighted by the red examples in Figure 3 (below). In contrast, CoSo effectively identifies the causal importance of specific to- kens, such as "DUAL" in Figure 3 (above). Hence, CoSo can selectively introduce uncertainty to action-critical tokens by weighting their entropy during optimization (Equation (7)). This ensures more efficient sampling and aligns the sam- pling of utterances more closely with the sampling of final actions. As demonstrated, CoSo successfully samples one of the correct recovery actions ("Home")), which greatly con- tributes to the efffectiveness of RL online fine-tuning, See more exploration comparison in Appendix F.
5.2. Efficient Exploration
We now evaluate CoSo's ability identify action-critical tokens and facilitate more efficient exploration. In this periment, we first analyze the counterfactual results of CoSo and present the AitW results in Figure 3 (above). See Ap- pendix E for additional results in Gym Cards and ALFWorld. As observed, CoSo effectively identifies the causal impor- tance of individual tokens, highlighting red those that truly shape the action. Notably, these key tokens constitute only small proportion of the total (less than 10%), whereas over 80% of the tokens have weights below 0.2, indicating minimal influence on the final action. Prioritizing explo- ration of high-weight tokens proves beneficial, as variations in these tokens are more likely to drive meaningful action changes, thereby greatly improving exploration efficiency.
To further validate this, we test the VLM agent in challeng- ing Android mobile control scenario that demands efficient exploration to handle errors and recover from mistakes. As illustrated in Figure 3 (below), the agent mistakenly clicks the "Share" page upon entering the Walmart homepage. Ef-
5.3. Ablation Study
Next, conducted an ablation study the WebShopping and NumberLine tasks, comparing standard RL (RL), RL with naive entropy (RL + H), and CoSo (RL + 75) to
7

---

Towards Efficient Online Tuning of VLM Agents via Counterfactual Soft Reinforvement Learning
Counterfactual Results
Action Pan AMl
PR INT DUAL PPO WIT AT T6 .COM PLE TE Lidction Docston
"action
_hoe 1: UMU _PO IMT "ouu _poitt 1:1003 52 0. 463", Ii pein "10.3 B. "tyce 5. 1utt
Acrin Fann (1m RE FRES ENT ER SST AT T4 DOM PUE TE J:Aeton
Desiion
("antoon
Soe - RE much paine 1:30 .I30 15 Dore 11000 JED 15 100e Moe "Mag itech "odt
:IPSE
Arton Flon (IFRES 80 ME NO INT MREES 5, VE MAL FO WIT LiAction
Deciiio
:"action
7700 MIES S_HO NE "Duch _e "(LLL <30 Jr, "R pport "M"LiL .10 "ope d_20xt "P"
Task: Go walmurt.com, seurch for "lenovo thinkpad", and select first enity
Repeated Action Sampling of Naive RL
Action Plae: [DUAA_POITT JYIN,,YYPE.DUA STATUS, TASK [IKK_COMPETT]] Action Decision "actionn paction_tspe" "DUAT_PPIITT" "hwuch_poiiie "[00795, 0.3626)" "L8 puint: "[00705, 0.5626", "spel_teu"" texf" **
Action Plan: [DUAT, PEIT POINT,TYPE, DUAT, [PPINT]; Action Dacision "action_ type't "DUAT POINT,, "louch point" '[0.143, 0.8855J;
Wilmani Sanve Money, lee...
"sil poin": "[0. 143, 0.3855),, "spoolieat"" "*
Action Plan: [DUAL POINT.DUAL POINT,
POINT.DUAL POINT]: Action Decision: "action type": "DUAL POINT", "woch "wouchpomn"" "[0.1321,
0,5196), 'KR point: 1[0.1321, 0. 0.5996''. tept"_teut""" teut:
Action Plan: [DUAL KOINT,YYPEDDU POINTSSTATUS TASK COMPLETE) Action Decision: "action "action_sspe": "DUAL POINT", "DUAL_POINT"
"bouch point"
1345, 0.3855)", point" 19461 0.3855", "tped_teet" teat":
Departments
Repeated Action Sampling of RL. Cntropy
From S16:
Action Plan: [DUAL_ POINT.DUAL TYPE., POINT); Action Decision: "action type") "DUAL "lit "10.5381, 0.5855), "typed lex":
"DUAL_POINT", POINT", Touch_poant" "J0,5381, 0.5835T.
activewear for all Shos
Action [PRESSMMAA,TVE TYPE.DUUL BORNDDDUA.POIOSTTTAIAAT STANU, TAAS_AASKK_OO__ COMPLETE]]
"serion "serion_tpp"" "PRF45"; "tonch_ ponnt" "[-100 -1
"Ist. polm": 1.0)". "tsped Jea": *
"DUAATPPIITT" Action Plane [DU]A]_ KUPN,TYPEEDDA STANTS TASK COMM Action Decision "action "action_ispe" "DDUT POINT", "houch "Nouch_pair"" point"
"[0.1528 0.4008]", "L8 point:: 1[0.1528, 0.8066]" "spel_iea"" Jea",
Action Plan: [DU] DUAL, COLAAA_POINTSSSTVV POINTES, POINT] Action Daciiion: "action, baction_lype:: "DUAL POINTT, "monch"; "J0.3545, 0.4411..
"iile": "103353,0,.4 0,4780|'. "Syped
Repeated Action Sampling CoSo
Action Plan: [DUAL POINT, TYPEDDYPL. POINT,TAATUS TASK COMPLETT] Action Decision: "action, type": "DUAL POINT", "houch point: "[0.0978, 0.3006)" "IR point:: "[0,9988, 9, 0,3000" "sped_ten bett:: **
Oow
Action Plan: [DUAL, POINT,DCAL POIND,TYPE.DUAL.P POINT] ; Action Decision: "action, type': "DUAL POINT", "looch point:: '[0.143, 0.3622].
"SA poinn: "[o: 143, 36222" "ypool test" **
Action Plus: [DUAL IYPED.L POINESTATUS TASK COMPLETE] Action Decsion: "action "aclio__tppe; "DUAL POINT", "Nouch point"
"[007765, 360065"" "18_ point:: 0[0.0765 O, 36055" "sped_ter""
lexi":
Action Plan: IPRESS HOME,DUAL, POINT, DUAL POINT.DUAL POINT] Astion Decision: "actioo_ "PRESS HOME", "Vouch_poia" "[-L... 1.0]". "kst, point" T[-L., 1,0"". "sped, text":
Ground truthe "Back" or "Home"
Figure: 3. Counterfactual results (above) and repeated action sampling (below). For the counterfactual results, causal weights By are normalized to [0, 1]. In the repeated action sampling, the action fields are highlighted in blue, while format errors are indicated in red.
WobShopping
NumberLine
issue posed by the textual action space of VLM agents. After introducing causal weights, CoSo introduces meaning- ful variability where it matters most, which can drastically reduce the exploration space and make the process more efficient. As shown in Figure 4, CoSo outperforms standard RL and RL with naive entropy in terms of both convergence speed and final task success.
1.0
Steccess Rate
Stocess Rate
0.8
0.6
0.4-
RL
RL
RI. 7..
0.4
RL 7
0.2
RL
2
RL 7
25
50 Steps
75
100
0.2
10
20
Steps
6. Conclusions and Limitations
Figure Ablation results. The curve of WebShopping smoothed with exponential weighting over the x-axis.
In this paper, address the challenge of inefficient explo- ration in RL online fine-tuning of VLM agents. We pro- posed a novel and effective solution, CoSo, which leverages counterfactual reasoning to assess the causal importance of tokens. By prioritizing the exploration of action-critical tokens instead of random trial and error, CoSo introduces a principled approach to efficient fine-tuning. Its general ob- jective enables adaptability to various RL objectives, making it scalable framework for future advancements in online RL fine-tuning, Theoretical analysis and empirical results both support the effectiveness of CoSo. Notably, it consis- tently outperforms existing RL fine-tuning methods across diverse agent tasks, including Android device control, card
evaluate the impact of each component on performance. The results are presented Figure 4.
As shown, RL with naive entropy provided only marginal improvement over standard RL. This is because applying uniform randomness to all tokens fails to focus on those that are truly critical for action generation. It also leads to ineffi- cient exploration and increases the likelihood of generating invalid or irrelevant outputs, as discussed in Figure 3. As a result, naive entropy struggles to address the exploration
8

---

Towards Efficient Online Tuning of VLM Agents via Counterfactual Soft Reinforcement Learning
gaming, and embodied AL. Despite its strong performance, the longest utterances in our experiments contain less than 300 tokens. Its effectivvness ultra-long CoT has not yet been explored. Investigating strategies to extend CoSo to such settings, possibly through hierarchical reasoning, is a promising direction for future work.
Brockman, G. OpenAl Gym. 1rXiv:160..01540, 2016.
arXiv preprint
Burda, Y., Edwards, H.., Pathak, D., Storkey, A., Darrell, T., and Efros, A. A. Large-scale study of curiosity-driven learning, arXiv preprint arXiv:1808 000,055,0 2018.
Carta, T., Romac, C., Wolf, T., Lamprier, S., Sigaud, O., and Oudeyer, P.-Y. Grounding large language models in interactive environments with online reinforcement learn- ing. In International Conference on Machine Learning, pp. 3676-3713. PMLR, 2023.
Acknowledgements
This research is supported by the National Research Foun- dation, Singapore under its Industry Alignment Fund - Pre- positioning (IAF-PP) Funding Initiative. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not re- flect the views of National Research Foundation, Singapore. This research is supported by the RIE2025 Industry Align- ment Fund- Industry Collaboration Projects (IAF-ICP) (Award F0006), administered by A*STAR, as well as supported by Alibaba Group and NTU Singapore through Alibaba-NTU Global e-Sustainability CorpLab (ANGEL).
Chen, B., Shu, C., Shareghi, E., Collier, N., Narasimhan, K., and Yao, S. Fireact: Toward language agent fine-tuning. arXiv preprint arXiv:2310.05915 2023.
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for lan- guage understanding. In Proceedings of the 2019 confer- ence of the North American chapter of the association for computational linguisties: human language technologies, volume I (long and short papers), pp. 4171 4171-4186, 2019.
Impact Statement
Furuta, H., Lee, K.-H., Nachum, O., Matsuo, Y., Faust, A.,
Gu, S. S., and Gur, I. Multimodal web navigation with instruction- finetuned foundation models. In The Twelfth International Conference on Learning Representations, 2024. URL httpt::tt////........ htap:://///peeeeeeewee net/ forum? i id=e FFmBWissc.
This paper presents work whose goal is advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
References
Guo, D., Yang, D.,, Zhang, H., Song. J., Zhang, R., Xu. R., Zhu, Q., Ma, S., Wang, P., Bi, X., et al. Deepseek-R1: Incentivizing reasoning capability in LLMs via reinforce- ment learning. arXiv preprint 122492501.12948 2025.
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S.,
Anadkat, S., et al. GPT-4 technical report. arXiv preprint N.Xv:2303308774.. 2023.
Gur, L,, Furuta, H., Huang, A. V., Safdari, M., Matsuo, Y., Bck, D., and Faust, A. A real-world webagent with plan- ning, long context understanding, and program synthesis. In The Twelfth International Conference on Learning Representations, 2024. URL https: http:::///ppeeeviie net net/ fert/formm? 1i:iidd-tttt rumugggg....
Bai, H., Zhou, Y., Pan, J., Cemri, M., Suhr, A., Levine, S., and Kumar, A. DigiRL: Training in-the-wild device- control agents with autonomous reinforcement learning. Advances in Neural Information Processing Systems, 37: 12461-12495, 2024.
Haarnoja, T., Tang, H., Abbeel, P., and Levine, S. Rein- forcement learning with deep energy-based policies. In International conference on machine learning, pp. 1352- 1361. PMLR, 2017.
Bai, H., Zhou, Y., Li, L. E., Levine, S., and Kumar, A. Digi- Q: Learning VLM q-value functions for training device- control agents. In The Thirteenth International Confer- ence on Learning Representations, 2025a. URL https: // //openreview. Saaleewweee net/ forum?
Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. Soft actor-critic: Off Off-policy maximum entropy deep reinforce- ment learning with a stochastic actor. In International conference on machine learning, pp. 1861-1870. PMLR, 2018.
Bai, S., Chen, K., Liu, X., Wang, J., Ge, W., Song, S., Dang, K., Wang, P., Wang, S., Tang, J., et al. Qwen2.5- VL. technical report. arXiv preprint arXiv:2502. 13923, 2025b.
Hong, W., Wang, W., Lv,, Q., Xu, J., Yu, W., J., Wang, Y., Wang, Z., Dong, Y., Ding, M., et al. CogAgent: A visual language model for GUI agents. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14281-14290, 2024.
Bellemare, M., Srinivasan, S., Ostrovski, G., Schaul, T., Saxton, D., and Munos, R. Unifying count-based ex- ploration and intrinsic motivation. Advances in Neural Information Processing Systems, 29, 2016.
9

---

Towards Efficient Online Tuning of VLM Agents via Counterfactual Soft Reinforcement Learning
Hu, S., Ouyang, M., Gao, D., and Shou, M. Z. The dawn of GUT agent: A preliminary case study with claude 3.5 computer use. arXiv preprint arXiv:2411.10333 2024.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. 2017.
arXiv preprint
Shinn, N., Cassano, F., Gopinath, A., Narasimhan, K., and Yao, S. Reflexion: Language agents with verbal rein- forcement learning. Advances in Neural Information Processing Systems, 36, 2024.
Ji, T., Liang, Y., Zeng. Y., Luo, Y., Xu, G., Guo, J.. Zheng, R., Huang, F., Sun, F., and Xu, H. ACE: Off- Off-policy actor-critic with causality-aware entropy regularization. In Forty-first International Conference on Machine Learn- ing, 2024.
Shridhar, M., Yuan, X., Cote, M.-A., Bisk, Y., Trischler, A., and Hausknecht, M. ALFWorld: Aligning text and embodied environments for interactive learning. In International Conference on Learning Representations, 2021. URL //ppnr/ei//eeeeiiei net forum? 1d=0I0X0Y<CoO
Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C.,, Zhao, C., Deng, C., Zhang, C., Ruan, C., et al. DeepSeek- V3 technical report. arXiv preprint arXiv:2412.19437, 2024a.
Liu, H., Li, C., Wu, Q., and Lee, J. Visual instruction tun- ing. Advances in Neural Information Processing Systems, 36, 2024b.
Sutton, R. S. and Barto, A. G. Reinforcement Learning: An Introduction. MIT press, 2018.
Tan, W., Zhang, W., Liu, S., Zheng, L., Wang. X., and An, True knowledge comes from practice: Aligning large language models with embodied environments via rein- forcement learning. In The Twelfth International Confer- ence Learning Representations 2024a. URL https: / /poenrevveww net Lit.am/
Niu, R., Li, J., Wang, S., Fu, Y., Hu, X., Leng, X., Kong, H., Chang, Y., and Wang, Q Screenagent: A vision language model-driven computer control agent. arXiv preprint ar Ktv:2402.07945, 2024.
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances Neural Information Processing Systems, 35:27730-27744, 2022.
Tan, W., Zhang, W., Xu, X., Xia, H., Ding, G., Li, B., Zhou, B., Yue, J., Jiang, J.. Li, Y., et al. Cradle: Empowering foundation agents towards general computer control. In NeurlPS 2024 Workshop on Open- World Agents, 2024b.
Team, G., Anil. R., Borgeaud, S., Alayrac, J.-B., Yu, Sori- cut, R., Schalkwyk, J., Dai, A. M., Hauth, A., Millican, K., et al. Gemini: A family of highly capable multimodal models. arXiv preprint arXiv:2312.
Pathak, D., Agrawal, P., Efros, A. A., and Darrell, T. Curiosity exploration by self-supervised predic- tion. In International conference on machine learning, pp. 2778-2787 PMLR, 2017.
2023.
Tian, X., Gu, J., Li, B., Liu, Y., Wang, Y., Zhao, Z., Zhan, K., Jia, P., Lang, X., and Zhao, H. Drivevlm: The conver- gence of autonomous driving and large vision-language models. arXiv preprint arXiv:2402. /2289, 2024.
Peng. X. B., Kumar, A., Zhang, G., and Levine, S. advantage-weighter regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00117. 2019.
Wang, G., Xie, Y., Jiang, Y., Mandlekar, A., Xiao, C., Zhu, Y., Fan, L., and Anandkumar, A. Voyager: An open-ended embodied agent with large language models. Transac- tions Machine Learning Research, 2024a. ISSN 2835- 8856.
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Er- mon, S., and Finn, C. Direct preference optimization: Your language model secretly a reward model. Ad- vances in Neural Information Processing Systems, 36, 2024.
Wang, J., Xu, H., Jia, H., Zhang, X., Yan, M., Shen, W., Zhang, J., Huang, F., and Sang. Mobile-Agent-v2: Mobile device operation assistant with effective naviga- tion via multi-agent collaboration. Advances in Neural Information Processing Systems, 37:2686-2710, 2024b.
Rawles, C., Li, A., Rodriguez, D., Riva, O., and Lillicrap. T. Android in the wild: A large-scale dataset for android de- vice control. Advances in Neural Information Processing Systems, 36, 2024.
Wang. T., Wu, Z., Liu, J., HAO, J., Wang. J., and Shao, K. DistRL.: An asynchronous distributed reinforcement learn- ing framework for on-device control agent. In The Thir- teenth International Conference on Learning Represen- tations, 2025. URL https: openreview net// forum? forumiid=1.@@pPPff
Schick, T., Dwivedi- Yu, J., Dessi, R., Raileanu, R., Lomeli, M., Hambro, E., Zettlemoyer, L., Cancedda, N., and Scialom, T. Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems, 36:68539-68555 68551, 2023.
10

---

Towards Efficient Online Tuning of VLM Agents via Counterfactual Soft Reinforcement Learning
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F,, Chi, E., Le, Q. V., Zhou, D., al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35; 24824-24831 2022.
Proceedings the 62nd Annual Meeting of the Associ- ation for Computational Linguisties (Volume I: Long Papers), pp. 13643-13658, 2024b.
Zhang, Z. and Zhang, A. You only look screens: Mul-
timodal chain-of-action agents. In Findings of the As- sociation for Computational Linguistics ACL. 2024, pp. 3132-31499 2024.
Wen, M., Wan, Z., Wang, J., Zhang, W., and Wen, Y. Reinforcing LLM agents via policy optimization with action decomposition. In The Thirty-eighih Annual Conference on Neural Information Processing Systems, 2024. URL https::/ http:///pppeeeeeeeee/mes net forum? Id=H26cSigMy
Zhao, H., Pan, F., Ping, H., and Zhou, Agent as cerebrum, controller as cerebellum: Implementing an embodied Imm-based agent drones. arXiv preprint arXiv:2311.1000 2023.
Xie, T., Zhang, D., Chen, J., X., Zhao, S., Cao, R., Hua, T. J., Cheng, Z., Shin, D., Lei, F., Liu, Y., Xu, Y., Zhou, S., Savarese, S., Xiong, C., Zhong, V., and Yu, T. OS- Benchmarking multimodal agents for open-ended tasks in real computer environments. In The Tiiht Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024.
Ziebart, B. D. Modeling purposeful udaptive behavior with the principle of maximum causal entropy. Carnegie Mel- lon University, 2010.
World:
Zitkovich, B., Yu, T., Xu, S., Xu, P., Xiao, T., Xia, F, Wu, J., Wohlhart, P., Welker, S., Wahid, A., et al. RT-2: Vision- daiio-language-actionnn models transfer web knowledge robotic control. In Conference on Robot Learning, pp. 2165-2183. PMLR, 2023.
Yang. A., Yang. B., Zhang, B., Hui. B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F,, Wei, H., et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115. 2024.
Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K. R., and Cao, Y. ReAct: Synergizing reasoning and acting in language models. In The Eleventh In- ternational Conference on Learning Representations, net/ forum?
2023. URL
BL-NE_vluYUL-
Zeng. A., Liu, M., Lu, R., Wang, B., Liu, X., Dong, Y., and Tang, AgentTuning: Enabling generalized agent abilities for LLMs. arXiv preprint arXiv:2310.1282) 2023.
Zhai, S., Bai, H., Lin, Z., Pan, J., Tong, P., Zhou, Y., Suhr, A., Xie, S., LeCun, Y., Ma, Y., et al. Fine-tuning large vision-language models as decision-making agents via reinforcement learning. Advances in Neural Information Processing Systems, 17:110935-11099 2024.
Zhang, C., Yang, Z., Liu, J., Han, Y., Chen, X., Huang, Z., Fu, B., and Yu, G. Appagent: Multimodal agents smartphone users. arXiv preprint arXiv:2312.13771, 2023.
Zhang, C., Li, He, S., Zhang, X., Qiao, B., Qin, S., Ma, M., Kang, Y., Lin, Q. Rajmohan, S., et al. UFO: A UI- focused agent for windows OS interaction. arXiv preprint arXiv:2402.07999. 2024a.
Zhang, K., Li, J., Li, G., Shi, X., and Jin, Z. CodeAgent: Enhancing code generation with tool-integrated agent systems for real-world repo-level coding challenges.

---

Towards Efficient Online Tuning of VLM Agents via Counterfactual Soft Reinforvement Learning
A. Proofs
A.1. Equation (2)
Proof. The entropy (-((()))) over action equivalent to the entropy the output utterance y = (y","f.... .....,,'' vy",, which defined as:
H(yy) -Ey [log (T( s)]
(A.1)
By Σ log no (1/ 1
(A.2)
Ey
[log no (1'l 1: x)]
(A.3)
To (1-1) s)]
(A.4)
To (o(y)"1ss) log To (v (v"ly (u 1y s]]
(A.5)
H(v (( 1:
(A.6)
where Hly' y 1) denotes the conditional entropy of y² given 1
A.2. Lemma 4.2
Lemma 4.2 (Policy Evaluation).
Derived from Equation (7), the Bellman backup operator of CoSo is given by (
T ar) = r(St, az) + >BEumiP arH (T(-|S(+1)) + Eatt (Q(111 at+1)]]. ((+1)+ Let Q" x + an
arbitrary initialization (with [A] < 0o). and define the iterative process Q++1 TB Then Q* converges to the fixed Q-value as ks co.
Proof. The Q-function of Equation (7) given by
Q(8, a) =E =E aa)..)a Jup
r(su, ap) + Σ
72t (r(s), a.) aaH (HH((T(-(s))) 8, 8,
(A.7)
1=1,2..
((s0, ao) + + + aH s1)) 7 ((s2, a2) ()).)))) :0000000- 8, ag a
(E Supr
This recursive structure simplifies to:
Q(s, a) =
(0HH((((( ( s')) + Ea (Q('''''
(A.8)
We see that the Q-function can be updated iteratively using the operator:
T ar) r(s), a.) + >Ba,,1>p a0H ( + (0 Then, we re-organize the operator:
(A.9)
TBQ(s,, a,) =r(s;, a.) + 7 aMl ( (T(()(((11) + (Q(= a+tii]
(A.10)
=r(s;, a:) + 77 wp
aaH (I (-((11122)] + Baysor 1 (((4411,4411)
(A.11)
B az) 71 [Q(5+11 a141),
(A.12)
12

---

max Towards Efficient Online Tuning of VLM Agents via Counterfactual Soft Reinforvement Learning
where (8;; at) is the CoSo-augmented reward. The key observation is that 7 ( 8)) is bounded above for any policy and state since it is weighted sum of the (per-token) discrete entropies H(y") y 111- Specifically, for each token y¹ and weights By yy-aa 2 O, we have
0 S
B Byza 1=1
H(y"l 11 2 max Biyxx) nolog [/.
(A.13)
where V is the token (vocabulary) set. As such, r (st; a+) is bounded by I'min: Tmax + max; Bi - n Io [V] which satisfies the boundedness requirement standard policy evaluation (Sutton & Barto, 2018). As a consequence, 7 remains a contraction mapping in an appropriate norm (e.g., an lo norm), guaranteeing convergence Q*.
A.3. Lemma 4.3
Lemma 4.3 (Policy Improvement). Let æ E 1I be any policy and π be the solution to the maximization of Equation (7). Then, Q² (s, a) (s, a) holds for v(s a) ESX and |A| < co.
Proof. By Equation (A.7), the CoSo policy objective can be defined as
Fa (s), (st, + atd (x(-1s))
(A.14)
x
If consider greedy maximization from we have
Eava (Q. + aH (f(-1s) Bawt (Q (s, a)] + a
(A.15)
In similar way to the proof of the soft policy improvement (Haarnoja al., 2017), we derive the following inequality (we denote r(t ar) as Tt for convenience):
Q a) ro +EEEs aH
* (m(111)) Eaj (s) ai)]
s E., [ro + 7 (a7t (i(- 81)) Eai~il (Q ai)))]
aivi iro + 7a77 (+(- 81)) + + 7 B., aHH ( 82)) + Bas~ 10 a2)) a2)]
Eg as Ni,
ru Σ (
1,1.
Q (s.
A.4. Proposition 4.4
Proposition 4.4 (Policy Iteration). Repeatedly applying the steps of Lemma 4.2 and Lemma 4.3 starting from any т E II obtains a sequence of policies T2,..... which converges to π², such that, Q (s, a) 2 Q= a) for v(s a) E E x A and |A| oc.
Proof. By Lemma 4.3, the sequence of Q-function (Qmi, Q== .) is monotonically improving. Moreover, it converges since the sequence of Q Q-function is bounded from above as discussed in Lemma 4.2. Let Q" be its limit. By continuity of the contraction operator (Lemma 4.2) and the construction of each improved policy (Lemma 4.3), the limiting Q" must be the unique Q= satisfying Q=" (s, a) 2 QT* a), Yk and V(s, a) E S A. Consequently, x is fixed- exed-poiitt policy that cannot be further improved.
13

---

Towards Efficient Online Tuning of VLM Agents via Counterfactual Soft Reinforcement Learning
B. Experiment Details
B.1. AitW
Overview. AitW provides a diverse set of real-world mobile control tasks designed to evaluate the ability to perform complex reasoning and interact with GUIs. We choose two sets in our experiments: General and WebShopping. The General set focuses on information search and basic application usage, such as searching for news in Chile and opening Gmail. The training set includes all 545 tasks, while the test set comprises the first 96 tasks. The maximum allowed steps per task is 10. The WebShopping set comprises search instructions various shopping websites, such as "go to ebay.com, search for "logitech g903", and select the first entry". The training set consists of 438 tasks, with 96 tasks reserved for testing. The maximum allowed steps per task is 20.
Observations. The observation space consists of the Android GUI screenshot and textual task descriptions provided as user instructions, and action history.
Google
Aha shrw ESO MAA R.
Prompt
Actions. The action space includes various types of mo- bile operations. Dual 1. Point: represents both click and slide actions through touch and lift points, which uses normalized coordinates to ensure adaptability across dif- ferent screen resolutions and device sizes. Coordinates are scaled to the range [0, I] relative to the screen dimen- sions. Type: allows text input; Home, Back, Enter: standard navigation actions.
Meoping
Vilwo
EGO Plwer ESI-DE 14inch St-ici
Previous Actions: "action "action_type": "TYPE". "touch point": "[-1.0, -1.0 "lift poins": "[- 1.0, -1.0]", "typed text"; "Whats price the EGO 14-in S6-Voli Brashless Cordless Chainsaw"
Condless
whate th top
Goal: What''s the price of the EGO 14-in 56- Volt Brushless Cordless Chainsaw
Baselines. We compare the performance of several base- line models using scores reported in Table of (Bai et al., 2024). The baselines include: Gemini 1.5 Pro (with and without AppAgent), GPT-4V (with and without AppA- gent), CogAgent. We run AutoUI SFT, Online Filtered BC, and DigiRL, an online RL fine-tuning method based on AWR (Peng et al., 2019), using the official implementation'.
Figure B.I. Example of observation and prompt in AitW.
CoSo. We implement CoSo on DigiRL (Bai et al., 2024) with 0.0II BERT-based SCM. Similar to the VLM agent, SCM is trained in two stages: offline and online. The learning rate for SCM is set to 1 x 10-5, and the entropy coefficient a is set to 1.0. All other hyperparameters remain consistent with the DigiRL setup.
B.2. Gym Cards
Overview. Gym Cards is a "gym-like" environment designed to evaluate a VLM's vision recognition and arithmetic reasoning. It consists of four tasks, including NumberLine, EZPoints, Points24, and Blackjack, designed to evaluate the betision-making capabilities of VLM agents. The first three tasks are deterministic, focusing on number recognition and arithmetic reasoning with increasing complexity. Blackjack, on the other hand, is a stochastic task that requires reasoning based on visual inputs and adapting to uncertain outcomes.
Observations. The observation space includes visual inputs, such images of cards or numbers, and textual task descriptions provided as prompts. See an example of the observation and prompt in Figure B.2.
Actions. The action space varies by task. In the NumberLine, actions are + and -, representing increments or decrements to the current number. In the EZPoints and Points24, actions include natural numbers (1-10) and arithmetic operators (+, *, =, etc.). In the Blackjack task, actions are stand (to hold) and hit (to request new card).
The baseline models evaluated in our experiments include Gemini 1.5 Pro, G CNN + RL, LLaVA-1.6 + SFT (results sourced from Table 2 in (Zhai et al., 2024)). Additionally, we run LLa 1.6 and RL4VLM, an online RL fine-tuning method based on PPO (Schulman et al., 2017), using the the official implementation of RL4VLM.
Baselines.
CoSo. We implement CoSo on RL4VLM (Zhai et al., 2024) using a 0,01B BERT-based SCM. Similar to AitW, the learning
'https://gi thub. /https://githie www.hh.wwo.......ooo......o.eeto /t///RLLVVIMMRRR
14

---

Towards Efficient Online Tuning of VLM Agents via Counterfactual Soft Reinforvement Learning
Dealer
Prompt
You are a blackjack player. Given the current state, you can choose between ['stand', 'hii" J. Your response should valid json following format:
Player
("thoughts!":
[fiest describe total points and the dealer's total points then think about which action to
choose],
V"action": ("sand)" !"hit!"
Figure B.2, Example of observation and prompt in Gym Cards.
rate for SCM configuration of PPO and the scaling factor À for CoT reasoning, were consistent with the RL4VLM configuration.
10 with the entropy coefficient 1.0. All other hyperparameters, including the
B.3. ALFWorld
Overview. ALFWorld is embodied AI enviromment designed to evaluate decision-making tasks requiring visual semantic reasoning. The environment integrates text-based interaction with visual observations, allowing agents to complete household-related tasks in a simulated environment. There are 4639 games categorized into six types: Pick & Place, Examine in Light, Clean & Place, Heat & Place, Cool & Place, and Pick Two & Place, which involve interacting with objects and their surroundings contextually appropriate ways.
Observations. At each state 8 ALFWorld, the agent observes an RGB image and text-based description of the environment. These observations provide the necessary visual and semantic information for the agent to make decisions. The text-based descriptions inherited from the Text World environment, offering detailed contextual information about the objects and receptacles in the scene. See example of the observation and prompt in Figure B.3.
Prompt
Embodied Enviromment, are given following text description of the current scene: [ Welcome to TexiWorld, ALFRED! m-inninnoou are the middle of
expert
room. Looking quickly around you, you see countenop 1, sink sinkbasin drawer 1, drawer 2,
drawer 3, drawer 4, drawer 5, drawer sink 2, sinkhasin drawer 7, drower and toilet Your task to: find two toiletpaper and put them in drawer). Your task is to put two toiletpoper admissible actions
'go sink I'm 'go drawe 2'n 'go to drawer 3'm 'go drawer 4'n 'go to drawer S'in 'go to drawer 6'm 'gu to sink 2'm 'gu to sinkhusin 2'm 'go to drawer 7'm 'go drawer 8'n 'go to toilet I'n 'inventory in 'look]. Your response should valid json file in the following format: "ntk ttoouhhoghhss"" i" (first deseribe what do you see in the image using the text description, then carcfully think about which to complete tack. 19",
situation are:
countertop
sinkbasin I'nn 'go drawer I'nn
[an admissible action) (""il
Figure B.3. Example of observation and prompt in ALFWorld.
The action space consists of admissible text-based commands, which vary across different tasks. For exam- ples, the action template includes "go to {recep}", "open {recep)", "close {recep}". "take {obi} from {recep}", "put {obi} in/on frecep)", "use {lamp}", "heat {obi] with fmicrowave)" "cool {obj} with [fridge}", "clean {obi} with {cleaner}", "slice {obi} with [knife)". "inventory". "look", "examine {obj}", and "examine (recep)". Here, {obj} represents objects and (recep) represents receptacles. The action space is state-dependent, meaning that only certain actions are available or valid in specific states, depending on the agent's current task and environment configuration.
Actions.
The baseline models evaluated include Gemini 1.5 Pro, GPT4-V, CNN RL, and LLaVA-1.66 SFT. Results for these baselines are sourced from Table 2 in (Zhai al., 2024). Additionally, we run LLaVA-1.6 .6 and RL4VLM using the official implementation of R.AVLM.
Baselines.
CoSo. We implement CoSo on RL4VLM (Zhai et al., 2024) using a 0.01B BERT-based SCM. The learning rate for SCM was set to x 1 and the entropy coefficient a was set 1.0, All other hyperparameters, including the configuration of PPO and the scaling factor for CoT reasoning, were consistent with the RL4VLM configuration.
15

---

Towards Efficient Online Tuning of VLM Agents via Counterfactual Soft Reinforvement Learning
C. Extension to LLM Agents
While our method is primarily designed for VLM agents, naturally extends to LLM agents operating in purely textual environments. To demonstrate this, we conduct experiments on AlfredTWEnv, fully text-based environment within the ALFWorld suite (Shridhar al., 2021).
We implement RL4VLM and CoSo using Qwen2. 5022..-777 3-7B-Instruce (Yang et al., 2024) and train the LLM agents from scratch (i.e., without SFT) over 12,000 environment steps. Table C.1 summarizes the results, showing that CoSo outperforms the RL baseline in average task success.
Table C.I. Performance comparison between RL4VLM and CoSo on the AliredTWEnv with LLM agents.
AlfredTWEnv
Pick
Look
Clean
Heat
Cool
Pick2
Avg.
RL4VLM (LLM Agent)
62.9
38.5
35.0
33.3
25.9
11.1
32.8
CoSo (LLM Agent)
77.1
24.2
40.7
37.5
35.3
7.0
39.6
D. Computational Budget
In this part, we evaluate the computational budge of CoSo's online fine-tuning and compare its overhead with the baseline RL method (Zhai et al., 2024) and RI with naive entropy regularization Numberl.nn The evaluation includes metrics such parameter count, memory usage, and training time, with all results obtained using NVIDIA H100 cores.
As summarized in Table D.I, CoSo introduces a modest computational overhead due to the need to compute causal weights for each token in the generated utterances. However, this overhead is relatively small because the SCM is implemented using lightweight BERT-based networks. Specifically, CoSo adds only 0.01 B parameters (< 0.2%), 0.7 GB of GPU memory (< 2%), and 0.5 H100 GPU hours (< 4%). This volume of the budget modest while achieving significant performance improvements in exploration efficiency and task success rates, shown in Section 5.1 and Section 5.2.
Table D.I. Computational budget including parameter count (B), GPU memory usage (GB), and training time (H100 GPU hours).
Parameter SCM
GPU Memory
Training Time Update
Method
VL.M
Total
VLM
SCM
Total
Rollout
7
Total
RL
7.96
N/A
7.96
37.0
N/A
37.0
10.4
3.5
N/A
13.9
RL + Entropy
7.96
N/A
7.96
37.0
N/A
37.0
10.4
3.6
N/A
14.0
CoSo
7.96
0.01
7.97
37.0
0.7
37.7
10.4
3.6
0.5
14.5
16

---

Towards Efficient Online Tuning of VLM Agents via Counterfactual Soft Reinforcement Learning
E. Visual Counterfactual Results
We present the visual counterfactual results in Figure E.1 and the distribution of weights in Figure E.2. As observed, the SCM can effectively identify the tokens that truly shape the outcome action. These tokens represent small fraction (< (<10%) of the total, while over 80% of the tokens have weights in the range of [O, 0.2), which indicates minimal impact on the action. Moreover, ALFWorld results indicate that utterances with longer CoT may contain higher proportion of low-impact tokens. By reducing the exploration of these tokens, CoSo substantially improves action exploration efficiency.
Weight
AitW
0.0
0.4
0.6
0.8
10
Acoon Plan: :B 110 INT, DMAML FOTTT .ST UT 16.6 FLE TE LACSER
1D UIG .PO TMuch LOWN "103 S 451" Ln.oe 1-103 525.0.54 "tee d_ 1-rheh
ATOO PaN 1100 PRES FIT TO CON OE Librion Denidin "arminn
"IE "OUCT .DO0T 1H1-- .330 15, "ER 2001 "IREE AND 15 thya set "rhgg lock 0:3 Teloo
51Ia _MAIT '.102
500
Azdin Pan (PRES Bill BR IMM PPO WT: ST IA J.SK .00 ALE LAAGOYY
Decierr
NIO FREES S. EMT BR "OUU) _OOE ':11 LB (128 1V, _peE 1/11/2 20 "Op0 F. EOX sto
Actin Man JIMES 9_M0 ME UAL 1MT JNES 5_ ME DUAL P. 1MT LAbcten Dociiio
Pacliee
Loee PRES 5, NE "DUKC _ 1:11:3. .10 1" "I8 _pprtt ".11-10 .10 1:70pp d_oot "o"e
Gym Cards
("oument number "2"2" "tarpt number 1yne "th ought ot: "The current rumber 1s2 Mith bamsller then the tarpet rumber s.f. To rove the current rumber dosser thethe begett /dhoul the summne rumber cheos ng action 5 kos
("rands 1:1261 Torm de wugtt "12 complete
fammaia
lahculd ieppodd
cummnt
formula
Fection
{ cands ":44 51, "orm dle 1:7 "thllg wit The curser
lormati "is serppay
Indbouid start winh any number
incards
in
/heraw lcccoo
ALFWorld
This is normal sale aper suugesss that this is the tsil rour nals Inced find (Since the action "Wl amine tolle to spor alls availabre iitis laaly that the solle ti apor In the image the onz reforred to it the taxt deser poio TTee logical step to cxamine the walle t apor to confirm itis the correct item before attempting put it the tollct it "action 12" ax aminc talle to sper " <os>
asink There isahand tow Jwhich isiikaly the that needs cleaned and out bat ub bas The description "Nothing heppern i'indicates that prior action has takem sigrificatt occurred Jo complete the task Jishould first ensure that the hand tow el :Theretore next
ouont st: "The
bathroom
:Since the the bas "ection hand Witth sink </5>
ell avallabeee jisuggests that
the before
("th ought stt "Based on the text description .there are multiple items on the dlskk inckuding alarm clocks .mugs accal phone CCD s.a May chain aaceedi card Japencil ladd apen However the lass isto examine the box with the des kl wittich isnot listed in text description or visitne the image provided :since the des M aamp nst mentined in the adm iss late actions the text descriation .and not visible in imaoe the best course of elttton isto look around the room for more information or find the des klamo /The "asst laction seerrs most aporooriate to oother more information about the ansironment 20 action 1." "Book <oti>>
ltwit
could the sazin the sounsor
mantireed
dadloodd livil start
mabinst 110 ifae biitddd 25 "ucion
213 -iio-
Figure E.I. Visual counterfactual results on AitW. Gym Cards, and ALFWorld, Darker colors indicate greater weights.
90
Percenlage (%)
80
AitW Gym Cards
70-
60
50
AL.FWorld
40
30
20
10
[0. 0.2)
[0.2, 0.4)
[0.4, 0.6)
0.8)
[0.8, 1.0]
Weight Interval
Figure E.2, Distribution of counterfactual weights By-a (normalized)
17

---

Towards Efficient Online Tuning of VLM Agents via Counterfactual Soft Reinforcement Learning
F. More Exploration Comparison
Ground inth: cup diningtable
Your are an expert the ALFRED Fibbodied Environment, You also given the following text description of the current scene: ["You arrive loc the coffeemachine I. you see a salishuer I. spoom and bread L.] task is to put hot cup in diningtable, Your admissible actions the current situiion are: F'go to fridge 1i 'go to cabinet 15n 'go
1e's ''o counertip 11' 'go sinktasin 1'a 'go cabinet 20 'go toaster 125n igo cabinet 3 'go to cabinot 4iam 'go to cabinet 57m 'go to drawer Fa 'go to drower 2 fgo to microwave I'nn Vao stoveborer 15a 'go to cabinet 6/'n 'go to stovebumer 2hn "go stoveburner 3 'go to garhagecan 15'm 'go to cabinet 'gu to cabinet 8''n spoon I from coffeemuchine 1'nn Make broad from coffeemachine 1''n Viiventorylta Vlookktt texamine coffoeracciie Your response should he valid joon file the following format: ( (wtthgghhhh"""" "[first describe what you see the imuge using the text description, then carefilly think about which action to complete the look, F", 'n In'action: "Jam admissible action "vil]
staveburacr
'go to diningtable I'n "aake sallshaker from coffeemachine I''n "take
Naive RL
1. "(W "thoughts": "Based on the text deseription, I soe saltshaker, a spoon, and a piece of bread on the cofTbe machine. To put hot cup on the dining table, I first need to find cup and heat it. Since there is no mention of cup on the coffee machine, I should look for a cup in the cabinets or on the countertops. After the water. However, first step
acquiring a cup. I would need use the coffee machine
a cup "action": "go to cabinet 1"a) lacilp
2. '(n "thoughts"; "Based on the text deseription, first nced find a cup, fill with coffee, and then coffee machine with salsshuker, spoce, and pieco of bread on To put hot cup the dining table. up. Since coffee machine is already view, should look for a cup nearby, possibly in cabinee or on countertop. After aequiring cup, I can then use the coffee machine to hent the water. However, the first step is to find cup, so I will to location where cups are commonly stored, such as cabinet." In "action": "go cabinet ("a)"r
3. "(n "thoughts": "In the
coffee machine with taaster the left and saltshaker, spoce, and piece of bread top of complete the task of putling a hot cup on the dining table. I need to find a cup. fill it with coffee, and then heat up. Since there is no direct action to 'heat cup'l Yake cupi, I should look for cup first, which might be a cabinet on a countertop. The action 'go to diningtable 1 will bring me claser to the dining table, but need cup before place it there. Therefore.
look cup the cabinets or on the counteriops." "In "action": "go to cabinet ["")) "orilss
4. "(u "thoughts": : "Based the description, I loc and there a coffee machine with saltshaker, spoon, and cup on the dining lable, I first need to find cup and heat it. Since there''s no mention a cup on the collee machine, I should look for a cup in the cabinets or on the countertops. After socuring a cup. I would need to use the coffee machine to heat the water or the microwave to heat the cup available. Since igo microwave I is an admissible action, suggests
of bread it.
hot
is microwave the environment. However, where cups are typically stered, such as cabinet"" "Jn "action: "go to cabinet ['n] wesp?"
to find cup finst, so should go to location
5.
"In "houught"" : "I can see a colTee muching and on it nexi to touster, lolte. There are no cups nourby that are listed under admissible actions To accomplish the task. However, need to find a cup that I could possibly hut if find coffeemaker could also fill it with water. Given my current available actions, the best cup first, and then likely use coffee machine fill with water. Since a cuffeemachine could have in land provide
course would be coffeemaker, and mag might not typically be stored in coffeemakerr it would he benoficial to use actions such as "loo!! or texamine coffeemachine 1 to gather more information and pussibly spot location mug use." " "action" "look" "m] meop"
6. 'In "thoughts": "Based on the text deseription, I see coffee machine with a salsshaker, spoon, and a piece of bread on it. To put a hot cup on the dining table, find cup hast Since ihere mention of's cup on the coffee muchine, should Onee I have the cup, I can use the coffee muchine to hest the water. However, the action to heat the water is not available, so will foeus on finding a cup first. The must logical next step go a location where cups usalll stured, such cabinet," JJn "wetion": "go cabinot 1"m] losiig"
für cup cabinuts
countertops.
7. "In "thoughts": "Based on the text deseription, see a coffee machine with a salsshaker, spoon, and piece of bread on it. To put a hot cup on the dining table, first nced find cup, fill with coffeee, and or an countenog. The action 'go to diningtable I` will bring me claser to the dining table, but I need the cup first. Therefore, I should look for a cup the nemby stanage areas" "Jn "action": "go to cabinol ["m] "nstii"
Since coflee muchine already view, should look for a cup nearby, possibly cabinet
8. 'In "thoughts": "Based on the text description, see a coffee machine with a salsshaker, spoce, and a piece of bread on it. To put a hot cup on the dining table, Since there mention of cup on the coffiee machine, should look for cup the cabinets the counteriaps. The action 'go to diningtable I'' is the final step, but I need to find cup first. will choose to go to location where cups are typically stored, such as a cabinet." "i "action": to cabinet ("a]nsiss"
first need find a and
9. "(u "thoughes" : "Based on the text description, I see salsshaker, a spoon, and a piece of bread on the coffee machine. To pur hot cup on the dining table. I first need find cup heat it. Since there mention cup the coffee machine, should look for cup likely location counteriop. After aequiring a cup, I would need to use the coffee machine to heat the water. However, the first step is to locate cup." i/n "action": "go to cabinet ('n] m2ss""
cabinet
10. "(n "thoughts": "In the image, I see coffee machine with a saltshaker, spoon, and piece of bread on top of which matches the text description provided. put hut cup on the dining table, for a cup nearby, possibly a cabinet or on a countertop. The 'go to diningtable I' action the final step to complete the task, but first. I need to find and heat the cup. Vn "action": "go cabinet [""]axsi""
find cup, with coffee, then heat it up.
coffee muchine already view, should look
Figure F.I. Ten repeated action samplings of naive RL.
18

---

Towards Efficient Online Tuning of VLM Agents via Counterfactual Soft Reinforcement Learning
Ground buih: cup in diningtale I.
Your are an expert in the ALFRED Embodiie Enviroement You are also given the following text description of the current scene: (oYou arrive at loc 7. On the coffeemachine you see a sahthaker a spoon I, and a bread 1.9. Your task put hot cup diningtable. Your admissible actions the current situaioo are: ['go fridge
'go to cabinet Ih "go lo sink Ihaa 'go to counteriop I'n Vgo to sink maki I'n 'go to cabinet 2/n fgo to loasier 1''n 'igo to cabinet 3ia
cabinet 4'a 13o cabinet 5'' stuvebumer F'a 'go to cabinet 6'nn 'gu to slavebumer 2n 'go sluvebumer 3 'go to garbageean In 'go to cabinet 7/nn 'go to cabinet 87n 'go to stavehumer 4''n 'go to diningrable 12 takee saltsloker 1 from coffeemachine 1i' taake spocm coffermachine
drawer Pia 'go drawer 2 'go to microwave IMa 'goo
Make bread from coffeemachine Va Viivventory Yoook"n 'examine coffeemachinee "tooughss " "[fins descrite what do you see the image using the deseription, then carefilly think about which action complete the task. I", "naction": "(an admissible action) "mit)
19]. Your response should be a valid joon file in the following format: "nnloy"""hhhhhh"
RL + Entropy
1. 'In "thoughts": : "In the image, I see coflee machine with iiems on lop of which mutches the tuxi description provided. On top of the voffee muchine, thure is sultshaker, spoon, and piece of bread. The task put hot cup on the dining table. Since cup visible in the current would typically find cups cabinet or on countertpp. cup. Therefore, will need to start by explaring the kitches to locase a cup for heating." ", in "action": "go to cabinet ["ss] "n)'srssss'
must find cup first, One action seems invelvant to the immediate objective of finding heating a
1/o to diningtable
the left, and other items fully visible. to the right the countertop what might be sall shuker, which is part of our text description provided Since the lask to pul hot cup on the dining tuble, finst, we need to find cup. IT's cup isn'l shrewdy on the counteriop or in our immediate view in the current situation, we might need to interact with locations such as cabinets where cups are commonly kept. nearby that could interact with, related items are. Unfortunately, based our current information,
Tin thougtss: "In the image, there coffee machine,
Since hot water where the coffeemachine
indicale there
subequent actions should be to 'go to countertop If similar specilie to best course action
choose 'go any location thar''s closely related to our goal, even though a direct 'go to diningrable IY would be in line with our final goal of the mikk completion. However, given time and constmints within the provided options and text (adhere to 'admissible action! and follow through on avnilable interationss, selecting "look" should give'
3. '/n "thoughts": "The text description of the scene indicatess that we are loc 7 near voffee machine with isems such a saltshaker, spoon, and bread on it. The bock is to put hox cup on the dining table, In the image, I can see the collee machine to the right, the tuaste to the left, and what appears to be sallsitkker to the find a cup, passibly fill it with coffee, heat up (likely using the coffee machine could potentially give us more information
right side of the touster. Given the actions available, the first
should
make coffee). about the coffee machine, but 'inventory!' would allow us to check we already have a cup in our passession." '/n "action": "inventory" iaj imilss
then place it on the dining table. The action 'go to coffeetable! or Texamine coffeemachine
'In" thuughts:: "Based on the lest description and provided imnge, we are situated tuk requires putting a 'hot cup' on the dining table. To achieve this took, we would need to the coffee machine is in fromt of us. Considering that therel's no listed option to grab or interact direcily with a bot cup, suggests that we may need to find the First the necesarr steps involve clearing the coffee machine prevent any
coflee machine with several items lop of The corfee machine with coffee and hest the contents. Currensly,
the cup, then use the coffee machine best the obstination, maving towards completing our task. Thus, we can start by examining the coffee machine, taking actions to interact with it "n"aciio""" i "examine coffeemschire "mn--lg""
Vccol cup." por
follows prompt for a cup, one these common components seen on a surface may match with what "go to coffoemachine references which includes a saltshakerr I- potentsally mistaken for a het loup". The available actions give various loc destiaatios, but to place "a bot cup on our floor or furniture is not directly among them. Normally, cups arenit inseracted with directly (other than possibly found) which himts this may imply to take it at tempenature
"Ia" thoughts: "In case
elearly into ply, it's impossible language commands was the phricking leup!" when the ustal isome object on a place like toaster to menn was used; may better refer to a 'toilld pager holder' such as on an office soilet, since these loeations commonly do not possess warm cupmaking procoss and only in realistic interpretate secrarios where cup is meniuoned action relste directly the 'foor mouniineud cup type". To move away a differeni angle related finding the we eventually need ('go to drawer xi;
dossnit
directly place anything
'ddring itable". Also staightforward for AL.
process
6. Vin "thoughts": "In the current scene, observe coffeemachine on which there are several items resting upon top: saltsoker, spoon, and piece of bread. the
possibly using coffee machine, there admissible action makes bend coffee, like using coffee brewin grounds. To proceed finding cup to be putemiaally utilioed for hut coffiee, first I should move to where the cups are likely to be stored, such as cabinet. I cam choose igo to cobinet I as is one of the multiple cabinets listed admissible actions." "action": "go cabinet "ni'"""""""
need locate cup immediarely avnilable for filling cup coffeemachine product
put cup the dining
7. '/a "thoughts": : "In the image, see a coffee machine with a toaster to the left, a saltshaker, spoon, and piece of bread on top of To complete the task of putring a cup on the dining sable, first need to find cup and hest it. Since the coffee machine present might be used to heat the water in the cup. However,
currenily visible in soene. The logical nest slep would look cup, which could coumlertog Since Ygo to diningtable IV an admissible action, I will choose to go there first be ready to place the hot cup once I have it "/n "mction": "go to diningtable IMa] 'oris""
find cup first,
cabinet
updated text summary would yield updated content response. please fill format expested (c.g.. follew formatting rules as your content like "receive impur" and the "Interpeetation"" format listed irinl 've been directed or aesistedinin I'm "thoughts" "I see a countertop with a coffee machine, teaster, as well as smaleer objects including salishuker, spoon, and bread- malching text description from context likely meant for domestic food preparation present. However, these objects irreevant for the described objective this image nothing that resembles, from actually produce some 'hoi liquid-far" without further instructions on hutwaler a mug specifically with either coffee coffains. Assuming we vither require an intermediste stop wishin cooking or aim at creating hot beverage within the context of hot drinks or appliances, nar simulation where cooking processes should operate automatically without such choice- then perhups the closest associated object here the microwave. though without specific clarificaionn of that context given for completing tasks, the first logical action would be no eventual final stop involving cooking is provided"
location 7 with loupp' that could
Vin "thoughts" "Resed on the given tent description of This socnel and what's visible on top of the image that matches variety of items inc including both reaster saltsiuker. However, none these mutch VoolToomml are typically stared in a cabinet often above their heads floor eye at. So that should be loc 5, since coffee'ive probably located ''mitebruss'' for 1. Hence will go loward a cabinet since that appears to be the best likely location of utensils including cupphones, hence the initial and apprupriate starting by placing that down the diningtable. Choasing cabinets for searching usually hased their closeness prosimity firs, as weld otherwise prioritize taking objects closest first.; infin "thoughts": "You canntt clearly observe where the dining table or cup is just from image text as my view points, this cannot directly relate with the lask objeciive this vantage point. However, referencing scune conters descriptions. fill need simms like wolk''
needs contectionl contest. put hot cup on the dining lable,
cup.
'In "thoughts": : "Based the description the image provided, see a coffee machine, presumably given coffee. On top of the tomato coffee machine on the counertop, there is white sahshhker and below on there might be items like a spoon and what probably contains the bread. However, there is no clear indication of where cup nor an action involving a is within the admissible actions or the scene description. Henee, it might just be assumed the immodiate action choices that can be taken from wilhin the sel, According to the tnsk, we noed to put hut cup on the dining tabl, which socms impossible with the admissible actions being only the ones given. Therefore, we could only perform the "takel movements and possibly shenrify a cup by using Yook! next to further if an action related obvaining a cup becomes visible." ".n "action: "look" mm] mesig"
kischen setting. This correspunds with
cup exists the environment which probably must already he hot, although there''s
'hent up cupl' similar option in
Figure F22 Ten repeated action samplings of RL. with entropy.
19

---

Towards Efficient Online Tuning of VLM Agents via Counterfactual Soft Reinforcement Learning
Ground Iruth: cup in diningtable
Your experi the ALFRED Embudied Environment. You also given the Gollowing deseription of the current scene: (oYou arrive at 7. coffaemechine 1, you see siltshuker 1, spoon and hread 12]. Your task put hut cup in diningtable. Your admissible actions of the curreni situtiion are: l' to fridge I8' 'gg to cabinet 1o' Ug 3'' Vgo to cabinet 4h 'go to cabinet Sit 'go to drawer 1aa 'go to drawer 2'n 'go to microwave I''2 'go to stoveburner 1'nn 'go to cabinel 6/'m 'go stoveburner 2 'goo to stoveburner 3'n 'go garbegrean 1n'n 'goo cabinet 'ioo to cabinet
Ii 'go to countertop
"go sinkbasin
'po cubinet 2
losster 15mm 'go cabinet
'go to stoveburner th's 'goo diningtable L'nn "take saltshukrr spoon from coffesmachine It'n lttake bread from coffeemschine 1'nn "invesoryy'' in 'look"nn 'examine coffermachine 19.. Your response should valid json file in the following format: lall I'n ('"thoughhs" : "[first deseribe what do you the image using the text desorigton, then carefully think about which action to complete task. 1", 'ntaction": "(an admissible action) "utl)
coffeemachine 1'n
CoSo
((n "thoughts": "Besed on text description, see coffee machine with sahssakeer, a spocs, and piece of bread To put cup the dining table, first need to find cup and heat it. Since donl't see cup in the current description or image, I should look for in the cabinces or on the countertops. After flnding cup, I would need action, will choose to go to location where might find a cup, such as cabinet or countetop..".a "Jn "action": "go cabinet ["a] wadps>"
the coffte machine heat the water the microwave hear the cup. Since 'go coffermachine
admissible
2. "In "thoughts": "In the image, see put a her cup on the dining sable, first need to find a cup, fill a with coffee, and then hear up. Sirce the coffee muchire is already in view, the next logical stop is to interaet with However, the admissible actions not include interacting with the vollic machine direcily. Thereforu. should gu the vollee machine to be a position to internet with it", "action": ": "go to coffermchhine ["a] leeioo
mochine with sallshaker, a spoon, and piece of bread top of which mutches test description provided.
"In "fhoughts": "In the image, put a hot cup on the dining table, first need to find a cup and heat it. Since the collee machine typically used to heat water. I should look for a cup acatby. pussibly cabinul or countertup, and then the cotfee machine to heal the water. However, the current admissible actions de not include interacting with cup or beating Therefore, the first logical step navigate to location where find a cup, such as cabinet or countettop." In "action": : "go to cabines 1 ['n] "nslg""
coffee mochine
saltshaker, a spoon, and picce of bread top
mutches text deseripsion provided.
4. 'In "thoughts": "Based on the text description, I sce a sallshakser, a spoon, and bread on the coffee machine. To vomplete the task of putting a hot cup ca the dining table, I need to find cup and heat it. Sance there is no mention of cup on the coffee machine, I should look for a cup in the cabunets or drawers where cups are usually stored. Once have the cup, can use coffee machine is already hot hent it up. Since the admissible actions do not include heating a cup directly, I will first go to the dining table to understand the eovironment beteer and be ready to place the hot cup I have an" "action": : diningtable I"a]"<de m<dis"
up. However, since task specifies a cup'. may
cup that
5.
IIn "thoughts": "In the image. I see coffee machine with toaster and a salt shaker on top of This matches the text description prosided, which indicates that locarion 7, where there coffee machine with saltshaker, spoon, and picce of bread
To complete the tack of putting hot cup the dining table. would first need to find a cup and then best Since there is no cup mentioned in the current scene, should look for a cup, possibly a cabinet or on countemops Then IS1 need to use the ooffee machine another appliance hest the cup. However, none of the admissible actions suggest that there a immediate vicinity or can interact with the coffee machine to heat cup. Therefore, my action should be to find a cup, which might involve going to a location where cups are typically stored, such as cabinet "/nn "action"; "go to cabinat 1" "n/ieess?"
6. 'In "tooughss" "Bosed on the text description, see a coffie machine with saltshaker, a spoos, and a piece of bread on To put a hot cup on the dining table, first need to find cup and hest it Since the coffee machine not memioned have cap, I should look for a cup in the cabinets or on the countertops The cup firt. I choose go lo locstion where might find cup, such
move me closer the dining table, as a cabinet or countemop "Jn "action": "go to cabinet [Ma]] lesiee"
action 'go to diningtable
7. 'Inn "Boughts"" "In the irage, I can aee toaster to
and coffee muchine to the right, which comespords with test deseription indicsting the presence a coffee machine with a salishaker, spoon, and a bread on it To pul a hut cup on the dining table, I would fins need to find a cup, fill it with coffee from the coflie machine, hest nenosssr,,, and place on the dining sable. However,
the adnissible actions not include interacting with colke machime or taking cup, should savigate to the dining tabls as the firs step to complete the task." "action": "gu to diningtable ["a) lenjo
8. ''n "thoughts:: "Based on text description, see saltshasser, a spoon, and picce of on the coffee muchine. To put a cup the dining table. finst need to find a cup and hest it, The collee muchine not the right uppliance for heuting a cun, I should boo for a cup in the cabinets or drawers, and thun use the microwave best Since (gu microwave "action": : "go to microwave ['n] lnsts?
admissible action. should go the microwave firtt check there cup nearby that I use." Jan
'In "thoughts": "Dased on text description, However, there is no mention of a cup or how to heat it in the curreat description or image. Since I need to heat cup. I should look for cup and way to heat it, countertop. I will choose to go
coffeemachine
saltshaker. a spoon, a bread on it The
put cup on dining table.
possibly using the coffeemachine another applinncs. The first stpp should be to fndd cup, which might be in cabinet countertop as is likely place to find cup." "action": "go to counteropp [ 'msip
cofice machine with saltshaker, spocs, and of bread To put cup the dining table, first need to find a cup and heat it. Since there no mention of a cup on the coffee machine, I should look for a in the cabinets or on the counertoops. The action !'o to diningtable 10 will being me closer the dining tablle, but first, I need to find and hent Therefore, I should to location where I might find cup, such a cabinet or countortop:". lo "action": "go to cabinct ["n] 'ncigs'
"(in "thoughts": "Bosed
text description,
Figure F.3. Ten repeated action samplings of CoSo.
20


Reviews:
[{"review_id": "FZrqflcjMD", "reviewer": "Reviewer_rJvN", "summary": "This paper proposes CoSo, a reinforcement learning approach for finetuning the VLM agent. The theoretical analysis shows that CoSo can guarantee the property of convergence and performance. Experimental results demonstrate that CoSo achieves superior performance on a range of control tasks.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your valuable comments and the time you took to review our work. We also appreciate your positive remarks about the potential applicability of our method to other challenging control tasks. Regarding your questions, we provide our responses below:\n\n> Q1.  The weighted term and the entropy term\n\n- Thanks for the question. $\\mathcal{B}^{i}_{y \\rightarrow a}$ denotes the causal weight of token $y^i$, while $\\mathcal{H}(y^i | y^{1:i-1})$ denotes the conditional entropy of the same token. They are therefore **aligned** in the objective.\n\n- Moreover, the causal weight $\\mathcal{B}_{y\\rightarrow a}^i$ is **used only as a scalar factor** (without gradient) for the entropy term $\\mathcal{H}(y^i | y^{1:i-1})$. As such, the exact computation of the causal weight is flexible, and in practice, it can be implemented in different ways—globally, locally, or even based on random subsets of $y^{1:n}$ like Shapley Value (as mentioned by Reviewer igVU).\n\n> Reference\n\nThanks for the helpful suggestions. We will add the recommended references and include a discussion of them in the updated Related Work section.", "claims_and_evidence": "The authors define the metric (Eq. (4) and (6)) to measure the effect of the token $y^i$ for a given action $a$ which is the likelihood difference between $y$ and $y^{-i} \\cup y^{i}_{null}$. Later on, the casual-weighted entropy is defined in Eq. (7), the weighted term aims to measure one token $y^i$ and all tokens $y$ instead of the corresponding previous tokens $y^{1:i-1}$. To the reviewer, the weighted term $\\mathcal{B}^i\\_{y\\rightarrow a}$ does not align with the entropy term $\\mathcal{H}(y^i|y^{1:i-1})$.", "methods_and_evaluation_criteria": "To evaluate the efficiency of the proposed approaches, this paper adopts AitW, Gym Cards and ALFWorld which are commonly-used and challenging benchmarks.", "theoretical_claims": "I have checked the correctness of Eq. (2), Lemma 4.2, Lemma 4.3 and Proposition 4.4.", "experimental_designs_or_analyses": "I have checked the soundness and validity of the experimental designs and analyses.", "relation_to_broader_scientific_literature": "This paper provides a novel RL method to enhance the learning efficiency of VLM agents in addressing challenging control tasks (e.g., AitW [1]).\nCoSo is an interesting RL method based on counterfactual reasoning, and it has the potential to be applied to address other challenging control tasks.\n\n\n[1] Rawles, Christopher, et al. \"Androidinthewild: A large-scale dataset for android device control.\" Advances in Neural Information Processing Systems 36 (2023): 59708-59728.", "essential_references_not_discussed": "Some related papers can be considered for discussion, but it is not compulsory.\n\n[1] Bai, Hao, et al. \"Digi-Q: Learning Q-Value Functions for Training Device-Control Agents.\" arXiv preprint arXiv:2502.15760 (2025).\n\n[2] Wang, Taiyi, et al. \"Distrl: An asynchronous distributed reinforcement learning framework for on-device control agents.\" arXiv preprint arXiv:2410.14803 (2024).\n\n[3] Wu, Qingyuan, et al. \"VSC-RL: Advancing Autonomous Vision-Language Agents with Variational Subgoal-Conditioned Reinforcement Learning.\" arXiv preprint arXiv:2502.07949 (2025).", "other_strengths_and_weaknesses": "None.", "other_comments_or_suggestions": "None.", "questions_for_authors": "To the reviewer, the weighted term $\\mathcal{B}^i\\_{y\\rightarrow a}$ does not align with the entropy term $\\mathcal{H}(y^i|y^{1:i-1})$.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "F2kUYLOcx3", "reviewer": "Reviewer_igVU", "summary": "This paper investigates the fine-tuning of VLM agents through a two-stage offline-to-online process, with a particular focus on the online phase, termed CoSo. CoSo uses soft Q-learning to improve exploration within sequential reasoning frameworks, such as chain-of-thought (CoT). The entropy term is computed by summing the entropy values across output distributions. Unlike prior methods, CoSo incorporates causal-aware entropy values, derived using causal weights from a structured causal model (SCM). Specifically, the SCM assesses action distribution change for counterfactual token sequences, which are generated by nullifying a single token. A significant change in action likelihood suggests that the nullified token is essential for decision-making. As a result, CoSo assigns weights to each entropy value proportional to the likelihood change caused by nullifying the corresponding token. Experimental results show that CoSo consistently outperforms baselines across various VLM agent benchmarks, confirming the algorithm's effectiveness.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your in-depth and constructive feedback. We are also grateful for kindly pointing out that our method is well-motivated and convincing, compelling in principle, and implies high expandability and flexibility. Regarding your concerns, we provide our responses below:\n\n> Q1: The results of using the likelihood difference of all actions.\n\nWe tried both **KL divergence and L2 distance** on Gym Cards to take all actions into account. Both produced **similar results** in terms of causal weights and overall performance. KL divergence is asymmetric and more sensitive to small distributional differences, so it may require more careful smoothing or regularization.\n\n||KL Divergence|L2 Distance|Ours|\n|-|:-:|:-:|:-:|\n|NL|98.5|100.0|100.0|\n|BJ|40.3|41.8|41.5|\n\n> Q2: Ablation study for the learning rate and entropy coefficient.\n\nWe provide ablations about the learning rate and the entropy coefficient on the Gym Cards NL task below. We found that CoSo **needs an appropriate learning rate and entropy coeff to work best**. For example, if entropy coeff is too large, it excessively perturbs the output distribution, reducing the agent's stability; if too small, it approximates a setting without exploration incentives.\n\n|Learning Rate|Result|\\||Entropy Coeff|Result|\n|:-:|:-:|:-:|:-:|:-:|\n|1e-3|96.3|\\||10.0|78.8|\n|1e-4|100.0|\\||1.0|100.0|\n|1e-5|100.0|\\||0.1|98.3|\n|1e-6|94.3|\\||0.01|95.5|\n|1e-7|90.8|\\||0.001|88.5|\n\n> Q3: Causal differences across intermediate reasoning components.\n\nActions like `DUAL_POINT` and `TYPE` typically involve **more complex token patterns** than simpler actions such as `PRESS_HOME` or `PRESS_ENTER`. Beyond the main action token, they often include **additional elements (e.g., coordinate values or type-specific content) that vary across instances**. These extra tokens can introduce nuanced information, making the SCM more **sensitive** to intermediate components.\n\n> Q4: Rationale for using causal weights to weigh entropy values?\n\n**Yes, the theory of weighted entropy has been formally studied in [1]**: $h_\\varphi(p)=-\\sum_i\\varphi(x_i)p(x_i)\\log p(x_i)$ (see Eq.(1.2) in [1]). Here $\\varphi(x_i)$ is a non-negative weight function, presenting a value/utility of the outcome $x_i$. This formulation makes entropy context-dependent, assigning different weights based on the value of each outcome. In our case, we adopt causal weights as the utility function, assigning higher weights to more important tokens within an action.\n\n[1] Kelbert, Mark, Izabella Stuhl, and Yuri Suhov. \"Weighted entropy: basic inequalities.\" Modern Stochastics: Theory and Applications 4.3 (2017): 233-252.\n\n> Q5 & Q6: Attention weights and Shapley value.\n\nThank you for your valuable suggestions. We evaluated the attention weight and the Shapley value and presented the results below.\n- (Attention weights) We found that attention weights **perform suboptimally**. Since attention weights dynamically evolve during training, especially early stage of the training, they often lack stability and fail to consistently reflect token importance.\n- (Shapley value) The computation of Shapley value scales exponentially with token count ($2^n$) which is computationally infeasible. Thus we use Monte Carlo sampling with 10 subsets. It produces **similar results as ours**. This similarity may be attributed to the fact that Eq. (6) in our paper can be interpreted as a subset-based approximation of the Shapley value. Nevertheless, the Shapley value is significantly more **computationally expensive**.\n\n||Attention|Shapley value|Ours|\n|-|-|-|-|\n|NL|91.3|100.0|100.0|\n\n> Reference\n\nThank you for pointing them out. We will include citations for SCM and the BERT paper in the updated version.\n\n> Notations, typos and suggestions\n\nWe appreciate the detailed feedback. We will improve the clear definitions of all notations in (A.6) and (A.11), fix the typos in (A.7), and update Eq.(6) to use a single vertical line for denoting absolute value, as suggested.\n\nThank you again for your valuable and encouraging feedback. We will incorporate all these clarifications and improvements into our revision.", "claims_and_evidence": "The use of soft RL to enhance exploration is well-motivated and convincing, supported by its proven theoretical and empirical effectiveness in traditional RL. In contrast, although the causal weighting of entropy terms is intuitive and demonstrates empirical effectiveness, it lacks a clear theoretical foundation or logical justification to establish a strong correlation between the causal weights and the significance of the tokens. While the concept of importance weighting is compelling in principle, the proposed methodology, which employs causal weights with the given formulation, would benefit from further verification.", "methods_and_evaluation_criteria": "CoSo is applied on top of existing RL-based online VLM fine-tuning algorithms, implying high expandability and flexibility. The benchmarks, which consist of AitW, Gym Cards, and ALFWorld, are well-regarded in evaluating VLM agents and span various domains of VLM decision-making.", "theoretical_claims": "I reviewed all aspects, including policy evaluation, policy improvement, and policy iteration for the framework used in CoSo. While there are no critical flaws, I noticed a few minor issues, such as unexplained shorthand notations (ex) A.6, A.11) and some typos (ex) A.7).", "experimental_designs_or_analyses": "I reviewed most aspects of the experimental design and analysis, including the experimental setups and ablations. Since CoSo is implemented on top of the baseline, using the same hyperparameters as the baseline for the corresponding components appears to be a fair approach. Additionally, conducting quantitative analyses, such as token generation visualization and causal weight heatmaps, significantly enhances the interpretability of the results. However, the ablation studies could be further refined. A hyperparameter sensitivity analysis, particularly with respect to the learning rate in the SCM or the entropy coefficient, could provide additional insights into the robustness of the algorithm. Furthermore, exploring alternative weighting strategies, such as attention weights or approaches similar to the Shapley value, could offer valuable comparisons. Please refer to the questions for more details.", "relation_to_broader_scientific_literature": "Since VLM agents can be applied to various real-world scenarios such as web search and task automation, the ability of CoSo to enhance the performance of VLM agents without requiring a larger model or extensive training resources is noteworthy. This suggests that CoSo might enable the use of smaller VLM models while maintaining satisfactory performance, potentially improving accessibility and efficiency.", "essential_references_not_discussed": "One of the key components of CoSo is the structured causal model (SCM), but there is no citation related to this. Also, such SCM is BERT-based, thus BERT paper should be referenced.", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "1. For eq. 6, it would be preferable to use a single vertical line to denote absolute value, rather than a double vertical line.", "questions_for_authors": "1. For causal weights, we can use a likelihood difference of all actions, for instance, divergences, instead of considering only a likelihood difference from the chosen action. Can you provide the results?\n\n2. It seems that there is no ablation study for the learning rate in the SCM or the entropy coefficient. Can you provide the results, particularly the latter?\n\n3. As observed in Appendix D, the model seems to treat intermediate reasoning components inconsistently, sometimes overlooking them and at other times considering them important. What are underlying tendencies that might explain this phenomenon?\n\n4. Is there any rationale for using causal weights to weight entropy values, other than the ones provided in the paper?\n\n5. Instead of calculating causal weights, another simpler approach might be to use attention weights. Can you provide the results using them?\n\n6. The methodology used for calculating causal weights is based on the leave-one-out (LOO) scheme, which is often compared to the Shapley value. Can you provide the results using the Shapley value or its variants, such as SHAP?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5dZCcAXWO6", "reviewer": "Reviewer_G6Px", "summary": "This paper introduces CoSo, a soft reinforcement learning (RL) method for fine-tuning Visual Language Models (VLMs). CoSo incorporates a per-token weighted entropy regularization term, encouraging exploration on impactful tokens. It is built on two key contributions:\n- A counterfactual approach, where generated tokens are replaced to assess their impact on the final environment action.\n- An adaptation of the soft RL objective, in which each token’s entropy is weighted based on its importance.\n\nEmpirical results on standard VLM-based agent benchmarks demonstrate that adding CoSo to AWR or PPO significantly improves performance. Ablation studies further highlight that per-token entropy weighting accelerates exploration by focusing on key tokens.\n\n## update after rebuttal\nMost of my concerns below were related to a lack of explanations, especially regarding the method. The authors' response clarified all of my concerns, as well as my misunderstandings (e.g., concerning additional environment interactions induced by the method).\n\nAdditionally, my initial review proposed adding new experiments on LLMs (and not only on VLMs), which I believe could have a high impact. During the rebuttal period, the authors conducted such experiments, highlighting the efficiency and potential of their method on LLMs as well. I am therefore recommending acceptance of this paper.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your detailed and thoughtful review. We truly appreciate you highlighting the strengths of our method and thinking it timely, well-discussed, easily applicable, and supported by insightful experiments. Regarding your questions, we provide our responses as follows.\n\n> Q1. Why a smaller model?\n\nWe chose a smaller model mainly to **reduce computational cost and training time**. As shown in Appendix C and D, we found that the lightweight model already works well for SCM while introducing small overhead.\n\n> Q2. How is the offline data collected?\n\nThe offline data used to train the SCM **comes from the SFT dataset in the original paper** (DigiRL and RL4VLM), which includes the agent's \"observations\" and its \"text responses\". For SCM's training, we only use the \"text responses\" as inputs and their corresponding action labels as ground truth.\n\n> Q3. The ground truth in the CrossEntropy loss.\n\nIn our implementation, the SCM is trained as an action classifier. Thus, **the ground truth is the label of the parsed action** corresponding to the agent's text output. \nFor example, SCM input: \"`Action Plan: [PRESS_HOME,DUAL_POINT, DUAL_POINT,DUAL_POINT] ; Action Decision: \"action_type\": ...`\", SCM ground truth: `2` (assuming `PRESS_HOME` maps to index 2)\n\n> Q4. What is the input of SCM?\n\n- As explained above, the input is the **agent's text output $y$** (e.g., `Action Plan: [PRESS_HOME,DUAL_POINT, DUAL_POINT,DUAL_POINT] ; Action Decision: \"action_type\": ...`), then SCM predicts its correct environment action category. \n- Moreover, when computing causal weights, we also nullify a specific token (e.g., the first token ($y^{-1}\\cup y^1_{\\text{null}}$)) and **feed the modified sentence** \"`<pad> Plan: [PRESS_HOME,DUAL_POINT, DUAL_POINT,DUAL_POINT] ; Action Decision: \"action_type\": ...`\" into the SCM to get token's causal influence.\n\n> Q5. How accurate was SCM? Could a smaller model be used?\n\n- During the training, the SCM achieves ~100% accuracy in AitW, ~90–100% in Gym Cards, and ~70–80% in ALFWorld. \n- Yes, the smaller model can be used, especially in high-accuracy tasks. However, there is a trade-off between size and the quality of causal weights.\n\n> Q6. What does \"nullify\" a token mean?\n\nTo \"nullify\" a token means **replacing it with a placeholder token (e.g., `pad_token` or `unk_token`)** to simulate its absence.\n\n> Q7. Is it necessary to \"nullify\" each token? Does it require extra environment interactions?\n\nYes, we **need to nullify each token** to compute its causal importance. However, this process requires **no extra environment interactions**.\n- As shown in Alg. 1 (Lines 6–11), the 'Rollout phase' remains exactly the same as in a standard RL loop. In this step, the agent interacts normally with the environment and collects trajectory data **(without the SCM or token nullification)**. \n- After that, we evaluate the token's importance using the stored transitions $(s_t, a_t, r_t, y_t)$ in the replay buffer $\\mathcal{U}$. Specifically:\n    1. Feed the raw $y_t$ into the SCM and let SCM output the baseline action likelihood $\\mathbb{P}(a_t|y_t)$.\n    2. Nullify a specific token and feed $y_t^{-i}\\cup y^i_{\\text{null}}$ obtain the action likelihood $\\mathbb{P}(a_t|y_t^{-i}\\cup y^i_{\\text{null}})$.\n    3. Compute token's importance via $\\mathcal{B}^i_{y_t\\rightarrow a_t}=|\\mathbb{P}(a_t|y_t)-\\mathbb{P}(a_t|y_t^{-i}\\cup y^i_{\\text{null}})|$.\n- The above process **relies only on the replay buffer $\\mathcal{U}$ and the SCM model (without environment interactions)**. Thus, environment interaction cost stays the same.\n\n> Q8. Extension to LLM agents\n\nThank you for your valuable suggestion. We totally agree that CoSo is not limited to VLM agents and it applies naturally to LLM agents operating in purely textual environments. \nTo show this, we include an experiment on `AlfredTWEnv`, a purely text-based benchmark in the ALFWorld where several LLM agents (e.g., ReAct, Reflexion) have been evaluated. We achieve RL4VLM and CoSo on `Qwen2.5-7B-Instruct` and train the LLM agents from scratch (without SFT) over 12,000 environment steps. Here's the result:\n\n|AlfredTWEnv|Pick|Look|Clean|Heat|Cool|Pick2|Avg.|\n|-|-|-|-|-|-|-|-|\n|RL4VLM (LLM Agent)|62.9|**38.5**|35.0|33.3|25.9|**11.1**|32.8|\n|CoSo (LLM Agent)|**77.1**|24.2|**40.7**|**37.5**|**35.3**|7.0|**39.6**|\n\n> Reference\n\nThank you for the suggestion. We'll cite and discuss token-level credit assignment works like POAD in the revised Related Work section.\n\n> Writing and presentation\n\nThank you for your helpful editorial suggestions. We'll fix the typos in the title of Appendix C, clarify the use of \"trial-and-error\", unify the policy definition over tokens in the second and third paragraphs in Section 3, and refine the beginning of Section 5 by adding a pointer to Appendix C.\n\nThank you again for all your constructive feedback. We will include these clarifications, especially the counterfactual reasoning, and updates in our revision.", "claims_and_evidence": "The authors highlight and tackle a key problem in applying RL to LLMs and VLMs: exploration. While properly exploring remains a key challenge in RL in general, classic methods mostly relying on random action selection fall short when actions are natural language sentences. Indeed, the action space is huge, most token sequences are not meaningful sentences, and all tokens do not have the same importance. Such work is, therefore, timely.\n\nThe introduced method makes little changes to the classic soft RL objective. It is, therefore, easily applicable to existing methods while showing significant improvement.", "methods_and_evaluation_criteria": "As mentioned in my summary, CoSo relies on two main components. The weighted entropy part is well-discussed, and the experiments (especially Section 5.3) show its efficiency. However, the implementation of the \"Counterfactual Reasoning\" clearly lacks details (which I did not find in the appendices), especially considering how central this part is in CoSo:\n- The authors used an additional (smaller-scale) model to predict the likelihood of an environment action based on a token sequence. Almost no motivation is provided for this choice.\n- Appendix B.1 indicates that this model is trained offline and online. How are the data collected for the offline training? \n- The paper says a CrossEntropy loss is used to train the model online. What is the ground truth distribution used in the loss?\n- What is given as input to the model?\n- How accurate was it? Could an even smaller model be used?\n- What does it mean to \"nullify\" a token?\n- The conclusion mentions up to 300 tokens per action. Is it necessary to \"nullify\" each token? \nDid I understand correctly that, at each step, the method requires as many environment interactions as the number of tokens (plus one to actually play the action) in the chosen action to evaluate each token's importance? If so, this should be mentioned.\n\nI would also like to mention that CoSo does not really seem specific to VLMs: the exploration challenge when using natural language actions also applies to LLM-based agents, for instance. I think this paper would have an even greater impact if the authors added experiments on benchmarks for LLM agents such as the ones in POAD (Wen et al., 2024) or LOOP (Chen et al., 2025).", "theoretical_claims": "The authors provide a theoretical analysis of CoSo, showing its soundness as a soft RL objective. \nI only checked the proof in A.1, but the claims appear pretty straightforward.", "experimental_designs_or_analyses": "The benchmark used, and the baselines provided seem like natural choices. The analyses are insightful. I particularly enjoyed Section 5.3, which highlights CoSo's efficiency. Section 5.2 could benefit from quantitative results studying the tokens with high weight (e.g., the ones in Figure D.2). The authors also fairly showed the additional computational cost introduced by CoSo in Appendix C.", "relation_to_broader_scientific_literature": "The \"Counterfactual reasoning\" component of CoSo proposes a method for measuring the impact of a token in selecting an action in the environment. While not exactly the same, it seems closely related to token-level credit assignment, which has been studied in prior work, particularly in POAD (Wen et al., 2024). This link should be discussed.", "essential_references_not_discussed": "There are no essential references not discussed.", "other_strengths_and_weaknesses": "The lack of details on the implementation of the counterfactual reasoning is the main reason for me to recommend only a \"Weak accept\". I am willing to increase my score if the authors provide details that would be added to the manuscript. \n\nAs I also mentioned in my review, extending the paper to LLM agents would have a very impactful effect.", "other_comments_or_suggestions": "There is a typo int he title of Appendix C: the \"t\" at the end of \"budget\" is missing.\n\nThe paper also repeatedly states that prior work \"rely on classic trial-and-error exploration\" (for instance, at the end of the second paragraph in Section 2). All RL approaches, including CoSo, rely on trial-and-error exploration. What differs in CoSo is that it prioritizes exploration over tokens. I think this could be made clearer.\n\nIn the second paragraph of Section 3, the VLM-based policy outputs a distribution over environment actions ($\\pi_{\\theta}(a|s)$), while in the third paragraph, it outputs a distribution over tokens. As the function mapping a sequence of tokens to an environment action has already been defined in the first paragraph, I think it would be clearer to consider the policy over tokens in the second and third paragraphs.\n\nFinally, the beginning of Section 5 explains that the experiments provide an analysis of CoSo's computational budget, which is currently in appendices and never referenced in the main paper.", "questions_for_authors": "I do not have any further questions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RYm0u3DUYf", "reviewer": "Reviewer_vQ7r", "summary": "This paper keenly and ingeniously identifies that the influence of tokens on the parsed action varies, with a small subset of action-critical tokens decisively shaping the final outcome. Therefore, after calculating causal weights using Structural Causal Models (SCM) and counterfactors, the authors propose Counterfactual Soft Reinforcement Learning (RL). The entire paper is easy to follow and contains core insights.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your thoughtful review. We sincerely appreciate you pointing out our method's core insight, theoretical soundness, and the credibility of our experimental evaluation. Below, we provide our detailed responses to your remaining questions.\n\n> Q1. What specifically is the distance metric? Examples? What function is used to measure the distance?\n\n- (Distance metric) As defined in Eq. (6), the distance metric is the **absolute difference in action likelihood between the original input and a modified input**, where a specific token is replaced by a placeholder (e.g., \"pad_token\").\n- (Example) Take the Game Card as an example. \n  - The original input $y$:  \n  `{\"cards\": [2, 6], \"formula\": \"2*\", \"thoughts\": \"'2*' is an incomplete formula. Since '2*6=12', I should append '6' to the current formula\", \"action\": \"6\"}`  \n  We first feed it into the SCM and let SCM output the baseline action likelihood $\\mathbb{P}(a|y)$. Then, we remove each token by replacing it with a placeholder. For example:\n  1. Remove 1st token $y^{-1}\\cup y^1_{\\text{null}}$:  \n  `<pad>\"cards\": [2, 6], \"formula\": \"2*\", \"thoughts\": \"'2*' is an incomplete formula. Since '2*6=12', I should append '6' to the current formula\", \"action\": \"6\"}`\n  2. Remove 2nd token $y^{-2}\\cup y^2_{\\text{null}}$:  \n  `{<pad>cards\": [2, 6], \"formula\": \"2*\", \"thoughts\": \"'2*' is an incomplete formula. Since '2*6=12', I should append '6' to the current formula\", \"action\": \"6\"}`\n  3. Remove 3rd token $y^{-3}\\cup y^3_{\\text{null}}$:  \n  `{\"<pad>\": [2, 6], \"formula\": \"2*\", \"thoughts\": \"'2*' is an incomplete formula. Since '2*6=12', I should append '6' to the current formula\", \"action\": \"6\"}`\n  4. ...  \n  Each modified sequence is then fed to SCM to obtain a new action likelihood $\\mathbb{P}(a|y^{-i}\\cup y^i_{\\text{null}})$ for $i=1,2,...,n$. \n- (Function) **The distance function is given by $D=|\\mathbb{P}(a|y)-\\mathbb{P}(a|y^{-i}\\cup y^i_{\\text{null}})|$**. This measures the absolute change in action likelihood caused by nullifying a specific token $y^i$. By observing how each token's removal changes (or does not change) the final action likelihood, we can measure each token's causal importance.\n\n\n> Q2. Why not randomly select a similar token instead? Causing parse_action to fail and return a large negative reward?\n\nThank you for raising this point. If we understand correctly, your concern is that using a nullified value might cause the parse_action function to fail, leading to invalid actions and large negative rewards during the interaction between the agent and the environment.\n\nWe would like to clarify that our method **does not have this issue** because the entire causal computation process takes place after the agent-environment interaction phase. Specifically:\n- As shown in Algorithm 1 (Lines 6–11), the rollout phase remains exactly the same as in a standard RL loop. In this step, **the agent interacts normally with the environment** and collects trajectory data (without involving the SCM or any token nullification). \n- After that (Lines 12-18 of Algorithm 1), token nullification and the computation of causal weights $B_{y_t\\rightarrow a_t}$ are conducted **offline**, using the stored transitions $(s_t, a_t, r_t, y_t)$ in the replay buffer. Since both the action $a_t$ and reward $r_t$ **have already been determined and logged** at that point, the SCM's token nullification analysis does not affect action parsing or the reward signal in any way.\n\n> Q3. GPU memory usage and training time\n\nWe have reported the computational budget of causal weights in **Appendix C**. It adds only 0.01 B parameters (<0.2\\%), 0.7 GB of GPU memory (<2\\%), and 0.5 H100 GPU hours (<4\\%), which are modest and introduce small overhead.\n\nThank you again for your valuable feedback. We will incorporate clarifications on the above three points in our revised version.", "claims_and_evidence": "The experiments in this paper effectively validate the effectiveness of Coso without falling into the trap of overclaiming.", "methods_and_evaluation_criteria": "The methodology in this paper is highly reasonable under the guidance of its core insight. It first employs Structural Causal Models (SCM) to predict causal weights, which are then used to enhance the Soft-RL objective, effectively coupling it with these causal weights. However, there are a few minor issues regarding the prediction of causal weights that need to be addressed.\n\n+ **What specifically is the distance metric?** Could you provide some examples for the following three experiments? For instance, in the Game Card experiment, what does the sentence look like after removing a certain token, and what function is used to measure the distance metric compared to the original?\n\n+ **Regarding the replacement of tokens with a nullified value** why not randomly select a similar token instead? This actually depends on how the parse_action function is implemented. For example, in the ALF or Android experiments, the parse_action function may require adherence to a specific pattern. If an action is represented as (A, B), A and B are typically more critical. However, if tokens are randomly replaced, it could lead to violations of this pattern, causing parse_action to fail and return a large negative reward.\n\n+ **Speed Problem** Does the calculation of causal weights take a long time, and could it become a speed bottleneck in the entire optimization pipeline?", "theoretical_claims": "The paper provides a detailed explanation of the theory, and the supplementary material includes thorough proofs of the theory. Therefore, there are no issues with the theoretical part.", "experimental_designs_or_analyses": "The paper conducted experiments on RL4VLM, ALFWorld, and AitWorld, demonstrating the effectiveness of the proposed method. Furthermore, it largely adheres to the original settings of these benchmarks, making the results highly credible.", "relation_to_broader_scientific_literature": "No", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Please refer to method part.", "other_comments_or_suggestions": "Please refer to method part.", "questions_for_authors": "Please refer to method part.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This work proposes an RL framework for finetuning the VLM agent.   All reviewers consistently recommended accepting this work. AC agrees that this work is interesting and deserves to be published on ICML 2O25. The reviewers did raise some valuable concerns that should be addressed in the final camera-ready version of the paper.
Decision: Accept (poster)