Digi-Q: Learning VLM Q-Value Functions for Training Device-Control Agents
Authors: Hao Bai
Yifei Zhou
Li Erran Li
Sergey Levine
Aviral Kumar
Venue: ICLR 2025 Poster (2025)
Keywords: Reinforcement learning
device control
digital agents
foundation models
Primary area: foundation or frontier models, including LLMs

Abstract: While a number of existing approaches for building foundation model agents rely on prompting or fine-tuning with human demonstrations, it is not sufficient in dynamic environments (e.g., mobile device control). On-policy reinforcement learning (RL) should address these limitations, but collecting actual rollouts in an environment is often undesirable in truly open-ended agentic problems such as mobile device control or interacting with humans, where each unit of interaction is associated with a cost. In such scenarios, a method for policy learning that can utilize off-policy experience by learning a trained action-value function is much more effective. In this paper, we develop an approach, called Digi-Q, to train VLM-based action-value Q-functions which are then used to extract the agent policy. We study our approach in the mobile device control setting. Digi-Q trains the Q-function using offline temporal-difference (TD) learning, on top of frozen, intermediate-layer features of a VLM. Compared to fine-tuning the whole VLM, this approach saves us compute and enhances scalability. To make the VLM features amenable for representing the Q-function, we need to employ an initial phase of fine-tuning to amplify coverage over actionable information needed for value function. Once trained, we use this Q-function via a Best-of-N policy extraction operator that imitates the best action out of multiple candidate actions from the current policy as ranked by the value function, enabling policy improvement without environment interaction. Digi-Q outperforms several prior methods on user-scale device control tasks in Android-in-the-Wild, attaining 21.2% improvement over prior best-performing method. In some cases, our Digi-Q ap-
proach already matches state-of-the-art RL methods that require interaction. The project is open-sourced at https://github.com/DigiRL-agent/digiq

Published as a conference paper at ICLR 2025
DIGI-Q: LEARNING VLM Q- Q-VALUE FUNCTIONS FOR TRAINING DEVICE-CONTROL AGENTS
Hao Bai UTUC
Yifei Zhou UC Berkeley
Erran Li Amazon
Sergey Levine UC Berkeley
Aviral Kumar CMU
ABSTRACT
While number of existing approaches for building foundation model agents rely on prompting or fine-tuning with human demonstrations, it is not sufficient in dynamic environments (e.g., mobile device control). On-policy reinforcement learning (RL) should address these limitations, but collecting actual rollouts in an environment often undesirable in truly open-ended agentic problems such as mobile device control or interacting with humans, where each unit of interaction is associated with cost. such scenarios, a method for policy learning that utilize off-policy experience by learning trained action-value function is much more effective. In this paper, we develop an approach, called Digi-Q. to train VL.M-based action-value Q-functions which are then used to extract the agent policy. We study our approach in the mobile device control setting. Digi- Q trains the Q-function using offline temporal-difference (TD) learning, on top of frozen, intermediate-layer features of VLM. Compared to fine-tuning the whole VLM, this approach saves compute and enhances scalability. To make the VLM features amenable for representing the Q-function, we need to employ an initial phase of fine-tuning to amplify coverage over actionable information needed for value function. Once trained, we use this Q-function via a Best-of-N policy extraction operator that imitates the best action out of multiple candidate actions from the current policy as ranked by the value function, enabling policy improvement without environment interaction. Digi-Q outperforms several prior methods on user-scale device control tasks in Android-in-the- -hill-dddd attaining 21.2% improvement over prior best rest-performing method. In some cases, our Digi-Q approach already matches wate-of-the-an art RL methods that require interaction. The project is open-sourced at htsps: //gi tuut.uu.... com commmDigiiRee agent/ /
INTRODUCTION
Foundation models (OpenAI, 2024a; GeminiTeam, 2024) open up the possibilities to build agents that make intelligent decisions in the real world (Liu et al., 2023). While prompting off-the-shelf language models with specific instructions is one way get them to make decisions, this is not good enough for attaining goals and maximizing rewards that are critical in downstream tasks (Zeng et al., 2023; Chen al., 2023). Part of the reason is the lack of sufficiently diverse decision-making data (Gur et al., 2023). But perhaps more fundamental challenge is that simply imitating Internet data is not good enough for training models "how" act intelligently, reduce uncertainty, and achieve goals in non-stationary real-world decision making settings (Bai et al., 2024; Ma et al., 2024).
Recently, the community has been turning towards using reinforcement learning (RL) for training agentic policies. RL avoids the shortcomings of imitation and prompting, by explicitly training the policy to solve tasks (Zhou al., 2024h; Verma et al., 2022; Snell et al., 2023; Abdulhai et al., 2023). That said, the best performing RL methods today for improving policy in multi-step agentic tasks rely critically on interaction due to the use policy gradient updates (Yao et al., 2023) coupled with Monte-Carlo values (Bai et al., 2024; Putta et al., 2024; Shao et al., 2024), which often require sufficient amounts of on-policy data to get low-variance learning signal. The amount of on-policy data needed is likely only larger in non-stationary and dynamic environments (Bai et al., 2024).
"qqual Contribution, correspondance haob2 @illinoissi.. yifei zhou @ herkeley.cdu

---

Published as a conference paper at ICLR 2025
Policy-based based Methods
the sunple discarded
train
value function
filter
Irain
π
(a,a,r,)) rollout buffer
(s, pairs to learn
Digi-Q
π
TD learning
VLM Q Function
the sample with new action) learned!
train
resample
judge
pick
actions
actions
Best-of-N
methods. (s, a) rollout pairs that are learned are marked green in the buffer. Typically, policy-based methods utilizes a state value function to filter out promising state-action pairs, and requires online data to improve. In contrast, Digi-Q learns a state-action (Q) value function through TD-Icarning on offline data, and re-samplc an amount of actions for each state. This Q-function is then used rank the re-sampled action learn policy using the best action under each state. Digi-Q results higher sample efficiency than policy-based methods to be applied even in fully offline setting.
Figure I: Comparing Digi-Q with on-policy
If on the other hand, we could train critic (i.e., action-value fanction) that could score policy's action reliably, without needing to actually simulate the policy behavior over multiple steps several times, we could simplify our recipe for policy improvement without costly simulations. With this motivation, in this paper, we build a simple approach to train a VLM Q-function. In our problem setting of mobile device control, this corresponds to training VLM-based Q-function that can provide score for every snapshot of the phone screen and an action, represented by laying the cursor over this snapshot. Our method, Digi-Q. trains Q-function for device control entirely using historical data collected by (potentially suboptimal) agents.
To train Q-functions effectively, Digi-Q Q handles a number of challenges posed by value learning at scale: (i) instability in running temporal -iiforalddifference (TD) learning for training value functions with large models (Kumar et al., 2021) and (ii) inefficiency of TD backups per unit amount of "compute" (i.e., gradient steps spent) (Chebotar et al., 2023) (see Appendix B. Digi-Q does so by training on top of a frozen intermediate layer representation of the VLM instead of training all parameters of the VLM. For attaining the best performance, representations from off -he--hh--h VLMs not good enough, since they often do not contain feature information crucial for predicting actions or their consequences. Therefore, Digi-Q prescribes running an initial phase of representation fine-tuning prime representations of a VLM to be more amenable to TD-learning.
To specifically understand benefits of learning the Q-function, we note that while state-only Monte- Carlo value functions (Bai et al., 2024; Zhai et al., 2024) can only evaluate the efficacy of a single action that was actually executed any given state, a good Q-function provides with reliable estimates of the future expected reward for multiple actions at the same state, without needing to actually roll these candidates out. This allows to develop a Best-of-N policy-extraction objective that trains the agentic policy to imitate the best-rated action per the Q-function without any additional interaction. The difference between policy-based methods and Digi-Q illustrated in Figure
The main contribution of this work is Digi-Q. an offline approach to train VLMe-bssed Q-function for building device-control agents. Digi-Q represents and trains Q-functions on top of intermediate representations from VLM, fine-tuned to especially consist of actionable information. Digi-Q unlocks the use of ofst-of-N policy extraction objective to make the most effective use of a Q- function in obtaining policy. The agent produced by running Digi Q offline data outperforms prior approaches that also extract policies from offline data in the problem setting of Android device control (Rawles al., 2023) with 21.2% of relative improvement over the best-performing prior
2

---

Published as a conference paper at ICLR 2025
method, even though these domains remain challenging for of proprietary models (Liu et al., 2024c; Bai et al., 2024). To the best of our knowledge, this work is the first to successfully scale state-action Q-value functions realistic agent tasks with VLMs.
RELATED WORK
RL for training GUI and device-control agents. Due to their reasoning and perception capabilities, LLMs and VLMs have been applied extensively to build agents to navigate web pages (Zhou et al., 2024a; Koh et al., 2024a; Deng et al., 2023; Zheng et al., 2024; He et al., 2024) and GUI interfaces (Bai et al., 2024; Yan al.. 2023; Hong et al., 2023: Rawles al., 2023; 2024; Zhang and Zhang, 2024). In contrast to using off off-the-shelf proprietary models (Zheng et al., 2024; Yan et al., 2023; Zhang et al., 2023; He et al., 2024) or fine-tuning them with a small amount of human demonstrations (Hong et al., 2023; Zhang and Zhang, 2024; Zeng et al., 2023), RL provides the advantage of optimizing task-specific reward and goal-oriented behavior, which is important in dynamic and non-stationary environments especially when human demonstrations are stale (Bai et al., 2024; Zhou et al., 2024b; Putta et al., 2024; Pan et al., 2024: Song et al., 2024). However, most successful applications of RL for real-world GUT agent tasks use less efficient algorithms such as (nearly) on-policy policy gradient or filtered imitation learning (Bai et al., 2024; Putta et al., 2024; Song et al., 2024; Koh et al., 2024b; Shao et al., 2024). This can be problematic for real-world GUT agent tasks where interaction with the actual environment presents a bottleneck and on-policy data is hard to collect due to practical issues such as privacy. In traditional RL, the approach avoid variance and learn without on-policy interaction (or massive simulation) is to actually train a Q- value function that can score given action at given state (snapshot of the phone screen). Using this Q-value function for training the policy results in substantially better performance (Mnih et al., 2013; Haarnoja et al., 2018 8; Fujimoto et al., 2018) and can be done entirely from historical data (Kumar et al., 2020; Fu et al., 2021). To the best of our knowledge, our work is the first to scale value-based Bellman backups to convert VLMs into device-control Q-functions, which serve as effective scoring functions to policy extraction.
From an algorithmic standpoint, the closest work to ours that trains agents with Q-functions is ArCHer (Zhou et al., 2024b), which builds a hierarchical framework for developing RL algorithms for training agents. Note that this prior work presents results simplified environments (Yao et al., 2023). While our use of a VL.M-based value function to train the policy can be interpreted as yet another algorithm under the hierarchical actor-critic abstraction in ArCHer, note that the methodology for running RL at scale is substantially different from this prior work. Specifically, we prescribe several important components along the use of frozen VLM representations and a policy extraction approach based on best-of-N policy extraction, which also enables scaling rest-time compute. Our experiments in Section 5 show that Digi-Q is much more effective (about a 20% improvement as shown in Section 5.2) than the policy-gradient algorithm used by Zhou et al. (2024b). This justifies the benefits of our seemingly simple, yet an effective design of using the value function. Other works (Zhai et al., 2024) train VLMs with on-policy PPO (Schulman et al., 2017; Chen et al., 2024). Finally, Chen et al. (2024) runs RL on top of frozen VLM representations as well, although unlike us they do not fine-tune the VLM to make these representations more amenable for fitting value functions. Instead, they use handcrafted prompts to prime the VLM into producing useful features. Our results highlight that the representation fine-tuning phase in Digi-Q is critical to obtaining good Q-function, but prompting alone is not as effective.
Challenges of training an off-policy Q function with foundation models. Despite the efficiency and data reuse benefits of training a Q function via off-policy TD-learning it can be unstable and computationally inefficient if not treated carefully, partieularly the case for large foundation models with billions of parameters. This instability stems from two aspects: (1) prior work has often found it hard and unstable to train value functions via Bellman backups and TD-learning (Kumar et al., 2021; 2022; Chebotar et al., 2023), which is challenging at scale. To address this, Chebotar et al. (2023) had to employ combination of conservative regularization (Kumar et al., 2020) and regularization with n-step returns (Hessel et al., 2018) resulting in a complex approach; (2) policy extraction from trained Q-functions often utilizes policy gradient approaches with a 'negative gradient" term (Tajwar et al., 2024) that can be unstable with offline data. This has largely resulted in the community focusing on on- on-policy or filtered imitation learning methods. However, Park et al. (2024) show that supervised regression methods such AWR (Peng al., 2019) can lead to slow convergence and poor asymptotic performance. To address challenge (1), Digi-Q runs TD-Icaniig on top of frozen VLM representations, but after a fine-tuning phase to make them more amenable to representing Q-
3

---

Published as a conference paper at ICLR 2025
functions and to address (2), we introduce Best-of-N based policy extraction loss, akin to concurrent work (Mark et al., 2024) in the domain of robotic learning.
3 PRELIMINARIES AND PROBLEM SETUP
We aim to build value functions for training agents in the domain of device control, where we wish to accomplish pixel-based interactions on virtual devices, following similar protocol as past work (Bai et al., 2024). In this section, will discuss the setup for this problem, followed by terminology, notation, and background that would be useful in developing our approach in the next section.
3.1 PROBLEM SETUP: ANDROID DEVICE CONTROL
We scope our study in the domain of pixel-based Android device control (Bai et al., 2024; Zhang et al., 2023; Rawles et al., 2023). Each episode in this domain starts with a functionnn Android emulator reset to the home screen, and a task is randomly drawn from task pool represented by natural language instructions. The agent needs to complete the task through pixel-based interactions with the device as illustrated in Figure 1. The actions that the agent can take are primitive pixel-level commands such as clicking at a coordinates and typing text. Concretely, given a screenshot of a phone, we want the agent to output string command such "click (0.8, 0.2)" to be executed in the environment at the current step, where 0.8 and 0.2 are 0-1 normalized x-y coordinates in the screen. This domain is known to be more general and challenging than web navigation alone link-based device control (Bai et al., 2024), and present many real-world challenges of device stochasticity and dynamism, such as unpredictable distractors like pop-ups and technical glitches like incomplete website loading. Following Pan et al. (2024); Bai et al. (2024), the agents are evaluated via binary 0/1 rewards from proprietary model (i.e.. Gemini 1.5 Pro (GeminiTeam, 2024)) that makes a verdict of whether the specific task has been completed at each step. We want to use this signal to learn a value function that can accurately predict the future outcome that the agent would attain if it were to execute given action at given snapshot, without actually executing this rollout. More importantly, we wish to learn this value function using static dataset 'D storing historical past interaction data and use it to learn agentic policy.
3.2 REINFOCCEMENT LEARNING DEFINITIONS
There two types of value functions we can model in our setting: (1) "turn-level" value function that can score each natural language interaction with the external environment (e.g.. "type box [2]: wikipedia of chocolate"), and (2) value function at the "token-level", where each step is an independent natural language token. Terminology wise, we define the state in the Markov decision process (MDP) in our setting to be the sequence of tokens denoting the log of the interaction history of the agent with the environment thus far concatenated to the current observation. Each action a, is a sequence of tokens directly applied to interact with the environment at the entire turn level.
The turn-level Q-function for given policy π is the expected cumulative reward of par- ticular action at the current step, and then following the policy т thereafter: (T)s,, an) = the ex-
E, IE 7'r($s+e, an+t)]. The value function of a policy т, VT(ss). is defined pected Q-value, (= a.). where actions On are sampled from the policy x. The advantage function A (s),,, ax) corresponds to the relative benefit of taking action as in state Sh, and is computed as the difference between the Q-value and the value of the state under the policy: A" Ar ) Q (sh, on) - (=ss)). The goal of RL is train policy that can produce token sequences that maximize discounted cumulative rewards over the course of a rollout.
For training the agentic policy, our approach seeks to train action-value Q-function Q parameterized by parameters 6, and a policy parameterized by ø. Additionally, we maintain a state-only value- function V parameterized by v to stabilize training. Both Q- and V- functions are instantiated by a small MLP layer on top of a VLM backbone, We use 0YLM, 1/YLM to represent the parameters of the VLM backbone and similarly fMLP, V'LLP for parameters of the MLP head. We will denote the last layer representations of these VLM backbones as fevix (s, a) and (
3.3 BACKGROUND: ARCHER FRAMEWORK FOR TRAINING AGENTIC POLICIES WITH RL
The ArCHer framework (Zhou al., 2024b) provides conceptual way pose training of foundation model value functions and agentic policies as a hierarchical RL problem. Although their framework is not specific to one particular RI. algorithm, Zhou et al. (2024h) show that a convenient
4

---

Published as a conference paper at ICLR 2025
way to instantiate this approach is to learn value function at the turn-level and policy to produce tokens in an autoregressive manner. The value function critic and the agentic actor are then optimized against each other similarly to standard actor-critic RL. The loss functions for training the cirtic are given by:
(Q) = Es,ann [Qo(s, - -
(vb) = EEDD (La)))())) [V Qz(s, a))
(2)
0 and é are the delayed target networks (Mnih et al., 2013) for stability and they are updated as an exponential moving average of 0 and ø. The instantiated algorithm from Zhou et al. (2024b) supports policy extractions through REINFORCE, policy gradient:
JT( = E.
E A(se: at) log TO se, aj 1
While our approach will utilize a similar framework to conceptualize the training of the value function critic and the agentic policy in our method, the design of actor and critic updates employed are substantially different in our method. As such, the actor update from Zhou et al. (2024b), which also corresponds to the standard policy gradient update, can also be unstable in certain problems.
DIGI-Q: TRAINING VLM Q- Q-VALUE FUNCTIONS FOR AGENTIC POLICY
To obtain an effective agentic policy for device control problems, Digi-Q trains a Q-value function on static data, which is then used to extract a policy. In the process of designing Digi-Q, we need to address challenges with running value- based RL scale. First, to avoid pathological behavior of TD-backups with large models (Zhou et al., 2024b; Snell et al., 2023; Abdulhai et al., 2023; Chebotar et al., 2023) and to avoid the computational costs associated with training billion-parameter VLM end-to-end with TD-learning, we train Q-functions on top of frozen VLM representations. Since VLMs are not trained on substantial quantities of sacision-making data, off-the-shelf VLMs largely do not accurately represent actionable elements of an input scene. To address this, Digi-Q fine-tunes VLM representations before running Q-function training. This fine-tuning procedure is not the same supervised fine-tuning on in-domain data, but rather is designed to emphasize actionable features that change from one snapshot to the other and hence help model the value function.
Unlike typical on-policy RL filtered imitation learning (e.g., AWR (Peng et al., 2019)) that only updates the policy with one action per state, training a Q-function allows us to simultaneously estimate returns from multiple action candidates, all of which can then be used for improving the policy. Using multiple action candidates can be more efficient, especially if the critic predictions are more liable, and even if not, it offers variance reduction benefits. Digi-Q utilizes this insight to develop a Best-of-N reranking based policy extraction objective for training the policy. This policy improvement operator is stable and more effective than policy gradient or advantage- weighted regression in our expriments. The method is illustrated in Figure 2, with each part described below.
advantage-weighted
4.1 TRAINING VLM Q-FUNCTIONS VIA TD-LEARNING AND REPRESENTATION
As discussed above, fine-tuning large VLMs end-to-end to represent value functions can present pathologies and perhaps a more practical approach is to train a separate value function on top of the frozen representation from the VLM. However, most VLM backbones are largely not trained to model actionable information for given scene or make predictions about possible scenes that could result from taking actions in an environment. If the internal representations of the VLM do not pay attention actionable information in a scene correctly, then training a state-action Q-function Q(s, a) will either degenerate into learning a state-only value function V (s) (i.e., it will ignore the action input) or diverge by incorrectly amplifying noise in partially-trained Q-value estimates at unseen, out-of-distributio actions that appear in the right hand side of TD-backup during training. Indeed, in our preliminary runs, we found that while open-source VLMs such as LLaVa-155 (Liu et al., 2024a) able to answer questions about scene, the same VLM does often fail at correctly answering questions about impact of current actions into the future, e.g., it answers "does this clicking operation lead to page in ebay.com?" incorrectly.
To address this issue with VLM representations, Digi-Q Q first fine-tunes representations of VLM with binary classification objective to enable pay attention to actionable features of input
5

---

Published as a conference paper at ICLR 2025
1. Representation Fine-tuning
2. Value-based Reinforcement Learning
8
Proonpt lcad different
a
(s.0)
0.15
dle, >
resample actions
actions
VLM representation
Binary Target
Enungt (i.i this actton tn ciffeeenttttii statel
VLM
Target
π
a1
Best-of-N Target
Figure 2: Overview of Digi-Q. Blue represent forward data flows, while red arrows represent how get learning targets used for back propagation. Our method first goes through representation fine-tuning stage to extract actionable features from the VLM. TD-learning is then performed on top of frozen VLM representations to learn reliable Q value function, followed by Best-of-N policy extraction approach.
scene. Once fine-tuned, the representations of this VLM are used to train a Q-function represented using small MLP on top of the frozen representation. Not only is this approach more stable and robust against pathologies that could emerge from fine-tuning, but it cuts down computational costs since only 1% of all the VLM parameters (the head) now trained via TD-learning.
Approach. Our representation fine-tuning objective constructed as follows: given a transition pair (St, ae, St+1) drawn from replay buffer, the representation fine-tuning objective attempts to model if and how the next state S141 will change from the current state and action (sc: a²). Our observation is that in device control problems, useful action should lead to a substantial visual change in the pixel values of a scene (e.g., successfully typing a search query and pressing enter on "google.com should cause the page change substantially to now show list of search results, whereas an unsuccessful search attempt, say due to imperfect clicks, with a very high probability will change none to few pixels the original scene and remain stuck on the "google.com" page ). Equipped with this insight, we construct positive and negative tuples of transitions (St, Ge, 8t+1), where the positive tuples consists transitions change the state significantly (i.e., larger than threshold on the & image distance) and the negative tuples are the remaining transitions. This is equivalent to assigning binary {0, 1} label to a transition:
O, d(st, <E otherwise
yt :={
Now, the VLM trained to produce this 0-1 label ye given state-action tuple (st, using a binary cross-entropy loss on its parameters OVLM:
(P(VL))) -.,,aa,alll (y, log Povis ('yo'' a.) + (1 yya log Povir (nno' at)l,
(3)
where Pavess is the next-token distribution obtained from the VLM backbone.
After this phase of representation fine-tuning, we freeze the parameters of VL.M, and extract the embedding of the yes/no token output to serve the input representation of (st, az) to the Q function. We now run a TD-learning objective from Equation in Section to train the Q-function.
Note that Equation also utilizes parameterized value function. Since the value function does not depend on the action, we are able to directly use internal representations without requiring any phase of initial fine-tume aae-ttning. On top of the frozen representations from the VLMs, our value functions 0MLP, L'LLP is optimized with the TD loss as in Equations and 5.
an off-the-shelf VLM,
JQ(OMLP) Ea,,..s'' laolllQQeaae (fevIm (s) - 7 VinaLe (frVise (s'))) Jy (1bML)) RAND
E (a' (ooe)))((VVV (Vonne (J (8)) Qéxar (J (8, a ()))
(5)
4.2 BEST-OF-N POLICY EXTRACTION AGAINST THE LEARNED Q- Q-FUNCTION
Given a learned funntioon, we will now use it to extract policy in an efficient and reliable manner. Perhaps the most straightforward approach for doing is to use the REINFORCE policy gradient

---

Published as a conference paper at ICLR 2025
estimator train the learned policy, however, this approach can be brittle with off off-policy data. The presence of a "negative gradient" term Tajwar et al. (2024) (i.e., a term where the policy gradient multiplies the log likelihood by negative-valued advantage) means that careful tuning of learning rates and interleaving policy and critic updates must be done to attain good performance (see Zhou al. (2024b) Section 5.7 for discussion of these challenges). While advantage-wae -winghhe supervised learning (i.e., AWR (Peng et al., 2019)) avoids this instability issue, it can be quite conservative in terms of moving away from the data collection policy.
To build a stable yet non-conservative policy training method, Digi-Q modifies weighted regression to make it more "aggressive", by leveraging the insight that access model of the Q-function allows for estimating values for multiple N actions at any given state. After computing Q-values for multiple action candidates, we can imitate the best action. This would produce updates that are substantially less conservative than single-action AWR, without needing negative gradient, and only employing a relatively more stable supervised learning update. Theoretically, this is because the implicit KL constraint against the data-generating policy that makes AWR conservative, is now much less a problem with our multiple-action approach, since this implicit constraint is enforced against the Best-of-N policy (Cobbe al., 2021), which already is much less conservative than the data collection policy for larger values of N. Moreover, akin to how verifiers have been used in reasoning problems, the value function critic can be used score multiple possible actions in offline manner without actually running the rollout or seeking a downstream correctness signal. This approach is similar concurrent work Mark al. (2024) from the domain of robotic policy learning.
Concretely, given any state 8, we sample N action token sequences from the initial policy: a1.-.. AN (=]-ss), where Ty is the policy used to rollout the offline dataset. Now, we rank these actions according to the Q-values obtained from the value function trained previously. The policy is then trained to imitate the highest Q-value action of these N actions as long as this best action also attains a positive advantage. Intuitivelyy this serves as proxy learning objective to maximize the advantage function per state without the risk of "negative gradient". Formally, this means that the policy is optimized as per the loss described in Equation 6.
N
I
(Ø) = E Ba.
Σ 1=1
((as)
Σ A=1
log TT(aa (e, oz 1
(6)
where 8(a.) = 1 {a; = arg max; (ss,, as) and Q(+t, a.)
V.,,) > 0}. This approach allows us to
make fairly non-conservative policy updates, while also being stable and efficient due to log loss.
4.3 PUTTING IT TOGETHER: IMPLEMENTATION DETAILS
A pseudocode of our overall algorithm is shown in Appendix A. After initially fine-tuning the VLM through the representation fine-tuning scheme in Section 4. Digi-Q trains and V-functions before performing gradient updates on the actor, where the VLM backbone for the V-function is kept frozen from the pre-trained checkpoint. The usage of V-functions follows from Zhou et al. (2024b); Snell et al. (2023) to improve training stability. The actor is represented on top of a separate VLM and is trained end-to-end, unlike the use of frozen features for the critic to improve training stability of TD-learning For our experiments, we sample N = 16 actions for computing the Best-of-N reranking policy learning objective in Equation 6: while the choice of N can differ from domain to domain, runs show that N 16 is a good choice for our domain of Android device control. We use LLa 1.5 (Liu et al., 2024a) for the backbone VLM for our Q- and V- functions. The architecture details can be found in Appendix B.2.
5 EXPERIMENTAL EVALUATION
The goal of our experiments to evaluate the efficacy of Digi-Q in producing effective Q-functions that in turn are able to train strong Android device control agents. Our experiments will answer the following questions: (1) How does Digi-Q compare with other state-of-the-art agent training algorithms, previously studied in the context of Android device control tasks? and (2) Can Digi-Q Q learn effectively from past interaction data? In addition, we perform several ablation experiments to understand the effects of various components of Digi-Q: to understand the benefits of using representation fine-tuning and to validate the efficacy of the Best-of-N reranking approach for training the policy using the value function.
7

---

Published as a conference paper at ICLR 2025
AitW General
AitW Web Shopping
Train
Test
Train
Test
GPT-4V
5.2
13.5
3.1
8.3
SET-OF OF-RAKS
Gemini 1.5 Pro
32.3
16.7
6.3
11.5
Prompting
GPT-4V
13.5
17.7
12.5
8.3
APPÅGENT
Gemini 1.5 Pro
14.6
16.7
5.2
8.3
SUPERVISED
CogAgent
25.0
25.0
31.3
38.5
TRAINING
AutoUI
27.7
22.9
20.7
25.0
Filtered BC
5.9
54.5 =4.5113
37.2 4.7
43.8 + 1.7
Learning
OFFLINE
DigiRL
53.5
59.0
43.1 #36
47.6 4.2
Digi-Q (Ours)
61.5 2.3
71.2
53.1
58.0
ONLINE
DigiRI.
63.5 #3.1
74.5 12.6
52.6 116
57.3 43.1
Table 1: Main comparisons of different agents across various settings. Each offline experiment is repeated three times and the mean and standard deviation reported. To be consistent with prior work (Bai al., 2024), results are evaluated with autonomous evaluator with the first 96 instructions the train and test set.
5.1 MAIN PERFORMANCE RESULTS
Comparisons. We compare Digi-Q with prior methods for building Android device control agents. First, we compare Digi-Q with prompting-basee based methods that extend off-the-shelf proprietary VLMs such as GPT-4V (OpenAl, 2024b) and Gemini 1.5 Pro (GeminiTeam, 2024) with the Set- of-Marks (Yang et al., 2023) and a chain-of-thought mechanism for producing actions. We also compare with existing VLMs trained via imitation learning for device control: CogAgent (Hong et al., 2023), a 18B model. We keep results for these three approaches to be the same as scores previously reported in the DigiRL paper (evaluations were done in March 2024.'). !). That said, evaluate AutoUl-1B (Zhang and Zhang, 2023) again. Finally, we compare to a state-of-the-ar approach for training device control agents, DigiRL, which does not utilize state-action Q-value function, but rather uses a state-only value function and MC return estimates to estimate advantages only on on-policy actions. We evaluate our results on Android-in-the-We Will (AitW) with offine dataset containing 1296 trajectories for AitW Web Shopping subset and 1008 trajectories from AitW General subset from pre-trained Auto checkpoint, following Bai al. (2024). More details on the offline dataset can be found in Appendix B.3. To understand the ballpark of performance gains from Digi-Q, we also compare with DigiRI. in the offline-to-online setting, which is given access online interaction starting from dataset of 512 initial trajectories.
Results. Our main results are presented in Table 1. We find that Digi - outperforms all prompting- based methods substantially (53.5% absolute improvement on average compared to the best prompting- based approach AppAgent with GPT-4V) and improves over the previous state-of-the-are in the offline setting, DigiRL by around 21.2 21.2% relatively averaged on General and Web Shopping test subsets, and 31.5% relatively over Filtered-BC. a simple but strong baseline. In fact, the performance of Digi-Q even roughly matches the performance of DigiRL when is allowed perform on-policy interaction. By visualizing the agent's rollouts on test examples, as we will show in Appendix C. find that training policy with value functions enhances the capability of RL perform dynamic programming with sub-optimal data to learn a better policy in the environment.
5.2 ABLATION STUDIES
'e expect the latest evaluations all methods largely underestimate results from evaluations in March 2024 on the AitW Web Shopping subset because certain websites have started blocking agents from taking actions on the website. Details are shown in Appendix C.2. This issue affects both the baseline approaches and method, and is a direct consequence of the non-stationarity and dynamism of the web enviromment. This also means that for given fixed offline dataset, baseline performance numbers from March 2024 expected to be higher than if the prior approach were run again as the time writing this paper. Hence Digi-Q Q outperforms March 2024 evaluations of prior method, we can reliably expect it to outperform that prior approach today. We also collected trajectory dataset with higher success rate for offline training. which we expect should lead to better results than March 2024, but also expect results to be uniformly underestimated due to the evaluation issue.
8

---

Published as a conference paper at ICLR 2025
Next, we will perform series of controlled ex- periments to understand the reasons behind the efficacy of Digi-Q. Q.gi-Q.ie In particular, to understand (1) the effect of representation fine-tuning (Stage I) for seeding the VL.M rep- resentations for subsequent Q-function training, (2) the behavior of Best-of-N reranking style policy extraction operator compared to AWR (used by DigiRL (Bai al., 2024)) and standard REINFORCE-style policy gradients (Williams, 1992), (3) the benefits of TDD-earnnng over the more conventional approach of supervised re- gression to Monte-Cate Carlo return for training the value function, and (4) the scaling performance with more data of Digi-Q Q compared to other baselines. Experimental details of the ablation studies can be found in Appendix B.4.
Representation
Performance
will attempt
Behavior Policy
25.0
Digi-Q (w/ MC return)
37.5 + 4.5
Digi-Q Off-the-Shee Shee-Shelf
31.9 1.3
Digi-Q w/ BLIP-2 BERT
47.6 +
Digf-Q (Ours)
58.0 2.1
Table 2: Efficacy of representation fine-tuning procedure on the Web-Shopping test set AitW.
(1) The effect of representation fine-tuning in Digi-Q. We first analyze the effect of fine-tuning the VLM representations by training them to ac- curately detect actions that led to a substantial change in the scene. To do so, we compare Digi-Q with alternate approaches that train Q- functions on top of two other natural choices of representations: (a) not using a generative VLM (i.e., Llava- 1.5), but instead using frozen BLIP- 2 (Radford et al., 2021) and tuned BERT (Devlin et al., 2019) representations, following Bai et al. (2024); (b) using an off-the-shed VLM, with- out any representation fine-tuning (Chen et al., 2024).
Actor Objective
Performance
KL
Behavior Policy
25.0
REINFORCE
AWR 37.5 = 4.7
7.15
19.4 = 1.3
2.84
Digi-Q
58.0 1 2.1
3.28
Table 3: (1) Performance and (2) token-level KL-
divergence value between the learned policy and the dataset when using different policy extraction methods on Web Shopping test set. We utilize the same critic for all the methods, and only train the policy differently.
As shown in Table 2, observe that simply using an afi-tf VLM only leads marginal improvement over the behavior policy (31.9% compared to 25.0%): this is perhaps expected because an off-the-shelf generative VLM introduces a representa- tion that captures features about the scene holistically, but does not necessarily judge whether these features are actionable. As we will also qualitatively show in Section C.I, off-the-shelf VLMs also do not pay enough attention to action information, resulting in a Q-function that degenerates a similar solution as the state-only value function in the absence of aggressive action coverage in the data. In fact, Digi-Q using off-the-shelff VL. falls short of Digi-Q w/ BLIP-2 BERT as well. contrast, the representation fine-tuning procedure employed by Digi-Q is able to unlock the advantage of using rich VLMs and achieves more than 10% absolute improvement over the counterpart with BLIP-2 + BERT.
(2) The effect of Best-of-N reranking style policy extraction operator. Next, we aim to understand the impact of using Best-of-N reranking for policy extraction. This operator differs from traditional policy extraction methods in several ways: (i) the use of multiple actions (ii) not using a "negative gradicnt" (Tajwar et al., 2024) as in REINFORCE (Williams, 1992). To understand the effect of the number of actions in (i), we ablate Digi-Q over multiple values of N E 11,4,8 8, 16) in Figure 3 (Left). Observe that Digi-Q improves monotonically as N increases, indicating clear benefit of sampling more actions and reranking them against the Q-function during training. More discussions on the ablations of number of actions resampled can be found in Appendix B.3.
Next, we answer (ii) by comparing Digi-Q with REINFORCE and supervised regression (AWR). Our results in Table 3 show that while REINFORCE is able to achieve some improvements compared to the data collection policy (37.5% compared to 25.0%). it also suffers from the highest variance among all ablated methods. We hypothesize that this is a direct consequence of the negative gradient, which is known to sometimes destabilize training. While AWR (Bai et al. (2024)) does not suffer from this issue, it is also not able to stably improve the policy (19. (19.4% compared to 25.0%), likely because it is conservative. On the other hand, Digi-Q is able to make substantial improvements.

---

Published as a conference paper at ICLR 2025
Bigh-) (Gansall DighO (Whihchop)
DigiRt (Fenwrsl) Digitt (Webbshop)
0.60
0.75
0.55
0.70
Success Rate
0.50
0.45
0.60
0.440
0.55
0.50
0.35
Digi-O Offine (Wwbhshop)
Digilti Offine (Webshopp
0.30
256
1296
of Figure 3: Left: Performance of Digi-Q when varying the number of actions N used for policy extraction. Observe that the performance of Digi-Q improves when more actions are used for policy extraction, indicating the efficacy of our approach and the benefits of learning Q-function. Right: Data efficiency of Digi-Q and D.giRL.. The success rate of Digi-Q increases significantly faster than offline DigiRI. given the same amount of more data.
of Trajectories
Next we attempt to understand how "non-conservative" the updates made by different approaches are since concern with AWR-style updates in prior work the extent to which they conservative. We wish to understand if our Best-of-N reranking based policy extraction approach also admits conservative updates. To do so, we measured the KL-divergence between actions from the dataset and the fine-tuned policies produced by Digi-Q. AWR, and REINFORCE in Table 3. Note that Digi-Q incurs larger KL-divergence value unlike AWR that incurs the smallest deviation and is most conservative. In contrast, REINFORCE attains larger divergence values but behaves unstably (see Appendix C.3 for some example rollouts). Some qualitative examples for these variants are shown in Appendix C.I. Our results are also consistent with findings in concurrent work from robotic control problems (Mark et al., 2024).
(3) The effect of TD-leanning as opposed to MC. To understand the importance of TD-learning for training the critic over Monte-Carlo (MC) regression that previous work based on, we run an ablation of Digi-Q, which uses MC regression. Observe in Table 2, that this version underperforms Digi-Q by 20% (58.0% compared to 37.5%). As we show in Appendix C.I, value functions from MC regression exhibit high variance, which inhibits them from producing good policies when used to rank multiple actions.
(4) Scaling performance of Digi-Q with different amount of data. We present the comparison between Digi-Q and DigiRL. along the axis of the number of training trajectories in Figure (Right). For fair comparison, for Digi-Q we rerun all stages of training while varying the amount training data. As shown in Figure 3 (Right), we observe that Digi-Q outperforms DigiRL in all regimes, even in the low-data regime with only 256 trajectories. We suspect this due to the ability reuse data and perform better per-step credit assignment, thanks to a reliable Q function.
CONCLUSION AND FUTURE WORK
We presented Digi-Q, an effective method for training VLM Q-value functions from offline data, specifically for training real-world device-control agents. At the core of our method is representation fine-tuning procedure that induces actionable features from VLM useful for later TD-learning and Best-of-N policy training method that makes the best use of the learned Q function from TD- learning. While we primarily focus on GUI agent tasks on Android devices, our methodology is general, compute efficient, and leads to substantial improvement performance. We believe that these ideas and approach should transfer to new tasks as well and applying Digi-Q to new domains in an interesting avenue for future work. That said, using the critic Digi-Q in an active online self-improvement loop will require more sophisticated system design to speed up the experiment iterations and methods to robustify the critic as the distribution the agent policy drifts far from the base data collection policy with more online improvement. Nonetheless, ideas from Kalashnikov et al. (2018) could provide a good starting point to build policy learning systems based on TD-learnin during real-wor world interaction.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
Hao Bai would like to thank Nan Jiang. Shengbang Tong and Jiayi Pan for early discussions during the project. This work was partly done when Hao Bai was visiting scholar UC Berkeley. Aviral Kumar would like to thank Amrith Setlur for discussions. This work is supported by NSF IIS-224681L, 1, ONR N00014-24-12206 and ONR N00014-21-1-2836 We thank Google Cloud for providing Gemini 1.5 Pro credit donations for academic use and some GPU and TPU resources. We also thank the NCSA Delta cluster admins for providing us with GPU resources for training.
REFERENCES
Marwa Abdulhai, Isadora White, Charlic Snell, Charles Sun, Joey Hong, Yuexiang Zhai, Kelvin Xu, and Sergey Levine. Lmrl gym: Benchmarks for multi-turn reinforcement learning with language models, 2023. URL wwtp::tpts//ooo httpss://///iiiee org/ abs 2211 18232.
Hao Bai, Yifei Zhou, Mert Cemri, Jiayi Pan, Alane Suhr, Sergey Levine, and Aviral Kumar. Digirl: Training wi-l device-control agents with autonomous reinforcement learning, 2024. URL https: http//aaivv.v/// wwwpxiiv.rrg/e... wtt/abss/////oooooeo.o./o...esss /tdbs//405 2406. 11896.
Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, Thomas Unterthiner, Daniel Keysers, Skanda Koppula, Fangyu Liu, Adam Grycner, Alexey Gritsenko, Neil Houlsby, Manoj Kumar, Keran Rong, Julian Eisenschlos, Rishabh Kabra, Matthias Bauer, Matko Bošnjak, Xi Chen, Matthias Minderer. Paul Voigtlaender, Ioana Bica, Ivana Balazevic, Joan Puigcerver, Pinelopi Papalampidi, Olivier Henaff, Xi Xiong. Radu Soricut, Jeremiah Harmsen, and Xiaohua Zhai. Paligemma: A versatile 3b vlm for transfer, 2024. URL https: //t////axxa wttp/iixvvvv.///// org/ abs/2401) 07726.
Yevgen Chebotar, Quan Vuong, Alex Irpan, Karol Hausman, Fei Xia, Yao Lu, Aviral Kumar, Tianhe Yu, Alexander Herzog, Karl Pertsch, Keerthana Gopalakrishnan, Julian Ibarz, Ofir Nachum, Sumedh Sontakke, Grecia Salazar, Huong T Tran, Jodilyn Peralta, Clayton Tan, Deeksha Manju- nath, Jaspiar Singht, Brianna Zitkovich, Tomas Jackson, Kanishka Rao, Chelsea Finn, and Sergey Levine. Q-transformer: Scalable offline reinforcement learning via autoregressive q- q-functions, 2023. URL https: /tttpp://arxiivi org/ abs .20...39111 10150.
Baian Chen, Chang Shu, Ehsan Shareghi, Nigel Collier, Karthik Narasimhan, and Shunyu Yao. Fireact: Toward language agent fine-tuning, 2023. URL https: hta/aaxxvvv..../.. abs /2110. 05915.
William Chen, Oier Mees, Aviral Kumar, and Sergey Levine. Vision-language models provide promptable representations for reinforcement learning, 2024. URL httpss///////iiv.o abs 2bss/202.. 02651.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint 2rXiv:2110.14J11 2021.
Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang. Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web, 2023. URL abs/2306 06070.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, 2019. URL /ocs/1B10.. 04805.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning, 2021. URL https: / /aaxivivvv//./////// wrgg/bbs//000 07219.
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods, 2018. URL tatt:p::n //tpr//aaaaaaa/.aa/eaa/eeaeaaaaa wwwpldv..gg//b///eeaeeooo 09477.
GeminiTeam. Gemini: A family of highly capable multimodal models, 2024. URL https:
///axxiv.rrg//b///33//3 11805.
11

---

Published as a conference paper at ICLR 2025
Izzeddin Gur, Ofir Nachum, Yingjie Miao, Mustafa Safdari, Austin Huang, Aakanksha Chowdhery, Sharan Narang, Noah Fiedel, and Aleksandra Faust. Understanding html with large language models, 2023. URL https: : arxiv. www.i.ptgwabs//oooooooot.tt.taaa wttp://ppp.........../// 222./22255..0..5...
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor, 2018. URL https:
//arxiv. www.oorg///oooooo abs/ /abs/// 11011111110 01290.
Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. Webvoyager: Building an end-to-end web agent with large multimodal models, 2024. URL https://aarttti //tpxaaaaivv////////a//eaaeaawaa org/ hatp/ab//220 2abb/2401... 13919.
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203. 2555,, 2022.
Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxuan Zhang, Juanzi Li, Bin Xu, Yuxiao Dong, Ming Ding, and Jie Tang. Cogagent: A visual language model for gui agents, 2023. https http/aaivvvvorrro. aatg///b///aa 2312 08911.009111 4.
Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, and Sergey Levine. Qt-opt: Scalable deep reinforcement learning for vision-based robotic manipulation, 2018. URL https:
//arxiv. //arxiv..rggabs///// hbt.g//ab//880e 10093..26.........111111
Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po- Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. Visualwebarena: Evaluating multimodal agents on realistic visual web tasks, 2024a. URL https: http:///axxvvvvo..oeeeoo . 13649.
Jing Yu Koh, Stephen McAleer, Daniel Fried, and Ruslan Salakhutdinov. Tree search for language model agents, 2024b. URL https: //tp///avvvv////////////ee///waa wwweiiiiorrrr/eeee wtw//aa/////oooooooeo./o//eaaaer 2200//1000001
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning, 2020. URL https:://///.. http::/://aaaii..oooaaa///////// 02t.//200000.79. 04779.
Aviral Kumar, Rishabh Agarwal, Tengyu Ma, Aaron Courville, George Tucker, and Sergey Levine. Dr3: Value-based deep reinforcement learning requires explicit regularization. arXiv preprini arXiv:2112.04716. 2021.
Aviral Kumar, Rishabh Agarwal, Xinyang Geng, George Tucker, and Sergey Levine. Offline q- learning on diverse multi-task data both scales and generalizes. arXiv preprint arXiv:2211.1514: 2022.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2024a. URL https:///// arxii. org/ abs 22100. 03744.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lec. next: Improved reasoning, ocr, and world knowledge, January 2024b. URL https:
//1lava-v..githun hotbbb/////ooo//////////// 01 30--30--110 10--lava-next././//
Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang. Shudan Zhang, Xiang Deng, Aohan Zeng. Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. Agentbench: Evaluating llms as agents, 2023, URL https: / arxiv. org. abs /2008. 03688.
12

---

Published as a conference paper at ICLR 2025
Xiao Liu, Tianjie Zhang, Yu Gu, lat Long long. Yifan Xu, Xixuan Song, Shudan Zhang, Hanyu Lai, Xinyi Liu, Hanlin Zhao, Jiadai Sun, Xinyue Yang, Yu Yang, Zehan Qi. Shuntian Yao, Xueqiao Sun, Siyi Cheng, Qinkai Zheng, Hao Yu, Hanchen Zhang, Wenyi Hong. Ming Ding, Lihang Pan, Xiaotao Gu, Aohan Zeng, Zhengxiao Du, Chan Hee Song, Yu Su, Yuxiao Dong, and Jie Tang. Visualagentbench: Towards large multimodal models visual foundation agents, 2024c. URL https://arxiv. org/ abs/2400 06327.
Xinbei Ma, Yiting Wang, Yao Yao, Tongxin Yuan, Aston Zhang, Zhuosheng Zhang, and Hai Zhao. Caution for the environment: Multimodal agents are susceptible to environmental distractions, 2024. URL https: wttpiviivvv.////////ee////aaa..a org/ /abs/ abs/2408.02544
Max Sobol Mark, Tian Gao, Georgia Gabriela Sampaio, Mohan Kumar Srirama, Archit Sharma, Chelsea Finn, and Aviral Kumar. Policy agnostic rl: Offline and online rl fine-tuning of any class and backbone. arXiv, 2024.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning, 2013. URL attps:://rrxt www.taiwww.....aaaaaoooooooooott org/ 2tos//112 5602.
OpenAl. Gpt-4 technical report, 2024a. URL
OpenAL. Gpt-4v(ision) technical work and authors, 2024b. URL https://openai. com/
contrrabutions//ptt- 4v/.
Jiayi Pan, Yichi Zhang, Nicholas Tomlin, Yifei Zhou, Sergey Levine, and Alane Suhr. Autonomous evaluation and refinement of digital agents, 2024. URL httph::t////atee
06474.
Seohong Park, Kevin Frans, Sergey Levine, and Aviral Kumar. Is value learning really the main bottleneck in offline rl?, 2024, URL https:
/ttpps://axxi...t..///// /ttpxaaxxiv/.////////e.///aata httpiiv.orggab//////000 abs//40.. 09329.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression. Simple and scalable off-policy reinforcement learning, 2019. URL wwtttttss//aaiiiee
abs/1910. 00177.
Pranav Putta, Edmund Mills, Naman Garg, Sumeet Motwani, Chelsea Finn, Divyansh Garg, and Rafael Rafailov. Agent q: Advanced reasoning and learning for autonomous agents, 2024. URL
https:://aran orgo htgg/ab///// 2408 07199.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021. URL https:
//arxiv. o abs
www.///.. /21///110 2203..11... 00020.
Christopher Rawles, Alice Li. Daniel Rodriguez, Oriana Riva, and Timothy Lillicrap. Android in the wild: A large-scale dataset for android device control, 2023. URL /ttp:vv//arxvooooo orgg/ abs /2007 1 10088.
Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Marybeth Fair, Alice Li, William Bishop, Wei Li, Folawiyo Campbell-Ajala, Daniel Toyama, Robert Berry, Divya Tyamagundlu, Timothy Lillicrap, and Oriana Riva. Androidworid: A dynamic benchmarking environment for autonomous agents, 2024. URL https: //tr///aaiii www.a.ivvvooooo wwwp:iii.ororroooooooooooo abs//ab///aaa /20...4055.. 14573.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy 06347.
optimization algorithms, 2017. URL https: a aax org//bs/ /trt:::///rxtt..or//bbb
Zhihong Shao, Peiyi Wang, Qihao Zhu, Rumxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseckmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint Sitv:2402.03000 2024.
Charlie Snell, Ilya Kostrikov, Yi Su, Mengjiao Yang, and Sergey Levine. Offline rl for natural language generation with implicit language q learning, 2023. URL https: // atxiv.org. abs/2206. 11871.
13

---

Published as a conference paper at ICLR 2025
Yifan Song. Da Yin, Xiang Yue, Jie Huang, Sujian Li, and Bill Yuchen Lin. Trial and emror. Exploration-base trajectory optimization for Ilm agents, 2024. URL https: BuRLhttss://axxttt http:///axxvv...../.. abs abbs/240.. 02502.
Fahim Tajwar, Anikait Singh, Archit Sharma, Rafael Rafailov, Jeff Schneider, Tengyang Xie, Stefano Ermon, Chelsea Finn, and Aviral Kumar. Preference fine-tuning of Ilms should leverage suboptimal, on-policy data, 2024. URL https: //axiiv..//iivv///a///// https////wwwiw...aaaa/eee /abs//ab//// 2////4444..33.
Siddharth Verma, Justin Fu, Mengjiao Yang, and Sergey Levine. Chai: A chatbot for task-oriented dialogue with offline reinforcement learning, 2022. URL ptattps:/aaxxv.ooo/ www.:iiiorrg/a.... 2204. 08426.
Ronald J Williams.
Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8:229 256, 1992.
An Yan, Zhengyuan Yang, Wanrong Zhu, Kevin Lin, Linjie Li, Jianfeng Wang. Jianwei Yang, Yiwu Zhong, Julian McAuley, Jianfeng Gao, Zicheng Liu, and Lijuan Wang. Gpt-4v in wonderland: Large multimodal models for zero-shot smartphone gui navigation, 2023. URL https://
arxiv. org/ abs
Jianwei Yang. Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set-of-mark prompting unleashes extraordinary visual grounding gpt-4v, 2023. URL https:://arxi org/abs/ 2310..231... 11441.
Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents, 2023. URL abs/ 01206.
Aohan Zeng, Mingdao Liu, Rui Lu, Bowen Wang. Xiao Liu, Yuxiao Dong, and Jie Tang. Agentuuning: Enabling generalized agent abilities for llms, 2023. URL httptttt////xiiieeee www.iiivorrgaaaoooo /ab//ba//////eeeeooeee/ 22310. 12823.
Yuexiang Zhai, Hao Bai, Zipeng Lin, Jiayi Pan, Shengbang Tong, Yifei Zhou, Alane Suhr, Saining Xie, Yann LeCun, Yi Ma, and Sergey Levine. Fine-tuning large vision-language models as Mecision-making agents via reinforcement learning, 2024. URL https: http://aaavvvvvo/oe /bs// 2405. 10292.
Chi Zhang, Zhao Yang, Jiaxuan Liu, Yucheng Han, Xin Chen, Zebiao Huang, Bin Fu, and Gang Yu. Appagent: Multimodal agents smartphone users, 2023. URL https: //tp///a.v..///////////.//.aal wrgg abss/ 2312.1377..
Zhuosheng Zhang and Aston Zhang. You only look at screens: Multimodal chain-of-action agents. arXiv preprint arXhv:2309.11436 2023.
Zhuosheng Zhang and Aston Zhang. You only look screens: Multimodal chain-of-action agents, 2024. URL https:ttt..... / htaaaxiivv.o/ org/ abs/ 220/22399.22 11436.
Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. Gpt-4v(ision) is generalist web agent, if grounded, 2024. URL /taptpps///xxxi....// 220p///1./221....//// 01614.
Shuyan Zhou, Frank F. Xu, Hao Zhu. Xuhui Zhou, Robert Lo. Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. Webarena: A realistic web environment for building autonomous agents, 2024a. URL 2307 13854.
orbbbs// abs/
Yifei Zhou, Andrea Zanette, Jiayi Pan, Sergey Levine, and Aviral Kumar, Archer: Training language model agents via hierarchical multi-turn rl, 2024b. URL https: :/aoss://///ivii http:/aww...gg............. 2402. 19446.
14

---

Published as a conference paper at ICLR 2025
Appendices
A
DETAILS ON THE ALGORITHM
For completeness, we include detailed pseudo-code of Digi-Q in Algorithm After initializing the parameters, we perform the representation fine-tuning procedure on top of VLM to obtain actionable features for later TD>learning. Then the VL.M parameters will be kept frozen and we train the Q- and V- functions using T'D-leariing on top of frozen VLM representations, After both value functions are trained, we perform gradient updates on the actor with Best-of-N policy extraction.
Algorithm 1 Digi-Q: Practical Framework
Initialize parameters é, &'LL, PMLP, BMLP, 0MLP.
2: Initialize replay buffer 'D (from an offline dataset).
3: for each VLM iteration do
4: 0YLM < V (OVIM)
Equation 3
5: end for
6: for each critic step do
7: ## Update high-level Q and functions by target function bootstrapping.
8: OMLP BMLP Jo MUNP (Q)
Equation b Equation
9:
PMLP + VMLP V J.inn (V)
## Update target Q and V functions.
11: OMIP + (1 (1- 1 + T0MIP
4'MLP (1 -1-1 (T)PP TV/MPP
13: end for
## Update low-level actor with high-level critic.
15: for each actor step do
Φ (TT) 17: end for
Equation
B EXPERIMENTAL DETAILS
B. COMPUTE EFFICIENCY COMPARISON
A common concern with deploying TD-learning methods to train large-scale foundation models is their compute inefficiency (Abdulhai et al., 2023; Chebotar et al., 2023). Therefore, we attempted to understand the cempute-performann end-to-end TD-Jearning on VLMs without using any representation fine-tuning or frozen pre-trained representations. We plot the performance-complue tradeoff curve for Digi-Q the web-shopping subset of the AitW dataset in Figure 4. We found a bit hard to Ene-tune an entire VLM with TD-learning, which required iteration hyperparameters such as learning rate and soft update rates for target networks. Due to the compute- -ompute-intensive nature, we use a 3B VLM (PaLiGemma (Beyer al., 2024)) for these runs instead of our 7B VLM (Liu al., 2024b), and evaluate the performance of the critic as measured by the correlation between advantage predictions and ground-truth notion of human judgement on a held-out set of trajectories. In particular, find that end-to-end TD- learning exhibits a much worse training FLOPS hurts performance. We conjecture that this behavior likely result of well-known pathologies of training large models with TD learning (Kumar et al., 2022), though we leave it for future work to fully understand these pathologies in our context. contrast, while Digi-Q invests an initial amount of computation for representation fine-tuning, its accuracy quickly rises up and results in much better frontiers, with no instability. The calculation the FLOPS is shown below.
tradeoffs associated with Digi-Q by comparing it against
cetoomannc-coomput frontier, to the extent that beyond a point more
FLOPS Calculation. The 3B VLM takes 45.6 x 10 FLOPS for each sample for forward plus backward process. As the end-to-end TD learning contains one VLM as part of the Q function and one VLM the target Q function (which only do forward pass), sample takes 68.4 x 10 FLOPS (according to Hoffmann et al. (2022), the FLOPS incurred by the forward prrcess is approximately half of the backward process). Thus, the longest takes 15k samples, the last point of the
15

---

Published as a conference paper at ICLR 2025
0.64
0.62
Critic Accuracy
0.60
0.58
0.56
0.54
0.52
0.50
Qurs (3B)
End to-end TD (3B)
0.48
1.0
FLOPS (x1018)
Figure 4: Offline critic evaluation accuracy as function of compute measured in terms of training FLOPS, compared across Digi-Q. end-to-end TD-leamning VLM, and MC return. Observe that the critic accuracy much better for our approach end-to-end TD-Icaming as the amount of compute increases.
end-to-end run in Figure 4 takes around x 10t8 FLOPS. Also, the first logged point takes 128 samples, so the starting point should have 8.3 x 10 FLOPS.
On the other band, in Digi-Q, we first finetune the 3B VLM, which incurs only one forward and backward process. Thus, finetuning the 3B VLM on 2000 samples takes 91.2 x 10a5 FLOPS. After that, we infer the representations of these samples with the 3B VLM, which includes one forward pass. This sums up to 136.8 x 15 FLOPs, which explains the starting point of the Digi-Q curve. Then we only train the value head using the VLM representations. The size of the value head is 0.07B, incurring 1 x 1012 FLOPS for each sample. The longest run of Digi-Q takes 0.46M samples, thus incurring 506.9 10t5 FLOPS (10 x 10
Thus, the end-to-end TD learning should range from 0.0083 10 to 1 x 10 FLOPS, while Digi-Q should range from 0. 137 10 FLOPS 0.644 10t8 FLOPS, which shown in Figure 4.
Critic Accuracy. We manually label 483 states with binary advantages, and normalize the advantages produced by the agents to have human annotations.
of zero before thresholding and calculating its accuracy with
B.2 CRITIC MODEL ARCHITECTURE
We show the details of the critic model architecture in Figure In our environment setting, the states are composed of task, observation (screenshot at step !), previous observation (screenshot at step t 1), and previous action (action at timestep t - 1). The task and previous action are text strings, while observations Then we concatenate all these feature vectors and pass them through a MLP that tries to predict the value. The target of the V value is calculated by Equation 5.
images. We encode the text strings with BERT and images with BLIP-2 model.
The state-action features are modeled by the current action as well, which is string passed into not only the BERT encoder but also a part of the prompt passed into the VLM. The prompt is described in Appendix D. In the end, the Q features include the BERT embeddings, the BL.P-2 embeddings, and the VLM intermediate- into the another MLP that predicts the Q value. The target of Q value is calculated by Equation 4.
representations. We concatenate all of these feature vectors and pass
B.3 TRAINING DATASET CONSTRUCTION
We use the pre-trained AutoUl checkpoint to collect offline trajectories. Specifically, to collect each trajectory, starting from the home screen, the agent generates an action, and then the environment takes the action and transitions to the next state. It iterates until a maximum number of steps have been reached the autonomous evaluator has decided to be success. We collect 1296 trajectories
In this experiment, fix the BERT model when running Digi-Q.
16

---

Published as a conference paper at ICLR 2025
Critic Model Architecture
@bert (8)
BERT
Previous Action
V(s)
VMPP
Observation
@blip (s)
BLIP 2
Observation
Task
Øbert (s)
Previous Action
Action
Øbert (a)
MT.P
Q(s,a)
@vim (8, a)
VLM
Previous Observation
BI.T--2
@blip
Figure 5: Q-function architecture. The modules marked orange are trained, otherwise the module is kept fixed.
this way for both AitW Webshop and AitW General subsets. The horizon H of the Webshop subset set to 20, and the horizon of the General subset is set to 10, which aligns with (Bai al., 2024). pairs (s, o, r, s')), which is also referred
Each trajectory is composed of rete-aaciion-ewar--eeette to as "transitions".
The N actions in the offline dataset used for Best-of-N loss sampled post-hoc from the pre-trained AutoUI checkpoint. When training actor offline, as we use the Best-of-N loss, we want to sample more than one action. From engineering aspect, collecting actions each time we sample from the offline dataset D during training is not efficient. Thus, in practice, we pre-collect K = 64 actions for each state, and store them in the offline dataset. As N {1, 4, 8, 16) is much smaller than 64, this strategy serves as good approximation and results in good performance. It suffices to give enough variety compared to sampling the actions when training the actor model. Note that in this case, the original action will always appear in the offline dataset.
B.4 ADDITIONAL МETHON DETAILS
Task set formulations. The two task sets (Webshop and General) in the AitW dataset have different horizons H (maximum number of steps allowed) in a trajectory to improve computational efficiency. Specifically, H 20 for AitW Webshop and H 10 for AitW General. Following tradition (Bai et al., 2024), we keep A > 1/H (e.g. 0.05 for AitW Wehshop) as a threshold for the actor model to learn the state-action pair.
Ablation on representation fine-tuning and TD learning as opposed to MC. In the ablation study on representation fine-tuning, for all configurations, train the actor model with Best-of-N loss where N = 16 to keep computation efficient. This is also the case for the ablation on the TD learning as opposed MC ablations.
Ablation on actor loss. For the ablation study on the actor loss, we keep the same trained Q function, while we ablate only on the loss used train the actor model. We use 30 actor epochs for the Best-of-N loss and AWR loss, and 120 epochs for the REINFORCE loss as the magnitide of the raw advantage is very small. We use N 16 for the Best-of-N loss, while REINFORCE and AWR both uses the original action in the offlin dataset.
Value function. In practice, we find the function significantly easier to train, and it suffices to only use the representations of the state from the vision encoder of the VI.M (CIIP) to train the V functions. This simplification significantly saves time and space required, and aligns with previous work (Bai et al., 2024).
C QUALITATIVE EXAMPLES
C.1 QUALITATIVE VISUALIZATIONS
Qualitative comparisons between different value function learning approaches. To qualitatively understand the quality of the Q-function learned, in Figure 6, visualize advantage estimates
17

---

Published as a conference paper at ICLR 2025
Task
waltsar alinneure
(8,
Humat
7D Learning
0.03
0.09
-1.26
-0.13
4.01
4..2
0.00
0.85
Mores Curls
6.14
0.24
0.15
0.18
2:33
6.16
0.00
(P4838T)
0.54
Figure 6: Qualitative examples showing the advantage estimations of several transitions of TD (ours), Monte-Carlo, and TD without VLM representation. Advantage estimations using TD-lcarnt value functions top of VLM representation better align with human judgements compared to MC and TD without using VL.M.
=) = Qo(s, a) a (s) computed from Q-functions produced by four methods: (1) Digi-Q Q (with representation fine-tuning and TD-learning), (2) Monte-Carlo regression, (3) Digi-Q but using BLIP-2 + BERT representations from Bai et al. (2024); and (4) Digi-Q without representation fine-tuning. We compare advantages with human judgments of whether the actions mark progress towards the desired task. Ideally, good actions should attain a positive advantage. We observe that advantage estimates from MC regression
A(s,
often erroneous and uncorrelated with the human notion of good actions, perhaps because of the use of high-variance MC estimator. Moreover, we find that (3) converges to a degenerate (s, a) that approximately matches state-only value function, with limited meaningful sensitivity to the action input, which is problematic for policy learning. (4) mitigates this problem and produces different action values under the same state, but still fails fine- Line-grained differences like clicking at different positions on the screen. Thus, all these ablation variants perform suboptimally at attaining good correlation with human judgement, only (1) Digi-Q is able produce advantage estimates that align well with human annotations and judgement.
C.2 ENVIRONMENT ERRORS
We observe that several tasks has problems working in the environment introduced in Bai et al. (2024). We observe that (1) the newegg, com domain has high probability of blocking the agent from accessing it, and (2) the costco, com domain prevents the agent from typing the <ENTER> key. Examples are shown in Figure 7. These problems
observed in Bai et al. (2024). This is the main reason why some scores on the AitW Webshop subset in this paper falls a little behind Bai et al. (2024).
to costco.com, search for 'razer blade'
Eans
cosico.com does not respond <ENTER>
newegg.com, search for "usb-c usb-a"
newegg.co banned Internet agents
Figure 7: Environment errors. These errots are systematic and can not removed by agent.
18

---

Published as a conference paper at ICLR 2025
Go to costco.com, search for 'razer blade'
REINFORCE (Vanilla Policy Gradient)
stuck clicking the same location
Best-of-N
<wait>
solved
Figure 8: Example trajectory of the agent trained with REINFORCE and Best-of-N loss. Results show that the agent trained with REINFORCE tends to get stuck at specific state because it's "stubborn", while agent trained with Best-of-N loss effectively solves the task.
Task: Go to Walmart.com
A: successful but lengthy (offline dataset)
B: failed but short (offline dataset)
C: successful and short (Digi-Q)
Figure 9: Trajectory examples showing henefits Q-functions. Our method can combine the hest of successful but lengthy (A) trajectory and a failed but short trajectory (B), to produce successful and short trajectories (C).
C.3 EXAMPLE TRAJECTORY COMPARING REINFORCE AND BEST-OP-N Loss
We show typical trajectory produced by the agent trained with REINFORCE in Figure 8. We observe that the agent frequently diverges from the target and is too "stubborn" to recover from errors.
In this task towards searching for item costco.com, the agent has successfully arrived at costco.com, but (1) it takes some bad actions and (2) cannot recover. Specifically, after the agent clicks the warehouse button, it keeps clicking the same button for 10 times until it clicks on somewhere else. This situation rarely appear in any trajectories collect by the agent trained with the Best-of-N loss.
19

---

Published as a conference paper at ICLR 2025
C.4 BENEFITS OF DYNAMIC PROGRAMMING
An appealing property of value-based RL is dynamic programmingg the ability to identify optimal behaviors from overall suboptimal rollouts. We present qualitative example in Figure that illustrates this ability of Digi-Q in learning optimal behaviors from sub-optimal data. In this example, trajectory (A) and (B) are from the offline dataset where trajectory (A) successfully completes the task but has many redundant actions while trajectory (B) does not have redundant actions but fails to complete the task. turns out that Digi-Q is able learn a policy that performs dynamic programming with trajectory (A) and (B) to produce trajectory (C) that completes the task in the most efficient way. Neither trajectory (A) nor (B) is the optimal trajectory for solving the task but this example shows the ability of Digi-Q to learn optimal policy from sub-optimal data, which is theoretically impossible through imitation alone.
D VLM PROMPTS
The prompt we use for fine-tuning and inferring the VLM is shown in Figure 10. The prompt template is designed to be action-tyye setioontype-ppecffee in order to facilitate the VLM better differentiate different types of actions, which promotes fine- gee-graaned representations within the same action type. The input to the VLM is constructed by the image and the text prompt. Note that the VLM only the current image (overlayed with a cursor if the action is to click), and the next image is only used to calculate whether the target should be "yes" or "no". The target single token promote computational efficiency. In practice, we find that long target sequence introduces challenges for the VLM to fine-tune the representations.
VLM Prompt
Input: [[mage] Tochre given user intersaace, There a cursor the screen.
(Acsien-spect Fio promps))
Respond only "Yes or 'Nol (without pertod; quotation marks) and don't respond anything else. Yarget: "No"
(Aesion spect fi proept)
[Image] (shown VLM)
[fost Image] (eet shown VLM)
Thore as cursor in the screen. The touch point located at (couch point)).
this cursor Clicking
interactive elements?
T
given user interface. Type action effectively tnput sext into somewbere on Sereershott
now Types [typod, bext))
It's user now Prosses the <HOME> button, will this action affectively navigate the user to the Home screen?
n
It user now Presses the <BACK> button, will this action effectircly navigatc the user the previous screen?
It's
button, this action affectionly sebbmit
Figure 10: Prompt template we use to fine-tune and infer the VLM. The input prompt consists of an input image and text input. The text input include template prompt concatenated with an action-specific prompt. The action-specific prompt includes specific information about the input image. The output (target) prompt just word "yes" "no".
E HYPERPARAMETERS
Hyperparameters for Digi-Q Q are carefully tuned through binary search on the training set of General and Web Shopping subsets. The final choice of hyperparameters for both methods can be found in Table 4. Results for all other methods (Filtered BC and DigiRL) are kept the same as discussed in the original paper (Bai al., 2024).
20

---

Published as a conference paper at ICLR 2025
Method
Hyperparameter
Value
actor Ir value function Ir batch size maximum gradient norm actor updates per iteration
lc-4
Ic-5 (gencral), 5c-6 (webshop)
128
0.01
Digi-Q
30
value function updates per iteration
20
number of iterations oftline actor updates
15. 30, 45,
number iterations for offline value function updates
30, 40, 45, 60, 90, 120
model checkpoint
liuhaotian/llaa--l1.-11 pad, no peenaiclip-vii-larg-pttchhhh 3,5,8,
image aspect ratio
VI.M SFT
vision encoder
number of training epochs
per device train batch size per device eval batch size
16, 32
Table 4: Hyperparameters for Digi-Q on both General and Web Shopping subset AitW. If multiple values are displayed, the bolded value represents the selected value after hyperparamemter sweeping.
21


Reviews:
[{"review_id": "TMwKLAmlCv", "reviewer": "Reviewer_h5me", "summary": "The paper proposes an approach, called Digi-Q, for learning useful behaviour for device-control by leveraging offline data and VLMs. The authors highlight the current difficulties of learning using temporal difference learning and large retrained models. To address this difficulty, the authors propose to pretrain the VLM with in-domain data with state and action pairs, together with labels indicating whether the resulting state has changed significantly after the taken action. Additionally, the authors propose to use a best-of-N action sampling strategy, where the best-of-N is calculated through an approximate Q function. On the Android-in-the-Wild (AitW) domain, experiments show that Digi-Q improves upon previous approaches. The authors also present compute efficiency comparisons and ablation studies on some of the choices within the algorithm.", "questions": "Why use a separate policy network from the value? This is mentioned on the way, but never explained or referred.\n\nIn DigiRL, the authors perform a curriculum over tasks, is this strategy also employed here?\n\nThroughout the paper, the best-of-N strategy is referred to as being novel. I do not care if a method is novel or not, but proposing a method that is not novel (it is an incremental improvement on AWR, filtered BC and BCQ [3]]), and referring to it as being novel is not great.\n\n[1] Deep Reinforcement Learning at the Edge of the Statistical Precipice, Agrawal et al., 2022\n[2] Deep reinforcement learning that matter, Henderson et al., 2018\n[3] Off-Policy Deep Reinforcement Learning without Exploration, Fujimoto et al., 2018", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "The paper builds on recent algorithms and evaluates on domains that are increasingly important in the space of decision making and AI agents. The paper also highlights some of the challenges associated with reinforcement learning and VLM (e.g. instabilities that appear in practice) and proposes an approach that seems to present favorable results in experiments. Some of the ablations also give reasonable answer to important questions, for example the number of actions in the best-of-N sampling strategy.", "weaknesses": "The paper is generally not very rigorous from a scientific point of view. There are numerous descriptions of problems, hypothesis as to why one things are not working, and empirical justifiications that are unsubstantiated. For example the negative gradient hypothesis being mentioned multiple times originates from a paper on a preference fine-tuning, not agentic tasks. Other examples include the whole motivation in section 4.2: \"REINFORCE [...] is brittle with off policy data\"  \"negative gradient [...] means careful tuning of learning rates must be done\" ( \"AWR is quite conservative and slow\". When reading such statements, together with a complete misunderstanding of the fact that value-based methods do not equal off-policy learning, which also does not equates to offline learning (see Introduction), indicates to me that there is limited understanding and insights into what's happening, and therefore the contribution of the paper is lessened.\n\nArguably, the text could be fixed and made more precise, however these issues also arise in the algorithms and experiments. The proposed method, mixes quite a few things together: using the ArCHer learning rules, pretraining on in-domain data and best-of-N action sampling. For each of these choices, there is far from enough evidence to understand what is its importance.\n\nConsider pretraining on in-domain data, the paper mentions that labels are created when s_{t+1} is significantly different from s_t using the l_2 distance. Is this a general or even reasonable objective? Does this assume that the environment is entirely controllable by the agent, or deterministic? Given many papers in the RL literature, it clearly does not seem to be the case. Also, there is no study on the sensitivity to value of the threshold epsilon for calculating labels. \n\nAlso, looking at Figure 3 right, we see that when the number of actions for best-of-N is set to 1, the performance is similar to Filtered BC. Why do we not see performance difference given the fact the Digi-Q is built on pretraining the VLM first?\n\nThe main results raise a few questions. First, only 3 seeds are being used, please see the numerous papers that indicate that this a bad practice [1, 2]. Second, why is the performance of DigiRL different than the one reported in the original paper? Third, concerning the ablation in Table 3, how is the performance of AWR so low? Is the procedure for AWR not the same as the one proposed in DigiRL?\n\nThe results on compute efficiency conflate a few things together. Finetuning whole LLMs with RL can be troublesome (although it is possible as reported in a few recent papers), but this problem is mixed with compute efficiency. If performance degrades in the reported experiments, and full fine-tuning brings does not give as high a score as partial fine-tuning, it has little to do with compute efficiency, but rather with the practical challenges of full fine-tuning. In this sense, it is a bit meaningless to claim that partial fine-tuning is more efficient than full fine-tuning, if the used update rules don't work with full fine-tuning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "J5Oh39ldyg", "reviewer": "Reviewer_BK8U", "summary": "This paper develops a new RL method called Digi-Q. It addresses how to effectively use value-based offline reinforcement learning (RL) to train visual-language model (VLM) agents in dynamic environments (such as mobile device control) by training on frozen intermediate layer representations of the VLM through temporal difference (TD) learning instead of training on the entire VLM.\nI think the main innovation is Digi-Q. According to the paper, they address a series of challenges brought about by large-scale value-based offline policy RL: training instabilities associated with running temporal difference (TD) learning on large models, and inefficiency of TD backup per unit of computation. If you solved any of them, I will think it's a good novelty.", "questions": "In the paper, you mentioned that Digi-Q first fine-tunes representations of a VLM with a binary classification objective to enable it to pay attention to actionable features of an input scene. The sample step is in Figure 9 and it looks like targeted. How do you solve the Overfitting and Catastrophic Forgetting problem? Can you try to do some case studies or use different datasets to prove your performance improvement?  \nYou mentioned that observation is that in device control problems, a useful action should lead to a substantial visual change in the pixel values of a scene (e.g., successfully typing a search query and pressing enter on google.com should cause the page to change substantially to now show a list of search results, whereas an unsuccessful search attempt will change none to few pixels of the original scene). But in my daily life, the more possible is that a failed attempt can also lead to a substantial visual change. For a very common example, the page changes to whole white or 404 at my iPhone App when you fresh the web page but lose internet. Can you provide some statistical evidence for your observation? I think your assumption is to decline your model in generalization ability again. I know what Reinforcement Learning is but sometimes it's not normal in the real world.\nCan you talk more about limitation? Do you ask some people to test your model and do some case studies? Do get some feedback? Like you said before, you have some observations, so I assume it's only only from you but also from your tester's feedback.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The article is well-structured, with references for every sentence. And the author didn't speculate on the causes of certain phenomena observed in the experiments, without having conducted the experiments or found relevant references in the literature.\n2. It solves a series of challenges brought about by large-scale value-based offline policy RL: training instabilities associated with running temporal difference (TD) learning on large models, and inefficiency of TD backup per unit of computation. It ensures innovation. \n3. It also gives annotations of the papers from which ideas were borrowed, so that reviewers will not always think of some familiar operations when reading the experimental part, and spend a lot of time to confirm whether there is plagiarism.\n4. All research is based on the latest literature", "weaknesses": "1. Overfitting and Catastrophic Forgetting\n2. Misclassification of Visual Changes\n3. Real-World Applicability\n4. General Limitations and Feedback\nCheck the Questions to see the details, thanks :)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Z5eyBMpKZv", "reviewer": "Reviewer_sUX9", "summary": "This paper presents a novel method for training VLM-based RL agents. It is well known that training a VLM-based value network is highly unstable when using TD learning. This method first fine-tunes the VLM using representation learning to differentiate between actions that lead to transitions in the state space. The VLM parameters are then frozen, and the Q function on top of those layers is updated using the TD target. The policy is updated using the best of N actions from this Q function.\n\nThe main results focus on web-based navigation tasks, and the improvements are substantial. The accuracy per FLOP of training is also efficient and stable compared to fine-tuning an entire VLM.  There are extensive ablations verifying each part of the method.", "questions": "This paper is very well written and does not warrant any immediate questions.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* Benchmarks are sufficient; GPT-4V and Gemini are both s strong benchmarks.\n* Qualitative visualizations strongly support the hypothesis.\n* The ablations are numerous, exploring different representation learning methods to train the Q function, comparisons with Monte Carlo learning, and divergence from the behavior policy.\n* The paper is extremely clear and well written.\n* Experimental evaluations are strong and address difficult web navigation tasks.\n\nI recommend acceptance of this work, it presents strong results in an area of RL that is of very high impact currently.  Using VLMs to perform RL tasks is currently a direction of interest to most of the RL community.", "weaknesses": "* Novelty is a bit lacking, the main contribution of this method is simply fine-tuning upon the frozen lays using the TD loss after representation learning.  This is especially apparent put into the context of DigiRL", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "l1mZWzLd2E", "reviewer": "Reviewer_3SUU", "summary": "This paper introduces Digi-Q, a novel approach to making reinforcement learning work with large vision-language models (VLMs) for device control tasks. The authors tackle a challenging problem: while value-based reinforcement learning methods like Q-learning are known to be efficient, they've been notoriously difficult to use with large language models. The key insight of this work is that instead of trying to train the entire VLM using temporal difference (TD) learning, they first fine-tune the model's internal representations to better capture action-relevant features, then freeze these representations and only train a small Q-function on top. They also introduce a \"Best-of-N\" policy extraction method that samples multiple potential actions and trains the policy to imitate the ones rated highest by the Q-function. The authors evaluate their approach on Android device control tasks, showing improvements over previous methods and better computational efficiency than end-to-end TD learning. While the improvements are modest (about 10% better than previous methods) and limited to one domain, the work presents a practical approach to combining value-based reinforcement learning with large vision-language models, supported by thorough empirical validation and ablation studies.", "questions": "1. Could you discuss whether and how this approach might generalize to other domains? Have you attempted any preliminary experiments with different types of agent tasks?\n2. The Best-of-N policy extraction method lacks strong theoretical justification. Could you provide more insight into why this approach works better than alternatives? How did you choose N=16 as the optimal value, and how sensitive is the method to this choice?\n3. While you show improved compute efficiency compared to end-to-end TD learning, could you provide more concrete details about the total computational resources required for training? This would help practitioners better understand the real-world applicability.\n4. Could you provide more detailed analysis of training stability across different random seeds and hyperparameters? The current results show standard deviations, but a deeper analysis would be valuable.\n5. Could you provide examples of scenarios where your method struggles and analyze why these failures occur? The paper would benefit from a more thorough discussion of failure cases.\n6. How dependent is your method on the specific VLM architecture used? Have you tested with different VLM backbones, and if so, how does the performance vary?\n7. The representation fine-tuning phase seems crucial to your method's success. Could you provide more details about how sensitive the method is to different fine-tuning objectives or architectures? Have you explored alternative approaches to making VLM representations more action-aware?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper demonstrates several notable strengths across different dimensions. On the technical side, it successfully adapts Q-learning to work with large VLMs in a practical way. The two-phase approach of fine-tuning representations before freezing them for Q-learning is clever and addresses real computational challenges, while the Best-of-N policy extraction method offers a more stable alternative to traditional policy gradients (though the improvements are modest). The empirical work is thorough, with comprehensive ablation studies and comparisons against strong baselines like GPT-4V and Gemini, backed by proper statistical reporting across multiple runs. The presentation is clear and well-structured, with effective use of figures and helpful qualitative examples that illustrate how the method works in practice. From a practical perspective, the work addresses a real problem that practitioners face when trying to use Q-learning with large models, and while the 9.9% improvement isn't revolutionary, it represents meaningful progress. Importantly, the authors provide complete implementation details and hyperparameter choices, making their work reproducible. While none of these strengths are groundbreaking on their own, together they represent a solid engineering advance that makes value-based RL more practical with large models. The work is particularly strong in its empirical validation and clarity of presentation, even if the core technical innovations are relatively straightforward extensions of existing ideas.", "weaknesses": "The paper has several notable limitations that temper its impact. Most significantly, the evaluation is restricted to a single domain (Android device control), making it unclear whether the approach generalizes to other types of agent tasks or VLM applications. While the authors show a 9.9% improvement over previous methods, this is a relatively modest gain that comes with considerable complexity in the training pipeline. The theoretical foundation for the Best-of-N policy extraction approach is somewhat thin - while it works empirically, we lack a clear understanding of why this particular method is effective or how to choose the optimal value of N. The computational efficiency claims, while promising, would benefit from more detailed comparisons across different model scales and task complexities. There are also some concerning gaps in the analysis: the authors don't thoroughly explore failure cases or limitations of their method, and the stability analysis across different random seeds and hyperparameters could be more comprehensive. From a technical perspective, while the idea of fine-tuning representations before freezing them for Q-learning is practical, it's a relatively straightforward combination of existing techniques rather than a fundamental advance in how we approach VLM training. The ablation studies, while thorough in some areas, don't fully explore the sensitivity of the method to various design choices, particularly in the representation fine-tuning phase. Finally, the paper would benefit from a more detailed discussion of the computational resources required for training, as this is crucial information for practitioners considering adopting this approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7keqaTAyCo", "reviewer": "Reviewer_NJc5", "summary": "The paper presents Digi-Q, a value-based offline reinforcement learning (RL) approach aimed at training vision-language models (VLMs) for device control, specifically in Android GUI tasks. Digi-Q introduces a stable temporal-difference (TD) learning method on frozen VLM layers, optimizing Q-values while avoiding end-to-end model instability. Digi-Q also introduces a unique Best-of-N policy extraction that selects the best action among multiple candidates to improve policy performance without using traditional policy gradients.", "questions": "1. In Table 1, the main comparisons of different agents across various settings raise significant concerns about the reliability and consistency of the reported results. The paper claims, \"To be consistent with prior work (Bai et al., 2024), results are evaluated with the autonomous evaluator with the first 96 instructions in the train and test set.\" However, it appears that results for GPT-4V, Gemini 1.5 Pro, and CogAgent were directly copied from the DigiRL paper [1], while experiments were \"reconducted\" only for AutoUI and DigiRL. Notably, the results for AutoUI show a significant improvement compared to the previously reported figures in [1], while DigiRL’s offline results are selectively reduced.  This selective approach to data raises substantial concerns. If the intention was to reproduce results, it would be expected that any shifts in performance would be consistent across all models, not selectively applied. Furthermore, a success rate fluctuation of up to **5%** relative to previously reported results, given that the reported improvements are relatively modest, calls into question the robustness and reliability of the findings. Such fluctuations suggest that the experimental setup or evaluation may not be sufficiently stable, casting doubt on the paper's claims of improvement. I would appreciate clarification regarding the rationale for selectively re-evaluating some baselines and not others, as well as an explanation for the considerable performance variance observed. Without such transparency, the contributions of this work appear uncertain and potentially unreliable.\n\n     **Hopefully, you can answer my concerns regarding these messy results or the \"free lunch results\" you used here.**\n\n2. Why did you only evaluate your model on two subsets of AiTW? Could you explain the decision not to include other tasks, such as app installation, which would offer a broader evaluation of your model’s capabilities?\n\n3. How similar are the evaluation tasks to those used during training? Please clarify the degree of overlap or differences, as this impacts how well the model generalizes beyond its training set.\n\n\n\n\n[1] Hao Bai, Yifei Zhou, Mert Cemri, Jiayi Pan, Alane Suhr, Sergey Levine, and Aviral Kumar. Digirl: Training in-the-wild device-control agents with autonomous reinforcement learning, 2024.", "limitations": "", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "1. The paper proposes an innovative Q-value-based RL approach, which integrates TD-learning with VLMs, to increase sample efficiency for complex environments.\n2. The paper introduces Best-of-N policy extraction, enhancing policy learning stability by leveraging multiple action candidates.\n3. The paper demonstrates improved compute efficiency over end-to-end TD learning, effectively addressing scalability in large models.\n4. Problem formulations are okay, evaluations over the AiTW subset are comprehensive\n5. Paper writing is good and presentations of results are good.", "weaknesses": "1. *Lack of Clear Motivation for Offline Value-Based Approach*: The paper does not sufficiently motivate the use of an offline, Q-value-based RL approach for device control, especially given the recognized stability and efficacy of methods like Advantage-Weighted Regression (AWR) and Generalized Advantage Estimation (GAE) as shown in previous work by [1] Bai et al. and [2] Pan et al. In particular, Q-value-based methods are known to introduce instability, especially in scenarios with partial observability, where AWR and GAE have demonstrated superior stability and simpler implementation when dealing with much more unstable and complex environments for on-device control.\n\n2. *Limited Novelty Compared to DigiRL*: The paper's novelty is questionable when compared with previous works, especially DigiRL [1] by Bai et al. While Digi-Q proposes certain adaptations, such as the Best-of-N policy extraction, these contributions appear to be incremental rather than fundamentally advancing the state of value-based RL for device control.\n\n3. *Concerns Over Experimental Data Reliability*: The experimental results lack reliability, particularly in light of my own testing experience. Many observed metrics and success rates in Digi-Q’s experiments suggest significant variance, casting doubt on the robustness of the results. Additional benchmarks and repeated trials would help validate these findings and ensure their reproducibility.\n\n4. *Assumption of High-Quality Offline Data Set*: The paper's methodology hinges on a high-quality, well-curated offline dataset (e.g., AiTW), assuming this accurately represents all relevant scenarios. However, for real-world device control applications, where app behaviors and mobile environments change frequently, the method should ideally support a combination of offline and online data collection. Relying solely on offline data for pretraining limits adaptability, and the paper does not provide sufficient insight into how the pretrained policy can improve performance in dynamic online interaction settings.\n\n5. *Lack of Evidence for Scalable Training at Scale*: Although the paper posits the question, “Can we train VLM agents at scale with value-based RL?” it falls short in demonstrating this scalability. There is a lack of empirical evidence supporting large-scale training or fine-tuning experiments, and the scalability of Digi-Q in practical, resource-intensive environments remains unclear without these demonstrations.\n\n[1] Hao Bai, Yifei Zhou, Mert Cemri, Jiayi Pan, Alane Suhr, Sergey Levine, and Aviral Kumar. Digirl: Training in-the-wild device-control agents with autonomous reinforcement learning, 2024.\n\n[2] Jiayi Pan, Yichi Zhang, Nicholas Tomlin, Yifei Zhou, Sergey Levine, and Alane Suhr. Autonomous evaluation and refinement of digital agents, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)