SlowGuess/NeurIPS_2023_OCR huggingface dataset:
CC-Net when each model is trained only with behavioral cloning, as CC-Net performance on this setting drops to 38.7 (results not shown in the Figure). Notably, CC-Net scores also drop by approximately  $10\%$  when the model is not given access to a dictionary of input strings provided by the environment. As shown in Figure 3, the key to our model's ability to achieve comparable performance without relying on DOM-based inputs is pixel-based pre-training. Another difference is that CC-Net uses a real time setting, which enables some forms of interaction not supported by our environment, and therefore can support a larger set of MiniWob++ tasks. On the other hand, for BC, CC-Net does not need to handle the shift in rendering format and potentially noisy action space conversion.
For WebShop, the best model on this setting is WebGUM (Furuta et al., 2023a), which leverages the HTML source, a custom action space for the shopping domain, and a Flan-T5-XL (Chung et al., 2022) backbone. WebGUM outperforms Pix2Act when compared on this setting. Some of this gap can be attributed to their simplified high-level action space, direct access to the relevant text on the page, and ability to transfer from Flan-T5's pretraining scale and instruction finetuning. Comparable improvements to the scale and pretraining of pixel-based models could reduce this gap.
We discuss other approaches that leverage DOM or HTML information further in §6. We also offer a complete comparison across all MiniWob++ tasks in Appendix C.
**Evaluating transfer across tasks** Training a pretrained, pixel-based model to interact with a GUI can intuitively lead to better generalization to new tasks that use common GUI design principles. To study this, we evaluate the ability of  $\mathbf{PIX2ACT}$  (without RL) to generalize to tasks unseen during training. Specifically, we hold out 9 out of 59 tasks and train on the remaining  $50$ . We then evaluate performance on the held-out tasks, comparing initializing with  $\mathbf{PIX2STRUCT}$  to random initialization. Table 1 illustrates that  $\mathbf{PIX2ACT}$  can reach a mean score of 28.3 on held out tasks compared to 65.5 when training on those tasks. Conversely, mean score is 7.6 when  $\mathbf{PIX2STRUCT}$  initialization is not used. This shows that combining pretraining with a general GUI interface can lead to non-trivial generalization to held out tasks.
For WebShop, we find that finetuning directly on WebShop (without intermediate finetuning on MiniWoB++ as mentioned in 5.1) results in a drop of 4.0 in Task Score, demonstrating transfer learning benefits across these datasets.
Tree search analysis Table 2 shows the improvement in MiniWob++ scores by training on episodes generated by tree search. After an initial round of training on episodes generated by tree search, the effectiveness of tree search also improves due to improvements in the underlying model used to guide the search. The best greedy policy achieves performance close to the best tree search policy, but does not require access to reward signals or additional exploration at inference time. Our results indicate that we could further improve performance with more iterations of policy improvement via tree search.
<sup>8</sup>We manually pick the 9 tasks to verify they include only actions or elements that would be reasonable to generalize to from the training tasks. The tasks are click-checkboxes-large, click-color, click-tab-2, click-tab-2-hard, count-shape, drag-shapes, use-color-wheel-2, useslider-2.
We focus on agents that interact with GUIs, such as operating system dialogs or web pages, to accomplish a given task. Many early approaches relied on the structured information from the GUIs (Zettlemoyer and St. Amant, 1999; Allen et al., 2007; Branavan et al., 2010). This information could range from a flat list of GUI components and their properties, to the full hierarchical structure of the components (e.g. the DOM tree). The output space also depends on this structured information, often using GUI components as action targets (e.g. clicking button #7). As discussed in §1, such structured information might not always be available, or might not align with what visually appears to the users.
When Shi et al. (2017) introduced the World of Bits tasks, which was the precursor to MiniWob++ (Liu et al., 2018), they proposed a model based on a convolutional neural network that takes both visual and structured inputs and then performs generic low-level computer actions (e.g. clicking at a coordinate or pressing a key), similarly to  $\mathbf{PIX2ACT}$ . However, the model performed poorly compared to humans. Follow-up work studied specialized architectures for incorporating structured DOM information and restricted the action space to clicking and typing predetermined texts on DOM elements (Liu et al., 2018; Gur et al., 2018; Jia et al., 2019). Humphreys et al. (2022) reconsidered incorporating both visual and structured information as well as a low-level action space that aligns better to the human demonstrations. We discussed their approach, CC-Net, in §5.3. Humphreys et al. (2022) also explored the benefits of large-scale human demonstrations, and we build on their work to utilize a large number of human demonstrations to train  $\mathbf{PIX2ACT}$ . This paper shows that  $\mathbf{PIX2ACT}$ , a model with pixel-only inputs, can outperform humans on MiniWob++ and match the state-of-the-art approaches that rely on DOM information.
Automating web-based tasks using large language models (LLMs) has also been broadly explored. For instance, WebGPT uses a text-based web browsing environment to search and navigate the web (Nakano et al., 2021). More relatedly, recent work has investigated prompting LLMs to produce agents that can generalize to tasks based on a small number of in-context examples. Yao et al. (2023) proposed ReAct, a few-shot prompted LLM, which uses observations derived from HTML and a custom action space to make predictions based on explicit reasoning steps. Similarly, Kim et al. (2023) proposed RCI, a prompted LLM that iteratively critiques and refines its outputs, also using HTML inputs and custom action spaces. These approaches achieve competitive performance on WebShop and MiniWob++, respectively, and are extremely sample-efficient, relying on just a handful of demonstrations per task. Gur et al. (2022) treated raw HTML as a string and fed it to LLMs pretrained on natural language. After fine-tuning them on demonstrations, the models improved MiniWob++ task success rate and sample efficiency compared to models that take DOM-based inputs and specialized architectures. Finally, WebGUM (Furuta et al., 2023b), discussed in §5.3, extends HTML-based models to integrate a vision encoder pretrained on ImageNet-21K.
Other work has focused on tasks related to mobile apps. Li and Li (2022) considered a model with pixel-based inputs similar to that of Lee et al. (2022), and included evaluations on tasks related to grounding instructions to screenshots, but did not consider interactive environments. Some work has considered instruction following tasks in mobile app environments (Li et al., 2020a; Burns et al., 2022), but has generally not studied observation and action formats similar to ours, instead relying on inputs based on the Android view hierarchy. We focused on web-based GUIs so that we could use a consistent environment framework for simplicity. Besides GUIs, several works on video game agents also considered visual-only input and low-level actions. For example, most works on Atari games used the screenshot as visual input and predicted the controller buttons to press (Mnih et al., 2015). More recently, Baker et al. (2022), which focuses on learning from unlabeled videos, proposes an agent for Minecraft that uses pixel-based inputs paired with keyboard and mouse actions, similarly to  $\mathbf{PIX}2\mathbf{ACT}$ .
Pixel-based vs. text-based representations Text-based representations may be practically useful when available, especially since they enable transferring knowledge from LLMs, demonstrating impressive few-shot learning with LLMs for MiniWob++ (Kim et al., 2023) and WebShop (Yao et al., 2023). When structured source is not available, OCR systems and models trained to predict the location and function of UI elements may also help connect models with the power of LLMs. On the other hand, similar advances in scaling and pre-training of vision or multimodal models could
potentially enable similar capabilities in a pixel-based setting in the future, as we have shown the effectiveness of pixel-based pre-training (albeit at a smaller scale) for GUI-based tasks. Nevertheless, beyond addressing the case where HTML or DOM information is unavailable, we hope our study contributes towards a better understanding of the potential of pixel-based representations for instruction following via GUIs.
Tree Search Our approach to policy improvement with tree search for MiniWob++ relied on the ability to procedurally generate new MiniWob++ environment and instruction variations and receive reward signals for task completion. Both aspects are unlikely to be available for some real world environments, and such an approach might need to rely on generative models of potential instructions and approximate reward models for task completion (e.g. Bahdanau et al. (2018); Du et al. (2023)). Our implementation also relied on the ability to reset the environment to an initial state, a useful feature for environments being used for exploration. Additionally, while we show that tree search can be sufficient to reach high performance on MiniWob++, we did not perform a detailed comparison relative to other search and RL algorithms in this study, which would be useful to better understand the most efficient approaches for learning from GUI-based environments.
Broader Impact In this paper we have trained and evaluated models only in offline environments. Responsibly deploying models in an environment where they can interact with online services would require additional considerations. Prior to enabling a model to access a new service, it would be important to sufficiently verify and/or constrain the behavior of the model to ensure that it is consistent with the terms-of-service for that service and does not otherwise cause harm. Ensuring sufficient data privacy could also be an important consideration for deploying models such as P1x2ACT that rely on capturing screenshots from browsers.
There would be many potential risks associated with deploying models that could interact with services in violation of their terms-of-service or otherwise engage in various forms of spam, fraud, or abuse. Examples of such behavior could include impersonating human users, generating harmful content or spam, or engaging in denial-of-service attacks. Models that use the same conceptual interface humans use could potentially be more capable of breaking security defenses (e.g. solving CAPTCHA) or engaging in forms of spam, fraud, or abuse that are more difficult to detect. It is therefore important for research related to security and techniques for detecting spam, fraud, and abuse to take such potential uses into account.
We would like to thank Peter Humphreys, Toby Pohlen, and Gregory Thornton for their assistance with the MiniWob++ demonstraions. We also thank Ming-Wei Chang, Austin Huang, Luheng He, Tianze Shi, David Gaddy, Jacob Eisenstein, and Yi Luan for useful discussions and comments.
James F. Allen, Nathanael Chambers, George Ferguson, Lucian Galescu, Hyuckchul Jung, Mary D. Swift, and William Taysom. Plow: A collaborative task learning agent. In AAAI Conference on Artificial Intelligence, 2007.
Thomas Anthony, Zheng Tian, and David Barber. Thinking fast and slow with deep learning and tree search. Advances in neural information processing systems, 30, 2017.
Dzmitry Bahdanau, Felix Hill, Jan Leike, Edward Hughes, Arian Hosseini, Pushmeet Kohli, and Edward Grefenstette. Learning to understand goal specifications by modelling reward. In International Conference on Learning Representations, 2018.
Bowen Baker, Ilge Akkaya, Peter Zhokov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, and Jeff Clune. Video pretraining (VPT): Learning to act by watching unlabeled online videos. Advances in Neural Information Processing Systems, 35:24639-24654, 2022.
S. R. K. Branavan, Luke Zettlemoyer, and Regina Barzilay. Reading between the lines: Learning to map high-level instructions to commands. In Annual Meeting of the Association for Computational Linguistics, 2010.
A Additional Dataset Details
A.1 MiniWob++ Supported Tasks
MiniWob++ consists of 104 tasks. Most prior work (Shi et al., 2017; Liu et al., 2018; Gur et al., 2018; Jia et al., 2019) has evaluated performance on only a subset of these tasks, with the notable exception of Humphreys et al. (2022), which evaluated on all 104 tasks. We evaluated on 59 of these 104 tasks, based on our best effort attempt to (1) design a general purpose set of actions that could be implemented using Selenium and (2) convert the demonstrations collected by Humphreys et al. (2022) to our observation and action format. While further development of the conversion process and Selenium-based actions could potentially support more tasks, the 59 tasks we support still include a wide range of instructions and interactions. Note that determining the set of 59 tasks was based solely on the feasibility of conversion to our observation and action format, and not based on model performance. Below we offer further details.
Several tasks in MiniWob++ feature animated elements. These tasks can require sampling observations in a real-time manner in order to capture the information needed to select the correct action. Also, the effects of an action may be delayed and therefore not captured by an observation sampled immediately after the action has executed. MiniWob++ provides a -nodelay version for several tasks which removes such animations. We train and evaluate on the -nodelay version of these tasks (choose-date, click-collapsible-2, click-collapsible, click-pie, use-autocomplete). We exclude choose-date-easy and choose-date-medium which offer simpler versions of choose-date but do not have a corresponding -nodelay version. Additionally, we exclude chase-circle, drag-cube, moving-items, and simon-says, which feature animation without a -nodelay version.
Many MiniWob++ tasks also involve vertical scrolling. In the human demonstrations, this can be implemented using a scroll wheel, or various clicking or dragging interactions with a vertical scroll bar rendered on the right side of a scrollable element. Mapping such interactions to actions that lead to equivalent scrolling in our Selenium-based environment is non-trivial. Therefore, for simplicity, we excluded tasks that involve scrolling: book-flight, click-Scroll-list, email-inbox, email-inbox-nl-turk, read-table, read-table-2, scroll-text, scroll-text-2, search-engine, social-media, social-media-all, social-media-some, terminal.
Demonstrations for many MiniWob++ tasks also include copying and pasting text. In many cases, this was executed in the human demonstrations by double clicking a text string and then clicking and dragging it into an input field. Such an interaction is not supported in Selenium, which made it challenging to support these tasks. This led us to exclude the following tasks: login-user-popup, copy-paste, copy-paste-2, email-inbox-forward, email-inbox-forward-nl, email-inbox-forward-nl-turk, email-inbox-nosscroll, email-inbox-reply, email-inbox-star-reply, enter-password, enter-text, enter-text-dynamic, find-word, login-user, multi-Layouts, multi-orderings.
Finally, we excluded several other tasks for various other reasons. The choose-list task uses the HTML <select> tag to implement a drop-down menu, which is not supported properly by our Selenium-based environment. The click-menu and click-menu-2 tasks require unsupported mouseover effects. Demonstrations for the text-editor task features click and drag interactions to highlight text which do not have the same effect when executed in Selenium. There also appeared to be differences in how Selenium implemented the number input field for guess-number. Finally, we excluded several tasks due to low demonstration conversion success rates (focus-text, focus-text-2, use-spinner). Upon further investigation, this was due to episodes completing immediately after a "pointer down" event without a complete click for focus-text and focus-text-2, and due to frequent double clicking for use-spinner.
A.2 MiniWob++ Rendering Differences
There are differences between the rendering of observations in the human demonstrations from Humphreys et al. (2022) and the rendering of environment state in our Selenium-based environment. We show an example in Figure 5, which shows subtle differences, e.g. in font style and in element sizes and positions.
Figure 5: Comparison of differences between the screenshots of the human demonstrations for MiniWob++ from Humphreys et al. (2022) (right) with how the same environment state is rendered in our Selenium-based environment (left).
DJVKEV
□Mf6RMO 
□yy
□DJVKFV
□Mf6RM0
□yy
B Additional Technical Details
B.1 Beam Search
As mentioned in §3, we use beam search over tokens in the text decoder to produce a set of top- $k$  actions for a given state, along with their approximate probabilities. We refer to these as approximate probabilities because they are subject to a length normalization factor (Wu et al., 2016) of 0.6 during beam search, following Raffel et al. (2020). For MiniWob and WebShop, our experiments used  $k = 8$  and  $k = 10$ , respectively.
B.2 Tree Search
Here we describe the details of the tree search approach described in §3.1. We adopt Monte Carlo Tree Search (MCTS) (Coulom, 2006), and follow prior work which has integrated MCTS with neural networks (Silver et al., 2017; Anthony et al., 2017), which we apply to MiniWob++ environments. We performed a minimal amount of tuning to determine an approach that yielded improvements in mean score over the greedy policy, even for the most challenging tasks.
Problem Setting We consider an environment with states  $S$  and actions  $\mathcal{A}$ . The reward function,  $r(s)$ , returns a scalar corresponding to the reward given for transitioning to state  $s \in S$ , and is described below. MiniWob++ environments are randomly generated, but transitions are deterministic within an environment generated by a particular random seed. The transition function,  $f(s, a)$ , returns the state resulting from taking action  $a \in \mathcal{A}$  in state  $s \in S$ .
Surrogate reward Rather than using the raw reward directly provided by the MiniWob++ environment, we consider a surrogate reward:  $r(s) = \alpha_s + r^t(s)$ , where  $\alpha_s$  provides a small negative reward that encourages shorter trajectories without unnecessary actions.  $r^t(s)$  is the raw reward from the MiniWob++ environment if  $s$  is a terminal state and the raw reward is  $> 0.8$ , or 0 otherwise. We use  $\alpha_S = -\frac{1}{30}$ . As all tasks can be completed within 30 steps, this is small enough to ensure a positive reward is possible for all tasks. Additionally, the penalty is small enough such that in practice the agent should not be incentivized to sacrifice raw reward to reduce the number of steps taken.
Value network The value function  $v^{\pi}(s)$  for a given policy  $\pi$  is the expected future rewards from state  $s$  if actions are selected according to policy  $\pi$ . The optimal value function,  $v^{*}(s)$ , is the expected future rewards if optimal actions are chosen. We attempt to learn an approximation of this function,  $\hat{v}_{\phi}(s) \approx v^{*}(s)$ , parameterized as a PIX2STRUCT-initialized model with parameters  $\phi$ , which we refer to as the value network. The model is trained on transitions from the human demonstrations, which demonstrate close to optimal behavior in many cases. For every state in the human demonstrations, we compute the actual future rewards for the given episode, according to the surrogate reward. We map these future rewards to discrete bins and represent them as integers in the PIX2STRUCT decoder. At inference time, we approximate the mean of the distribution over these discrete bins by considering the top-  $n$  predictions from the model using beam search (with  $n = 3$ ), weighted proportional to their respective probabilities.
Policy network For consistency with prior work, we will refer to the  $\mathrm{PIX2STRUCT}$  model tuned to generate actions (i.e.  $\mathrm{PIX2ACT}$ ) as the policy network, with parameters  $\theta$ . The greedy policy  $\pi_{\theta}(s)$  selects the action  $a$  with the highest approximate probability  $p_{\theta}(a|s)$  in the top- $k$  beam (see §B.1), subject to the conditions described in §3.
Search policy We can use lookahead search to implement a policy,  $\pi_{\theta}^{*}(s)$ , which leverages interactions with the environment  $(f(s,a)$  and  $r(s))$  to select actions in a more optimal way than the greedy policy  $\pi_{\theta}(s)$ . Both the policy network and value network are used to constrain and prioritize the search.
MCTS performs  $K$  rounds of traversing a search tree with nodes corresponding to states, and edges corresponding to actions. Due to the computational cost of the policy and value networks, we use a modest number of rounds,  $K = 16$ , for our experiments. The search tree is initialized with a single root node for state  $s$ . Each round starts at  $s$  and traverses the tree. At each step  $t$  of a given round, an action  $a_{t}$  is selected for state  $s_{t}$ , where  $a_{t} = \max_{a} Q(s_{t}, a) + U(s_{t}, a)$ .  $Q(s_{t}, a)$  is an average reward over all rounds that have traversed the associated edge. It is based on actual accumulated rewards during tree traversal and the value estimates of leaf states (described below).  $U(s_{t}, a) = c * p_{\theta}(a|s) * \frac{\sqrt{N(s_{t})}}{1 + n(s_{t}, a)}$  is a term that encourages exploration, where  $n(s_{t}, a)$  is the number of times action  $a$  has been selected from state  $s_{t}$ ,  $N(s_{t})$  is the total number of times state  $s_{t}$  has been visited, and  $c$  is a scalar hyperparameter that we set to 0.1. Following Silver et al. (2017), we use the policy network to bias this exploration term. To constrain the search, we only consider the top-  $k$  actions according to the policy network, where  $k = 8$  in our experiments.
If we select an action  $a_{t}$  for state  $s_{t}$  which has never been previously selected from  $s_{t}$ , then the simulation ends and we add a new leaf state,  $s_{L} = f(s_{t},a)$ , to the search tree. If  $s_{L}$  is not a terminal state, then we estimate its value (i.e. future returns) using both the value network and a rollout with the greedy policy. Specifically, following Silver et al. (2017), we estimate its value as  $\lambda *\hat{v}_{\phi}(s_L) + (1 - \lambda)*v^{\pi_\theta}(s_L)$  where  $v^{\pi_{\theta}}(s_L)$  is equal to the actual returns from following the policy  $\pi_{\theta}$  starting at  $s_{L}$  for a maximum of 20 steps, with actual returns clipped to a minimum value of 0. Is there  $\lambda$  is a mixing parameter that we set to 0.1. For challenging environments, rollouts may be unlikely to find a terminal state with positive reward, and in such cases rollouts may not be very informative. On the other hand, the value network can provide poor value estimates for certain states, especially if they are not well represented in the human demonstrations. By combining both methods we aim to provide a better approximation of the value of leaf states. Returns are propagated up the tree to each parent  $s'$  to update  $Q(s',a)$ . As  $Q(s_{L},a)$  is undefined prior to selecting  $a$  from  $s_{L}$  for the first time, we initialize  $Q(s_{L},a)$  for each action to be equal to the initial value estimate of  $s_{L}$  plus  $\alpha_{s}$ .