SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Peter Shaw $^{1*}$
Mandar Joshi\*
James Cohan
Jonathan Berant<sup>1</sup>
Panupong Pasupat<sup>1</sup>
Hexiang Hu
Urvashi Khandelwal
Kenton Lee
Kristina Toutanova<sup>1</sup>
<sup>1</sup> Google DeepMind
2 Google
Much of the previous work towards digital agents for graphical user interfaces (GUIs) has relied on text-based representations (derived from HTML or other structured data sources), which are not always readily available. These input representations have been often coupled with custom, task-specific action spaces. This paper focuses on creating agents that interact with the digital world using the same conceptual interface that humans commonly use — via pixel-based screenshots and a generic action space corresponding to keyboard and mouse actions. Building upon recent progress in pixel-based pretraining, we show, for the first time, that it is possible for such agents to outperform human crowdworkers on the MiniWob++ benchmark of GUI-based instruction following tasks.
Systems that can follow instructions to complete tasks through graphical user interfaces (GUIs) can help automate tedious tasks, improve accessibility, and expand the usefulness of digital assistants by allowing them to interact with tools and services. Despite the visual nature of GUIs, prior work has primarily focused on utilizing structured representations of the user interfaces (such as HTML sources, Document Object Model (DOM) trees, and Android view hierarchies) as well as custom, task-specific representations of high-level actions based on these structured representations (see §6). Recent efforts have achieved positive outcomes thanks to the advances of powerful language models (Gur et al., 2022; Kim et al., 2023; Yao et al., 2022).
While structured and task-specific representations may be useful, they are not always available – some examples are web applications that use extensive scripting, sandboxed environments where access to DOM is limited, and mobile applications which often do not expose the underlying structure to external modules. Even when structured application source data is available, it may be hard to interpret due to obfuscation and misalignment with what actually appears on the GUIs. Finally, aligning human demonstrations with task-dependent actions is often challenging.
In contrast, people interact with GUIs by perceiving the visual input and using generic mouse and keyboard actions, without needing to inspect the application's source code for cues on its functionality. They can quickly learn to interact with new applications that offer familiar visual interfaces, regardless of differences in implementation technologies. In this paper we ask: Can we build an agent that can
Figure 1: Our agent learns to follow instructions via Graphical User Interfaces (GUIs). Unlike most prior work studying instruction following for GUI-based tasks, our agent does not rely on text-based observations corresponding to DOM trees or HTML source code, or task-specific actions. Instead, our agent receives pixel-based observations and generates outputs corresponding to mouse and keyboard actions. The possible actions are encoded as text and shown on the top of the figure. We show examples of observations from various episodes for two benchmarks, MiniWob++ (top row) and WebShop (bottom row), that we adapt to study within the context of our general Chrome-based environment framework. See details in §2.
complete tasks for users while relying solely on pixel-level visual representations of the GUI state, and generic low-level actions?
Learning based on pixel-only inputs proved effective for game playing environments such as Atari (Mnih et al., 2015). However, for GUI-based instruction following tasks, learning from pixel-only inputs coupled with general low-level actions leads to several challenges. Interpreting GUIs visually requires understanding the interface layout, recognizing and interpreting visually-situated natural language, identifying visual elements, and predicting their functions and methods of interaction. A generic action space also poses the challenge of a more complex mapping between high-level textual instructions and corresponding sequences of low-level actions. As an example of the increased difficulty in this setting, on the MiniWob++ benchmark (Shi et al., 2017; Liu et al., 2018) of web GUI interaction, CC-Net (Humphreys et al., 2022) demonstrates human-level accuracy when accessing both screenshots and DOM structure, but its performance drops by  $75\%$  when the DOM information is removed from the agent's observations.
Here we present  $\mathrm{PIX2ACT}$ , a model that relies solely on pixel-based screenshots as input and selects actions corresponding to basic mouse and keyboard functionalities. We build on  $\mathrm{PIX2STRUCT}$  (Lee et al., 2022), a Transformer-based (Vaswani et al., 2017) image-to-text model pre-trained to map
$^{2}$ Code and models are available at https://github.com/google-deepmind/pix2act.
screenshot to structured representations derived from HTML on web-scale data. Pix2ACT tunes this model using a combination of human demonstrations and environment interactions, applying tree search to iteratively generate new expert trajectories for training. We develop a general browser-based environment framework, and adapt two benchmark datasets, MiniWob++ and WebShop (Yao et al., 2022), to our setting with a unified, general purpose observation and action format.
On MiniWob++, PIX2ACT outperforms human crowdworkers and improves task score nearly 4x compared to the best prior results in our proposed setting (CC-Net without DOM). Ablations show that a key ingredient for PIX2ACT's performance is the pixel-based pre-training of PIX2STRUCT.
1. We show, for the first time, that an agent using pixel-only inputs and a generic action space can outperform human crowdworkers on the MiniWob++ benchmark, significantly improving over prior work on this setting, and reaching performance comparable to that of state-of-the-art agents that access DOM information and use a comparable number of human demonstrations.
2. We adapt the WebShop benchmark to our setting, using pixel-based observations and general low-level actions. We establish the first baseline on this setting, although there is still a performance gap relative to larger language models using HTML-based inputs and task-specific actions.
3. We show that Pix2STRUCT's pre-training via screenshot parsing is effective for GUI-based instruction following with pixel-based inputs. In the behavioral cloning setting, pre-training improves task scores from 17.1 to 66.5 on MiniWob++ and from 1.1 to 46.7 on WebShop.
4. We demonstrate the successful application of tree search as a relatively simple method for policy improvement for MiniWob++.
2 Environment
Following the reinforcement learning literature, we model GUI interaction as a Markov Decision Process (MDP): at each time step, our agent receives an observation and selects an action. We develop a common environment framework with shared observation and action formats for browser-based tasks. Similarly to prior work on web-based agents (Liu et al., 2018), we use Selenium to programmatically interact with the Google Chrome browser.
Observations To form an observation, we first take a screenshot of the current browser window using Selenium and then augment it with additional information. First, if not already present, we render the natural language instruction on the top of the screenshot, following Lee et al. (2022). Second, as Selenium screenshots do not include cursors (which are typically rendered by the operating system), we draw a cursor on the screenshot to indicate the mouse pointer position. Finally, we render an indicator of whether the mouse button is currently pressed down, which is useful for dragging actions.
Actions Our action space consists of raw mouse and keyboard actions, as shown in Figure 1, where X and Y refer to discrete coordinate bins, K is one or more keys, M is an optional modifier key such as "shift", and Z refers to a vertical scroll amount, also represented as a discrete bin. The begindrag and enddrag actions can be used to execute "click and drag" actions. We use a configurable number of coordinate buckets per vertical and horizontal axis. Importantly, the DOM information is not provided by the environment and is therefore not used in any way to define observations or actions.
Episodes and Rewards Episodes continue until a terminal state or a configurable number of maximum steps is reached. For the environments we consider, the agent only receives a reward at a terminal state. This can be a binary reward based on whether the task was completed successfully or a partial reward based on how well the task was completed.
<sup>3</sup>We chose discrete bins because they enable a simple encoding of actions as tokens. Alternatives could include continuously-valued coordinates or relative movements with foveated binning (Baker et al., 2022).
Figure 2: An example episode of our agent on the MiniWob++ use-colorwheel-2 task. At each step, the agent receives a new observation and outputs the next action to take. The screenshots include a rendered instruction that the agent needs to follow to successfully complete the episode. For MiniWob++, we use 32 vertical and horizontal coordinate bins to specify locations. We show the click location visually for this figure.
3 Proposed Agent
Our agent, Pix2ACT, is based on the Pix2STRUCT model (Lee et al., 2022), which uses an image Transformer encoder and a text Transformer decoder. The architecture is based on Vision Transformer (Dosovitskiy et al., 2021) and T5 (Raffel et al., 2020). Pix2STRUCT is pre-trained on a screenshot parsing task: predicting simplified HTMLs from screenshots with visually-masked regions. Such pre-training was proven effective for tasks related to understanding user interfaces in a non-interactive setting, such as screen summarization and widget captioning (Wang et al., 2021; Li et al., 2020b). We use the Pix2STRUCT base variant with 282M parameters (12 encoder and 12 decoder layers; hidden size 768) for all our experiments. The model is called once per time step.
Input The only input to the model is pixel-based observation from the environment. We can also condition on multiple previous observations by concatenating multiple frames. In preliminary experiments, we did not observe significant gains from conditioning on past observations for MiniWob++, and thus we only use the screenshot of the current step in our experiments. We reuse Pix2STRUCT's image processing by scaling input images up or down so as to extract the maximal number of fixed-size patches that still fit within the sequence length limit. We use resolutions of  $160 \times 210$  and  $800 \times 600$  for MiniWoB++ and WebShop, respectively.
Output We encode actions as text tokens, which are predicted autoregressively by the Transformer decoder. We use beam search over tokens to output the  $k$ -best actions (see Appendix B.1 for details).
Greedy Policy For interacting with the environment, we adopt a standard greedy policy, selecting the highest scoring action at each step, with one modification. To help prevent the agent from getting stuck in cycles, we track which actions have been taken for a given observation, and select the highest probability action in the beam that has not previously been taken given the current observation, which provides a modest increase in performance.
We explore two methods for training models to follow instructions via GUIs. First, similarly to prior work, we use Behavioral Cloning (BC), where we train our model using standard supervised learning to predict the given action for each observation in a set of human demonstrations. Second, given access to environments with reward signals, prior work has also explored Reinforcement Learning (RL) to further improve agent performance. As an alternative to common reinforcement learning algorithms such as REINFORCE (Williams, 2004) and PPO (Schulman et al., 2017), we apply tree search as a simple method for policy improvement.
Tree Search For a given set of model parameters, tree search leverages the deterministic nature of the environment to look ahead at the consequences of possible actions to determine a more optimal policy than greedily selecting actions.
We adopt Monte Carlo Tree Search (MCTS) (Coulom, 2006), which outperformed more naive search algorithms in initial experiments, and has been successfully integrated with neural network policies in prior work (Silver et al., 2017; Anthony et al., 2017). Similarly to this prior work, we train a model to estimate a value function, which predicts the value (i.e., estimated future rewards) of a given state. We use a surrogate reward which penalizes the number of steps taken to encourage concise trajectories without unnecessary actions. We implement this value function approximator using the same Pix2STRUCT architecture used for our agent. However, instead of predicting actions, this model predicts state-values mapped to discrete buckets. To estimate the value of leaf states during MCTS, we use a combination of this value function approximator and rollouts using our greedy policy, similarly to Silver et al. (2017). See Appendix B for additional technical details.
We can then use successful episodes found with this stronger tree search policy to improve our model. As this stronger model then yields a more effective tree search policy, we can continue to iteratively improve our model using this method. Notably, this approach requires no modifications to the fine-tuning procedure of  $\mathrm{PIX2ACT}$ , as, for simplicity, we tune on episodes from the tree search policy using standard supervised learning.
4 Benchmarks and Demonstrations
We adapt two benchmarks, MiniWob++ and WebShop, to our environment framework (§2) which consists of pixel-based observations and generic low-level actions. We also map previously collected human demonstrations for these benchmarks to our observation and action spaces.
4.1 MiniWob++
MiniWob++ (Liu et al., 2018) is a set of over a hundred web-browser based tasks. See Figures 1 and 2 for task examples. Each task consists of an algorithm for generating variations of the task and an instruction template, controlled by a random seed, with up to billions of possible configurations per task. The task instruction is given as (mostly) natural language text in the top yellow part, which in our framework can only be accessed visually. An automatic reward is given at the end of the task.
Human Demonstrations We use the human demonstrations collected by Humphreys et al. (2022). However, their demonstrations were collected using an X11-based environment, which is different from our Selenium-based environment. This results in different renderings of the same underlying environment state, introducing a shift between the screenshots seen during training and those observed at test time. Additionally, we need to map from their real-time X11-based action sequences to our action space. We were able to perform this mapping with a reasonable degree of success for 59 tasks. Notably, not all behaviors in the human demonstrations are supported in our Selenium-based environment. For example, Selenium does not implement the ability to highlight text and drag it into a text field, and such an action is widely used in the human demonstrations for tasks where text is copied and pasted. Additionally, while our environment framework intends to cover the basic functionality of most web interfaces, aspects of some MiniWob++ tasks, such as capturing real-time observations for animated elements, are not supported. See Appendix A for additional details.
Starting with approximately 1.3 million demonstrations across the 59 supported tasks, we filtered demonstrations with a reward of  $< 0.8$ , or approximately  $6\%$  of demonstrations. We were able to successfully convert  $81\%$  of the remaining demonstrations to our action space. We reserve  $10\%$  of the data for a development set. Demonstrations contain approximately 3 steps per task on average, although this varies considerably across tasks.
4While it may be more efficient to share an encoder between these two Pix2STRUCT-based models that condition on the same inputs, we trained separate models for simplicity.
Other prior work has used the demonstrations from Liu et al. (2018), which cover a different subset of MiniWob++ tasks. However, these demonstrations do not include screenshots or sufficient information to replay the episodes in a browser environment to collect new screenshots, and therefore cannot be applied to our setting.
Figure 3: Main results evaluating P1x2ACT (ours) on MiniWob++ (left) and WebShop (right). In this paper we focus on approaches that do not have access to DOM or HTML information, and recieve pixel-based observations (blue). On this setting, P1x2ACT significantly improves over prior work on MiniWob++ and establishes the first baseline on WebShop. Our method performs competitively with humans (green) and with methods that have access to DOM or HTML information (red) on MiniWob++, although there is a gap with the best performing methods that access HTML on WebShop (see §5.3 for detailed analysis).
Evaluation We report the mean score across seeds and tasks. The score is the MiniWob++ raw reward (without time decay) mapped from the original range  $[-1, 1]$  to the range  $[0, 100]$ . The score is equivalent to the success rate (i.e., the proportion of episodes in which the agent receives a positive reward) for tasks with binary rewards. For episodes that do not complete due to reaching a maximum number of allowed steps, we assume a score of 0. For each task, we compute the mean over 100 random seeds, and then compute the mean over 59 MiniWob++ tasks.
4.2 WebShop
WebShop (Yao et al., 2022) is a web-based shopping environment with over 1.1 million products from Amazon. The task is to find and purchase a product based on a human-authored text instruction. Finding a suitable product requires entering search queries, clicking on results, and determining the relevance of various products to the instruction. An automatic reward is computed based on similarity between the purchased product and the gold target product.
Human Demonstrations We use the 1,566 human demonstrations (with a train/development/test split of 1012/54/500) collected in Yao et al. (2022). As with the MiniWob++ demonstrations, we need to map between the observation and action sequences used in their setup to our framework. Yao et al. (2022) used high-level actions (e.g. "search" or "click [item]", each of which could map to multiple lower-level actions in our environment. Specifically, for all actions involving a mouse click, we determine the coordinates of the center of the corresponding HTML element. For WebShop, the entire screen content is not always visible due to page heights exceeding the viewport dimensions. If the clicked element lies outside the visible area, we add scroll actions until the element is visible. Finally, we map search actions to two actions in our environment: clicking on the center of the search box and entering the search query followed by the enter key. We render the HTML inputs in the human demonstrations using our browser to obtain screenshots. Additionally we found that rendering the last 5 actions (separated by  $
$ ) on top of the screenshot to be helpful.
Evaluation Consistent with previous work, we report Task Score, which is the average reward across 500 test instructions.
5 Experiments and Analysis
We updated all model parameters during fine-tuning, including both the image encoder and text decoder. We used the Adafactor optimizer (Shazeer and Stern, 2018) with a learning rate of 0.01.
MiniWoB++ We finetuned a single model jointly on episodes from all tasks for a total of 26K steps using a batch size of 512, input/output sequence lengths of 512/16. We also evaluated using the tree search procedure described in §3.1 to improve our agent. We performed 2 iterations of policy
Figure 4: Comparing scores on MiniWob++ tasks of Pix2ACT (blue) with human crowdworkers (green), ranked from left to right by the relative difference in performance.
improvement with tree search, collecting a total of 826K episodes across all tasks, and tuning for a further 26K steps.
WebShop We used only the provided human demonstrations to train our model. Due to its larger resolution and text-heavy data, we used a higher input sequence length of 4096. We also found it useful to perform intermediate finetuning on MiniWoB++, followed by 10K steps of further finetuning on WebShop using a batch size of 256 (see §5.3 for details).
We report the results of our models on MiniWob++ and WebShop in Figure 3. For MiniWob++, we also provide task-level comparisons between P1x2ACT and human crowdworkers in Figure 4. There is limited prior work studying these tasks without access to DOM and HTML information. For MiniWob++, the only comparable baselines are from the CC-Net model of Humphreys et al. (2022), which mentions an ablation experiment where performance dropped by  $75\%$  from their primary results when the models conditioned on only screenshots without DOM information. As they did not provide per-task numbers for this ablation, we estimate the performance of CC-Net without DOM information by assuming that the drop in performance on the subset of tasks we study was also  $75\%$ . Regardless, it is clear that P1x2ACT significantly outperforms CC-Net on this setting. The difference in performance can be largely attributed to the screenshot parsing pre-training of Lee et al. (2022). For WebShop, there is no prior work exploring such a setting, so we establish the first baseline.
5.3 Ablations and Analysis
Pre-training ablations To study the impact of the pre-training on our model's ability to effectively learn to follow instructions via GUIs, we evaluate model performance without the pre-training procedure. For these experiments, we only compared performance of models trained using behavioral cloning. The results are shown in Figure 3, and demonstrate that pre-training is critical for our model's performance.
Comparison with models that use DOM or HTML as input We can also compare our results without access to DOM or HTML to previous methods that utilized these resources, including those which also leverage DOM information to construct specialized action spaces. The performance of the best model from prior work leveraging DOM or HTML information is shown in Figure 3.
For MiniWob++, the best model on this setting is CC-Net (Humphreys et al., 2022) trained with BC and RL and with access to both DOM and pixel-based observations.  ${}^{7}\mathrm{Pix2ACT}$  achieves comparable performance to their best model, while relying on only a subset of the information used by CC-Net, and using a comparable number of human demonstrations for training.  $\mathrm{Pix2ACT}$  also outperforms
<sup>6</sup>We did not explore applying RL techniques to WebShop in this work. Prior work (Yao et al., 2022) has not shown as significant an advantage to applying RL on WebShop relative to the large improvements shown by prior work on MiniWob++, which offers a near limitless variety of environments with reward signals for training.
<sup>7</sup>We compute mean scores for CC-Net by averaging their reported per-task results over the 59 tasks we study.
Pre-training	Included	Heldout
Yes	65.5	28.3
No	11.0	7.6
Table 1: We selected 9 MiniWob++ tasks and evaluated mean scores when they are heldout from the training set. Pretraining leads to non-trivial generalization (28.3) to held out tasks that were unobserved at training time compared to a randomly initialized model (7.6). We also include scores when the tasks are included during training for reference.
Policy	Iteration
0	1	2
Greedy	66.5	93.1	96.2
Tree Search	91.7	98.4	—
Table 2: We compare average MiniWob++ scores using the greedy policy with one that uses tree search and lookahead, given the same underlying model. The model is initially trained on human demonstrations and iteratively improved by training on episodes generated by the tree search policy.