From Pixels to UI Actions: Learning to Follow Instructions via Graphical User Interfaces
Authors: Peter Shaw
Mandar Joshi
James Cohan
Jonathan Berant
Panupong Pasupat
Hexiang Hu
Urvashi Khandelwal
Kenton Lee
Kristina Toutanova
Venue: NeurIPS 2023 spotlight (2023)
Keywords: instruction following
web tasks
user interface tasks
vision and language
representation learning
reinforcement learning
imitation learning
tree search
language grounding
web agents
computer control
TL;DR: We study GUI-based instruction following with a general observation and action space consisting of pixel-based inputs and low-level actions.

Abstract: Much of the previous work towards digital agents for graphical user interfaces (GUIs) has relied on text-based representations (derived from HTML or other structured data sources), which are not always readily available. These input representations have been often coupled with custom, task-specific action spaces.  This paper focuses on creating agents that interact with the digital world using the same conceptual interface that humans commonly use — via pixel-based screenshots and a generic action space corresponding to keyboard and mouse actions. Building upon recent progress in pixel-based pretraining, we show, for the first time, that it is possible for such agents to outperform human crowdworkers on the MiniWob++ benchmark of GUI-based instruction following tasks.

From Pixels to UI Actions: Learning to Follow Instructions via Graphical User Interfaces
Peter Shaw!*
Mandar Joshi
James Cohan?
Jonathan Berant
Panupong Pasupat
Hexiang Hu-
Urvashi Khandelwal'
Kenton Leek
Kristina Toutanova'
1 Google DeepMind
Google
Abstract
Much of the previous work towards digital agents for graphical user interfaces (GUIs) has relied on text-based representations (derived from HTML or other structured data sources), which not always readily available. These input representations have been often coupled with custom, task-specific action spaces. This paper focuses creating agents that interact with the digital world using the same conceptual interface that humans commonly use via pixel-based screenshots and a generic action space corresponding keyboard and mouse actions. Building upon recent progress in pixel-based pretraining, we show, for the first time, that is possible for such agents outperform human crowdworkers on the MiniWob++ benchmark of GUI-based instruction following tasks.
Introduction
Systems that can follow instructions to complete tasks through graphical user interfaces (GUIs) can help automate tedious tasks, improve accessibility, and expand the usefulness of digital assistants by allowing them to interact with tools and services. Despite the visual nature of GUTs, prior work has primarily focused on utilizing structured representations of the user interfaces (such as HTML sources, Document Object Model (DOM) trees, and Android view hierarchies) as well as custom, task-specific representations of high-level actions based on these structured representations (see $6). Recent efforts have achieved positive outcomes thanks to the advances powerful language models (Gur et al. 2022 Kim et al. 2023 Yao et al. (202)).
While structured and task sask-specific representations may be useful, they are always available some examples are web applications that use extensive scripting, sandboxed environments where access DOM limited, and mobile applications which often do not expose the underlying structure to external modules. Even when structured application source data is available, it may be hard to interpret due obfuscation and misalignment with what actually appears on the GUIs. Finally, aligning human demonstrations with task des-deeendent actions is often challenging.
In contrast, people interact with GUIs by perceiving the visual input and using generic mouse and keyboard actions, without needing to inspect the application's is source code for cues on its functionality. They can quickly learn interact with new applications that offer familiar visual interfaces. regardless of differences in implementation technologies. In this paper we ask: Can we build an agent that can
Equal contribution.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Possible
click
bagin_drag
and
scroll
Text Decoder
Perform Action
Chrome-based Environment
Pix2Act Agent
Image Encoder
Render Screenshot
Example Observations
Seleet 10/03/2016 the date subrret
Seleet wonts smirer sime. perume and dlck Butret
Belest al the shadess of grwer and press Submel
d smaller box thel infppekey mosee large lase.
emeil Courteey and the shar eer mark
Primary
Vovember 2014
miterable
Joanna
Shanoux
Courtney R
Submit
Rous suctor
Submit
B
Submit
Fougitt Stt
margarita do padlicta made with cily natural ngrodionts, and than 140.00
mangarit
packilaa made with only natural ingredients, and price than 140
dallar
WebShop
paciabs any nakure irpediant,
Baid saca aneetandase guaparta
Seachh
Original Margarita Mix Singles (8 Single-Serve Packets Per Box )- The Original Zero Sugar. Low Calorie. Low Carb, Keto Friendly, Skinny Cocktail Mixer. (3-Pack Makes 24 Individual Cocktails)
Reding
Figure 1: Our agent learns to follow instructions via Graphical User Interfaces (GUIs). Unlike most prior work studying instruction following GUI-based tasks, our agent not rely on text-basee observations corresponding to DOM trees or HTML. source code, or task-specific actions. Instead, our agent receives pixel- based ohservations and generates outputs corresponding to mouse and keybnard actions. The possible actions are encoded as text and shown the top of the figure. We show examples of observations from various episodes for benchmarks, Mini MobiWob++ (top row) and WehShop (bottom row), that we adapt study within the context of our general Chrome-based environment framework. details $2
complete tasks for users while relying solely on pixel-level visual representations of the GUI state, and generic low-level actions?
Learning based on pixel-only inputs proved effective for game playing environments such as Atari (Mnih al. 2015). However, for GUT-based instruction following tasks, learning from pixel-only inputs coupled with general low-level actions leads to several challenges. Interpreting GUIs visually requires understanding the interface layout, recognizing and interpreting visually-situe -istuley-stuuaed natural language, identifying visual elements, and predicting their functions and methods of interaction. generic action space also poses the challenge of more complex mapping between high-level textual instructions and corresponding sequences of low-level actions. As an example of the increased difficulty in this setting, on the Mini Wob++ benchmark (Shi et 2017 Liu al. 2018) of web GUI interaction, CC-Net (Humphreys et al. (2022) demonstrates human- level accuracy when accessing both screenshots and DOM structure, but its performance drops by 75% when the DOM information is removed from the agent's observations.
Here present PIX2ACT, model that relies solely pixel-based screenshots as input and selects actions corresponding to basic mouse and keyboard functionalities! We build on PIX2STRUCT (Lee al. 2022), Transe (Vaswani al. 2017 image-to-text model pre-trained map
"Code and models available httapp://gittuu.coo////e// http:///oooooo.oo.oooe/////a/aaa

---

screenshots to structured representations derived from HTML on web-scale data. PIX2ACT tunes this model using a combination of human demonstrations and environment interactions, applying tree search to iteratively generate new expert trajectories for training. We develop a general browser-based environment framework, and adapt two benchmark datasets, MiniWob++ and WebShop (Yao et al.. 2022). to our setting with unified, general purpose observation and action format.
On MiniWob++, PIX2ACT outperforms human crowdworkers and improves task score nearly 4x compared to the best prior results in our proposed setting (CC-Net without DOM), Ablations show that a key ingredient for PIx2ACT's performance is the pixel-based pre-training of PIX2STRUCT.
Our contributions are as follows:
1. We show, for the first time, that agent using pixel-only inputs and generic action space can outperform human crowdworkers on the MiniWobee benchmark, significantly improving over prior work on this setting, and reaching performance comparable that of state-of-the-ar agents that access DOM information and use a comparable number of human demonstrations.
2. We adapt the WebShop benchmark to our setting, using pixel-based observations and general low- level actions. We establish the first baseline on this setting, although there is still performance gap relative to larger language models using HTML-based inputs and task-specific actions.
3. We show that PIX2STRUCT's pre-training via screenshot parsing is effective for GUI-based instruction following with bxxel-baedd inputs. In the behavioral cloning setting, pre-training improves task scores from 17.1 to 66.5 on MiniWob++ and from 1.1 46.7 on WebShop.
We demonstrate the successful application of tree search as a relatively simple method for policy improvement for MiniWob++.
2 Environment
Following the reinforcement learning literature, we model GUI interaction as a Markov Decision Process (MDP): at each time step, our agent receives an observation and selects an action. We develop a common environment framework with shared observation and action formats for browser- based tasks. Similarly to prior work on web-based agents (Liu et al. 2018), we use Selenium to programmatically interact with the Google Chrome browser.
Observations To form an observation, we first take a screenshot of the current browser window using Selenium and then augment it with additional information. First, if not already present, we render the natural language instruction on the top of the screenshot, following Lee et al. (2022). Second, Selenium screenshots do not include cursors (which are typically rendered by the operating system), we draw a cursor on the screenshot to indicate the mouse pointer position. Finally, we render an indicator of whether the mouse button is currently pressed down, which is useful for dragging actions.
Actions Our action space consists of raw mouse and keyboard actions, as shown in Figure I where X and Y refer to discrete coordinate bins, K is more keys, is optional modifier key such as "shift", and refers to a vertical scroll amount, also represented as a discrete bin 3 The begin_drag and mnd_ actions can be used execute "click and drag" actions. We configurable number of coordinate buckets per vertical and horizontal axis. Importantly, the DOM information is not provided by the environment and is therefore not used in any way to define observations or actions.
Episodes and Rewards Episodes continue until a terminal state or a configurable number of maximum steps is reached. For the environments we consider, the agent only receives a reward at terminal state. This can be binary reward based on whether the task was completed successfully or a partial reward based on how well the task was completed.
We chose discrete bins because they enable simple encoding of actions as tokens. Alternatives could include continuously vatinuously-valludd valued coordinates relative movements with foveated binning (Baker et al. 2022).
3

---

Step
2
3
4
5
Observation
9Jamnl
Subriit
click 23 12
click 12 20
click 29 17
click 30 12
click 14 19
Action
Figure 2: An example episode of agent the MiniWob++ use-colorwheeli task. At each step. the agent receives a new observation outputs the next action to take. The screenshots include rendered instruction that the agent needs to follow to successfully complete the episode. For MiniWob++,, we use 32 vertical and horizontal coordinate bins to specify locations. We show the click location visually for this figure.
3 Proposed Agent
Our agent, PIX2ACT, is based on the PIX2STRUCT model (Lee et al. 2022). which uses an image Transformer encoder and former (Dosovitskiy et al. 202I) and T5 (Raffel et al. 2020). PIX2STRUCT is pre-trained on screenshot parsing task: predicting simplified HTMLs from screenshots with Aisually-masker re- gions. Such pre-training was proven effective for tasks related to understanding user interfaces in non-interacttvv setting, such as screen summarization and widget captioning (Wang al., [02T] Li et al. 2020b). We use the PIX2STRUC base variant with 282M parameters (12 encoder and T2 decoder layers; hidden size 768) for all our experiments. The model is called once per time step.
Transformer decoder. The architecture based Vision Trans-
The only input to the model is pixel- pixel-based observation from the environment. We can also condition on multiple previous observations by concatenating multiple frames. In preliminary experi- ments, we did not observe significant gains from conditioning on past observations for WiniWob++ and thus we only use the screenshot of the current step in our experiments. We reuse PIX2STRUCT's image processing by scaling input images up or down as to extract the maximal number of fixed-size patches that still fit within the sequence length limit. We use resolutions 160x210 and 800x 600 for MiniWoB++ and WebShop, respectively.
Input
Output We encode actions as text tokens, which are predicted autoregressively by the Transformer decoder. We use beam search over tokens to output the k-bhest actions (see Appendix B.T for details).
Greedy Policy For interacting with the environment, we adopt a standard greedy policy, selecting the highest scoring action at each step, with one modification. To help prevent the agent from getting stuck in cycles, we track which actions have been taken for a given observation, and select the highest probability action in the beam that has not previously been taken given the current observation, which provides a modest increase in performance.
3.1 Training
We explore two methods for training models to follow instructions via GUIs. First, similarly to prior work, we use Behavioral Cloning (BC), where we train our model using standard supervised learning to predict the given action for each observation in a set of human demonstrations. Second, given access to environments with reward signals, prior work has also explored Reinforcement Learning (RL) to further improve agent performance. As alternative to common reinforcement learning algorithms such as REINFORCE (Williams) 2004) and PPO (Schulman et al. 2017), we apply tree search simple method for policy improvement.

---

Tree Search For a given set of model parameters, tree search leverages the deterministic nature of the environment look ahead the consequences of possible actions to determine more optimal policy than greedily selecting actions.
We adopt Monte Carlo Tree Search (MCTS) [Coulom 2006), which outperformed more naive search algorithms in initial experiments, and has been successfully integrated with neural network policies in prior work (Silver et al., 2017 Anthony et al., 2017). Similarly to this prior work, we train model to estimate a value fumction, which predicts the value (i.e., estimated future rewards) of a given state. We use surrogate reward which penalizes the number of steps taken to encourage concise trajectories without unnecessary actions. We implement this value function approximator using the same PIX2STRUCT architecture used for our agent I However, instead of predicting actions, this model predicts state-values mapped to discrete buckets. To estimate the value of leaf states during MCTS, we use a combination of this value function approximator and rollouts using our greedy policy, similarly to (Silver et al. (2017). See Appendix B for additional technical details.
We can then use successful episodes found with this stronger tree search policy to improve our model. As this stronger model then yields more effective tree search policy, we can continue to iteratively improve our model using this method. Notably, this approach requires no modifications to the fine-tuning procedure of PIX2ACT, as, for simplicity, we tune on episodes from the tree search policy using standard supervised learning.
4 Benchmarks and Demonstrations
We adapt two benchmarks, MiniWob++ and WebShop, to our environment framework ($2) which consists of pixel-based observations and generic low-level actions. We also map previously collected human demonstrations for these benchmarks to our observation and action spaces.
4.1 MiniWob++
MiniWob++ (Liu et al. 2018) set of over a hundred web-browser based tasks. See Figures and for task examples, Each task consists of an algorithm for generating variations of the task and an instruction template, controlled by random seed, with up to billions of possible configurations per task. The task instruction is given as (mostly) natural language text in the top yellow part, which in our framework can only be accessed visually. An automatic reward is given the end of the task.
Human Demonstrations We use the human demonstrations collected by Humphreys et al. (2022). However, their demonstrations were collected using an XI I-based environment, which different from our Selenium- environment. This results in different renderings of the same underlying environment state, introducing a shift between the sereenshots seen during training and those observed at test time. Additionally, we need to map from their real-time XI 1-baseed action sequences to our action space. We were able to perform this mapping with a reasonable degree of success for 59 tasks. Notably, not all behaviors in the human demonstrations are supported in our Selenium-base environment. For example, Selenium does not implement the ability to highlight text and drag into a text field, and such an action is widely used in the human demonstrations for tasks where text copied and pasted. Additionally, while our environment framework intends to cover the basic functionality of most web interfaces, aspects of some MiniWo+++ tasks, such as capturing real-time observations for animated elements, are not supported. See Appendix A for additional details $
Starting with approximately 1.3 million demonstrations across the 59 supported tasks, we filtered demonstrations with a reward of < 0.8, or approximately 6% of demonstrations. We were able to successfully convert 81% % of the remaining demonstrations to our action space. We reserve 10% of the data for a development set. Demonstrations contain approximately 3 steps per task on average, although this varies considerably across tasks.
*While it may be more efficient to share encoder between these two PIX2STRUCT- models that condition the same inputs, we trained separate models for simplicity.
'Other prior work has used the demonstrations from Liu et al. (2018), which cover a different subset of MiniWob++ tasks. However, these demonstrations do not include screenshots or sufficient information to replay the episodes in browser environment to collect screenshots, and therefore cannot be applied to setting.
5

---

MiniWob++ Mean Scores
WebShop Scores
100
94.4
100
66.6
07.5
46.7
24.1
26
25
Ours
Ours (BC only)
Ours (BC only pretrain.)
CG Not (no DOM)
Human
So wi DOM or HTML
Ours
Qurs
SoTA wi
(no pretrain.)
DOM
HTML
Figure 3: Main results evaluating PIx2ACT (ours) MiniWob++ (left) and WebShop (right). In this paper we focus on approaches that observations (blue). On this setting. Pix2ACT significantly improves over prior work on MiniWob++ and establishes the first baseline on WebShop. Our method performs competitively with humans (green) and with methods that have access to DOM HTML information (red) MiniWoh+++, although there gap with the best performing methods that access HTML. WebShop (see $5.3 for detailed analysis).
have access DOM or HTML information, and recieve pixel-based
Evaluation We report the mean score across seeds and tasks. The score is the Mini HoniWob+++ raw reward (without time decay) mapped from the original range 1- 1] to the range [0, 100]. The score is equivalent the success rate (i.e. the proportion of episodes in which the agent receives a positive reward) for tasks with binary rewards. For episodes that not complete due to reaching a maximum number of allowed steps, assume score of O. For each task, we compute the mean over 100 random seeds, and then compute the mean over 59 MiniWob+: tasks.
WebShop
WebShop (aao et al. 2022) is web-based shopping environment with over 1.1 million products from Amazon. The task to find and purchase product based on a human-authored text instruction. Finding a suitable product requires entering search queries, clicking results, and determining the relevance various products to the instruction. An automatic reward computed based on similarity between the purchased product and the gold target product.
Human Demonstrations We use the 1,566 human demonstrations (with a train/develoopmen/ttst split of 1012/54/500) collected in Yao al. (022). As with the MiniWob++ demonstrations, need to map between the observation and action sequences used their setup to our framework. Yao al. (2022) used high-level actions (e.g. "search" "click [item] "), each of which could map to multiple lower-level actions in our environment. Specifically, for all actions involving a mouse click, we determine the coordinates of the center of the corresponding HTML element. For WebShop, the entire screen content is not always visible due to page heights exceeding the viewport dimensions. If the clicked element lies outside the visible area, we add scroll actions until the element visible. Finally, we map search ch actions to two actions in our environment: clicking on the center of the search box and entering the search query followed by the enter key. We render the HTML inputs in the human demonstrations using browser to obtain screenshots. Additionally we found that rendering the last actions (separated by 
) on top of the screenshot to helpful.
Evaluation Consistent with previous work, we report Task Score, which is the average reward across 500 test instructions.
Experiments and Analysis
5.1 Training Details
We updated all model parameters during fine-tuning, including both the image encoder and text decoder. We used the Adafactor optimizer (Shazeer and Stern 2018) with a learning rate of 0.01.
MiniWoB++ We finetuned single model jointly episodes from all tasks for total of 26K steps using batch size 512, input/output sequence lengths 512/16. We also evaluated using the tree scarch procedure described in $3.1 improve our agent. We performed 2 iterations of policy

---

Crowewarkers are better
PiX2ACT beteer
100
0
clck che desesboooa dobaanos-- softt Dreaoain dhats
COUr d2 be
mecceee de
drag Hare gnd
BRN Heme des
click Chat AeaAAAaa AarIurr largu
SUPT Pee D
bibl
dete eaay
center
long
auctcccoo piolo
2 delets
ba importannt
2
cole
Shapp
timo
Humberr
m eedium Shorrt
arthiam matte
button
Figure 4: Comparing scores on MiniWob++ tasks of PIX2ACT (blue) with human crowdworkers (green), ranked from left to right by the relative difference in performance.
improvement with tree search, collecting a total of 826K episodes across all tasks, and tuning for further 26K steps.
WebShop We used only the provided human demonstrations to train our model O Due to its larger resolution and texi-heavy data, we used a higher input sequence length of 4096. We also found useful to perform intermediate finetuning MiniWoB++, followed by 10K steps of further finetuning on WebShop using a batch size of 256 (see $5.3 for details).
5.2 Main Results
We report the results of our models on MiniWob++ and WebShop in Figure For MiniWob++ we also provide task-level comparisons between Pix2ACT and human crowdworkers in Figure 4 There limited prior work studying these tasks without access to DOM and HTML information. For MiniWob++, the only comparable baselines are from the CC-Net model of Humphreys et al. (2022). which mentions an ablation experiment where performance dropped by 75% from their primary results when the models conditioned on only screenshots without DOM information. As they did not provide per-task numbers for this ablation, we estimate the performance of CC-Net without DOM information by assuming that the drop in performance on the subset of tasks we study was also 75%. Regardless, it is clear that PıX2ACT significantly outperforms CC-Net on this setting. The difference in performance can be largely attributed to the screenshot parsing pre-training of Lee al. (2022). For WebShop, there is no prior work exploring such a setting, so we establish the first baseline.
5.3 Ablations and Analysis
Pre-training ablations To study the impact of the pre-training on our model's ability to effectively learn to follow instructions via GUIs, we evaluate model performance without the pre-training procedure. For these experiments, we only compared performance of models trained using behavioral cloning. The results are shown in Figure and demonstrate that pre-training is critical for our model's 's performance.
Comparison with models that use DOM or HTML input We can also compare our results without access to DOM or HTML to previous methods that utilized these resources, including those which also leverage DOM information to construct specialized action spaces. The performance of the best model from prior work leveraging DOM or HTML information is shown in Figure 3
For Mini Wob++, the best model on this setting is CC-Net (Humphreys et al. 2022) trained with BC and RI. and with access to both DOM and pixel-based observations Pix2 ACT achieves comparable performance to their best model, while relying only subset of the information used by CC-Net, and using a comparable number of human demonstrations for training. PIX2ACT also outperforms
"We did explore applying RL techniques WebShop in this work. Prior work Yao et al. 2022 has not
shown as significant an advantage to applying RL on WebShop relative to large improvements shown by prior work on Mini Wob++, which offers near limitless variety of environments with reward signals for training. We compute scores for Net by averaging their reported per-task results over the 59 tasks study.
7

---

Pre-training
Included
Heldout
Iteration
Yes
65.5
28.3
Policy
2
No
11.0
7.6
Greedy
66.5
93.1
96.2
Tree Search
91.7
98.4
Table selected MiniWob++ tasks and eval- uated mean scores when they are heldout from the training set. Pretraining leads to non-trivial general- ization (28.3) to held out tasks that were unobserved randomly initialized
We compare average Mini scores using the greedy policy with that tree search and lookahead, given same underlying model. The model is initially trained on human demonstra- tions and iteratively improved by training on episodes generated by the tree search policy.
Table 2:
at training time compared model (7.6). We also include scores when the tasks are included during training for reference.
CC-Net when each model is trained only with behavioral cloning, as CC-Net performance on this setting drops to 38.7 (results not shown in the Figure). Notably, CC-Net scores also drop by approximately 10% when the model is not given access a dictionary of input strings provided by the environment. As shown in Figure [ the key to our model's s ability to achieve comparable performance without relying on DOM-based inputs is pixel-based is that CC-Net uses real time setting, which enables some forms of interaction not supported by our environment, and therefore can support larger set of WiniWob++ tasks. On the other hand, for BC, CC-Net does not need to handle the shift in rendering format and potentially noisy action space conversion.
pre-training Another difference
For WebShop, the best model on this setting is WebGUM (Furuta et al. the HTML source, a custom action space for the shopping domain, and -lan-T5-XL (Chung et al.) 2022) backbone. WebGUM outperforms PIx2ACT when compared on this setting. Some of this gap can be attributed to their simplified high-level action space, direct access to the relevant text on the page, and ability to transfer from Flan-T5's pretraining scale and instruction finetuning. Comparable improvements to the scale and pretraining of pixel-based models could reduce this gap.
2023a 3a which leverages
We discuss other approaches that leverage DOM or HTML information further in $6 We also offer a complete comparison across all MiniWob++ tasks in Appendix C
Evaluating transfer across tasks Training a pretrained, pixel- model interact with a GUT can intuitively lead to better generalization to new tasks that use common GUT design principles. To study this, we evaluate the ability of PIx2ACT (without RL) to generalize to tasks unseen during training. Specifically, we hold out 9 out of 59 tasks and train the remaining 50 We then evaluate performance on the held-out tasks, comparing initializing with PIX2STRUCT to random initialization. Table iillutratts that PIX2ACT can reach mean score of 28.3 on held out tasks compared to 65.5 when training on those tasks. Conversely, mean score is 7.6 when PIX2STRUCT initialization is not used. This shows that combining pretraining with a general GUI interface can lead to non-trivial generalization to held out tasks.
For WebShop, we find that finetuning directly on WebShop (without intermediate finetuning on MiniWoB++ as mentioned in (5.T) results in a drop of 4.0 in Task Score, demonstrating transfer learning benefits across these datasets.
Tree search analysis Table 2 shows the improvement in MiniWob++ scores by training on episodes generated by tree search. After an initial round of training on episodes generated by tree search, the effectiveness of tree search also improves due to improvements in the underlying model used to guide the search. The best greedy policy achieves performance close to the best tree search policy, but does not require access to reward signals additional exploration at inference time. Our results indicate that we could further improve performance with more iterations of policy improvement via tree search.
*We manually pick the 9 tasks to verify they include only actions or elements that would be reasonable to generalize to from the training tasks. The tasks are click- www.-chheeeeeeeeeeeeeeeeeee -aaree, click-color, click-tab- tab--ab22 click- -lick-tab-2 2aaad, count-shape. drag-shapes, use-color-whedd -2, s1ider-2.

---

Related Work
We focus on agents that interact with GUIs, such as operating system dialogs or web pages, to accomplish given task. Many early approaches relied on the structured information from the GUIs (Zettlemoyer and St. Amant 1999 Allen et al. 2007; Branavan et al. 2010). This information could range from a flat list of GUI components and their properties, to the full hierarchical structure of the components (e.g. the DOM tree). The output space also depends on this structured information, often using GUI components as action targets (e.g. clicking button #7). As discussed in I such structured information might not always be available, or might not align with what visually appears to the users.
When Shi et al. 2017) introduced the World of Bits tasks, which was the precursor to MiniWob++ (Liu et al. 2018), they proposed model based on a convolutional neural network that takes both visual and structured inputs and then performs generic low-level computer actions (e.g. clicking at a coordinate or pressing a key), similarly to PIX2ACT. However, the model performed poorly compared to humans. Follow-up work studied specialized architectures for incorporating structured DOM information and restricted the action space to clicking and typing predetermined texts on DOM elements (Liu et al., 2018 Gur al., 2018 Jia et al., 2019). Humphreys et al. (2022) reconsidered incorporating both visual and structured information as well as a low-level action space that aligns better to the human demonstrations. We discussed their approach, CC-Net, in $5.3 Humphreys et al.) (2022) also explored the benefits of large-scale human demonstrations, and we build on their work to utilize a large number of human demonstrations train PIX2ACT. This paper shows that PIX2ACT, a model with pixel-only inputs, can outperform humans on MiniWob++ and match the state-of-the-art approaches that rely on DOM information.
Automating web-based tasks using large language models (LLMs) has also been broadly explored. For instance, WebGPT uses text-based web browsing environment to search and navigate the web
(Nkkano et al. 202T). More relatedly, recent work has investigated prompting LLMs to produce agents that can generalize to tasks based on a small number of in-context examples. Yao et al. (2023) proposed ReAct, a few-shot prompted LLM, which uses observations derived from HTML and a custom action space make predictions based explicit reasoning steps. Similarly, Kim al. (0023 proposed RCI, a prompted LLM that iteratively critiques and refines its outputs, also using HTML inputs and custom action spaces. These approaches achieve competitive performance on WebShop and MiniWob++, respectively, and are extremely sample-efficient, relying on just a handful of demonstrations per task. Gur al. (2022) treated raw HTML as string and fed it to LLMs pretrained on natural language. After fine-tuning them on demonstrations, the models improved MiniWob++ task success rate and sample efficiency compared to models that take DOM-based inputs and specialized architectures. Finally. WebGUM (Furuta et al. 2023b), discussed in $5.3 extends HTML-based models to integrate a vision encoder pretrained on ImageNet-21K.
Other work has focused on tasks related to mobile apps. Li and Li (2022) considered a model with pixel-based inputs similar to that of Lee et al. (2022), and included evaluations on tasks related to grounding instructions to screenshots, but did not consider interactive environments. Some work
has considered instruction following tasks in mobile app environments (Li et al.) 2020a Burns et al.) 2022), but has generally not studied observation and action formats similar to ours, instead relying on inputs based on the Android view hierarchy. We focused web-based GUIs that we could use consistent environment framework for simplicity. Besides GUIs, several works on video game agents also considered visual-only input and low-level actions. For example, most works on Atari games used the screenshot as visual input and predicted the controller buttons to press (Mnih et al. 2015). More recently. [Baker et al. (2022), which focuses on learning from unlabeled videos, proposes an agent for Minecraft that uses pixel-based inputs paired with keyboard and mouse actions, similarly to PIX2ACT.
7 Limitations and Discussion
Pixel-based vs. text-based representations Text-based representations may be practically useful when available. especially since they enable transferring knowledge from LLMs, demonstrating impressive few-shot learning with LLMs for MiniWob++ (Kim et al. 2023) and WebShop (Yao) et al. 2023). When structured source is not available, OCR systems and models trained to predict the location and function of UI elements may also help connect models with the power of L.MMs. On the other hand, similar advances in scaling and pre-training of vision or multimodal models could

---

potentially enable similar capabilitiees in pixel-based setting in the future, as we have shown the effectiveness of pixel-based pre-training (albeit a smaller scale) for GUI-based tasks. Neverthe- less, beyond addressing the case where HTMI, or DOM information is unavailable, we hope our study contributes towards a better understanding of the potential of pixel-based representations for instruction following via GUIs.
Tree Search Our approach to policy improvement with tree search for MiniWob++ relied on the ability to procedurally generate new MiniWob++ environment and instruction variations and receive reward signals for task completion. Both aspects are unlikely be available for some real world environments, and such an approach might need to rely on generative models of potential instructions and approximate reward models for task completion (e.g. [Bahdanau al. (2018); Du al. (2023)) )) Our implementation also relied on the ability to reset the environment to an imitial state, useful feature for environments being used for exploration. Additionally, while we show that tree search can be sufficient to reach high performance on Mini Wob++, we did not perform detailed comparison relative to other search and RL algorithms in this study, which would be useful to better understand the most efficient approaches for learning from GUI-based environments.
Broader Impact In this paper have trained and evaluated models only in offline environments. Responsibly deploying models an environment where they can interact with online services would require additional considerations. Prior to enabling model to access important to sufficiently verify and/or constrain the behavior of the model to ensure that is consistent with the cemssoo--sevvic for that service and does otherwise cause harm. Ensuring sufficient data privacy could also be an important consideration for deploying models such as Pix2ACT that rely on capturing screenshots from browsers.
service, it would be
There would be many potential risks associated with deploying models that could interact with services in violation of their terms-of-service otherwise engage in various forms of spam, fraud, abuse. Examples of such behavior could include impersonating human users, generating harmful content spam, engaging in -enial-of-service attacks. Models that use the same conceptual interface humans use could potentially be more capable of breaking security defenses (e.g. solving CAPTCHAs) engaging in forms of spam, fraud, abuse that are more difficult to detect. It is therefore important for research related to security and techniques for detecting spam, fraud, and abuse to take such potential uses into account.
Acknowledgments
We would like thank Peter Humphreys, Toby Pohlen, and Gregory Thorton for their assistance with the MiniWob+4 demonstraions. We also thank Ming-Wes Wei Chang, Austin Huang, Luheng He, Tianze Shi, David Gaddy, Jacob Eisenstein, and Yi Luan for useful discussions and comments.
References
James F. Allen, Nathanael Chambers, George Ferguson, Lucian Galescu, Hyuckchul Jung, Mary D. Swift, and William Taysom. Plow: A collaborative task learning agent. In AAAI Conference on Artificial Intelligence, 2007.
Thomas Anthony, Zheng Tian, and David Barber. Thinking fast and slow with deep learning and tree search. Advances in neural information processing systems, 30, 2017.
Dzmitry Bahdanau, Felix Hill, Jan Leike, Edward Hughes, Arian Hosseini, Pushmeet Kohli, and Ed- ward Grefenstette. Learning to understand goal specifications by modelling reward. In International Conference on Learning Representations, 2018.
Bowen Baker, 1lge Akkaya, Peter Zhokov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, and Jeff Clune. Video pretraining (VPT): Learning to act by watching unlabeled online videos. Advances in Neural Information Processing Systems, 35:24639-2466 2022.
S. R. K. Branavan, Luke Zettlemoyer, and Regina Barzilay. Reading between the lines: Learning to map high-level instructions to commands. In Annual Meeting of the Association for Computational Linguistics, 2010.
10

---

Andrea Burns, Deniz Arsan, Sanjna Agrawal, Ranjitha Kumar, Kate Saenko, and Bryan A Plummer. Interactive mobile app navigation with uncertain or under-specified natural language commands. arXiv preprint arXiv:2202.02312, 2022.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph. Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finttuee language models. arXiv preprint arXiv:2210.114 2022.
Rémi Coulom, Efficient selectivity and backup operators in monte-carlo tree search, In Computers and Games, 2006.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
Yuqing Du, Ksenia Konyushkova, Misha Denil, Akhil Raju, Jessica Landon, Felix Hill, Nando de Freitas, and Serkan Cabi. Vision-language models as success detectors. arXiv preprint arXiv:2303.07280, 2023.
Hiroki Furuta, Ofir Nachum, Kuang-Huce Lee, Yutaka Matsuo, Shixiang Shane Gu, and Izzeddin Gur. Instruction-finetuuee foundation models for multimodal web navigation. In Workshop on Reincarnating Reinforcement Learning at ICLR 2023, 2023a.
Hiroki Furuta, Ofrr Nachum, Kuang-Huet Lee, Yutaka Matsuo, Shixiang Shane Gu, and Izzeddin Gurt. Instruction-finetuueed foundation models for multimodal web navigation. In First Workshop on Multimodal Representation Learning ICLR, 2023b.
Izzeddin Gur, UIrich Rueckert, Aleksandra Faust, and Dilek Hakkani- -ur. Learning to navigate the web. arXiv preprint 1rXiv:1812.091 2018.
Izzeddin Gur, Ofir Nachum, Yingjie Miao, Mustafa Safdari, Austin Huang, Aakanksha Chowdhery, Sharan Narang, Noah Fiedel, and Aleksandra Faust. Understanding HTML with large language models. arXiv preprint 2210.03945, 2022.
Peter C Humphreys, David Raposo, Tobias Pohlen, Gregory Thornton, Rachita Chhaparia, Alistair Muldal, Josh Abramson, Petko Georgiev. Adam Santoro, and Timothy Lillicrpp A data-driven approach for learning to control computers. In International Conference on Machine Learning. pages 9466-9482. PMLR, 2022.
Sheng Jia, Jamie Ryan Kiros, and Jimmy Ba. Dom-q-net: Grounded rl on structured language. In International Conference Learning Representations, 2019.
Geunwoo Kim, Pierre Baldi, and Stephen McAleer. Language models can solve computer tasks.
arXiv preprint arXiv: 174.1 17491, 2023.
Kenton Lee, Mandar Joshi, Iulia Turc, Hexiang Hu, Fangyu Liu, Julian Eisenschlos, Urvashi Khan- delwal, Peter Shaw, Ming-Was Wei Chang, and Kristina Toutanova. Pix2Struct: Screenshot parsing as pretraining for visual language understanding. arXiv preprint 10.03:2210.03347, 2022.
Gang Li and Yang Li. Spotlight: Mobile ui understanding using vision- language models with a focus. In The Eleventh International Conference on Learning Representations, 2022.
Yang Li, Jiacong He, Xin Zhou, Yuan Zhang, and Jason Baldridge. Mapping natural language instructions to mobile ui action sequences. arXiv preprint 2000:02000.03776. 2020a.
Yang Li, Gang Li, Luheng He, Jingjie Zheng, Hong Li, and Zhiwei Guan. Widget captioning: Generating natural language description for mobile user interface elements. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pages 5495-5510, Online, November 2020b. Association for Computational Linguistics. doi: 18653/ 122220.emmll-main.444
org/ 2020. emnlp-main. 443
Evan Zheran Liu, Kelvin Guu, Panupong Pasupat, Tianlin Shi, and Percy Liang. Reinforcement learning on web interfaces using workfloo-gguide exploration. In International Conference on Learning Representations (ICLR), 2018. Utthhttps:/ http//axxvvvvo// org/ abs/ abs/1802..../////../ 0tte//802.08888
11

---

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Mare G. Belle- mare, Alex Graves, Martin A. Riedmiller, Andreas Kirkeby Fidjeland, Georg Ostrovski, Stig Petersen, Charlie Beattie, Amir Sadik, loannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg. and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 51 11:529-553. 2015.
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju. William Saunders, et al. Webgpt: Browser-assiste question-answering with human feedback. arXiv preprint arXiv:2112.09332. 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 21:1-67, 2020. URL https
//arxiv. awtpaaa/...... 10683
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. ArXiv, abs//777776 2017.
Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In International Conference on Machine Learning, pages 4596-4604. PMLR, 2018.
Tianlin Shi, Andrej Karpathy, Linxi Fan, Jonathan Hernandez, and Percy Liang. World of bits: An open-domain platform for web-based agents. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 3135-3144. PMLR, 06-11 Aug 2017. URL https:// proceedings mwr. www.mlr.rreeeeee wttpppees.//./...aaa...aa.aa shwl7a. html
David Silver, Julian Schritwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676):354 359, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017
Bryan Wang. Gang Li, Xin Zhou, Zhourong Chen, Tovi Grossman, and Yang Li. Screen2Words: Au- tomatic mobile UI summarization with multimodal learning. In The 34th Annual ACM Symposium on User Interface Software and Technology, UIST '21, page 498-510, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450386357. doi: (0.114534472749.3477765
URL httphttts////i..co org/ 10.
Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning. 8:229-256, 2004.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoe Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. Google's ' neural machine translation sys- tem: Bridging the gap between human and machine translation. arXiv preprint arXiv: 1609.08 144, 2016.
Shunyu Yao. Howard Chen, John Yang, and Karthik R Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL htttps:///ppnreviee wwt.:/ooorm//---Snnmmaa
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), 2023.
Luke S Zettlemoyer and Robert St. Amant. A visual medium for programmatic control of interactive applications. In Proceedings of the SIGCHI conference on Human Factors in Computing Systems, pages 199-206, 1999.
12

---

A Additional Dataset Details
A.1 MiniWob++ Supported Tasks
MiniWob++ consists of 104 tasks. Most prior work (Shi et al. 2017 Liu et al. 2018; Gur et al. 2018; Jia et al., 2019) has evaluated performance only subset of these tasks, with the notable exception of Humphress et al. (2022), which evaluated on 104 tasks. We evaluated on 59 of these 104 tasks, based on our best effort attempt (1) design general purpose set of actions that could be implemented using Selenium and (2) convert the demonstrations collected by Humphreys et al. (2022) to observation and action format. While further development of the conversion process and Sclenium- baacdd actions could potentially support more tasks, the 59 tasks we support still include wide range of instructions and interactions. Note that determining the set of 59 tasks based solely on the feasibility of conversion to our observation and action format, and not based on model performance. Below offer further details.
Several tasks MiniWob++ feature animated elements. These tasks can require sampling ob- servations in a real-time manner in order to capture the information needed to select the correct action. Also, the effects of an action may delayed and therefore not captured by an observation sampled immediately after the action has executed. MiniWob++ provides -nodelay version for several tasks which removes such animations. We train and evaluate on the -nodelay ver- daiik-collapsible
sion of these tasks (choose-date We exclude zhoose-date-ear date-easy and choose- dadtemmeiuum which offer
click-cole coock-oollppsbbee click-pie.
use-aue aut-autoooo www.auuppeeeeeee.eee.m..... simpler versions of choose-dats but do not have corresponding -nodelay version. Addition- ally, we exclude chase- circl, drag-cube, noving-items and s simon-says, which feature animation without -nodelay version.
Many MiniWob++ tasks also involve vertical scrolling. In the human demonstrations, this can be implemented using scroll wheel, various clicking or dragging inter- actions with a vertical scroll bar rendered on the right side of a scrollable element. Mapping such interactions to actions that lead to equivalent scrolling in our Selenium- based environment is non-trivial. Therefore, for simplicity, we excluded tasks that in- volve scrolling: book-flight, click-scroll-list. email-inbon email-inbox-nl-turrs read-table, see serollttext- searrh2eesarreeeeeeeeeee search-engine, social social-media-all, -ome-- terminal.
scroll
Demonstrations for many MiniWob++ tasks also include copying and pasting text. many cases, this was executed in the human demonstrations by double clicking a text string and then clicking and dragging into an input field. Such an interaction is not supported in exclude the fol- Selenium, which made it challenging to support these tasks. This led forward lowing tasks: login-user-popup. copy-paste, copy-paste-2, email-inbox email-inboxxforrardd..l email-inbox-forward--- email-inbox-reppl
email-inoox-noscroll. enter-password, enter-text, enter-teett-ynnmicc find-word, login wwwtm mww.illlyllussttatmmmaaaa www.uutsssmmiiioorrrrriiiee
email-inbox-star-repll
Finally, we excluded several other tasks for various other reasons. The choose-list task uses the HTML <select> tag to implement a drop-down menu, which is not supported properly by our Selenium-based environment. The click-menu and click-menu-2 tasks require unsupported mouseover effects. Demonstrations for the text-editor task features click and drag interactions to highlight text which do not have the same effect when exccuted in Selenium. There also ap- peared to be differences in how Sclenium implemented the number input field for guess-numbes number. Finally, we excluded several tasks due to low demonstration conversion success rates (focus- text,, bocus-text-ai use-spinnery Upon further investigation, this was due to episodes completing im- mediately after a "pointer down" event without a complete click for focus-text and and due to frequent double clicking for use-spinner.
A.2 MiniWob++ Rendering Differences
There are differences between the rendering of observations in the human demonstrations from Humphreys et al. (2022) and the rendering of environment state in our Selenium-based environment. We show an example in Figure which shows subtle differences, e.g. in font style and in element sizes and positions.
13

---

MIRMOO, DJVKFV
Select yy. MIBRMO, DJVKFVV and click Submit
DJVKFV
DJVKFFV
MPSRMO
MISRMO
yy
Submit
Submit
Figure 5: Comparison of differences between the screenshots of the human demonstrations for MiniWob++ from Humphreys et al. (2022) (right) with how the same environment state rendcred in our Selenium- based environment (left).
B
Additional Technical Details
B.1
Beam Search
As mentioned in } we use beam search over tokens in the text decoder produce set of top-k actions for a given state, along with their approximate probabilities. We refer to these as approximate probabilities because they are subject to length normalization factor (Wu et al. (0166) of 0.6 during beam search, following Raffel et al. (2020). For MiniWob and WebShop, our experiments used & = 8 and h 10, respectively.
B.2
Tree Search
Here describe the details of the tree search approach described in $3.1 We adopt Monte Carlo Tree Search (MCTS) (Coulom 2006), and follow prior work which has integrated MCTS with neural networks (Silver et al.) 2017 Anthony al. 2017), which apply MiniWob++ environments. We performed a minimal amount of tuning to determine an approach that yielded improvements in mean score the greedy policy, even for the most challenging tasks.
Problem Setting We consider environment with states S and actions A. The reward function, r(s), returns a scalar corresponding to the reward given for transitioning to state E S, and is described below. MiniWob++ environments are randomly generated, but transitions are deterministic within an environment generated by a particular random seed. The transition function, f(s, a), returns the state resulting from taking action e A in state E S.
Surrogate reward Rather than using the raw reward directly provided by the MiniWob++ environ- ment, we consider a surrogate reward: r(s) a, (s), where a, provides small negative reward that encourages shorter trajectories without unnecessary actions. r'(s) is the raw reward from the MiniWob++ environment if is terminal state and the raw reward is > 0.8, or otherwise. We use 30 As all tasks can be completed within 30 steps, this is small enough to ensure positive reward is possible for all tasks. Additionally, the penalty is small enough such that in practice the agent should not be incentivized to sacrifice raw reward to reduce the number of steps taken.
1
Value network The value function oz (x) for a given policy æ is the expected future rewards from state if actions are selected according to policy æ. The optimal value function, (s), is the expected future rewards if optimal actions are chosen. We attempt to learn an approximation of this function, (s) Rs o² (s), parameterized to as the value network. The model is trained on transitions from the human demonstrations, which demonstrate close to optimal behavior in many cases. For every state in the human demonstrations, we compute the actual future rewards for the given episode, according to the surrogate reward. We map these future rewards to discrete bins and represent them as integers in the PIX2STRUCT decoder. At inference time, we approximate the mean of the distribution over these discrete bins by considering the top-n predictions from the model using beam search (with n = 3), weighted proportional their respective probabilities.
model with parameters Φ. which we refer
14

---

Policy network For consistency with prior work, we will refer to the PIX2STRUCT model tuned to generate actions (i.e. PIX2ACT) as the policy network, with parameters B. The greedy policy (T(ss) selects the action with the highest approximate probability po(als) in the top-k beam (see $B.1) subject to the conditions described in 4
Search policy We can use lookahead search to implement policy, (s), which leverages interac- tions with the environment (f(s, a) and r(s)) to select actions in a more optimal way than the greedy policy (s). Both the policy network and value network are used constrain and prioritize the search.
MCTS performs K rounds of traversing search tree with nodes corresponding to states, and edges corresponding to actions. Due to the computational cost of the policy and value networks, we use modest number of rounds, K = 16, for our experiments. The search tree is initialized with a single root node for state s. Each round starts at 8 and traverses the tree. At each step of a given round, an action a, is selected for state St, where d₁ max, (8t, a) + Ust, a). Q( a) is an average reward over all rounds that have traversed the associated edge. It is based on actual accumulated rewards during tree traversal and the value estimates of leaf states (described below).
(M(s))
Use, a) = c* (ppo(aa)) * T+n(s..a) is a term that encourages exploration, where (1 a) is the number of times action a has been selected from state 8t, (s,) is the total number of times state 8 has been visited, and c is a scalar hyperparameter that we set to 0.1. Following [Silvr et al. (2017). we use the policy network to bias this exploration term. To constrain the search, we only consider the top-k actions according to the policy network, where = in our experiments.
If we select an action for state 8 which has never been previously selected from 8t, then the simulation ends and we add a new leaf state, 8L. = [(t, a). to the search tree. sy is not a terminal state, then we estimate its value (i.e. future returns) using both the value network and a rollout with the greedy policy. Specifically, following /iiveer et al. (2017), we estimate its value as A * (sr.) + (1 - A) * u (sr.) where t=* (sr.) is equal to the actual returns from following the policy Ty starting at 8r, for a maximum of 20 steps, with actual returns clipped to a minimum value of O. Is there A is a mixing parameter that set to 0.1. For challenging environments, rollouts may be unlikely to find a terminal state with positive reward, and in such cases rollouts may not be very informative. On the other hand, the value network can provide poor value estimates for certain states, especially if they are not well represented in the human demonstrations. By combining both methods aim to provide better approximation of the value of leaf states. Returns are propagated up the tree to each parent s' to update ((s', a). As (sL a) is undefined prior to selecting a from 8y for the first time, we initialize Q(s1. a) for each action to be equal to the initial value estimate of sr, plus Os.-
To understand the impact of rollouts and value estimates using the value network, in Table we compare mean scores over 12 challenging MiniWob++ tasks for different values of A: O (rollout only), 0.1 (both rollout and value network), and I (value network only). We also include the mean score using the greedy policy for reference. These results use the policy network and value network trained on the human demonstrations. The results show that using combination of rollouts and the value network gives the best results. The value network is primarily useful for challenging tasks that relative using rollouts only.
require longer trajectories, such
Greedy Policy = 0 (rollout only)
A 0.1 À
1 (value network only)
28.8
74.2
78.3
57.4
Table 3: Mean scores for different policies over 12 challenging Mini MiniWob++ tasks.
Once we have completed K rounds, Tī (s) selects the most visited action a for state s, and we begin the process again at the subsequent state. We reuse the search tree for subsequent time steps for efficiency, so we require only K - n= a) additional rounds for the subsequent state.
Policy improvement We can sample trajectories with Tÿ. then update by training ()()) to approximate nī (s) for each in the sampled trajectories. This then also improves (z), as 0 informs how the search space is constrained and prioritized. Therefore, we can continue to iteratively improve (T)s). To produce these trajectories, we randomly sample MiniWob++ tasks and seeds, and select
15

---

actions according to zg. We then filter trajectories where the raw reward is < 0.8. We then tune & on these new trajectories. For simplicity, keep the value network (i.e. σ) fixed.
We initially found that tuning on trajectories from MCTS could be unstable, leading to an early loss spike. To resolve this, slightly decreased the learning rate (from 1e to 5e 4) and increased the number of warmup steps (from 1000 to 4000) relative to the hyperparameters used for behavioral cloning.
B.3 Compute Details
We fine-tuned models using 64 Google Cloud TPU cores.
Additional Results
C.1 Variance Estimates
We evaluated results for MiniWob++ based on 100 randomly selected seeds for each of the 59 tasks. To understand how results vary depending which 100 seeds per task used for evaluation, ran 3 trials with different evaluation seeds for the strongest Pix2ACT model reported in Table 3 yielding mean scores of 96.2, 96.4, and 96.1; the standard deviation across these trials was 0.15. For WebShop, there is a standard test set consisting of 500 instances, so selecting seeds for evaluation is necessary.
C.2 MiniWob++ Results Per Task
We show the performance of PIx2ACT (ours) on each of the 59 MiniWob+t tasks we study, compared
other approaches, in Table 4 We compare with human crowdworker performance reported by
Humphreys et al. (2022), CC-Net (Humphreys et al. 2022). DOM-Q-Ne (Jia et al. 2019). DOMNET with workflow-guiide execution (Liu
2018). QWeb (Gur et 2018). RCI Kim et al. 2023). WebN-T5-3B (Gur et al. 2022). and WebGUM (Furuta et al. 2023a). We also report scores for PIX2ACT and CC-Net with behavioral cloning (BC) only. We do include scores for GlobalCNN (hi et al. 2017). which reported only human normalized success rates. Other than Humphreys et al. (2022). prior work has primarily reported success rate (i.e. the percentage of episodes with positive rewards), which can be equivalently mapped to the scores we report for tasks without partial rewards.
16

---

Task
Ours
Ours (BC)
Human
CC-Net
CC-Net (BC)
DOMNET
DOM-Q-Net
QWeb RCI
WebN-TS
WebGUM
bisect angle
92
97
choose-date
6
97
97
100
cende-cener
96
52
96
97
click Hick--btton
32
98
100
7%
100
100
100
100
100
100
click button
100
94
100
100
100
100
bock-chhelbbssss
9%
click biick--heebbosss
click-cheeliheesse
click checkboses transfer
98
99
64
100
click-colll colk--oppppiieee222
97
9%
99
02
click-on -ealllapaible
94
80
99
100
100
100
100
98
click-color
99
38
97
100
100
100
34
click--hh stick-dialag
100
12
100
100
95
100
100
100
100
100
100
click-d click-dialo-22
click click-link
option
clickestades
100
elicc-shopp
19
88
95
11
64
98
72
clack-sat
100
54
99
100
95
100
100
100
100
74
100
ciick-tab22
98
42
177
98
98
100
74
click -ab-------ewy
99
99
click sab hard
98
76
click--tal Bark--a----mddimm
100
cllick-tett lesk--es--2
100
click best transfer
100
100
99
100
click-n click-wwdger
100
87
83
100
56
93
100
100
100
count-shape
70
0
82
85
21
76
40
68
count-sides
100
78
98
100
drug box
100
drup
drag grid
dmggsshues
ding-sor-oumbers
92
97
mmil-inboo---llee
100
99
99
100
22
100
email inbox important
100
99
99
190
30
enter-date
100
59
97
100
96
100
96
100
enter-lext-2
100
98
100
modpoint
grid-cserdinate
identi shape
100
navigate aree
98
99
99
100
100
50
100
manist-chhoskwvee
26
96
99
resize testaren
99
100
04
100
27
right-angle
97
100
87
98
simple algebra
simplo-aritmmeii
tic Se-ta--lae
83
unicode test
99
100
use ee--atcccompltte
100
colorwheel
97
90
98
68
colorwheel-2
93
100
54
95
38
use dlide
92
98
91
uee-alides22
visual
average
Table Mean scores across 59 MiniWob++ tasks.
17


Reviews:
[{"review_id": "UR5qJOqNcd", "reviewer": "Reviewer_XzpC", "summary": "This work explores the possibility of building an agent that can complete tasks for users solely based on pixel-level visual representations of the GUI state and generic low-level actions, without relying on structured or task-specific representations. The authors demonstrate the effectiveness of their approach on two benchmarks, MiniWob++ and WebShop, adapted to their general Chrome-based environment framework.", "questions": "(1) In Figure 2, step 3, why are there two overlapping ‘+’? Does it mean the mouse is pressed down?\n\n(2) This is not a serious issue, but some Figures are quite blurry (e.g., MiniWob++ examples in Figure1 and top row in Figure 2), is this due to the way the dataset is constructed? \n", "limitations": "The authors have included discussions of the limitations.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This submission demonstrates significant strength in several key areas. Firstly, it presents a clear and compelling motivation for developing an agent capable of directly interacting with pixel-level GUI interfaces, aligning with a more natural and human-like manner. This aspect alone distinguishes the submission and makes it particularly appealing to the readers.\n\nMoreover, the use of well-crafted illustrations enhances the understanding of the content and effectively communicates the main ideas, further solidifying its strength. Readers can easily grasp the concepts presented, adding to the submission's overall impact.\n\nFurthermore, the proposed framework is rigorously validated on two datasets, and the comparison with baselines using DOM/HTML as input is conducted in great detail. This thorough analysis strengthens the submission's credibility and highlights its robustness.\n", "weaknesses": "I have several concerns regarding the experimental design in the submission:\n\n(1) One notable issue is the absence of certain ablations that would provide valuable insights into the effectiveness of each component in the proposed agent framework. Specifically, the authors should address the bottleneck of the framework, whether it lies in the visual encoder or the text decoder. Additionally, it would be beneficial to explore the potential performance improvements by utilizing alternative variants of visual encoders or text decoders. Furthermore, since the visual encoder plays a crucial role in extracting and understanding instructions embedded in the interface, it is essential to verify the effectiveness of using ViT in OCR accuracy.\n\n(2) The rationale behind embedding instructions into the UI screenshot instead of providing them directly as input remains unclear. The submission should elaborate on this choice and justify its benefits. Moreover, considering the availability of open-source Visual Language Models (VLMs) like mPLUG-Owl [1], OpenFlamingo [2], and Otter [3], which natively support instruction following and accept multimodal input, it would be valuable to include these VLMs as baselines in the revised version. This addition would offer a more comprehensive evaluation and better contextualize the proposed method's unique contribution in comparison to existing approaches.\n\n[1] https://github.com/X-PLUG/mPLUG-Owl\n[2] https://github.com/mlfoundations/open_flamingo\n[3] https://github.com/Luodian/Otter\n\nAddressing these concerns would significantly enhance the experimental design and strengthen the submission's overall validity and contribution to the field.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review!\n\n> Bottleneck is vision encoder or text decoder?\n\nThe text decoder is only responsible for decoding short strings corresponding to the closed set of actions shown in Figure 1. Therefore, it seems reasonable to assume that the ViT encoder is the “bottleneck” towards achieving strong performance. As discussed in the introduction, many of the key challenges are essentially representation learning challenges for the encoder, such as understanding the interface layout, recognizing and interpreting visually-situated natural\nlanguage, identifying visual elements, and predicting their functions and methods of interaction. Our ablations highlight the screenshot parsing pre-training task of Lee et al. 2023 as a critical factor towards improving model performance (Figure 3).\n\n> Architectural variations of Vision Transformers\n\nPix2Act builds off of the Pix2Struct model of Lee et al. 2023, which uses a standard ViT architecture. We believe their strong results on tasks related to UI understanding (e.g. RefExp, Screen2Words) support the choice of this model as a starting point for pushing the boundaries of a pixel-based digital agent.\n\n> “The rationale behind embedding instructions into the UI screenshot instead of providing them directly as input remains unclear.” \n\nWhile we agree this method for representing instructions can perhaps seem unorthodox at first, prior work (e.g. Lee et al. 2023) has used this method for incorporating instructions in the input of pixel-only models for tasks such as DocVQA. This was hypothesized to enable better transfer from the pre-training task (see section 2.5 of Lee et al. 2023), and validated by strong empirical results. We will add this justification to the paper.\n\nWe also note that while this is the most natural way to incorporate instructions when starting from Pix2Struct, providing the instructions as text may work better for other pre-trained models that are tailored to accepting a text instruction as input.\n\n> ​​Baselines for “Visual Language Models (VLMs) like mPLUG-Owl, OpenFlamingo, and Otter” \n\nWe are excited by the increasing public availability of larger and more capable VLM models, and we agree evaluating such models on our proposed setting would be interesting for future work (we include some related discussion in section 7). However, we note that the Neurips 2023 submission deadline was May 17, 2023. To the best of our knowledge, the mPLUG-Owl preprint was released on April 27, 2023. OpenFlamingo was announced on March 28, 2023 and a paper is not yet available. The Otter preprint was released on June 8, 2023. \n\nAdditionally, compared to Pix2Struct, VLM models such as mPLUG-Owl, OpenFlamingo, or Otter have not demonstrated as strong a capability for learning representations of inputs that contain visually-situated language such as UI screenshots, as measured on tasks such as RefExp, Screen2Words, etc. Indeed, one of the limitations of mPLUG-Owl is “complex OCR” and the ability to correctly interpret web page screenshots (Figure 12 of mPLUG-Owl paper, ​​https://arxiv.org/abs/2304.14178). In contrast to these publicly available VLMs, GPT-4 has demonstrated strong results on tasks involving visually-situated language, outperforming Pix2Struct on ChartQA and AI2D (https://openai.com/research/gpt-4), but as far as we know support for tuning this model on multimodal inputs is not publicly available, and the details of the model have not been published. Regardless, our ablations (Figure 3) suggest that pre-training on the screenshot parsing task of Lee et al. 2023 is critical to achieve strong performance on the tasks that we study, suggesting that this task or a similar one should be included in the pre-training of VLMs that are used to develop digital agents. \n\n> Figures are quite blurry\n\nWe will try to improve the rendering quality, but are limited by the resolution of the MiniWob environment, which is 160 by 210 pixels.\n\n> Figure 2 step 3\n\nThe color selection element renders a “+” at the currently selected color. When the pointer is positioned over this element, it is also rendered as a “+”, as this element has the css property “cursor=crosshair”. At step 3 of Figure 2, the pointer is positioned over the currently selected color, leading to two similar “+”s that are slightly offset.\n"}, {"review_id": "4XzOKyJlAh", "reviewer": "Reviewer_5HMX", "summary": "The paper introduces PIX2ACT, a model designed to interact with GUIs using only pixel-level visual representations. Unlike most prior works that depend on structured interfaces (like HTML or DOM trees), PIX2ACT relies solely on what it visually sees. This approach is motivated by the way humans interact with interfaces without necessarily knowing the underlying code , which is interesting. The study reveals that such a model can efficiently operate in GUI tasks with generic mouse and keyboard actions, and even outperforms humans in specific benchmarks. For this they use PIX2STRUCT pre-trained on mapping screenshots to structured representation.\n\nMajor Contributions:\n\n1. Demonstrated that an agent with pixel-only inputs and generic actions can outperform humans on the MiniWob++ benchmark, achieving performance similar to top-tier agents with access to DOM information.\n\n2. Adapted the WebShop benchmark to function with pixel-based observations and generic actions, establishing the initial baseline performance in this setting.\n\n3. Highlighted the efficiency of PIX2STRUCT’s pre-training through screenshot parsing for GUI-based instruction with pixel-only inputs, leading to notable performance improvements on benchmarks like MiniWob++ and WebShop.\n\n4. Successfully applied tree search for policy improvement in the MiniWob++ environment.", "questions": "Latency and Computation: Transforming screenshots into structured representations can be computationally intensive, which may lead to latency in real-time applications. So what is the typical time taken? Wanted to hear authors thoughts since , the system is based on interpreting pixel-level information from screenshots. This can be computationally intensive?? and may struggle with dynamic content that changes frequently, or with GUI elements that look visually similar but have different functionalities.\n\nHuman Demonstrations: The model seems to be heavily reliant on human demonstrations for training. Gathering these demonstrations can be time-consuming and might not be feasible for all applications. Plus, the quality of these demonstrations can significantly affect the model's performance. Does authors have any thoughts one this?\n\nScalability: How does PIX2ACT scale with different screen resolutions, GUI complexities, or dynamic content on the screens? The performance of models designed for specific benchmarks might not generalize well to real-world applications. Since even the original PIX2STRUCT paper talks about the impact of resolution.\n\nis there a scope for more sophisticated reward function, such as one that provides the agent with feedback about the progress of the task? also is the cumulative reward that accumulates in the miniwob++ benchmark based on the time taken to solve the task in one episode considered? since the agent receives a reward only at a terminal state. Sparse rewards can make learning more challenging, especially if the agent needs to take a long sequence of actions before receiving feedback?\n\nCursor Representation: The system manually draws a cursor on the screenshot to indicate the mouse pointer position. This might not capture the full nuance of the cursor's state or type (e.g., a hand cursor vs. an arrow cursor), which could provide additional context to the agent.\n\nGreedy Action Selection: The agent follows a greedy policy by selecting the highest scoring action. Does this kind of approach can be short-sighted and might not always result in the optimal long-term strategy??\n\nBeam Search Limitations: While beam search helps in narrowing down the set of most probable sequences, does it always yield the optimal sequence in practice. in the context longer sequences, where the algorithm might not explore sufficiently outside of its \"beam\" to find a better solution. It can sometimes prefer shorter sequences over more accurate longer ones??\n\nFuture work: An additional pre-training task that could potentially improve the model is predicting affordances of UI elements from screenshots. Affordances refer to the possible interactions that an element supports, like if a button can be clicked, a text box can be typed in, a menu can be opened, etc.\n", "limitations": "The authors have mentioned the limitations ( It was thoughtful to anticipate about CAPTCHA) .\nHowever:\nData Privacy: Since they are using screenshots to interpret and interact with GUIs, there might be concerns about data privacy, especially if sensitive information is displayed on the screen. This should be highlighted.\nI understand this would be issue while using DOM/HTML as well but highlighted might lead to responsible adoption.\n", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper studies a new approach to controlling GUIs using just pixel level information unlike alternate approach which uses DOM tree or HTML. While both has its pros and cons , this is one of the new papers ton explore the pixel based controls in this setting.\n\nAdaptable Benchmarking Capabilities: Authors successfully adapt and operate on the WebShop benchmark using solely pixel-based observations and broad actions, laying down a foundational performance standard.\n\nRobust Pre-training Mechanism: The strength of PIX2STRUCT’s screenshot parsing pre-training is evident. This pre-training strategy remarkably boosts the model's efficiency in GUI-based instruction tasks using pixel-based inputs, as observed in dramatic performance jumps on benchmarks. The authors have performed and demonstrated the importance of this through the ablation studies. This is very interesting.\n\nEffective Policy Enhancement with Tree Search: The model employs tree search effectively, which proves to be a straightforward yet potent method for enhancing its policy in environments like MiniWob++.\n\nA very good point that the paper has made - \" Finally, aligning human demonstrations with task-dependent actions is often challenging.\"", "weaknesses": "The paper talks about generalization but slightly fails to highlight the same- for example : The chosen datasets ( miniwob++ and webshop(which is a great start) ) are relatively simple datasets and does not capture not capture the full complexity of real-world GUIs. \nAddtionally , to perform well reasonable well on the webshop dataset the model seems to require a fine tuning on miniwob++ which is still unclear as to why it is needed. Thus does it actually generalize to webshop? Furthermore, the authors perform hold out set evaluation on miniwob++. However, the tasks are \"click-checkboxes-large, click-color, click-tab-2, click-tab-2-hard, count-shape, drag-shapes, use-color-wheel-2, use-slider-2\" which could be similar to tasks in the miniwob++ benchmark train set , for instance \"click-tab-2-hard\" etc. It would be great to test on more diverse and complicated datasets.\n\nOver-reliance on Pre-training: While the pre-training from PIX2STRUCT seems beneficial, depending heavily on pre-trained models can sometimes limit the adaptability and flexibility of the system.\n\nHuman Demonstrations: The model seems to be heavily reliant on human demonstrations for training. Gathering these demonstrations can be time-consuming and might not be feasible for all applications. Plus, the quality of these demonstrations can significantly affect the model's performance.\n\nScalability: The approach described in the paper works well for small, simple web applications. It is not clear how the approach would scale to larger, more complex web applications.\n\nThe limitation I see in the environment setup is the use of discrete bins for mouse coordinates and scroll amounts rather than allowing continuous values. A few potential issues with the discrete coordinate binning:\nIt may make precise clicking and dragging actions more difficult if the bin sizes are too coarse. The agent would need to learn to chain multiple discrete drag actions to move long distances. The optimal binning resolution may vary across different interfaces and tasks. Finding the right granularity could require environment-specific tuning. Discretization can potentially lead to suboptimal policies compared to allowing continuous coordinates.\n\nSimilarly for scroll amounts: Scrolling by discrete bins could be inefficient for long pages compared to direct scrolling by pixel amounts. The scroll bin size would again need tuning based on typical page lengths. Overall, the discrete coordinate and scrolling simplification may be necessary to limit the action space size. But it could negatively impact performance on some tasks compared to an environment with continuous values.\n\nNo fine-tuning of visual features. The Vision Transformer weights are frozen, so the model may not learn visual representations best suited for this task.\n\nMinor comments: Despite the visual nature of GUIs, prior work has primarily focused on utilizing structured representations of the user interfaces (such as HTML sources, Document Object Model (DOM) trees, and Android view hierarchies) as well as custom, task-specific representations of high-level actions based on these structured representations. ---- > Missing citations.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review! \n\n> Discrete bins for coordinates and scrolling\n\nWe utilized discrete coordinate bins primarily for simplicity. We agree with the limitations you mentioned. While some prior work has also used coordinate bins (e.g. Humphreys et al. 2022), other work has used regression objectives or relative coordinate bins of varying precision (e.g. Baker et al. 2022, https://arxiv.org/abs/2206.11795), which may be useful to adapt for future work. We will add a discussion of this to the limitations section.\n\n> No fine-tuning of visual features\n\nWe do in fact tune all parameters of the underlying model, including both the ViT encoder and the text decoder. We will make this clearer in the revised paper.\n\n> Citations in introduction\n\nThank you, we will add citations or a forward reference to Section 6 where we discuss such approaches in detail.\n\n> Latency and Computation\n\nWe did not focus on optimizing the latency of our model, but we measured the latency of processing a single screenshot and emitting an action to be 0.4s, when run on a single Google Cloud TPU v2. While our model is small (282M parameters) compared to some LLMs, further optimizations would likely be required to run in a real time setting at frame rate greater than 2 frames per second. Indeed, as we mention in section 4.1, to simplify our setting “capturing real-time observations for animated elements” is not supported, but could be interesting to consider for future work.\n\n> “GUI elements that look visually similar but have different functionalities”\n\nConversely, elements with similar functionality may look visually similar but have different source code implementations across applications. We hope our work encourages further investigation of the pros and cons of representing web pages based on their source code vs. visual rendering.\n\n> Human Demonstrations\n\nWhile we are building off of and comparing with prior work that used human demonstrations, we agree that reducing the quantity of human demonstrations necessary to achieve strong performance in a new environment is a desirable goal for future work.\n\n> Scalability\n\nWhile the underlying Pix2Struct model supports variable-sized aspect ratios and has demonstrated success across a range of resolutions, we are unfortunately limited by the evaluations available to us. While MiniWob++ and WebShop collectively evaluate a range of capabilities, we hope increasing interest in the community to develop more realistic benchmarks will enable better studying this question in future work. For example, some benchmarks that have been released very recently (after the Neurips submission date) include Mind2Web (https://arxiv.org/abs/2306.06070) and WebArena (https://arxiv.org/abs/2307.13854).\n\n> Reward function and long trajectories\n\nFor MiniWob, consistent with prior work, we do not use time-decayed rewards for evaluating and comparing different approaches. Additionally, the agent receives a reward only in the terminal state. We found that this can indeed cause challenges, e.g. for finding a correct trajectory using tree search for tasks that require longer trajectories. Perhaps related to your suggestion, for MCTS we trained a value network to estimate the future reward, based on a surrogate reward that encourages shorter trajectories. This provides information related to “progress on the task”, and Table 3 in Appendix B demonstrates that using this signal is useful. Appendix B also provides the relevant technical details. This approach to MCTS is not conceptually novel, but to the best of our knowledge this is its first application to the types of tasks we study.\n\n> Cursor representation\n\nWe do in fact dynamically change the cursor rendering based on the cursor type according to the `cursor` CSS property of the currently hovered element. This can be seen in Figure 2 step 3, where the color selector element has the css property `cursor=crosshair`. That said, we didn’t find much evidence that this actually improves performance.\n\n> Greedy action selection\n\nGreedy action selection is indeed limited, especially when the underlying model is weaker. This can be seen by the difference between the greedy policy vs. tree search policy (Table 2). However, at test time, we assume the agent cannot revise previously chosen actions so report the greedy policy results for consistency with prior work.\n\n> Beam search limitations\n\nBeam search is used in the text decoder to determine the top-k actions for a given step. Most actions have a similar token length (example action strings are shown in Figure 1). Additionally, following T5 and Pix2Struct, we use length normalization (briefly mentioned in Appendix B.1) to attempt to offset beam search’s bias towards shorter sequences, which we will clarify in the main text.\n\nWe also had initial experiments exploring beam search as an alternative to MCTS for identifying high-reward sequences of actions, but it did not perform as well as MCTS, especially for tasks requiring longer trajectories.\n\n> Future work and pre-training\n\nWe agree with your proposed direction for future work. This type of affordance information should be abundant on the web and provide a potentially useful pre-training signal.\n\n> Data privacy\n\nThank you for raising this potential concern. We will incorporate this into the revised paper.\n"}, {"review_id": "tsBgVD1bDI", "reviewer": "Reviewer_CAUm", "summary": "This paper presents PIX2ACT, a method that interacts with GUIs using pixel-level visual representations and generic low-level actions, emulating how humans interact with these interfaces. Unlike previous approaches, it doesn't rely on structured text-based data sources, but rather processes pixel-based screenshots, circumventing issues associated with obfuscation or misalignment in structured application data. The model showed improved performance compared to human crowdworkers on the MiniWob++ benchmark for GUI-based instruction following tasks.\n", "questions": "Would the model maintain high performance in real-time environments, given the training was conducted in offline environments?\nHow much of the performance difference is due to the size of the model? Can this method be easily adapted to take advantage of existing larger pretrained models? \n", "limitations": "The tree search approach relies on the ability to generate new environment and instruction variations and receive reward signals, which might not be feasible in some real-world applications.\nThe paper does not provide a clear solution for the performance gap on complex tasks and environments, like the WebShop benchmark.\n", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper introduces a novel approach to the challenge of automated GUI interactions, using pixel-based screenshots as opposed to relying on text-based representations.\nThe PIX2ACT model was tested on two benchmark datasets, MiniWob++ and WebShop, which ensured a robust and varied testing process.\nThe paper is clearly written and offers substantial and important contributions, namely the ability to build an agent that can outperform humans in task completion using pixel-based inputs and a generic action space.\nTheir findings indicate that the pre-training of PIX2STRUCT via screenshot parsing is effective for GUI-based instruction following with pixel-based inputs.\n", "weaknesses": "The performance on the WebShop benchmark is still significantly below larger language models using HTML-based inputs and task-specific actions.\nSince the method uses tree search, it seems to rely on offline environments. It’s not clear if this approach will be useful in real-world online environments, although I could see this working with perhaps re-settable virtual environments. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review!\n\n> Limitations of tree search in real-world environments\n\nWe tried to address some of the tree search limitations you mentioned in section 7, as well as some potential directions towards applying such an approach more broadly (e.g. generative models of potential instructions and approximate reward models). We will add that our tree search method also requires environments with the ability to reload an initial state, which we agree may not hold in all real-world environments. That said, there are also additional considerations (e.g. security) that may discourage training-time exploration beyond controlled environments in practice.\n\n> Would the model maintain high performance in real-time environments, given the training was conducted in offline environments?\n\nWhile MiniWob++ and WebShop collectively evaluate a range of capabilities, we hope increasing interest in the community to develop more realistic benchmarks will enable better studying this question in future work.\n\n> How much of the performance difference is due to the size of the model? Can this method be easily adapted to take advantage of existing larger pretrained models?\n\nWe have only evaluated Base Pix2Struct models (282M parameters) in this paper. However, conceptually, our method relies only on a pre-trained model implementing a generic image-to-text interface. Therefore, exploring larger pre-trained models is a great direction for future work, especially as such multimodal models become increasingly available.\n"}, {"review_id": "7pvbjEafth", "reviewer": "Reviewer_sYmC", "summary": "One of the main goals of intelligent agents is to interact with the internet in the same way that humans do. Prior state-of-the-art models relied on both the DOM structure and the graphical user interface (GUI) to achieve good performance on web browsing tasks that involve following instructions. In this paper, the authors propose a model that relies solely on pixel-based screenshots as input. The model then selects actions that correspond to basic mouse and keyboard operations. \n\nThe authors use a pre-trained model called Pix2Struct and a standard behavior cloning and reinforcement learning (RL) framework to improve the model's ability to follow instructions and perform various web tasks. The authors demonstrate that their pixel-based approach, which does not require access to DOM-related information, can achieve competitive performance on the MiniWob++ benchmark. Additionally, they show through ablation studies that the performance gains can be primarily attributed to the pre-training of Pix2Struct.\n\n**Update after rebuttal**\nI confirm that I have read the rebuttal and other reviewers feedback. Authors have addressed my concerns.", "questions": "- Hyperparameter on Webshop data: learning rate and optimizer details. If these are similar to MiniWoB++, please add a note \n- Choose-date: what kind of mistakes model make ? How does RL sampled data help with poor performing tasks? \n- Fig 3: for how many steps did you train Ours (BC only) vs Ours ? In particular, does BC training for additional steps help in improving BC baseline? \n- Authors suggest that intermediate fine-tuning on MiniWoB++ helps for WebShop tasks. Do they observe similar gains while using WebShop data for MiniWoB++ task?", "limitations": "Authors have addressed major limitations of their work and have also provided potential misuse of their research.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Teaching agents to interact with the web is an important research direction. The experiments presented in this paper provide valuable insights to the community. In particular, the authors demonstrate that large-scale pre-training on image-to-text tasks, such as Pix2Struct, can reduce the need to explicitly provide DOM information to the model. Additionally, the task transfer ablation study shows that the model is able to generalize to other tasks. These findings will help to shape the future research direction on this topic.\n\n- Paper is well written and is easy to follow. For MiniWob++ benchmark, authors performed sufficient ablation studies to quantify gains due to individual components i.e. pre-training, behavior cloning.", "weaknesses": "- It's not clear if authors have tuned all baselines correctly. In particular, authors show that their proposed RL data bootstrapping method leads to better performance than Behavior Cloning. If I understood the experiments correctly, they have not trained the model for the same number of steps so it's very likely that behavior cloning baseline is potentially under-trained. \n- There are multiple important design discussions that are added to the text without any particular justification. For example, authors fine-tune model on MiniWoB++ data for WebShop experiment but don't use WebShop data for MiniWob++ experiment. There is no justification for such design choices. Further, authors provide different learning rate and number of steps for various experiments without justifying these design choices.   \n- Authors present very sparse results and discussion on WebShop dataset and all experiments are mainly conducted on MiniWoB++ dataset. As a reader, it feels like that authors added that dataset mainly for the sake of adding it.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review!\n\n> WebShop details and hyperparameters\n\nWhile some prior work (e.g. Humphreys et al. 2022) evaluated only on MiniWob, we believe it was important to also evaluate on WebShop to better understand the generality and limitations of our approach. We will provide some additional details, discussion, and error analysis for WebShop as space allows in our revised paper. \n\nThank you for pointing out the missing optimizer hyperparameters for WebShop. We indeed use the same optimizer and learning rate as for MiniWob++. We will clarify this in our revised paper in section 5.1.\n\n> Tuning of BC-only model\n\nWe did tune the BC-only model to determine the optimal number of training steps. With 26K training steps on MiniWob++ the model completes multiple epochs over the human demonstrations. Training for more steps on the human demonstrations does not improve performance, and can reduce performance due to overfitting. \n\nNotably, our BC-only result is also strong even relative to prior work that uses DOM information (66.5 vs. 38.7 for Humphreys et al. 2022). Finally, the finding that leveraging environment interaction improves performance beyond BC on MiniWob is consistent with prior work that has applied RL methods to MiniWob++ (e.g. Humphreys et al. 2022).\n\n> Using MiniWob data for WebShop and vice versa\n\nThe improvement from using MiniWob data for WebShop is 4.0 points in task score (the relevant ablation is mentioned in section 5.4). While perhaps not a key result in the paper, we thought this improvement was significant enough to report.\n\nWe did not observe any improvement from using WebShop data for MiniWob. Notably, WebShop has far fewer human demonstrations (1.5K vs. ~15K per MiniWob task), which may explain this asymmetry.\n\n> Error analysis for “choose-date”\n\nThis task requires using a calendar interface to select a specific date. The calendar may be initialized to a month (e.g. June) that is far from the target month (e.g. December). The agent must navigate through the calendar one month at a time. A common error is that the model attempts to navigate the calendar in the wrong direction. Perhaps this implies a lack of knowledge related to the ordering of months, or an inability to robustly apply this knowledge in this context.\n\n> How does RL sampled data help with poor performing tasks?\n\nThe tree search algorithm can find examples of successful trajectories where the current policy would otherwise fail. Therefore, training on these trajectories can improve performance.\n"}, {"review_id": "OFW6rIszoj", "reviewer": "Reviewer_X3Sy", "summary": "This paper investigates to create agents that interact with the UIs based on pixel-based screenshots and a generic action space corresponding to keyboard and mouse actions. Extensive experiments on simulated environments, i.e., MiniWob++, WebShop, evaluate the effectiveness of proposed agents, show the benefits of pretraining, and demonstrate the successful application of tree search.", "questions": "1. What are the key differences between this work and the existing pixel-only UI agents and tree search based RL agents in terms of the methodology?\n2. What are the motivations, detailed formulation and beneits of applying tree search here? Are there anything special for completing UI tasks?", "limitations": "Pls kindly see the contents of weakness and questions as above.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "This paper builds a pixel-based agent for instruction-following interaction with UIs. This is an interesting and valuable research topic of high bussiness impacts as well for building AI-based UI interaction assistants. The proposed methods are reasonable, and the conducted experiments are clearly presented. The corresponding analysis and discussion for limitations are comprehensive and sufficient.", "weaknesses": "The biggest problem of this work is about the novelty of its proposed method. It's kind of difficult for readers to catch up with the core differences compared with the existing pixel-based agents, e.g., Pix2Act, and understraning its advantages. The authors should provide mode insight by highlighting the novelty of proposed methods and clarifying the rationale behind its advantages. \n\nAs for the application of tree search, it is widely seen in RL-based agent building. The corresponding introduction in this works, including its motivation, formulation and benefits, lacks of a detailed and clear statement.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review!\n\n> “The biggest problem of this work is about the novelty of its proposed method. It's kind of difficult for readers to catch up with the core differences compared with the existing pixel-based agents, e.g., Pix2Act, and understraning its advantages.”\n\nWe are confused by this comment. This paper introduces Pix2Act. Pix2Act is not an approach from prior work.\n\nThe only comparable pixel-based agent from prior work for the tasks that we study was a version of CCNet proposed by Humphreys et al. 2022. We discuss CCNet in the introduction, and offer several empirical comparisons in Section 5. One of the key differences is that Pix2Act is pre-trained on web-scale data, which we find demonstrates significant transfer to the instruction following tasks that we study. Pix2Act significantly outperforms the pixel-only version of CCNet on MiniWob++ with a mean score of 96.2 vs. 24.1 (Figure 3).\n\nPerhaps you are interested in the differences between Pix2Act and Pix2Struct? The underlying image-to-text model for Pix2Act is initialized from the pre-trained Pix2Struct model of Lee et al. 2023, but Pix2Struct image-to-text models have not previously been used to develop pixel-based agents. We will attempt to clarify this relationship in the introduction of Section 3.\n\n> “As for the application of tree search, it is widely seen in RL-based agent building.” “What are the motivations, detailed formulation and beneits of applying tree search here? Are there anything special for completing UI tasks?”\n\nWe agree that tree search has been used by prior work in the context of games and other environments, and we did not intend to claim any conceptual novelty in the MCTS algorithm that we apply. Indeed, we adopted MCTS because it is a well-studied algorithm and “has been successfully integrated with neural network policies in prior work” (Section 3.1). We will provide some additional context of prior work related to tree search for policy improvement to our revised paper. The formulation of our MCTS implementation, including integration with policy and value networks, is detailed in Appendix B. We hope the success of tree search on the tasks that we study will inspire future work to consider related methods.\n"}, {"review_id": "Ydfb5pd8U3", "reviewer": "Reviewer_uTRS", "summary": "The paper describes a system to perform pixel-based interaction with human-taylored screens based on GUIs. The proposed system relies on a transformation to a text-based, structured representation of the screen and the screen tasks performed using a larger, pre-trained model of almost 300M parameters. The proposed system achieves performance similar to human beings performing those tasks and SOTA using text representations of the screen. ", "questions": "1. Are the results of the proposed system statistically better than the other systems? With what level of confidence?\n2. Can you provide the standard deviations of the tests performed? Are they adequate?", "limitations": "The limitation section is very brief and not adequate for a paper with the ethical impacts that such technologies can create. ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The main strengths of the paper are:\n\n1. It demonstrates a successful system able to interact with the screen (web-based applications)  in a similar way that humans do.\n2. It does so by fine-tuning a pre-trained system with a small number of demonstrations. \n3. It performs an ablation test where it shows that the pre-training information is critical.", "weaknesses": "The main weaknesses of the paper are:\n\n1. The evaluation procedure is so succinctly described that it is hard to understand what was exactly done.\n2. The granularity of number of tests according to test type seems to be too coarse (500 tests for 59 task is less than 10 per task).\n3. Results are report with little extra information such as mean scores.\n4. No statistical significance analysis is performed justifying the claims that the scores are better than SOTA or human (therefore here I assume they are similar, not better).\n5. The discussion section is very limited and not clear about the actual contributions.\n6. The authors fail to discuss important ethical issues concerning their research. In particular, if AI systems can interact using GUI interfaces, they can produce content and affect web-based systems in very impactful ways, including altering political and social discourses, create hard-to-identify denial attacks, fool \"I am not a robot\" tests, and pose as people while interacting with real people.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review!\n\n> Ethical considerations\n\nWe agree there are important considerations for responsibly developing and deploying models that can interact with websites. While we attempted to identify some of these concerns (e.g. breaking CAPTCHAs) in the “Broader Impact” subsection of Section 7, you have raised several important additional concerns. We will expand the discussion of these issues, and potential mitigations, in our revised paper. See also our response to Ethics Reviewer Vxcz and Ethics Reviewer n367. \n\n> “The evaluation procedure is so succinctly described that it is hard to understand what was exactly done. The granularity of number of tests according to test type seems to be too coarse (500 tests for 59 task is less than 10 per task).“\n\nPlease note that there are two paragraphs that discuss evaluation separately for MiniWob++ (Section 4.1) and WebShop (Section 4.2). For MiniWob++, we evaluate on 100 random instances *per task*, for a total of 5900 instances. For WebShop, we follow the standard procedure of evaluating on the 500 test instances. In both cases, our evaluations are consistent and comparable with prior work. We will try to provide additional context for readers unfamiliar with MiniWob++ and WebShop, for our revised paper.\n\n> “Results are report with little extra information such as mean scores.”\n\nWe report mean scores across test instances as the primary metric, as described in the “Evaluation” paragraphs of sections 5.1 and 5.2. We also provide more detailed per-task scores in Table 5 of Appendix C.2. \n\n> Statistical significance analysis\n\nFor MiniWob++, we estimated variance by computing the mean score for 3 different sets of random seeds for generating test instances. This yielded mean scores of 96.2, 96.4, and 96.1; the standard deviation across these trials was 0.15. These results will be added to Appendix C for our revised paper.\n\nThe key comparison for Pix2Act on our proposed setting for MiniWob++ is with CCNet without DOM information, where Pix2Act outperforms with a mean score of 96.2 compared to 24.1. \n"}]

Author rebuttal:
We would like to thank all of the reviewers for their comments and suggestions! We tried to address any questions in the individual responses.

We would also like to respond to the ethical reviewers, as we did not see a way to respond individually to ethics reviews.

__Ethics Reviewer Vxcz__

Thank you for your review!

> Human Demonstrations

We have had previous correspondence with the authors of Humphreys et al. (ICML 2022) and Yao et al. (Neurips 2022). Our use of both datasets is consistent with the permission granted by the authors. We can also confirm that neither dataset contains PII. We will work with both groups of authors to confirm further details of the annotation process.

We agree that datasets that incorporate a more diverse set of instructions and example demonstrations, especially those that reflect the needs of diverse users with different abilities, would be a valuable contribution for future work, and will mention this in our revised paper.

> Broader Impacts

We agree there are important considerations for responsibly developing and deploying models that can interact with websites. While we attempted to identify some of these concerns in the “Broader Impact” subsection of Section 7, you have raised several important additional concerns. We will expand the discussion of these issues, and potential mitigations, in our revised paper draft. Taking your comments into consideration, below are some of our thoughts on these issues.

In this paper we have trained and evaluated models only in offline environments. Responsibly deploying models in an environment where they can interact with online services would require additional considerations. Prior to enabling a model to access a new service, it would be important to sufficiently verify and/or constrain the behavior of the model to ensure that it is consistent with the terms-of-service for that service and does not otherwise cause harm.

There would be many potential risks associated with deploying models that could interact with services in violation of their terms-of-service or otherwise engage in various forms of spam, fraud, or abuse. Examples of such behavior could include impersonating human users, generating harmful content or spam, or engaging in denial-of-service attacks. Models that use the same conceptual interface humans use could potentially be more capable of breaking security defenses (e.g. solving CAPTCHAs) or engaging in forms of spam, fraud, or abuse that are more difficult to detect. It is therefore important for research related to security and techniques for detecting spam, fraud, and abuse to take such potential uses into account.

__Ethics Reviewer n367__

Thank you for your review!

We will elaborate on the ethical considerations related to security, and possible mitigations, in our revised paper. Relevant security research would include work on CAPTCHAs. For example, Section 3.4 “Attacks against Behavior-based CAPTCHA” of  https://arxiv.org/abs/2103.01748 discusses CAPTCHA attacks by bots imitating human behavior, and the authors also discuss potential mitigations.

Please also see our response to Ethics Reviewer Vxcz for a broader discussion of other ethical considerations.


Meta-review: Overall I believe this paper should be accepted - I have one weak reject, but all others say accept. The authors and the weak reject have had several exchanges and I believe issues have been resolved but the score has not been changed. I recommended to accept two weeks ago and have received no objections. The authors have addressed ethics concerns in my opinion. 

I believe one reviewer summarizes the opinion well with the following:

This submission demonstrates significant strength in several key areas. Firstly, it presents a clear and compelling motivation for developing an agent capable of directly interacting with pixel-level GUI interfaces, aligning with a more natural and human-like manner. This aspect alone distinguishes the submission and makes it particularly appealing to the readers.

Moreover, the use of well-crafted illustrations enhances the understanding of the content and effectively communicates the main ideas, further solidifying its strength. Readers can easily grasp the concepts presented, adding to the submission's overall impact.

Furthermore, the proposed framework is rigorously validated on two datasets, and the comparison with baselines using DOM/HTML as input is conducted in great detail. This thorough analysis strengthens the submission's credibility and highlights its robustness.
Decision: Accept (spotlight)