The Impact of Element Ordering on LM Agent Performance

## Abstract

Abstract There has been a surge of interest in language model agents that can navigate virtual environments such as the web or desktop.
To navigate such environments, agents benefit from information on the various elements (e.g., buttons, text, or images) present.
However, it remains unclear
which element attributes have the greatest impact on agent performance,
especially in environments that only provide a graphical representation (i.e., pixels).
Here
we find that the ordering in which elements are presented to the language model is surprisingly impactful—randomizing
element ordering in webpages compromises average agent performance to a degree comparable to removing all visible text from webpages.
While web agents benefit from the semantic hierarchical ordering of elements available via the browser, agents that parse elements directly from pixels do not have access to any such ordering.
Here we endeavor to derive effective orderings and investigate the impact of various element ordering methods in web and desktop environments.
We find that dimensionality reduction provides a viable ordering for pixel-only environments.
We train a UI element detection model to derive elements from pixels and apply our findings to an agent benchmark—OmniACT—where we only have access to pixels.
Our method completes more than two times as many tasks on average relative to the previous state-of-the-art.

## 1 Introduction

There has been growing interest in using language model (LM) agents
to autonomously navigate virtual environments.
Autonomous web agents (Zhou et al., ; Kim et al., ; Zheng et al., ; Gur et al., ; He et al., ) have become a particularly popular area of research.
Typically, a web agent takes as input a task prompt from a user, observes a text and visual representation of the environment, and then outputs
one or more
actions to execute the task in the environment.
Recently, research interest has expanded to include agents that can navigate mobile (Rawles et al., ; Yan et al., ) and desktop (Xie et al., ; Kapoor et al., ; Bonatti et al., ) environments as well.

At a high level, a virtual environment consists of numerous elements—some are interactive (e.g. buttons or widgets), while others are not (e.g. plain text).
To allow for human navigation, these elements are usually represented in the pixel space via a Graphical User Interface (GUI).
In contrast,
agents often rely on distinct state representations to navigate virtual environments.
The exact format of a state representation varies between environments and approaches.
In web environments, common text representations include the HTML or accessibility tree (Zhou et al., ; Koh et al., ).
For visual representations, a popular approach is to label UI elements with bounding boxes and numeric identifiers (Koh et al., ; He et al., ), known as *Set-of-Mark* (Yang et al., ).
In either case, the state representation is derived from the underlying Document Object Model (DOM) (Zhou et al., ; Koh et al., ; He et al., ).
However, many environments lack a descriptive DOM and only provide pixel information, which we show is insufficient for existing agents (see Section 4.2).
To construct an effective state representation from only pixels, it is important to answer the following basic questions about these representations:
What aspects of a state representation are most important to an agent?
How can we derive these important aspects with only pixels?

Figure: Figure 1: We are motivated by the goal of enabling agents to act on environments where an underlying DOM does not exist. Instead, the agent must determine its next action using only the environment’s graphical representations. In Step 1, we first detect a list of unordered UI elements using an object detection model and identify them with bounding boxes. In Step 2, we convert these UI elements to their text representation. In Step 3, we order the elements via 2D-to-1D dimensionality reduction. Due to the sequential nature of a language model, elements are always presented in a specific order to the LM Agent. Finding an effective ordering is non-trivial, yet can significantly affect agent performance. Elements that are visually close together are often functionally associated with each other. t-SNE’s ability to retain local structure allows it to generate an effective ordering.
Refer to caption: extracted/5904312/imgs/fig1.png

In almost all implementations of the agent’s state representation, there exists a list of interactable or non-interactable elements which the agent uses to determine the next action (Koh et al., ; Zhou et al., ; He et al., ; Dupont, ; Yan et al., ; Ishan0102, ; Kapoor et al., ; Xie et al., ).
Elements are characterized by various attributes such as visual appearance, text descriptions, or type labels.
Because the state representation is the input to an LM,
this list of elements is always
presented in
a specific ordering.
For example, the default method to derive elements from a webpage performs a pre-order traversal of the DOM tree (World Wide Web Consortium, ).
We analyze various attributes of a popular state representation for agents and find element ordering to be the single most impactful attribute to agent performance.
We find that the ordering of elements can dramatically affect the performance of an agent, resulting in differences of up to 49% relative performance.

At a high level, a virtual environment consists of numerous elements—some are interactive (e.g. buttons or widgets), while others are not (e.g. plain text).
To allow for human navigation, these elements are usually represented in the pixel space via a Graphical User Interface (GUI).
While sometimes , environments provide only pixel-level information.
To construct an effective state representation in these scenarios, it is crucial to answer the following questions.
What aspects of a state representation are most important to an agent?
How can we derive these important aspects with only pixel-level information?

The exact format of a state representation varies between environments and approaches.
Common choices for a text representation include the HTML or accessibility tree (Zhou et al., ; Koh et al., ; Xie et al., ).
For visual representations, a popular approach is to label UI elements with bounding boxes and numeric identifiers (Yang et al., ; Koh et al., ; He et al., ).
In almost all implementations, the agent observes a list of interactable or non-interactable elements to determine the next action.
Elements are characterized by various attributes such as visual appearance, text descriptions, or type labels.
Because the state representation is the input to an LM,
this list of elements is always
presented in
a specific ordering.
For example, the default method to derive elements from a webpage performs a pre-order traversal of the Document Object Model (DOM) tree (World Wide Web Consortium, ).
We analyze various attributes of a popular state representation for agents and find element ordering to be the single most impactful attribute.
.
We find that the ordering of elements can dramatically affect the performance of an agent, resulting in differences of up to 49% relative performance.

This can prove problematic as many environments lack obvious methods to both derive and order elements.
For example, many mobile and desktop applications (see Figure 2) do not properly expose interactable elements (Chen et al., ; Ross et al., ; Zhang et al., ).
In such environments, pixels may be the only source of information available.
Previous approaches to deriving interactable elements from pixels either leverage off-the-shelf segmentation models (Yan et al., ; Kapoor et al., ) or build custom models that target accessibility features (Wu et al., ).
In our approach, we leverage common crawl (Common Crawl, ) to train an object detection model (Ren et al., ) that detects interactable UI elements specifically for agents.
To the best of our knowledge,
the elements detected through these approaches are ordered arbitrarily (e.g. based on confidence scores);
visually, the ordering is effectively random.
Our experiments indicate that a random ordering consistently results in the lowest performance across multiple scenarios.

Figure: Figure 2: Many software applications lack informative accessibility trees or DOMs. The accessibility tree for a popular game development engine (Godot, left) contains only the exit, minimize, and full screen buttons. For a presentation slide (Google Slides, right), no interactive elements (e.g. title and subtitle boxes) are present in the DOM. Language model agents rely on this information to navigate applications, and agent performance can accordingly be compromised in scenarios where it is incomplete.
Refer to caption: extracted/5904312/imgs/bad_7.png

Here we propose and evaluate strategies for deriving effective element orderings in scenarios where a hierarchical ordering based on the GUI design is not explicitly provided by the environment.
Across multiple agent benchmarks, we find that ordering elements via a 2D-to-1D dimensionality reduction (Van der Maaten & Hinton, ) reliably yields improvements to agent performance relative to other baselines.
We experiment on the VisualWebArena (Koh et al., ) and OmniACT (Kapoor et al., ) benchmarks and achieve new state-of-the-art performance on OmniACT.

Out contributions are as follows.

- •
We conduct a thorough ablation of VisualWebArena’s state representation for agents by including or removing each element attribute individually. Despite advancements in vision language models, we find that a text representation is still necessary for web and desktop agents. We find that element ordering is, perhaps surprisingly, more impactful than any other attribute in the text representation.
- •
We demonstrate that ordering via dimensionality reduction consistently provides performance improvements over random ordering. Additionally, we find that ordering via dimensionality reduction performs better than
a simple position-based
ordering in most scenarios.
- •
We achieve a new state-of-the-art result on OmniACT, an agent benchmark that considers the scenario of operating on pixels. Our approach more than doubles the expected average task success rate compared to the previous state-of-the-art.

## 2 Related Work

#### Agent Benchmarks.

World-of-bits provided the first environment for evaluating web GUI navigation using an agent (Shi et al., ).
Over time, more realistic web (Zhou et al., ; Koh et al., ; Yao et al., ), desktop (Kapoor et al., ; Xie et al., ), and mobile (Rawles et al., ) agent benchmarks have been created.
Kim et al. () provided one of the first LM agent approaches, successfully navigating World-of-bits.
However, existing agents are still unable to properly navigate more realistic benchmarks, completing only 15% of web tasks (Zhou et al., ) and 12% of desktop tasks.

#### Agents With Direct Access to Elements.

Despite work on multimodal agents (Koh et al., ), existing techniques in navigating web and desktop environments still rely heavily on ground-truth text representations.
Zhou et al. (); Koh et al. () both utilize the accessibility tree and its elements as their state representation.
Koh et al. (); He et al. () consider approaches where interactable elements are also represented in an image via Set-of-Mark (Yang et al., ) bounding boxes and labels.
Xie et al. () provides an agent that navigates desktop applications by observing a filtered down version of the accessibility tree.
All of these approaches require access to either a webpage’s underlying DOM or an accessibility tree to derive elements;
however, our focus is on environments that only give access to their graphical representations which is significantly more challenging.

#### Agents With Access to Only a Graphical Representation.

There have been several approaches—primarily focused on desktop and mobile environments—to directly navigating a GUI via its pixels.
Kapoor et al. () and Yan et al. () focus on navigating desktop and mobile applications respectively.
Both leverage an off-the-shelf-segmentation model—Segment Anything (Kirillov et al., )—to find icons in the image.
These icons are then either represented in text (Kapoor et al., ) or labeled with Set-of-Mark (Yang et al., ) bounding boxes and labels in the image (Yan et al., ).
We instead train an object detection model that detects interactable UI elements directly.
While previous UI element detection models are trained to detect accessibility features (Wu et al., ), our model is trained specifically to detect interactable elements that would be useful to an agent.
All three of our approaches use Optical Character Recognition (OCR) modules such as EasyOCR (AI, ) to extract text from pixel information.

#### Agent Input Ablations.

While most agent studies include some ablations, few focus on detailed analysis of an agent’s input.
To our knowledge, Huq et al. () is the only other study that directly studies this.
Their study focuses on broader components to an input prompt such as the selection of few-shot examples used, while we focus on specific element attributes such as element ordering.

## 3 Problem Definition

We define the environment state $\mathcal{E}$ as a set of elements ${\mathcal{E}=\{e_{1},e_{2},\ldots,e_{n}\}}$,
where each element $e_{j}$ is a tuple $\langle i_{j},C_{j},A_{j},S_{j}\rangle$ defined by the following parameters:

- •
$i_{j}\in\{0,1\}$ denotes the interactability of element $e_{j}$. An element with $i_{j}=1$ is interactable, while an element with $i_{j}=0$ is not.
- •
$C_{j}=\{\langle x_{1},y_{1}\rangle,\langle x_{2},y_{2}\rangle\}$ is the set of pixel coordinates that form the bounding box around $e_{j}$. $\langle x_{1},y_{1}\rangle$ is the top left coordinate and $\langle x_{2},y_{2}\rangle$ is the bottom right coordinate.
- •
$A_{j}=\{a_{1},a_{2},\ldots,a_{m}\}$ is the set of potential actions that can be taken on that element. For example, the potential actions for a search bar might be $\{\texttt{click},\texttt{type}\}$; a non-interactive text element has the action set $\varnothing$.
- •
$S_{j}$ represents the set of other environment-specific attributes for element $e_{j}$. These attributes can include image captions, type labels (e.g., Button, Text Field), or accessibility information.

While certain environments may provide full access to this environment state,
here we are focus on environments where only the pixel information, $\mathcal{P}$, is available.
We then must predict elements from the pixel information to construct a state representation for the agent.
In other words, we must find must find a function $g\colon\mathcal{P}\rightarrow\mathcal{E}$

Elements can be represented in both visual and text modalities.
For images, the most common approach is to overlay bounding boxes with numeric identifiers around each interactable element (Koh et al., ; He et al., ; Yan et al., ) in a manner inspired by Set-of-Mark Prompting (Yang et al., ).
In text, a common approach is to represent each element as a string containing its index, coordinates, and other attributes, such as "[1] [x,y] [Description]".
Because LMs operate on sequential data, elements must be given an ordering;
in most approaches, this is implicitly defined by the method used to identify the elements.

#### Ordering.

The ordering is defined as a permutation $\sigma$ of the indices $\{1,2,\ldots,n\}$ where $\{e_{\sigma(1)},e_{\sigma(2)},\ldots,e_{\sigma(n)}\}$ represents a specific sequence of elements.
An ordering function is a function $f\colon\mathcal{E}\rightarrow\sigma$ that takes the environment as an input and yields a specific ordering $\sigma$.

## 4 Which Aspects Of Agent State Representations Are Most Impactful?

Here we describe a series of ablation experiments designed to examine which aspects of an LM state representation are most impactful to the performance of LM agents.
In particular, we experiment on the VisualWebArena (VWA) (Koh et al., ) benchmark and ablate attributes of the state representation of the state-of-the-art agent (proposed in the same paper).
We pick this representation in particular due to both its strong performance on VWA, as well as its similarity to common practices seen in other agent research (He et al., ) and open source projects (Ishan0102, ; Dupont, ).
Our experiments in turn ablate the impact of
(1) multimodal (image and text) aspects of the state representation, and
(2) individual element attributes within the text component alone.

### 4.1 VisualWebArena

VisualWebArena focuses on multimodal tasks in the web and provides a self-hostable environment for language agents to navigate (Koh et al., ).
Agents operating on VisualWebArena have full access to the DOM.
The current best approach (Koh et al., ) utilizes a multimodal representation where elements are parsed in a pre-order traversal of the DOM tree (World Wide Web Consortium, ). Each element $j$ has attributes
$S_{j}=\{id,tag,text\}$ where $id$ is a unique numeric identifier for interactable elements and $\varnothing$ otherwise, $tag$ is the HTML tag (e.g. BTN or IMG), and $text$ is the alt text and captions for images and the HTML text otherwise.
In the text representation, an example of an element would be “[1] [IMG] [alt text, caption]”.
In the image representation, each element is labeled with bounding boxes and numeric labels.

The original agent in (Koh et al., ) only achieves 15% success rate across all tasks.
Since our goal is not to improve agent performance on VisualWebArena but rather to understand the importance of attributes in the state representation,
we examine a subset of tasks to reduce costs.(^0^00A full run of the state-of-the-art agent on VisualWebArena can cost up to $800 with GPT-4V.)
Specifically, we explore tasks marked as “easy” within tasks that the original agent completed successfully.
Due to variance associated with stochastic LM outputs, our reproduction of these originally-successful tasks yields a success rate around $74.07\%\pm 5.56\%$.
The exact list of tasks can be found in Appendix A.4.
We reuse the action space from the original agent which consists of executing high-level actions (e.g., click, hover) on individual elements—see Appendix A.2 for more details.

### 4.2 Ablation Setup and Findings

The agent state representation we explore is multimodal and consists of image and text information.
The image consists of a screenshot of the webpage along with Set-of-Mark annotations, while the text consists of a DOM-ordered list of elements with the attributes outlined above.
Our ablation protocol consists of removing individual attributes from the image or text representation and measuring task success rate—we say an attribute has high “impact” if its removal leads to substantial reduction in task success rate.
To provide evidence that these findings may be robust across different LM backbones,
we explore both GPT-4V as used in the original agent,
and Gemini 1.5 Pro.
Some experiments were not run on GPT-4V due to high associated costs,
though we found ordering to be consistent between these two LM backbones on all ablations where we ran both.

In Table 1, we report results ablating aspects of the multimodal representation.
In Table 2, we report the impact of ablating various attributes in the text representation specifically.
Across all experiments, we consider the pre-order traversal of the DOM tree as the ground truth element ordering, and define the “removal” of ordering information as substituting an ordering $\sigma_{rand}$ picked uniformly at random from all possible permutations.
We summarize a few key findings from both sets of ablations below.

**Table 1: Ablating the multimodal aspects of state representation in VisualWebArena. ✓ indicates ground truth obtained from the HTML. x indicates removal of the attribute. We find that removing the text representation can dramatically harm agent performance.**
| Observations | Success Rate ($\uparrow$) |  |  |  |
| --- | --- | --- | --- | --- |
| Screenshot | Set-of-Mark | Text Representation | Gemini 1.5 | GPT-4V |
| ✓ | ✓ | ✓ | 64.20% | 74.07% |
| x | x | ✓ | 46.30% | 38.89% |
| ✓ | x | ✓ | 50.00% | - |
| ✓ | ✓ | x | 3.70% | 38.89% |

#### Text Representation is Still Necessary.

While adding a visual representation clearly improves performance, we find that it alone is insufficient even with Set-of-Mark labels.
This contradicts previous findings on agents for mobile applications which found that a screenshot with Set-of-Mark labels achieves similar performance with or without text (Yan et al., ).
We
speculate
that this is due to the
substantial
difference in viewport sizes between mobile and desktop environments.
Specifically, the average mobile device has a viewport size of 360x800 while the average desktop has a viewport size of 1920x1080 (Statcounter Global Stats, ).
Additionally, larger viewport sizes have been shown to improve agent performance in desktop environments (Xie et al., ).
We speculate that this may be because
current agents almost never understand when to change the screen view (e.g. by scrolling).

**Table 2: Ablating attributes of the text component of the VisualWebArena state representation. All results include the screenshot with Set-of-Mark bounding boxes and labels. TAG is the HTML tag. CAPTIONS are image captions generated using BLIP-2-T5XL(Li et al., ). TEXTAlt, TEXTInteract, and TEXTStatic are the alt text, text for interactable elements, and text for non-interactable elements respectively. ORDER is element ordering. ✓ indicates ground truth obtained from the HTML. x indicates removal of the attribute. x Element Ordering indicates a random shuffling of the elements. - denotes experiments that were not run due to cost.**
| Element Attributes | Success Rate ($\uparrow$) |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| TAG | CAPTIONS | TEXTAlt | TEXTInteract | TEXTStatic | ORDER | Gemini 1.5 | GPT4-V |
| ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | 64.03% | 74.07% |
| x | ✓ | ✓ | ✓ | ✓ | ✓ | 61.11% | 61.11% |
| ✓ | x | ✓ | ✓ | ✓ | ✓ | 46.30% | - |
| ✓ | ✓ | x | ✓ | ✓ | ✓ | 68.15% | 66.67% |
| ✓ | ✓ | ✓ | x | ✓ | ✓ | 53.70% | - |
| ✓ | ✓ | ✓ | ✓ | x | ✓ | 57.40% | - |
| ✓ | ✓ | ✓ | x | x | ✓ | 35.18% | - |
| ✓ | ✓ | ✓ | ✓ | ✓ | x | 37.04% | 44.44% |

#### Removing Ordering Information Harms Performance More Than Removing Any Other Attribute.

Although most element attributes are important, we find that ordering is the single most important attribute for agent performance. Random ordering results in a similar performance drop to removing all HTML text descriptions.

#### Captions Impact Performance More Than Alt Text.

Removing captions causes a greater decrease to performance than removing alt text.
From our experience, captions almost always provide more information than alt text.
In fact, captions frequently include the alt text directly in its description.

## 5 Experimental Setup

For the remainder of this paper, we leverage the insights gained from our state representation ablations on VisualWebArena to tackle a more challenging task: enabling LM agents to act in environments that only expose pixel information.

Most applications are built on top of an underlying hierarchical representation.
For example, a webpage is modeled by the DOM which is hierarchical.
When exposed, this hierarchy can be used to determine a strong element ordering.
However, the availability and quality of an underlying hierarchical representation can vary greatly between environments.
For example, Chen et al. () found that 77% of mobile applications had missing labels and Ross et al. () found that 53% of image buttons had missing labels and were incorrectly sized.
Additionally, Ross et al. () found that
$8\%$ of
applications lacked interactable element information altogether.
In such scenarios, we may only have access to the application’s pixel information.
We continue to experiment with VisualWebArena and also experiment with the OmniACT benchmark as a scenario where we only have access to pixel information. Details on the LM agent backbones used in our experiments can be found in Appendix A.3.

### 5.1 OmniACT

OmniACT provides both web and desktop environments for agents to benchmark on.
OmniACT contains 177 application screenshots overall and 2021 tasks in the test set.
Agents are tasked with generating pyautogui code that can navigate the application screenshot.
We consider OmniACT as a setting where only a pixel information is available.

To detect UI elements $\{e_{1},e_{2},\ldots,e_{n}\}$ when given only pixel information $\mathcal{P}$ we train an object detection model (Ren et al., ) to detect interactable UI elements in the screenshot and use EasyOCR (AI, ) to extract visible text.
In other words, the function $g\colon\mathcal{P}\rightarrow\mathcal{E}$ is defined by the trained object detection model.
We add visible text and captions to each interactable UI element.
We gather a dataset by finding 67,530 interactable elements over 1468 Common Crawl webpages.
We selected our webpages based on top websites from Similarweb ().
Despite the domain shift from webpages to desktop applications, we found that our object detection model worked reasonably well on the OmniACT benchmark in the end-to-end agent setting.
We publicly release this model along with our paper.
Training details can be found in the Appendix A.1.

OmniACT provides partial human annotations for each screenshot; multiple, but not all, interactable UI elements are annotated with bounding boxes.
The original intent of these bounding boxes is for evaluation only.
As a result, there are significantly less UI elements annotated compared to possible UI elements in the application.
We experiment with elements derived from these human annotated bounding boxes to understand a) impacts to ordering performance in an easier setting and b) the potential performance that can be gained by improving UI element detection.

### 5.2 Metrics

Our primary metric is task success rate which is the standard for agent evaluations (Koh et al., ; Zhou et al., ; He et al., ).
VisualWebArena provides an evaluation framework for task success.
Task success criteria include achieving an expected final webpage state or receiving a desired response from the language agent.
OmniACT does not provide task success rate directly, instead introducing the sequence score and action score metrics.
Sequence score evaluates if the output contains the correct high level action (e.g. click or type), but does not check if the action element or parameter (e.g. click [1] or type [parameter]) are correct.
Action score evaluates if the output contains both the correct high level action and the correct element or parameter.
Thus, for OmniACT we focus our evaluations on the action score as it is more similar to task success rate.

### 5.3 Ordering Methods

In addition to random ordering, we experiment with two different ordering methods.

#### Random.

We pick the ordering $\sigma_{rand}$ uniformly at random from all possible permutations. This provides a baseline performance.

#### Raster.

Elements $\{e_{1},e_{2},\ldots,e_{n}\}$ are ordered in a left-to-right raster scan. We define a raster scan as an ordering $\sigma_{raster}$ where $i<j$ iff $\lfloor\frac{y_{i}}{8}\rfloor<\lfloor\frac{y_{j}}{8}\rfloor$ and $x_{i}<x_{j}$. We chose to discretize the scan to prevent jumps in ordering from minor pixel variations.
This method mimics the natural way English speakers read text and images from left-to-right and top-to-bottom.

#### t-SNE.

We apply dimensionality reduction techniques to better capture the spatial relationships. Using t-SNE (Van der Maaten & Hinton, ), we reduce the dimensionality with the function $g\colon\langle x,y\rangle\rightarrow z$.
The set of values $Z=\{z_{1},z_{2},\ldots,z_{m}\}$ is used to determine the ordering $\sigma_{tsne}$. We use the scikit-learn (Pedregosa et al., ) implementation of t-SNE and keep the default parameters.

Our intuition for choosing t-SNE stems from visually and qualitatively inspecting our agent trajectories. We observed that agents often use adjacently ordered elements as context clues to determine the correct action. Furthermore, agents have difficulty reasoning about functionally associated elements that are separated from each other in the ordering. t-SNE generates an effective ordering as elements that are visually close together (i.e. nearby in 2D pixel space) are often functionally associated with each other as well. When reducing dimensions, t-SNE retains local structure which increases the odds that functionally associated elements are adjacent in the induced 1D ordering.

### 5.4 Action Space

**Table 3: The set of possible actions in OmniACT.**
| Action | Description |
| --- | --- |
| Click | Perform a single click on an element. |
| Double Click | Perform a double click on an element. |
| Right Click | Perform a right click on an element. |
| Move/Hover | Move the cursor over an element. |
| Drag | Click and drag an element to a new position. |
| Scroll | Scroll up or down the page. |
| Horizontal Scroll | Scroll left or right on the page. |
| Press | Press a key on the keyboard. |
| Keyboard Hotkey | Use a keyboard shortcut or hotkey. |
| Write | Type text using the keyboard. |

## 6 Results

Our main findings on the impact of ordering are in Table 4.
We utilize our various findings to improve upon OmniACT; our experiments against their baseline are in Table 6.

**Table 4: Performance of different ordering methods across various models and information scenarios. The baseline approach for VisualWebArena is the same as their paper. Human annotations are from OmniACT’s annotation files. The Faster-RCNN model is trained to detect interactable UI elements from CommonCrawl webpages. VisualWebArena is evaluated on success rate. OmniACT is evaluated on unweighted action score (i.e. each task is weighted equally). We use Llama3-70B for VisualWebArena and Llama3-8B for OmniACT due to its large size(^2^22We use the Groq API for our Llama3 models.). GPT-4v is evaluated on a 100 random tasks for OmniACT; the exact list is in Appendix A.4. Gemini-1.5 and GPT-4v are multimodal while Llama3 is text only.**
| Experimental Settings | Success Rate |  |  |  |  |
| --- | --- | --- | --- | --- | --- |
| Element Source | Benchmark | Ordering Method | Gemini-1.5 ($\uparrow$) | GPT-4v ($\uparrow$) | Llama3 ($\uparrow$) |
| Ground Truth (DOM) | VWA | Pre-order | 64.03% | 74.07% | 27.79% |
|  |  | Random | 37.04% | 37.04% | 20.37% |
|  |  | Raster | 38.88% | 53.70% | 29.63% |
|  |  | t-SNE | 44.44% | 61.11% | 24.07% |
| Human | OmniACT | Random | 57.29% | 61.52%^∗ | 28.67% |
| Annotations |  | Raster | 61.04% | 65.88%^∗ | 33.65% |
|  |  | t-SNE | 59.17% | 62.11%^∗ | 31.99% |
| Detected | OmniACT | Random | 39.59% | 44.63%^∗ | 18.88% |
| (Faster-RCNN) |  | Raster | 45.21% | 47.38%^∗ | 21.58% |
|  |  | t-SNE | 47.16% | 49.18%^∗ | 24.61% |

### 6.1 Impact of Ordering

#### Ordering Consistently Impacts Performance.

Ordering
has a significant impact to performance across all of our experiments.
Random ordering decreases performance in VisualWebArena by 50% and 42% relative performance for GPT-4v and Gemini-1.5 respectively.
In all experiments, random ordering decreases performance over a proper ordering method.

#### t-SNE Best For Larger Models And More Challenging Tasks.

Navigating by using detected elements is a harder task than navigating by using human annotated bounding boxes;
not only are there more elements—on average double the amount—there is the possibility that the correct element is missing from the detected elements.
We see that when elements are derived from the DOM and our UI detection model, t-SNE ordering generally outperforms raster ordering.
Additionally, more powerful models see an increased benefit from t-SNE ordering with Gemini-1.5 and GPT-4v seeing larger improvements than LLama3.

#### Raster Ordering Performs Best With Human Annotations.

Raster ordering performs the best with human annotated elements.
Unfortunately, these annotations are fewer and partially guaranteed to contain important information.
Additionally, high quality human annotations are difficult to scale across applications.

Ordering Impact Increases with Element Count As element count increases, we observe that the impact of ordering increases as well. This is consistent across all datasets, although VisualWebArena shows the greatest discrepancy. Our results on Gemini are in Figure 3 and the remainder are in the Appendix.

Figure: Figure 3: Differences in reward from random ordering. Element counts are divided based on the median. Across all datasets and element sources, the impact of a proper ordering method increases with element count.
Refer to caption: extracted/5904312/imgs/element_count_gemini.png

Findings:

- 1.
Tag and Alt Text are more important for better models.
- 2.
Captions, text for interactive elements, and text for static elements are important.
- 3.
Ordering of elements is surprisingly important.

I feel like I need another model here. Maybe an OSS one since GPT-4V is expensive.

Findings:

- 1.
The accessibility tree is the most important.
- 2.
Set-of-Mark labels are important. It seems as though Set-of-Mark is what enables GUI understanding. I did an experiment with a white background and bounding boxes. I noticed almost no difference from having the actual screenshot + set-of-marks. Should I write about that?
- 3.
Our approach is not able to reach the same performance as with full Accessibility Tree info.

### 6.2 State-of-the-Art Performance on OmniACT

We achieve a new state-of-the-art performance on Omniact.
Due to cost, we look to our previous experiments to pick the best combination of features for our approach.
We observe that multimodal representations are still helpful.
We find that t-SNE ordering improves performance the best in OmniACT when elements are detected by our model.
Koh et al. () states that high level actions are easier for a LM to reason with.

We analyze the differences between our best approach and OmniACT’s baseline.
These are as follows.

- •
Element Source:
OmniACT obtains elements by searching for icons with Segment Anything (Kirillov et al., ) and text with EasyOCR. Unfortunately, there are no shared artifacts for their icon detection system.
We obtain UI elements through an object detection model and text with EasyOCR (AI, ).
- •
Ordering:
It is unclear how elements are ordered in OmniACT. Considering how most approaches don’t pay specific attention to ordering, we assume the ordering in OmniACT is effectively random.
We order our elements using our t-SNE ordering.
- •
Action Space:
OmniACT directly outputs pyautogui code as their actions.
We consider a higher level action space that maps to pyautogui code.
- •
Intractability OmniACT lists out each element, but does not indicate which element is interactable. We specify whether elements are interactable or not.
- •
Multimodal Representation: OmniACT evaluates their full test set using a text only representation(^3^33OmniACT evaluates the impact of adding a visual representation on smaller subset; however, this subset is not shared and varies empirically from the full test set.). We experiment with a multimodal representation.

We apply our findings and achieve more than one-fold increase over the existing best action score. Our results can be found in Table 6.

**Table 5: GPT-4 with various action spaces and ordering methods. Actions indicates actions such as CLICK [1] which are converted to pyautogui code. Code indicates an action space where pyautogui code is directly generated. Experiments are run on a random sample of 100 tasks. t-SNE ordering provides a significant improvement to performance over random when directly generating pyautogui code.**
| Action Space | Ordering | Screenshot | Action Score ($\uparrow$) | Sequence Score ($\uparrow$) |
| --- | --- | --- | --- | --- |
| Code | Random | x | 15.89 | 21.14 |
|  | Random | ✓ | 25.16 | 33.12 |
|  | t-SNE | x | 27.80 | 35.43 |
|  | t-SNE | ✓ | 24.64 | 29.51 |
| Actions | Random | x | 13.83 | 18.02 |
|  | Random | ✓ | 19.86 | 25.55 |
|  | t-SNE | x | 15.03 | 20.86 |
|  | t-SNE | ✓ | 25.04 | 29.39 |

**Table 6: ✓ and x indicates whether the feature is available when building a representation for the model. Ours indicates a high level actions such as click [1]. Code indicates that pyautogui code is directly generated. bold is best and italics are second best. While sequence score only checks for the correct high level action (e.g. click), action score checks for both the correct action and the correct element or parameter (e.g. click [1]). Thus, action score is the most equivalent to task success rate. * is as reported in Kapoor et al. ()**
| Model | State | Actions | Screenshot | Action Score ($\uparrow$) | Sequence Score ($\uparrow$) |
| --- | --- | --- | --- | --- | --- |
| GPT-4 | OmniACT | Code | x | 11.60* | 32.75* |
| Gemini 1.5 | Ours | Code | x | 16.53 | 21.67 |
| Gemini 1.5 | Ours | High Level | x | 22.29 | 29.42 |
| Gemini 1.5 | Ours | High Level | ✓ | 22.86 | 28.91 |
| Llama8b | Ours | High Level | x | 18.64 | 26.22 |
| GPT-4v | Ours | High Level | ✓ | 23.34 | 30.47 |

## 7 Future Work

#### Further Improvements to Ordering

We provided two simple methods to apply ordering when a default ordering is not given.
However, both approaches still fall short when compared to the hierarchical ordering derived from the DOM.
We hope that future research can introduce more sophisticated methods to find ordering with only pixel information.

#### Image Only Ordering

We focused on the impact of element ordering in a text representation (although element labels re-ordered accordingly in the visual representation as well).
The impact of element ordering may or may not generalize to an image only representation.
Unfortunately, our results indicated that a visual representation alone was insufficient for web and desktop environments which prevents us from conducting this experiment.
However, Yan et al. () found that a visual representation in mobile environments was able to achieve comparable performance with and without a text representation.
In the future, we hope to experiment with various ordering methods on an image only representation.

#### Expanded Scenarios and Benchmarks

In this paper, we explored two benchmarks—VisualWebArena and OmniACT—as web and desktop scenarios.
In the future, we hope to explore other benchmarks and settings with our approach.
For example,  Xie et al. () uses the OS level accessibility tree for desktop agent navigation.
We hope to compare our approach against theirs and believe that combining both approaches may lead to further improvements.
Additionally, mobile environments often have only pixel-level information (Chen et al., ; Ross et al., ); we hope to apply our approach to a mobile benchmark such as Rawles et al. ().

## 8 Conclusion

We conducted thorough ablations to show that element ordering has a significant impact on the performance of agents.
We provided a method of ordering elements through dimensionality reduction and showed that it performed best in realistic environments.
We trained a UI element detection model on Common Crawl data and publicly share the model.
We demonstrated an end-to-end method which allows a LM agent to act on environments that provide only pixel information.
Using this method, we were able to achieve a new state-of-the-art performance on OmniACT.

## Appendix A Appendix

### A.1 Faster-RCNN Training Details

We trained our model using the detectron2 Wu et al. () implementation of faster-rcnn. We did not change much from the default implementation and recognize that there are significant improvements that could be made to the model.

**Table 7: Key hyperparameters for the Faster-RCNN model.**
| Hyperparameter | Value |
| --- | --- |
| Base Learning Rate | 0.00025 |
| Number of Classes | 1 |
| Iterations | 200000 |
| Optimizer | SGD |
| Backbone | Resnet-50 (ImageNet Pretrained) |
| ResNet Depth | 50 |
| Images per Batch | 16 |
| Objects per Image | 128 |
| Devices | 8 |

We share the remaining hyperparmeters in a config file.
We also share the model artifacts and dataset.

### A.2 VisualWebArena Agent Action Space

We use the same action space as VisualWebArena for all VisualWebArena experiments.

**Table 8: The set of possible actions in VisualWebArena.**
| Action | Description |
| --- | --- |
| click [id] | Click on element id. |
| hover [id] | Hover on element id. |
| type [id] [text] | Type text on element id. |
| press [key_comb] | Press a key combination. |
| new_tab | Open a new tab. |
| tab_focus [index] | Focus on the i-th tab. |
| tab_close | Close current tab. |
| goto [url] | Open url. |
| go_back | Click the back button. |
| go_forward | Click the forward button. |
| scroll [up|down] | Scroll up or down the page. |
| stop [answer] | End the task with an optional output. |

### A.3 LM Agent Hyperparameters and Settings

**Table 9: Settings for different LM agent backbones.**
| Setting | Language Model Backbone |  |  |
| --- | --- | --- | --- |
|  | GPT-4v | Gemini-1.5 | Llama3 |
| Input Token Limit | 3840 | 900000 | 3840 |
| Temperature | 1.0 | 1.0 | 1.0 |
| Top-p | 0.9 | 0.9 | 0.9 |

### A.4 VisualWebArena and OmniACT Subset

We experimented with subsets of VisualWebArena and OmniACT to save on costs. We list them here for reproducibility.

For all VisualWebArena experiments we used the following:

[13, 15, 50, 129, 164, 167, 0, 77, 86, 89, 98, 99, 100, 101, 105, 130, 131, 142, 143, 146, 150, 189, 16, 29, 37, 38, 39, 47, 49, 52, 53, 56, 60, 61, 62, 69, 73, 76, 77, 81, 148, 173, 193, 196, 201, 212, 216, 231, 273, 314, 315, 322, 445]

For GPT-4v ordering ablations on OmniACT we used the following:

[4, 58, 115, 147, 156, 162, 165, 178, 179, 194, 204, 218, 235, 240, 248, 297, 353, 374, 391, 392, 395, 404, 409, 419, 434, 462, 487, 492, 517, 533, 556, 573, 598, 658, 667, 673, 678, 719, 795, 827, 896, 910, 944, 961, 975, 1018, 1025, 1038, 1084, 1093, 1101, 1103, 1128, 1130, 1138, 1142, 1147, 1181, 1192, 1219, 1252, 1284, 1291, 1353, 1427, 1442, 1448, 1514, 1521, 1538, 1580, 1590, 1594, 1600, 1606, 1622, 1636, 1641, 1665, 1684, 1694, 1696, 1710, 1711, 1719, 1726, 1731, 1740, 1743, 1845, 1877, 1883, 1918, 1924, 1951, 1960, 1993, 1994, 1997, 2011]

### A.5 Prompt

Figure: Listing 1: Language Model Prompt

### A.6 Full Omniact Experiments

**Table 10: Comparison of different approaches and models on performance metrics. 500 samples. Random Seed 777**
| Approach | Model | Sequence Score | Action Score |
| --- | --- | --- | --- |
| Omniact | GPT-4V | 39.43 | 20.76 |
| Ours (Actions) w/ GT BBox | Gemini 1.5 | 28.23 | 21.67 |
| Ours (Actions) w/ GT BBox | GPT-4V | 32.00 | 24.81 |
| Ours (Actions) w/ GT BBox and Short | Gemini 1.5 | 22.8 | 17.29 |
| Ours (Pyautogui) w/ GT BBox | Gemini 1.5 | 28.28 | 22.38 |
| Ours (Actions) w/ GT BBox and filtering | Gemini 1.5 | 25.75 | 19.10 |
| Ours (Actions) w/ GT BBox and text interaction | Gemini 1.5 | 26.35 | 20.59 |

**Table 11: Comparison of different approaches and models on performance metrics. 500 samples. Random seed 77.**
| Approach | Model | Sequence Score | Action Score |
| --- | --- | --- | --- |
| Omniact | GPT-4V | 39.43 | 20.76 |
| Ours (Actions) w/ GT BBox | Gemini 1.5 | 30.45 | 22.75 |
| Ours (Actions) w/ GT BBox and Emphasize Ordering | Gemini 1.5 | 33.53 | 25.16 |
| Ours (Actions) w/ GT BBox, Emphasize Ordering, OCR + Captions | Gemini 1.5 | 31.92 | 25.34 |
| Ours (Actions) w/ VWA BBox, Emphasize Ordering, OCR + Captions | Gemini 1.5 | 27.41 | 18.55 |

### A.7 Table with Scaled Values

**Table 12: Performance of different ordering methods across various models and information scenarios. The baseline approach for VisualWebArena is the same as their paper, the baseline approach for OmniACT* is taken from their annotation files, and the baseline approach for OmniACT is the ordering given by our object detection model. VisualWebArena is evaluated on success rate. OmniACT* and OmniACT are evaluated on action score. We use Llama3-70B for VisualWebArena and Llama3-8B for OmniACT* and OmniACT.**
| Scenario | Dataset | Ordering Method | Gemini-1.5 ($\uparrow$) | GPT-4v ($\uparrow$) | Llama3 ($\uparrow$) |
| --- | --- | --- | --- | --- | --- |
| Object | VWA | Baseline | 64.03% | 74.07% | 27.79% |
| Model |  | Random | 37.04% | 44.44% | 20.37% |
|  |  | Raster | 37.04% | 44.44% | 20.37% |
|  |  | t-SNE | 44.44% | 61.11% | 24.07% |
| Human | OmniACT | Random | 57.29%(25.31) | - | 28.67%(17.34) |
| Annotations |  | Raster | - | - | - |
|  |  | t-SNE | 59.17%(25.01) | - | 31.99%(17.95) |
| Screenshot | OmniACT | Random | 39.59%(20.27) | - | 18.88%(16.84) |
| Only |  | Raster | - | - | - |
|  |  | t-SNE | 47.16%(22.86) | - | 24.61%(18.64) |

### A.8 Sequence Score Values

**Table 13: ✓ and x indicates whether the feature is available when building a representation for the model. $\triangle$ indicates the feature is predicted using our object detection model. Actions indicates actions such as CLICK [1] which are converted to pyautogui code. Code indicates an action space where pyautogui code is directly generated.**
| Approach | Model | Action Space | Annotations | Screenshot | Sequence Score ($\uparrow$) | Action Score ($\uparrow$) |
| --- | --- | --- | --- | --- | --- | --- |
| Ours | llama8b | Code | ✓ | x | 8.25* | 5.26* |
| Ours | Gemini 1.5 | Code | ✓ | ✓ | 28.43 | 23.8 |
| Ours | llama8b | Actions | ✓ | x | 23.19 | 17.95 |
| Ours | Gemini 1.5 | Actions | ✓ | ✓ | 28.82 | 25.01 |
| Ours | Gemini 1.5 | Actions | ✓ | x | 32.07 | 27.00 |
| Omniact | GPT-4 | Code | N/A | x | 32.75 | 11.6 |
| Ours | Llama8b | Actions | $\triangle$ | x | 26.22 | 18.64 |
| Ours | Gemini 1.5 | Actions | $\triangle$ | ✓ | 28.91 | 22.86 |
| Ours | Gemini 1.5 | Actions | $\triangle$ | x | - | - |
| Ours | GPT-4V | Actions | $\triangle$ | ✓ | 30.47 | 23.34 |
| Ours | GPT-4 | Actions | $\triangle$ | x | - | - |

## Peer Reviews

confidence: 5
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. How does element ordering affect results at different orders of magnitude (e.g., fewer than 10 elements vs. more than 50 elements)?

2. While the ablation section examines how different inputs affect experimental results, to draw more convincing conclusions, the authors should conduct additional ablation experiments on datasets beyond VisualWebArena.
soundness: 4
strengths: 1. The paper provides the first in-depth investigation of how element ordering affects agent performance and demonstrates its significance.

2. The proposed dimensionality reduction-based ordering method performs well in pixel-only scenarios and achieves new state-of-the-art performance on the OmniACT agent benchmark.

3. The paper introduces a UI element detection model which has been made publicly available for other researchers to use and improve upon.
summary: This paper investigates language model agents' navigation capabilities in virtual environments and focuses on exploring how element ordering affects agent performance. The paper's main finding is that element ordering has a significant impact on agent performance, with random ordering leading to substantial performance degradation - comparable to the effect of completely removing all visible text from web pages. While web-based agents can benefit from the semantic hierarchy provided by browsers, agents that parse elements directly from pixels cannot access this ordering information. The paper proposes an ordering method based on dimensionality reduction (such as t-SNE) that performs well in scenarios where only pixel information is available.
weaknesses: 1. While the paper focuses on the t-SNE dimensionality reduction ordering method, it lacks in-depth analysis and comparison with other ordering methods. Additionally, all ordering methods show significant performance gaps compared to Pre-ordering (Table 4).

2. The paper briefly introduces the training process of the UI element detection model but lacks more detailed specifics.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: Please refer to the weaknesses section above for specific points that require clarification or further detail.
soundness: 2
strengths: 1. Reveals the Critical Impact of Element Order on Agent Performance: Through systematic experiments, the paper effectively demonstrates the significant influence of element ordering on language model agents operating in pixel-only environments. This finding presents a new perspective for developing efficient virtual environment navigation algorithms. Previous research has often concentrated on accuracy in image recognition and text analysis, overlooking the importance of element order in contextual understanding. This paper addresses this gap by introducing order optimization as a novel method for enhancing agent performance.

2. Provides an Ordering Method in Environments Lacking Hierarchical Information: In situations where structural information (e.g., HTML or DOM trees) is not available, traditional methods struggle to identify an optimal element order. By utilizing t-SNE for dimensionality reduction, the paper effectively maps pixel-level interface elements to a one-dimensional sequence, establishing a practical ordering method for pixel-only environments. Despite its simplicity, this approach achieves performance gains in pixel-based contexts, setting new performance benchmarks on platforms like OmniACT.
summary: This paper investigates the impact of element ordering on the performance of language model agents. The empirical study highlights that the sequence in which UI elements are presented to the agent significantly influences performance, especially for elements represented solely by pixels, where traditional hierarchical ordering is unavailable.

The authors propose an ordering method based on dimensionality reduction, necessitating the training of a UI element detection model to extract elements from pixel data. This approach offers a viable solution for ordering in pixel-only environments. The proposed method consistently demonstrates performance improvements over random ordering and achieves a new state-of-the-art result on the OmniACT benchmark.
weaknesses: 1. While the baseline of random ordering is understandable as detrimental to large language models in interpreting UI, it is overly simplistic and is not a strong baseline. The study would benefit from incorporating more heuristic baselines. For instance, could a vision-language model, such as GPT-4V, assist in determining an optimal ordering when seeing the UI directly?

2. In Table 6, the proposed method consistently outperforms other ordering techniques only when elements are detected using Faster R-CNN, failing to demonstrate consistent advantages in other scenarios.

3. Although the paper validates the effectiveness of various ordering methods (e.g., t-SNE ordering surpassing random ordering), it lacks a systematic analysis of why specific ordering methods enhance performance. The authors primarily illustrate the effects of ordering through experiments but do not investigate the mechanisms by which different strategies influence the language model's understanding and contextual construction.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: - The paper mentions using default parameters for t-SNE, but this method can be sensitive to parameter choices, such as perplexity and learning rate. Please provide a sensitivity analysis of key t-SNE parameters (e.g., perplexity, learning rate) and their impact on ordering performance.
- While t-SNE shows promising results for ordering, it may also be beneficial to explore other dimensionality reduction techniques like UMAP and MDS. Please compare t-SNE with at least one other dimensionality reduction technique (e.g., UMAP) on a subset of data, and report the relative performance in terms of agent task success rate.
- While the appendix provides training details for the UI detection model, the paper would benefit from additional performance metrics specific to this scenario. Please provide a breakdown of the training data composition and to report related object detection metrics (e.g., mAP, precision-recall curves) for the UI detection model.
- Tables 4 and 5 report key results on different ordering methods, yet they lack detailed context, such as specific configurations or experimental conditions for each row. Could the authors expand on these tables with additional descriptions or footnotes? For example, is the trained model used for UI detection consistent across all experiments, does the success rate and action score reflect the same metric on OmniACT, etc.
soundness: 3
strengths: - Originality: The paper tackles a relatively unexplored area—optimizing UI element ordering for LM agents in pixel-only environments. The use of t-SNE for ordering based on spatial relationships is a novel application, offering a fresh perspective on improving agent navigation performance.
- Quality: The research employs ablation studies on VisualWebArena and OmniACT. The methodological depth and comparison across multiple ordering methods highlight the improvement of the approach.
- Clarity: The paper presents the problem and solution clearly, with intuitive descriptions of complex concepts like dimensionality reduction for ordering.
- Significance: Achieving state-of-the-art performance on OmniACT, the paper has practical implications for LM agent design in unstructured environments. The release of the trained UI detection model will further enhances its contribution to the research community.
summary: The paper explores the effect of element ordering on LLM agent performance in virtual environments like web and desktop. Through ablation studies on the VisualWebArena and OmniACT benchmarks, they find that random element ordering significantly degrades performance, similar to removing all textual information. To address this, the authors propose using dimensionality reduction techniques, specifically t-SNE, to create effective element orderings based on spatial relationships in pixel-based environments. The authors contribute a trained UI detection model and achieve state-of-the-art performance on OmniACT, demonstrating the impact of ordering strategies for agent navigation in pixel-only environments.
weaknesses: - The paper could clarify its discussion of the dimensionality reduction approach, specifically addressing the parameters used in t-SNE. As t-SNE can be sensitive to parameter tuning, a detailed analysis of how parameter choices affect ordering outcomes would strengthen the validity of the results.
- Another weakness is the limited exploration of alternative ordering methods. While t-SNE provides performance gains, the study would benefit from a broader examination of ordering techniques, especially methods more aligned with semantic or functional groupings, which could be valuable in complex UI layouts where spatial proximity does not equate to functional relevance. 
- While the authors release the trained UI detection model, the study would benefit from greater transparency in terms of training data diversity and model performance metrics, such as precision and recall for UI element detection, to better assess its effectiveness across domains.
- The ablation study lacks rigorous control over experimental variables, which makes it difficult to isolate the exact impact of individual attributes on agent performance. For instance, the experiments appear to vary multiple aspects of the state representation without consistently controlling. A more systematic approach to ablation studies would enhance the credibility of the findings.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: I am not very familiar with this field, so I will adjust my rating after reading other reviewers' feedbacks. 

Please see weaknesses W1 to W4. Writing mistakes do not need a response.
soundness: 3
strengths: 1. The investigated problem (ie, ordering’s effects) is somewhat novel.
2. The findings that ordering can influence the effects are intuitive and consistent with human perception.
3. The paper is presented clearly and easy to follow.
summary: This manuscript investigates the influence of ordering (in which elements are presented) to the performance of language model agents on navigating virtual environments, and proposes two simple ordering methods (raster and t-SNE) that are better than random ordering.
weaknesses: W1. Some conclusions are not well supported by experimental results.
- 1.1 In L299, the manuscript claims that “Random ordering results in a similar performance drop to removing all HTML text descriptions”. However, random ordering results in 37.04% (Gemini 1.5) and 44.44% (GPT4-V) in Table 2, while removing all texts brings 3.7% (Gemini 1.5) and 38.89% (GPT4-V) in Table 1. The performance gap between these two is still large.
- 1.2 In the Introduction, the manuscript claims that the t-SNE ordering improves performance. However, in Table 4, t-SNE is worse than raster in (a) VWA with Llama3, and (b) OmniACT with human annotations. Here, 4 out of 9 metrics, almost half, show raster is better than t-SNE. 

W2. The analyses and explanations of ordering’s effects are not sufficient. The manuscript notices that t-SNE is worse than raster in OmniACT with human annotations and provides these results in L455. However, why raster is better than t-SNE with human annotations has not been further analyzed.

W3. The necessity of using t-SNE to reduce dimension from 2D to 1D is not well stated. It is well known that t-SNE is more suitable for high-dimensional features, but the dimension of the original space is small (ie., 2D). In such case, using t-SNE will roughly equal the process that (1) choosing the top-left one element A, (2) finding its nearest neighbor element B, (3) then finding the nearest neighbor of both element A and element B (with a weighting strategy like exponential moving average), and so forth. 

W4. The experimental settings are not clear.
- 4.1 In L704, the manuscript states that only a subset of OmniACT is used in experiments. However, in Sec. 6.2, the manuscript directly compares the results with previously reported results tested on the full OmniACT. This raises concerns about the comparison results. 
- 4.2 In Table 5, the manuscript claims state-of-the-art performance. However, the sequence score is much lower than previous methods. Also, the foundational models are different. It is hard to say whether the performance difference is from the foundational models or the designs. 

W5. There are some writing mistakes.
- In L212, the words “must find” are repeated two times.
- In L461, “on Omniact” should be “on OmniACT”.
- In L465, “a LM” should be “an LM”.

Meta-review: This paper reveals that the ordering of UI elements significantly impacts language model agent performance in virtual environments - randomly ordering elements reduces performance as much as removing all visible text. The authors propose using dimensionality reduction (specifically t-SNE) to create effective element orderings in pixel-based environments, achieving state-of-the-art results on the OmniACT benchmark with more than double the task completion rate compared to previous approaches.

While the paper introduces an important consideration about element ordering in LM agent performance, it has several key limitations. The core claims about random ordering effects are not well-supported by data, and the proposed t-SNE approach shows mixed results compared to simpler baselines. The paper lacks thorough analysis of why different orderings work or fail, and the evaluation methodology makes it difficult to isolate the impact of the ordering approach. Despite raising interesting questions, the work needs substantial revision to strengthen its scientific contributions. Besides, the authors accidentally revealed their identity during the rebuttal (but the decision would be the same regardless).

Decision: Reject