OpenApps: Simulating Environment Variations to Measure UI Agent Reliability
Authors: Karen Ullrich
Jingtong Su
Claudia Shi
Arjun Subramonian
Amir Bar
Ivan Evtimov
Nikolaos Tsilivis
Randall Balestriero
Julia Kempe
Mark Ibrahim
Venue: ICLR 2026 Oral (2026)
Keywords: reinforcement learning
agents
envrionment
reliability
Primary area: datasets and benchmarks
TL;DR: We introduce a new environment, OpenApps, for generating thousands of versions of apps to test UI agent reliability.

Abstract: Reliability is key to realizing the promise of autonomous UI-agents, multimodal agents that directly interact with the apps humans use, as users must be able to trust an agent to complete a given task. Current evaluations rely on fixed environments---often clones of existing apps--- which are limited in that they can only shed light on whether or how often an agent can complete a task within a specific environment. When deployed however, agents are likely to encounter variations in app design and content that can affect an agent’s ability to complete a task. To address this blind spot of measuring agent reliability across app variations, we develop OpenApps, a light-weight open-source ecosystem with six apps (messenger, calendar, maps, etc.) that are configurable in appearance and content. OpenApps requires just a single CPU to run, enabling easy generation and deployment of thousands of versions of each app. Specifically, we run more than 10,000 independent evaluations to study reliability across seven leading multimodal agents. We find that while standard reliability within a fixed app is relatively stable, reliability can vary drastically when measured across app variations. Task success rates for many agents can fluctuate by more than 50\% across app variations. For example, Kimi-VL-3B's average success across all tasks fluctuates from 63\% to just 4\% across app versions. We also find agent behaviors such as looping or hallucinating actions can differ drastically depending on the environment configuration. These initial findings highlight the importance of measuring reliability along this new dimension of app variations.

Published as a conference paper at ICLR 2026
OPENAPPS: SIMULATING ENVIRONMENT VARIATIONS TO MEASURE UI-AGENT RELIABILITY
Karen UlIrich*!
Jingtong Su Claudia Shit! Arjun Subramonian" Amir Bar'
Ivan Eytimov! Nikolaos Tsilivis'
Randall Balestriero 133 Julia Kempe', 1.2 Mark Ibrahim"
FAIR at Meta New York University
BBrwnn University
Code: http:://cccbbkeeeeecccchhhhhhhhp
ABSTRACT
Reliability is key to realizing the promise autonomous UT-agents, multimodal agents that directly interact with the apps humans use, users must be able to trust an agent to complete a given task. Current evaluations rely on fixed onvironments-ooteen clones of existing apps- which are limited in that they can only shed light on whether or how often an agent can complete a task within a likely encounter
specific environment. When deployed however, agents variations in app design and content that can affect an agent's 's ability to complete task. To address this blind spot of measuring agent reliability across app vari- ations, we develop OPENAPPS, a light-weight open-source ecosystem with six apps (messenger, calendar, maps, etc.) that are configurable in appearance and content. OPENAPPS requires just single CPU to run, enabling easy generation and deployment of thousands versions of each app. Specifically, we run more than 10,000 independent evaluations to study reliability across seven leading mul- timodal agents. We find that while standard reliability within a fixed app is rela- tively stable, reliability Task success rates for many agents can fluctuate by more than 50% across app variations. For example, Kimi-VL-3BB average success across all tasks fluctuates from 63% to just 4% across app versions. We also find agent behaviors such as looping or hallucinating actions can differ drastically depending on the environ- ment configuration. These initial findings highlight the importance of measuring reliability along this new dimension of app variations.
vary drastically when measured across app variations.
OpenApps
Configurable
Reliability Across App Variations
Apo
U1-Tor
Kimikl
Gownd S-V1
Figure 1: OPENAPPS generate thousands of configurable versions of apps. OPENAPPS contains six apps covering common digital tasks with configurable appearance and app data for measuring a new dimension of reliability: across app variations agents are likely to encounter. OPENAPPS can be deployed anywhere Python can run with a single CPU (without specialized hard- ware, emulators or setup). In the right panel, we see average success rates for the same tasks and agents can fluctuate across app versions, suggesting app variation is a key axis of reliability.
"Core contributors, Secondary contributors.

---

Published as a conference paper at ICLR 2026
INTRODUCTION
Recent advances in foundation models have spurred growing interest in building autonomous UI- agents capable of executing complex, multi-step workflows across digital environments. Such agents hold the promise of serving capable assistants in everyday and professional contexts, from coor- dinating schedules to managing documents. Key to realizing the promise of Ul-agents is reliability; users must be able to trust an agent can successfully complete the task. Researchers have invested considerable effort to measure the reliability of agents by cloning existing apps or web sites. For example environments such OSWorld (50), (Visual)WebArent (64) 2I), and TheAgennCoompny (52) allow researchers to measure reliability in terms of how often an agent can successfully com- plete task within fixed app clone.
When deployed, however, agents are likely to encounter numerous variations in app design, appear- ance, and content. For example, there are (conservatively) dozens of calendar apps, each with an ever evolving style and configurable content and appearance. An agent may struggle when the con- tents of the calendar are dense with events or find it easier to navigate Ul-elements dark mode. In short, the reliability of an agent's 's task performance depends on the variation in app versions an agent encounters. This dimension of reliability across app variations however, can not be measured in current environments that rely on fixed clones of apps.
To capture this crucial dimension of reliability, we develop OPENAPPS. OPENAPPS is light- weight open-source ecosystem for generating thousands of versions of apps with transparent logic and state. OPENAPPS includes calendar, messenger, todo, maps, shopping, and code editing apps. In contrast to environments that pre-record trajectories or pre-defined tool APIs 57), multimodal agents directly interact with OPENAPPS using the same actions as humans (click, type, scroll, etc.). OPENAPPS allows researchers to
Generate thousands of versions of each App. Each app comes with configurable appear- ance and content variables allowing researchers to generate thousands of versions of each app to study reliability of agents across app variations (see Figure IT.
Access app logic and ground truth underlying state. The full app state and logic of each app is exposed in Python for researchers to study or extend. Additionally, each task reward is based on the underlying app state thereby avoiding noisy reward signals and reward hacking behavior (66).
Deploy OPENAPPS on any machine that runs Python for lightweight scalable exper- iments. Unlike many existing environments that require specialized emulators or con- tainers, OPENAPPS requires single CPU and runs on any machine that can run Python thereby enabling scalable parallel experiments across app variations.
Using OpenApps, we study agent behaviors across app variations. We run 10,000 trials across seven leading agents spanning both closed and open multimodal foundation models includ- ing Claude, OpenAI, Qwen-VL. and specialized Ul-models (UI-Tars). We find that while agents are reliable within a single app variation, reliability across app variations can differ drastically, and task success rates can vary by more than 50% across app variations. For example, Kimi-VL-3B average success across all tasks fluctuates from 63% to just 4% across app versions. Furthermore, agent behaviors such as looping or hallucinating actions can also heavily depend on the app variations an agent encounters. For example, UI-Tars is 5x more likely to hallucinate actions depending on the app variation. The findings suggest app variations play a crucial role in ensuring agents are reliable.
2 RELATED WORK
While text-only tool calling agents can interact with Apps via pre-defined interfaces such as MCP 28 33), here we focus on Ul-agents that directly interact with the multimodal environment (with- out requiring a pre-defined API interface). In Appendix C we review how foundation models are adapted for agentic workflows, including post-training strategies that boost performance in such settings. We also highlight the critical role of simulators in reinforcement learning. This section fo- cuses specifically on environments and benchmarks for digital autonomous agents. Aforementioned can be distinguished by the platform and the capabilities they target, how they assign rewards, what
2

---

Published as a conference paper at ICLR 2026
modalities they support, where they lie the the sim2real spectrum, their scale and how easily they are deployable.
Web Agent Benchmarks. Benchmarks designed for web applications typically target consumer
tasks of web browsing such as online shopping (56) 12), travel and food planning (58, (7), and more
generally search and navigation (64, 21
18 59). While some benchmarks crowd-source T7 58)
the tasks they test on, most provide relatively small list of meta tasks such as "Search for expensive x". All Ul-agent benchmarks we are aware of rely on website clones to ensure realism, the complexity of these clones makes it hard to design strategic interven- tions to website design or content. As a consequence of this lack of controllability, when reporting agent failure, authors typically rely anecdotal evidence alone. Reward functions are typically based either on human demonstration trajectories or on state-change signals. Our approach follows the latter, but differs in that evaluate the full environment state (see section (.3), ensuring that rewards cannot be gamed by completing adversarial side tasks. A key distinction of our environment its position in the trade-off' between realism and computational efficiency that allows for deploy- ment at scale. Full website clones, as used in WebArena (64), can require over 100GB of memory per site, which severely limits scalability or very expensive to run scale. At the other extreme, lightweight environments such as MiniWoB(AA (40) match our efficiency but fall short in realism. Fi- nally, BrowserGym (3) provides a standardized interface across web benchmarks, and underpins both REAL (17), Aheeggooon (52), and our environment.
the best/leat
OS-Level Benchmarks. Beyond everyday everydayy web tasks, several benchmarks target general computer use. WorkArena (13), WorkArena++ (6) and Themaayn (52) are UI-benchmarks that focus on corporate workflow automation (e.g.. HR, customer support) in a few specific apps. OSWorld (5I) on the other hand simulates full Linux based operating system, where the distribution of tasks reflects more general use. AgentBench (26) focused on an even broader range of tasks, such as puzzle solving, knowledge retrieval, operating systems or web browsing. Equivalently, there exist environments targeting mobile OSs (8) 43 25). Although the tasks in OPENAPPS are less complex than those in many of these environments, agents still struggle to complete them. Moreover, the heavy compute demands of virtual machine- based environments make them impractical for large- scale trajectory generation just as web-agent- -beneettbeenchaars...
Mobile device control benchmarks. Another line of work focuses mobile device control, usn- ally targeting Android environment simulation. Android in the Wild (36) presents the AITW dataset, designed to encourage robustness evaluation new task descriptions, applications, platform ver- sions. The B-MoCA benchmark rates randomized device configurations. AndroidWorld (37) provides another live Android emulator environment where tasks are dynamically instantiated. LlamaTouch (62) introduces testbed for on-device mobile UI task execution and evaluation. Compared to these benchmarks, OPENAPPS is not restricted to any specific device-emulation framework and offers the distinctive advantage of supporting large-scale experimentation through its lightweight deployment requirements. Its config- urable content and customizable appearance further make it highly extensible, allowing researchers to casily define specialized tasks interest.
(22) covers daily tasks over an Android emulator, which incorpo-
3 OPENAPPS: FRAMEWORK AND ENVIRONMENT
Orchestrating an agent to interact with an environment of apps towards a goal involves many moving pieces. Consequently, we organize agent interactions in OPENAPPS within the established termi- nology of reinforcement learning. As shown in Figure 3 the agent receives visual (and in some cases simplified text representations of Ul-elements) observations O from OPENAPPS then directly acts with an action from the space A consisting of common actions available to humans such as cl click, type, scroll, etc. (sce Section 3.2) Finally, we assess using OPENAPPS underly- ing state, such as the list of events in the calendar for example, whether at a given step t the agent
successfully completed the task (see Section 3.3
3

---

Published as a conference paper at ICLR 2026
3.1 OPENAPPS ENVIRONMENT
OPENAPPS is browser-based, highly customizable, open-source, low resource and isolated en- vironment for UI agents. This ecosystems comes with six fully functional apps written in Python. OPENAPPS contains a range of apps needed for common digital tasks: OpenCalendar, OpenMessen- ger, OpenMaps, OpenToDo, OpenCodeEditor, and OpenShop. OpenCalendar for example, allows the user to view, create, and delete events in a fully functional calendar app. To our knowledge this the first UI agent environment written in Python the lingua franca of Al researchers and practi- tioners, making it easy for researchers (and possibly coding agents) to understand and modify the internal logic of each app. This choice improves accessibility relative to existing environments that often depend languages less known researchers (Kotlin, javascritt, CSS) or require specialized infrastructure such as Android emulators, large databases, or containers. We accomplish this by building OPENAPPS using the FastHTML framework. Configuring the apps is made easy by pro- viding access to appearance variables and underlying data via editable YAML files. Because these treat them the initial environment state 8o. When
YAML files fully represent the environment, the agent takes an action az, the state 8; is updated accordingly.
OpenTooD
OpenCalendar
OpenMessenger
Wetcome Openthop
OpenMaps
OpenCodeEditorr
OpenShop
Figure 2: Screenshots with example appearance variations of all six apps in OPENAPPS: OpenToDo, OpenCalendar, OpenMessenger, OpenMaps, OpenCodeEEiior, and OpenShop. Each app is fully functional Python application with editable state and appearance. OpenApps be configured via simple YAML files.
Configuring OPENAPPS. As shown in Figure D the data underlying each app is configurable via yaml files. For example, to modify the existing list of todos in OpenToDo, users edit or supply their own list of todos as strings in the target yaml file. Along with the granular control each element, variations. For appearance variations, we include a light theme, a black-and theme, and the use of challenging fonts with Brush Script MT, illustrated in Figures 2 and 7 For content variation, we enrich the environment with extended descriptions for each application, intentionally drafted misleading descriptions, adversarial text, and German translations alongside the default English contents. We also explore popular font, color, and language choices used in the web Appendix E.3) Rather than attempting to cover every possible variation, our approach focuses on a curated selection. All granular appearance and content variables (titles, colors for Ul-elements, etc.) available OPENAPPS for researchers modify via simple yaml generate thousands of versions of each app.
provide pre-populated high-level configurations for appearance and content
theme, a dark
Large scale reproducible experiments with OPENAPPS. Because each instance of OPENAPPS runs in single lightweight Python process, deploy many parallel experiments modest
4

---

Published as a conference paper at ICLR 2026
Task
"Delete the Brooklyn Bridge from ny saved placea"
OpenApps environment
Agent
Sty Swrgenshot
Inaruction
Prompt
observation Oz
Histry 1:911
Anaiiabl odtions (BrowserGyn API
Absinuet and explati onampiea
policy <(a
...
action O2
state 8
reward Stompe
Figure 3: We organize agents interactions with OPENAPPS using the standard terminologylof re- inforcement learning. The environment state 8, defined by design and content variables, and initialized from YAML. specification. At each step 4, the agent receives observations (screen- shot and accessibility tree) and issues an action a, (a (a;-)a;;,, 0,)%) through BrowserGym API calls. Task success is evaluated using the underlying app state St.
hardware (even a single CPU) and memory (< 10MB). guarantee reliable and reproducible execution, every starts from local copy of the full environment state inceudinniiiing all app data and appearance variables- which which footprint low while ensuring that experiments
reset at initialization. Our design keeps the overall memory
be reproduced exactly, with ground-truth reward signals available by default (thus following best practices from (66)).
OPENAPPS be combined with any agent scaffolding and task orchestration approach. In the next section, we show one example of how this can be realized using the BrowserGym framework implement tasks and agents with OPENAPPS.
3.2 DEPLOYING AGENTS IN OPENAPPS: OBSERVATION AND ACTION SPACE
To implement agent actions and capture observations with OPENAPPS we use BrowserGym (23) popular web agent framework used in prior works (56, 65). Given prompt with goal, time step the agent receives observations Og from OPENAPPS as visual screenshots (akin to what a human would see) and for agents that support text inputs also simplified text representations of UI- elements (using AX Tree generated by browsers). The agent then sends an action a; from the set of actions available humans such as click, type, scroll, etc. that directly interacts with OPENAPPS to update the state from to 8141 (set of available actions detailed in Appendix D.4).
For specialized agents post-trained trained other ecosystems, is sometimes necessary translate their native APIs into the BROWSERGYE interface. We have implemented such parsers, for instance for the popular open-weights agent UI-TARS, a specialized ixer-interface agent 32) Results presented later in this work omit these translation details. Whenever standard system prompts model config- urations (e.g., temperature) have been proposed in the original works, we adopt them to ensure fair and optimal evaluation of all agents (we provide additional temperature ablations in Figure D). It is worth emphasizing that not all agents are equally compatible with all observation modalities, often due to their (partially unknown) training regimes. Accordingly, report results under optimized configurations for each agent. For example, UI-TARS can only operate as a visual agent and is not compatible with text-based APIs that rely text representation inputs.
5

---

Published as a conference paper at ICLR 2026
3.3 TASK DESIGN AND SUCCESS
We provide a set of simple fifteen tasks such as adding an item to the calendar or saving location your favorites in maps. We ensure each task has multiple goal prompts and that each app has least two tasks (full set of tasks in Section 3.3 and sample goals in Table D.
To evaluate task success, existing benchmarks typically adopt one of two alternatives: (a) human- trajectory rewards, where an agent is rewarded for imitating demonstrations. This approach is overly restrictive, since many valid action sequences may lead the same goal ("many roads lead to Rome"); (b) change-based checks, where only the presence of specific modification is verified. This can be exploited by agents taking unintended or malicious actions (e.g., purchasing flight but also submitting credit card information to third party) as shown in Zhu et al. (66).
We avoid both pitfalls by granting the reward function access to the complete app state at each time step f. For example, the state may include the full set of calendar events or all messages together with their metadata (see Appendix [ for examples). In our implementation, environment states are serialized into lightweight yaml files, which can be represented structured vectors. Rewards are defined as a deterministic indicator function of whether the target state has been reached,
so that task is considered complete only if all state conditions are satisfied. This design provides an objective and reproducible measure of an agent's 's ability to perform precise state changes (a list of tasks is available in Table and Table Because app logic and reward definitions are implemented in Python, researchers can easily extend or redefine reward functions to measure alternative notions of task success.
MULTIMODAL AGENT RELIABILITY WITH OPENAPPS
We introduce how OPENAPPS can be used to study agent reliability along the dimension of app variations. Typically, agent success is studied within fixed app version, with fixed choices for colors, fonts, content etc. In other words, agent success is evaluated on a fixed distribution A defined by
A1 (f1,1 f12, f1.3...
(1)
where each f1, denotes fixed choice of factor of variation, such as font, color, and content. When deployed however, agents are likely to encounter one of many app variations AI, A2.... that differ in appearance and content. Our goal is expand the dimension of agent reliability capture the distribution shifts across app variations. Specifically, across app reliability captures agent success reliability across mixture of app variation distributions (A1, 3......
We first show agents are sensitive to app variations in Section 4.1 We find fixed app environments do not capture the considerable fluctuations in agent success rates across app variations. Next in Sections 4.2 and we study how agent behaviors such as looping or hallucinating actions as well as deployment configuration can also differ across app variations, which in all confirms app variations is important axis of agent reliability.
EXPERIMENTAL SETUP
For each task, we apply each of the eight content and appearance variations (shown Section 3.1 and Figure 7 all apps simultaneously. For example, all apps would be set to their dark theme (we explore variation interactions in Appendix I.. Rather than create a challenging benchmark, we focus on fifteen simple tasks, such as add buy milk to my todo 1ist, that require only a few steps to isolate changes in reliability (shown in Table 4). We include additional experiments on complex multi-step tasks in Appendix E..2 We then launch more than 10.000 independent eval- uations with seven agents spanning both closed and open multimodal foundation models including Claude, OpenAl, Qwen- VI., and specialized Ul-models (UI-Tars).
Reliability of task success. Beyond average success rates. reliability captures fluctuations in suc- cess rates when an agent is deployed. To measure reliability, we measure fluctuation in agent success rates using the standard deviation of rewards across runs for given task. Given set of rewards

---

Published as a conference paper at ICLR 2026
90
82
81
median
max
73
Task Success
63
49
28
19
4
GPT-40
Claude Sonnet
UI-TARS-1.55 5-78
Kimi Kimi-VL-A3B VL---V----BB--- WW--------s---uud Instruct
Qwen2. S.
LLaVA-11 6-78
GPT-4o (vision only)
Figure 4: Agents are sensitive to app variations. We show that the average success rate all tasks differs across app variations. Each bar represents the average success rate over tasks within single app version (with three random seeds per task). Note filter tasks where agent has success rate of 0 across all app variations. We see success rates can differ considerably across app versions.
Rei from an agent deployed in a fixed app version (Ai), we call sld(Rws) the deviation within fixed app version. When deployed, however, an agent is likely to encounter many versions of an app. For example, for a calendar there are several dozen (if not more) calendar apps, each with ever evolving updates and configurations, yielding many app variations. To measure reliability erally to include the many app variations an agent is likely to encounter, A1, A2,..., we compute std( R2,, Ro2,-..3) which call the overall deviation across app variations. Finally, com- pare the ratio of Fixed App Reliability/ / Overall Reliability to assess how performance fluctuation can stems from variations apps, which we show can be quite large in the coming sections.
gen-
4.1 AGENTS ARE SENSITIVE TO APP VARIATIONS
Agents are sensitive to app variations. In Figure we show each agent's performance in terms of the average task success rate across app variations. Each bar measures average task success rate across different app versions. We find agents are sensitive app variations with some agents exhibiting more sensitivity than others. For example, Kimi-VL performance can vary between 4% and 63% task success depending on the app variant (a more than 10x difference), suggesting agent success can dramatically differ across app variations. Even for closed models that have high overall performance such Claude Sonnet and GPT-4o (when inputs also contain simplified text AX tree representation), success rates on individual tasks can fluctuate drastically as shown in Table 3 For example, the send message task success fluctuates from 42% 0% for GPT-4o and 75% 20% for Claude 4 Sonnet depending the app variation the agent encounters. We report the breakdown by task of agent performance and reliability in the default app version in Tables 2
and
Task success within fixed app version overestimates reliability. In Figure 5 we compare devia- tions task success within a fixed version of apps (as the case with existing environment clones) versus the overall deviation across app variations agent is likely to encounter. We find fixed apps overestimate reliability across the app variations that agents are likely to encounter, as task success consistently fluctuates more than within fixed app version. In many cases, for example 5wen2.5- VL, Kimi-VL, and UI-Tars, standard deviations in task success across app variations are more than twice those observed within fixed apps. These finding suggest studying reliability within fixed app clone underestimates fluctuations in agents' task success. We also measure absolute deviation in task success in Figure TO which provides similar conclusions.

---

Published as a conference paper at ICLR 2026
Claude Sonnet 4
31.9
deviation
Within Fixed Appe
Claude Sonnet (Fixed Apps)
23.8
App Versions
GPT-4o
17.7
GPT-4o (Fixed Apps)
12.4
Aimi-VL-ABB-lnstrutt
40.5
Kimi-VL-A3B Inntrucc (Fixed Apps)
16.5
LLaVA-v1.8 1.-77
18.5
LLaVA-v1. 6-7B (Fixed Apps)-
10.7
Qwen2.5-VL
32.0
Qwen2.5-VL (Fixed Apps)
16.8
11-TARS-1.5-78
37.5
UI-TARS-1. (Fixed Apps)
17.5
-40
-30
-20
-10
10
20
30
40
STD Task Success
Figure 5: Reliability within fixed app versions underestimates fluctuations in performance. In the middle of each bar we show the standard deviation of task success. We compare two settings: within fixed app version compared overall deviation that also accounts for difference in agent success rates across app variations.
challeiii
long Scotrpttro
Mbcc wait
oerear
1014 there
acomsuria plana
3.55
830
0.1 0.5
Ary. kaik 3000e0
Ang 500 940003
Figure 6: Agent reliability can be low terms of performance across app variations. Model performance (as measured by the task success rate across random seeds, averaged over all tasks) can differ greatly across appearance and content variations (shown Figure 7. For example, we notice sizable drop in the performance of UI-TARS- 1.5-7B (a vision-only model) compared to the default when the app has a dark theme, and likewise drop the performance of Kimi-VL-A3BB-Insmn when the app is in German or contains adversarial page descriptions. The black bars capture the standard deviation of rewards across seeds, averaged over tasks. The task prompt is explicit and fixed. Kimi uses visual and AX tree inputs while UI-TARS is UI-visual only.
8

---

Published as a conference paper at ICLR 2026
App appearance can affect agent success (UI-TARS case-study). When OPENAPPS has dark theme, the performance of UI-TARS degrades (see Figure 6). This could be due to lower contrast in the dark theme setting. Given that dark themes common real-world websites, this finding highlights the importance of measuring agent reliability with respect to appearance variations. Here, choose UI-TARS susceptible to appearance variations. We provide the full breakdown of task success rates for all agents and appearance variations in Table 5 the appendix. While the trend for UI-TARS does not hold for the other agents, we still observe that appearance variations can change model performance. Upon qualitative inspection, we find that Qwen2.5-V VL can struggle to remove saved place in the map when a dark theme is deployed. In a similar vein, OPENAPPS enables agent developers to stress test their agents across different appearance variations and dig deeper into failure modes.
case study because it is Ul-visual only agent and is thus likely most
App content can also affect agent success (Kimi-VL. case-study). The performance of Kimi- VL-A3B-Inssen Instruct degrades most when OPENAPPS is in German contains adversarial descriptions (see Figure (). This highlights the importance of testing agents on languages besides English and malicious content, among other content variations. The performance of Kimi-VL does not degrade as much for long descriptions, which could be due to its focus on long-contex understanding. We provide the full breakdown of task success rates for all models and content variations in Table 6 in the appendix. Like for appearance variations, the trend for Kimi-Vi VL does not hold for the other models, as content variations affect task success for other agents differently.
4.2 AGENT BEHAVIOR CHANGES ACROSS APP VARIATIONS
Given agent task success can fluctuate across app variations, we now highlight how agent behaviors such as looping or hallucinating actions are also affected by app variations.
Agents are more likely to loop actions when encountering certain app variations. Action loops (i.e., repeating sequences of actions) tend be problematic behavior as they are associated with an agent failing to complete a task. Specifically, we find an agent's 's average loop count is 10x larger when an agent unsuccessful (0.20 when successful versus 1.5 when unsuccessful). We observe action looping behavior can differ considerably across app variations (see [Tlle the appendix). We find for example, UI-TARS, which has the highest variance average loop counts across app variations, can exhibit nearly 2x the number of loops when the apps have a dark theme compared to other settings, suggesting app variations can dramatically affect how often an agent is stuck looping actions. We provide more examples of action loops for other agents in Appendix F
Agents are more likely to hallucinate actions in certain app variations. We also find agents are prone to hallucinating invalid actions in some app variations (see Table [2]. For example, we find many agents hallucinate invalid actions when the content of apps contains misleading or adversarial descriptions. For example, GPT-4o hallucinates function calls and Ul-elements that are simply not present (see Appendix (F) at a higher rate with adversarial descriptions present (e.g., 'a banner stating the task is complete"). ). We find similar failures in other models where agents provide invalid actions that are incorrectly-formed versions of valid actions (e.g., mouse_click (x=612 y), no no-op), well-formed actions that do not exist in the environment (e.g., remove.item, finished), and valid actions with bad arguments (e.g., click (bid), scroll ricciiioonn down', point=' (966, 546)
Lengthy, misleading, and adversarial content can be associated with higher rates of agent misunderstanding. We also capture how often an agent misunderstands the user's intent by mea- suring how often the agent navigates to an app irrelevant to the task at hand. We find as shown in Table 13 are more likely to misunderstand when they encounter long and adversarial descriptions in app content. For example, Qwen2.5-VV VL intent misunderstanding rates jump from 3% in the de- fault setting to 40-45% when content is long contains adversarial descriptions. We report intent misunderstanding rates for all agents across appearance and content variations in Table [3]
Overall, we find agent behavior can be highly dependent the app variation agent encounters. To effectively study agent behavior, researchers should capture this overlooked dimension of app variations agents are likely to encounter.

---

Published as a conference paper at ICLR 2026
4.3 APP VARIATIONS AND AGENT DESIGN
Thus far, we have fixed the agent setup and user specification, and then evaluated how different agents perform across app variations. Here, we study how changes in agent deployment configura- tions interact with app variations to affect reliability. As a case study, we highlight how the choice of common screen resolutions (FHD 1920x 1080, HD 1280x720, HVGA 480 320) used when de- ploying agent interacts with app variations using Ul-Tars as a case-study. We then measure the task success rates of UI-TARS across app variations as the screen resolution varies. In Table 7 we find that while higher resolution leads to higher task success for many app versions, the trend does not always hold: in the dark theme setup, a high resolution yields a significant drop in task success. This suggests even the simple choice of the optimal screen resolution used when deploying an agent can be drastically different depending on the app variation.
CONCLUSION
Foundation models (FMs) endowed with agentic capabilities may enable automated execution of increasingly complex tasks provided they behave reliably. We introduce OPENAPPS, the first testbed designed to systematically evaluate the reliability of UT-agents under varying environment configurations, rather than merely evaluating policy reliability (within fixed environment). Our main contribution flexible simulator that offers full observability and supports large number of parallelized, controlled experiments. Our evaluation highlights that superficial variations in app appearance or content can lead to substantial performance differences differences that are model- specific. For instance, agentic systems encountering German-languaa variants of the interface hibit both significantly improved and degraded performance, depending on the model. We further demonstrate that such variations provoke distinct failure modes, including action loops and halluci- nated behavior. These diagnostics may offer actionable insights for agent development. Importantly, our findings also inform deployment: we find that although agents deployed with higher resolution inputs tend to have higher task success rates, for some app variations the opposite is in fact true. Together these findings highlight that app variations are key axis of reliability in terms of agent performance, behaviors, and deployment.
dffferences-diifereenees
Limitations and future work OPENAPPS offers calendar, messenger, maps, etc., apps which encompass many common digital tasks. In this work however, we focus on simple tasks such as adding an item to todo 1ist that require only few steps complete and do not necessarily represent the complexity or distribution of real world tasks. Even with such simple tasks, we see considerable fluctuation in agent success rates. Future work can extend the set of tasks to include more complex or longer-horizon tasks to form a benchmark for Ul-agent reliabil- ity. Furthermore, here we focus on varying each app appearance or content factor independently. Of course interactions between multiple app variation factors can also expose interesting behaviors that we leave future work. Finally, here we focus on autonomous agents, though agents can cer- tainly also incorporate human validation or interaction when completing task. Beyond evaluating reliability, OPENAPPS can also serve as a wealth of training data. Thanks to the thousands of app variations OPENAPPS can generate, OPENAPPS can be used to scale digital agent training pipelines, provide a safe sandbox for deploying agents without weal-world risk, and allow researchers to study generalization across app variations. We elaborate these exciting possibilities in Appendix B
REPRODUCIBILITY STATEMENT
To ensure transparency and reproducibility, we soen-s all components of our work, including the environment, experimental setups, and evaluation code. This enables other researchers to fully replicate our results and build upon our framework without restrictions.
ACKNOWLEDGMENTS
We're grateful for discussions and feedback from Kamalika Chaudhuri, Candace Ross, and Maxi- milian Nickel. JK thanks the Simons Foundation for support through the Collaborative Grant "The Physics of I Learning and Neural Computation".
10

---

Published as a conference paper at ICLR 2026
REFERENCES
[1] Tamer Abuelsaad, Deepak Akkil, Prasenjit Dey, Ashish Jagmohan, Aditya Vempaty, and Ravi Kokku. Agent-E: From Autonomous Web Navigation Foundational Design Prin- ciples in Agentic Systems, July 2024.
13032
arXiv:2407.13032 [es] version:
[2] Saaket Agashe, Jiuzhou Han, Shuyu Gan, Jiachen Yang, Ang Li, and Xin Eric Wang. Agent An open agentic framework that uses computers like human. arXiv preprint arXiv:2410.08164, 2024.
[3] Pierre Andrews, Amine Benhalloum, Gerard Moreno- Torres Bertran, Matteo Bettini, Amar Budhiraja, Ricardo Silveira Cabral, Virginie Do, Romain Froger, Emilien Garreau, Jean- Baptiste Gaya, Hugo Laurençon, Maxime Lecanu, Kunal Malkan, Dheeraj Mekala, Pierre Ménard, Grégoire Mialon, Ulyana Piterbarg, Mikhail Plekhanov, Mathieu Rita, Andrey Rusakov, Thomas Scialom, Vladislav Vorotilov, Mengjue Wang, and Ian Yu. Are: Scaling up agent environments and evaluations. arXiv preprint arXiv: 2509. 17158, 2025.
[4] Szilárd Aradi. Survey of deep reinforcement learning for motion planning of autonomous vehicles. arXiv preprint 1rXiv:2001. 2020.
[5] Mare G. Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, 2013.
[6] Léo Boisvert, Megh Thakkar, Maxime Gasse, Massimo Caccia, Thibault de Chezelles, Quentin Cappart, Nicolas Chapados, Alexandre Lacoste, and Alexandre Drouin. Workarena+4:: To- wards compositional planning and reasoning-basee common knowledge work tasks, Advances in Neural Information Processing Systems, 37:5996-660 605166600112550 2024.
[7] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint / 2016.
[8] Andrea Burns, Deniz Arsan, Sanjna Agrawal, Ranjitha Kumar, Kate Saenko, and Bryan A. Plummer. A dataset for interactive vision-language navigation with unknown command feasi- bility. European Conference on Computer Vision, 2022. doi: 10.1007/978:3-031-2200
[9] Minghao Chen, Yihang Li, Yanting Yang, Shiyu Yu, Binbin Lin, and Xiaofei He. Automan- ual: Constructing instruction manuals by IIm agents via interactive environmental learning. Advances in Neural Information Processing Systems, 37:589-631, 2024.
[10] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information pro- cessing systems, 30, 2017.
[11] DeepSeck-Ar AI, Daya Guo, Dejian Yang, Haowei Zhang. Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang. Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjic Qiu, Junlong Li. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen. Kai Dong, Kai Hu, Kaige Gao. Kang Guan, Kexin Huang, Kuai Yu. Lean Wang, Lecong Zhang. Liang Zhao, Litong Wang. Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang. Qiancheng Wang. Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang. Shuiping Yu, Shunfeng Zhou, Shuting Pan. S. S. Li, Shuang Zhou, Shaoging Wu. Shengfeng Ye, Tao Yun. Tian Pei. Tianyu Sun, T. Wang, Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. I.. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao

---

Published as a conference paper at ICLR 2026
Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang. Y. X. Wei, Yang Zhang. Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang. Yi Yu, Yichao Zhang. Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang. Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang. Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning, 2025. URL / wwwpv..oo///b///5......1 12948
[12] Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2Web: Towards Generalist Agent for the Web, December 2023. URL http://arae http:/awwwv..ooaa////aaaaaoa 2306. 2200000066000.060000010000000000 3Xiv:23066060700
[13] Alexandre Drouin, Maxime Gasse, Massimo Caccia, Issam H. Laradji, Manuel Del Verme, Tom Marty, Léo Boisvert, Megh Thakkar, Quentin Cappart, David Vazquez, Nicolas Chapa- dos, and Alexandre Lacoste. WorkArena: How Capable Are Web Agents Solving Common Knowledge Work Tasks?, July 2024. URL http://arxiv. abs/ http://aaw...ooo//b///0......... arXiv:2403.07718.
[14] Linxi Fan, Guanzhi Wang, Yunfan Jiang, Ajay Mandlekar, Yuncong Yang, Haoyi Zhu, Andrew Tang, De-An Huang, Yuke Zhu, and Anima Anandkumar. Minedojo: Building open-ended embodied agents with internet-scale knowledge. Advances in Neural Information Processing Systems, 35:18343-18362, 2022.
[15] Shiqing Fan, Xichen Ding, Liang Zhang, and Linjian Mo. Mcptoolbench++: A large scale ai agent model context protocol mep tool use benchmark, 2025. URL/https://arxiv
abs/ 07575
[16] Yao Fu, Dong-Kt Ki Kim, Jaekyeom Kim, Sungryull Sohn, Lajanugen Logeswaran, Kyunghoon Bae, and Honglak Lee. Autoguide: Automated generation and selection of context-aware guidelines for large language model agents. Advances in Neural Information Processing Sys- tems, 37:1 19919 110199-1999999 19948, 2024.
[17) Divyansh Garg, Shaun VanWeelden, Diego Caples, Andis Draguns, Nikil Ravi, Pranav Putta, Naman Garg, Tomas Abraham, Michael Lara, Federico Lopez, et al. Real: Bench- marking autonomous agents on deterministic simulations of real websites. arXiv preprint arXiv:2504. 11543, 2025.
[18] Hongliang He, Wenlin Yao. Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. Webvoyager: Building an end-to-end web agent with large multimodal models. Annual Meeting of the Association for Computational Linguistics, 2024. doi: 10.
485500arXiv.2401113311
[19] Shengran Hu, Cong Lu, and Jeff Clune. Automated design of agentic systems. arXiv preprint Stt1:240080043333 2024.
[20] B. Ravi Kiran, Ibrahim Sobh, Victor Talpaert, Patrick Mannion, Ahmad A. A. Sallab, Senthil Yogamani, and Patrick Pérez. Deep reinforcement learning for autonomous driving: A survey. IEEE Transactions on Intelligent Transportation Systems, 4999:4999 2022. doi: 10. 1109/TITS. 1021.356655
[21] Jing Yu Koh, Robert Lo. Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po- Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks, June 2024. URL http://
arxiv. arxiv..rgg/abs/2400 13649 arXiv:2401.13644
12

---

Published as a conference paper at ICLR 2026
[22] Juyong Lee, Taywon Min, Minyong An, Dongyoon Hahm, Haeone Lee, Changyeon Kim, and Kimin Lee. Benchmarking mobile device control agents across diverse configurations. arXiv preprint arXiv:2404. 166600 2024.
[23] Thibault Le Sellier de Chezelles, Maxime Gasse, Alexandre Lacoste, Massimo Caccia, Alexandre Drouin, Léo Boisvert, Megh Thakkar, Tom Marty, Rim Assouel, Sahar Omidi Shayegan, Lawrence Keunho Jang, Xing Han Lü, Ori Yoran, Dehan Kong, Frank F. Xu, Siva Reddy, Graham Neubig, Quentin Cappart, Ruslan Salakhutdinov, and Nicolas Chapados. The browsergym ecosystem for web agent research. Trunsactions on Machine Learning Research (TMLR), 2025. URLhttps: to appear.
/ttp:/ttssspppeeiiiiiieeo http://pppeevveeeeeee net/ fewee/ooommidd522222 fKGmv3 Preprint,
[24] Jiajian Li et al. Open-world reinforcement learning over long short-term imagination. arXiv preprint 2400032400...618 2024.
[25] Yang Li, Jiacong He, Xiaoxia Zhou, Yuan Zhang, and Jason Baldridge. Mapping natural language instructions mobile ui action sequences. Annual Meeting of the Association for Computational Linguistics, 2020. doi: 10.18663/.112002aacll.miii
[26] Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. Agentbench: Evaluating Ilms as agents. arXiv preprint arXiv:2308. 03688, 2023.
[27] Xinbei Ma, Yiting Wang, Yao Yao, Tongxin Yuan, Aston Zhang, Zhuosheng Zhang, and Hai Zhao, Caution for the Environment: Multimodal Agents are Susceptible Envi- ronmental Distractions, August 2024. URL http:://axxi, wwwpvv..rrraab///eeeee arXiv:2408.025 0224444....
[28] Grégoire Mialon, Clémentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom. Gaia: a benchmark for general ai assistants. In The Twelfth International Conference on Learn- ing Representations, 2023.
[29]
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Mare G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Pe- tersen, Charles Beattic, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep rein- forcement learning. Nature, 518(7540):5295535 2015.
[30] Janosch Moos, Kay Hansel, Hany Abdulsamad, Svenja Stark, Debora Clever, and Jan Peters. Robust reinforcement learning: A review of foundations and recent advances. Machine Learn- ing and Knowledge Extraction, 4(1):276-315 315,, 2022.
[31] Zehan Qi, Xiao Liu, lat Long long, Hanyu Lai, Xueqiao Sun, Wenyi Zhao, Yu Yang, Xinyue Yang, Jiadai Sun, Shuntian Yao, et al. Webrl: Training IIm web agents via self-evolving online curriculum reinforcement learning. arXiv preprint arXiv:2411.025: 00237, 2024.
[32] Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang. Wanjun Zhong, Kuanye Li, Jiale Yang, Yu Miao. Woyu Lin, Longxiang Liu, Xu Jiang, Qianli Ma, Jingyu Li, Xiaojun Xiao, Kai Cai, Chuang Li, Yaowei Zheng, Chaolin Jin, Chen Li, Xiao Zhou, Minchao Wang, Haoli Chen, Zhaojian Li, Haihua Yang, Haifeng Liu, Feng Lin, Tao Peng, Xin Liu, and Guang Shi. UI-TARS: Pioneering Automated GUI Interaction with Native Agents, January 2025. URL attp: org/ abs/2501. 12326 arXiv:2501.12322 [cs].
[33] Ella Rabinovich and Ateret Anaby-Tavor. On the robustness of agentic function calling. arXiv preprint arXiv: 5504.00144 2025.
[34] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Er- mon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances Systems, volume 36, pp.
Neural Information Processing
ht httro://prooeeciigg...rrr...p..p Curran Associates, Inc., 2023. URL
a e
13

---

Published as a conference paper at ICLR 2026
[35] Shyam Sundhar Ramesh, Yifan Hu, lason Chaimalas, Viraj Mehta, Pier Giuseppe Sessa, Haitham Bou Ammar, and Iija Bogunovic. Group robust preference optimization in reward- free rihf. arXiv preprint arXiv:2405.2000 2024.
[36] Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy Lillicrap. An- droidinthewild: A large-scale dataset for android device control. Advances in Neural Informa- tion Processing Systems, 36:59708-5977 2023.
[37] Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Mary- beth Fair, Alice Li, William E Bishop. Wei Li, Folawiyo Campbell-Ajala, Daniel Kenji Toyama, Robert James Berry, Divya Tyamagundlu, Timothy Lillicrap. and Oriana Riva. Androidworld: A dynamic benchmarking environment for autonomous agents. In The Thirteenth International Conference on Learning Representations, 2025. URL https: //openreviee ew. www.:eee//aee http://ooummmmm11yypaeee
[38] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. 2017.
[39] Junhong Shen, Atishay Jain, Zedian Xiao, Ishan Amlekar, Mouad Hadji, Aaron Podolny, and Ameet Talwalkar. Scribeagent: Towards specialized web agents using production-scale work- flow data. arXiv preprint 2rXiv:241..15000 2024.
[40] Tianlin Shi, Andrej Karpathy. Linxi Fan, Jonathan Hernandez, and Percy Liang. World of bits:
An open-domain platform for web-based agents. In International Conference Machine Learming, pp. 1444. PMLR, 2017.
[41] Paloma Sodhi, SRK Branavan, Yoav Artzi, and Ryan McDonald. Step: Stacked IIm policies for web actions. arXiv preprint 130.0370, 2023.
[42] Hongjin Su, Ruoxi Sun, Jinsung Yoon, Pengcheng Yin, Tao Yu, and Sercan Ô Ank. Learn- by-interact: A data-centric framework for self-adaptive agents in realistic environments. arXiv preprint arXiv:2501. 2025.
[43] Liangtai Sun, Xingyu Chen, Lu Chen, Tianle Dai, Zichen Zhu, and Kai Yu. Meta-gui: To- wards multi-modal conversational agents on mobile gui. Conference on Empirical Methods in Natural Language Processing. 2022. doi: 10.485500axXx.2255.1022
[44] Victor Talpaert, Ibrahim Sobh, B. Ravi Kiran, Patrick Mannion, Senthil Yogamani, Ahmad
A. A. Sallab, and Patrick Pérez. Exploring applications of deep reinforcement learning for real-world autonomous driving systems. arXiv preprint arXhv:1901.01536, 2019.
[45] Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS), pp. 23- 30. IEEE, 2017.
[46] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012.
[47] Dheeraj Vattikonda, Santhoshi Ravichandran, Emiliano Penaloza, Hadi Nekoei, Megh Thakkar, Thibault Le Sellier de Chezelles, Nicolas Gontier, Miguel Muñoz-MMrmol, Sa- har Omidi Shayegan, Stefania Raimondo, et al. How to train your Ilm web agent: A statistical diagnosis. arXiv preprint arXiv:2507.044 04103, 2025.
[48] Zora Zhiruo Wang, Jiayuan Mao, Daniel Fried, and Graham Neubig. Agent workflow memory. arXiv preprint 240::2209900422,, 2024.
[49] Zhepei Wei, Wenlin Yao, Yao Liu, Weizhi Zhang, Qin Lu, Liang Qiu, Changlong Yu, Puyang Xu, Chao Zhang, Bing Yin, et al. Webagent-rl: Training web agents via end-to-end multi-turn reinforcement learning. arxiv preprint arXiv:2505. 16421, 2025.
14

---

Published as a conference paper at ICLR 2026
[50] Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. Osworld: Benchmark- ing multimodal agents for open-ended tasks in real computer environments. arXiv preprint WIXv::444.077972. 2024.
[51] Tianbuo Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng. Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Vietor Zhong, and Tao Yu. OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments, May 2024. URL
http://arxin org/ ab///////////// 2404. 07972 arXiv:2404.07972 [cs].
[52] Frank F Xu, Yufan Song. Boxuan Li, Yuxuan Tang, Kritanjali Jain, Mengxue Bao, Zora Z Wang, Xuhui Zhou, Zhitong Guo, Murong Cao, et al. Theagentcompany: benchmarking Ilm agents on consequential real world tasks. arXiv preprint arXiv:2412.14100 2024.
[53] Jianwei Yang. Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V, November 2023. URL
attp:///axivv.oor//bb////22 hwwpgg/ab//3300 1144 arXiv:2310.1 111441 [cs].
[54] John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineering. Advances in Neural Information Processing Systems, 37:50528-5063 2024.
[55] Ke Yang. Yao Liu, Sapana Chaudhary, Rasool Fakoor, Pratik Chaudhari, George Karypis, and Huzefa Rangwala. Agentoccam: A simple yet strong baseline for Ilm-based web agents. arXiv preprint arXiv:2410. 13825, 2024.
[56] Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents, February 2023. URL http:
a arXiv:2207.01 00000 [es].
[57] Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. r-bench: A benchmark for tool-agent-user interaction in real-world domains, 2024. /ttp::s:s///rrxi..orr
0bs/2406.122 12045
[58] Suyu Ye, Haojun Shi, Darren Shih, Hyokun Yun, Tanya Roosta, and Tianmin Shu. Realwe- bassist: benchmark for long-horizon web assistance with real-world users. arXiv preprint arXiv: 2504. /0445, 2025.
[59] Ori Yoran, Samuel Joseph Amouyal, Chaitanya Malaviya, Ben Bogin, Ofir Press, and Jonathan Berant. AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks?, July 2024.
15711 arXiv:2407.15711.
[60] Chao Yu, Jiming Liu, Shamim Nemati, and Guosheng Yin. Reinforcement learning in health- care: A survey. ACM Computing Surveys, 55(1):1-36, 2021. doi: 10.1 155/3776777
[61] Jenny Zhang, Shengran Hu, Cong Lu, Robert Lange, and Jeff Clune. Darwin godel machine: Open-ended evolution of self-improving agents. arXiv preprini arXiv:2505.22954 2025.
[62] Li Zhang, Shihe Wang, Xianqing Jia, Zhihan Zheng, Yunhe Yan, Longxi Gao, Yuanchun Li, and Mengwei Xu. Llamatouch: faithful and scalable testbed for mobile ui task automa- tion. In Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology, pp. 1-13, 2024.
[63] Yanzhe Zhang, Tao Yu, and Diyi Yang. Attacking Vision Language Computer Agents via Pop- ups, November 2024. URL HtRLhtt::/ http:::://aaii.or//bbbssss//////
02391 1iXi:2411.00234
[64] Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Yonatan Bisk, Daniel Fried. Uri Alon. et al. Webarena: A realistic web envi- ronment for building autonomous agents. arXiv preprint 1XXi:2307.13854 2023. URL https: /twpba rena. dev
15

---

Published as a conference paper at ICLR 2026
[65] Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk. Daniel Fried, Uri Alon, and Graham Neubig. WebArena: re- alistic web environment for building autonomous agents. arXiv preprint arXhv:2307.13854 2024.
[66] Yuxuan Zhu, Tengjun Jin, Yada Pruksachatkun, Andy Zhang, Shu Liu, Sasha Cui, Sayash Kapoor. Shayne Longpre, Kevin Meng, Rebecca Weiss, Fazl Barez, Rahul Gupta, Jwala Dhamala, Jacob Merizian, Mario Giulianelli, Harry Coppock, Cozmin Ududec, Jasject Sekhon, Jacob Steinhardt, Antony Kellermann, Sarah Schwettmann, Matei Zaharia, Ion Stoica, Percy Liang, and Daniel Kang. Establishing best practices for building rigorous agentic benchmarks, 2025. /tLLatttss////a22.oo////a///////
02825
[67] Daniel M. Ziegler. Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv: 100. 2019.
16

---

Published as a conference paper at ICLR 2026
A STATEMENT ABOUT USAGE OF LLMS
We used LLMs in two ways: (1) to edit grammar, style, and suggest alternative phrasings during manuscript preparation, and (2) assist in coding and generating application data for experiments. All conceptual contributions, study design, and analysis were carried out by the authors.
B FUTURE DIRECTIONS
Beyond the design of benchmarks to evaluate reliability, OPENAPPS also lays the foundation for future advances in post-training agentic of foundation model:
Simulators for Agentic Post- Training. de facto approach for adapting FMs-whethe for alignment via RLHF (10 67) or DPO (4), or for improving reasoning performance via methods like GRPO (35 II. These approaches require
Reinforcement learning (RL) fine-tuning has become the
extensive interaction data-be it human preference annotations or curated reasoning datasets. In these settings, effective learning typically requires large volumes of interaction data human prefer- ence annotations for alignment, or curated reasoning problems for mathematical and logical tasks. Agentic learning poses even greater data challenge: unlike single-turn settings, agents must of- ten take multiple sequential actions, expanding the state-action space exponentially with planning horizon, and thus becoming substantially compute-effective solution for generating large-scale agentic interaction trajectories in a controlled setting.
sample-inefficient. OPENAPPS offers fast and
Safe Training Without Real-World Risk. Even with abundant data, training agents directly on production systems poses unacceptable risks-ranging from leaking private user data and corrupting critical files to executing harmful operations or triggering unintended purchases. In similarly high- stakes domains such as healthcare (60) and autonomous driving (20 44 4), the RL community has long relied on simulators to train and test agents safely. OPENAPPS provides
analogous capability: it supports the generation of risky, logically inconsistent, or noisy trajectories that would be infeasible or dangerous to collect in real-world systems. This opens the door to adversarial training, robust optimization, and curriculum learning over safety-critical scenarios.
Self-Improving Agents via Simulation. Inspired by systems like AlphaZero, future work could use OPENAPPS not only for evaluation but also for generating new tasks and configurations to support aef-impprvveeet. For example, agents could generate increasingly difficult task variants or use judge-based based verification to refine internal policies.
judge
Sim2Real Transfer and Generalization. The vision layed out in this section assumes that skills learned in the simulator transfer meaningfully the real world-a a chalennc widely studied in sim- to-real transfer literature (45). Future research should explore the generalization of agent behavior across task distributions and under robustness requirement (30). Specifically, researchers can select partitions of app versions, {A1, A2,... }. for training and evaluation to carefully probe generaliza- tion properties of agents. Given the scale of data generation possible, researchers can ensure training and evaluation splits do not overlap in terms of key correlations. For example, probing questions such as the training data distribution necessary for agents to generalize to particular factors such as fonts, colors, or layouts. Part of this program could also consider overfitting, shortcut learning, and memorization by probing learning dynamics in terms of training and evaluation splits.
C ADDITIONAL RELATED WORK
C.1 AGENT FAILURE EVALUATION
Recent studies show digital agents are vulnerable to adversarial conditions. Zhang al. (63) report that adversarial pop-ups reduce success rates dramatically (e.g.. VisualWebArena from 92.7% to 73.1%) Ma et al. (27) show that distractions such coupon banners derail agent trajectories. Ope- nApps, with fine-grained controllability and noise injection, provides a testbed for systematically stress-testing such failure modes.
17

---

Published as a conference paper at ICLR 2026
C.2 BUILDING AND TRAINING AGENTS
The pipeline for building an autonomous agent mainly involves selecting an appropriate, fixed foun- dation model (LLM VLM), defining input/output spaces, and optionally configuring memory modules. AgentOccam (55) reveals that simplifying both input structures and output action sets can unlock remarkable performance gains. SWE-agent (54) highlights the advantages of specialized Agent-Computer Interface for foundation models. Sodhi et al. (41), Chen et al. (9), Fu et al. (16) demonstrates that numan--writen/data--rriin
rules foster greater generalization. Agent Workflow Memory (48) introduces the concept of storing annotated trajectories as reusable workflows, en- abling agents recall and apply them in analogous scenarios, Agent S (2) envisions dual mtuarm system, episodic and narrative, evolving in tandem to enrich the agent's 's adaptability. Mcanwhile, Automated Design of Agentic Systems (99) and Darwin Godel Machine (6I) advocate the auto- matic design and dynamic updating of system wrappers through code, thereby minimizing the need for manual design or intervention.
While much progress has been made in boosting agent performance with fixed foundation models, a new wave of research now focused on training these models directly within their deployment environments. ScribeAgent (39) demonstrates that fine-tuning with large-scale, real-world workflow data can yield significant gains. Learn-by-nnteract (42) generates hindsight semantic labels for agent trajectories, which are then leveraged for further fine-tuning. WebRL (31) introduces a self-evolving curriculum to address the challenge of task scarcity, enabling agents to adapt and learn more effec- tively. WebAgent-R1 (49) investigates end-to-end, multi-turn RL for agents. Vattikonda et al. (47) provides empirical insights into balancing computational resources between supervised fine-tuning and on-policy reinforcement learning for optimal agent training.
Beyond benchmarks, methods to boost agent performance include labeling visual web elements (53) and hierarchical architectures with HTML simplification ①. New agent models such AgentOc- cam and GAIA have also been introduced, specializing in web-based tasks via fine-tuning, More recently, GAIA2 expose app functionality via text using the MCP protocol (3). Our OpenApps complements these works by providing a platform to systematically evaluate such methods under controlled conditions and can potentilaly be intergated with orchestration frameworks such as ARE 3 15)
C.3 SIMULATED ENVIRONMENTS IN RL
Simulators have long played central role in reinforcement learning (RL). Because RL. agents typi- cally require vast amounts of interaction data, direct deployment in the real world is often infeasible due to cost, safety, or logistical constraints. Simulated environments offer several advantages: they can be run at accelerated speed, reset deterministically. and instrumented for complete state access, thereby enabling reproducibility and controlled experimentation. Foundational work such as the Ar- cade Learning Environment (ALE) (5) and OpenAl Gym 7 provided standardized benchmarks and interfaces that allowed the community to compare algorithms under shared conditions. The land- mark DQN work (29) further demonstrated the effectiveness of simulators by showing human-level performance on Atari games through large-scale training in ALE.
In robotics and control, physics-based simulators such as MuJoCo (46), PyBullet, and Isaac Gym have become indispensable. These platforms make it possible to train agents in environments that approximate real-world dynamics without exposing hardware to risk degradation. They also en- able advanced techniques such as domain randomization (45), where simulated environments are deliberately varied improve transferability the real world. By serving as safe and scalable prox- ies for embodied interaction, these simulators have been central to progress in continuous control and sim-to-real transfer.
More recently, RL. research has expanded toward open-ended and high high-dimensional environments, where agents must master long-horizon exploration and compositional skills. Beyond Atari, ALE has been extended with more challenging tasks, and platforms such as MineDojo (14) and LS- Imagine (24) leverage Minecraft-stylee open worlds to study the challenges of exploration, planning, and credit assignment across vast state spaces. These environments highlight the role of simula- tors not just for safe and efficient data collection, but also as a means of stress-testing agents on increasingly realistic and unstructured tasks.
18

---

Published as a conference paper at ICLR 2026
Our work seeks to join the spirit of this line of work for the domain of digital agents.
D
METHOD
D.1
EXAMPLE OF CONFIGURATION YAML FILE
Listing Example YAML to configure OpenCalendar in OpenApps
style:
# Event visual placeholder for UI agent, text aria label for AXTree _dd_even__display:
placeholder:
title: 'Event Title'
date: 'YYYY-MM-DD' # Date format
description: 'Event description..."
url: htttps::/examppe.coom
invitees: 'John, Jane, etc.
location: 'None'
aria_label:
title: 'Event title'
date:
'Event date, YYYY-MM-DD'
description: 'Event description' url: 'Event URI'
invitees: 'Event invitees'
location: 'Event location'
Color scheme
colors:
primary: '#1095c1'
Primary color for buttons, links, etc.
primary_hover: 1#0a6d8a'
Hover state for primary elements
secondary: "46c757d'
Secondary color for less important elements
background: 'ffffffE'
Main background color
text: '/212529'
Main text color
error: 'ddc3555'
# Error messages color
border: '#ced4da'
Border color
Typography typography:
font_family: 'sans-serif' heading_font: 'sans-serif base_font_sizer '16px'
Main font family
Font for headings
Base font size
heading_size: '1.5rmm
Size for headings
Button styles
buttons:
border_radius: 10. 30.37mmmm'
Border radius for buttons Button padding
padding: 10.5rem 1rem'
# Layout
layout: container width: '1008'
Width of the main container
spacing: '1rem'
Standard spacing between elements
events:
title: WACV 2026 Abstract Deadline date: 2025-07-11
description:
$ WACV 2026 Abstract Deadline
19

---

Published as a conference paper at ICLR 2026
Winter Conference Applications of Computer Vision abstract submission deadline.
##
Important Dates:
Abstract Beadlines July 11, 2025 - Full Paper Deadline: July 18, 2025
Submit abstracts through the conference portal.
url: https: https://wacv202: coeet..t
location: Online
invitees: null
title: RACV 2026 Paper Deadline date: 2025-07-18
description:
# WACV 2026 Full Paper Deadline
Final deadline for Winter Conference on Applications of Computer Vision paper submissions.
Papers must be submitted the required format with all supplementary materials. url: bttp:::/wwavv225.thaae location: Online
invitees: null
D.2 VISUAL EXAMPLES OF APP VARIATIONS
lwdes
88
Chillppigg
Figure 7: Examples of the OpenTodo page with different app variations.
D.3 TASKS
We pre-populated OPENAPPS with 15 high level task objects, each with several natural language goal variants. We show a sample of the task goals in Table [
20

---

Published as a conference paper at ICLR 2026
Table Sample goals for the evaluated tasks
Ask Bob Are we playing baskethall on Saturday?"
Save "Bockelwitz, Germany' to the 'Saved Locations' panel in maps.
Make calendar entry for Christmas shopping the 14th (2025). need a calendar entry for the team lunch on July 22nd, 2025 noon. Create an empty python (.py) file under the developing folder in code editor. Can you add Play piano' to my todo list?
Create empty python (.py) file under the code editor directory. Go to the Calendar app and add my meeting with Dennis on April 1st of 2026. Return to the home page Can you delete the AAAI paper submission deadline from my calendar? It should be early August 2025. Please remove the 'WACV 2026 Abstract Deadline' event from July 11th, 2025 from my calendar. Can you mark Answer emails' done in my todo list?
Mark 'Update calendar' as done in my todos.
Please remove the 'WACV 2026 Abstract Deadline' from my calendar. Can you remove Answer emails' from my todo list. I need to duplicate the CHI 2026 Paper Deadline event. Please create copy next year.
Go to code editor, and remove the entire folder with name empty_folder' Add the following item, Mens Casual Cargo in 'orange', to my cart.
Create an empty C++ (.pp) file under the code editor directory. Go code editor and remove the entire folder with name 'developing'.
Send Charlie my last message from Bob
Forward Charlie my most recent message from Bob. Can you add 'Buy milk' to my todos?
Add the following item, OWYN 100% Vegan Bant-Based Protein Shakes my cart. Add 'get grandma gift' to my todos.
Navigate to the todo app page
Go to code editor, and delete all existing files and folders.
Demande Bob "est-ce qu' on va jouer du Basketball samedi?"
Can you mark Buy groceries" done in my todos?
Go to the todo app page
Add the following item, Magical Hair Treatment Mask in 'I '120ml', my cart. Find the conference deadline on July 18th, 2025 and duplicate it to October 15th, 2025. Remove 'water plants' from my todos.
Help me delete everything in my current cart.
Add Bockelwitz my places.
Ajouter Bockelwitz sur mes licux préférés.
Can you remove 'Make fruit salad' from my todos.
I need to duplicate the CHI 2026 Paper Deadline event.
Go the shop app. click on cart, and remove all items.
21

---

Published as a conference paper at ICLR 2026
D.4 ACTIONS
Categoory
Primitive
Deecription
fi11
PiI inpat field with
click(bs4, button)
Click an elenent.
dblelick(bid, button)
Doablo-llik an element.
bid
preas (bad, key_comb)
Focus an element and press a combination of Joys.
focua(bid)
Poens element.
clear(bid)
Clear on inpat fildd
select, optionnbid, optiooa)
Select one multiple options drop-down element.
drag. and drop.drotttronbee btd, to_bid)
Drag drop
another.
upload file file)
Click a 'Glechooser' elenennt, then select one multippe Inpat files for upkaad
mouse_nove(i. nove(x, 7) y,
mouse,
More the mouse location. Move
pross hold
mouse_ utt, y, button)
Move the mouse then release button.
click(s, 7, button)
Move the mouse double-click button.
and click button.
mouse, dblclick(x, y, button)
coord
mouse drag and drop_dptptttoo.... fron_y, te.x, to.y)
Drag and drop from location to location. Click 'ficchooser' location, them solort multippe Inpat for upload.
uplundd file(x, y,
keyboari__owmm(er)
Press and holde bayboard koy.
keyboard_ ukay)
keyboard ley.. Prees combination of keys.
keyboard_preee press (key_comb)
keyboard_tyyy Eepeardd__pppeeeee (tpcc))))
Types string of text through keyboard.
keyboard (eoprrtttexx(tx))))
Insert string sext in the currently focused elerment.
Open
tab
tab, closa()
Close the current tab.
tab__ Sob_ffocuStIddes
Bring tab front (activate tab).
go. back()
Navigate to the provious pagy history.
forvord goto(ur))
Navigate to the next page in history.
Navigate to a
rrag. (raasage)
Send message the in the chat.
report, snfeasible wcroll (dr, dy)
a special massage the chat and terminate. Scroll piools X and/or Y direction.
misc
nocp(secoads)
aothing.
Figure 8: The action set provided through BrowserGym, copied from Appendix A 23
Our agents rely on subset the action set provided through the BrowserGym API, see Figure for detail. Here we provide the full set of actions available agents:
click, fill dblclick, clear, select.option
drag arag-andddrop. hover, go-back, go-forward goto, scroll, mouse click, mouse dblclick, mouse move. mouse down, mouse up, mouse click, mouse dblclick. mouse wewpeddraaaaddddedde keyboard.up, keyboard.press, keyboard keyboard.insert.ceee inw.dtttteeettteeeeees
mouse casseeepplddd__ee file keyboard,down.
For visual only agents such as UI-TARS provide the subset supported by Ul-agents:
back go. forward, goto mouse-clie
mouse .dllllcck scroll moosee mouse_down, mouse puuse_upp mouse click mouse dblelick. mouse mouse_drag and_droo drop mouse mouse.upload file keyboard down, keyboard_up keyboard_presss keyboard.type. ke keyboard.insert.tex.
D.5 MEAN ABSOLUTE DEVIATION
In addition to standard deviation, to measure reliability, we also measure fluctuation in agent success rates using the mean absolute deviation (MAD) of rewards across for given task. Given set of rewards Ry from an agent deployed in a fixed app version (vk), we measure reliability for app wk
22

---

Published as a conference paper at ICLR 2026
Table 2: Tasks with the largest fluctuaatinn in success rate across application variations. maximum and minimum success rates across app variations.
We show
Model
Task
Maximum pass@1
Minimum pass@1
Difference
GPT-4o
ForwardMessageTaskk
0.43
0.00
0.43
Kimi-VL-A3B -L--ABB-nstttt
Navigate TaPageTTask
1.00
0.00
1.00
Qwen2.5-VL.
Navigate Task
1.00
0.00
1.00
UI-TARS-I.5-7B
RemoveSavedPlace
1,00
0.00
1.00
claude sonnct
ForwardMessage Task
1,00
0.00
1.00
llava-v 1-bm
Navigate PorigateeoPageeee ToPage Task
0.50
0.00
0.50
Fixed ok App Reliability
Σ N.R..
Ti
Σ TyER.
7
Tb
where is the number of times agent attempts the given task. Equation 2 is measure of reliability that can be captured with common agent environments that rely on fixed clones of apps.
When deployed, however, an agent likely encounter many versions of app. For example, for a calendar there are several dozen (if not more) calendar apps, each with ever-evolving updates and configurations, yielding many app variations. To measure reliability more generally in way that includes the many app variations an agent is likely to encounter, we compute:
Overall Reliability
Σ
T1
Σ
Tt
nd
nd ry E
r E.E.Re....RRdd Roa]
where d is the number of variations we simulate. Crucially, Equation 3 also captures fluctuations in agent performance across app variations that an agent is likely to encounter when deployed. In evaluations, compare the ratio of Fixed App Reliability / Overall Reliability to assess how performance fluctuation can stems from variations in apps, which we show can be quite large in the coming sections.
E ADDITIONAL EXPERIMENTAL RESULTS
We show in Table the task with the largest change in success rate app variations. We find all agents success rates can fluctuate from 0 to 100% success depending on the app variations; with the exception of GPT-4o that fluctuates from 0 to 43%. We also show task success across all app variations in Table 3 where we see even closed source agents based GPT and Claude can have very different success rates depending on the app variation encountered.
In Figure 9 we show many temperature values yield reasonable performance. We show performance for Claude and Kimi-VL on three tasks across varying values for temperature. Figure 10 we show
agent reliability across app variations using mean absolute deviation (see Appendix D.5). In Table
we report the success rate and reliability of each agent by task. Table we show the performance across appearance variations, and across content variations in Table 6 We show resolution analysis across app variations Table 7
23

---

Published as a conference paper at ICLR 2026
Table Tasks with variable success rate across app variations. We show all app variations for each task.
Agent
Task
App Variation
Task Success
GPT-4o
ForwardMessage
adversarial descriptions black and white challenging_font dark_theme default german
0.00
GPT-4o
FoswardMessage Task
14.29
GPT-4o
ForwardMessage Task Tass
42.86
GPT-4o
ForwardMessagel Task
28.57
GPT-4o
ForwardMessage Task
7.14
GPT-44
ForwardMessage" TaaaaaaeTTs Task ForwardMessinn Task Task Navigate' ToPage'Tass Task Navigate ToPage PageTTask Task
0.00
GPT-44
long descriptions misleading descriptions adversarial adversarial _descriptions black_and and white challenging-fon dark theme default german
0.00
GPT-4o
Forward Messagee lessage'Task
0.00
Kimi-VL--A3BBInstrmu
12.50
Kimi-VL-A3B-Insn
100.00
Kimi-VL-A3B-Instne
Navigate ToPage PageTTask
87.50
Kimi-VL-A3B-Inste
Navigate ToPage PageTaask Task
100.00
Kimi-VL--A3B-Instmee
Navigate' ToPage' PageTaast Task
92.86
Kimi-VL--ABB-Instnu
Navigate ToP ToPage' Task
0.00
Kimi-VL-AABB-nnstruet
Navigate ToPage Task PageeTask
long siggddecrrptions misleading deecriptionss
75.00
Kimi-VL--ABB-Instruut
Navigate MavigateToPaal ToPage Task
75.00
Qwen2.5-VL
Navigate ToPage' Task
adversarial descriptionss black_and. wwhte challenging font dark theme default
12.50
Qwen2.5-VL
Navigate ToPage
PageeTask Task
0,00
Qwen2.5-VL
Navigate Page' Task
0.00
Qwen2.5-VI.
Navigate' ToPage'Tase T'ask
0.00
Qwen2.5-VI.
Navigate' ToPage' Taeesass
5.88
Qwen2.5-VL
Navigate PageTTssk
german
0.00
Qwen2.5-VL
Navigate' ToPage
Page'Task Task
long descriptions misleading descriptions
90.00
Qwen2.5-VL
Wavigate'T To Page'aaaaa
100.00
UI-TARS-1.5-7B
SemoveSaved!P
adversarial descriptions
0.00
U--ARRS-.5--B
RemoveSavedPlant
black and white challenging_fone
100.00
UI-TARS-1.5-7B
RemoveSavvdPPlae
100.00
U1-TARS-I.5-7B
RemoveSavedPlace
thrk_ default
0.00
UI-TARS-1.5
RemoveSavedPlace
93.33
UI-TARS-1.5-75
RemoveSavedPlace
german
100.00
UI-TARS-1.5-7B
RemoveSavedPlace
long descriptions
0.00
UI-TARS-1.3 --55-7
RemoveSavedPPe
misleading deseriptions
20.00
Claude 4 _sonnet
ForwardMessann MessageTssse Task ForwardMessager
adversarial descriptions
50.00
Claude 4 _onnnet
ForwardMessan sageeTask
black_and white challenging dark_theme default
50.00
Claude_4 _snneet
MessageTaak Task ForwardMesseee eesageTaask Task ForwardMessager
33.33
20.00
Claude sonnet
22.22
Claude 4 sonnet
Forward TaaageeTass Task
german
75.00
Claude soonnet
ForwardMessage" Task Task
long descriptions misleading _descriptionn adversarial adversarial.descciptionn black andd.andd.vee white challenging.font dark theme default
100.00
Claude_4_sonne
ForwardMessage Task
25,00
Llava-vl. Llava-vl.--miitraal--hhh
NavigateToof ToPage Task
0.00
Navigate ToPage'Tast Task Navigate' ToPage' Task
50.00
Llava-vl.--mistrall-7bbb
12.50
Llava-vl.--mistral--b-ht
Navigate ToPage' Task ToaggTTask
10.00
Navigate' ToPage' Task
35.29
Llava-v1.6-mistae
Navigate ToPage'Tast Task
german
0.00
Llava-v...-miittra-----ff
NavigateTof ToPage
PageTask Task
long den_deesrritions misleading descriptions
0.00
Llava-vl.6-mistral--b-ttt
NavigateToPage ToPage
0.00
24

---

Published as a conference paper at ICLR 2026
Wittbe pezbae
60
Figure 9: We compare agent performance for Claude (left) and Kimi- Limi-VL (right) as the temperature varies.
Claude Sonnet
26.3
deviation
Within Fixed Apps
Claude Sonnet (Fixed Apps).
18.3
Across App Versions
GPT-4o
12.1
GPT-4o (Fixed Apps).
9.5
Kimi-VL-A3B-Instre
34.2
Kimi-VL-ABB-Instruc (Fixed Apps)-
12.7
LLaVA-11.6-7
10.8
LLaVA-v1.6.. Apps)
8.2
Qwen2.5AV 5-VL.
24.6
Qwen2.5-V (Fixed Apps)
13.0
UI- TARS-1.5-7B
33.7
UI-TARS-1.55 5---B (Fixed Apps)
MAD Task Success
Figure 10: Performance within a fixed app version overestimates reliability. We compare the mean absolute deviation of task success rates in two settings: 1) within fixed app version (in blue), compared to 2) overall deviation that also accounts for difference in agent success rates across app variations.
25

---

Published as a conference paper at ICLR 2026
Tolk
Mede
Keen
E16
1na-l Leveara Thir
0.0
0.20
E/0
UETARS 5m
0.55
8.5
6446
GPT-4
00:33
610
0.00
Gwed SA
0.80
010
dace foone
0.64
0.50
6-16
UPSAKS Ling
(03)
ETC
A
0.0
610
088
0.53
0.48
0.33
610
METARS 7-
6:32
diace
OPT44
10377
110
Deplnxkrceellad
080
0.30
6:0
dasc
E/0
0:31
610
Bea-lS-aara Thia
0.00
0.20
040
dlase
0.86
6:18
Gead K.
C:4
148
0.30
510
1757
1131
MasayXTaat
0,25
6.22
6:14
610
Tave Lenaara Thia
033
0.49
546
0.33
0/0
0.00
112
10:77
110
Kaul A18 Saardc
080
0.30
610
045
0:33
1
010
Ina-l Lenaraa Thoa
0.82
0.20
0.0
0.55
0.53
8:50
0.00
Owad2.
6.11
GP1-4
148
0,30
510
1153
1131
E10
6.28
TARS
0117
0100
031
0.0)
2:44
0.32
0110
0.0
Suarace
Owod S.
03:3
01.55
0:h
0313
0.17
6:10
Table Breakdown of model performance by task. Model performance (as measured by pass @ over random seeds) on all tasks. We report the mean absolute deviation (MAD) and standard devi- ation (std) of rewards over random seeds. We use the default environment content and appearance, and the task prompt is explicit and fixed. All models use visual and AX tree inputs, with the excep- tion UI-TARS, which a UI-visual only model.
26

---

Published as a conference paper at ICLR 2026
avg. pass@1
avg. std
avg. MAD
Model
Appearance
dark theme
0.69
0.29
0.23
default
0.75
0.27
0.21
claude sonnet
challenging font
0.76
0.26
0.21
black and white default
0.77
0.23
0.19
0.78
0.15
0.12
dark theme
0.81
0.13
0.11
GPT-4o
black and white
0.82
0.08
0.06
challenging font dark theme default
0.84
0.11
0.10
0.12
0.10
0.08
0.21
0.15
0.13
Kimi-VI--ABB-Instmut
challenging font black and white
0.23
0.16
0.13
0.29
0.15
0.12
dark theme challenging font default
0.01
0.02
0.01
0.01
0.02
0.01
sera-vl.6-mistalll
0.02
0.03
0.03
black and white challenging font black and white dark theme default
0.04
0.06
0.05
0.03
0.06
0.05
0.05
0.11
0.08
Qwen2.5-VL
0.05
0.09
0.08
0.08
0.17
0.12
dark theme
0.05
0.10
0.07
default
0.41
0.24
0.18
UI-TARS-L.5-BB
black and white
0.42
0.18
0.14
challenging font
0.43
0.15
0.12
Table 5: Agent reliability can be low in terms of performance across appearance variations. Model performance (as measured by the pass@1 across random seeds averaged over all tasks) can differ greatly across content variations. We report the standard deviation and mean absolute devia- tion of rewards across seeds, averaged over tasks. The task prompt is explicit and fixed.
27

---

Published as a conference paper at ICLR 2026
avg. pass
avg. std
avg. MAD
Model
Content
default
0.75
0.27
0.21
adversarial descriptions
0.75
0.22
0.17
claude 4 sonnet
misleading descriptions
0.75
0.26
0,20
long descriptions
0.78
0.23
0.18
german
0.82
0.21
0.16
default
0.78
0.15
0.12
misleading descriptions
0.79
0.11
0.08
GPT-4o
german
0.79
0.09
0.07
adversarial descriptions
0,81
0.09
0.07
long descriptions
0.82
0.07
0.05
adversarial descriptions
0.02
0.05
0.03
german
0.02
0.06
0.04
Kimi-VL-ABa -L--aut-Inttruct
misleading descriptions
0,07
0.06
0.05
long descriptions
0.12
0.03
0.03
default
0.21
0.15
0.13
misleading descriptions
0.00
0.00
0.00
german
0.00
0.00
0.00
long descriptions
0.00
0.00
0.00
adversarial descriptions
0.01
0.02
0.01
default
0.02
0.03
0.03
german
0.02
0.06
0.04
default
0.08
0.17
0.12
Qwen2.5-VL
adversarial descriptions
0.10
0.15
0.12
long descriptions
0.14
0.11
0.08
misleading descriptions
0.23
0.12
0.09
long descriptions
0.19
0.11
0.09
adversarial descriptions
0.23
0.12
0.10
U--TARS-L.5-BB
misleading descriptions
0.23
0.16
0.14
default german
0.41
0.24
0.18
0.42
0.17
0.13
Table Agent reliability can be low terms of performance across content variations. Model performance (as measured by the pass @ across random seeds averaged over all tasks) can differ greatly across content variations. We report the standard deviation and mean absolute deviation of rewards across seeds, averaged over tasks. The task prompt is explicit and fixed.
Appearance screen resolution
black and white
challenging font
dark theme
default
FHD (1920 x 1080)
0.64
0.76
0.06
0.69
HD (1280 x 720)
0.52
0.70
0.61
0.63
Low (480 320)
0.42
0.42
0.47
0.48
Table 7: Higher HD is longer always better when we fix the the content and vary the appearance.
28

---

Published as a conference paper at ICLR 2026
E. VARIATION INTERACTIONS
To capture possible interactions across variations, we explore combinations of appearance and con- tent changes. We apply combinations of the appearance and content variations across apps to capture their interactions. We then evaluate 12 tasks covering app navigation, adding items to the todo list, and adding calendar events across three agents (GPT-4o vision only, GPT-4o vision + AXTree, and UI-Tars vision only). We report results for these experiments below highlighting interactions can lead to different success rates (see Table g For example, we see Ul-Tars and GPT-4o vision agents task success drops when apps contain combination of English and German compared to German alone.
E.2 MORE COMPLEX MULTI-STEP TASKS WITH PARTIAL REWARDS
We implement 10 multi-step tasks across todo, calendar, and messenger. These tasks involve for example. looking up an event on the calendar, messaging its title to friend, and adding a related todo the todolist. With these multi-step tasks, we also measure incremental rewards if the task was partially completed (say a message was successfully sent, but the todo item wasn't 't added). We report the reward, capturing incremental rewards as well as a column indicating whether at least one step was successfully completed. As shown in Table 9 we observe agents often struggle to complete multi-step tasks, even GPT-4o with full access to the AXTree, We see for example, GPT- 4o and Ur-Tars complete single steps towards the task at 3x higher rate than the full multi-step task.
E.3 VARIATIONS GROUNDED IN POPULAR DESIGN CHOICES AND LANGUAGES USED IN THE WEB
To ground our variations in real world distributions, we run additional experiments using the most popular fonts, colors, and language found on the web. Specifically, we select the three most popular fonts based on data from https: http://inoootdeeeeeee wttp:/ooppaoaaaaooooooooiooo fonts (Inter, Roboto, Open Sans), three most popular color palettes from httpp::/cooorruutttco wttp:/rtttuuttccpapeeeeeeeeeeeee popular and translate the todo app contents into 3 additional languages based on popular- ity on the web here: htttps: / /en. wik pedi a. o Internet to cover 4 out of 5 most commonly used languages.
We then evaluate three agents (GPT-40 vision, GPT-4o with vision + AXTree, and UI-Tars vision only Ul-model) on three tasks requiring the agent to add items to the todo list. We report these new results grounded in real world variation below as a demonstration of the flexibility of OpenApps in modeling real world variations. The community simply by modifying yaml variables for font, button colors, and language via the content.
of course easily extend these experiments
We report in Table 10 the task success rate for adding items the todo list across these popular design choices and languages. We find even among these popular choices, task success rates can vary.
F AGENT BEHAVIORS
Loops. In Table we show the average loop count across models, appearance varia- tions, and content variations. When inspecting loops qualitatively, the most common loops are sequences of the same action being repeated, e.g., GPT-4o produces click (47) click (47) click (47) click (47) ates oal.appaikiiiiiiiiooooddiiaaaaaa and UI-TARS- click
click (47) click click (47), Kimi-VL gener- click (17)
1.5-7B outputs keyboard.press (key=' h Keyboarddresss (key=' ctrl ) These examples reveal that loops can be, but are not always problematic. The bid [17] corre- sponds to Section ', which is task-irrlevantt empty section header. In contrast, the bid [47]
29

---

Published as a conference paper at ICLR 2026
variations
task success
agent
GPT-4o (vision AxTree)
dark theme dark theme + default dark theme + german default
1.00
1.00
1.00
1.00
english + german
1.00
german
1.00
GPT-4o (vision only)
dark theme
0.11
dark theme + default
0.11
dark theme + german
0.11
default
0.11
english german
0.00
german
0.11
UI-TARS-1.5-77
dark theme
0.00
dark theme + default
0.11
dark theme + german
0.11
English
0.67
english + german german
0.67
0.75
Table 8: Variations can interact to form new combinations of app versions. We show agent task success across 15 tasks covering navigation, adding items todo, and add calendar events.
agent
variations
reward
at least one correct step
GPT-4o (vision + AxTree) GPT-4o (vision only)
default
0.22
0.67
default
0.00
0.00
UI-TARS-1.5-7B
default
0.10
0.29
Table We show performance on tasks requiring multiple steps across todo, messenger, and calen- dar. We measure partial reward with respect to whether the agent completed each incremental step in each app. We also show whether at least on step was correctly completed.
30

---

Published as a conference paper at ICLR 2026
agent
variations
task success
GPT-4o (vision + AxTree)
Inter
1.00
Open Sans
1.00
Roboto
1.00
default
1.00
french
1.00
german
1.00
popular colors
1.00
popular colors 2
1.00
popular colors 3
1.00
spanish
1.00
GPT-4o (vision only)
Inter
0.00
Open Sans
0.00
Roboto
0.00
default
0.00
french
0.00
german
0.00
popular colors
0.00
popular colors 2
0.00
popular colors
0.00
spanish
0.00
U.-TARS-1.5-7B
Inter
1.00
Open Sans
1.00
Roboto
0.33
default
1.00
french
1.00
german
1.00
popular colors
1.00
popular colors 2
1.00
popular colors 3 spanish
1.00
1.00
Table 10: We measure agents task success when asked to add items to the todo list across popular choices for fonts, colors, and languages. We find even among the most popular choices, agent success rates can vary.
31

---

Published as a conference paper at ICLR 2026
avg. loop count
Model
Appearance
Content
default
german
0.10
long descriptions
0.11
dark theme
default
0.14
default
adversarial descriptions
0.14
claude sonnet
challenging Tont
default
0.14
black and white
default
0.15
misleading descriptions
0.15
default
default
0.15
challenging font
default
0.32
black and white
default
0.34
dark theme
default
0.41
default
0.41
GPT-4o
misleading descriptions
0.42
default
adversarial descriptions
0.44
german
long descriptions
default
long descriptions
1.16
black white
default
1.20
dark theme
delault
1.27
delault
misleading descriptions
1.33
Kimi-VL-Ar Bomi-VL-A3B-Instruel
challenging lont
default
1.40
default
1.43
default
german
1.66
adversarial descriptions
1.74
black and white
default
1.18
def'ault
udversarial descriptions
1.18
challenging font
default
1.23
llaa-vv.6-mistree 1.6-mistral-766
dark theme
default
1.24
german
1.25
long descriptions
1.28
default
misleading descriptions
1.29
default
1.35
misleading descriptions
1.22
default
long descriptions
1.32
dark theme
default
1.33
challenging font
default
1,38
Qwen2.5-VV VL
black and white
default
1.41
default
1.43
default
adversarial deseriptions
1.45
german
1 1.80
black and white
default
0.93
detault
german
1 .09
challenging font
default
1.21
default
1.24
UI-TARS-L.5-BB
misleading descriptions
1.47
default
adversarial descriptions
1.54
long descriptions
1.63
dark theme
delault
2.21
Table 11: Agents are more or less prone loop actions depending on the app variations they encounter: The average count of loops (across all tasks and random seeds) varies across appearance and content variations. We define loops as maximal sequences of actions entirely consisting of repeated subsequence. The task prompt is explicit and fixed.
32

---

Published as a conference paper at ICLR 2026
avg. invalid action count
Model
Appearance
Content
default
default
0.00
GPT-4o
dark theme
default
0.01
default
adversarial descriptions
0.07
default
0.01
default
german
0.07
Kimi-VL-A3BB-nstmun
long descriptions
0.07
black and white
default
0.07
default
adversarial descriptions
Ilava--..--mistall-
default
german
0.05
misleading descriptions
0.13
default
detault
0.01
black and white
default
0.02
Qwen2.5-VL
misleading descriptions
0.07
default
german
0.15
long descriptions
0.20
dark theme
default
0.71
default
german
1.84
black and white
default
1.88
challenging font
default
1.91
UI-TARS-L.5-BB
delault
2.12
misleading descriptions
2.23
default
long descriptions
2.25
adversarial descriptions
2.60
Agents more likely hallucinate actions in certain app variations than others. The average count of invalid actions (across all tasks and random seeds) seems to be higher when the content of apps contains misleading or adversarial descriptions. We define invalid actions as generated actions that do not have the correct syntax, are not part of the custom or default actions of a model, or have bad arguments (e.g., type error, argument does not exist). Any unreported models, appearance variations, and content variations have an average invalid action count of 0. The task prompt explicit and fixed.
Table 12:
corresponds to button Next in OpenCalendar, which agents are often required to click multiple times to navigate to the next calendar page to complete calendar tasks.
Invalid actions. Models may hallucinate actions more due to content than appearance variations, and when an application contains distractor information, c.g., adversarial or misleading descriptions (see Table 12). The five most common invalid actions for each model include:
GPT-4o: click (23), noop, check.ax.tree () mouse.click (x=612 y)
Kimi-VL-A3B-Instten
click (bid) 'n'',, click (bid) |n<)action>
click (bid) in<laction>]n "python|npyautttnt htwpol/laaatttuu..iiio... click (x=0.523, y= 466) In " click (bid) \n "In" "ptttooo http::a..."""""pooooooooaaaaaaaa
(x=0. 000, 000,
y=0 000) n n' click (bid) "ptctb"""""""""to"/"""o "prthooninyaattopii....... y-0. \n'..
llava-v..--mistrallBboo remove.item (water plants), duplicate (17)
click([67] link 'main-page', enter (searchInput, Bockelwitz, Germany), no-op, (' http:://wwwwwwwmmpleepm cttpeeeccmmmmmpm type (OWYN 100% Vegan Plant-Based Protein Shakes Cold Brew Coffee, 12 F1 Oz Dairy-Free, Gluten-Free, Soy-Free, Tree Nut-Free, Egg-Free, Allergy-Free, Vegetarian)
Qwen2.5-VL:
finished (), w)it(), scroll (direction=/ point=' (966, ") finished, None
UI-TARS-1.5-7B:
Qualitatively, we observe diverse kinds of invalid actions: incorrectly-formed versions valid actions (e.g. mouse click (x=612 y), no-op), well-formed actions that do not exist in
33

---

Published as a conference paper at ICLR 2026
intent misunderstanding rate
Model
Appearance
Content
GPT-4o
default
german
0.06
default
mislcading descriptions
0.01
challenging font
default
0.18
black and while
detault
0.24
Simi-VL-A3BBIIssin
default
detault
0.25
dark theme
default
0.30
default
german
0.50
misleading descriptions
0.03
default
adversarial descriptions
0.06
long descriptions
0.08
challenging font
default
0.11
Ilava-v1. -----stl--itt
dark theme black and white
dctault
0.12
default
0.18
default
german
0.21
default
0.23
dark theme
default
0.02
detault
detault
0.03
challenging font
default
0.04
black and white
default
0.05
Qwen2.5-VL
german
0.31
misleading descriptions
0.36
default
long descriptions
0.40
adversarial descriptions
0.45
challenging font black and white
default
0.02
default
0.05
default
0.05
UI-TARS-I.5-7B
german
0.14
default
adversarial descriptions
0.31
long descriptions
0.35
misleading descriptions
0.37
dark theme
default
0.50
Table 13: Agents may be prone misunderstand users' intent when the apps content lengthy, misleading, or adversarial content. We report the rate of intent misunderstandings over tasks and random seeds for each appearance and content variation. An agent "misunderstands intent" when navigates to page irrelevant to the task. Any unreported models, appearance variations, and content variations have intent misunderstanding rate of O. The task prompt is explicit and fixed.
the environment (e.g., remove.item, finished), and valid actions with bad arguments (c.g., click (bid), scroll (direction=' down', point=' (966, 546) )))
Instruction copying. We do not observe that any models regurgitate in-context examples from their system prompt (e.g.. f' 'my text"), mouse-click (200, 300, outton="left!)
Intent misunderstandings. In Table we show the intent misunderstanding rate across models, appearance variations, and content variations. We note that we do not consider DovigateTToPageTT as the explicit goal of this task is to navigate to a page rather than to understand user intent. An agent navigating to incorrect page is not always fatal (i.c., yields an unsuccessful run); however, unnecessary steps in runs can yield higher costs for agent developers. Additionally, agents may navigate to irrelevant pages outside the OPENAPPS environment, e.g., https: // //www/ example. com/ onl ine- shop, https:/ htap:s://llfffaaa /leaflet js. com/.
34


Reviews:
[{"review_id": "1WRazLG6wU", "reviewer": "Reviewer_MLr5", "summary": "The paper addresses the question of reliability of UI agents when the underlying environments change (without affecting overall functionality). The key idea is to create multiple appearance and content variations of a set of UI apps and measure agent performance across these variations. The authors find that all models have significant variance in performance, with larger models being relatively more stable compared to smaller models. The analysis further illustrates that certain specific types of variations in the environment lead to consistently larger drops in agent performance (e.g. using darker themes for apps leads to consistent failures).", "questions": "- I appreciate the state based evaluation and the rigorous evaluation. However, it seems like partial progress towards the task is not accounted for in the current analyses. Is there a way to extend the analysis to partial progress? \n\n- How are the within app fluctuations observed? Are these deviations computed across all queries? Or averages of standard deviations over multiple attempts for each query?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The question of reliability is an important one, especially when considering how dynamic real world apps often are. Assessing agent performance under these variations is quite useful.\n\n- The analysis shows that multimodal models have large variability in performance, raising important questions about their suitability in practical settings, where the underlying environments change. \n\n- This benchmark could be a useful additional evaluation for multimodal agent solutions in addition to the existing benchmarks.", "weaknesses": "The main issue I have with this work is that the motivations for the variations is not adequately justified and the choices for curation are not well explained. \n  - Content variations seem somewhat arbitrary. If one were to include misleading descriptions and adversarial perturbations performance is going to drop. What is the point of this exercise? There is not much in terms of motivation for why these were done and what specific ways in which these were created. Why use German translations? Why not others? \n  - Similarly the choice of stylistic variations also appears somewhat arbitrary and adversarial. It would have been much more natural to target the most frequently used variances alongside some of the more rarely used variations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WBlG3cORVl", "reviewer": "Reviewer_CTPq", "summary": "This paper introduces OpenApps, a lightweight and scalable benchmark designed to evaluate the **reliability of multimodal UI agents under app-environment variations**. It exposes an overlooked weakness in current evaluation protocols—agents’ fragility to minor visual or structural changes in otherwise identical tasks. The framework is technically sound, reproducible, and empirically comprehensive across seven models. Results show dramatic reliability drops (up to 50% across app versions) and reveal new failure behaviors such as looping and hallucination. The contribution is both **timely and impactful**, offering a reproducible infrastructure that can underpin future robustness research.\n\nWhile methodologically solid, the work would benefit from a **clearer theoretical framing of “reliability,” richer task diversity, and improved presentation**. The current tasks are simple and limited to single-app workflows; including long-horizon tasks would strengthen generality. Despite these limitations, the paper’s novelty, execution quality, and potential community value are strong.", "questions": "+ How are app variations generated — random parameter changes, manually curated modifications, or rule-based templates? Could the process introduce artificial correlations that models might exploit?\n+ How do you ensure that a task instance in version A is semantically equivalent to the same task in version B (e.g., identical goal, only UI difference)?\n+ Did you observe any model types that adapt better to variations (e.g., multimodal LLMs vs. RLVR UI-trained agents)? If so, what might explain this?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. The paper identifies a key blind spot in existing agent benchmarks: reliability across app variations. Unlike prior environments that focus on fixed app clones, OpenApps systematically quantifies how changes in design and content affect UI-agent performance. This is a genuine conceptual contribution to the field of multimodal agent evaluation.\n2. OpenApps is implemented in pure Python and runs on a single CPU, removing the heavy dependencies of prior environments (e.g., emulators, Docker containers, or large memory requirements). This design choice makes it widely accessible and reproducible.\n3. The authors conduct 10,000+ trials across seven state-of-the-art agents, including GPT-4o, Claude Sonnet, Kimi-VL, Qwen-VL, and UI-TARS. The scale and comprehensiveness of these experiments convincingly demonstrate the practical significance of app variation as a factor in agent reliability.\n4. Results are striking: task success can fluctuate by more than 50% across app versions, and specific models show massive degradation (e.g., Kimi-VL drops from 63% to 4%). The paper also documents behavioral shifts such as looping and hallucination, revealing that environmental variability induces new failure modes.\n5. OpenApps can be extended for safe training, adversarial robustness testing, or sim2real transfer studies. The discussion section articulates a credible roadmap for future research directions, showing awareness of the broader ecosystem.", "weaknesses": "1. **Task Simplicity and Limited Scope**\n    - The current experiments focus on **15 simple tasks** (e.g., adding a to-do item). While the paper demonstrates large variability even on these, such simplicity limits conclusions about generalization to *complex or long-horizon* tasks seen in real-world apps.\n2. **Insufficient Theoretical Framing of “Reliability”**\n    - While empirical results are strong, the paper lacks a deeper theoretical formalization of *reliability across app variations*—for example, framing it as an expected reward stability problem under distributional shifts could strengthen the conceptual rigor.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PdLXnPAWQo", "reviewer": "Reviewer_1P4o", "summary": "This paper introduces OpenApps, a novel, lightweight, open-source ecosystem designed to measure a new dimension of reliability for autonomous UI-agents: performance fluctuations across app variations. Current evaluations rely on fixed environments, failing to capture how changes in app design, appearance, or content affect agent success. OPENAPPS provides six common apps (calendar, maps, messenger, etc.) that are highly configurable via simple Python/YAML files, enabling large-scale, reproducible experiments (over 10,000 trials conducted). The study finds that while reliability within a fixed app is stable, reliability across app variations fluctuates drastically (e.g., Kimi-VL-3B's success rate varied from 63% to 4%), and that failure modes like looping and hallucination are highly environment-dependent. This highlights app variation as a critical, overlooked axis of agent reliability.", "questions": "1. The flexibility of OpenApps allows generating training data across thousands of versions. Did the authors explore whether fine-tuning a model (e.g., UI-TARS) on a synthetic dataset generated by OpenApps could significantly improve its overall deviation (Figure 5) compared to training only on a single fixed version?\n\n2. The paper notes GPT-4o performs highly when given simplified AX tree representations along with the screenshot. Which specific content variations (e.g., German language, adversarial text) affected the reliability of AX tree parsing more than the visual recognition, or was the failure primarily a model-level reasoning issue?\n\n3. The reward function is deterministic. For complex, open-ended tasks, a continuous partial reward is often more useful. Could the authors confirm that the Python logic of OpenApps could support easily pluggable, continuous reward functions (e.g., L2 distance between the current state vector and the target state vector) for future RL training efforts?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper correctly identifies the critical gap between testing on fixed clones and real-world deployment, where app style, content density, and language constantly change. This significantly advances the utility of reliability metrics.\n\n2. OpenApps is designed for parallel, large-scale study (single CPU, < 10MB memory, pure Python). This low overhead is a crucial enabler for large-scale RL training and comprehensive testing that previous VM-based or complex web-clone environments could not afford.\n\n3. Beyond average success rates, the study provides valuable diagnostics on how environment changes induce specific failure modes.", "weaknesses": "1. The paper focuses exclusively on 15 simple, short-horizon tasks (e.g., adding a to-do item). While justified to isolate reliability, the performance of agents on complex, multi-app workflows (which is the ultimate promise of UI-agents) remains unevaluated. Future work needs to extend the task set to long-horizon, multi-step tasks to form a comprehensive reliability benchmark.\n2. The study primarily varies each app factor (appearance or content) independently. The authors acknowledge that interactions between multiple variations (e.g., dark theme + German language + dense content) could expose novel failure modes. A brief initial exploration of combined variations would strengthen the argument.\n3. The paper provides excellent diagnostics but offers limited concrete proposals for agent development (e.g., does VLM fine-tuning on synthetic variants solve the problem? Should contrast-boosting pre-processing be used for dark themes?). While the goal is diagnosis, suggesting a simple architecture or training technique that could leverage the synthetic data would enhance the paper's prescriptive value.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RwzBirdG1b", "reviewer": "Reviewer_5Ghj", "summary": "This work presents a light-weight version of a benchmark for GUI agents, which presents high reproducibility and easy/cost-effective evaluations. Especially, the authors focused on evaluating the reliability of the agents across UI variants. Across six applications that are popularly employed in many routines, authors configure various combinations of UI diversities, constituting more than 10,000 independent cases. Based on the cases, the vulnerability of agents is revealed, and their qualitative analysis is presented.", "questions": "For brevity, I included questions and suggestions in the section above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 4, "strengths": "On top of all, I believe that this work is highly notable because a light-weight benchmark for GUI agents is a missing yet demanding block in this community. I describe several other strengths of this work.\n1. Important problem: the reliability of agents in practice is a highly crucial problem. I believe that this work represents an important step forward for the community.\n2. Robust success detection logic: the authors put considerable effort into creating success detectors that can work around diverse UI variations. I believe that the efforts that the authors put into this part should be recognized, as this is known to be a challenging problem.\n3. Extensive experiments: the volume of the experiments is notable. I believe that such a massive numbers of experiments make the results presented more reliable.\n4. Interesting observations: the authors also present a detailed behavioral analysis. These observations highlight the shortcomings of current agents, enabling the design of future solutions.", "weaknesses": "I discuss the weaknesses of this paper, including questions and suggestions.\n1. Mobile device control benchmarks: while the paper presents fruitful discussion on web benchmarks, discussion on mobile device control benchmarks [1,2,3,4] (which also overlaps with OS-level benchmarks) is absent. Notably, B-MoCA [2] and AndroidWorld [3] both tackle the robustness of the agents, where the former one demonstrates the feature of UI variations and degradation of agents’ performance with respect to the variations, similar to this work. Yet, I do think that this work presents unique and distinct features from prior works (e.g., focus on easy reproduction), which I hope to read in the revision. \n2. HTML diversity: while the use of FastHTML is desirable, I question if it would have negative effects in terms of HTML diversity. To elaborate, I think evaluating the agents with diverse formats can be more appealing if the agents take text input.\n3. Intentionally misleading description: I worry this is out of scope in this work. There are many works tackling the robustness of the agents in an adversarial manner [5]. However, from my understanding, the UI variations are a ‘natural’ perturbation challenging the agent's robustness rather than ‘intentional’ (line 188-189). Such features should be handled differently, in my opinion.\n4. Simplicity of tasks: the proposed test suite suffers from both (1) a lack of diversity and (2) a lack of challenges. There is not enough headroom for improvements in this benchmark for the state-of-the-art agent (i.e., GPT-4o), which I assume would be a bigger problem with the recent agent (e.g., GPT-5). I suggest diversifying the tasks in terms of both volumes (i.e., more than 15 task templates) and difficulties.\n\nI do believe that this work has a strong potential to be a highly noteworthy work that can function as a standardized benchmark.\n\n--- \n\nReferences:\n\n[1] Rawles et al., “Android in the Wild: A Large-Scale Dataset for Android Device Control” (2023).\n\n[2] Lee et al., “Benchmarking Mobile Device Control Agents across Diverse Configurations” (2024).\n\n[3] Rawles et al., “AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents” (2024).\n\n[4] Zhang et al., “LlamaTouch: A Faithful and Scalable Testbed for Mobile UI Task Automation” (2024).\n\n[5] Wu et al., “Dissecting Adversarial Robustness of Multimodal LM Agents” (2025).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)