OSUniverse: Benchmark for Multimodal GUI-navigation AI Agents
Authors: name: Mariya Davydova; name: Daniel Jeffries; name: Patrick Barker; name: Arturo Márquez Flores; name: Sinéad Ryan
Published: 2025-05-06 14:29:47

Abstract: In this paper, we introduce OSUniverse: a benchmark of complex, multimodal
desktop-oriented tasks for advanced GUI-navigation AI agents that focuses on
ease of use, extensibility, comprehensive coverage of test cases, and automated
validation. We divide the tasks in increasing levels of complexity, from basic
precision clicking to multistep, multiapplication tests requiring dexterity,
precision, and clear thinking from the agent. In version one of the benchmark,
presented here, we have calibrated the complexity of the benchmark test cases
to ensure that the SOTA (State of the Art) agents (at the time of publication)
do not achieve results higher than 50%, while the average white collar worker
can perform all these tasks with perfect accuracy. The benchmark can be scored
manually, but we also introduce an automated validation mechanism that has an
average error rate less than 2%. Therefore, this benchmark presents solid
ground for fully automated measuring of progress, capabilities and the
effectiveness of GUI-navigation AI agents over the short and medium-term
horizon. The source code of the benchmark is available at
https://github.com/agentsea/osuniverse.