Read Anywhere Pointed: Layout-aware GUI Screen Reading with Tree-of-Lens
  Grounding
Authors: name: Yue Fan; name: Lei Ding; name: Ching-Chen Kuo; name: Shan Jiang; name: Yang Zhao; name: Xinze Guan; name: Jie Yang; name: Yi Zhang; name: Xin Eric Wang
Published: 2024-06-27 15:34:16

Abstract: Graphical User Interfaces (GUIs) are central to our interaction with digital
devices. Recently, growing efforts have been made to build models for various
GUI understanding tasks. However, these efforts largely overlook an important
GUI-referring task: screen reading based on user-indicated points, which we
name the Screen Point-and-Read (SPR) task. This task is predominantly handled
by rigid accessible screen reading tools, in great need of new models driven by
advancements in Multimodal Large Language Models (MLLMs). In this paper, we
propose a Tree-of-Lens (ToL) agent, utilizing a novel ToL grounding mechanism,
to address the SPR task. Based on the input point coordinate and the
corresponding GUI screenshot, our ToL agent constructs a Hierarchical Layout
Tree. Based on the tree, our ToL agent not only comprehends the content of the
indicated area but also articulates the layout and spatial relationships
between elements. Such layout information is crucial for accurately
interpreting information on the screen, distinguishing our ToL agent from other
screen reading tools. We also thoroughly evaluate the ToL agent against other
baselines on a newly proposed SPR benchmark, which includes GUIs from mobile,
web, and operating systems. Last but not least, we test the ToL agent on mobile
GUI navigation tasks, demonstrating its utility in identifying incorrect
actions along the path of agent execution trajectories. Code and data:
screen-point-and-read.github.io