Harnessing Webpage UIs for Text-Rich Visual Understanding
Authors: name: Junpeng Liu; name: Tianyue Ou; name: Yifan Song; name: Yuxiao Qu; name: Wai Lam; name: Chenyan Xiong; name: Wenhu Chen; name: Graham Neubig; name: Xiang Yue
Published: 2024-10-17 17:48:54

Abstract: Text-rich visual understanding-the ability to process environments where
dense textual content is integrated with visuals-is crucial for multimodal
large language models (MLLMs) to interact effectively with structured
environments. To enhance this capability, we propose synthesizing general
multimodal instructions from webpage UIs using text-based large language models
(LLMs). Despite lacking direct visual input, text-based LLMs are able to
process structured text representations from webpage accessibility trees. These
instructions are then paired with UI screenshots to train multimodal models. We
introduce MultiUI, a dataset containing 7.3 million samples from 1 million
websites, covering diverse multimodal tasks and UI layouts. Models trained on
MultiUI not only excel in web UI tasks-achieving up to a 48\% improvement on
VisualWebBench and a 19.1\% boost in action accuracy on a web agent dataset
Mind2Web-but also generalize surprisingly well to non-web UI tasks and even to
non-UI domains, such as document understanding, OCR, and chart interpretation.
These results highlight the broad applicability of web UI data for advancing
text-rich visual understanding across various scenarios.