GPT-4V(ision) is a Generalist Web Agent, if Grounded
Authors: name: Boyuan Zheng, name: Boyu Gou, name: Jihyung Kil, name: Huan Sun, name: Yu Su

The recent development on large multimodal models (LMMs), especially
GPT-4V(ision) and Gemini, has been quickly expanding the capability boundaries
of multimodal models beyond traditional tasks like image captioning and visual
question answering. In this work, we explore the potential of LMMs like GPT-4V
as a generalist web agent that can follow natural language instructions to
complete tasks on any given website. We propose SEEACT, a generalist web agent
that harnesses the power of LMMs for integrated visual understanding and acting
on the web. We evaluate on the recent MIND2WEB benchmark. In addition to
standard offline evaluation on cached websites, we enable a new online
evaluation setting by developing a tool that allows running web agents on live
websites. We show that GPT-4V presents a great potential for web agents - it
can successfully complete 50% of the tasks on live websites if we manually
ground its textual plans into actions on the websites. This substantially
outperforms text-only LLMs like GPT-4 or smaller models (FLAN-T5 and BLIP-2)
specifically fine-tuned for web agents. However, grounding still remains a
major challenge. Existing LMM grounding strategies like set-of-mark prompting
turns out not effective for web agents, and the best grounding strategy we
develop in this paper leverages both the HTML text and visuals. Yet, there is
still a substantial gap with oracle grounding, leaving ample room for further
improvement.