Creating General User Models from Computer Use
Authors: name: Omar Shaikh; name: Shardul Sapkota; name: Shan Rizvi; name: Eric Horvitz; name: Joon Sung Park; name: Diyi Yang; name: Michael S. Bernstein
Published: 2025-05-16 04:00:31

Abstract: Human-computer interaction has long imagined technology that understands
us-from our preferences and habits, to the timing and purpose of our everyday
actions. Yet current user models remain fragmented, narrowly tailored to
specific apps, and incapable of the flexible reasoning required to fulfill
these visions. This paper presents an architecture for a general user model
(GUM) that learns about you by observing any interaction you have with your
computer. The GUM takes as input any unstructured observation of a user (e.g.,
device screenshots) and constructs confidence-weighted propositions that
capture that user knowledge and preferences. GUMs can infer that a user is
preparing for a wedding they're attending from messages with a friend. Or
recognize that a user is struggling with a collaborator's feedback on a draft
by observing multiple stalled edits and a switch to reading related work. GUMs
introduce an architecture that infers new propositions about a user from
multimodal observations, retrieves related propositions for context, and
continuously revises existing propositions. To illustrate the breadth of
applications that GUMs enable, we demonstrate how they augment chat-based
assistants with context, manage OS notifications to selectively surface
important information, and enable interactive agents that adapt to preferences
across apps. We also instantiate proactive assistants (GUMBOs) that discover
and execute useful suggestions on a user's behalf using their GUM. In our
evaluations, we find that GUMs make calibrated and accurate inferences about
users, and that assistants built on GUMs proactively identify and perform
actions that users wouldn't think to request explicitly. Altogether, GUMs
introduce methods that leverage multimodal models to understand unstructured
context, enabling long-standing visions of HCI and entirely new interactive
systems that anticipate user needs.