SIGCHI Lifetime Research Award Talk: Thinking with Computers

We think with things, with our bodies, with marks on paper, and with other people. Thinking is a distributed, situated, and social activity that exploits the extraordinary facilities of language, representational media, and embodied interaction with the world. Today we increasingly think with computers. This includes desktop and laptop computers but more and more involves massive computer networks that deliver search results and provide access to a vast array of cloud-based services. At the same time, the monolithic computer of the recent past is coming apart and being reassembled in myriad new forms, including cell phones, tablets, digital pens, cameras that allow focusing after a picture is taken, glasses that can take a picture of what we see and tell us about it, sensors that capture our activity and support interaction by movement of hands and bodies as well as voice and eyes, and even cars that drive themselves. We are on the threshold of an era that is being described as the Internet of Things or, perhaps more accurately, as the Internet of Everything. The historical moment when people primarily worked in front of a single computer has passed. Computers are ubiquitous. They are being embedded in virtually every new device and the Internet and wireless communication enable their connection to each other and to ever-expanding information resources with previously unimaginable computational power. This is being accelerated by a radically changing cost structure in which the price is the same or less to use a thousand computers for a minute or a day than to use one computer for a thousand minutes or days. These and related developments are fundamentally changing the world in which we live and the ways we think, communicate, and interact within it. Yet for all their capacity and speed, using computers too often remains difficult, awkward, and frustrating. Even after six decades of design evolution there is little of the naturalness, spontaneity, and contextual sensitivity that characterize other activities nor is there the conviviality and flexibility of working with tangible media. Most time using computers is spent within applications, each designed to support specific functions and allowing interaction with information in only predetermined ways that are rarely sensitive to overall activity context or history of use and far too often we can look, but not touch, annotate, or personalize the information involved. Access to information needed to conduct daily life is spread across multiple applications and resources, resulting in fragmentation of activities and increased complexity. The rapidly expanding world of mobile apps, which are often walled off from each other and from other information sources, and the increasing use of multiple devices further fragments the ways we work and interact. Although we commonly use paper and digital work materials together, they remain elements of two distinct worlds. Disconnects between digital and physical worlds, lack of naturalness and context sensitivity, inability to tailor and personalize information and our interaction with it, and fragmentation of real activities across multiple applications and devices are systemic problems and unfortunately becoming accepted features of modern life. In this talk I argue that we need to challenge the legacy document- and application-centric view of information. We must design environments that move beyond simple metaphor-based imitation of static information entities to ones in which information is dynamic, operating according to cognitively-inspired physics that exploit human abilities, are sensitive to context and history, link the digital and physical worlds, and support web-scale information activities.