Toward a Software Model and a Specification Language for Next-Generation User Interfaces

As user interfaces evolve from traditional WIMP to ‘reality based interfaces’, developers are faced with a set of new challenges which are not addressed by current software tools and models. This paper discusses these challenges and presents a software model and specification language which are designed to simplify the development of non-WIMP interfaces such as virtual environments. Finally, we discuss our plans to extend this model to support physical interaction and dynamic adaptation of interfaces. INTRODUCTION Current HCI research covers a diverse range of new interaction styles, including augmented reality, multi-modal interfaces, tangible user interfaces, physical user interfaces and lightweight interaction. Common to all of these is the change in interaction with computers, from a segregated and specialized activity to one more closely related to the real physical world. Thus, we identify these interaction styles as an emerging next generation of user interfaces and refer to them as reality-based interfaces. Challenges for User Interface Software Tools By leveraging experience, knowledge and skills that users already possess, reality-based interfaces offer the promise of interfaces that are easier to use. However, these interfaces are currently more difficult to build than traditional WIMP interfaces. To explore why current user interface software tools are not applicable to reality-based interfaces, consider some of the characteristics of current WIMP interfaces vs. next-generation interfaces as they are specified in table 1. Many of the characteristic of WIMP interfaces such as single thread, discrete and precise tokens as well as explicit commands correspond to the characteristics of noninteractive programming languages processed by compilers. Indeed, many of the current user interface tools are built around compiler technology. As next-generation interfaces violate these assumptions, they are not well served by compiler-based approaches. A recent practice in WIMP interfaces has been to codify available interaction techniques into a toolkit which contains a standard set of widgets. However, nextgeneration interfaces introduce a far richer range of interaction techniques and interactive behaviors and while the effort to codify existing interaction techniques has begun [2], the search for new interaction techniques and support technologies continues. Thus, software tools which are easily extended to support new interaction techniques and sensing technologies are needed. Current Next Generation Single-thread dialogue Concurrent, asynchronous dialogues Discrete and precise tokens Continuous and discrete inputs and responses, probabilistic input Sequence, not time, is meaningful Real-time requirement, deadline-based computations Explicit commands Intentional and passive interaction Consists of predefined interactive objects and behaviors. Introduces new types of objects with new types of interactive behavior. Single output channel Parallel digital and physical output channels Standard IO hardware Employ a variety of devices and sensing technologies Operate in a static and isolated computational environment Operate in dynamic, ad-hoc, heterogeneous environments Table 1. The characteristics of current vs. next-generation UI. To date, WIMP interfaces primarily operated in the homogeneous and computationally isolated desktop environment. Next-generation interfaces will operate in dynamic heterogeneous environments where a large variety of devices communicate and share resources such as storage and interaction capabilities. These interfaces may spread across multiple devices which are temporarily connected. As devices join and leave a computational environment, changing the available landscape of resources, user interfaces should be constantly adapted to employ available interaction devices. User interface software tools will thus need to provide ways to build interfaces given uncertainty about the devices they will employ and supply services such as dynamic adaptation of user interfaces. Though existing techniques could be extended in ad-hoc ways to address some of the aspects unique to nextgeneration UI’s, they would fall short of producing a comprehensible specification at a high level abstraction. The approach of User Interface Management System (UIMS) [3] seems like a solid foundation for the development of software tools for next-generation user interfaces. However, to date, few UIMS’s address the unique characteristics of next-generation interfaces.