A visual language for specifying and programming tangible user interfaces.

Augmenting the physical world by integrating digital information with everyday physical objects, tangible user interfaces offer a more natural, intuitive and accessible form of interaction than traditional user interfaces. They draw upon users' knowledge and skills of the everyday non-digital world, such as pre-existing spatial, motor and social skills. With a broad range of application-domains, including education, planning and design, problem solving, and entertainment, these interfaces show a potential to enhance the way people interact with and leverage digital information. However, tangible user interfaces are currently considered challenging to design and build. Designers and developers of these interfaces encounter several conceptual, methodological and technical difficulties. Among others, these challenges include: the lack of appropriate interaction abstractions, the shortcomings of current user interface software tools to address continuous and parallel interactions, as well as the excessive effort required to integrate various input and output technologies. This dissertation addresses these challenges by proposing a new paradigm for developing tangible user interfaces. Rather than using a specific toolkit, we propose to specify the structure and behavior of a tangible user interface using high-level constructs, which abstract away implementation details. Thus, developers would be able to use a comprehensive set of abstractions to specify, discuss and program tangible interactions. An important benefit of this approach is that these specifications can be automatically or semi automatically converted into different concrete TUI implementations by a Tangible User Interface Management System (TUIMS). In addition, such specifications could serve as a common ground for investigating both design and implementation concerns by TUI developers from different backgrounds. Thus, the primary contribution of this dissertation is a high-level user interface description language that provides developers, from different disciplinary backgrounds, means for effectively specifying, programming and refining a broad range of tangible user interfaces. There are four distinct elements to this contribution: (1) a set of abstractions for describing the structure and behavior of tangible user interfaces, (2) visual specification techniques that utilize the above set of abstractions, for effectively modeling tangible interaction while capturing its unique characteristics, (3) a declarative XML-compliant language, based on the above specification techniques, that provides a canonical format for specifying tangible user interfaces, (4) a high-level architecture and a set of proof-of-concept prototypes that demonstrate the feasibility of building a Tangible User Interface Management System that semi-automatically translates high-level specifications into a program controlling specific target technologies. This user interface description language (UIDL) was evaluated using three evaluation methods: cognitive dimensions for notations, specification of benchmark user interfaces and application in a classroom environment over several semesters. Our evaluation found that the proposed language is capable of describing a broad range of TUIs and can be easily used by developers from a variety of disciplinary backgrounds to specify, refine and discuss tangible interaction effectively.