Our approach is to use the multimodal object as a unified design paradigm. Another direction for our research efforts is to investigate the use of many types of sounds in the interface and their integration. Music can be used purely aesthetically, only to create sounds we enjoy, without being tied to real-world events. Speech is essential for human communication. Music comes to bear in helping to realize the meaning of the film, in stimulating and guiding the emotional response to the visuals.
Music manipulates our feelings and moods and our internal rhythms.
A third direction for our investigations is to examine how sound can be used to reveal visually hidden objects and interactions. Sound is an important channel of information that is unlike the visual channel. During the daylight hours, with few exceptions, objects in our field of view are seen, so our experience of an object is usually based on what it looks like. An object may not be seen, yet its presence is often discovered through its interaction with another object, such as footsteps, coughing, doorbells, etc. Below we describe three research efforts where we study the problems posed above.