Excerpt from a new book on multimodal interface design
By: Rosenfeld Media
Tags: Interface Design, Multimodal, Interfaces, Inclusive Design, UX

Chapter 7: The Spectrum of Multimodality

You’ve now explored the full spectrum of options (currently) available to you when breaking beyond the old-fashioned paradigm of a screen, mouse, and keyboard. But pulling those things together requires a broader perspective. A galactic perspective, perhaps?

In Star Trek: The Next Generation, the officers of the Starship Enterprise use technology to navigate through space, scan and catalog planets, communicate in short range and across solar systems, and even to entertain themselves. The crew may have ready access to a variety of touch panels and physical throttles, but they can also ask nearly any question in spoken language. The system can reply with language in kind, but when the answer isn’t well-suited for a spoken reply, they may move seamlessly back to a screen (or even the Holodeck). They might ask for a damage report and hear a spoken summary, while seeing a map of the damage. By tapping on a part of the ship, they can ask the computer to seal off that specific section; or they can ask the computer verbally to take action.

Starfleet Dreams

Yes, like many technologists I’m obsessed with Star Trek. I’ve immersed myself in that world for years as a member of an improvised parody of the original series called “Where No Man Has Gone Before.” The more I explore the world of Star Trek, the deeper my admiration becomes for its optimistic, (usually) inclusive futurism.

The bridge of the Starship Enterprise (Figure 7.1) is a shining example of what today’s designers often refer to as a multimodal interface. The Enterprise supports multiple input modalities, like touch and voice, and it supports multiple output modalities, like screens, VR, and speech. What’s most remarkable is how effortless it is for the crew to change the way they interact with the ship’s computer and its various modes of input at a moment’s notice.

Figure 7.1 The bridge of the Starship Enterprise in the TV series Star Trek: The Next Generation is the ultimate multimodal interface. Officers interact meaningfully with the system in different ways, depending upon their proximity to physical affordances. Star Trek and the Starship Enterprise are © ViacomCBS

Now that next-generation technologies like conversational user interfaces and computer vision have arrived, the most immediate challenge that today’s multimodal experiences face is the tension between visual interface elements and audio interface elements. No matter how much hype surrounds voice-user interfaces, voice is not necessarily best suited to all interactions.

Early voice-enabled interfaces like the Xbox Kinect relied on a “see it, say it” model where customers could speak the written name of an interface element to interact with it, as depicted in Figure 7.2. In contrast, early automotive systems treated voice and touch as separate ways of performing the same task with no synchronization or support for transitions. Mercifully, those experiences are fading away in favor of more integrated approaches to multimodality.

Figure 7.2 Some early voice-enabled multimodal interfaces like Microsoft’s Xbox Kinect relied heavily on direct reference of UI elements. Photo credit: Howcast.com

Years later, the experience of designing for Amazon’s voice-enabled countertop tablet device, the Echo Show, put us in uncharted territory. As we worked on one of the first devices to attempt a tight pairing between interactive visuals and natural language speech, we faced a number of new interaction design questions. Could we use the same responses as the Echo itself? What level of display was appropriate? How did an interaction with the Echo Show differ from the Fire TV? How would we help customers switch between physical interactions and spoken interactions?

As we waded more deeply into the product design process for the Echo Show, it became evident that the introduction of the screen to what had previously been a voice-only interaction was far from cosmetic. This particular combination of screen and natural language interface was yet another new paradigm. It seemed silly to read a full forecast when the screen could display it so much more efficiently. Some requests, like “show me stock prices,” might not even warrant a spoken response due to the context.

Coping with this added complexity required us to make intentional choices about the relationship between our primary input and output modalities. Over time, those choices became patterns, and my first multimodal interaction model was born. But much of my thinking has evolved since those early days when I, like the industry, was still quite device-focused.

Before you begin designing specific interactions for your experience, you must first make a conscious choice about what multimodal interaction models your product will support.

Dimensions of Multimodal Experiences

Early attempts to chart these new voice-enabled experiences placed a heavy focus on the voice aspect of the interaction. At Google I/O 2018, a “Multimodality Spectrum” with a single axis was presented: the support for voice interactions, from voice only to no voice.

A single-axis “multimodality spectrum” depicted in “Design Actions for the Google Assistant beyond smart speakers” from Google I/O 2018.

While voice is undeniably a critical element of many multimodal interactions, simple support of voice features isn’t necessarily the defining feature of multimodal interactions. Furthermore, the terms voice forward and screen forward are inherently device-centric.

In order to create a more robust interaction model that will stand the test of time, take a step back and consider the human impacts of multimodal systems. While you’ve explored many input and output modalities thus far, there are two dimensions that have the greatest impact upon your customer experience:

Proximity: The typical or average distance between your customer and the device(s) involved in the interaction.

The typical or average distance between your customer and the device(s) involved in the interaction. Information density: The amount of information presented to your customer in a typical interaction. In this case, information includes length and complexity of spoken prompts; any visual information like images, lights, or written text; and any tertiary information provided over channels like haptic feedback.

To understand what multimodal interaction model applies to your experience, begin by asking yourself these two questions about those critical dimensions of the experience:

Where will your customer be in relation to the device? (See Table 7.1.)

How much information is presented to your customer during a typical interaction? (See Table 7.2.)

Table 7.1

As discussed in Chapter 6, “Expressing Intent,” the nature of the input sensors you’ve selected will directly impact the assumptions you can make about the distance between your customer and the device they’re interacting with. Note that close proximity is determined by the location of the input sensor, which is not always on the primary device, as seen in streaming devices with microphone-enabled remote controls.

Table 7.2

High-density information is often provided on high-resolution screens. Multimodal rich output also typically includes robust voice and sound interactions. Low-information density interactions usually include very constrained visuals: either a small screen, an LCD readout, or an LED display of some sort. Figures 7.4 and 7.5 illustrate two ends of the information density spectrum.

Figure 7.4 Low information density: a Fitbit wristband.

Figure 7.5 High information density: Netflix on Amazon’s Fire TV

Dynamic Devices

Observant readers may have already noted that these dimensions aren’t necessarily fixed, even for a single device. A person can be near or far from their Amazon Echo. Some devices are more constrained — customers are unlikely to use streaming video devices out of eyeshot. But, in other cases, a single device might support multiple “stops” on both of these spectrums, depending upon customer context. While the examples here tend to be device-based, in reality you could say that an adaptive device is technically switching between intangible and anchored experiences. A single device could support multiple interaction types. The difference is that an adaptive experience has affordances that allow customers to choose how to interact in the moment.

Mapping the Multimodal Quadrants

Once we’ve answered these two questions, we can begin to understand where our product fits in what was once a dizzying spectrum of experiences. What have our customers learned to expect from experiences like this one, and where have those experiences fallen short?

By placing relative customer proximity on the X axis and information density on the Y axis, you can place most multimodal scenarios firmly into one of four quadrants, as shown in Figure 7.6. Note that there is no superior quadrant: each has its own strengths and weaknesses.

To apply this multimodal interaction model to your designs:

For your experience, answer the two questions specified earlier.

Where will your customer be in relation to the sensors you’re using during the scenario?

How much information is presented to your customer during a typical interaction?

Use the answers to these questions to place your experience in one of the four multimodal interaction quadrants.