Between the (book) covers: Design Beyond Devices
By: Cheryl Platz
Tags: User Experience, Technology, Voice Assistant, Artificial Intelligence, Design

Between the (book) covers: Design Beyond Devices

Four themes woven throughout my new book on multimodal and cross-device design

On December 1, 2020, my first book Design Beyond Devices will ship worldwide. As so many authors do, I wrote this book partially because I wish it had existed when I was tackling the big unknowns on projects like Alexa Notifications, the Echo Look, the Echo Show, and Windows Automotive.

But I was also inspired to write because as an industry, we still seem to be focusing on individual touchpoints rather than focusing on the customer’s journey between touchpoints. And I wrote the book because so many of our devices are now capable of interacting with us in so many ways.

These days, it should less about “designing for a smart speaker” or “designing for a desktop” — many of these devices share capabilities. The real skill is figuring out exactly how your customer wants to interact in the moment. Or better yet, how might you allow your customer to choose their mode of interaction IN the moment?

Design Beyond Devices by Cheryl Platz: Available from Rosenfeld Media and on Amazon in print and digital.

The elevator pitch

Over the past year, when folks outside the industry asked me what I was writing about I’d say:

“I’m writing a design book that I hope will help teach designers how to design the bridge of the Starship Enterprise.”

If you’ve ever watched Star Trek: The Next Generation, you’ve seen the vision of the future I’m chasing. Interactions that span tablets, touchscreens, physical controls, voice interactions — even augmented reality, all while allowing crew members to seamlessly move between modes of interaction as needed. But underneath all that simplicity must lie a framework of great complexity. Still waters run deep.

The full title of the book is “Design Beyond Devices: Creating Multimodal, Cross-Device Experiences.” I haven’t seen any confusion about the term ‘cross-device,’ but the term ‘multimodal’ is still unfamiliar to many designers. What the heck am I talking about, anyway?

Ironically, the term “multimodal” is itself multimodal. It’s been defined multiple ways in different contexts. In THIS context:

A mode is a type of communication, and humans receive communication using their senses. I go on to define 5 categories of communication in the book: Visual, auditory (audio), haptic (direct/physical), kinetic (motion-based), and ambient (indirect).

is a type of communication, and humans receive communication using their senses. I go on to define 5 categories of communication in the book: Visual, auditory (audio), haptic (direct/physical), kinetic (motion-based), and ambient (indirect). A multimodal interaction is an exchange between two parties (a device and a human being) where multiple input or output modalities may be used simultaneously or sequentially depending upon context and preference.

Why include both multimodality AND cross-device design?

It’s short-sighted to assume ANY experience exists in a vaccum. Our customers are swimming in devices. Even websites are cross-device now: most websites have to function on desktop and mobile, which means interruption, context, and notifications become relevant. And the limits of multimodality on one device may cause a customer to turn to another device.

The deeper concepts

But for better or worse, this is a design book released in the throes of a turbulent 2020. It’s not enough just to throw some design ideas at you and run away. It was extremely important to me that this book provide you with a sound foundation for evaluating your ideas through lenses of inclusion, bias, impact, and beyond. Just because we CAN build it doesn’t mean we SHOULD.

The following four themes are woven deeply into the book’s 15 chapters (chapter list available at the Rosenfeld Media site):

Customer context & ethics Multimodal frameworks Ideation and Execution Emerging technology

Let’s dive more deeply into each of these themes so you know what we’re getting into.

Theme 1: Customer context and ethics

It was important to me that this book be more than just an exploration of what we CAN do — it’s an exploration of HOW we should move forward and what we owe our customers.

Creating the World We Want to Live In (Chapter 1) provides definitions and context that will frame the rest of our explorations: from definitions of multimodality and gender equity to deeper discussions on disability, inclusive design, anti-racism, and anti-neutrality.

Capturing Customer Context (Chapter 2) provides you with a functional, improv-inspired framework for deepening your customer outreach and research to include the extra context you need when designing multi-device, multimodal experiences.

Understanding Busy Humans (Chapter 3) helps you break down human behavior into patterns that can be represented within your system or platform as a part of teaching your devices to interrupt and interact politely and responsibly.

Should You Build It? (Chapter 15) concludes the book with a new framework for ethically querying your work: PICS or It Shouldn’t Happen. You’ll find a series of prompts to challenge your problem definition, your degree of inclusion, your theory of change, and the systems you’re influencing.

Theme 2: Multimodal Frameworks

Once you’ve fully explored your customer’s context and assessed the potential impact of your project on the environments and systems with which it will engage, it’s time to start exploring how multimodality might be applied to those situations.

Activity, Interrupted (Chapter 4) walks you through a process I first spearheaded for Alexa Notifications: creating an interruption matrix that takes your customer activity model and maps it against your potential interruptions, defining predictable interaction patterns for each combination.

The Spectrum of Multimodality (Chapter 7) helps you figure out what approach to multimodality will make most sense for your customer based on two key dimensions: their proximity to their device(s) and the amount of information being communicated.

Read an excerpt from Chapter 7 on the Rosenfeld Media Medium page!

Let’s Get Proactive (Chapter 10) What are the common patterns of interruption? What are the philosophies and pitfalls behind proactive interactions in interactive systems? This chapter draws from my experience on notification systems to help you chart your own path.

From Envisioning to Execution (Chapter 12) In the midst of the discussion of implementation, we explore an extension of Brad Frost’s Atomic Design to cope with the added complexity of multimodality — multimodal atomic design.

Theme 3: Ideation and execution

But how do you take these amorphous, big, intimidating ideas and make them real? How do you communicate these designs in a way that makes them understandable by your implementation partners?

It’s a (multimodal) trap! (Chapter 8): Seasoned veterans of multimodal and cross-device design will know when to ask difficult questions, but will you? This chapter covers some of the stickiest areas you’re likely to encounter — from ergonomics to multi-user scenarios.

Lost in Transition (Chapter 9): Whether you’re focusing on a single multimodal device or an expansive cross-device experience, transitions will make or break your experience. We inventory the most difficult types of transitions and what to look out for along the way.

Breathe Life into the Unknown (Chapter 11): From the Opti-Pessimism framework from exploring the best and worst cases of your idea to storyboarding, bodystorming, and a variety of prototyping philosophies — this chapter will kick-start your attempts to make your ideas real.

From Envisioning to Execution (Chapter 12): Learn a common visual language for multimodal flows, review tangible examples of multiple approaches to documenting time-bound multimodal designs, and explore how you might expand existing design systems to account for multimodality.

Theme 4: Emerging Technology

Many readers won’t yet feel fully comfortable with multimodal technologies, or perhaps may not even see themselves as multimodal designers at all. That’s OK! This book contains plenty of background to bring folks of all experience levels up to speed.

The Language of Devices (Chapter 5): In this chapter, we take a complete inventory of the available output modalities and technologies available to you, along with inclusivity considerations and case studies along the way.

Expressing Intent (Chapter 6): An exploration of the current state of input technologies, from natural language understanding through gestural interfaces — with plenty of inclusivity considerations and case studies to flesh out your understanding.

Beyond Devices: Human + AI Collaboration (Chapter 13): It is near-impossible to work with modern multimodal systems without dealing with AI. Learn the basics: types of machine learning, types of bias, and how to cope with designing with AI on your project.

Beyond Reality: XR, VR, MR, AR (Chapter 14): At the bleeding edge of multimodality, extended reality experiences are just pushing from niche to mainstream. Learn the difference between virtual reality, mixed reality, and artificial reality — and what makes design for XR different.

Learn More

If you’re intrigued by the concepts described here, the book is available via Rosenfeld Media or your local Amazon storefront. If you pick up the book, find me on Twitter at @funnygodmother or @ideaplatz to share your thoughts. And may 2021 see us to better outcomes — the final frontier is waiting for you!

Podcasts

Upcoming events (Talks & Workshops)