Experimental Prototyping (Prototyping User Experience 3/3)
By: Carey Hill Smith
Tags: Human Centered Design, Design Thinking, Experimental Design, UX, Interaction Design

This article focuses on how to construct focused design experiments

The Prototyping User Experience articles are an overview of selected lectures on the topics of prototyping, interaction design, and UX. The concepts here borrow from my time at Stanford’s d.school, and were developed further while teaching at California College of the Arts, and Art Center College of Art & Design. Each article looks at different core elements of understanding and prototyping experiences. These articles do not focus on software tools, but rather look at how the experience itself can be understood, pushed, pulled, and prototyped. After all, every good human-machine interaction is built on top of a solid human-human interaction.

At the bottom of each article are exercises that can be used as a reference by students, by educators/designers planning a lesson or workshop, or really just anyone interested in these topics :)

The other articles: Design Interventions, and Use, Usability, and Meaning

What is Experimental Prototyping

Experimental prototyping sounds fancy. In reality it means constructing your designs and prototypes as an experiment, with hypotheses that can be disproved or validated. I like the tension between the common conception of experimental design, and the practical application of design experiments. On the one hand, we are engaging in an experiment. This experiment strives for consistent methods and variables, and a testable hypothesis. On the other hand, we are working in an ambiguous, nonscientific field of design. Our designs are not held to the same rigorous review as a scientific experiment**. What we can learn from these experiments gives us increasingly deeper knowledge about the experiences and products we design.

Why Experimental Prototyping is Important

“If you already know the outcome of a prototype, it’s not a prototype, it’s a demonstration.”

This quote strikes to the core of why prototype experiments are important. When designing something new, with lots of unknowns, we build prototypes to answer questions. In experience design, those questions often revolve around what a person will do in an experience, how the experience will unfold, and how they will feel during that experience (See Use, Usability and Meaning). Prototype experiments are a way of ensuring that the questions we want answered are clear, testable with our prototype, and that we test the prototype consistently with the different people who use the prototype. It is totally acceptable for the outcomes of your experiment to generate more questions than answers. In fact, that’s what you want at early stages of the design process, especially when working with design interventions.

How to Construct a Prototype Experiment

Constructing these experiments requires a large amount of critical thinking, which builds on a deep understanding about the context of your design project. Prototype experiments have key overlaps between design research and prototyping. Understanding the different prototyping methods available to you will help greatly in the process of building an effective experiment.

Running an effective experiment takes repeated practice and collaboration. When working on a professional design team, this process often works best when a designer (within the field of focus for the project) collaborates closely with a design researcher. When learning these methods in an academic setting, consider your first round of the experiment to be a pilot test, and after testing it a couple times, refine and improve from there. When working professionally, pilots are always performed before the real test.

Design Experiments > Prototypes

Prototypes are part of design experiments, but a design experiment has many more pieces than just a prototype. The core building blocks of a design experiment are:

The Experiment Protocol:

Your protocol is a document used in preparation and execution of your experiments. It should contain the following sections.

An Experiment Overview: A brief paragraph that explains the parameters of your experiment and the goals of the tests. Your general project and design goals should go here.

A brief paragraph that explains the parameters of your experiment and the goals of the tests. Your general project and design goals should go here. Prototype Description: As the title suggests, you need a clear description of your prototype. If it’s software, describe the flow of the software and what can be done with the prototype (same if it’s a paper prototype). If it’s an intervention (see design interventions), describe the intervention.

As the title suggests, you need a clear description of your prototype. If it’s software, describe the flow of the software and what can be done with the prototype (same if it’s a paper prototype). If it’s an intervention (see design interventions), describe the intervention. Method & Test Guide: This should detail how you plan to test your prototype. The gold standard should be that you can hand this written method to someone who is not working on your project, and they can execute the experiments and tests without further consultation (that’s not always how it plays out :). The guide should detail the main sections of the test. It covers introductions, activities, questions you’ll ask your participant, and how long each section should roughly take.

This should detail how you plan to test your prototype. The gold standard should be that you can hand this written method to someone who is not working on your project, and they can execute the experiments and tests without further consultation (that’s not always how it plays out :). The guide should detail the main sections of the test. It covers introductions, activities, questions you’ll ask your participant, and how long each section should roughly take. Hypothesis: What is the single question you’re trying to answer with the prototype? Is it about how someone might use your prototype, a certain way they might accomplish a goal, or are you trying to spur a meaningful reaction from them? One important component of a good hypothesis is that it needs to be extremely specific. Many design hypotheses fail because they are too broad.

Experiment Testing:

Testing is often seen as the meat of the experiment, and it is where the results of your experiment are generated. But often the tests are the smallest chunk of work. My former colleague Mathis often referred to the hourglass description of experiments, the bulk of work happens during preparation and synthesis, with testing being the lowest volume of work.

For qualitative in-person testing, it works best if you conduct three to four 1-hour tests over each day of consecutive testing. This schedule allows for setup and debrief, and doesn’t overload the team. The whole team executes the tests, observes and documents their observations. When possible, tests should always be video and audio recorded.

You can use this format for taking notes on sticky notes while observing a test. Tip: use different colored stickies for each category.

Nice-to-have elements of testing:

A naturalistic setting (if it’s about in-car digital systems, hold the test inside a car. If it’s about testing an interactive restaurant kiosk, test in a restaurant.

Stream the test to a remote observation room. Use a go-pro/skype setup to allow your team to watch.

Take notes on Post-its:

Pains: Things with which your participants struggled.

Gains : Parts of the test they excelled at and enjoyed.

Quotes : What they said.

Wow: Things that surprised you, or the participants.

Experiment Synthesis & Analysis:

Synthesis & Design analysis are core skills of design research. I won’t go into too much depth on how to perform effective analysis, but at the bottom of the article are some great resources.

Exercises:

Activity 1 — Build an Experiment Protocol:

You’ll do this activity before you conduct experiments with your prototype. Use is as a way of planning out your experiments, and the team/group working from the same document. The experiment protocol is essentially the description of what’s supposed to happen when you test your prototypes. Remember from the reading above that protocol should contain:

Experiment Overview

Prototype Description

Method and Guide

Hypothesis

Learning Objectives:

Becoming comfortable with planning design experiments and being critical and focused about what you’re testing with your prototypes; learning how to work as a team with a shared goal & plan for a design.

Activity 2— Synthesis Session:

Use these guidelines to run an effective working session where you analyze your experiment. Synthesis works best in teams of three to six people.

Analysis and synthesis of your prototypes and experiments should be deep and structured. There is a sweet spot between too casual (everyone sitting around discussing what happened) and too structured (coding every transcript of every test for keywords and running machine analysis on it).

Have everyone review a transcript of the test, or review documentation of the test before synthesis. Use Post-its to capture key chunks of data (Pains, Gains, Quotes, Wow).

— Hopefully your team took notes during the testing Structure your analysis test-by-test on a white board or wall.

— Do separate analysis for each test/participant. Summarize your findings from the individual analysis of each test.

— These categories help to organize the findings: Insights, Open Questions, Modifications for future Experiments, Design Implications

Notes & Attributions: