Usability Testing: Moderated vs Unmoderated
By: Nick Babich
Tags: User Testing, Design Best Practices, UX, UX Design, Usability Testing

In this article, we will explore both types of testing and share practical recommendations on how to conduct each of them.

Moderated usability testing

Moderated usability testing is a usability testing technique that requires the active participation of a moderator (a real person who will facilitate the testing). The moderator will have solid experience in the field of usability testing and user research and they will work directly with the test participant, guiding them through the testing process.

Moderated testing can be done either in-person (in a lab or work environment, where a moderator can observe user interactions with a product), or remotely (via a video-conference call, during which a moderator asks test participants to share their screens).

Moderated usability testing in the lab. Image credit projekt202.

When to conduct moderated usability testing

Moderated testing is better suited for collecting qualitative feedback (times when it helps to find the answer to “Why?” questions, such as “Why did participants interact with a product in a certain way?” or “Why is a certain problem happening?”). For that very reason, it’s recommended to conduct moderated testing in the earlier stages of design — when a team is working on a product concept and wants to explore various solutions. Moderated usability testing helps the product team understand the mental model of a user and use this information to inform their product design.

Pros of moderated testing

Moderated testing gives you more control. This testing allows guidance and support. The facilitator can get a test participant back on track if they notice that the participant has misunderstood or is confused. Facilitators can also ask clarifying questions to better understand user behavior.

This testing allows guidance and support. The facilitator can get a test participant back on track if they notice that the participant has misunderstood or is confused. Facilitators can also ask clarifying questions to better understand user behavior. This form of testing results in participants that are more engaged. Moderators can have a natural conversation with test participants, which helps establish trust and makes test participants more motivated to complete tasks.

Cons of moderated testing

It is often expensive. Moderated testing requires you to hire relevant test participants, experienced moderators, and for you to find a place for the testing.

Moderated testing requires you to hire relevant test participants, experienced moderators, and for you to find a place for the testing. It also requires preparation. Moderated testing needs upfront planning — you’ll need source the necessary equipment and coordinate the time and date with your test participants.

Best practices for moderated testing

Define goals and plan the scope of testing

Knowing what parts of the product you want to test isn’t enough. You need to define key goals you want to achieve by testing your product with users and selecting tasks according to this goal. For example, if your goal is to improve the usability of a specific flow, it’s better to avoid vague goals (such as, “Our goal is to streamline the signup process.”). Be more specific on what you want to achieve (say, “Our goal is to reduce the number of dropouts during the sign up from 60% to 30%”). These specifics will help you to define the standards for your product and measure whether your existing product meets those standards.

Clearly define criteria for test participants

Test participants should represent your target audience. And for that very reason, you need to know what kind of people you want to hire. Make sure those you hire are in the right demographic and demonstrate the right behavioral traits you’re seeking. You can use the information from your user personas to define the criteria for test participants.

Follow the classic structure of moderated usability testing

Setting up a usability test involves carefully creating a scenario. The classic structure for a moderated usability test (a proven way of testing a product) includes the following steps:

Pre-session questions. Asking questions about the participant’s lifestyle, background, and/or general behavior. Pre-session questions will help identify typical habits and other relevant personal information about your target audience.

Asking questions about the participant’s lifestyle, background, and/or general behavior. Pre-session questions will help identify typical habits and other relevant personal information about your target audience. Actual tasks. In this step, the test participant performs a list of tasks using the product. All tasks should be selected according to the goal of the testing.

In this step, the test participant performs a list of tasks using the product. All tasks should be selected according to the goal of the testing. Post-session questions. This involves asking test participants general questions about the experience and collecting their thoughts about a product. Post-session questions allow test participants to provide their opinions and ask questions about the product, as well.

As soon as you create a structure with actual tasks, run a pilot test with one or two of the test participants to ensure that this structure works well for them.

Select the right fidelity for testing

As a tool, moderated usability testing can be run during the product design process. In this case, the testing is conducted using a prototype, not a real product. While it’s possible to use low-fidelity and high-fidelity prototypes for moderated testing, it’s always better to test with hi-fi prototypes because such testing will give you a more realistic idea of how users perceive your design. When test participants see a prototype that looks like a finished product, they will evaluate it accordingly. On the other hand, testing with low-fidelity prototypes is valuable in the early stages of product design when you want to collect opinions about your design from real users.

Ask test participants to think out loud during the testing

When it comes to completing actual tasks using a product, it’s recommended to ask test participants to talk through what they’re doing as they’re doing it. When test participants think out loud as they complete tasks, it helps moderators to understand the rationale behind every decision, which reveals their motivations/perceptions.

Record the interaction on video

Both a user’s reactions and their screens should be recorded during testing. The video can be used during the results analysis phase to draw a conclusion from the research. It’s recommended that you share the video with the entire product team to make them aware of any problems users face when they interact with the product.

Take notes during the testing

A lot of things might happen during moderated testing, and it’s relatively easy to forget something. Thus, it’s recommended that you write down any interesting observations and facts that you can later use during the test results analysis.

Analyze results and present them in a report

It’s not enough to run a moderated test. You need to invest time in analyzing the results of testing. The report you create after the testing should contain the most important insights and help the product team and stakeholders prioritize design decisions.

Here are some recommendations for creating a report:

Avoid UX jargon. Since you will show your report to various types of readers, you need to ensure that they won’t need to decode the meaning of any information provided in the report. Thus, it’s better to avoid industry-specific terms like “interaction cost” or “attention span” and replace them with clear synonyms.

Since you will show your report to various types of readers, you need to ensure that they won’t need to decode the meaning of any information provided in the report. Thus, it’s better to avoid industry-specific terms like “interaction cost” or “attention span” and replace them with clear synonyms. Use visuals to aid the information provided in a report. Humans are visual creatures, and it’s easier for us to comprehend information when it’s’ provided in visual format.

Humans are visual creatures, and it’s easier for us to comprehend information when it’s’ provided in visual format. Have a clear summary of the most important insights from the testing. This is especially important for long reports as people tend to skip over long documents. A quick summary will help readers get a sense of the information provided in a report at a glance.

Unmoderated usability testing

Unmoderated usability testing is completed by test participants in their own environment without a moderator present. This technique is usually used to test specific parts of a product (specific interaction scenarios), rather than providing an overall review of the user journey. Unmoderated testing is typically done using special tools for automated testing. The chart below visualizes the difference between testing tools in terms of the data they help to collect.