A beginner’s guide to organizing a product usability testing
By: Hafidz Zulkifli
Tags: User Research, User Experience, Usability, UX, UX Research

For those that frequent my blog, you’d know that I mostly write about data science, machine learning, and sometimes — the occasional book review. This post isn’t going to be about that (unfortunately?).

This post will talk about how to organize a Usability Test, essentially defined as assessing whether our product’s target audience are capable of completing certain tasks within a prototype/mock environment that we’ve developed.

For context — I’m not a trained UX person, nor have I any experience in this subject matter. The opportunity to work on this came about after my recent job change, where the folks in my unit are somewhat putting on multiple hats as we’re setting up our new business unit (like any other startup). It’s a refreshing change of pace, from doing traditional data science work to meeting up with our target audience to validate our design hypotheses.

And did I mention that all these were done during a nationwide mandatory quarantine (due to Covid19)?

What this meant was physically meeting up with your users weren’t an option at all, and severely limit the tools that we’re able to use.

Now with all of that out of the way, let’s first understand why people would want to do a usability test at all in the first place.

Why do Usability Testing?

A usability test is useful when we want to validate stuff like:

Will this feature be useful to our users? Will our users be able to …? Can our user navigate their way from the main page to making a purchase? Are they any gaps in our user journey that we haven’t thought of?

The diagram on the above describes the typical product design process. Having gone through design workshops and built the prototype, we want to conduct a usability test to ensure that any gaps or issues that relate to the design can immediately be identified and dealt with before actual development takes place.

It saves us from wasting time developing things that people can’t use (which also meant saving money).

After we’ve extracted and examined the feedback from our group of users, we can then proceed to build our product accordingly. Another round of usability test can later be run to validate whether our final product (which has all the bells and whistles complete with the inputs that we’ve gotten earlier) meets our customer’s expectations.

The benefit of developing a product using this approach is that you’re able to learn about the gaps a lot sooner (ie. prototype stage), save time (from developing the wrong thing) and reduce the risk of launching a dud to the market.

Each learning experience ensures that we’re able to pivot to what the user actually needs. credit: Google Ventures

Now that we’ve established the why — let’s get on with the how.

How To Do It

In explaining how to do this, let us try to work on a hypothetical prototype.

A hypothetical prototype

We’ve built an AI solution that aims to help college students save money — better. Now that we have a prototype, how do we know that it’s good for production?

1. Be clear on what you’re trying to learn.

Based on our problem statement earlier, we’ve built a prototype that aims to help students be better savers. The way that it works is by sending a push notification to the user every now and then based on data that is derived from user’s past spending pattern in an attempt to encourage good behaviours (like investing) and punish bad ones (like … ehem … buying overpriced boba teas 😜 .. cue pitch forks and torches ).

There are essentially 2 types of questions that you can dive into during the interview:

Discovery

Usability

Discovery questions revolve around understanding how the user typically behaves, the problems that they face (students keep running out of cash), and how do they go about solving them. We’re interested in learning about the tools or workaround that they adopt (a money box?), why certain things that seem obvious to us aren’t in their radar (saving money obviously is good for the future, but why aren’t they doing it?)

Usability on the other hand covers the users’ experience in using the app/solution. Some example of things that we might be interested in:

Is the account easy enough to setup?

Where are the typical pain points in configuring the app?

Do they how how to turn the notification on/off?

As the prototype itself is full of features and rather open-ended in its navigation (let’s just assume that our 10X full stack developer have actually put tons of cool features inside), there are simply too many things that you can test and ask the users to give an opinion on. Thus, it’s important that from the get go that you — as the person in charge of organizing this UX research — be aware of this and limit the number of tasks (ie. can user successfully complete their in-app behavioural test) or features that we want to test our users with.

For an hour worth of interview, it’s generally good to settle on just 3–5 main tasks.

2. Source candidates. Screen them first.

Once we have a focus on what we want to learn, we now need to source some candidates.

Size

In terms of numbers, research [3] shows that 5 users are already enough to provide you with the necessary results to answer your questions.

The diagram above shows that the number of usability problems that can be found diminishes as more users are added to the test. Essentially, you can already identify 80% of the problems with just 5 users.

With that in mind, what’s important in terms of maximizing our learning outcome isn’t really the size of our sample — but rather how much iterations of small tests that we can run (using just 5 users for each tests). As we introduce small fixes in each of our iterations, we gain more understanding of our user’s expectation, behaviour and needs (and heading closer to our goal).

Type of user

Next, we need to consider who can we invite.

When we built the app, we already have in mind a set of users that we intend to market it to (ie. students). Hence it doesn’t make sense for us to invite users who are employed, aged 40, married with 3 kids to participate in our test.

In short, we need to identify the types of people that we should both include, and exclude, from our test.

As a general rule — avoid family members and friends (more details in [5]).

Some of the criteria of people we want to invite:

Aged 13–23 (essentially young of age) A student Uses an Android or an Iphone (since we’re only developing for those 2 types of phone, and not some Blackberry or Windows phone) Have the means to get some money on a monthly basis (either allowance, working part-time, or even running a business)

With our user segment clearly defined, we can now craft a questionnaire that can be used to screen/filter out potential candidates. Some tips on writing the questions:

Don’t reveal what you’re trying to test for to the potential user. Doing this might influence their decision/answers (ie. changing answers to be eligible) should the incentives look too appealing (ie. users will get MYR 500 once the UX test is completed). Don’t write leading questions. The questionnaire should take a neutral tone, where there is no clear right or wrong answer. For example, to check whether a user owns an Android or Iphone, you could ask them to (a) Write down their phone model name; or (b) A multiple checkbox list of devices that they own and use like android phone, iphone, blackberry, windows phone, android tablet, ipad, kindle etc (such as below).

Questions shouldn’t be leading users to certain outcome.

At the same time, be wary of experts joining the tests. In the case of our app, if we assume that the underlying technique on how things will be communicated to our users are based on a series of “nudges” (refer to [8] to understand what a “nudge” is), we wouldn’t want someone with a PhD in economics to be part of the test as their feedbacks would most likely be overly critical and wouldn’t represent what our typical customer segment would look like.

Implementation

Google Form is usually good enough to be used. Other options include Microsoft Forms, Typeform, and Jotform.

Once you have the screening form (ie. Google Form) ready, the cheapest way to distribute them are via:

Friends and family: Are any of them connected to our target audience (ie. lecturers, teachers, educators, friends at school) Online community: School’s Facebook page, forum threads, meetups (Pre-covid19) Physically going to schools and universities and meeting up with the students directly.

Alternatively, you can also outsource this task to an agency, as they’d have better reach and/or partnerships with the right market segment. The plus side of engaging a vendor to do the sourcing is that it saves you the time and effort. It also ensures that you’re not biased when selecting your candidate.

On the other hand; agencies are also more expensive, probably takes longer to source and you have lesser control of the relationship (ie. you can’t engage with the user again for follow up interview without paying fees).

Doing it yourself can be hard the first time around, as you’ll be figuring out a lot of things like what questions to write, how should the invite be sent out, how do I use Google Form and stuff. But once it’s done, you already have a few templates of your own that you can reuse for future tests. Aside from being cheaper and taking a shorter time to do (we sourced ours within 4 days.. pretty much depends on how vast your network is) — what I find most valuable is to have that list of users that I can always go back to should I need to do a quick follow up or enlist for future studies.

Pro tip: You can always add a few more questions than needed, spread the questionnaires to a wider audience — and end up with a good market research survey result.

Scheduling

Once you’ve managed to secure around 7–10 suitable candidates based on your set of criteria, it’s time to reach out to them with interview meeting invitations containing information such as:

URL of online meeting room (or address of your office where you decide to hold the interview).

For online meeting, do inform them of any required peripherals that they might need such as a working microphone, audio speaker, video camera (to record their face) and a stable/fast internet connection. Certain video conferencing apps require some software installation and registration to be done first, so it’ll be good to remind them of this. Also, since we’re recording the session, a well-lit space will also help with the quality.

Time and date

A reminder that the session will require a signed NDA (or some sort of consent form). We will not be able to proceed with the session without them being able to sign this later on, so it’s good to preempt them of this. Alternatively, just include the form along with the interview meeting invitation and have them sign it before accepting the invitation.

Pro tip: Never abandon those that didn’t make the cut. Have some *ehem* decency *ehem* and send out a courtesy email to explain why they were not forwarded an invitation.

3. Get your candidates to sign an NDA.

It is important that they’re clear on what the interview entails in terms of:

Confidentiality. All personal information will be kept safe and only used internally.

Ownership of ideas. Users need to be clear that what is being discussed or ideas brought forward during the interview will be owned by you (this is to prevent any future legal disputes).

Informed consent. Users need to be aware what the research is about, what data is being collected, how will the data be used etc.

Permission to record. Interviews will mostly be recorded (ie. video) for future reference.

The “NDA” can be shared to selected candidates either prior to the interview or at the start of the interview. This can be done simply on Google Form and sending the link over to the users for confirmation.

With regards to the content of the form, always refer to your legal team for counsel.

4. Prepare a script for the interview.

Interview conversation arc — Michael Margolis [9]

The above diagram describes how you can structure your 1-hour long interview. The intention of having an arc/structure is to allow users to ease into the interview questions and tasks naturally and more comfortable.

Throughout the interview, it’s important to note that what we’re trying to do is to understand what is going on in the user’s mind as they do things. This might seem obvious to some, but part of recognizing this fact means that the interviewer needs to be able to separate any of his/her preconceived notion of how the world works.

Things that you think “should” work in a certain way might not make sense to them. What they think as “normal” might seem weird to you.

Being able to remove your view of the world and self-bias will go a long way towards helping you develop questions that stems out of pure curiosity and discovery mindset. Granted, you might at times want to compare how they do things against what you assume to be the “conventional” method — but do so in a non-judgmental way, stating the question from a neutral standpoint and open-ended manner.

Pro Tip: Don’t ask leading questions. For example, “Don’t you agree that AI-assisted notification can help you save better?” preempts the user that the feature is a better option, and we’re trying to get a confirmation to our worldview. Hence an obliging user would more often than not will simply agree to the statement. An open question such as “What are your thoughts on AI-assisted notification?” is a better choice, since we’re not suggesting any answer to the user.

Greet and intro (1–5 min)

At this stage, we want to build an initial rapport and give them some idea of what the whole study is all about.

Discovery (5–10 min)

Once our candidate is at ease, we can start to ask some general questions to learn more about their behaviour and experience. At this stage, we ask questions such as:

How are you solving this problem (ie. saving money) now? Are you using any apps that is currently solving that problem? Can you reenact exactly how that app works, how you use it, and how it solve/not solve your problem?

Tasks (30–40 min)

At this juncture, we’re asking the user to complete certain tasks using the prototype that we’ve built. This ties back to the learning objectives that we’ve defined earlier.

To recap, what we wanted to learn was:

Is the account easy enough to setup?

Where are the typical pain points in configuring the app?

Do they (the user) know how to turn the notification on/off?

The corresponding tasks then could be:

Task 1: You’ve now installed the app on your phone. Please show me how you would (a) Register your profile (b) Setup an investment account

Task 2: Configure the app such that it understands your spending behaviour

Task 3: From the main page, please show me how you would disable the notification alerts.

In each task, we want to be asking follow up questions, to better understand why users does certain things. For example, our user might take a long pause before clicking at a button. There can be many reasons why the user might be stalling. For example:

He’s not sure what the button does.

He was looking for some other button to complete the task.

He didn’t understand what needs to be done to complete the task.

He’s not familiar with English — thus couldn’t understand the instruction.

It’s important that the interviewer is able to pick up on these things, ask follow up questions, and act accordingly. Among some actions that you can take, given some scenario:

If user is asking for validation from you, try “What would you do if I wasn’t here sitting beside/not in this Zoom call with you?”

If user is silent and you’re not sure what he’s thinking, try “What’s on your mind?”

If user makes a comment, but you’re not sure how it was triggered, try “Was there something in particular that made you think that?”

If user seems to be wandering away from the task at hand, try “What are you trying to do now?”

Pro tip: Give participants some time to answer. A silent pause from interviewer can push participants to think harder (people get uncomfortable during an awkward silence). Allocate some time for this when planning the interview. Of course, don’t hesitate to guide them if they pause for too long.

Debrief and Cool Down (5–10 min)

Once our user have completed all the tasks at hand, it’s time to take a step back and review what we’ve learned so far with them and ask follow up questions on any doubts or gaps that we’ve uncovered throughout the entire process.

Typically, this is the time where you’d ask them stuff like:

What are your thoughts about the overall look and feel of the prototype? Was there, at any point in time, that you felt loss or didn’t know where to go? Which part of the app did you dislike the most? Why? The app is currently trying to remind users to save money by analyzing their daily spending pattern. Now that you’ve seen how it works within the prototype, do you think that it’ll help you save better? How would you do it differently if resources isn’t an issue?

Pro tip: Aside from verbally asking our users questions, you can also opt to ask them to answer a survey form, which would ensure that results can be attained in a more objective fashion. Scale ratings (ie. On a scale of 1–10, how would you rate your experience using the app) are common at this stage.

De-greet (1–2 min)

At this stage, we want take the time to thank the users for their time, as well as inform them how their gift/incentives will be distributed.

Pro tip: For an end to end example of an entire interview script, refer to [10].

5. Source your observers.

Most UX study that I’m aware of will have at least 1 person acting as an observer. Observers will typically join the remote session along with the interviewer and user, but will not be turning on their audio and camera (hence the term — observer).

The role of the observer will be mostly to:

Take notes during the interview. As the interviewer focuses on how the user behaves and guiding them along the UX testing process; it falls onto the observer to jot down any interesting quotes (ie. “this feels smooth like butter”), moments (ie. amazement, disgust), mistakes (ie. user didn’t click on icon as expected), paths taken or suggestions (ie. “this button feels out of place”) from the user. This would enable the interviewer to pay more attention to the user feedback and reactions.

Prevent bias in the final result compilation.

It is also encouraged to ask your stakeholders and team members to participate as observers. The advantage of this is three-fold.

Stakeholders can better empathize on user’s pain points by listening to their feedback directly. This can help trigger questions that can be relayed to the interviewer in future sessions.

Neutralize any strong opinions that any your team members might have since everyone is observing at the same time, so there’s no longer any dispute on how the user “should” be using the prototype (since the user already “is” using it and everybody saw how it’s being used).

As there’s a lot of angle where notes can be derived from (facial/body reaction, remarks, paths taken etc); having many observers provides us the ability of assigning each angle to be covered by a specific member. Intentionally assigning each observer with a specific task also ensures that more insights can be extracted (as opposed to everyone jotting the down the same quotes) and making them feel more engaged throughout the session.

Refer to the observer guidelines at [11] for more details.

6. (optional) Ensure that your prototype can be used by the user.

With the interview candidates ready, and the script prepared — if time permits, try to allocate some time to do a mock interview with an internal volunteer. This person can be your family member, a close friend or even your colleague.

The goal for this activity is to:

Iron out any gaps that you have in your interview script narrative.

Fix any glaring bugs that went unnoticed in your prototype app.

Test out the video conferencing tool, the screen sharing capability, and it’s recording feature.

To let the entire team be familiar with what they need to do on D-Day.

7. Do interviews. Jot down notes. Record them.

The highlight of the UX testing is of course, the interview itself. If you haven’t done so already by this stage, get the candidates to sign off on your NDA/consent form before proceeding with the interview.

As the organizer, be aware that there will always be chances that your scheduled users might not be able to attend the interview that you’ve carefully setup.

People forget, have other overlapping commitments — and some just over slept.

The onus is on you to have plans on how to mitigate the risk (ie. reschedule, backup candidates, 10-min courtesy call etc) and pray your best that your session runs as smooth as can be. 🤣

Once everyone is online and are in the meeting room, record the session until the end. Remember to remind the user of this and about the presence of observers (which should have their mics and cameras turned off).

With the prepared interview script, we can now proceed with the interview.

8. Compensate your users for their time. Get confirmation from them.

In terms of compensation, it’s really up to you to decide how much it is that you want to spend. With the limited experience that I have, the best I can say is — it depends.

It depends on how big your product is (ie small startup vs big pharma), where your company is based at (ie. Malaysia vs US), what the market rate is, who is it that you’ll be asking to spend an hour with (ie. an hour with investment bankers vs school kids) and many others.

Method of payment (that I’ve seen) can range from Amazon/Grab vouchers, PayPal, to direct bank transfer if need be.

Once payments have been delivered, the final thing that you’ll need to get from them are a confirmation on whether they’ve received it or not. You might want to check with your finance team on whether this is required or not for audit purposes (and what type of confirmation do they need).

9. Review the result with your team. Decide on the next action item.

With the interview over, it’s time to gather all the results (hand notes, recorded videos) and compile them so that you can review them together with your team. Working from home (due to social distancing) meant that digital whiteboards had to used to substitute for everyone being in the same room and thrashing it out.

Personally, I can vouch for Miro or Figma for this task (Trello or any other collaboration tool can also be used similarly), as it allowed all the team members to add in their ideas and work together in the same space in real-time. We use it in tandem with Microsoft Teams (essentially any video conferencing tool would work) so that we’re able to discuss about the board while we’re making the changes.

It’s quite fun, to be honest.

The above picture is an example of how you could summarize your findings on a physical board. You can list all of the users in each column, and the tasks at each row. Texts in red denotes the negative things that you noticed from the user in relation with the task, while green denotes the positive things. Black denotes something that is neutral in nature.

As we fill each boxes with our findings (maybe by going through the notes, reviewing the video or recalling from memory), similar themes will start to emerge. This is when we’ll know which issues are glaring enough that it’s been mentioned multiple times by different participants, and which ones are less important. At the end of this activity, you should already have a clear reading on whether your hypotheses are validated (both positive and negative) and have a sense of direction on what your next action item would be.

Basically there are few methods that you can choose from to fill up the above table:

Do it together, perhaps the next day after all the interviews have ended. This has the added benefit of keeping up the momentum and preventing any one person from discounting the notes and opinion that was noted by others. On the down side, it requires a heavy time commitment of all those who participated as interviewers and observers — and this might not be ideal for an organization with plenty of other things that needs to be attended to (which is common in big organizations). The organizer/interviewer summarizes all the findings by him/herself — then presents the findings over to the entire team a later date. This might seem more time efficient, but it opens up the possibility of bias being introduced and disagreements from team members who have a different interpretation of how the interview went. Something in between 1 and 2. Organizer/interviewer can organize all of the findings beforehand. At a later time, all the team members can mettup and discuss about the result, grouping similar items and prioritizing the task backlog.

Summary

In this post, we’ve covered the steps needed to organize a successful product UX testing.

Topics we touched on:

Understanding the need for UX testing. Be clear on the objective of the UX testing. How to source the right candidate. The importance of having a signed user agreement. Components of an interview script. Gathering your observers. (optional) Advantages of a mock interview Doing the interview. Rewarding your participants Reviewing the results

While the list above may seem like a lot of work — within a few iteration you’ll find that some parts of the steps can in fact be turned into templates, hence making it reusable for future works. Things like NDA forms, email invitations, follow up questions and such can always be recycled in some way or another. And of course, practice makes perfect. While your mind might be racing to prepare for that next question during the first interview, you’ll eventually feel more relaxed as you work on your third and subsequent interviews.

I’ve included the sources that I’ve relied on during my learning journey in the reference section for those who want to dive more into the topic.

All of the above wouldn’t have been possible without the endless input, feedback and script revisions provided my teammates. A UX test isn’t something that can be pulled of alone, as you’ll need to work closely with everyone (developer, stakeholder, observer etc) to ensure that everything moves together in a synchronized manner.

Credit also goes to my friend Eddy, currently doing product design and UX at Seek, for being a great bouncing board on how to dive into UX testing. Thanks.