Quantifying Iterations | Case Study
By: Naomi Hochberg
Tags: UX Research, Usability, Case Study, UX Design, Usability Testing

Think about the last redesign you were a part of. How could you tell if one iteration was definitively better than the one before? Were your users more satisfied if you moved a button 16 pixels to the right and changed button text from “edit” to “move?” How did you find the balance between usability and satisfaction? Did that balance make the experience more delightful for the user?

Designing for delight is about establishing an emotional connection between the user and our product. We are used to qualitatively interviewing users to evaluate usability and infer delight. But, can we quantify human emotions and design reactions to know for sure?

Moreover, can qualitative data be quantified? And does that data tell you enough?

Usability & Satisfaction

I was involved in a redesign to introduce an organizational hierarchy in Pluralsight’s Admin tools so that users could see analytics that would match the organization of their company. Instead of seeing data on individual learners, or small teams of those learners, Admins could continue to see data for these small groups, but they could also see data on a larger scale to track usage and skill improvement. For this redesign, I wanted to quantify both usability and satisfaction. Would delight emerge from this balance?

After attending Front in 2019 in Salt Lake City, UT, I decided to adopt a scorecard approach, which was presented by Lauren Treasure from Chatbooks, as a way to track progress from one iteration to the next.

Instructions given by Lauren Treasure of Chatbooks.

I tested both tasks and components of the design on a scale from 0 to 2.

For usability, 0 was a total failure to perform the task or use the component correctly, and a 2 was a total pass. For satisfaction, a 0 represented a negative reaction and a 2 was a positive reaction. All scores were solely based on observations.

To avoid designer-bias, I always had at least one other person scoring each interview with me. Assigning scores led to a holistic discussion about user behavior, which led to a synthesis of the needs of the user participating. These scores and discussions helped drive the next iteration.

Scores of each user were tallied and then calculated to a percentage. My goal for both usability and satisfaction was to reach an 80% confidence to ship, a Pluralsight framework. With that confidence, I would be ready to hand over designs to engineers and feel confident that this redesign would benefit the account Admin.

Scorecard used for every interview. This is from Dana’s interview in Iteration One (below).

Overall score (%) = total points / potential points

Iteration one

For the first iteration, we interviewed 5 Admins. They consisted of two tech leaders, like software engineers, two Learning & Development (L&D) leaders, and one Pluralsight Customer Success Manager. These Admins represented accounts ranging from 15 up to 3,000 licenses.

Tim

First, we spoke with a software engineer, Tim, who managed 24 licenses. He understood and correctly anticipated affordance actions and behaviors. Some naming conventions made him hesitate slightly before correctly performing the task. Overall, Tim scored 86% in usability and 95% in satisfaction.

We felt great! Such high scores on the first iteration? Easiest redesign ever, right?

Not quite.

Going into these interviews we thought our users were tech leaders, like Tim. We assumed they were tech-savvy and familiar with common icons, like ellipses, to know where to find more actions.

What we learned was that only about half of our users were tech leaders like Tim. The other half were L&D leaders. These L&D leaders were big promoters of using Pluralsight, however, they were much more accustomed to a hand-held experience. They wanted to see everything on one page. They didn’t want much, if any, navigation. They weren’t really interested in new affordances to perform jobs to be done.

Dana

Dana was an L&D leader focused on IT learning development. Her initial reaction was hesitation because she was used to a different design and it was a hurdle for her to assimilate to a new environment and layout. There was a lot of hesitation before clicking anywhere. Dana scored 75% in usability and 68% in satisfaction.

This is how most of the interviews went in the first iteration — alot of hesitation. We saw large gaps between the usability and satisfaction scores, where users were eventually able to figure out how to use the product and navigate through it, but they weren’t very happy about it. This redesign got the job done, but not how userswanted it to.

Overall, the first iteration scored 79% in usability and 75% in satisfaction.

Not quite that 80% confidence.

Iteration two

What worked and what didn’t work in the first iteration?

I averaged the scores for each task and component across the first five users to see what passed and what didn’t.

We had two total fails: a visualization tool to see your organization’s hierarchy in a tree structure and the pin icon in the tree to indicate what level you’re on.

But other tasks and components failed too. So I needed to determine which were the most critical for the user. I created a table determining task criticality, its impact on the user, the frequency we were seeing it across all interviews, which calculated the severity (criticality x impact x frequency).