Measuring design quality with heuristics
By: Jasmine Friedl
Tags: Usability, Product Design, Design, UX, Heuristics

How do we measure design quality? This question has been asked at every company where I’ve led design, and it comes not only from designers but from stakeholders and executives as well. Solutions range from “we’ll know it when we see it”, to principle-based evaluation, to detailed and rigorous product and design reviews. Other suggested solutions have included surveys like NPS (Net Promoter Score), CSAT (Customer Satisfaction Score), or custom user surveys like “How delightful do you find our product?”

Each of these solutions is problematic. Internally-led evaluations may be biased by internal motivations or blindspots. Surveys may not be able to isolate design quality, or they might add an unnecessary burden to the user.

A few years back, I was leading a design and research team at a company that was newly obsessed with John Doerr’s philosophy and application of OKRs. We found ourselves measuring everything, including design quality.

The evaluation mechanism we landed on was an expanded set of product and usability heuristics based on Jakob Nielsen’s 10 Usability Heuristics for User Interface Design. We came to the conclusion that while there may be bias in a design team evaluating their own work, it’s a no-brainer that designers should know what “good” is, as critique is an essential part of our practice.

I worked with a few teammates to develop and iterate on a set of heuristics mapped to our general breakdown of product design: product thinking, UX design, and UI design. An additive benefit was that we looked at designer qualifications and skills in similar groupings, so as we were evaluating quality, we could easily have input to the question “where has the designer exhibited strengths or areas of opportunity in this project?”

To evaluate, a small group of us sat down after every product launch and did a lightweight review of the work that was shipped. We answered a simple yes or no to each question and implemented a formula to give a quality percentage score (100 for yes, 0 for no).

While of course, this wasn’t a a perfectly precise conclusion, we logged resonant variants from project to project and found it useful to have the language to give feedback to designers as well as share a state of quality with non-designers.

Here are the heuristics.

Product thinking

User problem

Is the user problem to be solved clear?

Does the research at hand support the proposed direction?

Does the data at hand support the proposed direction?

Does the product or feature solve the user problem stated in the product brief?

Audience

Is the product or feature solving the problem for the target audience?

Does the product or feature avoid negative tradeoffs for the target audience?

Does the product or feature have minimal negative impact on a global audience/other audiences?

Value

Does the product or feature bring clear added value to the user’s experience?

Does the product or feature avoid removing value in other areas?

Design principles

Does the product or feature guide without overwhelming?

Has the product or feature been pressure tested?

Does the product or feature champion quality?

UX Design

System status

Is it clearly evident where the users are in the experience?

Is it clear to users what has happened previously and what will happen next?

Clear task at hand

Can users easily find what they are expected to do?

Do the users know how they are expected to accomplish the task(s)?

Straightforward and ease of use

Does the task(s) at hand require minimal effort to learn?

Does the task(s) at hand follow a logical order?

Does the task(s) at hand avoid unnecessary friction?

User expectations

Does the task(s) at hand follow the user’s mental models?

Does the product or feature follow common interaction patterns?

Is the product or feature flexible or customizable to meet the needs of different user groups?

Is the product or feature cohesive with other relevant product or feature systems?

Error handling

Do users have a clear path when something goes wrong?

Is error messaging clear, descriptive, and concise?

Have edge cases been considered?

Do users have access to help?

UI Design

Consistent standards

Is the design of the feature or product accessible?

Does the design of the feature or product use the give design system?

Does the UX writing in the design follow content strategy standards (appropriate voice, tone, style)?

Recognition over recall

Does the design of the feature or product make objects, actions, and options visible?

Are instructions or information easily retrievable?

Familiarity

Does page layout, hierarchy, etc or the feature or product feel familiar to other highly used features or products?

Does the design of the feature or product follow appropriate UI conventions for respective platforms?

Craft