How to understand your Design System’s health, and eventually, its success.
By: Christos Kastritis
Tags: Systems Thinking, Surveys, Research, Design Systems, Metrics

Creating a baseline survey

The above two methods are great for getting the ball rolling. But we got to think long term. How might we turn those qualitative insights into something quantifiable? Can we do this on a recurring basis? Enter our baseline survey. I decided on doing this for a few reasons:

To see if there is any similarity between this and the previous rounds of research in terms of feedback.

To track, over time, how these perceptions change, allowing us to better prioritize our focus areas.

To branch out and include as many designers and engineers as possible.

We already have insights. But now we are merging common themes between design and engineering into a recurring survey to capture what progress has been made.

To make these insights quantifiable, in a language that our stakeholders, or really anyone can understand.

Setting it up

A Google Form was sent out with multiple-choice questions. These were for product designers and frontend engineers across each platform. I like Google Forms because you can export the results into a spreadsheet, speeding up the processing of survey data.

The options were mostly numbers-based (1–7) from strongly disagree to strongly agree, although it did vary when a question suited more to a yes/no response. People also had the option to provide more feedback if they felt like it.

I focused on statements around their overall experience which consisted of quality, speed, confidence, contributing, completeness, and comms. I’ll be breaking these down in the next section as I find that’s where most of us can get stuck. Which is where those previous insights and research studies come in handy.

The ECTS grading scale (on page 6) was used as I believe it is the fairest when it comes to grading a design system. Memories of your teen years flash before your very eyes.

It’s all about the KPIs

Statements. Questions. Topics. KPIs. Whatever you want to call them, they are beyond important. Especially for your baseline. I split our survey into two sections: General and Experience. Let’s dive in.

1 — Role

“What is your role?”

Content/Product designer, Frontend engineer (platforms are separate options as they have different experiences), Other (for any scallywags).

2 — Tenure

“How long have you been at Deliveroo?”

Having different options for this will give us an insight into whether new hires have very different opinions in comparison to those who have been here longer. What are your assumptions within your own company? At the time of running this survey I had only started initiatives around educating new hires across our Tech org, so expected the results to vary.

3 — Quality

“Our design systems make it easier for me to build a high-quality UI.”

Ratings-based answer

Quality is a staple and widely-understood benefit of a design system, thus it should be included.

4 — Speed

“Our design systems help me implement or design features faster.”

Ratings-based answer

Much like quality, speed is a staple and widely-understood benefit of a design system, thus it should be included.

5 — Confidence

“I feel confident in using our design systems.”

Ratings-based answer

It is important that people feel they are armed with the right information and are getting the most out of our design systems. I feel the word confidence captures this well. We know from previous insights that there are a lot of uncertainties around many things so for us it was imperative to include this and monitor it going forward. Remember our vitals? I feel like this is an opportunity for a pun, but I can’t lose all credibility just yet.

6 — Contributing Back: Ease

“I can contribute back to our design systems with ease.”

Ratings-based answer

This is broken out from Speed based on the insights from our own research. Our insights signal that different platforms have trouble contributing back, which can be either Ease or Speed related or even both.

7 — Contributing Back: Speed

“I can contribute back to our design systems with speed.”

Ratings-based answer

The same rationale behind number 6.

8 — Completeness: Availability

“I find myself using something that is outside of our design systems.”

Frequency-based answer

One of our main insights is around the disparity between what is available in our design tool libraries vs. our code-based component libraries. There is a different percentage of components available in each platform and there will be some cases where:

What they require is in Figma but not code (yet)

There is something similar available but does not fit their needs

There is nothing similar available

So it’s something that our team should definitely include.

9 — Completeness: UI Coverage

“Design systems covers at least 80% of my use cases when building a UI.”

Yes/No answer

Our design system will not cover 100% of use cases, which is totally fine. 80% is based on the Pareto principle (80/20 rule). I believe this is a realistic number for us to be aiming for in terms of adoption across each application. Those 100%-ers are living in a pipe dream. For now, doing this through a survey is a good place to start until we can properly quantify how much is actually covered in each application.

10. Comms

“I know about the latest updates in our design systems.”

Ratings-based answer

I don’t need to say anything here, do I?

Pointers

Base your themes and questions on the problems that surfaced from previous studies. This helps you hone in on the areas you should be focusing on.

If your question includes an “and”, separate into two questions. This is because you are technically asking two questions. A participant’s response may lean towards one of them. Separating them removes that ambiguity and leads to a more accurate response. I originally did this for my question on Contributing Back, so thank you, Audrey, for the tip.

I mentioned this before when running interviews, but please, run your questions past several disciplines. Researchers, content writers, designers, engineers. What makes sense to you might sound completely alien to others. We want to be certain that what we send out is understandable, reusable but most importantly is not a waste of effort.

Think about what metric you wish to communicate with your audience. I felt that a grading system would be the most straightforward and easily understood within our business.

Leave space for extra feedback as optional. Let’s face it, surveys can be time-consuming. Those who wish to finish it quickly can do so by answering the MCQs. Those who feel invested and wish to contribute have that option to do so but now, we are no longer blocking the rest of our participants.

The analysis should separate Designers and Engineers into separate streams.

I decided on this because:

They’re using the systems in different ways, thus the feedback may be drastically different.

From an analysis point of view, the understanding of our designers is not as detailed as engineers. Water cooler chats excluded.

New feedback may differentiate the next steps that have already been defined.

Analyzing for days

By taking the responses of each statement per discipline, I calculated the median for each one. In this scenario, the median makes more sense because it is likely this data will have some extreme values (high or low). In such cases, the median gives a more realistic estimate of central value. We can then turn that into a percentage, and from there we use our grading scale and end up with a very simple grade that anyone can understand. Create a report. Have a high-level summary at the top. But also go deep into your analysis. For each statement include things such as:

Why you chose this statement

A breakdown that is separated into each discipline

A graph per discipline

Quotes directly related to the statement. I find including the rating they gave and their tenure helps gain better context around their feedback.

What the feedback signals

What the opportunities are

The grade, followed by a brief summary.

I could go into more detail around analyzing and structuring reports, but let’s leave that for another blog post.