Do your ML metrics reflect the user experience?
By: Danny Doan
Tags: Machine Learning, AI, User Experience, Technology, Product Development

‘Wind tunnels’ provide a testing ground for faster feedback loops

Developing ML products shares some surprising parallels with developing a Formula 1 car. The costs of delivering a race car to the track and a web-scale ML product to production are both high. To manage costs, Formula 1 teams don’t have every race car prototype go straight from the blueprint to the race track for a trial run.

Instead, cheaper small-scale models first go through rigorous testing and refinement in a wind tunnel. In that wind tunnel, metrics that are relevant to the car’s aerodynamics — such as downforce and drag — are measured as an indicator of the design’s performance.

The wind tunnel allows a much shorter feedback cycle, allowing the team to gauge the performance of the car after every iteration of the research. They only take the race car to the track for a proper trial run once the results from the wind tunnel show satisfying performance, because trial runs take a lot of time and effort.

Similarly, we can test our product iterations by conducting UX interviews with real customers. By asking targeted questions, we’re having a trial run with customers about the performance of our product. However, these interviews are expensive to schedule, run, and analyse. And they don’t scale to a large number of customers. Without alternative testing strategies, we run the risk of exposing a subpar product that can potentially damage customers’ trust and Xero’s reputation.

So if our product were a race car, a UX interview might be a trial run on a race track. Which raises the question: what’s our wind tunnel?

In ML, our wind tunnel is the evaluation pipeline

We needed a wind tunnel for our ML product, to perform offline evaluations of each iteration and benchmark its performance before showing it to customers. So we built an evaluation pipeline for this purpose, which reduced the feedback cycle from weeks (organising user interviews) to minutes (calculating and analysing metrics).

How we develop and measure our ML product’s performance

What did we measure in our wind tunnel? Good question. We started with two basic ML metrics — precision and recall:

Precision : What proportion of predicted transactions actually happened?

: What proportion of predicted transactions actually happened? Recall: What proportions of transactions that should have been predicted were correctly predicted?

They might sound simple, but can quickly become more complicated once we put them in the context of cash flow. For example, what is a correct prediction? Will our customers care if the predicted amount is off by 5%? How about 20% or 30%? And if the predicted amount is correct, what if the predicted date of the transaction is out by a few days?

As for recall, many business transactions only occur once, so which ones should be considered predictable for our recall? Are all transactions equal? Our customers care a lot more about a correctly predicted rent payment, than a bank fee. So how do we account for the relative importance of transactions?

These are just a few of the open-ended questions we had to address when designing our metrics. Without considering the answers, we would have had different variations of precision and recall, without knowing if one was better than the other in terms of how they reflect user experience.

The metrics we measure are complex, and require validation

Clearly, designing metrics is a complex endeavour, and it’s tricky to get right. Therefore, we needed a way of knowing if our metrics were measuring the right things. Without this evidence, we ran the risk of satisfying ourselves and our stakeholders on irrelevant numbers that didn’t actually reflect our customers’ product experience. It would be like putting a Formula 1 race car in the wind tunnel, and feeling confident about its performance by measuring its shininess or paint colour.

We needed to find evidence to determine whether our ML metrics were a good proxy for user experience. To do this, we correlated our ML metrics with the subjective, amorphous concept of user experience.

Correlating ML metrics with user feedback

Step 1: Scoring qualitative feedback

Our team works closely with a group of early adopters to get regular user feedback on our product. Every month or so, we interview these small business owners to talk through the latest changes we’ve made. This might be new features, UI components, or improvements to predictions. We used the feedback from these interviews to set the context for our ML metrics.

We did this by giving each interview two scores — one for what we expected their precision value to be, and another for recall. The quality of predictions were categorised using a score out of five:

0/5 — Terrible, all predictions are wrong (precision). No predictions, missed everything (recall)

1/5 — Bad, not useful

2/5 — Meh

3/5 — OK

4/5 — Good

5/5 — Perfect, couldn’t be better. Andrew Ng’s face appears in a dream and pats you on the back

Taking precision as an example, we estimated how precise our predictions were based on their comments:

“Almost all of these predictions are wrong!” → 1/5 expected precision score

“Wow, every single one of these predictions is correct!” → 5/5 expected precision score

We used the same process for recall — if a customer indicated that we failed to predict some key things, we expected the recall score to reflect that.

“You’ve correctly predicted some, but you’ve missed many transactions that are important to me!” → 2/5 expected recall score

“Wow, it predicted everything I can think of!” → 5/5 expected recall score

This process of assigning a score to qualitative feedback can be subjective, so we put some measures in place to reduce bias:

We scored the qualitative feedback before looking at the metrics, because we didn’t want the actual values to bias our judgement

We agreed beforehand on the user experience that each score represents

We got our scores checked and signed off by our product owner and designer, since they attended all the interviews and had more context on these small businesses

TIP: We were able to bootstrap off our existing usability testing schedule, but you don’t need a formal UX research plan to get a rough gist of how good your product is to a human. Maybe you could show your feature to some people from another team who are less familiar with the product, and get informal feedback that way.

Step 2: Calculating ML metrics for the interviewed users

After we attempted to put some numbers against our qualitative feedback, we calculated our algorithm’s performance metrics (precision and recall) to begin our correlation study. Using our evaluation pipeline, we were able to quickly generate metrics for each of the users in our testing group.

Step 3: Analysing correlation between qualitative feedback and quantitative metrics

Once we had both the expected qualitative scores and the actual quantitative metrics, we analysed their correlation.