Evaluating Recommender Systems
By: Saumyadeepta Sen
Tags: Statistics, Recommendation System, Machine Learning, Recommender Systems, Data Science

Accuracy isn’t the only thing that matters with recommender systems. There are other things we can measure if they’re important to us. For example, coverage. That’s just the percentage of possible recommendations that our system is able to provide. Let’s think about a movies’ data set of movie ratings. It contains ratings for several thousand movies, but there are plenty of movies in existence that it doesn’t have ratings for. If we were using this data to generate recommendations on, say, IMDb, then the coverage of this recommender system would be low, because IMDb has millions of movies in its catalog, not thousands. It’s worth noting that coverage can be at odds with accuracy. If we enforce a higher quality threshhold on the recommendations we make, then we might improve your accuracy at the expense of coverage. Finding the balance of where exactly we’re better off recommending nothing at all can be delicate. Coverage can also be important to watch, because it gives us a sense of how quickly new items in our catalog will start to appear in recommendations.

Another metric is called diversity. We can think of this as a measure of how broad a variety of items our recommender system is putting in front of people. An example of low diversity would be a recommender system that just recommends the next books in a series that we’ve started reading, but doesn’t recommend books from different authors, or movies related to what we’ve read. This may seem like a subjective thing, but it is measurable. Many recommender systems start by computing some sort of similarity metric between items, so we can use these similarity scores to measure diversity. If we look at the similarity scores of every possible pair in a list of top-N recommendations, we can average them to get a measure of how similar the recommended items in the list are to each other. We can call that measure S.

Diversity is basically the opposite of average similarity, so we subtract it from 1 to get a number associated with diversity. It’s important to realize that diversity, at least in the context of recommender systems, isn’t always a good thing. We can achieve very high diversity by just recommending completely random items. But those aren’t good recommendations by any stretch of the imagination. Unusually high diversity scores mean that we just have bad recommendations more often than not. We always need to look at diversity alongside metrics that measure the quality of the recommendations as well.

Similarly, novelty sounds like a good thing, but often it isn’t. Novelty is a measure of how popular the items are that we’re recommending. And again, just recommending random stuff would yield very high novelty scores since the vast majority of items are not top sellers. Although novelty is measurable, what to do with it is in many ways subjective. There’s a concept of user trust in a recommender system. People want to see at least a few familiar items in their recommendations that make them say, “Yeah, that’s a good recommendation for me. This system seems good.” If we only recommend things people have never heard of, they may conclude that our system doesn’t really know them, and they may engage less with our recommendations as a result. Also, popular items are usually popular for a reason. They’re enjoyable by a large segment of the population, so we would expect them to be good recommendations for a large segment of the population who hasn’t read or watched them yet. If we’re not recommending some popular items, we should probably question whether our recommender system is really working as it should. This is an important point. We need to strike a balance between familiar, popular items and what we call serendipitous discovery of new items the user has never heard of before. The familiar items establish trust with the user, and the new ones allow the user to discover entirely new things that they might love. Novelty is important, though, because the whole point of recommender systems is to surface items in what we call “the long tail.”

Imagine this is a plot of the sales of items of every item in your catalog, sorted by sales. So the number of sales, or popularity, is on the Y axis, and all the products are along the X axis. We almost always see an exponential distribution like this. Most sales come from a very small number of items, but taken together, the “long tail” makes up a large amount of sales as well. Items in that long tail, the yellow part in the graph, are items that cater to people with unique interests. Recommender systems can help people discover those items in the long tail that are relevant to their own unique niche interests. If we can do that successfully, then the recommendations our system makes can help new authors get discovered, can help people explore their own passions, and make money for whoever we’re building the system for as well. Everybody wins. When done right, recommender systems with good novelty scores can actually make the world a better place. But again, we need to strike a balance between novelty and trust.

Another thing we can measure is churn. How often do the recommendations for a user change? In part, churn can measure how sensitive your recommender system is to new user behavior. If a user rates a new movie, does that substantially change their recommendations? If so, then our churn score will be high. Maybe just showing someone the same recommendations too many times is a bad idea in itself. If a user keeps seeing the same recommendation but doesn’t click on it, at some point should we just stop trying to recommend it and show the user something else instead? Sometimes a little bit of randomization in our top-N recommendations can keep them looking fresh, and expose our users to more items than they would have seen otherwise. But, just like diversity and novelty, high churn is not in itself a good thing. We could maximize your churn metric by just recommending items completely at random, and of course, those would not be good recommendations. All of these metrics need to be looked at together, and we need to understand the trade-offs between them.

One more metric is responsiveness; how quickly does new user behavior influence our recommendations? If we rate a new movie, does it affect our recommendations immediately or does it only affect our recommendations the next day after some nightly job runs? More responsiveness would always seem to be a good thing, but in the world of business we have to decide how responsive our recommender really needs to be, since recommender systems that have instantaneous responsiveness are complex, difficult to maintain, and expensive to build. We need to strike your own balance between responsiveness and simplicity.

Online A/B tests

Doing online A/B tests to tune our recommender system using your real customers, and measuring how they react to your recommendations.

We can put recommendations from different algorithms in front of different sets of users, and measure if they actually buy, watch, or otherwise indicate interest in the recommendations we’ve presented. By always testing changes to our recommender system using controlled, online experiments, we can see if they actually cause people to discover and purchase more new things than they would have otherwise. That’s ultimately what matters to our business, and it’s ultimately what matters to our users, too. None of the metrics we’ve discussed matter more than how real customers react to the recommendations we produce in the real world. We can have the most accurate rating predictions in the world, but if customers can’t find new items to buy or watch from our system, it will be worthless from a practical standpoint. If we test a new algorithm and it’s more complex than the one it replaced, then we should discard it if it does not result in a measurable improvement in users interacting with the recommendations we present. Online tests can help us to avoid introducing complexity that adds no value, and remember, complex systems are difficult to maintain. So remember, offline metrics such as accuracy, diversity, and novelty can all be indicators we can look at while developing recommender systems offline, but we should never declare victory until we’ve measured a real impact on real users from our work. Systems that look good in an offline setting often fail to have any impact in an online setting, that is, in the real world. User behavior is the ultimate test of our work. There is a real effect where often accuracy metrics tells us that an algorithm is great, only to have it do horribly in an online test.

Percieved Quality

Another thing we can do is just straight up ask your users if they think specific recommendations are good. Just like we can ask for explicit feedback

on items with ratings, we can ask users to rate our recommendations, too. This is called measuring the “perceived quality” of recommendations, and it seems like a good idea on paper, since, as we’ve learned, defining what makes a “good” recommendation is by no means clear. In practice though, it’s a tough thing to do. Users will probably be confused over whether we’re asking them to rate the item or rate the recommendation, so we won’t really know how to interpret this data. It also requires extra work from our customers with no clear payoff for them, so we’re unlikely to get enough ratings on our recommendations to be useful. It’s best to just stick with online A/B tests, and measure how our customers vote with their wallets on the quality of our recommendations.