What similarity metric should you use for your recommendation system?
By: Brett Vintch
Tags: Geometric, Recommendation System, Similarity, Data Science, Recommendations

The other obvious metric in Euclidean space is Euclidean proximity (that is, the inverse of Euclidean distance), although it has not historically been as popular as the others for Collaborative Filtering or Matrix Factorization. This may be because the operation is less efficient than inner product-based metrics, but it may also be because odd things can happen for short vectors near the origin. The nearest neighbor of a vector with a small magnitude could be a vector 180 degrees away, reflected across the origin. In cases where the vector direction doesn’t have a lot of meaning this may not be very important, but this is usually not the case for recommendation systems.

So which metric should you choose?

There’s no simple rule of thumb. The best way to choose a metric is to optimize predictions offline with cross-validation, or optimize them online with an A/B test. The right metric for one data set may not be the right metric for another because of the distribution of feedback values over users, items, and interactions. However, combining the geometric interpretation above with a bit of experience allows us to make a few observations:

Cosine similarity and Pearson correlation naturally normalize rating scales across users because they don’t care about the magnitude of the user vectors.

It doesn’t make sense to use Cosine similarity on unary or binary data, which is common in the case of implicit feedback, or for ordinal data that is not normalized or centered. This is because these rating vectors will always be in the first quadrant for every user, which vastly limits the dynamic range of the Cosine metric. That is, for two vectors in this quadrant, they can be a maximum of 90 degrees from one another which gives a floor of 0 to the similarity metric, versus 180 degrees, or -1, for centered data.

If we are seeking the nearest items for a user with vectors obtained from Matrix Factorization, there are few other things to note:

The most common formulation for Matrix Factorization models ratings as the dot product between the user vector and the item vector. Therefore, when looking for the best items for a user the natural similarity metric should be a dot product; the model was optimized to predict the ratings matrix with this assumption.

Assuming user and item data is not normalized, the dot product formulation means that popular items and optimistic users are likely to have long vectors. This is because these items and users have high average ratings, and long vectors are more likely to give a high predicted rating when multiplied.

Using Cosine similarity or Pearson correlation helps to mitigate the bias towards popular items, but can also end up recommending very unpopular, niche items.

Euclidean proximity gives a nice balance between being sensitive to vector direction (for vectors far enough from the origin) without being overtly biased to popular items. In my own work I’ve found that optimizing models that represent distance with Euclidean proximity often gives the most intuitive results.

Your mileage may vary. The most important things to remember are: 1) the performance of each metric is dependent upon the distribution of your data, and 2) your specific business objectives may help clarify the choice (for example, you may be willing to accept a slight dip in accuracy if the results are more diverse). It’s worth noting that there are many other ways to measure distance and we didn’t cover metrics well suited to implicit feedback, such as Jaccard distance, or its nearest neighbor approximation MinHash. Even then, a geometric approach can yield a new perspective and aid in the decision making process.