Popular evaluation metrics in recommender systems explained
By: Giorgos Papachristoudis
Tags: Machine Learning, Recommender Systems, Recall, Precision, TensorFlow

What are recall@k and precision@k?

Precision and recall are evaluation metrics that are commonly used in classification settings. In the context of recommender systems, we use metrics like recall@k and precision@k, since it is more common to provide k recommendations per example. An example can be a user query in our case. I.e., if a user likes a particular set of movies (the query), the output will be a list of k movies (of potential interest to her). Sticking to our movie paradigm, a user likes 70 out of 200K total movies (which translates to this example having 70 labels), we can represent this with 1x200,000 vector where all elements would be 0 except for 70 being 1 (in the positions corresponding to the movies the user has expressed interest in). This is called multi-hot encoding.

In that respect, we can treat this problem as a multi-label classification problem, where an example is labeled as multiple different things. A similar and more familiar concept borrowed from Computer Vision is when we have an image and we try to label it with the different animals present in the image. On the other hand, multi-class classification is when we have multiple classes but only one label per example (e.g., image is classified only as cat, only as dog, as horse etc). Let’s proceed to define recall@k and precision@k.

Recall@k is defined as the percentage of true labels captured when we recommend k labels per example. Its exact definition is:

On the other hand, precision@k is defined as the percentage of predictions we get right. Its exact definition is:

Let’s borrow an example from medicine (hopefully not too graphic) to explain the different goals of recall and precision. Assume a doctor is to remove a patient’s metal, non-life threatening implants that are used to speed up the recovery of a broken leg. Recall is concerned with removing as many of these implants as possible. Precision is concerned with removing the implants with as few attempts as possible. If the doctor is sloppy, a patient could have all of her implants removed in four consecutive operations which would result in very high recall, very low precision. On the other hand, a minimally intrusive operation that lasts only half an hour but only removes one of the implants would result in very low recall, very high precision. Both of these situations are unacceptable. The best of both worlds would be to optimize for one metric, keeping the other metric above a minimum acceptable threshold. We will talk more about this later on.

Going back to the definitions of recall@k and precision@k, assume an example has three true labels and the algorithm gives 5 recommendations. If two of these recommendations correspond to true labels, then # of true labels captured = 2 , # of true labels = 3 and # of predictions made = 5 . Therefore, recall@k = 2/3 = 0.67, precision@k = 2/5 = 0.4. For simplicity, we will denote the # of true labels captured as TP@k, which is known as true positives@k. In the general case of batch_size examples and k recommendations per example, # predictions made = batch_size * k . Also, # of true labels will refer to the total number of labels in the batch. Both recall and precision take values between 0 and 1.

There is usually an inverse relationship between recall and precision. In addition, there’s a monotonic relationship between recall@k and k (the number of recommendations per example), while precision@k tends to lower with k. The monotonicity between recall@k and k is very easy to show. The denominator in recall@k is # of true labels which remains constant. The more recommendations we add per example, the more likely it is one of them to be a true label, which would increase the numerator TP@k . Now, when it comes to precision@k both the numerator ( TP@k ) and denominator ( # predictions made ) are affected with the addition of new recommendations. Let’s assume we increase the resulting output to one more recommendation per example. That is, we go from k to k+1 . What condition should hold so that precision@(k+1) ≥ precision@k?

When do we have precision@(k+1) ≥ precision@k?

TP@(k+1)-TP@k = 1 , if the k+1 ᵗʰ recommendation is a true label and 0 , otherwise. Since, 0 ≤ precision@k ≤ 1, precision@(k+1) ≥ precision@k if and only if the k+1 ᵗʰ recommendation is a true label. In general, if we add m new recommendations per example, we can show (we will avoid the proof here but it follows the same logic to the proof above) that precision@(k+m) ≥ precision@k if any only if

which basically translates to requiring the precision of the additional m recommendations being larger than the current precision. The above equation shows also why precision@k is more likely to decrease with k. If an example has L true labels and we already predicted some of them, it is more and more difficult to uncover the remaining labels for every recommendation we add. Lastly, let’s give an example with TensorFlow to make this a bit more clear.

Here, we have just one example in the batch (we can easily generalize to batch_size examples) that has two true labels: 0 and 3. We assume we can have up to 6 labels per example, which is the number of columns in predictions . We will try different values of k (number of recommendations per example) and check the effect they are having in recall@k and precision@k.

The output of the above snippet is:

recall@1 = 0.00, precision@1 = 0.00

recall@2 = 0.00, precision@2 = 0.00

recall@3 = 0.50, precision@3 = 0.33

recall@4 = 0.50, precision@4 = 0.25

recall@5 = 1.00, precision@5 = 0.40