International Workshop on Industrial Recommendation Systems
in conjunction with ACM KDD 2020

Multi-sided Exposure Bias in Recommendation
Himan Abdollahpouri

Masoud Mansoury∗

himan.abdollahpouri@colorado.edu
University of Colorado Boulder
Boulder, USA

m.mansoury@tue.nl
Eindhoven University of Technology
Eindhoven, The Netherlands

ABSTRACT
Academic research in recommender systems has been greatly focusing on the accuracy-related measures of recommendations. Even
when non-accuracy measures such as popularity bias, diversity,and novelty are studied, it is often solely from the users’ perspective. However, many real-world recommenders are often multistakeholder environments in which the needs and interests of several stakeholders should be addressed in the recommendation process. In this paper, we focus on the popularity bias problem which
is a well-known property of many recommendation algorithms
where few popular items are over-recommended while the majorityof other items do not get proportional attention and address its
impact on different stakeholders. Using several recommendation
algorithms and two publicly available datasets in music and movie
domains, we empirically show the inherent popularity bias of thealgorithms and how this bias impacts different stakeholders suchas users and suppliers of the items. We also propose metrics tomeasure the exposure bias of recommendation algorithms from theperspective of different stakeholders.

KEYWORDS
Multi-sided platforms, Recommender systems, Popularity bias, Multistakeholder recommendation

1

INTRODUCTION

Popularity bias is a well-known phenomenon in recommender systems: popular items are recommended even more frequently thantheir popularity would warrant, amplifying the long-tail effect already present in many recommendation domains. Prior researchhas examined the impact of this bias on some properties of therecommenders such as aggregate diversity (aka catalog coverage)[4, 22]. One of the consequences of the popularity bias is disfavoring less popular items where the recommendations are not fair interms of the amount of exposure they give to different items withvarying degree of popularity: an exposure bias. However, as we
discuss in [1], many recommender systems are multi-stakeholder
environments in which the needs and interests of multiple stakeholders should be taken into account in the implementation andevaluation of such systems.
In many multi-stakeholer recommenders as described in [1]two main stakeholders (or what often is being referred to as sides
in multi-sided platforms [11] ) can be identified: consumers (akausers) and suppliers. For instance, in a music platform such as
Spotify, on one side there are users who get recommendations forsongs in which they are interested and, on the other side, there areartists whose songs are being recommended to different users. The
popularity bias can be investigated from both sides’ perspective.
Regarding the users, not everyone has the same level of interestin popular items. In the music domain as an example, some users
might be interested in internationally popular artists such as Drake,
Beyoncé, or Ed Sheeran and some might be more interested inartists from their own culture that might not necessarily have thesame popularity as the aforementioned artists (such as the Iranianmusician Kayhan Kalhor) or generally they prefer certain typeof music that might not be popular among the majority of otherusers (such as country music). With that being said, we expect thepersonalization to handle this difference in taste but as we will see
in section 4.1 that is certainly not the case.
The suppliers also do not have the same level of popularity.
In many recommendation domains including movies, music, oreven house sharing, few suppliers have a large audience while
the majority of others may not be as popular though they stillmight have their fair share of audience. Now the question is, dorecommender systems let different suppliers with varying degreeof popularity reach their desired audience? Again, the short answer
is No as we will see more details in section 4.2.
Investigating the impact of recommendation algorithms on the
exposure bias on both users and suppliers is the focus of this paper.
We study several recommendation models in terms of their inherent
popularity bias and propose metrics that can measure such impact.

2 EXPERIMENTAL SETTING
2.1 Data
We have used two publicly available datasets for our experiments.
We needed datasets that either had information about the supplierof the items or we could extract them. We found two: the firstone is a sample of the Last.fm (LFM-1b) dataset [19] used in [9].
The dataset contains user interactions with songs (and the corresponding albums). We used the same methodology in [9] to turnthe interaction data into rating data using the frequency of theinteractions with each item (more interactions with an items will
result in higher ratings). In addition, we used albums as the items
to reduce the size and sparsity of the item dimension, therefore
the recommendation task is to recommend albums to users. We
considered the artists of each album as the supplier. Each album isassociated with an artist. Artists could have multiple albums. We
removed users with less than 20 ratings so only consider users forwhich we have enough data. The resulting dataset contains 274,707
ratings by 2,697 users to 6,006 albums from 1,998 artists.
The second dataset is the MovieLens 1M dataset1 . This dataset
does not have the information about the suppliers. We considered
the director of each movie as the supplier of that movie and we
extracted that information from the IMDB API. Total number ofratings in the MovieLens 1M data is 1,000,209 given by 6,040 users to
3,706 items. Overall, we were able to extract the director informationfor 3,043 movies reducing the ratings to 995,487. The total numberof directors is 831.
We used 80% of each dataset as our training set and the other
20% for the test.

∗ This author also has affiliation in School of Computing, DePaul University, Chicago,

1 Our experiments showed similar results on MovieLens 20M, and so we continue toUSA, mmansou4@depaul.edu.

use ML1M for efficiency reasons.

IRS@KDD2020, August 2020, San Diego, CA, USA(a) MovieLens

Abdollahpouri and Mansoury

The bias in rating data could be due to two different reasons:• External Bias: Some items and products are inherently
more popular than others even outside of the recommendersystems and in the real world. For instance, even before themusic streaming services emerge, there were always few
artists that were nationally or internationally popular suchas Shakira, Jennifer Lopez, or Enrique Iglesias. As a result of
this external bias (or tendency) towards popular artists, users
also often listen to those artists more on streaming services
and hence they get more user interactions.
• Feedback Loop: Since the recommendation algorithms havea higher tendency towards recommending popular items,
these items have a higher chance to be recommended tothe users and hence garnering a larger number of interactions from the users. When these interactions are logged
and stored, the popularity of those items in the rating data
increases since they get more and more interactions overtime [12, 20].

(b) Last.fm

Figure 1: The Long-tail Distribution of Rating data. 𝐻 items
are a small number of very popular items that take up
around 20% of the entire ratings. 𝑇 is the larger number ofless popular items which collectively take up roughly 20%of the ratings, and 𝑀 includes those items in between thatreceive around 60% of the ratings, collectively.

2.2

Algorithms

For our analysis, we have used three personalized recommendation
algorithms: Biased Matrix Factorization (Biased-MF) [14], Userbased Collaborative Filtering (User-CF) [5], and Item-based Collaborative Filtering (Item-CF) [8]. All algorithms are tuned to achieve
their best performance in terms of precision. The size of the recommendation lists for each user is set to 10. We also include a nonpersonalized Most-popular algorithm which only recommends
the 10 most popular items to every user given they have not rated
the items before. We used LibRec [10] and librec-auto [16] for running the algorithms.

3

POPULARITY BIAS

Skew in wealth distribution is well-known: The richest 10% of
adults in the world own 85% of global household wealth while
the bottom half collectively owns barely 1% 2 ; in recommendersystems a similar problem exists: a small number of popular items
appear frequently in user profiles and a much larger number ofless popular items appear rarely. This bias can originate from two
different sources: data and algorithms.

3.1

Bias in Data

Rating data is generally skewed towards more popular items. Figure 1 shows the percentage of users who rated different items inMovieLens and Last.fm datasets: the popularity of each item. Items
are ranked from the most popular to the least with the most popular
item being on the far left on the x-axis. Three different groups ofitems can be seen in these plots: 𝐻 which represents few items
that are very popular and take up around 20% of the entire ratings
according to the Pareto Principle [18]. 𝑇 is the larger number ofless popular items which collectively take up roughly 20% of theratings, and 𝑀 includes those items in between that receive around
60% of the ratings, collectively. The curve has a long-tail shape[6, 7] indicating few popular items are taking up the majority of
the ratings while many other items on the far right of the curve
have not received much attention. This type of distribution canbe found in many other domains such as e-commerce where few
products are best-sellers, online dating where few profiles attract
the majority of the attention, social networking platforms where
few users have millions of followers, to name a few.
2 https://www.wider.unu.edu/publication/global-distribution-household-wealth

3.2

Bias in Algorithm

Due to this imbalance property of the rating data, often algorithms inherit this bias and, in many cases, intensify it by overrecommending the popular items and, therefore, giving them ahigher opportunity of being rated by more users: the rich get richer
and the poor get poorer [2].
Figure 2 shows the percentage of users who have rated an itemon the x-axis (the popularity of an item in the data) and the percentage of users who received that item in their recommendations using
four different recommendation algorithms Biased-MF, User-CF,
Item-CF, and Most-popular in both datasets. The plots aim atshowing the correlation between the popularity of an item in the rating data versus how often it is recommended to different users. It isclear that in all four algorithms, many items are either never recommended or just rarely recommended. Among the three personalizedalgorithms, Item-CF and User-CF show the strongest evidence
that popular items are recommended much more frequently thanthe others. In fact, they are recommended to a much greater degree
than even what their initial popularity warrants. For instance, thepopularity of some items have been amplified from roughly 0.4
to 0.7 indicating a 75% increase. Both Item-CF and User-CF areover-promoting popular items (items on the right side of the x-axis)
while significantly hurting other items by not recommending themproportionate to what their popularity in data warrants. In fact,
the vast majority of the items on the left are never recommended
indicating an extreme bias of the algorithms towards popular items
and against less popular ones. Biased-MF does not show a positive
correlation between popularity in data and in recommendations although some items are still over-recommended (have much higher
popularity in recommendations versus what they had in ratingdata). However, this over-recommendation is not concentrated on
only popular items and some items from lower popularity values
are also over-recommended. Most-popular obviously shows thestrongest bias but, unlike the other three, it is not a personalized
method3 .

4

MULTI-SIDED EXPOSURE BIAS

We measure the impact of popularity bias on different stakeholders
in terms of exposure: how the bias in algorithms prevents users
3 If an item in the top 10 is already rated by a user it will be replaced by another popular

item. That is why there is an inflection point in the scatter plot for this algorithm on
MovieLens dataset.

Multi-sided Exposure Bias in Recommendation

IRS@KDD2020, August 2020, San Diego, CA, USA(a) MovieLens(a) MovieLens(b) Last.fm

Figure 3: Users’ Propensity towards item popularity(b) Last.fm

Figure 2: Item popularity versus recommendation popularity
to be exposed to the appropriate range of items and also how it
stops items from different suppliers to be exposed to their desiredaudience.

4.1

Exposure Bias From the Users’ Perspective

Not every user is equally interested in popular items. In cinema, forinstance, some might be interested in movies from Yasujiro Ozu,
Abbas Kiarostami, or John Cassavetes, and others may enjoy moremainstream directors such as James Cameron or Steven Spielberg.
Figure 3 shows the ratio of rated items for three item categories
𝐻 , 𝑀, and 𝑇 in the profiles of different users in the MovieLens 1M
and Last.fm datasets. Users are sorted from the highest interest

towards popular items to the least and divided into three equalsized bins 𝐺 = {𝐺 1 ,𝐺 2 ,𝐺 3 } from most popularity-focused to least.
The y-axis shows the proportion of each user’s profile devoted todifferent popularity categories. The narrow blue band shows the
proportion of each users profile that consists of popular items (i.e.
the 𝐻 category), and its monotone decrease reflects the way theusers are ranked. Note, however, that all groups have rated many
items from the middle (green) and tail (red) parts of the distribution.
The plots in Figure 4 are parallel to Figure 3, with the users
ordered by their popularity interest, but now the y-axis shows the
proportion of recommended items using different algorithms fromdifferent item popularity categories. The difference with the originaluser profiles in rating data especially in the case of Most-popular,
Item-CF, and User-CF is stark where the users’ profiles are richin diverse popularity categories, the generated recommendations
are nowhere close to what the user has shown interest at. In fact,
in Item-CF almost 100% of the recommendations are from thehead category, even for the users with the most niche-oriented
profiles. Tail items do not appear at all. We demonstrated here that
popularity bias in the algorithm is not just a problem from a global,
system, perspective. It is also a problem from the user perspective[3]: users are not getting recommendations that reflect the diversityof their profiles and the users with the most niche tastes (𝐺 3 ) are
the most poorly served.
To measure the impact of popularity bias on users, we need tocompare two lists together: the list of the items in a users’ profile andthe list of the items recommended to the user. To do this comparison,

IRS@KDD2020, August 2020, San Diego, CA, USA

Abdollahpouri and Mansoury

unlike the genre labels in [21] where it is possible for a movie tohave multiple genres, each item only has one popularity category.
Í𝑖 ∈𝜌𝑢 𝑟 (𝑢, 𝑖) 1 (𝑖 ∈ 𝑐)
Í𝑝 (𝑐 |𝑢) = Í(1)𝑐 𝑗 ∈𝐶 𝑖 ∈𝜌𝑢 𝑟 (𝑢, 𝑖) 1 (𝑖 ∈ 𝑐 𝑗 )Í𝑖 ∈ℓ𝑢 1 (𝑖 ∈ 𝑐)Í(2)𝑞(𝑐 |𝑢) = Í
𝑐 𝑗 ∈𝐶 𝑖 ∈ℓ𝑢 1 (𝑖 ∈ 𝑐 𝑗 )1 (.) is the indicator function returning zero when its argument
is False and 1 otherwise. Once we have 𝑃 and 𝑄, we can measure
the distance using Jensen–Shannon divergence, which is a modification of KL-Divergence that has two useful properties which
KL-divergence lacks: 1) it is symmetric: 𝔍(𝑃, 𝑄) = 𝔍(𝑄, 𝑃) and 2)
it has always a finite value even when there is a zero in 𝑄. For
our application, it is particularly important that the function bewell-behaved at the zero point since it is possible for certain item
categories to have zero items in them in the recommendation list.

(a) MovieLens

1
1
1
𝔍(𝑃, 𝑄) = 𝐾𝐿(𝑃, 𝑀) + 𝐾𝐿(𝑄, 𝑀), 𝑀 = (𝑃 + 𝑄)
(3)
2
2
2
where 𝐾𝐿 is the KL divergence.
We introduce the following metric to quantify the exposure bias
from the users’ perspective.
Users’ Popularity propensity Deviation (UPD):
Having different groups of users with varying degree of interest
towards popular items, we measure the average deviation of therecommendations given to the users in each group in terms of item
popularity. More formally,ÍÍ𝑢∈g 𝔍(𝑃 (𝜌𝑢 ),𝑄 (ℓ𝑢 ))

g∈𝐺

|g|

(4)
|𝐺 |
where |g| is the number of users in group g and |𝐺 | is the numberof user groups. 𝑈 𝑃𝐷 can be also seen as the average miscalibrationof the recommendations from the perspective of users in different
groups.
𝑈 𝑃𝐷 =

4.2(b) Last.fm

Figure 4: Users’ centric view of popularity bias

we need to compute a discrete probability distribution 𝑃 for eachuser 𝑢, reflecting the popularity of the items found in their profile
𝜌𝑢 over each item category 𝑐 ∈ 𝐶 (in this paper, 𝐶 = {𝐻, 𝑀,𝑇 }). We
also need a corresponding distribution 𝑄 over the recommendation
list given to user 𝑢, ℓ𝑢 , indicating what popularity categories arefound among the list of recommended items. In Steck [21] and Kaya
et al. [13] where, unlike this paper, the calibration is done according
to genres, the rating data is binarized to reflect just “liked” items.
We are retaining the original ratings when computing over user
profiles and, instead, using Vargas et al.’s [23] measure of category
propensity which has the users’ rating component in it. Note that

Exposure Bias From the Suppliers’
Perspective

As noted above, multi-stakeholder analysis in recommendation
also includes providers or as termed here suppliers, “those entities
that supply or otherwise stand behind the recommended items” [1].
We can think of many different kinds of contributors standing
behind a particular movie: for the purposes of this paper, we will
focus on movie directors. In the music domain, often the artists are
considered as the suppliers of the songs [17]. In this paper, we also
make the same assumption on Last.fm dataset.
We create three supplier groups 𝑆 = {𝑆 1, 𝑆 2, 𝑆 3 }: 𝑆 1 represents
few popular suppliers whose items take up 20% of the ratings, 𝑆 2 arelarger number of suppliers with medium popularity whose items
take up around 60% of the ratings, and 𝑆 3 are the less popular suppliers whose items get 20% of the ratings. Figure 5 shows the rankof different directors in MovieLens and artists in Lats.fm datasetsby popularity and the corresponding recommendation results fromdifferent algorithms. The recommendations have amplified the popularity of the popular suppliers (the ones on the extreme left) while
suppressing the less popular ones dramatically. Strikingly, using
Item-CF, movies from just 3 directors in 𝑆 1 (less than 0.4% of the
suppliers here) take up 50% of the recommendations produced,
while items items from the 𝑆 3 are seeing essentially zero recommendations.

Multi-sided Exposure Bias in Recommendation

IRS@KDD2020, August 2020, San Diego, CA, USA

To quantify the impact of popularity bias on the supplier exposure, we measure the amount of deviation different groups of suppliers experience in terms of exposure. In other words, the degreeof over-recommendation or under-recommendation of suppliers
from different groups.
Supplier Popularity Deviation (SPD):Í|𝑞(𝑠) − 𝑝 (𝑠)|𝑆𝑃𝐷 = 𝑠 ∈𝑆
(5)
|𝑆 |
Í
𝑞(𝑠) =𝑢 ∈𝑈𝑖 ∈ℓ𝑢 1 (𝐴(𝑖) ∈ 𝑠)ÍÍ𝑛 × |𝑈 |, 𝑝 (𝑠) =𝑢 ∈𝑈𝑖 ∈𝜌𝑢 1 (𝐴(𝑖) ∈ 𝑠)ÍÍ𝑢 ∈𝑈 |𝜌𝑢 |where 𝑞(𝑠) is the ratio of recommendations that come from itemsof supplier group 𝑠. 𝑝 (𝑠) is the ratio of ratings that come from itemsof supplier group 𝑠. 𝑈 is the set of users and 𝐴(.) is a mapping
function that returns the supplier for each item.
In fact, 1−𝑆𝑃𝐷 can be considered as Proportional Supplier Fairness[15] since it measures how the items from different supplier groups
are exposed to different users proportional to their popularity inrating data.

5(a) MovieLens

DISCUSSION AND FUTURE WORK

One important consideration regarding the deviation of popularity
from the users’ perspective (𝑈 𝑃𝐷) and suppliers’ perspective (𝑆𝑃𝐷)is how these two metrics behave with respect to one another. In
other words, whether calibrating the recommendations for the users
in terms of popularity would make the experience for the suppliersalso better. Figures 6(a) and 7(a) show the connection between these
two in the MovieLens and Last.fm datasets, respectively. We cansee that, generally, the lower the 𝑈 𝑃𝐷 (better calibration from theusers’ perspective) the lower the 𝑆𝑃𝐷 (better proportional fairness
for the suppliers) will be. This indicates the advantage of optimizing
for the popularity calibration in the recommendations since it will
also benefit the suppliers.
Another important finding is the fact that more accuracy does notnecessarily leads to a better calibration and vice versa as can be seenfrom Figures 6(b) and 7(b). For instance, on Last.fm, Most-popularand Biased-MF have roughly equal precision but the 𝑈 𝑃𝐷 for
Biased-MF is significantly lower (better) than Most-popular. The
reason is, the majority of the items are usually not very popularand, therefore, exclusively recommending popular items would notmatch the original distribution of the ratings (Figure 1) which leads
to high 𝑈 𝑃𝐷. In addition, if an algorithm randomly recommends
items, the likelihood of having items from 𝑀 and 𝑇 increases compared to many personalized recommendations where they sufferfrom popularity bias. However, improvement in 𝑈 𝑃𝐷 by randomly
recommending items would happen under the cost of having an
extremely low precision. Therefore, in practice, both 𝑈 𝑃𝐷 and𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 should be taken into account simultaneously in the optimization process.
The relationship between catalog coverage and supplier popularity deviation is also interesting to look at. Will an algorithm
that covers many items necessarily have lower (better) 𝑆𝑃𝐷? The
answer is No as can be seen from Figures 6(c) and 7(c). The reason is, item coverage (aka aggregate diversity) is the number of
unique items that an algorithm has recommended even if an item
is only recommended only once. However, what matters in 𝑆𝑃𝐷 isgiving appropriate exposure to the items from different suppliers
proportional to their popularity.
The impact of 𝑈 𝑃𝐷 and 𝑆𝑃𝐷 on the success of a real-world
recommender system can be evaluated using online A/B testing and(b) Last.fm

Figure 5: Suppliers’ centric view of popularity bias

see how metrics such as user engagement, retention, and also thesatisfaction of suppliers (e.g. artists in the music recommendation
platforms) would be influenced.

6

CONCLUSION

Recommender systems are multi-stakeholder environments; in addition to the users, some other stakeholders such as the supplierof the items also benefit from the recommendation of their items
and gaining a larger audience. The algorithmic popularity bias cannegatively impact both users and suppliers on a recommender system platform. In this paper, we demonstrated the severity of the
popularity bias impact on different sides of a recommender systemusing several recommendation algorithms on two datasets. We also
proposed metrics to quantify the exposure bias from the perspective of both the users and suppliers. Our experiments showed thatwhen the recommendations are calibrated for the users in terms of

IRS@KDD2020, August 2020, San Diego, CA, USA(a) UPD vs SPD

Abdollahpouri and Mansoury(b) UPD vs Precision(c) SPD vs Coverage

Figure 6: The relationship between different metrics (MovieLens)(a) UPD vs SPD(b) UPD vs Precision(c) SPD vs Coverage

Figure 7: The relationship between different metrics (Last.fm)
popularity (lower 𝑈 𝑃𝐷), it will also benefit the suppliers of the recommendations by giving them proportional exposure (lower 𝑆𝑃𝐷).
We believe, it is extremely crucial for the recommender systems
researchers to see the implications of real-world recommenders
where a single-stakeholder focus might not address all the complexities.

REFERENCES[1] Himan Abdollahpouri, Gediminas Adomavicius, Robin Burke, Ido Guy, Dietmar
Jannach, Toshihiro Kamishima, Jan Krasnodebski, and Luiz Pizzato. 2020. Multistakeholder recommendation: Survey and research directions. User Modelingand User-Adapted Interaction 30 (2020), 127–158. Issue 1.
[2] Himan Abdollahpouri, Robin Burke, and Bamshad Mobasher. 2017. Controlling
Popularity Bias in Learning to Rank Recommendation. In Proceedings of the 11th
ACM conference on Recommender systems. ACM, 42–46.
[3] Himan Abdollahpouri, Masoud Mansoury, Robin Burke, and Bamshad Mobasher.
2019. The unfairness of popularity bias in recommendation. In RecSys Workshop
on Recommendation in Multistakeholder Environments (RMSE).
[4] Gediminas Adomavicius and YoungOk Kwon. 2011. Improving aggregate recommendation diversity using ranking-based techniques. IEEE Transactions on
Knowledge and Data Engineering 24, 5 (2011), 896–911.
[5] Charu C Aggarwal. 2016. Neighborhood-based collaborative filtering. In Recommender systems. Springer, 29–70.
[6] Chris Anderson. 2006. The long tail: Why the future of business is selling more forless. Hyperion.
[7] Òscar Celma and Pedro Cano. 2008. From hits to niches?: or how popular artists
can bias music recommendation and discovery. In Proceedings of the 2nd KDD
Workshop on Large-Scale Recommender Systems and the Netflix Prize Competition.
ACM, 5.
[8] M. Deshpande and G. Karypis. 2004. Item-Based Top-N Recommendation Algorithms. ACM Transactions on Information Systems 22, 1 (2004), 143–177.
[9] Kowald Dominik, Schedl Markus, and Lex Elisabeth. 2019. The Unfairness ofPopularity Bias in Music Recommendation: A Reproducibility Study. arXiv
preprint arXiv:1912.04696 (2019).
[10] Guibing Guo, Jie Zhang, Zhu Sun, and Neil Yorke-Smith. 2015. LibRec: A Java
Library for Recommender Systems. In UMAP Workshops.
[11] Andrei Hagiu and Julian Wright. 2015. Multi-sided platforms. International
Journal of Industrial Organization 43 (2015), 162–174.

[12] Ray Jiang, Silvia Chiappa, Tor Lattimore, András György, and Pushmeet Kohli.
2019. Degenerate feedback loops in recommender systems. In Proceedings of the2019 AAAI/ACM Conference on AI, Ethics, and Society. 383–390.
[13] Mesut Kaya and Derek Bridge. 2019. A comparison of calibrated and intent-aware
recommendations. In Proceedings of the 13th ACM Conference on Recommender
Systems. 151–159.
[14] Y. Koren, R. Bell, and C. Volinsky. 2009. Matrix factorization techniques forrecommender systems. Computer 42, 8 (2009), 30–37.
[15] Matt J Kusner, Joshua Loftus, Chris Russell, and Ricardo Silva. 2017. Counterfactual fairness. In Advances in Neural Information Processing Systems. 4066–4076.
[16] Masoud Mansoury, Robin Burke, Aldo Ordonez-Gauger, and Xavier Sepulveda.
2018. Automating recommender systems experimentation with librec-auto. In
Proceedings of the 12th ACM Conference on Recommender Systems. ACM, 500–501.
[17] Rishabh Mehrotra, James McInerney, Hugues Bouchard, Mounia Lalmas, andFernando Diaz. 2018. Towards a fair marketplace: Counterfactual evaluationof the trade-off between relevance, fairness & satisfaction in recommendation
systems. In Proceedings of the 27th acm international conference on informationand knowledge management. 2243–2251.
[18] Robert Sanders. 1987. The Pareto principle: its use and abuse. Journal of Services
Marketing (1987).
[19] Markus Schedl. 2016. The lfm-1b dataset for music retrieval and recommendation.
In Proceedings of the 2016 ACM on International Conference on Multimedia Retrieval.
103–110.
[20] Ayan Sinha, David F. Gleich, and Karthik Ramani. 2016. Deconvolving feedback
loops in recommender systems. In Advances in neural information processing
systems. 3243–3251.
[21] Harald Steck. 2018. Calibrated recommendations. In Proceedings of the 12th ACM
Conference on Recommender Systems. ACM, 154–162.
[22] Saúl Vargas and Pablo Castells. 2011. Rank and Relevance in Novelty and Diversity
Metrics for Recommender Systems. In Proceedings of the Fifth ACM Conference
on Recommender Systems (Chicago, Illinois, USA) (RecSys ’11). ACM, New York,NY, USA, 109–116. https://doi.org/10.1145/2043932.2043955[23] Saúl Vargas and Pablo Castells. 2013. Exploiting the diversity of user preferences
for recommendation. In Proceedings of the 10th conference on open research areas
in information retrieval. 129–136.