A reproducibility study of “User-item fairness tradeoffs in recommendations”
Authors: Sander Honig
Elyanne Oey
Lisanne Wallaard
Sharanda Suttorp
Clara Rus
Venue: Accepted by TMLR ()

Abstract: Recommendation systems are necessary to filter the abundance of information presented in our everyday lives. A recommendation system could exclusively recommend items that users prefer the most, potentially resulting in certain items never getting recommended. Conversely, an exclusive focus on including all items could hurt overall recommendation quality. This gives rise to the challenge of balancing user and item fairness. The paper “User-item fairness tradeoffs in recommendations” by Greenwood et al. (2024) explores this tradeoff by developing a theoretical framework that optimizes for user-item fairness constraints. Their theoretical framework suggests that the cost of item fairness is low when users have varying preferences compared to each other, and may be high for users whose preferences are misestimated. They empirically measured these phenomena by creating their own recommendation system on arXiv preprints, and confirmed that the cost of item fairness is low when users have preferences that differ from one another. However, contrary to their theoretical expectations, misestimated users do not encounter a higher cost of item fairness. This study investigates the reproducibility of their research by replicating the empirical study. Additionally, we extend their research in two ways: (i) verifying the generalizability of their findings on a different dataset (Amazon books reviews), and (ii) analyzing the tradeoffs when recommending multiple items to a user instead of a single item. Our results further validate the claims made in the original paper. We concluded the claims hold true when recommending multiple items, with the cost of item fairness decreasing as more items are recommended.

Published in Transactions on Machine Learning Research (06/2025)
A reproducibility study of "User-item fairness tradeoffs in recommendations"
Sander Honig University of Amsterdam
conder.konig@@sttueent...o.
Elyanne Oey University of Amsteriam
Lisanne Wallaard University of Amstendam
lisonne. Assnme.vwllaav@@sttuddd.uuu.u
Sharanda Suttorp
shammnda.suttorrsstttdddd.....
University of Amsterdam
Clara Rus University of Amstendam
c.0.rus@uvvo.r ml
Reviewed on OpenReview: https: openreview. net/ forum? id= Ad=ultzen UitzhhaaL
Abstract
Recommendation systems are necessary to filter the abundance of information presented our everyday lives. A recommendation system could exclusively recommend items that users prefer the most, potentially resulting in certain items never getting recommended. Conversely, exclusive focus including all items could hurt overall recommendation quality. This gives rise to the challenge of balancing user and item fairness. The paper "User-item fairness tradeoffis in recommendations" by Greenwood et al. (2024) explores this tradeoff by developing a theoretical framework that optimizes for user-item fairness constraints. Their theoretical framework suggests that the cost of item fairness is low when users have varying preferences compared to each other, and may be high for users whose preferences are imsest They empirically measured these phenomena by creating their recommendation system arXiv preprints, and confirmed that the cost of item fairness is low when users have preferences that differ from another. However, contrary to their thcoretical expectations, miscstimated users do not encounter higher cost of item fairness.
This study investigates the reproducibility of their research by replicating the empirical study, Additionally, we extend their research in two ways: (i) verifying the generalizability of their findings on a different dataset (Amazon books reviews), and (ii) analyzing the tradeoffs when recommending multiple items to a user instead of single item. Our results further validate the claims made in the original paper. We concluded the claims hold true when recommending multiple items, with the cost of item fairness decreasing more items are recommended.
1 Introduction
Recommendation systems have become incrcasingly popular due to the rapid expansion of the internet and the vast amount of information it holds (Lü et. al. (2012)). They are necessary to filter the abundance of information, ensuring users are just shown small selection of relevant items rather than being overwhelmed by information overload. One approach to exclusively recommend items that the user prefers the most, however, this could result in certain items never getting recommended. To prevent this, algorithmic tech- niques have been developed to improve item fairness in recommendations (Mehrotra et al. (018): Wang et al.

---

Published in Transactions on Machine Learning Research (06/2025)
2021 However, too much focus on item fairness may lead to users getting less relevant recommendations, thereby compromising user fairness and, in turn, hurting the overall recommendation quality Wang et al. h023) This shows the intricacy in the balance between user and item fairness (multi--ided fairness).
Previous studies have focused on designing algorithms that strive to balance user fairness, item fairness and overall recommendation quality (Burke et al. (2018); Wann & Joachims (2021)): yet little research exists on the effects and the tradeoffs of this multi-sided optimization. [Greenwood et al. (2024) address this gap by developing theoretical and empirical framework to analyze these effects and tradcoffs in the context of multi-sided fairness- constrained optimization. Theoretically, they concluded two phenomena: (i) the cost (i.c. decline in user fairness, defined as minimum normalized user utility) of item fairness is low when users have preferences that differ from one another, and (ii) item fairness may have high cost on users whose preferences are misestimated, such as users who are new to the system (i.e. cold start users). They empirically measured these phenomena by creating their own recommendation system for arXiv preprints. Similar to their thcoretical findings, they found the cost of item fairness to be low when users have preferences that differ from one another. However, they concluded that the cost of having iisestimated users was already high such that imposing further item fairness constraints them did not further increase costs.
This reproducibility study focuses on examining the reproducibility of Greenwood et al. (2024) by repli- cating the experiments using their published code on the same arXiv dataset and Semantic Scholar data. Additionally, we extend their research in two ways. Firstly, we make an extension by examining whether the proposed claims hold for a novel dataset consisting of Amazon books reviews. This aims to evaluate the generalizability of the claims regarding multi-sided fairness tradeoffs across more widely used domains, such book e-commerce. Recommending books is a more common application than academic paper rec- ommendation, primarily due to its commercial relevance, which makes it useful extension for evaluating the generalizability of the original work. Secondly, we further extend the research by tackling one of its limitations: analyzing the effects of user-item fairness tradeoffs when recommending multiple items to each user. Greenwood et al. (2024) experiment with recommending only one item, however, recommending mul- tiple items better reflects a real-world scenario as recommendation systems typically suggest multiple items to users rather than a single one. To support this scenario, we adapted their theoretical framework and performed the same experiments.
This paper successfully reproduces the empirical findings of the experiments on the original and newly introduced dataset, further validating the claims made by Greenwood et al. (2024). Moreover, we observed a decreased cost when imposing item fairness constraints as the expected number of items recommended increases. The GitHub repository containing the code discussed in this paper can be found at the following link: https://gii trggittub.ccom cot. /tta://aaaahhoi/B---ee--e------e faateerreiet----eeee--eeeeeeeees htt.://eeeeeeataaaaaaaaaaaoreeee
2 Scope of reproducibility
This research investigates the empirical claims concerning the identified phenomena by Greenwood et al. 2024h. Their work focuses the implications of including item fairness constraints alongside user fairness in recommendattion system for arXiv preprints. The fairness definition they adopt follows an individual egalitarian approach, meaning maximizing the utility of the worst-off individual. Therefore, their definition of user fairness is given by the normalized minimum user utility. By employing an in-proccssing technique for fairness that integrates item fairness directly into the recommendation algorithm, they discovered the following phenomena:
Discovered phenomenon 1: "When user preferences are diverse, there is 'free' item and user fairness." Free fairness implies that item fairness can be imposed with minimal cost to the user while it is beneficial for the items.
Discovered phenomenon 2: "Users whose preferences are misestimated can be especially disadvan- taged by item fairness construints."
Building on these theoretical insights, they attempted to empirically validate these phenomena using an arXiv recommendation system, leading to the following claims:
2

---

Published in Transactions on Machine Learning Research (06/2025)
"More homogeneous groups of users have steeper user-item fairness tradcoffss as theorelically predicted, diverse user preferences decrease the price of ilem fairness." Notably, the price of fairness becomes substantial only when strict item fairness constraints are imposed.
Empirical claim 1:
Empirical claim 2: "The 'price of misestimation' is high (users for whom less training data is available receive poor recommendations), but on average item fairness constraints do not increase this cost."
This study seeks to validate these empirical claims by reproducing the experiments of Greenwood et al. (2024) on the arXiv dataset. To further explore the robustness of their claims, this paper will extend the original work by examining whether the proposed claims hold for a novel dataset about Amazon books reviews. Additionally, this rescarch expands the original study by increasing the expected number of recommended items, allowing for deeper analysis of the user-item tradeofT.
3 Methodology
For the reproduction of these empirical findings, we lean on the original theoretical framework and use the GitHub repository of the original paper I We further extend their research by reproducing their claims on a novel dataset and examine the effect of increasing the expected number recommended items per user on user fairness.
3.1 Theoretical framework
Greenwood ct al. (2024) propose a recommendation framework that aims to balance user fairness and item lairness, leading to multi-sided fairness problem. The principle of egalilarian fairness (i.e. the utility of all agents is given by the utility of the worst-off agent) is used to quantitatively define both user and item fairness, where the balance between the two is parametrized by fairness level This leads to the following optimization constraint:
(7( max, min; Wp) subject to miny Iy(p,w) 2 max, miny ())
(1)
Here, p denotes a set of parameters referred to as the recommendation policy (ó is the recommendation policy for the item fairness optimization problem, is the recommendation policy for the user fairness optimization problem and since subjected to the item fairness constraints to the entire optimization problem). Pt is the distribution over items recommend to user i, with Σ, Pig = 1, Pis E [0, 1]. Utility matrix w is caleulated using the cosine similarity between user and item embeddings, where Wij > 0 denotes the utility of recommending item to user i. U.(www denotes user i's expected utility following P normalized by the utility they would receive from being recommended their best match, and (p, denotes item j's expected utility following p normalized by the utility it reccives if it is recommended to every user. Their values are given by:
U:(pww) = E, Psj wij max, wiy
(2)
[)(,w) E. Pij Wij
E,wij
In words, Equation socks to find the parameters p for which the minimum normalized expected user utility is maxinized (i.e. most fair according to egalitarian fairness), while the minimum normalized expected item utility is at least a fraction 7 of its optimal value. Note that for U*(y = 0,w) = 1 since the optimal recommendation policy would doterministically recommend each user their most preferred item.
The price of the item fairness constraint can be calculated as the normalized decrease in user fairness when subjected to the item fairness, given by the formula:
( = U*(7 = w/ - 07=
(3)
U-(: = w
Accessible htttpp////ghhbb..cc/////hhhh/hh/ Aot.oaacee.......aeeeeeeeeeerooo Research
3

---

Published in Transactions on Machine Learning Research (06/2025)
Utility matrix w can be estimated based on the embeddings of users and items to be recommended, however, it should not be ignored that these utilities are mere estimations. This price of misestimation can be computed by:
To (v),w,))))))) U- = ,w) - min, Ui(((r'). (' ,
(4)
Here, denotes the true utility matrix, @ denotes the estimated utility matrix, and p(r') denotes recom- mendation policy that solves the recommendation problem (Equation with respect to the misostimated utilities, i.e. ((7') solves U (7))
For a new user without known preferences (a so-called cold start user), the platform estimates their preference as the average of the preferences of existing users. This would, without item fairness constraints, result in the platform recommending generally popular items to all cold start users, regardless of their actual preference. Introducing item fairness constraints could worsen this now not only items recommended regardless of actual preferences, but (near) all items will be recommended including many generally unpopular ones. Therefore, theoretically one would expect user fairness to be worsened as increases.
3.2 Datasets
In this paper, we use the arXiv datasot to reproduce and extend the work of Greenwood al. (2024). Furthermore, we use the Amazon books reviews dataso to investigate how their findings generalize a different domain, namely book e-commerce.
3.2.1 Original dataset: arXiv
The aiviv dataset consists of 2,639,142 papers, containing papers published on arXiv from 1991 up until now Each entry covers a paper, which is defined by fourteen attributes. Just as Greenwood et al. (2024). we only considered papers that possess one or more Computer Science categories and dropped all remaining papers. This left 707,763 papers. The train and test sets are created based on a paper's year of publication: all papers published before 2020 are selected for the train set, and all papers from 2020 are selected for the test set, resulting in 255, and 65,948 papers respectively. The category distributions of these sets are visualized in Appendix A.I Figure The distribution of our dataset differs slightly from the ones presented
by Greenwood et al. (2024) shown in Appendix A.2 F Figure
b Additional information on all test set papers is acquired through API calls to Semantic Scholar using the Semantic Scholar corpus-IDs. All papers where the corpus-ID could not be acquired are discarded, leaving 26,254 papers in the lest. set for further use. To match the size of the test set with the original authors, we sampled 14,307 papers such that the proportions of all subcategories remained unchanged. Section 3.3.1 explains how we again used sampling on the test set when making additional API calls to retrieve extra information for logistic regression. The category distributions of these samples are visualized in Appendix A.1 Figure
We kept the train set size unchanged as it was unclear how the dataset sizes by 2024 were obtained, and explicit sampling could lead to significant performance cost
Greenwood et al.
3.2.2 Novel dataset: Amazon books reviews
The Amazon books reviews dataset consists of 3,000,000 reviews with each review accompanied by attributes of the corresponding book, including title, author, deseription, and year of publication. The reviews span
2 Accessible via hwtss///wwwkawtt www.leagggl.cmm.oooo www../oo//daaattttsssaae datasets/ aarohhamebbaahelee waweto/mazzoooooa wtw.hh/w//aaaooomooooeeeeeeee books-reviews
com /tt////aaatttorr/aaaee Slattaeett/Coooelll-tttetrrrrrrs University/arsii
Accessible via https::/
"The dataset is updated weekly. These numbers correspond to the time of writing this, January 11 2025.
SSince these are the ones which recommendations will take place. The train set is merely used to create embeddings for authors, described in Section 3.3.1
More information about the Semantic Scholar API be found via: https:: http//wwwwttt. www.anticsccooan....mmmmm org/ wwtp:/ooop///pppp/////ooo
EEgg unlucky sampling could lead to sampling papers of which many authors not preseut the test sct, making the
effective size of usable authors small.
Later correspondence clarified this, as discussed in Section

---

Published in Transactions on Machine Learning Research (06/2025)
from 1996 until the end of 2013 and include 212,404 unique books. Books that missed either a description the year of publication were excluded from our analysis, as the description is essential for creating the user embeddings, and the year of publication is required to split the dataset. After cleaning the data, we retained 78,571 unique books. For the training set, we considered books published before 2007. This training set was used to create embeddings for the reviewers, who represent the users in the recommendation system. It contains 30,506 books, 475,382 reviews, and is contributed to by 237,310 unique reviewers. The test set included books published from 2007 until the end of 2011, and contained 14,548 books. The test set is the set of books to be recommended. We chose these years to align the test set size with the original paper.
3.3 Experimental setup
To replicate the original study, the README files in the paper's repository were taken as a leading ref- erence, in coordination with the description in the paper itself. The code provided largely required minor adjustments, however, some files were missing, which pipeline starts with general data preparation, where the train and test set are separated and additional infor- mation about papers is gathered through API calls. Subsequently, embeddings for papers and authors, and associated similarity scores are calculated. After all preparation, logistic regression is performed to evaluate whether calculated similarity scores are a good measure of successful recommendations. Finally, we run the experiments to validate the main claims of the paper. Below we describe the detailed experimental set-up of reproducing the results the arXiv datasct (Section 3.3.1 and of our proposed extensions: (i) validating the claims on different dataset (Section 3.3.2 and (ii) examining the effect of user fairness when multiple items are recommended for each user (Section 3.3.3
reintroduced to fully run the original pipeline. This
3.3.1 Original setup: arXiv dataset
We maintained the below pipeline of the original paper with minor adjustments.
General data preparation Identically to Greenwood et al. (2024), ; we retrieved all papers of the arXiv dataset as described in Scction 3.2.1 and discarded all papers that do not possess Computer Science as a main category. We added an extra script to produce all mappings from calegory-ID to main and subcategory, as this file seemed to be missing. We moved the script to gather additional features of the test set forward; according to the original README, this script is performed later, lcading to problems later scripts, hence we moved it forward. Furthermore, we introduced a script to sample the subcategories proportionally.
Logistic regression data preparation Additional API calls to the Semantic Scholar API are made to obtain information for logistic regression regarding: all authors that contributed to any paper in the test set, all papers that have a paper from the test set in their citation, and all citations of papers in the test set. Since we could not obtain a Semantic Scholar APT key (due to a pause in issuances by high demand), we were restricted to the 1,000 API calls per second shared across all unauthenticated requests to the Semantic Scholar servers 10 This lod to many unsuccessful responses and eventually to the API server blocking our requests, as described in Appendix A.33 To mitigate this problem, we retried failed responses four more times before moving on to the next paper. Still, the number of failed responses forced us to significantly reduce the test set available for logistic regression down to 1/12th of our initial test set by sampling proportionally to the existing subcategories, visualized in Appendix A.1 Figure 6. This ended up in successfully gathering additional information for 2,188 papers, approximately 1/6.5th 5t of the number of papers in the test set of Greenwood et al. [2024].
Embeddings and similarity scores Paper embeddings are computed for both train and test sets by considering each paper's title, abstract and categories, removing stopwords, and letting scikit-learn's TF- IDF vectorizer create embeddings. Subsoquently, utility matrix (w), containing all similarity scores, is constructed in two-step process. First, for every author (that occurs in the test set.) i, the cosine similarity is calculated between each of their train set papers and a test set paper j. Say author i has papers in the train set, this would result in n distinct similarity scores. Second, to calculate Wij, the maximum out of the n similarity scores is considered. In the original code, author embeddings are created for 1,000 uniquely
9 As explained in Section our approach in line with previous version of the original paper.
10 As stated here: batps://wus htt.://wwwetticcco.....c www.amttiicccolar..orrrrrrtttece prg/pppduct/app
5

---

Published in Transactions on Machine Learning Research (06/2025)
sampled authors from the test set. Since embedding can be created for authors that do not appear in the train set, we modified the code to create embeddings for all authors present in both sets, after which we sample 1,000. The results of the original approach are shown in Appendix A.4. and the results of our approach are discussed in Section g Furthermore, we adjusted how the calculated similarity score is stored for logistic regression. In the original codebase, the similarity score is stored by iterating for each author over all items, starting from the first item entry every time. This effectively overwrote all previously stored similarity scores for previous authors. We corrected this by starting to write the similarity scores from the first entry of the corresponding author.
Logistic regression To evaluate the recommendation system, we performed logistic regression. Since the original repository did not include this code, we implemented it based on the details provided in the paper. The same sample size of 1,128 was used as the original paper. We added all recommendations per user for our logistic regression, instead of only the top-1 recommendations. The logistic regression was conducted using Python's statsmodels module between the similarity score and whether the author cited the recommended paper, the author was referenced, or the user was an author on the item. The similarity score was computed as described in the paragraph above. For robustness, performed logistic regression three times with random sceds 42, 999, and 123. We calculated the coefficient, standard error, Z-value, and P-value.
Experiments We added a script to generate the data source file for all experiments based on the README file of the original authors, since this script was missing. For each experiment, the utility matrix w is computed again analogously. The first experiment examines the difference in user-item fairness tradeoff between heterogeneous and homogeneous users. For heterogeneous users, we sampled 40 authors and 20 papers out of the entire test set. Then, for 50 values of 7 between O and 1, U is calculated and plotted. In total 10 curves are calculated, after which the mean and (two) standard deviations are plotted. For homogeneous users, all authors are first grouped into 25 clusters. For this clustering, the original code casts the sparse into dense embeddings, resulting in significantly more memory usage. We ensured the sparse roprosentations were maintained to significantly save time and memory without a change in result. The process of creating the remainder of the graph is identical to that of the helerogeneous graph, except that for each curve 40 authors of the same cluster are sampled. The second experiment examines the difference in user-item fairness tradcoff between users for whom preference data is present and cold start users. The latter category is constructed by treating 10% of the sampled users cold start users by removing their embedding. Then again, for 50 values of y between 0 and 1, U is calculated and plotted.
3.3.2 Extension: Amazon books reviews dataset
User embeddings are defined by all book embeddings to which reviewer made a review prior to 2007. In the original paper, user embeddings were constructed using the title, abstract, and category of each item. To align with this, we generated user embeddings based on each book's title, description, and category, using seikit-learn's 's TF-IDF vectorizer in the same manner as the original implementation. To evaluate the rec- ommendation system, we performed logistic regression following the methodology described in section 3.3.1] adjusted for the Amazon books reviews dataset. The logistic regression was conducted between the similarity score and whether the user actually left review for the recommended book. The similarity score reflects the similarity between the books reviewed by the user before 2007 (i.e. user embedding), and the possible recommended books from the test set.
For the experiments, we sampled 1,000 users, consistent with the original paper. We held all experimen- tal parameters, for conducting the experiments, constant to ensure comparability with the original paper. Specifically, we tested 50 values of 7 between and 1, clustered the 1,000 users into 25 clusters using the k-means algorithm for the homogeneous population for the first experiment, and again treated 10% of the population to be misestimated for the second experiment.
3.3.3 Extension: Multiple recommendations per user
We extended the original study on the arXiv datasot by recommending more than one item to each user to better simulate a real scenario. In real-world recommendation systems (e.g. streaming, e-commerce platforms, etc.), users are often provided with multiple recommendations instead of single item.
6

---

Published in Transactions on Machine Learning Research (06/2025)
We noticed that the optimal policy P which maximizes the minimal user utility (Equation under the constraint Σj Pij = 1, leads to the following without item fairness: for each user i, the item with index h twitth wi(j=)) the highest utility score for user i- is allocated Pi(j=h) + 1, while all other items have Piligh) + 0. In other words, for user i the distribution over items to recommend p; is a one-hot encoding. This makes sense as we aim lo assign the highest recommendation weights the items with the greatest utility to achieve the maximum expected user utility. When item fairness constraints are imposed, Pi not stricily a one-hot encoded vector anymore, but rather becomes somewhat. more distributed across multiple items. The element-wise multiplication of Pr and W/; and subsequent summing computes the expected user utility (Equation 2 Intuitively, this can be interpreted as sampling each item j with a probability Pij-
When generalizing to recommending multiple (k) items in the optimal policy P, modifying the constraint to Ej Ply= = k, lets the i distribution become a k-hot encoding since the constraint p E [o. 1] stays unchanged. This leads to the following without item fairness: for each user i, the set of item indices H -a set size k, with the & highest utility scores Wijelr) for user i-is allocated Pigjer) + 1, while all other items have P((ie)) - As previously, when item fairness constrraints are imposed, Pi is nol strielly k-hot encoded vector anymore, but rather becomes slightly more distributed across multiple items. The element wise multiplication of pt and wi: and subsequent summing now leads to the expected user utility of recommending h items (Equation 5 This can still be interpreted as sampling each item j with a probability Pij. leading to k recommended items in expectation. Note that, when item fairness constraints are imposed, the top-k items with the greatest utility are not necessarily chosen.
In this experiment, & is set to three and five, to analyze the impact on U of recommending a set of three, and five items respectively. As recommending multiple items automatically increases the expected user utility, the normalization of the user utility should be adjusted to maintain equivalent range for the expected user utilities across different & values. Therefore, we adjusted the normalization for user i's utility in Equation 2 to:
Pig Wiy where H is the set of indices with the & highest utility scores for user w.(p,w)= = Ejen wiy"
Esprs
This ment ensures that user i's expected user utility is normalized by the sum of their & highest utility scores. It generalizes the idea of normalizing by their highest utility score when recommending only a single item. To clarify, the only differences from the original setup in section 3.3.1 are setting to values other than 1 and modilying the normalization.
3.4 Computational requirements
For the experiments described in Section 3.3 we used a node containing nine cores of the Intel Xeon Platinum 8360Y, an NVIDIA A100 GPU, and 60GB of DRAM. In total, all computing time took 99 hours. Wo calculated the CO2 emission to be approximately 12 kg CO.
4 Results
In this section, we first briefly review the original results of the paper. Secondly, we present our findings from the reproducibility study on the arXiv dataset using the original setup. Thirdly and fourthly, we discuss the results of the two extensions. Since no seed is set when sampling authors and papers for the curves, the curves vary slightly every run due to randomization.
4.1 Original results
Greenwood et al. 2024) validated their arXiv recommendation system by performing a logistic regression on the similarity score and whether the paper was actually cited by the author. Table shows their results with a highly significant positive coefficient, confirming reliable recommendation system.
HTaaing the average Carbon Efficiency of The Netherlands from Moro & Lonza 2018 and the calculation tool of Lacoste et. al. (2019)

---

Published in Transactions on Machine Learning Research (06/2025)
The results of their experiments presented in Figure Figure la shows that item fairness imposes higher cost on homogeneous users compared to diverse users, consistent with empirical claim 1. This cost remains low except when reaches 1, matching strict item fairness. Figure Ib demonstrates a substantial cost of miscstimation. This cost is so high that item fairness does not have a negative impact on user fairness, corresponding to empirical claim
Coefficient
Standard Error
Z-value
P-value
Similarity score
12.4100
0.058
212.178
0.000
Table 1: Logistic regression results validate the recommendation system with the arXiv dataset
Green-
wood al. 2024).
extmmation
1.6
Random users Homogenccus uSCrs
00
0.5 1.0 94
0.6
Normalized utility guarenteed. y
Memalooed for
(a) Homogeneous versus diverse users the arXiv dataset.
(b) With and without misestimation on the arXiv dataset.
Figure The empirical findings of Greenwood et al. 2024 between user fairness (normalized minimum user utility U* Eq. and item fairness constraint (%, Eq.
4.2 Original setup: arXiv dataset
This section presents the results of our reproducibility study on the arXiv subset. Due to API rate limits discussed in Section 3.3.1] we ran the logistic regression for the arXiv dataset on the subset of 2,188 papers instead of 14,307 papers. To ensure that using subset of the test set for the experiments would not impact the results, conducted the experiments
both the full test set and subset of it. Since results the full test set (Appendix A.5 Figure g and on the subset of the test set (Figure 2 show similar trend and no significant differences, we decided to present results on the subset of 2,188 papers for consistency with the dataset on which logistic regression is performed.
Table shows the results of the logistic regression. The coefficient of 6.21 suggests a strong positive rela- tionship between the similarity score and whether or not the author cites the paper in the future. This led to conclude that recommending items based on similarity score is valid. The coefficient is, however, lower than what is reported in the original paper (see Table which might be caused by the faulty storage of similarity scores as described in Section 3.3.1
Coefficient
Standard Error
Z-value
P-yalue
Similarity
6.2092 1 0.3429
0.1650 + 0.0093 37.8554 4.0966
0.000
Table 2: Logistic regression results using three random seeds to validate the recommendation system with the arXiv dataset.
Figure 2a demonstrates the comparison between heterogeneous and homogeneous users. The resulting curves are similar to those observed in the original paper. For moderate item fairness constraints (i.e. 0 < 7 / 0.9), we observe a slight negative effect on user fairness by increasing item fairness. As y approaches 1, the tradeoff becomes significantly steeper. Homogencous users exhibit higher price of fairness, consistent with empirical claim in the original study.
8

---

Published in Transactions on Machine Learning Research (06/2025)
presents the results of the comparison between users with and without miscstimation (i.e. users without and with prior known preferences). The curves of this experiment closely resemble those of the original paper. A similar drop in user fairness is present as for random users in Figure 2a while user fairness stays consistent, with even a slight increase, for cold start users when increasing 7. This shows, like stated in empirical claim the original paper, that the cost. of misestimation already so high that it does not seem to be worsened with item fairness constraints as we analyzed in Section 3.1
Figure 2b
32
Pasta olean Wnvernaled sericiity Aunene..
(a) Homogeneous versus diverse dataset.
the arXiv
(b) With and without misestimation on the arXiv dataset.
Figure 2: Empirical findings between the user fairness (normalized minimum user utility U", Eq. and item fairness constraint (y, Eq. by sampling from 1,000 authors from a subset of 2,188 papers.
4.3 Extension: Amazon books reviews dataset
This section presents the results of the original experiments conducted the Amazon books reviews datasel. Table 3 presents the results of the logistic regression. The coefficient. of 15.05 suggests strong positive relationship between the similarity score and whether or not a reviewer leaves a review for a book in our recommendation selection. Thus, we concluded our recommendation system to be valid.
Coefficient
Standard Error
Z-value
P-value
Similarity score
15.05 + 0.78
0.365 = 0.012
41.23 +: 1.04
0.000
Table 3:
Logistic regression results using (hree random seeds to validate the recommendation system with Amazon books reviews dataset.
Ua un mill aiduuee
47-
05
(a) Homogeneous versus diverse users on the Amazon books reviews dataset.
(b) With and without misestimation on the Amazon books reviews dataset.
Figure 3: Empirical findings between the user fairness (normalized minimum user utility) and item fairness (normalized minimum item utility), where y is the item fairness constraint, using the Amazon books reviews dataset.

---

Published in Transactions on Machine Learning Research (06/2025)
Figure 3a demonstrates the comparison between heterogeneous and homogencous users. The resulting curves are similar to those observed in the original paper. Homogeneous users exhibit. higher price of fairness for the novel dataset, which aligns with empirical claim I reported in the original paper. Notably, the two curves are closer to each other compared to the original study. possible explanation for this difference could be the more homogeneous nature of the Amazon books reviews datasel. Further details about this analysis can be found Appendix A.6
Figure 3b shows the comparison between users with and without misestimation. The curves of this experi- ment closely resemble those of the original paper, further supporting the robustness of the proposed empirical claim 2 that item constraints do not increase the cost of miscstimation.
4.4 Extension: Multiple recommendations per user
Homogeneous versus diverse users for K- 3.
With and without misestimation for k 3.
32 08
onlly
(c) Homogeneous versus diverse users for k == 5.
(d) With and without misestimation for k = 5.
Figure 4: Empirical findings between the user fairness (normalized minimum user utility U Eq. 1 and item fairness constraint (7, Eq. D by sampling from 1,000 authors from a subset of 2,188 papers. Respectively, k ==3 and & = 5 represent the recommendation of three and five items.
This section displays the experimental results of recommending multiple items, conducted on the arXiv subset of 2,188 papers. Figures 4a and 4c show the differences between homogencous and heterogeneous users when recommending three and five items respectively. The observed curves have a similar trend to those in the original paper, where homogeneous users encounter a higher cost of item fairness than heterogencous users, and the curves become substantially steeper when approaches 1. So, this is consistent with proposed empirical claim 1. However, a notable observation that both curves become less steep as k increases. Intuitively, this makes sense as the negative impact of an individual recommendation mitigated by considering a larger variety of items. Due this enlarged variety in recommendations, the gap bet recommendation differences for homogeneous and heterogeneous users decreases.
4b and 4d illustrate the disparity between miscstimated and truly estimated users when recommend- ing three and five items. As expected, the line of misestimatee users shifts upwards for a higher k since misestimations are less problematic when more items are recommended. In this context, when recommend- ing more arXiv preprints, the likelihood of the user encountering a relevant paper increases, which leads higher (normalized minimun user) utility. Furthermore, we observe that item constraints do not increase the cost of misestimation, which aligns with empirical claim in the original paper.
Figures
10

---

Published in Transactions on Machine Learning Research (06/2025)
5 Discussion and conclusion
We reproduced the empirical framework described the original paper by Greenwood et al. (2024). First, we performed logistic regression on the subset of the original dataset, confirming that similarity scores provide a valid estimation for future citations which act a proxy for user preferences. We then performed the same empirical experiments on both the original and novel Amazon books review dataset to prove generalizability Socondly, we extended the experiments to examine the effect of incrcasing the number of recommended items per user on user fairness with item fairness constraints.
The results of both datasets validate empirical claim 1: (i) diverse user preferences decrease the price of item fairness and (ii) the price of item fairness becomes substantial when y approaches 1. We similarly validate empirical claim 2: the price of misestimation is high, but imposing item fairness constrrints does not. increase this cost. Additionally, increasing the expected number of recommended items does not influence these two claims, however, the cost of imposing item fairness does seem to lessen.
The revalidation of empérical claim 1 further validates the findings of the original paper. These observations are in line with points discussed in carlier research on item fairness in recommendation systems. Zhao et al. (2025), Koutsopoulos & Halkidi (2018), and Castellini et al. (2023) investigated the relationship between user fairness, item fairness, and diversity in recommendation systems, and similarly observed that increasing user diversity leads to improved item fairness. [Zhao et al. (025 attributed this to the fact that, in this manner, generally unpopular items have naturally higher chance of being recommended. This corresponds to our observation of decreased price of item fairness when user diversity is higher. The importance of this diversity is further elaborated on by Kunaver & Požrl (017)) however, it is considered beyond the scope to discuss it in this reproducibility study.
The revalidation of empérical claim again shows evidence that the theoretical intuition of further worsening user fairness when increasing item fairness for cold start users does not seem to hold, instead user fairness is stagnantly low with slight increase near strict item fairness (observed by and the original paper). We suspect that, due to some of the naturally present item variability in recommendations for cold start users, it is of no value to impose minimum variability up to a certain level (by item fairness). When increasing minimum variability beyond this level, while recommendations are still not personalized, the increased diversity raises the odds of providing better matches. Since (normalized) minimum utility measures the most underserved user in the cold start group, near striet fairness uplifts the bottom the one user who had terrible matches before now has a higher probability to get something relovant, nudging the minimum up. However, as the graph drops at striet item fairness, there seems to be an optimum; stricl levels will also force the recommendations of the most unpopular items to some users, dropping user fairness again. It would be valuable to further investigate this hypothosis in future work. Future work should investigate the relation between the amount of variability in item recommendations and (normalized minimum) user utility for cold start user. Additionally, we suggest. running the experiments more than 10 times to obtain statistically significant results for a better-founded conclusion.
Increasing the number of recommended items from one to three and five showed no influence on these two claims. However, increasing k does seem to lessen the cost of imposing item fairness. The recommendation of one, three or five papers, made up substantial part of the entire recommendation pool of 20 papers. Future work could investigate whether this effect weakens on a larger sample size, i.e. when a smaller portion of the total items is recommended. This would further decrease the high standard deviation we observed due to our small sampling.
The revalidation of both claims on the Amazon books review datasel demonstrated the generalizability beyond the domain of academic paper recommendations a more widely used domain. The item-user fairness curves between heterogencous and homogencous groups were slightly closer to cach other than in the original work, likely caused by the more homogeneous nature of users within the dataset. Future research could explore datasets with more heterogeneous random users, as well as those from other widely adopted
Larger sampling was already implemented in newer version of the original paper, but due time and computational limitations, we were unable to do this ourselves, as discussed Section 0
11

---

Published in Transactions on Machine Learning Research (06/2025)
domains (e.g. social media, music streaming, etc.). Since recommendation systems are broadly applied, enhancing robustness across different domains is a valuable direction for future research.
While we have addressed the limitation of recommending only a single item, various other constraints of the original work still remain. These include the need to theoretically explore the user-item tradeoffs to other definitions of fairness, the challenge of platform hesitation to adopt an egalitarian approach, and the limited number of user types in the theoretical proofs (in Appendices D and E of the original paper). Addressing these limitations is still relevant area for future research, and could enhance the framework's applicability to real- world scenarios.
Overall, in line with the original findings, our results suggest that recommendation system designers should keep in mind that it is beneficial to have a diverse user population and that item fairness constraints should be imposed on the entire population rather than subgroups.
6 Final remarks
How we deviated from the original authors. Due to time constraints, limited computing resources and the computational complexity of optimizing the convex problem, we were confined to reproduce the older original NeurIPS version of the paper, accessible via ltttps:/neeurip..
This version deviates from the most recent version in three aspects. Firstly, it draws smaller samples of papers (20 instead of 200) and authors (40 instead of 500) for cach curve. Secondly, it first samples 1,000 papers to sample each curve from. Both the old and new versions of the paper share very similar graphs and identical conclusions, Thirdly, for creating homogeneous authors, it creates more clusters (25 instead of 10). These choices resulted in significantly less memory usage and faster computing times, with very similar plots and an identical conclusion section. Lastly, as described in Section 3.3.1 we ensured the clustering for creating homogencous groups was performed on sparse embeddings.
What was easy. The original code was easy to find and publicly available. Together with well-rrgaizzed repository, it allowed us to reproduce their results. Furthermore, the original research already extensively discussed several extensions in their appendix, answering quite some of our questions and therefore enabling efficient progress. The foundation of enabling multiple-item recommendations was already present in the original code, enabling easier integration of this extension.
What was difficult. Loading the original dataset from Kaggle resulted in more data than described by the original authors, which made it unclear whether the originally reported number of papers in the train and test set was obtained. Also, some adjustments in code and execution order were necessary to obtain working pipeline, which was challenging since only parts of the code contained comments. Moreover, to obtain additional information for each paper for logistic regression, we needed data from the Semantic Scholar API. Lacking an API key, we were unable to make all the requests needed lo get the data for every paper. Besides, the original paper did not provide code or a clear explanation to perform the logistic regression, hence we had to make assumptions solely based on the paper. Furthermore, extending their theoretical framework for multiple-item recommendation was challenging. because their framework did not explicitly incorporate that only one item is recommended, only implicitly in both their framework and code.
Communication with original authors. During this replication study, we contacted the authors for the following two points: Since the linked GitHub repository did not provide a file for the logistic regression, we asked for clarification on their implementation. In their answer, they provided the code for the logistic regression, which led us to conclude that our implementation slightly differed. [reenwood et al. (2024) ran logistic regression three times: depending on whether a citation was present, whether the author was referenced, and whether the user was an author on the paper. The first metric was discussed in their paper, Our implementation assumed that if either of those were true, it was good recommendation, and used this as a variable for the logistic regression. Secondly, our train and test set sizes differed significantly from their train and test set sizes. They clarified that they have used the 'update date', while we made use of the ppullshedd date'. Since many papers have been updated recently, Greenwood et al. 2024h have significantly fewer papers in their train and test sets. Due to time constraints, we were not able to change both of these deviations.
12

---

Published in Transactions on Machine Learning Research (06/2025)
References
R. Burke, Nasim Sonboli, and Aldo Ordonez- Gauger. Balanced neighborhoods for multi-sided fairness in rocommendation. In FAT, 2018. URL httpss///ap...omannicccchoooll org/CorpusID:46611
Jacopo Castellini, Amelia Fletcher, Peter L.. Ormosi, and Rahul Savani. Supplier competition on subscription-
based platforms in the presence of recommender systems. SSRN preprint, (4428125), April 2023, doi: 10.2139 httss/ss....8881 URL https://papers. swreeerrsssss......... ssrn. htt..//aaataaaatt42222222
Sophie Greenwood Sudalakshmee Chiniah, and Nikhil Garg. User-item fairness tradeoffs recommen- dations. The Tiihtt- Annual Conference on Neural Information Processing Systems, 2024. URL
Tordanis Koutsopoulos and Maria Halkidi. Efficient and fair item coverage in recommender systems. Tn 2018 IEEE 16th Intl Conf on Dependable, Autonomic and Secure Computing, 16th Intl Conf on Pervasive Intelligence and Computing, 4th Intl Conf Big Data Intelligence and Computing and Cyber Science and Technology Congress DASC/PiCom/ httpS///iiC////aaaaoonmma Dwt/DDaCooC/eeerrreeeeeeeaaae /CyberSciTech), pp. 912-918, 2018. doi: 10.1109 /DA/C/ 000.00000......221000000009999
Matevž Kunaver and Tomaž Poärl. Diversity in recommender systems survey. Showledge-Basse Systems, 123:154- 162, 2017. ISSN 0950-7051. doi: http://d..o.rr./.....//////oo///
URL https:
Alexandre Lacoste, Alexandra Luceioni, Victor Schmidt,, and Thomas Dandres. Quantifying the carbon emissions of machine learning. arXiv preprint arXie:1910.09700 2019.
Linyuan Lü, Matúš Medo, Chi Ho Young, Yi-Cheng Zhang, Zi-Ke Zhang, and Tao Zhou. Rocommender systems. Physics Reports, 519(1):1 49, 2012. ISSN 0370-1573. doi: http:///doo.orrr//...0////ppss/p 0012.02.006. URL http:///www..aaaccccccc.o.oo///a ommender Systems.
Rec-
Rishabh Mehrotra, James Mclnerney, Hugues Bouchard, Mounia Lalmas, and Fernando Diaz. Towards fair marketplace: Counterfactual evaluation of the trade-off between relevance, fairness & satisfaction in recommendation systems. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management, CIKM '18, pp. 2243 2251, New York, NY, USA, 2018. Association for Computing Machinery. ISBN 9781450360142. doi: 10.1145//2869226.3277728 URL https://doi... 10. 00.1145/3269206.3222 3272027
Alberto Moro and Laura Lonza. Electricity carbon intensity in curopcan member states: Impacts on ghg emissions of electric vehicles. Transportation Research Part D: Trunsport and Environment, 64:5 14, 2018. ISSN 1361-9209. doi: http::/ddo..rrg/10100//// http:////...001......11111111111 (ttcce/cceerrttceeee///////11110
URL https: //www. scieeccdiirer..t The contribution of electric vchicles environmental challenges in transport. WCTRS conference in summer.
Lequn Wang and Thorsten Joachims. User fairness, item fairness, and diversity for rankings in two-sided markets. In Proceedings of the 2021 ACM SIGIR International Conference on Theory of Information Retrieval, ICTIR '21, pp. 23-41, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450386111 11. doi: 10. 00.1455/44715558347222
URL http:::://ooi.rr//1/// 11t.///0011115//71155155 3472260
Loqun Wang, Yiwci Bai, Wen Sun, and Thorsten Joachims. Fairness of exposure in stochastic bandits. International Conference on Machine Learning, 2021. URL https://api semantcccchoaa... org/ CorpusID:23211000
Yifan Wang, Weizhi Ma, Min Zhang, Yiqun Liu, and Shaoping Ma. survey on the fairness of recommender systems. ACM Trans. Inf. Syst., 41(3), February 2023. ISSN 1046-8188. doi: 10.1145/3544333, URL
:tttp:://ai..org/10.1111
13

---

Published in Transactions on Machine Learning Research (06/2025)
Yuying Zhao, Yu Wang, Yunchao Liu, Xueqi Cheng, Charu C. Aggarwal, and Tyler Derr. Fairness and diversity in recommender systems: A survey. ACM Transactions on Intelligent Systems and Technology. 16(1):1 28, February 2025. ISSN 2157-6904, 2157-6912. 1141/1664//39
doi: 10.1145/3664922
URL https://dl.ace acm.
14

---

Published in Transactions on Machine Learning Research (06/2025)
A Appendix
A.1 arXiv dataset distribution of this study
This section provides the details regarding the categories distribution of the arXiv dataset used in this reproducibility study. Figure 5 shows all papers published before and in 2020 in the arXiv dataset at the time of writing. Out of all the papers published in 2020, we sampled 14,307 papers to align with the test set size of Greenwood et al. (2024), the distribution of which is shown in Figure 6a Due to the failed API calls required for logistic regression -as described Section 3.3.1 our final test sct contained 2,188 papers published 2020, of which the distribution shown Figure 6b
35000
Number of Publications
30000
20000
15000
10000
5000
Companer appaation Saaee Coddiaie samun cooiiiig
Compote tose Naten
gee Retveni
De
corli Lage with
Computer Science Sub categaries
aa) Distribution of papers by subcategory of the 255,138 papers in the train set (L.e. papers published before 2020).
20000
Number Publications
15000
5000
0
wote the Coppent
crosage segertii Sestamt
Computer Science Sub-categories
(b) Distribution of papers by subcategory of the 65,948 papers published 2020.
Figure 5: Distribution of papers by subcategory per dataset. A paper possessing multiple subcategories is counted multiple times.
15

---

Published in Transactions on Machine Learning Research (06/2025)
Distribution of research paper publications in the test dataset over time
5000
4000
Number Publicatioons
3000
2000
1000
Comeen saee and anpeettii
Computer Science Sub sabb-attegries
(a) Distribution of papers by subcategory the 14,307 sampled test set papers. These papers were used for experi- ments.
700
Number of Publicattons
600
500
300
200
100
sape an
Computer Science Sub-categories
(b) Distribution of papers by subcategory of the 2,188 sampled test set papers for which Semantic Scholar API calls succeeded. These papers were used for experiments and for logistic regression.
Figure 6: Distribution of papers by subcategory per sampled dataset. The x-axis order of categories is equivalent to Figure 55 for comparability. A paper possessing multiple subcategories is counted multiple times.
16

---

Published in Transactions on Machine Learning Research (06/2025)
A.2
arXiv dataset distribution of original paper
Figure 7 provides the details regarding the categories distribution of the arXiv dataset used by Greenwood et al. 2024
Mitee rutson Raregrttae
Lanila halemas Tawey
and wat aereen
compees
Acanetes
Seery
computer bcanct
Distribution of papers by subcategory in train set (i.e. papers published before 2020).
Computts
fane Soumeityy
Sapeen and Dteme therry
brenttaeeyy Comeense
Largen and thaeems Thary sangeges
Engrases
Colenn dengaiinn
Comcunt Sub sa-catgerin
(b) Distribution of papers by subcategory in the test set (L.e. papers published in 2020).
Figure 7: Distribution of papers by subcategory per dataset Greenwood et al. 2024
17

---

Published in Transactions on Machine Learning Research (06/2025)
A.3 API rate limit error
When making API request when more than 1,000 other unauthenticated requests were performed, the following error was given:
429: Too Many Requests. Please wait and try again or apply for key for higher rate limits.
After a large number of requests (sometimes in the hundreds, sometimes in the few thousands), occasionally the server kicked all requests for dozens of minutes and the Python script was terminated. This is likely caused by the server blocking our IP as common rate-limiting strategy.
requests. exceptionnn
ConnectionError: oxtoppiioos.ommmttiiiorrroo HITPSConnectionPol) (host= appi. e semattcsccooaa. 'org port=443): Max retries exceeded with url: *URI.* * (Caused by NameResolutionErrro wwt..o...........mmeeeee........ HrTPSConnectioo object 0x7fd8ab3a7f20>: Failed to resolve
("<urllib3. wwwencciinnccoooo repanicaanticcccllaa..ooo org' ([Errno -3] Temporary failure in name resolution)")) "))
A.4 Test set sampling method in provided code
Figure 8 shows plots produced by the sampling method in the originally provided code. This method differed from what was described in the paper, by how authors were sampled to generale the curves. The original code first sampled 1,000 authors randomly from the test set, after which embeddings were made for authors that also appeared in the training set. This resulted a significantly smaller pool of authors (659 authors) from which 40 authors were sampled to generate each curve in the plots. Contrary, the results in the body of the paper first created all author embeddings, after which 1,000 authors were sampled.
Rendom users
10
Norellled oaer utilt
0.8
0.0
Min usar under mis estimation
0.0
0.2
0.0
Fraction normalized utility quaranteed
Fraction normalized item quaranteed, y
(a) Homogeneous diverse the original
(b) With and without misestimation the original dataset.
dataset.
Figure Empirical findings between the user fairness (normalized minimum user utility U*, Eq. and item fairness constraint (y, Eq. I by sampling from 659 authors from subsct of 2,188 papers.
18

---

Published in Transactions on Machine Learning Research (06/2025)
A.5
Test set sample of 14.307
Figure g shows the empirical results of the full test set. The smaller sample in the paper aligns with the dataset on which logistic regression was performed. These two sets differed due to the difficulties with API calls, as explained Scction 3.3.1
Random Homogenecus users
10-
10-
Nonmellad) user utlly
0.8
Min user under miseestimaiion
Nomalized
0.4
0.4
0.0
0.2
0.4
0..6
1.0
0.0
0.2
0.4
0.6
0.6
1.0
Fraction nommllzzdd utility guarantaed
Fraction mm normalized itom utility guaranteod, y
(a) Homogeneous versus diverse users on the arXiv
(b) With and without misestimation on the arXiv dataset.
datasel.
Figure 9: Empirical findings between the user fairness (normalized minimum user utility U* Eq. and item fairness constraint (7, Eq. 1h by sampling from 1,000 authors from a subset of 14,307 papers.
19

---

Published in Transactions on Machine Learning Research (06/2025)
A.6 Amazon books reviews homogeneity
For the Amazon books reviews datasel, homogeneous users seemed less negalively impacted by item fairness than suggested by the original arXiv dataset. A possible explanation for this difference is that homogeneous users in the Amazon books reviews datasct may be clustered morc heterogencously. To test this, we computed the silhouette scores for clusters in both datasets, a metric measuring within-chuster similarity relative to other clusters, with lower scores indicating greater heterogeneity. However, as Table f shows, both datasets yield similar scores, albeit hinting towards slightly more heterogeneous distribution of the arXiv dataset.
Silhouette score
ar preprints dataset
0.3401 + 0.0038 0.3447 + 0.0096
Amazon books reviews dataset
Table 4: Silhouette scores for three random seeds (42, 999, 123).
To further investigate why the two curves are closer to each other, plotted the authors and the clusters to which they belong. Figure 10 shows greater homogencity of authors in the Amazon books reviews dataset; the user embeddings are gathered in a small interval of values, compared to the arXiv dataset with more spread. This leads to a higher probability of more similar users being randomly sampled as heterogeneous users in the Amazon dataset compared to the arXiv dataset. The more homogeneous nature the random users in the Amazon dataset can lead to more similar user utility between the two curves. This can explain the two curves being closer to each other compared to the original study the arXiv dataset.
PCA Author Embeddings and Mears Clustering
PCA of Author tmbeddings and KMeans Clussering
01
0.59
00
0.00
6.3
022
-3.05
0d5
0.5
035
PCA Companert
Component
(a) arxiv dataset.
(b) Amazon books reviews dataset. We left out approx- imately 5 data points that were outliers.
Figure 10: Scatterplots for the test set of both datasets depicting 25 clusters and two PCA components.
20

---

Published in Transactions on Machine Learning Research (06/2025)
A.7 Amazon books reviews dataset
This section provides details regarding the data distribution of the Amazon books reviews dataset. Figure and Figure 12 show the distribution of books and reviews, respectively, based on their publication date.
11
Number of Bocks Published Per Year
Number o' Books
1000
0
3850
3910
1800
2000
201
200
200
Bar
Figure 11: Number of books published per year in the Amazon books reviews dataset (48 books between 1802 and 1950 were dropped for visualization).
Reviews per year
300000
Number of reviews
200000
150000
100000
50000
1869
1995
1996
1897
1998
1999
2000
2001
2002
2003
2005
2006
2007
2008
2009
2010
2011
2013
Yar
Figure 12: Number of reviews per year in the Amazon books reviews datasct.
21


Reviews:
[{"review_id": "GPs0pu2XMl", "reviewer": "Reviewer_egGN", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n- The authors provide a comprehensive description of their process of reproducing Greenwood et al. (2024), including clear descriptions and justifications for necessary changes.\n- The authors provide results on a new, distinct dataset from a different recommendation setting, which shows not only that the original analysis is replicable but also that the original results replicate _beyond_ the original setting.\n- The paper explores several hypotheses to explain the observation that the tradeoff curves for homogeneous users and for randomly sampled users are closer together in the new Amazon dataset compared with the arXiv dataset.\n\n## Weaknesses\n- The paper does not provide details for how the embeddings were generated for the new Amazon dataset.\n- In Section 3.3.3, which discusses the generalization to top-$k$ recommendations, the authors claim that the recommendation policy which solves Equation 1 (optimizes user fairness subject to item fairness) converges to a one-hot vector, and use this claim to imply that replacing the constraint $\\sum_j \\rho_{ij} = 1$ with $\\sum_j \\rho_{ij} = k$ results in a $k$-hot recommendation policy, so that this new optimization problem finds a recommendation set of $k$ items. While this is true for when there are no item fairness constraints, the theoretical results of Greenwood et al. include a closed-form solution to Equation 1 when item constraints are maximal, and this solution is not necessarily one-hot -- rather, this recommendation policy defines a distribution from which a single item is sampled. I suspect that using the constraint $\\sum_j \\rho_{ij} = k$ will not produce a $k$-hot policy. This policy could be used to construct a recommendation set by including item $j$ in the set with probability $\\rho_{ij}$, but this set would have *expected* size $k$ rather than deterministically have size $k$.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper is a replication study of the paper \"User-item fairness tradeoffs in recommendation\" by Greenwood et al. (2024). This paper reproduces the empirical results of Greenwood et al. (2024) which showed in a recommender system for arXiv preprints that 1) the tradeoff between recommendation user fairness and item fairness is lessened in populations of users with more heterogeneous preferences, and 2) the cost to user utility of mis-estimating user preferences is high but not increased by imposing item fairness constraints. The paper also validates these claims by extending the empirical analysis of Greenwood et al. to a different dataset of Amazon book reviews, and shows that the same results hold. Finally, the authors note that Greenwood et al. only analyzed the setting of providing a *single* recommended item, and extend the empirical analysis to recommendation sets with $k > 1$ recommended items, observing similar patterns but a flatter user-item fairness tradeoff.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "## Critical changes\n\n1. It would be helpful if the authors could provide further implementation details on the Amazon book recommender.\n2. I propose that the authors should either (a) justify why the generalized Equation 1 will produce one/$k$-hot policies for all values of $\\gamma$, or (b) reframe this section to clarify that the recommendation policy is not produce a recommendation set but rather a distribution from which a set (of expected size $k$) can be sampled, and argue that this will resemble the tradeoffs that would be seen in top-$k$ recommendation.\n\n## Additional minor changes (not critical)\n\n1. At the bottom of page 1 the authors state that \"the cost (i.e. decline in user utility) of item fairness is low\" and that Greenwood et al. \"found the cost of item fairness to be low for users with diverse preferences\". In the context of Greenwood et al., the cost is the decline in the user *fairness* (that is, minimum user utility), and the cost of item fairness is low for *populations of users* with diverse preferences, that is, when users do not agree with each other on which items are \"good\" and \"bad\" (as opposed to individual users' preferred items being a diverse set of items). I would recommend the authors clarify the phrasing to reduce ambiguity.\n2. Section 3.1 states that Greenwood et al. \"propose a recommendation system that aims to balance user fairness, item fairness, and overall recommendation quality\"; however, Greenwood et al. do not propose a recommendation *system* but propose a *framework* for studying the fairness tradeoff, and this framework does not take into account the overall recommendation quality.\n3. What is the distribution of subcategories in the test set papers the authors were able to obtain from the Semantic Scholar API, and how does this differ from the distribution of the authors' full test set?\n4. The authors state that \"in the original codebase, the similarity score is stored by overwriting all previously stored similarity scores for each author. We corrected this by writing the similarity score to the data entries which belong to the author it relates to\". I'm not sure I understand the issue and amendment here -- it would be helpful if the authors could provide further explanation.", "broader_impact_concerns": "n/a", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "N8o8ycsb6I", "reviewer": "Reviewer_XJP2", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The authors successfully replicated the experiments from Greenwood et al. (2024), although they encountered challenges due to time constraints.\n- The paper is well-structured and the reasoning is presented clearly.\n\nMajor Weaknesses:\n- Lack of novel insights beyond the confirmation of the original findings (for example, more generalizable insights regarding failure modes, unexpected behaviors, and limitations of the original work). While validating the empirical results of the original paper is important, identifying its limitations and drawbacks is also valuable. \n\n- Only a single new dataset (Amazon Books Reviews) is explored, and the random users in this dataset are inherently more homogeneous. This weakens the conclusion on robustness and is also linked to the weakness mentioned above, which fails to provide more generalizable insights. The original paper uses a narrow domain to empirically validate their framework, which is not representative of other recommendation settings like e-commerce or social media. Recommendation systems in practice have very broad application scenarios. This new dataset’s structure and task are very similar to the arXiv one. Why not try other widely adopted benchmark datasets in the area of recommendation systems, such as MovieLens, Last.fm.....?\n\n- In the original paper, Greenwood et al. mentioned that the framework is restricted to a small number of user types and assumes symmetric utilities. While the authors extend this to alternative fairness definitions (section 3.3.3) and multiple recommendation per user (section 4.4), the scope still remains narrow. A broader exploration of user types and utility models should be integrated to enhance the generalizability of the findings and better align with real-world application scenarios.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper focuses on replicating and verifying the empirical findings of the proposed method by Greenwood et al. (2024) and the key contributions including successfully replicating the main results and extending the experiments with a new Amazon Books Review dataset to further validate the original paper's results.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "Please refer to the Weaknesses part.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "750hBbbksg", "reviewer": "Reviewer_Bv8X", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Assessing the impact of AI technologies that we commonly see deployed in the wild, such as recommender systems, is always an interesting research direction. Given their ubiquitous presence in almost every online platform or service, their (possibly unintended) effects are a critical aspect that regulators and policy makers need to take into account. However, the paper provides a poor advancement in our understanding of these aspects: the main focus in on reproducing a setting that has already been proposed and investigated by another existing work (with little modifications to their whole pipeline), basically only corroborating their findings. The second, novel, investigated dataset adds little to these findings, given that all the results go (as expected indeed) in the same direction. The most interesting part is that extending the results to multiple recommendations for each user, which is indeed not present in the original work used as a basis. However, also these results are a bit expected, and also similar works, such as [[Castellini et al., 2023]](https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4428125), [[Fletcher et al., 2023]](https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4319311) and [[Calvano et al., 2023]](https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4448010), already proposed and investigated similar issues and properties. I am unsure about how the current work is adding value to the already existing literature in its current form.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper reproduces the empirical analysis in [[Greenwoord et al., 2024]](https://arxiv.org/abs/2412.04466) on the trade-off between recommendations quality and fairness, and extends it to both a new dataset and to the case of multiple recommended items for each user. Their empirical results strengthen the empirical findings already proposed in the original paper.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Q1:** The paper is not extremely clear when it is exposing the entire pipeline for the arXiv dataset experiment. Given that this is the core result of the work (as it is the reproduction of the original results), I would suggest to make the text clearer, especially on the parts describing embedding preparation, their scoring and the following logistic regression process.\n\n**Q2:** The most interesting aspect of [[Greenwood et al., 2024]](https://arxiv.org/abs/2412.04466) to me is the discrepancy between their theoretical claim that fairness has an high cost for misestimated users, while the empirical results suggest that such a cost mainly stems from the misestimation itself, and fairness is instead even alleviating the problem somehow. I would have liked to see some more discussion on this, as a valuable contribution could be to better understand this gap, and try to give the respective merits to the fairness constraint and to the misestimation part. How can we measure this aspect more precisely? Is the impact of fairness really negligible when users are misestimated?\n\n**Q3:** The work is absolutely not related to any existing literature other than its reference [[Greenwood et al., 2024]](https://arxiv.org/abs/2412.04466)... This does not help really is positioning the current contribution within the current landscape, and indeed pointing out how your work is similar or differ from other similar analyses (for example those I mentioned above) would greatly help is making your own contribution stand out from the work.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Two of the reviewers recommend acceptance while one recommends rejection. Overall, I do not see the reviewers' concerns as serious in the context of a reproducibility study and thus recommend acceptance with minor revision. The revision should address the following remaining comments from Reviewer egGN:
1. In Section 3.3.3, please clarify the sentences "In this setting, the $k$ items with the highest utility tend to receive the highest recommendation weights. As a result, we expect the top-$k$ recommendations to align with this sampled set of expected size $k$." Their meaning is also not clear to me.
1. Please make the histograms of test set subcategories (Figures 6(a) and 6(b) I believe) more comparable as recommended by the reviewer.
1. The last line on page 1 still has the ambiguous phrase "users with diverse preferences."
Decision: Accept (poster)