On the Generalizability and Predictability of Recommender Systems
Authors: Duncan C. McElfresh
Sujay Khandagale
Jonathan Valverde
John P Dickerson
Colin White
Venue: NeurIPS 2022 Accept (2022)
Keywords: recommender systems
algorithm selection
meta-learning
collaborative filtering

Abstract: While other areas of machine learning have seen more and more automation, designing a high-performing recommender system still requires a high level of human effort. Furthermore, recent work has shown that modern recommender system algorithms do not always improve over well-tuned baselines. A natural follow-up question is, "how do we choose the right algorithm for a new dataset and performance metric?" In this work, we start by giving the first large-scale study of recommender system approaches by comparing 24 algorithms and 100 sets of hyperparameters across 85 datasets and 315 metrics. We find that the best algorithms and hyperparameters are highly dependent on the dataset and performance metric. However, there is also a strong correlation between the performance of each algorithm and various meta-features of the datasets. Motivated by these findings, we create RecZilla, a meta-learning approach to recommender systems that uses a model to predict the best algorithm and hyperparameters for new, unseen datasets. By using far more meta-training data than prior work, RecZilla is able to substantially reduce the level of human involvement when faced with a new recommender system application. We not only release our code and pretrained RecZilla models, but also all of our raw experimental results, so that practitioners can train a RecZilla model for their desired performance metric: https://github.com/naszilla/reczilla.

On the Generalizability and Predictability of Recommender Systems
Duncan McElfresh" Sujay Khandagale", Jonathan Valverde*13
John P. Dickerson Colin White?
Abacus.AI, ArthhrAI, University of Maryland
Abstract
While other areas of machine learning have seen more and more automation, designing high-performing recommender system still requires high level of human effort. Furthermore, recent work has shown that modern recommender system algorithms do always improve over well-tuned baselines. natural follow-up question is, "how do we choose the right algorithm for a new dataset and performance metric?" In this work, we start by giving the first large-scale study of recommender system approaches by comparing 24 algorithms and 100 of hyperparameters across 85 datasets and 315 metrics. We find that the best algorithms and hyperparameters are highly dependent on the dataset and performance metric. However, there is also a strong correlation between the performance of each algorithm and various meta-features of the datasets. Motivated by these findings, we create RecZilla, meta- weta-ne approach to recommender systems that uses model to predict the best algorithm and hyperparameters for new, unseen datasets. By using far more meta-training data than prior work, RecZilla is able to substantially reduce the level of human vnvolvement when faced with a new recommender system application. We not only release our code and pretrained RecZilla models, but also all of our raw experimental results, that practitioners can train a RecZilla model for their desired performance metric: http:/htw...mm////////////aaaaaa
Introduction
Due to the large computational resources for training machine learning models, researchers have found many ways to repurpose existing computation. For example, is common to start with pretrained ImageNet classification model for computer vision tasks 30 50 75). or pretrained BERT model for natural language tasks [3 65) These approaches work well because the core problems are largely homogeneous; for example, any computer vision model its core must be able distinguish edges, colors, and shapes. Even a task-specific architecture can be found automatically through neural architecture search 36), since the building blocks such convolutional layers stay the same.
On the other hand, recommender system (rec-sys) research has followed different trajectory: despite their widespread usage across many e-commerce, social media, and entertainment companies such as there is far less work reusing models or automating the process of selecting models. Many rec-sys techniques are designed and optimized with just Intuitively, this might be because each rec-sys application
Amazon, YouTube, and Netflix 21
41 777
single dataset in mind is highly unique based the dataset and the target metric. For example, typical user session looks very different among e.g. YouTube, Home Depot, and AirBnB 2 48 58) However, this intuition
Equul Contribution. Work done while the first two authors were employed at Abacus.AI. Correspondence to: (duncan, sujay, colin)@abacoe aii valverde@es und, edu, john@arthur. ai.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

RecZilla Training Pipeline (Offline)
Algorithm Selection
Dataset Meta-feature Selection
Meta-Model training
Select paramaterced rec-sys algonthona with high coverage for matri Mon dillssoss
Select defcset mota-foctures which are corrolated with monio for the selected slgs
Learn funciion 5 22 mat mape mete meta-festurae festersttrr to matric for all salected aigs.
User input
Performance metric:
function
in the mata data-dolasof M
RecZilla for Prediction (Online)
New rec-sys dalaset:
Calculate Meta-features
Performance Prediction
Output
Calculate meta-features
Estinate performance
Algorithm Hyperparameters Predicted metric d
dataset D
parameterized alg, datasat
Figure 1: RecZilla recommends parameterized rec-sys algorithm for user-provee aser-provided dataset and performance metric. The RecZilla pipeline is built using a meta-dataset M that includes many different performance metries evaluated on many different rec-sys algorihtms on many different datasets; we estimate algorithm performance using dataset meta-features. To avoid over-fitting and reduce runtime, users can limit the number of algorithms and meta-features constants n and m.
has not been formally established. Furthermore, recent work has shown that neural recommender system algorithms do not always improve over well-tuned baselines such as k-nearest neighbor and matrix factorization [2]] A natural question is then, "how do we choose the right algorithm for new dataset and performance metric?"
In this work, we show that the best algorithm and hyperparameters are highly dependent on the dataset and user-defined performance metric. Specifically, we run the first large-scale study of rec-sys approaches by comparing 24 algorithms across 85 datasets and 315 metrics. For each dataset and algorithm pair, we test up to 100 hyperparameters (given a 10 hour time limit per pair). The codebase that release, which includes unified API for a large, diverse set of algorithms, datasets, and metrics, may be of independent interest. We show that the algorithms do not generalize the set of algorithms which perform well changes substantially across dataset and across performance metrics. Furthermore, the best hyperparameters of a rec-sys algorithm on one dataset often perform different dataset. Although show that there
significantly worse than the best hyperparameters are no universal algorithms that work well on most datasets, we do show that various meta-features of the dataset can be used to predict the performance of rec-sys algorithms. In fact, the same meta- features are also predictive of the runtime of rec-sys algorithms as well as the "dataset hardness" how challenging is find a perffrrnfn model on a particular dataset.
Motivated by these findings, we introduce RecZilla, a meta-learning-baseee based algorithm selection approach (see Figure [ inspired by SATzilla 833 At the core of RecZilla is model that, given user-defined performance metric, predicts the best rec-sys algorithm and hyperparameters for a new dataset based meta dataset features such as number of users and items, and spectral properties of the interaction matrix. We show that RecZilla quickly finds high-performing algorithms on datasets has never seen before. While there has been prior work on meta-learning for recommender systems no prior work is metric-independdet, searches for hyperparameters as well as algorithms, or considers more than nine families of datasets. By running an ablation study on the number of meta-training datasets, we show that more datasets are crucial to the success of RecZilla. We release ready-to-use, pretrained RecZilla models for common test metrics, and release the raw results from our large-scale study, along with code so that practitioners can easily train new RecZilla model for their specific performance metric of interest.
26 27
Our contributions. We summarize our main contributions below.
We run a large-scale study of recommender systems, showing that the best algorithm and hyper- parameters are highly dependent on the dataset and user-defined performance metric. We also show that dataset meta- meta-ffcaures are predictive of the performance of algorithms.

---

We create RecZilla, meta- weea--earming-bee -banid algorithm selection approach which, given perfor- mance metric, efficiently predicts the best algorithm and set of hyperparameters on new datasets. We release a public repository containing 85 datasets and 24 rec-sys algorithms, accessed through unified API. Furthermore, release both pretrained RecZilla models, and raw data so that users can train a new RecZilla model on their desired metric.
Common ap- proaches include k-nearest neighbors M. matrix factorization [57 63 and deep learning approaches A recent meta-study showed
Related Work Recommender systems are a widely studied area of research 100)
221 41 77 For a survey on recommender systems, see
that of the 12 published neural rec-sys approaches published at top conferences between 2015 and 2018, 11 performed worse than well-tuned baselines (e.g. nearest neighbor search or linear models) Another recent paper found that the relative performance of rec-sys algorithms can change significantly based on the choice of datasets used [E]
28
first studied in 2011 52 by using a graph representation of item ratings. Follow-up work used dataset meta-features to select the best nearest Subsequent work focused on improving the model and framework [27] including studying 74 meta-features systematically [23]] More recent approaches from 2018 run meta-learning for recommender systems by casting the meta-problem itself as collaborative filtering problem. Performance is then estimated with subsampling landmarkers No prior work in algorithm selection for rec-sys includes open-source Python code. There is also work on automated machine learning (AutoML) for recommender systems, without meta-learning 6 46 47 82). Finally, we note that meta-learning across rec-sys datasets is not be confused with the body of work on meta-learning user preferences within single rec-sys dataset 17 18 [9]. To the best of our knowledge, no meta-learning or AutoML rec-sys paper has run experiments on more than nine dataset families or four test metrics, and no prior work predicts hyperparameters in addition to algorithms.
Algorithm selection for recommender systems
neighbor and matrix factorization algorithms K 35 [
251 20
2 Analysis of Recommender Systems
In this section, we present large-scale empirical study of rec-sys algorithms across large, diverse set of datasets and metrics. We assess the following two research questions.
1. Generalizabilitt. If a rec-sys algorithm or set of hyperparameters performs well on one dataset and metric, will it perform well on other datasets or on other metrics?
2. Predictability. Given a metric, can various dataset meta-features be used to predict the perfor- mance of rec-sys algorithms?
Algorithms,, datasets, and metries implemented. We present full results for 20 rec-sys algorithms, including methods from recent literature and common basclines. Methods include five similarity
and clustering- based methods: User-KNN (74). Item-KNN 760 P3Alpha [20]. RP3Beta (72). and
Co-Clustering [3]] six Matrix-Factorizttion (MF) methods: MF-FunkSVD MF-Asy [36].
MF-BPR 7
IALS 51. Pure-SVD, Non-negative matrix factorization (NMF)
five methods
based on linear models: Global- SLIM-BPH BPR (7]. SLIM- MeLLM---laticNee Net 62). EASE-R [78), and
SlopeOne 61:: two simple baselines: Random and Top-Pop: and two neural network based methods: and Mult- VAE (64] We also include partial results (on 8- 10 datasets each) for
User-NeuRec [
four more neural network based methods: DELF-EF [33, DELF-MLP [I3, Item-NeuRee (84), and
Spectral-CF 35 (with results included in Tables y 10 and MT. These algorithms were chosen due
to their high performance, popularity, and speed. For many algorithms, we used the implementations from the codebase of Dacrema et al. 28 For full details of the algorithms, Appendix A
We run the algorithms on 85 datasets from 19 dataset "families": Amazon BookCrossing [87). CiaoDVD 45
Anime 16
59 Dating (Libimseti.com [59 60) Epinions 67 68 Film
and MarketBias- ModCloth 180], MovieTTeettnngs 3 Movielens 49. NetflixPrize 19. Recipes [66]. Wikilens [37].
44. Frappe [ Gowalla (5). Jester2 40 LastFM [II, MarketBias-Eleetrooe
and Yahoo 34 Here, a "dataset family" refers to an original dataset, while "dataset" refers to a single train-test split drawn from the original dataset, which may be a small subset of the original. We implemented the majority of rec-sys datasets commonly used for research; to the best of our knowledge, this is the largest number of rec-sys datasets accessible in single open-source repository.
3

---

Table The relative performance of each rec-sys algorithm depends on the dataset and metric. This table shows the mean, min (best) and max (worst) rank achieved by all 20 algorithms over all 85 datasets, over 10 aceuracy and hit-rate metries all cutoffs tested. This includes metries NDCG, precision, recall, rrc.-RRc.-Min-denne hit-rate, F1, MAP, MAP-Min-density, ARHR, and MRR.
SLIM SLIM-ElastiNNe ElasticNet
GlobalEffects
CoClustering
Item-KNN -NNN
PSalpha
SLIM-BPR
EASE-R
RP3beta
User User-KNN KNN
MF Funk
TopPop
MF-Asy
MF BPR
Mult- VAE
U U-neural
Random
SlopeOne
Item-
SVD
IALS
NMF
Rank
Min.
1
1
1
1
1
1
1
1
1
Max.
14
18
14
18
17
16
17
19
14
17
18
19
16
17
20
20
20
19
20
20
Mean
2.3
4.2
4.7
5.3
6
6
7
7
7.1
7.6
9.4
10.4
10.7
11.2
11.7
12.3
13.3
14.9
16.2
16.7
We use 23 different "base" metrics: ARHR, Average Popularity, Diversity Similarity, Fl Score, Gini Index, Herfindahl Index, Hit Rate, Item Coverage, Item-hit Coverage, Precision (PREC), Precision- Recall Min Denominator, Recall, MAP, MAP Min Denominator, Mean Inter-List Diversity, MRR, NDCG, Novelty, Shannon Entropy, User Coverage, User-hit Coverage. All of these metrics are computed at cutoff's f1, 2, 3, 4,5, 6, 7, 8, 9, 10, 15, 20, 30, 40, 50}, for a total of 315 different metrics. See Appendix A.6 for more details. These metrics include the most popular from the literature, and we use the Dacrema et al. 23 implementations.
Experimental design. Each dataset's 's train, validation, and test split is based on leave-latt-k-out (and our repository also includes splits based on global timestamp). We use a random hyperparameter search for all methods, with the exception of neural network based methods. Since neural networks require far more resources to train (longer training time, and requiring GPUs), we use only the default hyperparameters for neural algorithms. For each non-neural algorithm, we expose several hyperparameters and give ranges based on common values. For each dataset, we run each algorithm on a random sample of up to 100 hyperparameter sets. Each algorithm is allocated a 10 hour limit for each dataset split; we train and test the algorithm with at most 100 hyperparameter sets on an n1 m1-highmme -2ggmmmm-m22 Google Cloud instance, until the time limit is reached. Each neural network method trained on each dataset using the default hyperparameters used in its respective paper, with a time limit of 15 hours an NVIDIA Tesla T4 GPU. All neural network methods are trained with batch size 64, for up to 100 epochs; early stopping occurs loss does not improve in 5 epochs.
Each algorithm is trained on the train split, and the performance metrics are computed on the test split. We refer to each combination of (algorithm, hyperparameter set, dataset) as an experiment. By running 24 algorithms, most with up to 100 hyperparameters, on 85 datasets, this resulted in 84 850 successful experiments, and by computing 315 metries, our final meta-dataset of results includes more than 26 million evaluations. We give more detailed look at the breakdown of experiments in Appendix A and we discuss any potential biases in the resulting dataset in Section
2.1 On the generalizability of rec-sys algorithms
If a rec-sys algorithm or set of hyperparaneters performs well on one dataset and metric, will perform well other datasets on other metrics?
Our first observation that all algorithms perform well on some datasets, and poorly others. First we identify the best-performing hyperparameter set for each (algorithm, dataset) pair-to simulate hyperparameter optimization using our meta-dataset. We then rank all algorithms for each dataset, according to several performance metrics.
If we focus single metric, then many algorithms are ranked first according to this metric at least one dataset. Take for example metric NDCG@1: 1: 17 of 20 algorithms are ranked either first or second on least dataset. The same is true for metric RECALL @50: all algorithms except for SlopeOne, GlobalEffects, and Random are ranked either first or second on at least one dataset. The same is true for many other metrics and cutoffs (see Table in Appendix C
Average performance more varied: some algorithms tend to perform better than others. Table D shows the mean, min (best) and max (worst) ranking of all 24 algorithms over all datasct and all

---

*
Item-KNN
U-neural
MF-BPR JALS
SLIM-BPR
Global Efffectsstt
Randem
User-KNN
MUIL-VE
SLIM Elassiicti
TopPop
1.0
PREC@50
0.5
COV-ITEM HI@@10
1.0
0.5
1.0
NDCG@5
0.5
0.0
Amz- Amz-Movie
Amzz Homelmpry
Anime
BookCrass
ClaoDVD
Epinions
FilmTrust
Jester
MktBias-Mon Mad
MvTweet
Mvins- Mvins-100K -00K
Mvins- Mvins-I0M 10M
Mvins-Hetre Hetre-
Recipes
Wikilens
YahooMv
Figure 2: Relative algorithm performance depends both the dataset and metric: no algorithm dominates across all dataests or metries. Each plot shows different metric, normalized [o. 1] for each dataset; the horizontal axis shows different dataset, ordered alphabetically. Each series corresponds to different algorithm: similarity-bai -based methods are red, matrix factorization methods are green, baseline methods are black, and neural network methods are in gray.
accuracy and hit-rate metrics. This includes metries NDCG, precision, recall, Prec--Rec.-Min-eeeee hit-rate, F1, MAP, MAP-Min-densigg ARHR, and MRR (see Appendix A.6 for descriptions of these metrics). Nearly all algorithms are ranked first for least one metric on at least one dataset. Many algorithms perform well on average. Furthermore, most algorithms perform very poorly in some cases: the maximum rank least 14 (out of 20) for all algorithms.
To illustrate the changes in algorithm performance across datasets, Figure shows the normalized metric values for eight algorithms across 17 dataset splits. Some algorithms tend to perform well (Item-KNN and SLIM-BPR) and others poorly (Random, TopPop), but no algorithm clearly dominates for all metrics and datasets. This is a primary motivation for our meta-learning pipeline descirbed in Section 3 different algorithms perform well for different datasets on different metrics, so is important to identify appropriate algorithms for each setting.
Generalizability hyperparameters. While the previous section assessed the generalizability of pairs of (algorithm, hyperparameters), now we assess the generalizability of the hyperparameters themselves while keeping the algorithms fixed. dataset i, and then evaluate the normalized performance of the tuned method on a dataset j. compared to the normalized performance of the best hyperparameters from dataset j. In other words, we another.
given rec-sys algorithm, we can tune it on
compute the performance of tuning method on one dataset and deploying
In Figure we run this experiment for all pairs datasets (one dataset per dataset family). We plot the hyperparameter transfer for three different algorithms, as well as the average over all algorithms which completed sufficiently many experiments across the set of hyperparameters. For each given j. we create the set of hyperparameters that completed for the given algorithm on both datasets and j. and then we use min-max scaling for the performance metric values of these hyperparameters on and on j separately. Therefore, all matrix values are between 0 and 1; value close to 1 indicates that the best hyperparameters from dataset are also nearly the best on dataset j. value close to 0 indicates that the best hyperparameters from dataset i are nearly the worst for dataset j. Across
5

---

HFaram.
Matrivssctrraation.
ommender
AmazoniBecuryy
1.0
AmatondiBeerry
10
Mime
Anme
RockCmsiin
BockCrnsing
Cleoton
Deling
Deting
Epmana
Epraan
hrasp
0.6
Frasps
06
Jen0r2
Jepor2
Movicllecctng
Hoviclers35N
Moviclorc20M
0.2
NedflePriee
02
Secpes
Keimn
Buok BualCross Crg
Denino
Gowel
Jester?
tohooMorcc
Amareekk Do
Keima
Cosorr
Jerrers
Lintri
Wuichhe Coccsoii
fietze
"a cooMooocc
Ammraan tree
porr
Wa
Transfer: AFShetaRecommeeddr
Transfer
AmarondiBesrry
10
AmatonWiBesrry
1.0
Atme
Anme
RockCmeiing
BockCinssing
Cleotvn
Dating
Dating
spmans
Epmaas
Nimiuust
Frasps
0.6
Frasps
0.
bowala
Jeste22
MarsetBlauumazoon
Moviclonc2SW
Moviclor220
52
02
Recpes
Koime
Book BuorCrosin
Danino
Estion
Russe
Goodls
Jester
Yo neddosiii on
Rools Tooleeooooe Teooo and
Amaren theaty
Anima
CooDVo
Dating
Jeste's
Wa Nes sessssttooo aod un
Movi Movic ceciingg
Wetfuuhon
Figure 3: Transfcrability of hyperparameters across datasets, for three different algorithms, and the average of all algorithms (bottom right). For each plot, row i, column j denotes the relative performance of an algorithm tuned dataset and then evaluated dataset j. A value close to 1 indicates that the hyperparameters transfer well from to j. while a value close indicates that the hyperparameters transfer poorly.
all algorithms, we see that is particularly hard for hyperparameters transfer and from some datasets such as Gowalla and Jester2. Furthermore, the majority of pairs of datasets do not have strong hyperparameter transfer. Overall, these experiments give evidence that tuning the hyperparameters of an algorithm on one dataset and transferring to another dataset does not give high performance, motivating RecZilla which predicts the best hyperparameters for given algorithm and dataset.
2.2
On the predictability of rec-sys algorithms
Can attributes of the rec-sys dataset be used predict the performance of rec-sys algorithms??
Dataset meta-fcatures. We calculate 383 different meta-features to characterize each dataset. These meta-features include statistics on the rating matrix matrix-including basic statistics, the distribution- based features of Cunha et al. [3] and landmark features 24 wwhic measure the performance of simple rec-sys algorithms on a subset of the training dataset. Since these meta-features are used for algorithm selection, they calculated using only the training split of each dataset. For more details on the dataset meta-features, see Appendix A.4
Algorithm performance prediction. Table shows the meta-features are most highly- correlated with the performance (PREC@ @ 10) of each algorithm, using their default parameters.

---

Table 2: Highest absolute correlations computed across datasets and weighed inversely proportional to dataset family frequency, over all pairs of algorithm families and meta- feta-fea,uree, for the PREC 10 performance metric and the default algorithm hyperparameters.
Abs. Correlation
Algorithm Family
Meta-feature
0.941
SlopeOne
Mean item rating count distribution
0,933
CoClustering
Median of item rating count distribution
0.887
MF-BPR
Sparsity of rating matrix
0.855
RP3heta
Mean of item rating count distribution Landmarker, Pure SVD, mAP@5
0.846
UserKNN
Several meta- meta-features aare highly-correlated with algorithm performance; one of the simplest metrics the mean of the item rating count distribution-is highly correlated with performance of two rec-sys algorithms. This experiment motivates the design of RecZilla in the next section, which trains model using dataset meta-features to predict the performance algorithms on new datasets.
As toy-model version of RecZilla, we train three different meta-learner functions (XGBoost, KNN, and linear regression) using our meta-dataset, predict performance metric PREC@ 10 for 10 rec-sys algorithms with high average performance (see Appendix A for details). We use leave-one-out evaluation for each meta-learner: one dataset family held out for testing, while m are used for training. Figure shows the mean absolute eπror (MAE) of each meta-learner; these results are aggregated over 200 random samples of randomly-selecterr training dataset families. MAE decreases as more dataset families are added, suggesting that it is possible to estimate rec-sys algorithm performance using dataset meta-features.
We also find that performance metrics are not the only values that can be predicted with dataset meta-features. In particular, we find that the runtime of rec-sys algorithms also highly correlated with different meta-fcatures, Furthermore, we compute a simple measure of dataset hardness, which we compute as, given performance metric, the maximum value achieved for that dataset across all algorithms. For example, if all 20 algorithms do not perform well on the Movie Tweetings dataset, dataset is "hard". Once again, we find that certain
then we can expect that the MovieTweetimg dataset meta- meta-features are highly correlated with dataset hardness. For more details on meta-feature correlation with algorithm runtimes and dataset hardness, see Appendix
The fact that dataset meta-features are correlated with algorithm performance, algorithm runtimes, and dataset hardness strong positive signal that meta-learning is worthwhile and useful in the context of recommender systems. We explore this direction further in the next section.
3 RecZilla: Automated Algorithm Selection
In the previous section, found that (1) the best algorithm and hyperparameters strongly depend on the dataset and user-chosen performance metric, and (2) the performance of algorithms can be predicted from dataset meta-features. Points (1) and (2) naturally motivate algorithm selection approach to rec-sys powered by meta-learning.
In this section, we present RecZilla, which is motivated by practical challenge: given a performance metric and a new rec-sys dataset, quickly identify an algorithm and hyperparameters that perform well on this dataset. This challenge arises in many settings, e.g., when selecting good baseline algorithms for academic research, or when developing high-performing ree-sys algorithms for commercial application. We begin with overview and then formally present approach.
Overview. RecZilla is an algorithm selection approach powered by meta-lcarning. We use the results from the previous section the meta-training dataset. Given user-specified performance metric, we train a meta-model that predicts the performance of each of a set of algorithms and hyperparameters on a dataset, by using meta-features of the dataset. Given new, unseen dataset, compute the meta-feature of the dataset, and then use the meta meta-model to predict the performance of each algorithm, returning the best algorithm according to the performance metric. See Figure] ||
7

---

Preliminaries. We start with notation for the rec-sys problem. Let D denote a rec-sys dataset, consisting of a set of user-item interactions, split into a training and validation set. Let a denote rec-sys algorithm parameterized by set of k(a)
XGB KNN
0.010
Linear
0.008
hy perparameters (a) C R*(a), which is Aggorthm-ddependmt Suppose we train algo- rithm on the training split of dataset D, us- ing hyperparameters w; we denote the perfor- mance of algorithm with hyperparameters w on dataset D as ó(a, w. D) E R. The function ((.,,) represents performance metric for the recommender system that selected by the user; throughout this paper refer to this function and its numerical value simply as performance. In this paper, larger values of ((:;,;) indicates better performance. We report the normalized performance defined as
MAE
0.006
0.004
10
12
14
16
18
Num. training dataset families
Figure 4: Three basic meta-learners (KNN, lin- regression, and XGB) are trained randomly- selected dataset families to predict performance metric PREC@ 10. As more dataset families added, the meta-learners are better able to predict rec-sys algorithm performance, suggesting that our dataset meta-features are useful for predicting rec- sys algorithm performance. Vertical axis shows mean absolute error (MAE), over over all folds of leave- one-out validation, and 200 random trials; in each trial a different set of training datasets are chosen. Error bars show the 40th and 60th percentile.
Pman Pman b(a w. D) = 100 p(a, w, D)
Phar P pmai
where Piaz (resp. PDin) are the max (resp.
min) performance of any algorithm on D.
Next we define notation for the meta-learning problem. Given a fixed performance function Φ,
let .M- {(D;, a, w;, y;)) denote meta- datasel consisting of M tuples. Each tuple con- sists of a dataset D, algorithm a parameterized by w. and performance y. where yi plas, Wi, D.). Since many algorithms have a wide range of hyperparameters, we refer to the set of parameterized algorithms in the meta-dataset where each element of d is meta-fcature of the dataset. In this setting, the meta-lcarning task is to identify a function that estimates the performance of parameterized algorithm (a, w) on dataset D.
= {(a w: ) We represent each dataset using vector RT
3.1
The RecZilla Algorithm Selection Pipeline
The RecZilla algorithm selection pipeline takes as input a meta-dataset M built with user-chosen performance function for the recommendation task ((, RecZilla can accommodate any perfor- mance function that is a computable function of the performance metrics present in the meta-dataset. The RecZilla pipeline proposed here returns both algorithm a A and set of hyperparameters w E H (a). so that RecZilla can be used without additional hyperparameter optimization.
Since the meta-learning problem is in the low-data regime, to guard against overfitting, we select subset of the algorithms that have good coverage over the training dataset. We similarly select only the metaurfeturess which are most predictive of the user-selected performance metric for the selected algorithms. Below we outline each of the steps used to build the proposed RecZilla pipeline.
1. Algorithm subset selection: We select subset S' S of n parameterized algorithms that collectively perform well on all datasets in the meta-training dataset M, according to performance function é. We aim to select an algorithm subset with high coverage over the set of known datasets, where coverage of a subset S' is defined as
C(S') D
Σ
max Hla,w, D).
DED (a.))SS)
In other words, the coverage of subset S' the normalized performance metric the best performing algorithm in S', averaged over all datasets D. Selecting subset with maximum coverage is itself a difficult problem; we use a greedy heuristic as follows. We begin with S' = O and iteratively add the parameterized algorithm (a" w²) E arg (a))es C(S' U {(a ()) to n. That is, we greedily ensure that the coverage is maximized at each step.
S' until |S'|
8

---

Table Comparison between RecZilla and two representative algorithm selection approaches from prior work. To give fair comparison, the approaches are given the same meta-training datasets. We compute Difff as defined Section 3.2 as well as the Precision 10 for the predicted best algorithm. We report the mean and standard deviation across 50 trials for 19 test sets, for 950 total trials. The runtime the average time it takes to output predictions on the meta-test dataset.
Approach
Runtime (sec)
% Diff (4)
pREC@ 10 of best pred. (1)
cunha2018 27
0.39
52.9 23.0
0.00813 0.0113
cf4cf-mets 2
6.68
43.5 + 21.8
0.00808 + 0.00773
RecZilla 6.69
33.2 22.8
0.00915+t 0.00840
2. Meta-feature selection: Similar to the previous point, select subset of meta-features with good coverage over the meta-training dataset M, where here coverage defined in terms of the correlation between algorithm performance and each meta-feature (see Appendix A.4 for details). Let M : + R denote the resulting function that maps a dataset to a vector of meta-features.
3. Meta-learning: We learn function f : Rm - R", where f performance metric of all parameterized algorithms in S on dataset meta meta-features d.
is a vector of the estimated
Note that the RecZilla pipeline has two hyperparameters: n, the number of parameterized algorithms in 8"; and m, the number of dataset meta-features used by the meta-learner. In our experiments, we run an ablation study with both and m, as well as different functions f for the meta-learning model.
Using RecZilla for Algorithm Selection. After developing the RecZilla pipeline, we use the following steps to select an algorithm for a new dataset D':
1. Calculate m meta-features of the dataset d' + -D').
2. Estimate the performance of all parameterized algorithms: y' + f(d).
3. Return the parameterized algorithm in S' with the best estimated performance.
3.2 Experiments
In this section, we evaluate the end-to-end RecZilla pipeline. We start by describing the specific versions of RecZilla used in our experiments. We use four different meta-learning functions within RecZilla: XGBoost [[2], linear regression, k-nearest neighbors, and uniform random. For KNN, we set k = 5 and use the L2 distance from the selected meta-features.
Experimental setup. Focusing on performance metric PREC @ 10, we build meta-dataset M using all rec-sys datasets, algorithms, and meta-features described in Section 2 We first use the algorithm selection and meta-feature selection procedures described above to select n m 100 parameterized algorithms and meta-features. For all experiments, algorithms selected during this process. We vary both the number of training meta-datapoints and meta-features; the datapoints and features are randomly selected over 50 random trials. All RecZilla meta-learners are evaluated using ceeee-on--dtttte--eeeeeeeeeeeeee family the meta-test dataset, and run the full RecZilla pipeline using the remaining datasets the meta-traine veta-training data. Splitting on dataset families rather than datasets ensures that there is no test data leakage. Then for each dataset D in the test set, compare the performance metric of the predicted best parameterized algorithm (a', w²) to the performance metric of the ground-truth best algorithm y", using the percentage-Oe wewegeeddfffffee-fffeeee frombbbeto-- %Diff 100 x (y* ((D, d' w'v)""". %Diff is between O and 100, and smaller values indicate better performance.
use the best 10 paramcterized
evaluation: we iteratively select each dataset
Results and discussion. In Table 3. we compare RecZilla with two prior algorithm selection approaches: cunha2018 [27] and sf4ca-meta [25], which are a comprehensive depiction of all prior work (see Appendix B.4 for justification, and for details of the experiment). Furthermore, note that cunha2018 has no open-source code, and cf4c4-meta only has code in R. We find that RecZilla outperforms the other two approaches in both % Diff and terms of the PREC@ 10 value of the rec-sys algorithm outputted by each meta-learning algorithm.

---

Figure 5 (left) shows %Diff vs. the size of the meta-training set, and Figure 5 (right) shows the results of an ablation study on the number of selected meta-features m. See Appendix for more details and discussion.
In Appendix C
Pre-trained RecZilla models. We release pre-trained RecZilla models for PREC@ 10, NDCG@ 10, and Item-hit Coverage @ 10, trained with XGBoost on all 18 datasets, with algorithms n 10 and meta-features m 10. We also include a RecZilla model that predicts the Pareto-front of PREC @ 10 and model training time, so that users can select their desired trade-off between performance and runtime. Finally, we include a pipeline so that users can choose metric from the list of 315 (or any desired combination of the 315 metrics) and train the resulting RecZilla model.
4 Conclusions, Limitations, and Broader Impact
In this work, we conducted the first large-scale study of rec-sys approaches: we compared 24 algo- rithms and 100 sets of hyperparameters across 85 datasets and 315 metrics. We showed that for a given performance metric, the best algorithm and hyperparameters highly depend on the dataset. We also find that various meta-features of the datasets are predictive of algorithmic performance and runtimes. Motivated by these findings, we created RecZilla, the first metrie-indeeenndet, hyperparameter-an algorithm selection approach to recommender systems. Through empirical evaluation, we show that given a user-defined metric, RecZilla effectively predicts seg-pperforiin algorithms and hyperpa- rameters for new, unseen datasets, substantially reducing the need for human involvement. We not only release our code and pretrained RecZilla models, but we also release the raw experimental results so that users can train new RecZilla models on their own test metrics of interest. This codebase, which includes a unified API for 85 datasets and 24 algorithms, may be of independent interest.
Limitations. While our work progresses prior work along several axes, there are still avenues for improvement. First, the meta-learning problem in RecZilla low-data. Although we added nearly all common rec-sys research datasets into RecZilla, the result is still only 85 meta-datapoints (datasets). While we guarded against over-fitting to the training data in numerous ways, RecZilla can still be improved by more training data. Therefore, as new recommender system datasets are released in the future, our hope to add them our API, so that RecZilla continuously improves over time. Similarly, our hope is to add the most recent high- fethhhrrffrmingggg rec-sys approaches to our work, as well as algorithms released in the future. This includes adding neural network- -baed approaches, in addition to the six that we have already included. Another limitation is that RecZilla does not directly predict the performance of hyperparameters for algorithms on a given dataset. Although care must be taken to not overfit, modifying RecZilla to predict the performance of an algorithm together with a set of hyperparameters is an interesting avenue for future work. Finally, the magnitude of our evaluation (84 850 models trained) leaves our meta-data susceptible to biases based on experiment success/failures. While we fixed many common errors such as out-of-memory errors, it was infeasible to give each experiment specific attention. Therefore, RecZilla may have higher uncertainty for the datasets and algorithms that are more likely to fail. An interesting future improvement to RecZilla would be to predict the likelihood that a new dataset will successfully train on a new dataset.
Broader impact. Our work is "meta-research": there is not one specific application that we target, but our work makes it substantially easier for researchers and practitioners to quickly train recommender system models when given a new dataset. On the research side, this is a net positive because researchers can much more easily include baselines, comparisons, and run experiments on large numbers of datasets, all of which lead to more principled empirical comparisons. On the applied side, day-to-day lives are becoming more and more influenced by recommendations generated from machine learning models, which comes with pros and cons. These recommendations connect users with needed items that they would have had to spend time searching for [54] Although these
recommendations may lead to harmful effects such as echo chambers 38 55 techniques to identify
and mitigate harms are improving 42 69
Acknowledgments and Disclosure of Funding
This work was supported by a GEM Fellowship, NSF CAREER Award I1S-1346,37 NIST MSE Award #20126334, DARPA GARD #HR001 12020007, and DoD WHS Award #HQ003420F0035..
10

---

References
[1] David Adedayo Adeniyi, Zhaoqiang Wei, and Yongquan. Automated web usage data mining and recommendation system using k-nearest neighbor (knn) classification method. Applied Computing and Informatics, 12(1):90-108,, 2016.
[2] Gediminas Adomavicius and YoungOk Kwon. Improving aggregate recommendation diversity using ranking-based techniques. IEEE Transactions on Knowledge and Data Engineering, 2465):896-91,, 2011.
[3] Gediminas Adomavicius and Jingjing Zhang. Impact of data characteristics on recommender systems performance. ACM Transactions on Management Information Systems (TMIS), 3(1):1- 17, 2012.
[4] Charu C Aggarwal et al. Recommender systems, volume 1. Springer, 2016.
[5] Fabio Aiolli. Efficient top-n recommendation for very large scale binary rated datasets. In Proceedings of the 7th ACM Conference on Recommender Systems, RecSys '13, page 273-280, New York, NY, USA, 2013. Association for Computing Machinery.
[6] Rohan Anand and Joeran Beel. Auto-surprise: An automated recommender-system (autorecsys) library with tree of parzens estimator (tpe) optimization. In Fourteenth ACM Conference on Recommender Systems, pages 585-587, 2020.
[7] Krisztian Balog, Filip Radlinski, and Shushan Arakelyan. Transparent, scrutable and explainable user models for personalized recommendation. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR' 19, page 265-274, New York, NY, USA, 2019. Association for Computing Machinery.
[8] Linas Baltrunas, Karen Church, Alexandros Karatzoglou, and Nuria Oliver. Frappe: Under- standing the usage and perception of mobile app recommendations in-the-wild. arXiv preprint 1rXiv:1505.03011 2015.
[9] James Bennett, Stan Lanning, et al. The netflix prize. In Proceedings of KDD cup and workshop. volume 2007, page 35. Citeseer, 2007.
[10] Bobadilla, F. Ortega, A. Hernando, and A. Gutiérrez. Recommender systems survey. Knowledge-Bused Systems, 46: 109-132, 2013.
[11] Iván Cantador, Peter Brusilovsky, and Tsvi Kuflik. 2nd workshop on information heterogeneity and fusion in recommender systems (hetrec 2011). In Proceedings of the 5th ACM conference on Recommender systems, RecSys 2011, New York, NY, USA, 2011. ACM.
[12] Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, pages 785-794, 2016.
[13] Weiyu Cheng, Yanyan Shen, Yanmin Zhu, and Linpeng Huang. Delf: A dual-embedding based deep latent factor model for recommendation. In Proceedings of the Twenty-Seventh Interna- tional Joint Conference on Artificial Intelligence, IJCAI-18, pages 3329-3335. International Joint Conferences on Artificial Intelligence Organization, 7 2018.
[14] Jin Yao Chin, Yile Chen, and Gao Cong. The datasets dilemma: How much do we really know about recommendation datasets? In Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining, pages 141-149, 2022.
[15] Eunjoon Cho, Seth A. Myers, and Jure Leskovec. Friendship and mobility: User movement in location- based social networks. In Proceedings of the 17th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '11, page 1082-1090, New York. NY, USA, 2011. Association for Computing Machinery.
[16] Hyerim Cho, Marc L Schmalz, Stephen A Keating, and Jin Ha Lee. Analyzing anime users' online forum queries for recommendation using content analysis. Journal of Documentation, 2018.
11

---

[17] Andrew Collins and Joeran Beel. Meta-learned per-instance algorithm selection in scholarly recommender systems. arXiv preprint arXiv:19/2.0866 2019.
[18] Andrew Collins, Dominika Tkaczyk, and Joeran Beel. novel approach to recommendation algorithm selection using meta-learning. In AICS, pages 210-219, 2018.
[19] Andrew Collins, Dominika Tkaczyk, and Joeran Beel. One-at-a-time: A meta-learning recommender-system for aecommendation-aagoriiaii selection on miero level. arXiv preprint arXiv: /2//8 2018.
[20] Colin Cooper, Sang Hyuk Lee, Tomasz Radzik, and Yiannis Siantos. Random walks in recom- mender systems: exact computation and simulations. In Proceedings of the 23rd international conference on world wide web, pages 81 811-816, 2014.
[21] Paul Covington, Jay Adams, and Emre Sargin. Deep neural networks for youtube recommenda- tions. In Proceedings of the 10th ACM conference on recommender systems, pages 191-198, 2016.
[22] Paolo Cremonesi, Yehuda Koren, and Roberto Turrin. Performance of recommender algorithms on top-n recommendation tasks. In Proceedings of the fourth ACM conference on Recommender systems, pages 39-46, 2010.
[23] Tiago Cunha, Carlos Soares, and André CPLF de Carvalho. Selecting collaborative filtering algorithms using metalearning. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 409. Springer, 2016.
[24] Tiago Cunha, Carlos Soares, and André CPLF de Carvalho. Recommending collaborative filtering algorithms using subsampling landmarkers. In International Conference on Discovery Science, pages 189-203. 203. Springer, 2017.
[25] Tiago Cunha, Carlos Soares, and André CPLF de Carvalho. Cf4cf-meta: Hybrid collaborative filtering algorithm selection framework. In International Conference on Discovery Science, pages 114-128. Springer, 2018
[26] Tiago Cunha, Carlos Soares, and André CPLF de Carvalho. Cf4cf: recommending collaborative filtering algorithms using collaborative filtering. In Proceedings of the 12th ACM Conference on Recommender Systems, pages 357-361, 2018.
[27] Tiago Cunha, Carlos Soares, and André CPLF de Carvalho. Metalearning and recommender systems: A literature review and empirical study on the algorithm selection problem for collaborative filtering. Information Sciences, 423:128-144: 2018
[28] Maurizio Ferrari Dacrema, Simone Boglio, Paolo Cremonesi, and Dietmar Jannach. A troubling analysis of reproducibility and progress in recommender systems research. ACM Transactions on Information Systems (TOIS), 39(2):1-49, 2021.
[29] Maurizio Ferrari Dacrema, Paolo Cremonesi, and Dietmar Jannach. Are we really making much progress? A worrying analysis of recent neural recommendation approaches. In Toine Bogers, Alan Said, Peter Brusilovsky, and Domonkos Tikk, editors, Proceedings of the 13th ACM Conference on Recommender Systems, RecSys 2019, Copenhagen, Denmark, September 16-20, 2019, pages 101-109. ACM, 2019.
[30] Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-22882 255.. Ieee, 2009.
[31] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Confer- ence of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume I (Long and Short Papers), pages 4171-4188 Minneapolis, Minnesota, June 2019, Association for Computational Linguistics.
[32] Lee R. Dice. Measures of the amount of ecologic association between species. Ecology, 26(3):297-302, 1945.
12

---

[33] Simon Dooms, Toon De Pessemier, and Luc Martens, Movietweetings: a movie rating dataset collected from twitter. In Workshop on Crowdsourcing and Human Computation for Recom- mender Systems, CrowdRec at RecSys 2013, 2013.
[34] Gideon Dror, Noam Koenigstein, Yehuda Koren, and Markus Weimer. The yahoo! music dataset and kdd-cup' 11. In Proceedings of KDD Cup 2011, pages 3-18. PMLR, 2012.
[35] Michael Ekstrand and John Riedl. When recommenders fail: predicting recommender failure for algorithm selection and combination. In Proceedings of the sixth ACM conference on Recommender systems, pages 233-236, 2012.
[36] Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey.
In JMIR, 2019.
[37] Dan Frankowski, Shyong K. Lam, Shilad Sen, F. Maxwell Harper, Scott Yilek, Michael Cassano, and John Riedl. Recommenders everywhere: The wikilens community- maintained recommender system. In Proceedings of the 2007 International Symposium on Wikis, WikiSym '07, page 47-60, New York, NY, USA, 2007. Association for Computing Machinery.
[38] Yingqiang Ge, Shuya Zhao, Honglu Zhou, Changhua Pei, Fei Sun, Wenwu Ou, and Yongfeng Zhang. Understanding echo chambers in e-commerce recommender systems. In Proceedings of the 43rd international ACM SIGIR conference research and development in information retrieval, pages 2261-2270, 2020.
[39] Thomas George and Srujana Merugu. A scalable collaborative filtering framework based on co-clustering. In Fifth IEEE International Conference Data Mining (ICDM'05) pages pp. IEEE, 2005.
[40] Ken Goldberg, Theresa Roeder, Dhruv Gupta, and Chris Perkins. Eigentaste: A constant time collaborative filtering algorithm. Information Retrieval, 4(2):133-151, 2001.
[41] Carlos A Gomez-Uribe and Neil Hunt, The netflix recommender system: Algorithms, business value, and innovation. ACM Transactions on Management Information Systems (TMIS), 6(4):1-
19, 2015.
[42] Pietro Gravino, Bernardo Monechi, and Vittorio Loreto. Towards novelty-driven recommender systems. Comptes Rendus Physique, 20(4):371-379, 2019.
[43] Josephine Griffith, Colm O' O'Riordan, and Humphrey Sorensen. Investigations into user rating information and predictive accuracy in a collaborative filtering domain. In Proceedings of the 27th Annual ACM Symposium on Applied Computing, pages 937-942, 2012.
[44] G. Guo, Zhang, and N. Yorke-Smith. A novel bayesian similarity measure for recommender systems. In Proceedings of the 23rd International Joint Conference on Artificial Intelligence (IJCAI), pages 2619-2625, 2013.
[45] Guibing Guo, Jie Zhang, Daniel Thalmann, and Neil Yorke-Smith. Etaf: An extended trust antecedents framework for trust prediction. In 2014 IEEE/ACM International Conference on Advances Social Networks Analysis and Mining (ASONAM 2014), pages 540-547, 2014.
[46] Garima Gupta and Rahul Katarya. Enpso: An automl technique for generating ensemble recommender system. Arabian Journal for Science and Engineering, 46(9):8677-8695, 2021.
[47] Srijan Gupta. Auto-caserec: A novel automated recommender system framework, 2020.
[48] Malay Haldar, Mustafa Abdool, Prashant Ramanathan, Tao Xu, Shulin Yang, Huizhong Duan, Qing Zhang, Nick Barrow-Willin deep learning to airbub search. In Proceedings of the 25th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, pages 1927- 2019.
Bradley C Turnbull, Brendan M Collins, et al. Applying
[49] F. Maxwell Harper and Joseph A. Konstan. The movielens datasets: History and context. ACM Trans. Interact. Intell. Syst., 5(4), dec 2015.
[50] Kaiming He, Ross Girshick, and Piotr Dollár. Rethinking imagenet pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4918-4927, 2019.
13

---

[51] Yifan Hu, Yehuda Koren, and Chris Volinsky. Collaborative filtering for implicit feedback datasets, In 2008 Eighth IEEE international conference on data mining, pages 263-272. Icee, 2008.
[52] Zan Huang and Daniel Dajun Zeng. Why does collaborative filtering work? transaction- based recommendation model validation and selection by analyzing bipartite random graphs. INFORMS Journal on Computing, 23(1):138-155 2011.
[53] Nicolas Hug. Surprise: A python library for recommender systems. Journal of Open Source Software, 5(52):2174, 2020.
[54] Dietmar Jannach, Markus Zanker, Alexander Felfernig, and Gerhard Friedrich. Recommender systems: an introduction. Cambridge University Press, 2010.
[55] Ray Jiang, Silvia Chiappa, Tor Lattimore, András György, and Pushmeet Kohli. Degenerate feedback loops in recommender systems. Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, pages 383-390, 2019.
[56] Yehuda Koren. Factorization meets the neighborhood: A multifaceted collaborative filtering model. In Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '08, page 426-434, New York, NY, USA, 2008 Association for Computing Machinery.
[57] Yehuda Koren, Robert Bell, and Chris Volinsky. Matrix factorization techniques for recom- mender systems. Computer, 42(8):30-37, 2009.
[58] Pigi Kouki, Ilias Fountalis, Nikolaos Vasiloglou, Xiquan Cui, Edo Liberty, and Khalifeh Al Jadda. From the lab production: A case study of bession-based recommendations in the nome-improvement domain. In Fourteenth ACM conference on recommender systems, pages 140-149, 2020.
[59] Jérôme Kunegis. Konect: The koblenz network collection. In Proceedings of the 22nd International Conference World Wide Web, WWW "13 Companion, page 1343-1350, New York, NY, USA, 2013, Association for Computing Machinery.
[60] Jérôme Kunegis, Gerd Gröner, and Thomas Gottron. Online dating recommender systems: The split-complex number approach. In Proceedings of the 4th ACM RecSys Workshop on Recommender Systems and the Social Web, RSWeb '2, page 37-44, New York, NY, USA, 2012. Association for Computing Machinery.
[61] Daniel Lemire and Anna Maclachlan. Slope one predictors for online rating-based collaborative filtering. In Proceedings of the 2005 SIAM International Conference on Data Mining, pages 471-475. SIAM, 2005.
[62] Mark Levy and Kris Jack. Efficient top-n recommendation by linear regression, 2013,
[63] Omer Levy and Yoav Goldberg. Neural word embedding as implicit matrix factorization. Advances neural information processing systems, 27, 2014.
[64] Dawen Liang, Rahul G. Krishnan, Matthew D. Hoffman, and Tony Jebara. Variational autoen- coders for collaborative filtering. In Proceedings of the 2018 World Wide Web Conference, WWW '1, page 689-698, Republic and Canton of Geneva, CHE, 2018. International World Wide Web Conferences Steering Committee.
[65] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach, 2019.
[66] Bodhisattwa Prasad Majumder, Shuyang Li, Jianmo Ni, and Julian McAuley. Generating personalized recipes from historical user preferences. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP--JCNIPP) pages 5976-5982, Hong Kong, China, November 2019. Association for Computational Linguistics.
14

---

[67] Paolo Massa and Paolo Avesani. Trust-aware recommender systems. In Proceedings of the 2007 ACM Conference on Recommender Systems, RecSys '07, page 17-24, New York, NY. USA, 2007. Association for Computing Machinery.
[68] Paolo Massa, Kasper Souren, Martino Salvetti, and Danilo Tomasoni. Trustlet, open research on trust metries. Scalable Computing: Practice and Experience, 9(4), 2008.
[69] Virginia Morini, Laura Pollacci, and Giulio Rossetti. Toward standard approach for echo chamber detection: Reddit case study. Applied Sciences, 1 1112))3390, 2021.
[70] Allan H. Murphy, The finley affair: signal event in the history forecast verification. Weather and Forecasting, 11(1):3 200 1996.
[71] Jianmo Ni, Jiacheng Li, and Julian McAuley. Justifying recommendations using distandly labeled reviews and fin-grained aspects. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLL) pages 188-199, Hong Kong, China, November 2019. Association for Computational Linguistics.
[72] Bibek Paudel, Fabian Christoffel, Chris Newell, and Abraham Bernstein. Updatable, accurate, diverse, and scalable recommendations for interactive applications. ACM Transactions on Interactive Intelligent Systems (Tiis), 7(1):1-34, 2016.
[73] Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme. BPR: Bayesian personalized ranking from implicit feedback. In Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence, UAI '09, page 452-461, Arlington, Virginia, USA, 2009. AUAI Press.
[74] Paul Resnick. Neophytos lacovou, Mitesh Suchak, Peter Bergstrom, and John Riedl. Grouplens: An open architecture for collaborative filtering of netnews. In Proceedings of the 1994 ACM Conference Computer Supported Cooperative Work, CSCW '94, page 175-186, New York, NY, USA, 1994. Association for Computing Machinery.
[75] Tal Ridnik, Emanuel Ben-Baruch, Asaf Noy, and Lihi Zelnik. Imagenet-21K pretraining for the masses. In J. Vanschoren and S. Yeung, editors, Proceedings of the Neural Information Processing Systems Truck on Datasets and Benchmarks, volume 1, 2021.
[76] Badrul Sarwar, George Karypis, Joseph Konstan, and John Riedl. Item-based collaborative filtering recommendation algorithms. In Proceedings of the 10th International Conference on World Wide Web, WWW '01, page Computing Machinery.
New York, NY, USA, 2001, Association for
[77] Brent Smith and Greg Linden. Two decades of recommender systems at amazon. com. leee internet computing, 21(3))12-18, 2017.
[78] Harald Steck. Embarrassingly shallow autoencoders for sparse data. In The World Wide Web Conference, pages 3251 3251-3257, 2017,
[79] Amos Tversky. Features of similarity. Psychological review, 84(4):327, 1977.
[80] Mengting Wan, Jianmo Ni, Rishabh Misra, and Julian McAuley. Addressing Marketing Bias in Product Recommendations, page 618-626. Association for Computing Machinery, New York, NY, USA, 2020.
[81] Fan Wang. Xiaomin Fang, Lihang Liu, Yaxue Chen, Jiucheng Tao, Zhiming Peng, Cihang Jin, and Hao Tian. Sequential evaluation and generation framework for combinatorial recommender system. arXiv preprint arXiv: (ttv:/902.000 102400025, 2019.
[82] Ting- Hiig-H Wang, Xia Hu, Haifeng Jin, Qingquan Song, Xiaotian Han, and Zirui Liu. AutoRec: An Automated Recommender System, page 582-584. Association for Computing Machinery, New York, NY, USA, 2020.
[83] Lin Xu, Frank Hutter, Holger H. Hoos, and Kevin Leyton-Brown. SATzilla: portfolio-based algorithm selection for SAT. Journal of Artificial Intelligence Research, 32:565-666 606, June 2008.
15

---

[84] Shuai Zhang, Lina Yao, Aixin Sun, Sen Wang, Guodong Long, and Manqing Dong, Neurec: On nonlinear transformation for personalized ranking. In Proceedings of the Twenty-Seventh Inter- national Joint Conference on Artificial Intelligence, IJCAI-18, pages 3669-3675. International Joint Conferences on Artificial Intelligence Organization, 2018.
[85] Lei Zheng, Chun-Ta Lu, Fei Jiang, Jiawei Zhang, and Philip S. Yu. Spectral collaborative filtering. In Proceedings of the 12th ACM Conference on Recommender Systems, RecSys '18 page 31 511-319, New York, NY, USA, 2018. Association for Computing Machinery.
[86] Tao Zhou, Zoltán Kuscsik, Jian-Guo Liu, Matúš Medo, Joseph Rushton Wakeling, and Yi-Cheng Zhang. Solving the apparent diversity- dilemma of recommender systems. Proceedings of the National Academy of Sciences, (77(10)4551-4511 2010.
[87] Cai-Nicolas Ziegler, Sean M. McNee, Joseph A. Konstan, and Georg Lausen. Improving recommendation lists through topic diversification. In Proceedings of the 14th International Conference World Wide Web, WWW '05, page 22 22-32, New York, NY, USA, 2005. Associa- tion for Computing Machinery.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] [The main claims in the abstract and introduction reflect the paper's contributions and scope.]
(b) Did you describe the limitations of your work? [Yes] [See Section
(c) Did you discuss any potential negative societal impacts of your work? [Yes] [See Section
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] [We read the ethics review guidelines and ensured our paper conforms to them.]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A] [We did not include theoretical results.]
(b) Did you include complete proofs of all theoretical results? [N/A] [We did not include theoretical results.]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed reproduce the main ex- perimental results (either in the supplemental material or as a URL)? [Yes] [We include the code, data, and instructions reproduce the results here: https //github. cocsuuu..oo//aaaaaeeeeeeeaaeaaaa coom//asszill/rrciil111
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] [See Section and Appendix A
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes] [See Section 5
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] [We include this information in Section
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] [See Section 2 and Appendix |
(b) Did you mention the license of the assets? [N/A] Our experiments were conducted only publicly available datasets.
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
We did not include new assets.
16

---

(d) Did you discuss whether and how consent was obtained from people whose data you'le re using/curating? [N/A] Our experiments were conducted only on publicly available datasets.
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [N/A] Our experiments were conducted only on publicly available datasets.
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A] [We did not conduct research with human subjects.]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A] [We did not conduct research with human subjects.]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A] [We did not conduct research with human subjects.]
17


Reviews:
[{"review_id": "uS_9dY4zB4a", "reviewer": "Reviewer_3P6Z", "summary": "This paper conducts extensive experiments on 18 algorithms and 85 datasets in recommendation domain. Specifically, the authors found that “the best algorithms and hyperparameters are highly dependent on the dataset and performance metric”. As a result, the authors proposed RecZilla that uses as a model to predict the best recommenders with hyperparameters for new, unseen datasets.", "questions": "Some of my concerns/suggestions for improvements: \n\n1. Although the experiments are great, the contributions are not strong enough in my opinion. The conclusion of the first contribution is already stated in various previous works [1, 2, 3]. Thus, in my opinion, this work is great as a survey paper, but the contributions are incremental and not novel enough. \n2. Moreover, [4] also shows the idea of selecting algorithms for recsys, and I don’t see much difference in the performance of [4] compared to RecZilla according to Table 3.\n3. Why didn’t we consider deep learning based algorithms? The 18 proposed algorithms are already introduced in [1, 2]. In fact, the authors also used the implementations from [1, 2]. Any specific reasons to discard the deep learning approaches such as GRU4Rec, etc?\n4. Why only focus on Prec@10 in the Experimental Setup section? Isn’t it yield ‘bias’ as well since we only focus on one single metric and optimize for that metric?\n5. [Minor] Typo in line 172: “are” instead of “aare”, missing related work [3]\n\n[1] Are We Really Making Much Progress? A Worrying Analysis of Recent Neural Recommendation Approaches. RecSys 2019.\n[2] A troubling analysis of reproducibility and progress in recommender systems research. TOIS 2021.\n[3] The Datasets Dilemma: How Much Do We Really Know About Recommendation Datasets?. WSDM 2022\n[4] Deep Learning based Recommender System: A Survey and New Perspectives. ACM Computing Surveys 2019.\n[5] CF4CF: Recommending Collaborative Filtering algorithms using Collaborative Filtering. RecSys 2018.\n\nOverall, I’m impressed by the extensive experiments, but I’m not convinced from the novelty of the contributions. The contributions are incremental only to me. Since the authors mentioned a lot about “practice” and “practitioners”, I believe the authors could emphasize more on this perspective in subsequent versions to make the contributions become more interesting. For example, how the researchers and practitioners benefit from this work? Any insights from practical perspective? \n", "limitations": "The authors did address the limitations and potential negative social impact very well.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Some strengths:\n\n1. The experiments of the paper are very extensive and great. \n2. The paper considers 19 algorithms with 85 datasets, which cover the majority of recsys datasets \n3. The Appendix shows good results with supplementary materials\n4. It’s also good that the authors are going to release the source code for reproducibility  \n\n\nSome of my concerns/suggestions for improvements: \n\n1. Although the experiments are great, the contributions are not strong enough in my opinion. The conclusion of the first contribution is already stated in various previous works [1, 2, 3]. Thus, in my opinion, this work is great as a survey paper, but the contributions are incremental and not novel enough. \n2. Moreover, [4] also shows the idea of selecting algorithms for recsys, and I don’t see much difference in the performance of [4] compared to RecZilla according to Table 3.\n3. Why didn’t we consider deep learning based algorithms? The 18 proposed algorithms are already introduced in [1, 2]. In fact, the authors also used the implementations from [1, 2]. Any specific reasons to discard the deep learning approaches such as GRU4Rec, etc?\n4. Why only focus on Prec@10 in the Experimental Setup section? Isn’t it yield ‘bias’ as well since we only focus on one single metric and optimize for that metric?\n5. [Minor] Typo in line 172: “are” instead of “aare”, missing related work [3]\n\n[1] Are We Really Making Much Progress? A Worrying Analysis of Recent Neural Recommendation Approaches. RecSys 2019.\n[2] A troubling analysis of reproducibility and progress in recommender systems research. TOIS 2021.\n[3] The Datasets Dilemma: How Much Do We Really Know About Recommendation Datasets?. WSDM 2022\n[4] Deep Learning based Recommender System: A Survey and New Perspectives. ACM Computing Surveys 2019.\n[5] CF4CF: Recommending Collaborative Filtering algorithms using Collaborative Filtering. RecSys 2018.\n\nOverall, I’m impressed by the extensive experiments, but I’m not convinced from the novelty of the contributions. The contributions are incremental only to me. Since the authors mentioned a lot about “practice” and “practitioners”, I believe the authors could emphasize more on this perspective in subsequent versions to make the contributions become more interesting. For example, how the researchers and practitioners benefit from this work? Any insights from practical perspective? \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ppycnMAdaBp", "reviewer": "Reviewer_J9aR", "summary": "The paper presents a large-scale analysis of the performance of existing recommender system algorithms on multiple (85) datasets on 315 performance metrics. The study reveals that the algorithm’s performance is highly susceptible to the performance metric and dataset’s meta features. Thus, they propose an automated algorithm selection approach that chooses the best algorithm for a new, unseen dataset based on these findings.", "questions": "See above.", "limitations": "The authors have adequately discussed limitations of the work and their broader impacts on the society. ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n•\tTheirs is the first large scale study that compares recommender system algorithms across many metrics and datasets. The study is thorough, and the authors have covered almost all metrics, datasets and meta-features of the dataset.\n•\tThey open source their training data as well as pretrained models that can be beneficial for practitioners to choose best performing model for their problem specifications.\n\nWeakness\n•\tThe models used in the study are traditional and simplistic in the recommender literature. For instance, they have compared clustering models, Matrix factorization models and linear models. The literature has evolved to use more sophisticated models that might be less susceptible to the observed performance variations. Thus, selecting the best algorithm out of those selected from their proposed model might be best suited for a reliable baseline. Do the authors have plan to extend it to more sophisticated (non-linear, graph-based models) in the future? What will be the challenges for using such models in the current framework?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "V0dDaPfvgbt", "reviewer": "Reviewer_a5Pn", "summary": "The paper addresses the problem of model selection\nfor recommender systems. The authors investigate the \nperformances of 18 different recommender models on \n19 datasets and 23 metrics and find, that item-based \nnearest neighbor performs best on average, but each \nmodel performs best for one dataset/metric pair and\nvery bad for another one. In a second step they transfer\nthe SatZilla algorithm selector [76] to selecting recommender\nsystem models and show that it outperforms two \nexisting such model selectors from the literature.\n", "questions": "The paper addresses an import question of current research\nin recommender systems: can some existing models be fitted\nuniversally to any dataset/metric pair? And if not, can we predict,\nwhich model will have the best performance for a specific \ndataset/metric pair?\n\nThe authors conduct a large-scale study that alone will be valuable\nfor the field in my opinion. Also their meta learning method \n\"RecZilla\" sets a good basis for future research. \n\nGiven all the positive aspects, I see nevertheless some weak points:\nw1. it is not clear if the hyperparameters of the methods\n  have been optimized on train or test.\n  On p.4 you just write \"First we identify the best-performing \n  hyperparameter set for each (algorithm,dataset) pair\". \n  But do you use the performance on train or on test? If on test,\n  you systematically overestimate the performance of the algorithms.\n  And can you clarify: you do this separately for each metric? \n\nw2. The authors use 18 different models from the literature,\n  but there is no neural network among those models.\n  I understand the motivation: that the choosen models likely\n  are way faster to train. And that Dacrema et al. [26] found\n  them not to perform competitively. But nevertheless, it feels\n  the study misses an important aspect this way.\n  \nw3. The comparison against other model selectors for recommender\n  systems is only in the appendix B.4.\n  To me this looks like a crucial aspect of your paper and \n  I wondered why it has been banned to the appendix. \n  For example, are the sensitivity analyses in fig. 4 really\n  more important?\n\nSmaller points:\n- p. 4 \"for each dataset, we compute a train and test split\". \n  Given your earlier definition '\"dataset\" refers to a single \n  train-test split\" (p.3), this is a little bit confusing.\n- several of your references miss their venue, e.g., [6] is a \n  RecSys paper.\n\ntypos:\n- p.5 \"aare\"\n- p.6 \"leave-one-out leave-one-out\"\n", "limitations": "Yes.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "strong points:\ns1. large-scale meta study for recommender systems.\ns2. transfer of the SatZilla algorithm selector to recommender\n  systems.\n\nweak points:\nw1. it is not clear if the hyperparameters of the methods\n  have been optimized on train or test.\nw2. no neural networks are among the tested models.\nw3. The comparison against other model selectors for recommender\n  systems is only in the appendix.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "JRpUdnQCZKc", "reviewer": "Reviewer_s1pX", "summary": "This paper studies the impact of different datasets, algorithms, and hyperparameters on the performance of recommender systems through a large number of experiments. It proves that different datasets and algorithms in the recommender system significantly impact the performance, and they are not general but predictable. Further, this paper proposes RecZilla, a meta-learning method that predicts the best-performing algorithm and hyperparameters on new datasets by inputting meta-features. The authors show that RecZilla quickly ﬁnds high-performing algorithms on datasets it has never seen before.", "questions": "1. In terms of algorithm selection, why most of the selected methods are earlier works? Although it might be because their computational complexity is low and it is easy to conduct experiments, I still think that more advanced algorithms should be considered and evaluated.\n2. How do the authors justify the value of the proposed method in practical recommender systems, as the datasets are larger and the algorithms are more complex.", "limitations": "Yes, but not good enough. There is still a large room for discussing the remaining gap in the presented work from the realistic recommendation scenarios.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1. Since choosing algorithms and hyperparameters for recommender systems has always depended on human experience, this paper targets an important problem in applying ML in industrial practices.\n2. The claims of this paper have been validated by a large number of experiments. To verify the generality and predictability of the recommender system, the authors have selected a large number of datasets, algorithms, and hyperparameters to conduct experiments.\n3. The authors have open-sourced the experimental code, which is essential for an empirical study.\n\nWeakness\n\nDespite the extensive experiments by the authors, I remain skeptical about the generalizability of this method. \n1. First of all, in the selection of algorithms, most of the algorithms selected by the author were published earlier, such as SVD and MF, and many papers were published before 2015. These methods have rarely been used in industrial recommendation systems nowadays, as neural network-based methods have become popular in recent years. \n2. Secondly, most of the datasets used by the author come from public datasets, which makes this paper highly credible and reproducible. However, in real-world recommendation systems, the user's behavior is not consistent in different scenarios, such as different platforms, user interfaces, etc., which may affect the distribution of datasets. Although the author provides a pre-trained model, it is still doubtful that this model can work across different empirical datasets.\n3. If new datasets are added, the cost of model training and updating appears to be very large because of the large number of new data points that need to be collected.\n\nSeveral typos:\\\n    line 172: 'aare'\\\n    line 182: 'leave-one-out'", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept