On the Generalizability and Predictability of Recommender Systems

Abstract: While other areas of machine learning have seen more and more automation, designing a high-performing recommender system still requires a high level of human effort. Furthermore, recent work has shown that modern recommender system algorithms do not always improve over well-tuned baselines. A natural follow-up question is, "how do we choose the right algorithm for a new dataset and performance metric?" In this work, we start by giving the first large-scale study of recommender system approaches by comparing 24 algorithms and 100 sets of hyperparameters across 85 datasets and 315 metrics. We find that the best algorithms and hyperparameters are highly dependent on the dataset and performance metric. However, there is also a strong correlation between the performance of each algorithm and various meta-features of the datasets. Motivated by these findings, we create RecZilla, a meta-learning approach to recommender systems that uses a model to predict the best algorithm and hyperparameters for new, unseen datasets. By using far more meta-training data than prior work, RecZilla is able to substantially reduce the level of human involvement when faced with a new recommender system application. We not only release our code and pretrained RecZilla models, but also all of our raw experimental results, so that practitioners can train a RecZilla model for their desired performance metric: https://github.com/naszilla/reczilla.

Reviews and author responses:

 Thank you for your additional feedback. We agree with your minor comment about giving more details for practitioners leveraging Section 2. We have now updated Section C.2 with concrete examples and more details.

Note that the particular use cases and goals of a practitioner may be very specific (they may be concerned with one particular dataset, or many datasets; they may be very concerned with the training time and/or latency of the model; they may need to use a specific algorithm for various reasons). We give three examples: (1) computing the “hardness” of a dataset in advance, with concrete numbers, and what to do next, (2) predicting the training time and/or latency of a model, and (3) gaining concrete insights for a particular algorithm. 


 Thanks for the additional explanation. My concern regarding the algorithm selection has been resolved. I am also somewhat more convinced to adopt the practical value of this work, so I will raise my score from 4 to 5.

 Thanks for addressing the feedback. I have a look at the latest version on 03 August 2022, and it looks good to me. Thus, I increase my score from 4 to 7. Well done!

[Minor] I have one minor comment for Section C.2 from Line 950-959: the authors should give further details of key insights for practitioners as the current version seems too general in my opinion. For example, after practitioners calculate the entropy of the rating matrix following Table 8, do we have a range of values that we consider as 'low'? What would happen **next** if their dataset is considered as 'hard'? Should they change their algorithms (if yes, what's best practice)? I believe give as much details as possible for C.2 would further strengthen the paper, but it's just a minor comment. 



 What do you think of the clarifications brought by the authors? In particular  s1pX and 3P6Z do you have changed your mind ?

 Thank you for your excellent review. We are pleased to hear that you are impressed by the extensive experiments. Overall, your suggestions helped us to improve our paper, for example by adding six new deep learning algorithms which also further improves our meta-learner, running Section 3 with other objectives, and including a guide to practitioners. Thank you for your suggestions, and we give the details below.

**"1a. The contributions are not strong enough."**

We respectfully remind the reviewer that our work introduces the largest public repository (and analysis) of recommender systems datasets and algorithms, which in itself is a sizeable contribution. We have now added a guide to practitioners, so that users can get the most out of our work (Section C.2).

**"1.b not novel enough"**

We respectfully remind the reviewer that experimental survey-type papers have been recently accepted to ICML [6], ICLR [7,8], and NeurIPS [9]. Just in case the reviewer would like a refresher on these discussions, please see [here](https://openreview.net/forum?id=SJgIPJBFvH&noteId=8EatWJ_2_U), [here](https://openreview.net/forum?id=HygrdpVKvr&noteId=C5rPJ2sLzP), and [here](https://openreview.net/forum?id=6RB77-6-_oI&noteId=l5H13jujQKo).

- [6] Descending through a Crowded Valley - Benchmarking Deep Learning Optimizers, ICML 2021.
- [7] Fantastic Generalization Measures and Where to Find Them, ICLR 2020.
- [8] NAS evaluation is frustratingly hard, ICLR 2021.
- [9] How Powerful are Performance Predictors in Neural Architecture Search, NeurIPS 2021.

**"2. Similarities to [4]."**

We agree that Section 3 shares similarities with [4], however, the main focus of our work is studying the generalizability and predictability of a large variety of recommender system algorithms and datasets. Unlike [4], we open-source our code, which includes a common framework for all 85 datasets.

 [4] Deep Learning based Recommender System: A Survey and New Perspectives. ACM Computing Surveys 2019.

**"3. No deep learning based algorithms."**

We agree with your suggestion, and so we have added six deep learning based algorithms. We have now updated the results of Sections 2 and 3 with these new algorithms. We find that the deep learning algorithms also further improve our meta-learner compared to prior work [4], which helps to address your question 2 above, as well. In particular, the average %Diff (percent difference of predicted performance from best performance) improves from 35.1 to 33.2. while [4] achieved 52.9 and [5] achieved 43.5.

- [4] Deep Learning based Recommender System: A Survey and New Perspectives. ACM Computing Surveys 2019. 
- [5] CF4CF: Recommending Collaborative Filtering algorithms using Collaborative Filtering. RecSys 2018.

**"4. Only focus on PREC@10 in Section 3."**

We thank you for pointing this out. We already released pretrained models on three different objectives (mentioned at the end of Section 3.2). Following your suggestion, we also re-ran the experiments in Section 3 with COVERAGE@50 and HIT-RATE@5 (chosen to give a good variety) in addition to PREC@10. See the new results in Section C.3.

**"5. Minor comments."**

We have fixed the typo and included that reference. Thank you for pointing these out.

**"Overall emphasis on practitioners"**

We thank you for this suggestion. We have now added a guide to practitioners (Section C.2) which explains the key takeaways and insights from our analysis, as well as how to use our pre-trained models, so that practitioners can get the largest value from our work.

We once again thank you for these excellent suggestions (especially adding deep learning algorithms, adding experiments with more objectives, and adding a guide to practitioners). We respectfully ask that you please consider increasing your score if you find that our responses help to address your questions. We are also happy to continue answering follow-up questions. Thank you!


 Thank you for your insightful review. We are glad to see that you list our thorough comparison, open-source code, and pretrained models as strengths of our work. We address your concern below.

**"The models are traditional and simplistic"**

We agree, and so we have now updated our work to include six more sophisticated (deep learning based) algorithms: DELF_EF, DELF_MLP, INeuRec, MultiVAE, SpectralCF, and UNeuRec. See the updated list of algorithms [here](https://anonymous.4open.science/r/anon-reczilla-51FC/RecSys2019_DeepLearning_Evaluation/algorithm_handler.py), and see our updated paper for the new results.

We find that indeed some of the deep learning approaches are less susceptible to issues with generalizability. However, we still do see issues with generalizability across all algorithms (In Table 11, the best average rank achieved by the best deep learning approach, Mult-VAE, is still only 8.6) which is consistent with prior work [1]. However, the addition of the deep learning algorithms **do** improve the overall performance of RecZilla: %Diff (percent difference of predicted performance from best performance) improves from 35.1 to 33.2.

We thank the reviewer once again for their suggestion, since we agree that it improves the impact of our work. We respectfully ask that you please consider increasing your score, if you find the additions satisfactory. Otherwise, we are happy to answer additional questions or concerns.

[1] [A Troubling Analysis of Reproducibility and Progress in Recommender Systems Research](https://arxiv.org/abs/1911.07698)


 Thank you for your thoughtful review. We are glad to see that you find our work addresses an important question of current research, and that you believe the large-scale study is valuable to the field. We reply to your questions below.
 
**"W1. Unclear if hyper parameters are optimized on train or test"**

We use the validation set to optimize hyperparameters, and then we report the results on the test set. Thank you for clarifying; we have now made it more clear in the paper. Please see this excerpt from our [README](https://anonymous.4open.science/r/anon-reczilla-51FC/README.md)
as an example output from our DataSplitter function.

```
DataSplitter_global_timestamp: DataReader: Movielens100K
    Num items: 1682
    Num users: 751
    Train         interactions 79999,     density 6.33E-02
    Validation     interactions 1535,     density 1.22E-03
    Test         interactions 1418,     density 1.12E-03
```
 
**”W2. No neural networks."**

We agree that this is an important aspect, and therefore we have added six neural network approaches to our analysis. We have updated the experiments from Sections 2 and 3 to include these new algorithms. Please see more details in our [general comment](https://openreview.net/forum?id=wO53HILzu65&noteId=Bxmi2cLJW72), and in the updated paper.
 
**"W3. Comparisons to other models are in the appendix."**

We agree, and so we have now moved this to Section 3. Thank you for the suggestion.
 
**"Minor comments."**

We thank the reviewer for their close reading of our paper - we have fixed all the typos and confusing points.
 
We thank the reviewer once again for their comments and suggestions. If you have any follow-up questions, please let us know.

 Thank you for your thoughtful review. We are glad to hear that you feel we target an important problem and that we validate our claims with a large number of experiments. We reply to your comments below.

**"1.  Most algorithms are older methods."**

We agree with this concern, and so we have added six deep learning algorithms to our experiment codebase. We have incorporated these algorithms into our results in Sections 2 and 3 (see the details in the [general comment](https://openreview.net/forum?id=wO53HILzu65&noteId=Bxmi2cLJW72), and in the updated paper). We agree with you that this update gives our work more practical significance.

**" 2. User’s behavior is not consistent."**

We are not sure we understand the reviewer’s question (please let us know, thank you). All 85 datasets used in our work **do** come from real-world settings, from different platforms and user interfaces, and therefore we have a diverse distribution of datasets. Since our work considers by far the largest number of datasets out of any recommender system paper, this is our biggest strength compared to prior published work. See all datasets [here](https://anonymous.4open.science/r/anon-reczilla-51FC/RecSys2019_DeepLearning_Evaluation/dataset_handler.py).

If the reviewer meant that in real-world settings, the user’s preferences are constantly changing over time, then we completely agree, and including approaches that specifically focus on dynamically changing preferences is an exciting avenue for future work. 

**"3. Cost of adding new datasets is large."**

We agree that ironically, since our experiments were so thorough, adding one new dataset across **all** settings would require running it on 20 algorithms and 100 hyperparameters each. However, even evaluating on a subset of these settings would give a very useful addition to our meta-dataset. For example, due to the limitations mentioned in Section 4 and Appendix A.1, our current results do not include all possible combinations of datasets, algorithms, and hyperparameters (but they still include 84,769 successful experiments).

**"Justify the value in practice, such as the datasets are larger and the algorithms are more complex."**

To help answer your overall concern about the value in practice, we added a guide for practitioners in Section C.2, including key takeaways from our analysis, as well as how to use our pre-trained RecZilla models. Regarding datasets, as shown in Table 6 of Section A.3, the size of our datasets range from <100 interactions to more than 77 million interactions (Netflix, Yahoo, Amazon), which are among the *largest that we can include while keeping our repository public* (i.e., the largest public recommender system datasets). To address your concern about algorithms, we incorporated six new deep learning-based algorithms, as described above. 

**"Limitations"**

As requested, we have extended Section 4 to address the limitations you pointed out.

We thank you once again for raising these important points. We kindly ask that you consider updating your score if you think the additions we made improve our work. We are also happy to answer any follow-up questions or new comments. Thank you!


 Dear reviewers and AC, we have now addressed all of the suggestions and concerns mentioned by the reviewers. We thank the reviewers very much for these comments, which we agree has improved our work.

The primary weakness noted by all four reviewers was the lack of implementation of deep learning based recommender system algorithms. We are pleased to report that we have **now included six deep learning algorithms** in our experiments: [DELF_EF](https://www.ijcai.org/proceedings/2018/0462.pdf), 
[DELF_MLP](https://www.ijcai.org/proceedings/2018/0462.pdf), 
[INeuRec](https://arxiv.org/abs/1805.03002), 
[Mult-VAE](https://arxiv.org/abs/1802.05814), 
[SpectralCF](https://arxiv.org/abs/1808.10523), 
and [UNeuRec](https://arxiv.org/abs/1805.03002). 
See our [algorithm handler](https://anonymous.4open.science/r/anon-reczilla-51FC/RecSys2019_DeepLearning_Evaluation/algorithm_handler.py) and see our updated paper for the new results in Sections 2 and 3.

The full list of changes are as follows
 - Section 2 is updated with the six deep learning algorithms. Two of the algorithms (Mult-VAE and U-NeuRec) are comparable in performance to many of the 18 non-neural algorithms. However, the best algorithm across all settings is still Item-KNN.
 - Section 3 is updated with the deep learning algorithms. Reczilla now outperforms prior approaches by an even larger margin: average %Diff (percent difference of predicted performance from best performance) improves from 35.1 to 33.2 (the next-best, cf4cf-meta, is 43.5).
 - New section, Appendix C.2, **“A Guide for Practitioners”**, that describes the key takeaways for practitioners (including insights from our analysis, and how to use our pre-trained models) so that practitioners can use our work most effectively.
 - New section, Appendix C.3, that re-runs Section 3 experiments (which used PREC@10) with the base metric set to COVERAGE@50 and HITRATE@5.
 - The table comparing RecZilla with prior work is moved to Section 3.
 - Fixed other minor typos and clarifications.

We thank all reviewers once again for these suggestions. We are happy to address any new follow-ups or concerns.


 This paper studies the impact of different datasets, algorithms, and hyperparameters on the performance of recommender systems through a large number of experiments. It proves that different datasets and algorithms in the recommender system significantly impact the performance, and they are not general but predictable. Further, this paper proposes RecZilla, a meta-learning method that predicts the best-performing algorithm and hyperparameters on new datasets by inputting meta-features. The authors show that RecZilla quickly ﬁnds high-performing algorithms on datasets it has never seen before. Strengths
1. Since choosing algorithms and hyperparameters for recommender systems has always depended on human experience, this paper targets an important problem in applying ML in industrial practices.
2. The claims of this paper have been validated by a large number of experiments. To verify the generality and predictability of the recommender system, the authors have selected a large number of datasets, algorithms, and hyperparameters to conduct experiments.
3. The authors have open-sourced the experimental code, which is essential for an empirical study.

Weakness

Despite the extensive experiments by the authors, I remain skeptical about the generalizability of this method. 
1. First of all, in the selection of algorithms, most of the algorithms selected by the author were published earlier, such as SVD and MF, and many papers were published before 2015. These methods have rarely been used in industrial recommendation systems nowadays, as neural network-based methods have become popular in recent years. 
2. Secondly, most of the datasets used by the author come from public datasets, which makes this paper highly credible and reproducible. However, in real-world recommendation systems, the user's behavior is not consistent in different scenarios, such as different platforms, user interfaces, etc., which may affect the distribution of datasets. Although the author provides a pre-trained model, it is still doubtful that this model can work across different empirical datasets.
3. If new datasets are added, the cost of model training and updating appears to be very large because of the large number of new data points that need to be collected.

Several typos:\
    line 172: 'aare'\
    line 182: 'leave-one-out' 1. In terms of algorithm selection, why most of the selected methods are earlier works? Although it might be because their computational complexity is low and it is easy to conduct experiments, I still think that more advanced algorithms should be considered and evaluated.
2. How do the authors justify the value of the proposed method in practical recommender systems, as the datasets are larger and the algorithms are more complex. Yes, but not good enough. There is still a large room for discussing the remaining gap in the presented work from the realistic recommendation scenarios.

 The paper addresses the problem of model selection
for recommender systems. The authors investigate the 
performances of 18 different recommender models on 
19 datasets and 23 metrics and find, that item-based 
nearest neighbor performs best on average, but each 
model performs best for one dataset/metric pair and
very bad for another one. In a second step they transfer
the SatZilla algorithm selector [76] to selecting recommender
system models and show that it outperforms two 
existing such model selectors from the literature.
 strong points:
s1. large-scale meta study for recommender systems.
s2. transfer of the SatZilla algorithm selector to recommender
  systems.

weak points:
w1. it is not clear if the hyperparameters of the methods
  have been optimized on train or test.
w2. no neural networks are among the tested models.
w3. The comparison against other model selectors for recommender
  systems is only in the appendix.
 The paper addresses an import question of current research
in recommender systems: can some existing models be fitted
universally to any dataset/metric pair? And if not, can we predict,
which model will have the best performance for a specific 
dataset/metric pair?

The authors conduct a large-scale study that alone will be valuable
for the field in my opinion. Also their meta learning method 
"RecZilla" sets a good basis for future research. 

Given all the positive aspects, I see nevertheless some weak points:
w1. it is not clear if the hyperparameters of the methods
  have been optimized on train or test.
  On p.4 you just write "First we identify the best-performing 
  hyperparameter set for each (algorithm,dataset) pair". 
  But do you use the performance on train or on test? If on test,
  you systematically overestimate the performance of the algorithms.
  And can you clarify: you do this separately for each metric? 

w2. The authors use 18 different models from the literature,
  but there is no neural network among those models.
  I understand the motivation: that the choosen models likely
  are way faster to train. And that Dacrema et al. [26] found
  them not to perform competitively. But nevertheless, it feels
  the study misses an important aspect this way.
  
w3. The comparison against other model selectors for recommender
  systems is only in the appendix B.4.
  To me this looks like a crucial aspect of your paper and 
  I wondered why it has been banned to the appendix. 
  For example, are the sensitivity analyses in fig. 4 really
  more important?

Smaller points:
- p. 4 "for each dataset, we compute a train and test split". 
  Given your earlier definition '"dataset" refers to a single 
  train-test split" (p.3), this is a little bit confusing.
- several of your references miss their venue, e.g., [6] is a 
  RecSys paper.

typos:
- p.5 "aare"
- p.6 "leave-one-out leave-one-out"
 Yes.

 The paper presents a large-scale analysis of the performance of existing recommender system algorithms on multiple (85) datasets on 315 performance metrics. The study reveals that the algorithm’s performance is highly susceptible to the performance metric and dataset’s meta features. Thus, they propose an automated algorithm selection approach that chooses the best algorithm for a new, unseen dataset based on these findings. Strengths
•	Theirs is the first large scale study that compares recommender system algorithms across many metrics and datasets. The study is thorough, and the authors have covered almost all metrics, datasets and meta-features of the dataset.
•	They open source their training data as well as pretrained models that can be beneficial for practitioners to choose best performing model for their problem specifications.

Weakness
•	The models used in the study are traditional and simplistic in the recommender literature. For instance, they have compared clustering models, Matrix factorization models and linear models. The literature has evolved to use more sophisticated models that might be less susceptible to the observed performance variations. Thus, selecting the best algorithm out of those selected from their proposed model might be best suited for a reliable baseline. Do the authors have plan to extend it to more sophisticated (non-linear, graph-based models) in the future? What will be the challenges for using such models in the current framework?
 See above. The authors have adequately discussed limitations of the work and their broader impacts on the society. 

 This paper conducts extensive experiments on 18 algorithms and 85 datasets in recommendation domain. Specifically, the authors found that “the best algorithms and hyperparameters are highly dependent on the dataset and performance metric”. As a result, the authors proposed RecZilla that uses as a model to predict the best recommenders with hyperparameters for new, unseen datasets. Some strengths:

1. The experiments of the paper are very extensive and great. 
2. The paper considers 19 algorithms with 85 datasets, which cover the majority of recsys datasets 
3. The Appendix shows good results with supplementary materials
4. It’s also good that the authors are going to release the source code for reproducibility  


Some of my concerns/suggestions for improvements: 

1. Although the experiments are great, the contributions are not strong enough in my opinion. The conclusion of the first contribution is already stated in various previous works [1, 2, 3]. Thus, in my opinion, this work is great as a survey paper, but the contributions are incremental and not novel enough. 
2. Moreover, [4] also shows the idea of selecting algorithms for recsys, and I don’t see much difference in the performance of [4] compared to RecZilla according to Table 3.
3. Why didn’t we consider deep learning based algorithms? The 18 proposed algorithms are already introduced in [1, 2]. In fact, the authors also used the implementations from [1, 2]. Any specific reasons to discard the deep learning approaches such as GRU4Rec, etc?
4. Why only focus on Prec@10 in the Experimental Setup section? Isn’t it yield ‘bias’ as well since we only focus on one single metric and optimize for that metric?
5. [Minor] Typo in line 172: “are” instead of “aare”, missing related work [3]

[1] Are We Really Making Much Progress? A Worrying Analysis of Recent Neural Recommendation Approaches. RecSys 2019.
[2] A troubling analysis of reproducibility and progress in recommender systems research. TOIS 2021.
[3] The Datasets Dilemma: How Much Do We Really Know About Recommendation Datasets?. WSDM 2022
[4] Deep Learning based Recommender System: A Survey and New Perspectives. ACM Computing Surveys 2019.
[5] CF4CF: Recommending Collaborative Filtering algorithms using Collaborative Filtering. RecSys 2018.

Overall, I’m impressed by the extensive experiments, but I’m not convinced from the novelty of the contributions. The contributions are incremental only to me. Since the authors mentioned a lot about “practice” and “practitioners”, I believe the authors could emphasize more on this perspective in subsequent versions to make the contributions become more interesting. For example, how the researchers and practitioners benefit from this work? Any insights from practical perspective? 
 Some of my concerns/suggestions for improvements: 

1. Although the experiments are great, the contributions are not strong enough in my opinion. The conclusion of the first contribution is already stated in various previous works [1, 2, 3]. Thus, in my opinion, this work is great as a survey paper, but the contributions are incremental and not novel enough. 
2. Moreover, [4] also shows the idea of selecting algorithms for recsys, and I don’t see much difference in the performance of [4] compared to RecZilla according to Table 3.
3. Why didn’t we consider deep learning based algorithms? The 18 proposed algorithms are already introduced in [1, 2]. In fact, the authors also used the implementations from [1, 2]. Any specific reasons to discard the deep learning approaches such as GRU4Rec, etc?
4. Why only focus on Prec@10 in the Experimental Setup section? Isn’t it yield ‘bias’ as well since we only focus on one single metric and optimize for that metric?
5. [Minor] Typo in line 172: “are” instead of “aare”, missing related work [3]

[1] Are We Really Making Much Progress? A Worrying Analysis of Recent Neural Recommendation Approaches. RecSys 2019.
[2] A troubling analysis of reproducibility and progress in recommender systems research. TOIS 2021.
[3] The Datasets Dilemma: How Much Do We Really Know About Recommendation Datasets?. WSDM 2022
[4] Deep Learning based Recommender System: A Survey and New Perspectives. ACM Computing Surveys 2019.
[5] CF4CF: Recommending Collaborative Filtering algorithms using Collaborative Filtering. RecSys 2018.

Overall, I’m impressed by the extensive experiments, but I’m not convinced from the novelty of the contributions. The contributions are incremental only to me. Since the authors mentioned a lot about “practice” and “practitioners”, I believe the authors could emphasize more on this perspective in subsequent versions to make the contributions become more interesting. For example, how the researchers and practitioners benefit from this work? Any insights from practical perspective? 
 The authors did address the limitations and potential negative social impact very well.

Meta-review: The core idea is to specialize meta-learning approaches to recommender systems. The specialization is done using features on the dataset themselves so is different from usual autoML approaches. Some code is provided allowing easy comparison to a lot of well tuned baselines in the domain. Yet easily reusable it is also demonstrating that several papers accepted in the past few years were overclaiming because of lazy comparisons. It also formalize some experience that many practitionner have about the "good" algorithms to use depending of the metric ans data for recommender systems. 

Reviewers significantly updated their scores during the discussion phase as the authors ran a new set of comparisons and clarified some sections. I feel the work can be reused so I recommend an accept.



Decision: Accept