Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper studies the impact of different datasets, algorithms, and hyperparameters on the performance of recommender systems through a large number of experiments. It proves that different datasets and algorithms in the recommender system significantly impact the performance, and they are not general but predictable. Further, this paper proposes RecZilla, a meta-learning method that predicts the best-performing algorithm and hyperparameters on new datasets by inputting meta-features. The authors show that RecZilla quickly ﬁnds high-performing algorithms on datasets it has never seen before.
strengths_and_weaknesses: Strengths
1. Since choosing algorithms and hyperparameters for recommender systems has always depended on human experience, this paper targets an important problem in applying ML in industrial practices.
2. The claims of this paper have been validated by a large number of experiments. To verify the generality and predictability of the recommender system, the authors have selected a large number of datasets, algorithms, and hyperparameters to conduct experiments.
3. The authors have open-sourced the experimental code, which is essential for an empirical study.

Weakness

Despite the extensive experiments by the authors, I remain skeptical about the generalizability of this method. 
1. First of all, in the selection of algorithms, most of the algorithms selected by the author were published earlier, such as SVD and MF, and many papers were published before 2015. These methods have rarely been used in industrial recommendation systems nowadays, as neural network-based methods have become popular in recent years. 
2. Secondly, most of the datasets used by the author come from public datasets, which makes this paper highly credible and reproducible. However, in real-world recommendation systems, the user's behavior is not consistent in different scenarios, such as different platforms, user interfaces, etc., which may affect the distribution of datasets. Although the author provides a pre-trained model, it is still doubtful that this model can work across different empirical datasets.
3. If new datasets are added, the cost of model training and updating appears to be very large because of the large number of new data points that need to be collected.

Several typos:\
    line 172: 'aare'\
    line 182: 'leave-one-out'
questions: 1. In terms of algorithm selection, why most of the selected methods are earlier works? Although it might be because their computational complexity is low and it is easy to conduct experiments, I still think that more advanced algorithms should be considered and evaluated.
2. How do the authors justify the value of the proposed method in practical recommender systems, as the datasets are larger and the algorithms are more complex.
limitations: Yes, but not good enough. There is still a large room for discussing the remaining gap in the presented work from the realistic recommendation scenarios.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper addresses the problem of model selection
for recommender systems. The authors investigate the 
performances of 18 different recommender models on 
19 datasets and 23 metrics and find, that item-based 
nearest neighbor performs best on average, but each 
model performs best for one dataset/metric pair and
very bad for another one. In a second step they transfer
the SatZilla algorithm selector [76] to selecting recommender
system models and show that it outperforms two 
existing such model selectors from the literature.

strengths_and_weaknesses: strong points:
s1. large-scale meta study for recommender systems.
s2. transfer of the SatZilla algorithm selector to recommender
  systems.

weak points:
w1. it is not clear if the hyperparameters of the methods
  have been optimized on train or test.
w2. no neural networks are among the tested models.
w3. The comparison against other model selectors for recommender
  systems is only in the appendix.

questions: The paper addresses an import question of current research
in recommender systems: can some existing models be fitted
universally to any dataset/metric pair? And if not, can we predict,
which model will have the best performance for a specific 
dataset/metric pair?

The authors conduct a large-scale study that alone will be valuable
for the field in my opinion. Also their meta learning method 
"RecZilla" sets a good basis for future research. 

Given all the positive aspects, I see nevertheless some weak points:
w1. it is not clear if the hyperparameters of the methods
  have been optimized on train or test.
  On p.4 you just write "First we identify the best-performing 
  hyperparameter set for each (algorithm,dataset) pair". 
  But do you use the performance on train or on test? If on test,
  you systematically overestimate the performance of the algorithms.
  And can you clarify: you do this separately for each metric? 

w2. The authors use 18 different models from the literature,
  but there is no neural network among those models.
  I understand the motivation: that the choosen models likely
  are way faster to train. And that Dacrema et al. [26] found
  them not to perform competitively. But nevertheless, it feels
  the study misses an important aspect this way.
  
w3. The comparison against other model selectors for recommender
  systems is only in the appendix B.4.
  To me this looks like a crucial aspect of your paper and 
  I wondered why it has been banned to the appendix. 
  For example, are the sensitivity analyses in fig. 4 really
  more important?

Smaller points:
- p. 4 "for each dataset, we compute a train and test split". 
  Given your earlier definition '"dataset" refers to a single 
  train-test split" (p.3), this is a little bit confusing.
- several of your references miss their venue, e.g., [6] is a 
  RecSys paper.

typos:
- p.5 "aare"
- p.6 "leave-one-out leave-one-out"

limitations: Yes.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper presents a large-scale analysis of the performance of existing recommender system algorithms on multiple (85) datasets on 315 performance metrics. The study reveals that the algorithm’s performance is highly susceptible to the performance metric and dataset’s meta features. Thus, they propose an automated algorithm selection approach that chooses the best algorithm for a new, unseen dataset based on these findings.
strengths_and_weaknesses: Strengths
•	Theirs is the first large scale study that compares recommender system algorithms across many metrics and datasets. The study is thorough, and the authors have covered almost all metrics, datasets and meta-features of the dataset.
•	They open source their training data as well as pretrained models that can be beneficial for practitioners to choose best performing model for their problem specifications.

Weakness
•	The models used in the study are traditional and simplistic in the recommender literature. For instance, they have compared clustering models, Matrix factorization models and linear models. The literature has evolved to use more sophisticated models that might be less susceptible to the observed performance variations. Thus, selecting the best algorithm out of those selected from their proposed model might be best suited for a reliable baseline. Do the authors have plan to extend it to more sophisticated (non-linear, graph-based models) in the future? What will be the challenges for using such models in the current framework?

questions: See above.
limitations: The authors have adequately discussed limitations of the work and their broader impacts on the society.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper conducts extensive experiments on 18 algorithms and 85 datasets in recommendation domain. Specifically, the authors found that “the best algorithms and hyperparameters are highly dependent on the dataset and performance metric”. As a result, the authors proposed RecZilla that uses as a model to predict the best recommenders with hyperparameters for new, unseen datasets.
strengths_and_weaknesses: Some strengths:

1. The experiments of the paper are very extensive and great. 
2. The paper considers 19 algorithms with 85 datasets, which cover the majority of recsys datasets 
3. The Appendix shows good results with supplementary materials
4. It’s also good that the authors are going to release the source code for reproducibility  


Some of my concerns/suggestions for improvements: 

1. Although the experiments are great, the contributions are not strong enough in my opinion. The conclusion of the first contribution is already stated in various previous works [1, 2, 3]. Thus, in my opinion, this work is great as a survey paper, but the contributions are incremental and not novel enough. 
2. Moreover, [4] also shows the idea of selecting algorithms for recsys, and I don’t see much difference in the performance of [4] compared to RecZilla according to Table 3.
3. Why didn’t we consider deep learning based algorithms? The 18 proposed algorithms are already introduced in [1, 2]. In fact, the authors also used the implementations from [1, 2]. Any specific reasons to discard the deep learning approaches such as GRU4Rec, etc?
4. Why only focus on Prec@10 in the Experimental Setup section? Isn’t it yield ‘bias’ as well since we only focus on one single metric and optimize for that metric?
5. [Minor] Typo in line 172: “are” instead of “aare”, missing related work [3]

[1] Are We Really Making Much Progress? A Worrying Analysis of Recent Neural Recommendation Approaches. RecSys 2019.
[2] A troubling analysis of reproducibility and progress in recommender systems research. TOIS 2021.
[3] The Datasets Dilemma: How Much Do We Really Know About Recommendation Datasets?. WSDM 2022
[4] Deep Learning based Recommender System: A Survey and New Perspectives. ACM Computing Surveys 2019.
[5] CF4CF: Recommending Collaborative Filtering algorithms using Collaborative Filtering. RecSys 2018.

Overall, I’m impressed by the extensive experiments, but I’m not convinced from the novelty of the contributions. The contributions are incremental only to me. Since the authors mentioned a lot about “practice” and “practitioners”, I believe the authors could emphasize more on this perspective in subsequent versions to make the contributions become more interesting. For example, how the researchers and practitioners benefit from this work? Any insights from practical perspective? 

questions: Some of my concerns/suggestions for improvements: 

1. Although the experiments are great, the contributions are not strong enough in my opinion. The conclusion of the first contribution is already stated in various previous works [1, 2, 3]. Thus, in my opinion, this work is great as a survey paper, but the contributions are incremental and not novel enough. 
2. Moreover, [4] also shows the idea of selecting algorithms for recsys, and I don’t see much difference in the performance of [4] compared to RecZilla according to Table 3.
3. Why didn’t we consider deep learning based algorithms? The 18 proposed algorithms are already introduced in [1, 2]. In fact, the authors also used the implementations from [1, 2]. Any specific reasons to discard the deep learning approaches such as GRU4Rec, etc?
4. Why only focus on Prec@10 in the Experimental Setup section? Isn’t it yield ‘bias’ as well since we only focus on one single metric and optimize for that metric?
5. [Minor] Typo in line 172: “are” instead of “aare”, missing related work [3]

[1] Are We Really Making Much Progress? A Worrying Analysis of Recent Neural Recommendation Approaches. RecSys 2019.
[2] A troubling analysis of reproducibility and progress in recommender systems research. TOIS 2021.
[3] The Datasets Dilemma: How Much Do We Really Know About Recommendation Datasets?. WSDM 2022
[4] Deep Learning based Recommender System: A Survey and New Perspectives. ACM Computing Surveys 2019.
[5] CF4CF: Recommending Collaborative Filtering algorithms using Collaborative Filtering. RecSys 2018.

Overall, I’m impressed by the extensive experiments, but I’m not convinced from the novelty of the contributions. The contributions are incremental only to me. Since the authors mentioned a lot about “practice” and “practitioners”, I believe the authors could emphasize more on this perspective in subsequent versions to make the contributions become more interesting. For example, how the researchers and practitioners benefit from this work? Any insights from practical perspective? 

limitations: The authors did address the limitations and potential negative social impact very well.
review_title: null

Meta-review: metareview: The core idea is to specialize meta-learning approaches to recommender systems. The specialization is done using features on the dataset themselves so is different from usual autoML approaches. Some code is provided allowing easy comparison to a lot of well tuned baselines in the domain. Yet easily reusable it is also demonstrating that several papers accepted in the past few years were overclaiming because of lazy comparisons. It also formalize some experience that many practitionner have about the "good" algorithms to use depending of the metric ans data for recommender systems. 

Reviewers significantly updated their scores during the discussion phase as the authors ran a new set of comparisons and clarified some sections. I feel the work can be reused so I recommend an accept.


confidence: Less certain
Decision: Accept