Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The work focuses on a very practical problem of designing a good a two-stage recommender systems. The authors shed light on interaction between the two stages (called nominator and ranker) and its effects on an overall performance of the solution. They theoretically prove that the way nominators are trained may significantly affect the end quality and provide an empirical evidence for their findings as well. The authors also provide a recipe for designing a better first stage based on a mixture-of-experts approach that turns out to advantageous over naïve approach in certain cases.

main_review: I totally agree with the authors that there's a lack of research devoted to two-stage recommenders. Indeed, there are many reports from different companies on the use-cases where such systems provide more advantages over standalone monolithic solutions. This observation is also supported by various recsys challenges where two-stage solutions are also demonstrated to provide highly competitive results. Hence, attempts to better understand their internals and working mechanics are more than welcomed. The work provides good theoretical grounds for the considered cases within the specified constraints and nicely connects it to experimental part.

It should be noted though, the focus is made specifically on bandits-based solutions, which is not the only possible setup. It would be great to see (at least at high level) some connections to other approaches, e.g. when ranker is based on Decision Trees and nominator is based on matrix factorization, which is a very popular setup. Surely, this may lead to many difficulties with analysis and may not even give a way for proper theoretical investigation, but it would be helpful to at least specify the limitations. Nevertheless, the lack of such analysis doesn't decrease the value of the provided results.

Related to the previous comment, an abstract creates an impression of a general solution, while only the case of bandits is considered. I'd suggest to modify the abstract and the introductory part of the text to more clearly position the work and define its boundaries.

I also have a few concerns regarding the experimental part and the assumptions made there. In Section 3.1, the synthetic dataset is generated under the assumption that context follows normal distribution. However, in almost all practical cases, items in recommender systems follow power law or zipf-like distributions, which, I believe, would contradict to the described assumption. It's not clear how this would affect further derivations and provided intuitions. Describing how different assumptions on the nature of contexts may affect the theoretical and/or experimental results of this work would help to better understand its practicality.

The data preprocessing part also seems to deviate form real scenarios. E.g., in Section 4 (lines 273-275): why sampling based on label frequency is performed? How does it affect distribution of data? How it relates to the assumptions on data used in theoretical part?  What is the connection to real world datasets? Please, add more details on the reasons for the described preprocessing.
limitations_and_societal_impact: The authors briefly mention connections of recommender systems to some of the issues like biases, fairness, etc. and do not explore it further. However, even for single-stage recommenders this is a very challenging topic. Given that the problem of two-stage recommender systems is underexplored, I believe that putting too much focus on societal and ethical concerns in the presented work wouldn't be helpful. The demonstrated awareness of the problem is enough in this case.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper studies on the properties of two-stage recommendation architecture. Both empirical and theoretical analyses have been conducted to reveal the impact of pool allocation on recommendation performance. The authors further propose a MOE-based learning strategy for pool allocation, which yields empirical improvement over the random pool allocation.
main_review: Strengths:
1. This paper studies on the properties of the two-stage recommendation architecture. It is an interesting and under-explored topic.
2. Both empirical and theoretical analyses are conducted to validate the importance of pool allocation on recommendation performance.
3. The authors further propose to learn pool allocation with MOE strategy, which yields empirical improvement over the random pool allocation.

Weaknesses:

1. The analyses are based on some strong assumptions, which may not be consistent with the setting of real-world two-stage recommendation system.

For example, in real-world RS, we usually use complex model (e.g. MF, FM, DeepFM) instead of simple linear model; Also, each nominator may return a list of items rather than single item; What’s more serious, the assigned pool of nominators in real-world RS usually covers all items (ie. A_n=A). I do not understand why we need pool allocation in this scenario. 

2. The proof of Proposition 2 is not complete. 
The proof is on the specific case with two nominators and three items, which lacks generalization.

3. Some important concepts lack clear interpretation, making the paper hard to follow. 
For example, what does “context distribution” mean? Does it mean \pi(x), or mean the distribution of x over the items in the pool? Also, in figure 2, what does “ G” or “U” mean? Does it mean one-stage recommender?

4. Insufficient motivation of using MOE.

We do agree that learning pool allocation is promising. But whether MOE can learn a proper allocation is questionable. More concretely, we do not understand why the weights returned by the MOE is a useful signal for pool allocation. The authors should give more deep analyses on MOE and explain why MOE is so special for pool allocation.
 

In summary, although this paper studies an interesting and important problem, it has some serious limitations in terms of presentation and theoretical soundness. As such, I think this paper is potential but this version is not touch the bar of NIPS.

limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: In this paper, the authors are trying to optimize two stage recommendation systems (first stage to generate candidates and second stage to rank the candidates for a given context).
main_review: The idea of being able to do combined optimization on the rankers and candidate generators together is extremely useful. The authors proposed solution to use MOE model to create multiple candidate/item pool (nominators) specialized on different sub-tasks and final ranker to select the best candidate can find a lot of applications. 

The authors have given extensive explanation with solid theoretical proof for their proposed solution. They have presented a thorough study of different components in their system and how it effects the final stage performance of the ranker.

The evaluations presented in the paper support the author’s claims and observations. However, the paper is not extremely easy to read especially the experimental setup could have been better explained, and I have listed down some opportunities the authors could have presented better.

Opportunities:
1.	There is no evaluation against any Learning to rank models, extremely popular with the two stage recommendation systems. Their main performance comparison is against single stage performance, but the setup for that evaluation/theoretical justification remains the same (EE based system).
2.	The related work section can be better explored.
limitations_and_societal_impact: The authors have identified no limitations or  potential negative impact.

Potential Limitation :
One of the main reasons for popularity of 2 stage recommendations is the scalability & flexibility they provide for large scale recommender systems (independence of creating a candidate set and a separate ranker). Increasing the number of nominators & MOE model adds more complexity and increases computational cost for the proposed 2-stage recommender system by the authors.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This work compares one-stage and two-stage systems from both of empirical and theoretical observations, which finds that interactions between the ranker and the nominators substantially affect the overall system performance. A Mixture-of-Experts approach is adopted to train the nominators to specialize on different subsets of the item pool. This improves the performance conducted on a Amazon review dataset. Overall, this submission is well presented and easy to follow.
main_review: Strength: 

1.This work targets at looking for two-stage specific properties, an interesting topic for the audience in the area of online large scale information system.

2.Empirical and theoretical observations well motivate its research problem.

3.Literature review is well embedded in the body of this submission.

Weak:

1.Since this work focus on recommender system. My main concern is what are the differences between RS and others. The mentioned ranker and nominator are also existed in some information systems like Q&A, Web search, sentence matching, and so on.

2.Without clearly addressing those differences, readers may feel that the current version can be applied mechanically to other IS.

3.Back to RS, there are various benchmark datasets. The reason of selecting “AmazonCat-13K” is not known. Experiments are conducted on review texts, so if only using rating scores, does it have different effects on the current findings?

4.The way to define two-stage RS is based on the bandit based approach. The reason and its advantages should be further addressed.
limitations_and_societal_impact: Yes.
review_title: null

Meta-review: Two-stage recommenders are critically important in production recommender systems, yet as the authors and reviewers point out, there has been very little theoretical analysis of this problem.  While the reviewers do reasonably question whether bandit-based analysis is really the right theoretical framework, the author response rightfully points out that their analysis captures generic characteristics of some deployed two-stage recommenders.  Post-rebuttal, the majority of reviewers agreed with a decision to accept and believe this work can inspire follow-on research in this potentially high impact area.  The authors are strongly encouraged to address review concerns (e.g., on power law and zipf-like distributions) and integrate their insightful rebuttal discussion into the paper (or Appendix), even when it may be self-critical of the present work.
Decision: Accept (Poster)