Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: In this work, a tree-based recommendation method RecForest is proposed to improve the model efficiency of tree-based method. Also, the model training cost can be improved with the multiple K-ary trees. The retrieval of near-boundary items becomes more effective. Experiments are performed on several recommendation datasets to show the model effectiveness.
strengths_and_weaknesses: Strengths:
1.The limitation of near-boundary item retrieval has been addressed in this work.
2.The ablation study is provided to investigate the effects of forest construction.
3.Several widely used experimented datasets are used for model evaluation.

Weaknesses:
1.In the evaluation section, it would be better to describe the detailed settings of some compared baselines, such as DIN and YoutubeDNN.
2. It would be better to discuss the potentials of the proposed method in tackling other types of recommendation scenarios, in addition to the sequential recommendation scenario.

questions: Considering that the model evaluation tackles the sequential recommendation scenario, it would be better to further compare the new proposed RecForest approach with Transformer-based sequential recommender systems (e.g., Bert4Rec), or discuss the advantage of the new mode. The new method is also built over the Transformer architecture for sequence encoding.

What are the specific parameter settings of some compared baselines in the evaluation section?

limitations: Please find the limitations in the weakness part.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors are interested in efficiently retrieving personalized items to users based on users' preferences. They focused on the efficacy aspect of the recommendation problem. The empirical results consist of several datasets with different ranges for # users, # items, and # interactions. They also included an analysis of performance on the quality of suggestions.

strengths_and_weaknesses: The efficacy of recommender systems is often a neglected problem in tradition RecSys literature. Most of the works are interested in improve the quality of recommendations. However, this is a challenging and important problem, mainly, in practical settings.

It would be helpful to include statistical significance tests of the empirical results.
questions: The authors listed the following limitations of previous approaches to justify their framework:

- it is challenging to route to items located around the partition boundaries, given that the item set is hierarchically partitioned,
- the accuracy of the beam search can be limited by the routing decision, which is made without consideration of the previous trajectory,
- the tree-based index can be memory-consuming, given that the number of internal nodes is at the same magnitude as the leaf nodes,
- there is a high cost for the training stage of existing approaches when calling for the joint adaptation of the representation model and tree index, given that the tree structure needs to be repetitively updated.

It would be helpful to link the above statements/following features of RecForest and empirical results. For instance, a reference to which section contains the results that present the advantages to RecForest over baselines.

The authors correctly included the complexity analysis of essential times and index size. This helps understand the tradeoffs between previous literature and the proposed framework.

limitations: The authors need to highlight the limitations of their method, preferably contrasting it and previous approaches.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: Approximate nearest neighbour search (ANNs) is used to retrieve top-N results for a user from a massive item set in personalized recommendation.
The paper creatively formulates the approximate nearest neighbour search (ANNs) problem into the sequence-to-sequence problem: the user profile and history is fed into the encoder of a Transformer while the top-N results are obtained by the beam-search of the decoder of that Transformer. To support this, the items are organized into a K-ary tree. To enhance the robustness, they construct several trees to form a forest. 
strengths_and_weaknesses: Strength: 

(1) Hats off to the core idea "Given the hierarchical numbering of the items, the recommendation turns out to be a sequence-to-sequence problem: based on the encoded user information, paths to the most preferable items are progressively decoded via beam search, from which the top-n recommendation is made. "  Hence, the problem of ANN and item embedding learning are merged and solved in one system.

Is this the first paper that using such beam search way based on Transformer decoder to generate the Top-N results for recommendation?

(2) The experiments show RecForest significantly outperforms all efficient recommenders with indexes on all datasets w.r.t NDCG@20 and NDCG@40. 

Weakness:
I don't see obvious weakness but hope the authors can carefully answer my questions presented in the following section because I feel some ablation study might be missing.
questions: (1) What if they use other decoder instead of Transformer? like RNN or CNN-based decoder?

(2) Yes, RecForest significantly outperforms all efficient recommenders with indexes. But what contributes most to this improvement? the Transformer's power or the effectiveness of joint representation learning?

(3) Why index size of JTM and TDM is O(ND) since they are also tree-based methods as shown in Figure 1?

(4) The K-ary tree means the vocabulary of the decoder is K, right? In NLP, K is a large number like 30,000. But in your case, K should be a very small number like 2 if binary, right? Hence, I just wonder will such big difference post any training issues on Transformer?

(5) Based on (4), since K is a small number, is that necessary to share the "word" embeddings among the trees to save the memory? 

(6) In real world recommendation, there are two stages: top-N retrieve and rerank. So this RecForest is for the top-N retrieve stage but not for the rerank stage, right? Since the performance is still worse than DIN in the experiments.
limitations: I don't see negative societal impact of their work,
review_title: null

Meta-review: metareview: The paper introduces a method for top-n item recommendation based on approximate nearest neighbor search (ANN). The authors formulate ANN as a sequence to sequence problem, the input being the user profile and activity, and the output being the top-n recommendations. The focus of the paper is on the computational efficiency of the ANN process. The proposed method learns jointly a tree-based index for organizing the items and a transformer based decoder for the top-n recommendation. The index is composed of multiple trees. Experiments are performed on classical benchmarks.


The reviewers consider that this is an original contribution with a convincing experimental evaluation. The authors have added several complementary experiments, including additional baselines, during the rebuttal and answered satisfyingly to the reviewers’ comments and questions. All the reviewers recommend acceptance.

confidence: Certain
Decision: Accept