Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper proposed a large-scale multipurpose benchmark dataset for recommender systems and verified it on 10 recommendation tasks. 
strengths: Different from existing recommendation datasets, the proposed Tenrec have following strengths:
1) Large-scale real-world dataset, which contains 5 million users and 140 million interactions;
2) Containing multiple types of behavior and true negative feedback, which could be utilized to support CTR scenario.
3) Consisting of overlapped users/items, which could support the CDR and TF research;
4) Including additional user and item features, which could be used for side information aware methods.

Owing to these merits, I think this dataset have potential to become a widely used benchmark dataset in the future, which could advance the recommendation research in the academia.

weaknesses: Here are some concerns about this paper:
1) In Table 1, why the #exposure of QK-article and QB-article are missed?
2) Section 3.8, 3.9, 3.10 is weakly related to the contributions of this dataset. The authors should choose the tasks that could better associate with the merits of the Tenrec, such as utilizing true positive negative feedback and additional user and item features. Although model depression and acceleration are also important topics, it is more appropriate to put them to the appendix or as the future work. 
3) The authors did not clearly illustrate what kind of user behavior they used for some tasks like session-based recommendation, transfer learning for recommendation ...
4) Given the fact that the SASRec performs better than BERT4Rec in Table 3, why the authors still use BERT4Rec for cold-start recommendation? It is hard to convince me that "bidirectional encoder enables better transfer learning than the unidirectional encoder" without comparing with SASRec.
5) In line 234, the authors said that "We only study item recommendation for users who have interactions less than 5 in the QK-article dataset ...", however, in the dataset description part, the author just said "the requirement that each user had at least 5 video clicking behaviors ..." "We perform similar data extraction strategy for QK-article, QB-video, and QB-article.", so the question is, why does the QK-article dataset have interactions less than 5? The authors should make it more clear.
6) To summary, I think the authors should focus on the tasks that are highly related to the five characteristics described by them. If possible, please include more baselines to compare.
correctness: The claims seem correct for me. The dataset is constructed in a sound way. And most of the evaluation methods and design seems appropriate and are performed correctly.
clarity: This paper is well-written and easy to follow.
relation to prior work: This paper has clear discussion about the difference between Tenrec and other recommendation datasets.
additional feedback: Some typos:
1) Line 230, "Tenure" should be "Tenrec";
2) Line 231, "by apply" should be "by applying".
review_title: A Large-scale Multipurpose Benchmark Dataset for Recommender Systems

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The authors proposed Tenrec, a large-scale and multipurpose real-world dataset, which have several characteristics. They also examine its properties by ten recommendation tasks by running some popular or state-of-the-art baselines. 
strengths: I really enjoyed this dataset that it has the potential to become a useful benchmark for recommendation community since it has several strengths. 
+ Compared with majority existing datsets, Tenrec has five characteristics: (1) large-scale, containing around 5 million users and 140 million interactions; (2) contains not only positive user feedback, but also true negative feedback (vs. one-class recommendation); (3) contains overlapped users and items across four different scenarios; (4) contains various types of user positive feedback, in forms of clicks, likes, shares, and follows, etc; (5) contains additional features beyond the user IDs and item IDs.
+ They verify Tenrec on ten diverse recommendation tasks by running several classical baseline models per task. 
weaknesses: As mentioned above, I am very excited about this paper, however, I think that it still have some limitations need to clarify and need more details.
+ Some experiment settings and strange results for benchmark need to be clarify. See more details in **Correctness**.
+ The document about the dataset processing for specific tasks are missed, which is also important to promote reproducibility for our community.  
+ The authos have no mention of some other multiple (more than 2) feedback datasets, which is very related.  
+ The methods for each task is too few and more popular and strong baselines are not included. 


More concerns are described in the following sections, please check them.
correctness: The paper is correct for the most part. The dataset construction is sound. Details of different methods and experiments are included. However, for benchmark, I still have several concerns as follows.  
1. Since you have used the DeepCTR to run baselines, and I see that you copyed the class 'VarLenSparseFeat' from DeepCTR (line 41 in model/ctr/inputs.py), however, for sequence features (i.e., user’s past 10 clicked items in your paper), you embeded them separately(line 84 in utils.py), I wonder whether it is a better idea to share the same embedding with item embedding rather than separately? Could you please give some explanation?  
2. Since Tenrec have timestamp, why don't split the data into the training set, validation set, and testing set according to the time, which is a common setting in industry?  
3. More metrics should be evaluated, for example, GAUC and Logloss for CTR prediction task.
4. For MTL task, why only-click and only-like better than MTL methods? And what is the training data for like prediction? Is there also selection bias here like CTR and CVR? Maybe you could add more details.    
5. Some strong and popular models for the tasks you mentioned can also be included, for example, DIN[1] and DIEN[2] for CTR prediction task. 

[1] Deep Interest Network for Click-Through Rate Prediction, KDD 2018.   
[2] Deep Interest Evolution Network for Click-Through Rate Prediction, AAAI 2019.
clarity: The paper is well written, however, some parts are not clear. Consider giving some examples.    
1. I could not find any description for Figure 1 in the main text (i.e. not explained at one place).   
2. In section 2, are videos and articles mixed and displayed on the same platform? Or displayed separately? In figure 1, I think you want to mean the latter case. More introductions to scenarios in which raw data are collected should be included.  

relation to prior work: The authors have mentioned some existing recommender systems datasets and compared them in Table 1 in Appendix.  However, there are some other datasets that contain multiple (more than 2) user feedbacks (e.g. UserBehavior(https://tianchi.aliyun.com/dataset/dataDetail?dataId=649), Ali_Display_Ad_Click(https://tianchi.aliyun.com/dataset/dataDetail?dataId=56)), which also need to be compared since one of your Tenrec's biggest merits is that it contains multiple types of positive user feedback.
additional feedback: All the comments and suggestions are provided above. This is a dataset full of potential, and if you can address my concerns and add more details about the questions above, I would like to increase my score.
review_title: Review of "Tenrec: A Large-scale Multipurpose Benchmark Dataset for Recommender Systems"

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: 1. This paper provides a large-scale dataset Tenrec valuable for RS, containing multiple scenarios, various user feed back, user profiles, etc.  It is useful for the development of recommendation communities. 
2. This paper illustrates 10 tasks to which the Tenrec dataset can be applied, and provides some of the experimental results.
3. This paper points out the lack of sufficiently large new datasets for recommender systems, which hinders the growth of the community. This motivation is very clear and worthy of recognition.
strengths: 1. The provided dataset Tenrec is one of the largest datasets for RS, covering a majority of recommendation scenarios and tasks(e.g. CTR prediction, top-n, sequential recommendation and so on).
2. They provide the corresponding code and will give the leaderboard of the related tasks.
3. This paper is well organized and easy to read.
weaknesses: 1. The introduction of the data collection process and the application platform (QQ Kandian and QQ browser) is insufficient, which may affect the assessment of the data characteristics.
2. Lack of citation and comparison of relevant work providing datasets. Especially in the last two years some competitions/companies also provide multi-target, cross-domain or even multi-modal datasets for recommendation purposes.
3. The experimental results need to be improved. Although the code is provided and experiments are performed on 10 tasks, the majority of the experimental results are only on the sampled QK-video-10M. And some of the experimental settings and results may be questioned. 
4. The extensibility of the dataset is limited. Although the dataset is large and provides auxiliary information on users and items, it lacks some rich semantic information. Specifically, for example, QK-video lacks multimodal features, and QB-article lacks content text.
correctness: I think the core contribution of this paper is the multi-purpose dataset provided. 
1. Empirically, the dataset is correctly constructed and corresponds to the description in the paper.
2. The raw data is provided, but the processed dataset is temporarily unavailable. Therefore, although I have some doubts about the experimental setup and results, I cannot absolutely judge the correctness.
clarity: This paper is very well written and I can understand it clearly and easily.
However, some details about figures(e.g. Figure 1) need to be added more clearly. Also, the github address of the code could be placed more prominently. 
relation to prior work: This paper discusses the limitations of datasets for recommendation systems and uses ImageNet and GLUE as examples to illustrate the necessity for large-scale recommendation datasets. Of course, they also mention some datasets, but there is a lack of intuitive comparison and references to some other large-scale datasets such as Criteo, Taobao, etc.

additional feedback: - Line 19-21. “Many improvements have been achieved in prior literature, however, much
20 of them performs evaluation on non-benchmark datasets…”.  Although other recommendation datasets may not be as large as Tenrec, they can be considered as benchmark datasets.
- Line 47-48. The provided dataset contains mainly ID features and does not seem to be well suited for content-based recommendations.
- Fig. 1.  Does the location of the subfigure have any special meaning, especially the QK-video subfigure.
- Fig. 2(a) and (b). Please explain these two figures more clearly. 
- L116-118.  I'm confused about the reason for sampling. Without sampling, I ran CTR experiments and multi-task experiments on the full QK-video dataset using an 11G GPU. 
- Line 119-120.  The original data set has a positive to negative sample ratio of 1:2.4, and I believe that additional sampling is not necessary.
- Table 2. According to my experimental results on the unsampled QK-video, AFM hardly outperforms xDeepFM by more than 1%.  I strongly encourage the authors to tune the experimental results more carefully and to add the experimental results of DCN and DCNv2.
- Line 158. Sampled softmax seems to be a more general approach. If sampled softmax is used, please provide the number of negative samples.
- I suggest that the authors use both gAUC and AUC as evaluation metrics, especially for CTR prediction and Multi-task Learning.
- Table 4. The experimental results of PLE should be added. In addition, in the experiments I did on the unsampled QK-video, the AUC of click is significantly higher than that reported in the paper. Therefore, I doubt that the authors' sampling is biased.
- Line 183-185. All samples in QK-article are click-positive samples, so QK-article is not applicable to six-task learning.


Overall, I think Tenrec is valuable to the development of RS. I encourage and expect the authors to refine the experiments and provide more solid leaderboards and the corresponding reproduction details.
review_title: Reviews of Tenrec

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The paper proposes a novel, large and versatile datasets -- Tenrec. Tenrec contains around 5 million users and 140 million interactions, which make it one of the largest datasets for recommender systems. It covers 4 different real-world scenarios with overlapped users and items, as well as various types of user feedback, including clicks, likes, shares, and follows, etc. Besides, it has not only positive user feedback, but also true negative feedback. It also contains additional features beyond the user IDs and item IDs.

The authors study the proposed datasets on 10 different recommendation tasks, including CTR Pred, session-based rec, MTL, transfer learning, user profile pred, cold-start, user representation learning, model compression, and model training/inference speedup. And they also benchmark sota models such as W&D, DeepFM, xDeepFM for CTR, and ESMM, MMoE for MTL, etc.
strengths: 1. The dataset will be useful towards better evaluating recommender systems for the development of the whole community. Since the tasks it involves is versatile, it has the potential to become a useful benchmark dataset for a majority of popular recommendation tasks. 

2. The dataset is quite large and diverse in comparison with existing recommender systems datasets.

3. The authors have done a good job implementing several classes of baselines and the results look interesting.
weaknesses: 1. The sample space of the negative sampling part seems not complete. The paper only considers the "with exposure but no user action" as the negative feedback. However, since all the exposures have been selected by the recall & rank modules, they are somewhat relevant with the users. Actually there are more negative items which are not exposed. For the CTR models, these "not-exposed" items should also been considered and showed in the training set so that the model could learn the distribution from them. With only the "with exposure but no user action" items, the sample space is not complete.

2. The comparison of the proposed benchmark with the existing benchmarks in recommender systems could be discussed more clear. The paper does not mention a lot about the competitive benchmarks. Maybe the authors could add more comparisons with the existing benchmarks in the areas of datasets size, datasets scenarios, types of tasks, etc.

3. The experimental results in CTR prediction show that the DeepFM performs even worse than W&D. This is little weird since DeepFM is the upgrade of W&D with an FM at the wide side. Maybe the authors could give more explanations for the phenomenons like this.

4. In the MTL part, the sota maybe PLE[1] rather than MMoE, the author could also add the results of PLE.

[1] Progressive layered extraction (ple): A novel multi-task learning (mtl) model for personalized recommendations. Tang, et al. 2020.

5. Some typos, such as in line 165 The → This.
correctness: Yes. The dataset is constructed in a sound way and the evaluation methods are appropriate.
clarity: Overall, I feel it is easy to follow the paper.
relation to prior work: The comparison of the proposed benchmark with the existing benchmarks in recommender systems could be discussed more clear. The paper does not mention a lot about the competitive benchmarks. Maybe the authors could add more comparisons with the existing benchmarks in the areas of datasets size, datasets scenarios, types of tasks, etc.
additional feedback: none.
review_title: review

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The benchmark datasets presented in this article are plentiful and collected from two big different real-world recommendation platforms of Tencent. These datasets have many desirable characteristics: (1) it is large-scale; (2) it includes both positive and negative user feedback; (3) it contains overlapped users and items across four different scenarios; (4) it contains various forms of feedback; (5) it includes additional features of users and items beyond the user IDs and item IDs. These characteristics enable the proposed datasets to suit different recommendation tasks. The proposed datasets make up for the shortcomings of classical datasets and have great potential.  

strengths: (1)	The proposed datasets make up for the shortcomings of the existing public datasets; 

(2) This paper conducts experiments on ten distinct recommendation tasks for illustrating the usages of the proposed datasets.  
weaknesses: (1)	Causality-inspired recommendation has gained much attention in recent years, this paper lacks discussion and illustration for this line of literature. 

(2)	The examples/tasks given in Section 3.8-3.10 are not representative tasks for recommender systems, they are more of some model storage and speedup tricks.

correctness: Yes
clarity: This paper is well written. Here are some minor comments. 

(1)  A user’s behaviors will affect the behaviors of other users in his social network. Do the proposed datasets contain information about social networks?

(2)  Are there any missing variables (e.g., features and feedback) in the proposed datasets?

(3)  Can the proposed datasets be used as benchmarks for causality-inspired debiasing methods? 

(4)  Why the metrics in Tables 2-8 are not unified?

relation to prior work: Yes
additional feedback: Error bars in the experimental results should be reported.
review_title: This paper provides a publicly available data collection for recommender system that makes up for the shortcomings of the existing classic datasets and is therefore of great value.

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: This paper releases a real large-scale dataset for recommender systems. Different from previous classic datasets like Movielens, which only have limited feedback forms, the proposed dataset, i.e., Tenrec, contains multiple types of user behaviors and is collected from four scenarios of QQ (a popular social platform). The authors also comprehensively establish benchmarks of their Tenrec dataset on up to ten concrete and important recommendation tasks. Considering the severe data scarcity problems for academic research of recommender systems, I think the dataset proposed in this work is precious.
strengths: 1. The paper was very well written.

2. Proposing a real large-scale recommendation dataset containing various user/item/behavior information.
At present, there is a great lack of high-quality real large-scale datasets in the field of recommendation systems. This work will help academic research better promote actual industrial progress, which would be a great boost to the community.

3. For the proposed dataset, the authors established comprehensive benchmarks over ten popular recommendation tasks, which facilitates the subsequent research based on this dataset. The results are also well analyzed.
weaknesses: 1. It's better to specify more training details of baselines, for example, the stop rules (e.g., training fixed epochs or the details of your early stopping strategy).
2. Collaborative filtering, as one of the most classical recommendation tasks, should be tested. Besides, as the proposed dataset contains some content features, it would be grateful to additionally fuse the content information beyond user feedback. 
3. It seems that only ''read_percentage'' is available. Could you also release the ''watch_percentage''?
correctness: I do not see obvious errors in their claims and experiments.
clarity: The paper is very well written.
relation to prior work: Clearly discussed.
additional feedback: Please see above.
review_title: A very valuable large-scale multipurpose benchmark dataset for recommender systems

Meta-review: metareview: In the last few years hyperscalers such Meta and Google have stated, more or less plainly, that the vast majority of their "AI Cycles" in the datacenter are devoted to various forms of recommendation tasks. Unfortunately, it has been hard to perform research on this problem, outside of those companies, due to the private and proprietary nature of the data. Large open benchmark sets have been limited to more narrow domains such as movie reviews. This paper offers a more representative large-scale real-world dataset which contains 5 million users and 140 million interactions.  The authors have gone to great lengths to address reviewers feedback, and, following their rebuttals, the reviews of this paper are now uniformly positive. This paper should be accepted.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept