RecFlow: An Industrial Full Flow Recommendation Dataset
Authors: Qi Liu
Kai Zheng
Rui Huang
Wuchao Li
Kuo Cai
Yuan Chai
Yanan Niu
Yiqun Hui
Bing Han
Na Mou
Hongning Wang
Wentian Bao
Enyun Yu
Guorui Zhou
Han Li
Yang Song
Defu Lian
Kun Gai
Venue: ICLR 2025 Poster (2025)
Keywords: recommendation system
recommendation dataset
Primary area: datasets and benchmarks

Abstract: Industrial recommendation systems (RS) rely on the multi-stage pipeline to balance effectiveness and efficiency when delivering items from a vast corpus to users. Existing RS benchmark datasets primarily focus on the exposure space, where novel RS algorithms are trained and evaluated. However, when these algorithms transition to real-world industrial RS, they face two critical challenges: (1) handling unexposed items—a significantly larger space than the exposed one, profoundly impacting their practical performance; and (2) overlooking the intricate interplay between multiple stages of the recommendation pipeline, resulting in suboptimal system performance. To bridge the gap between offline RS benchmarks and real-world online environments, we introduce RecFlow—an industrial full-flow recommendation dataset. Unlike existing datasets, RecFlow includes samples not only from the exposure space but also from unexposed items filtered at each stage of the RS funnel. RecFlow comprises 38 million interactions from 42,000 users across nearly 9 million items with additional 1.9 billion stage samples collected from 9.3 million online requests over 37 days and spanning 6 stages. Leveraging RecFlow, we conduct extensive experiments to demonstrate its potential in designing novel algorithms that enhance effectiveness by incorporating stage-specific samples. Some of these algorithms have already been deployed online at KuaiShou, consistently yielding significant gains. We propose RecFlow as the first comprehensive whole-pipeline benchmark dataset for the RS community, enabling research on algorithm design across the entire recommendation pipeline, including selection bias study, debiased algorithms, multi-stage consistency and optimality, multi-task recommendation, and user behavior modeling.

Published as a conference paper at ICLR 2025
AN INDUSTRIAL FULL FLOW RECOMMEN- DATION DATASET
RECFLOW:
Qi Liu',
Kai Zheng,,
Rui Huang,
Wuchao Li',
Kuo Cai²,
Yuan Chai²,
Yanan Niu?,
Yiqun Hui?,
Bing Han?,
Na Mou
Hongning Wang",
Wentian Bao?,
Yunen Yu,,
Guorui Zhou?,
Han Li?,
Yang Song",
Defu Lian';
Kun Gai
University of Science and Technology of China Kuaishou
IIndependent 'Tsinghua University
{qiliu67, wiw.caa/mmmaaaaa.........a uste edu. en, fLiandefu]@uutcc.e en
bhbengkai,,hangruiie caikuo, niuyanan, chaiyuan
( /Kuuashhu. com
buntyiqun,,anbiinin zhouguorui, songe,,s @Kua ishuu com (ww5))evirginia.c (ob [yuenyun]@126.. com, {285208254, gai (an.:2/a2,..ga...uungegggegg com
[wb2328]@collubbb
edu
ABSTRACT
Industrial recommendation systems (RS) rely on the multi-stage pipeline to balance effectiveness and efficiency when delivering items from vast corpus to users. Existing RS benchmark datasets primarily focus on the exposure space, where novel RS algorithms are trained and evaluated. However, when these algorithms transition to real-world industrial RS, they face two critical challenges: (1) handling unexposed items a significantly larger space than the exposed one, profoundly impacting their practical performance; and (2) overlooking the intricate interplay between multiple stages of the recommendation pipeline, resulting in subopti- mal system performance. To bridge the gap between offline RS benchmarks and real-world online environments,
introduce RecFlow industrial full-flow recommendation dataset. Unlike existing datasets, RecFlow includes samples not only from the exposure space but also from unexposed items filtered each stage of the RS funnel. RecFlow comprises 38 million interactions from 42,000 users across nearly 9 million items with additional 1.9 billion stage samples collected from 9.3 million online requests over 37 days and spanning 6 stages. Leveraging potential in design-
RecFlow, we conduct extensive experiments demonstrate ing novel algorithms that enhance effectiveness by incorporating stage-specific samples. Some of these algorithms have already been deployed online at KuaiShou, consistently yielding significant gains. We propose RecFlow as the first comprehen- sive whole-pipeline benchmark dataset for the RS community, enabling research algorithm design across the entire recommendation pipeline, including selection bias study, debiased algorithms, multi-stage consistency and optimality, multi-task recommendation, and user behavior modeling.
INTRODUCTION
Recommendation systems (RS) are pivotal modern web and mobile applications that handle vast amounts of information. Their primary objective is to deliver personalized recommendations from an extensive corpus of items, based on estimated user preferences. To meet stringent online latency requirements, industrial RS predominantly employs multi-stage funnel-like pipeline (Covington et al. 2016), striking a balance between effectiveness and efficiency. Substantial efforts have been
"Corresponding author

---

Published as a conference paper at ICLR 2025
devoted designing algorithms within this system, aiming to enhance its effectiveness as measured by user feedback on selected items. A typical multi-stage RS consists of successive stages: retrieval - pre-ranking + ranking - re-ranking. During online serving, the retrieval stage (Hidasi et al. 2015; Kang & McAuley 2018 Zhu et al. 2018) retrieves thousands of preferred items from the entire corpus. The pre-ranking stage (Huang et al. 2013; Wang et al. 2020) filters out less favorable items from the retrieved set, forwarding hundreds of more promising items to the ranking stage. In turn, the ranking stage (Cheng et al., 2016, Zhou et al. 2018 Bian al.
2022) selects the most appealing items from this refined set. Finally, the re-ranking Pee et al. 2019 Bello et al.) 2018)) stage determines the final items to be displayed, considering both diversity and business objectives. Notably, as we progress through the stages, the model complexity tends to increase, incorporating additional features and interleaving them at shallow layers of deep neural network models. Importantly, the latter three stages typically learn from the exposure space, which captures actual user feedback (both positive and negative) on the displayed items.
Despite the maturity of industrial RS, two significant shortcomings persist. First, a discrepancy exists between the data distribution in the training space and that in the serving space (Qin et al. 2022). The former corresponds to the exposed space, while the latter primarily resides in the unexposed space. This discrepancy, which we refer to as the distribution shift problem, poses challenges. For instance, consider the pre-ranking model (Wang et al., 2020): It must score thousands of items, yet only a few of these items are exposed to users and stored as training data in each request. Most of the remaining samples have not been exposed even once. Consequently, a pre-ranking model trained solely on the exposure space may inaccurately predict preferences in the retrieved space, leading to suboptimal recommendations (Wei et al. 2024). Similar issues arise in the ranking and re-ranking stages. Second, there is a discrepancy between the learning and serving environments. Although models at different stages are learned and evaluated separately, they must collaborate as a cohesive system to meet user preferences. Insufficient knowledge about other stages during the learning process can result in suboptimal performance when these learned models serve online. For example, the online performance of retrieval algorithm not only depends itself but is also influenced by subsequent stages. Incorporating knowledge from subsequent stages can enhance the retrieval algorithm's 's performance (Ding al. 2019. Lou al., 2022: Zheng al. 2024).
Large-scale datasets serve as the bedrock for advancing various machine learning algorithms. For instance, ImageNet (Deng et al., (009) has significantly contributed to computer vision, while GLUE (Wang et al.] 2018) has played a crucial role in natural language processing. However, in the
RS domain, existing datasets (Harper & Konstan 2015: Ni et al. 2019 Asghar 2016 Zhu et al. 2018: Yuan YYuan et al. 2022; Gao et al. al.] 2022a bj /Sun et al.) 2023) though instrumental in fueling RS research-have a limitation: they are exclusively collected from the exposure space. These datasets cannot fully capture the true dynamics of online recommendation services. Moreover, this inherent bias prevents them from effectively addressing the discrepancy between training and serving in RS.
We propose RecFlow, an industrial large-scale full-flow dataset collected from the real industrial RS. The industrial RS's multi-stage funnel-like pipeline encompasses the following stages: retrieval, pre- ranking, coarse ranking, ranking, re-ranking, and edge ranking. Unlike all previous RS benchmarks, RecFlow samples representative unexposed items from each stage of the funnel in a single request alongside all the exposed items. The inclusion of full-stage samples in each request provides several merits. (I) By recording items from the serving space, RecFlow enables the study of how to alleviate the discrepancy between training and serving for specific stages during both the learning and evaluation processes (Qin et al., 2022). (2) RecFlow also records the stage information for different stage samples, facilitating research joint modeling of multiple stages, such stage consistency or optimal multi-stage RS (Zheng et al. 2024). (3) The positive and negative samples from the exposure space are suitable for classical chick-thruuhh rate prediction or sequential recommendation tasks (Zhou et al. 2018 Kang & McAuley 2018). (4) RecFlow stores multiple types of positive feedback (c.g., effective view, long view, like, follow, share, comment). supporting research on multi-task recommendation (Ma et al. 2018a| Zhao et al. 2019 Tang et al. 2020 Liu et al.) (023). (5) Information about video duration and playing time for each exposure video allows the study of
learning through implicit feedback, such as predicting playing time (Covington et al.) 2016 Lin
cct al. 2023). (6) RecFlow includes a request identifier feature. which can contribute to studying the re-ranking problem (Pei et al. 2019 Bello et al. 2018). (7) Timestamps for each sample enable the aggregation of user feedback in chronological order, facilitating the study of user behavior sequence
modeling algorithms (Zhou et al. 2018 2019 Chang et al. 2023 Hou et al. 2023). (8) RecFlow
2

---

Published as a conference paper at ICLR 2025
Retrieval
Pre ranking
Course Ranking
Ranking
Re Ranking
Edge Ranking
mink pos
rerank pos
8000
3000
500
120
Tog-
Top-I0
Exposure
Preferenc
10
5000
2500
380
Feedbuck
sampling
sempling
sampiing
sunpliig
40
terank
Collected into RecFlow
premank Beg
Figure 1: The overall collection process of RecFLow.
incorporates context, user, and video features beyond identity features (e.g., user and video ID), making it suitable for context-based recommendation (Huang et al. 2019: Wang et al. 2022). (9)
The rich information recorded about RS and user feedback allows the construction of more accurate RS simulators data may help estimate selection bias more accurately and design better unbiased algorithms (Chen) al. 2023). Furthermore, RecFlow is a large-scale dataset, containing 38 million exposure samples and 1.9 billion stage samples, ensuring the credibility of algorithm improvements based on its data.
models in feed scenarios (Shi et al.] 2019 Zhao et al. 2023). (10) Rich stage
Given these characteristics, RecFlow can be utilized across a broad spectrum of RS algorithms. In this paper, we primarily conduct pioneering experiments to explore its potential in each stage of the RS funnel. In the retrieval stage, we investigate the effectiveness of using filtered videos from each stage as hard negative samples and explore the interplay between retrieval and subsequent stages. For the coarse ranking stage, we leverage corresponding stage samples address the distribution shift problem and model mutual effects between stages. Motivated by existing works, we explore how to exploit stage samples for designing auxiliary ranking tasks and behavior sequence modeling algorithms to improve classical AUC metrics. Similar exploration experiments the ranking stage. Notably, RecFlow also introduces a new recall metric to assess the performance of different methods based stage samples mitigate the gap between training and serving environments. RecFlow the first RS dataset containing stage samples. It stands as one of the largest and most comprehensive datasets for RS, covering nearly all recommendation tasks. We have made the dataset and source codes publicly available to promote reproducibility and advance RS The dataset is licensed under
also conducted for
research at wttptt:://// / thub. com/ Rec# ow- TCLR/ RecFl low CC-BY-NC-SA-440 1E--CC--A-400OOtemaatiiiiia International License.
2
DATASET CHARACTERISTIC
2.1 COLLECTION
RecFlow is the first RS dataset containing intermediate filtered videos of each stage in the industrial RS funnel. The multi-stage funnel-like pipeline of the industrial RS contains six stages, including retrieval + pre-ranking - coarse ranking - ranking - re-ranking - edge ranking. The
number of videos output at each stage is 8000 - 3000 - 500 + 120 10 6. We collected the online request logs from January 13 to February 18, 2024. The collection process is as follows. We randomly sample 42K seed users on January 12, 2024, and store each recommendation request of the seed users from January 13, 2024. As shown in Figure [[] we sample some filtered videos from each stage but adopt a stage-wise strategy. From January 13 to February 04, 2024, which is called the 1st period, we sample 10 filtered videos of the pre-ranking stage named pre-rank_neg 10 filtered videos of the coarse ranking stage named coarse_neg, top 10 ranking videos as rank-pos, 10 sampling filtered videos after the 120-th re-ranking video as rank neg in the ranking stage, top 10 re-ranking videos as rerank rerank-pos and 10 sampling filtered videos after the 80-th re-ranking video as rerank neg in the re-ranking stage, and the user's 's various feedbacks the exposed videos. Note that the recommendation scenario is feeds-style, the user can only watch one video on the screen. So, the 6 output videos of the RS may not all be exposed to the user because the user can leave the APP any
3

---

Published as a conference paper at ICLR 2025
Table 1: Detail quantity information of various aspects in RecFlow.
#Stage Sample
#Request
#Users
#Realshow _vidoos
#All_vidcos
1st Period
352, 420,410,401
6,062,348
38,193
5,984,924
30,305,725
2nd Period
1.572,217,303
3,308,233
35,073
3,627 69,
55,665,503
Total
1,924.337,704
9,370,581
42,472
8,773,147
82,216,301
#Realshow
#Like
#Long-view
#Effective_
#Follow
1st Period
24,523,473
1,027,013
5,853,054
9,343,776
69,495
2nd Period
13.721.842 38,245,315
618,158
3,111,439
5,063.751
37.558
Total
1,645,171
8,964,493
14,407,527
107,053
#Forward
#Comment
#Prerank_neg
#coarse
#Rank.pos
1st Period
45,966
175,896
60,623,480
60,623,480
60,624,430
2nd Period
23,769
114,741
132,329,320
132,329.320
33,082.330
Total
69,735
290,637
192,952,800
192,952,800
93,706,760
#Rank neg
#Rank
#Rerank pos
#Rerank neg
#Re-rank
1st Period
60,624,012 33,082,330
121,248,442
60,624,613
60,623,606 33,082,330 93,705,936
121,248,219
2nd Period
1,307,558,66
33,082,330
1 1,307,558,66
Total
93,706,342
1,428,807,105
93,706,943
1, 1,428,806,882
time. We define the realshow field to identify whether the user has watched the video. From February 05 February 18, 2024, which called the 2nd period, we expand the amount of stage samples. Both the pre-ranking neg and the coarse neg go up to 40. For the ranking, re-ranking, and edge ranking stages, save all the videos that appear in these stages. We still obtain the rank _pos, rank_neg. rerank-pos, rerank_neg, and realshow under the same stage-wise strategy as the previous period. We collect stage samples in this way, considering the storage pressure and information integrity. The 2nd period has more complete stage information compared to the 1st period, which gives the researchers more choices to further process the dataset based on their needs. We sample 10/40 filtered videos from the pre-ranking and coarse ranking stages because keeping all of the filtered videos has huge storage pressure. Besides, the videos filtered by the first three stages are less important. For the latter three stages, we keep the information integrity of the stage. The videos appearing in these stages closer to the user's preference and have a small scale.
2.2 FEATURES
The formation of each instance in RecFlow is {request.id. request.timestanee user device iid, age, gender. province, video id, author id, category level one, category level two, upload type, upload timestamp, duration, realshow, rerank pos, rerank neg, rank pos, rank neg, coarse neg, pre-rank neg, rank index, rerank index, playing time, effective .iew, long view, like, forward.. comment). realshow indicates whether the user has watched the video. The same procedure is applied to the other "_pos/neg fields. For example, when the video ranks top 10 the ranking stage, then the rank_pos is set to otherwise O. To reserve the original industrial RS information, we also retain the ranking position of each video in the ranking and reranking stages through the rank_index and rerank index fields. We record seven types of positive feedback that reflect the user's varying degrees of preference towards videos. playing time is the time the user spends watching the video. The other features' details are in the subsection Feature Description of Appendix.
2.3 ANALYSIS
In this section, we conduct basic statistical analysis to show RecFlow's characteristics. We collect 9 million requests. It has 38 million exposure samples and 1.9 billion stage samples (including exposure samples). Among these samples, there are 42K users, 8.7 million exposed videos, and 82 million videos. Nearly 89% of videos are not exposed. This new character does not exist in existing RS datasets. During the first period, the quantity of each defined stage's ' samples is about 60 million. Stage samples are 14.8x larger than exposed samples. The difference between stage samples and exposure samples has increased to 236 times in the 2nd period. The huge quantity difference is the foundation for studying the distribution shift problem. The detailed quantities of
4

---

Published as conference paper at ICLR 2025
the dataset shown in Table Figure whose horizontal axis represents the range of the number of videos interacted with by users and the vertical axis shows the number and percentage users within that range, illustrates that the frequency of users exhibits long-tail distribution. In Figure 4 the horizontal axis represents the logarithm of the frequency of video appearances, while the vertical that frequency. The left chart only includes videos
axis shows the video quantity corresponding marked as realshow with 1, which are the exposed vidcos, while the right chart includes videos from all stages. It shows the frequency of videos in exposure space and all stages' space, respectively. The left chart shows that exposure video frequency follows long-tail distribution. The right chart reveals that video frequency in all stages also obeys the long- tail distribution, which new discovery.
Logo #Realshow Videos
#Users
30908
390
3.7
12
Log:: Frequency
Logso Frequency
Normal Interactions
Figure 3: User Distribution.
Figure 4: Video Distribution.
2.4 COMPARISON
We compare RecFlow with existing RS datasets. MovieLens (Harper Konstan 2015) contains the user's rating data for movies. Amazon (Ni et al. 2019) dataset contains the user's s review information on the product. Yclp (Asghar 2016) is a dataset for location recommendation. The three datasets only contain the user's ' single type of positive feedback. Taobao (Zhu et al.) (2018). an e-commerce dataset, has four types of the user's positive feedback. Tenrec (Yuan et al. 2022) is a comprehensive recommendation dataset that captures multiple types of user feedback across four distinct recom- full-observed video recommedation dataset. KuaiRand (Gao et 2022b) an unbiased sequential video recommendation dataset with randomly exposed videos. KuaiSAR (Sun et al. 2023) a unified search and recommendation dataset. The three datasets are opened for dedicated research problems. RecFlow differs from those datasets because of the existence of samples from each recommendation stage. Table 8 in the subsection Dataset Comparison A.2 of Appendix gives a detailed comparison between RecFlow and existing RS datasets. We also discuss the limitation of RecFlow in subsection A.3 of the Appendix.
recommendation
mendation scenarios. KuaiRec (Gao et al. 2022a)
2.5
USER CONSENT AND PRIVACY PROTECTION
We only collect interaction data from the user who has made his/her personal information publicly (like user id, age, gender, province, etc), and this public information allows for some level of data sharing, according to the privacy policy that users voluntarily agreed to when they signed up for an account. Besides, we anonymize all features that contain personal information. In detail, anonymize each feature ID by adding the raw ID value with a random large integer first and remapping it to new ID through the Hash algorithm. It can not know who the person in the real world is from the anonymous data. The General Data Protection Regulation of the European Union has confirmed that "personal information that has been anonymized does belong to personal information. Therefore, personal information that has been anonymized does not have the corresponding personal information compliance obligations, and companies can freely process without the consent of individuals." Thus, open-source dataset meets legal requirements.
We have anonymized all features which contain personal information including request.id, user iid, device .id, age, gender, province, video.id. author id, category level_eveeeeee category Ltw and pppled_ttpe. We first anonymize each feature ID by adding the raw ID value with random large integer and then remapping owns a unique larger integer. The rest of the features are stage labels and the user's feedback labels, which not related to privacy. Anonymizing data with random noise and the Hash algorithm
ID through the Hash algorithm. Note that each raw ID value
5

---

Published as a conference paper at ICLR 2025
satisfies the privacy protection requirements of the law of the European Union. The way of RecFlow's
anonymization is more strict than previous public recommendation datasets, including Amazon (Ni et al. 2019), Taobao (Zhu et al. 2018), KuaiRec (Gao et al. 2022a), and Tenrec (Yuan et al. 2022).
We add random large integer noise before the Hash algorithm and others not. It is nearly impossible to recover raw personal information, such who the person in the real world is.
3 EXPERIMENTS
We explore how to utilize stage samples to alleviate distribution shift and distill knowledge of subsequent stages for improving RS's performance. We focus on typical retrieval, coarse ranking, and ranking stages. For each stage, we briefly introduce its duty and existing learning paradigm. Then, we state the motivation and the ways of exploiting stage samples. Finally, we report the experiment results and analysis. We run all experiments five times with Pytorch (Imambi et al. 202Tb Db on Nvidia 32G V100. We report the average result and standard deviation. For all methods and all experiments, we train the neural models for only one epoch, and there is no early stopping. Thus, all methods are compared fairly. There are two reasons for only one epoch. First, all online recommendation models of the industrial RS are trained by one epoch. We keep consistency with the online configuration. Second, there exists a one-epoch phenomenon (Zhang et al.) 2022) of the training recommendation model, which indicates that multi-epoch training does not bring improvement.
3.1 RETRIEVAL.
Retrieval is the first stage of the industrial RS. It aims at retrieving thousands of videos that the user potentially prefers from the 100 million scale video corpus. Given the large candidate pool, the retrieval stage mostly adopts the lightweight two-tower model together with approximate nearest neighbor search to retrieve items quickly. To ensure that the user's preferred videos are obtained, the retrieval models usually learn with positive feedback videos as positive samples and randomly sampling videos as negative samples. We choose SASRee (Kang & McAuley) 2018) with one head and one layer for exploration experiments. We apply the effective. videos positive samples and randomly sample 200 videos as negative samples for each positive. To keep consistency with the real industrial RS's online learning mode, we train SASRec with the first 36 days' data day by day. The data from the last day is for evaluation. We utilize the standard top-N ranking metrics, including hit Recall @ K and NDCG@K. K. K is set to 100, 500, 1000. The feature is the user's 50 past effective_viee _vew videos. We apply embedding for the video_d _id feature and set the embedding dimension to 8. The batch size is 4, 096 and the learning rate is 1e-1. BPR (Rendle al. 2012) is the loss function, and Adam (Kingma& & Ba 2014) is used for optimization.
3.1.1 HARD NEGATIVE MINING
Recent research (Zhang et al. 2013 Rendle & Freudenthaler 2014 Lian et al. 2020) has shown that hard negative mining usually not only accelerates the convergence but also improves the model accuracy for the retrieval model. The hard negative samples are those videos that are similar to the positive videos but uninteresting to the user. The multi-stage RS pipeline aims estimating the user's preference. Videos that fail to be exposed to the user during the pipeline are similar to the displayed positive video but very likely less attractive to the user. Thus, we think the unexposed stage samples indeed satisfy the definition of hard negative samples. We conduct experiments to explore the effectiveness of the stage samples as hard negative samples. In the experiments, we replace some randomly sampled easy negative samples with the same number of hard negative stage samples. The total number of negative videos for each positive video is 200.
We have the following findings from the result in T (1) Applying filtered videos from each stage as hard negatives all gains performance improvement on the RecalI/NDCG metric. (2) As the in Recall/NDCG@@ @ K becomes smaller, the performance improvement becomes better. For example, when we add 1 pre-rank as hard negative, the relative promotion of Recall @ 100, 500, 1000 are 24.7%. 18. 18.2%. 9. 2% respectively, and the relative promotion of NDCG@ @ 100, 500, 1000 are 28.3%, 20.7%, 12.6% respectively. (3) The hard negative video from rerank-po ppss outperforms than the other stages. We think that videos from perank-pos are negative samples of appropriate difficulty. We also vary the number of hard negative samples to observe the changes in effectiveness. The experiment result and analysis are in the subsection A.4 of the Appendix.

---

Published as a conference paper at ICLR 2025
Table 2: Recall(R) and NDCG(N) results (mean + std) obtained by using single different stage sample as the hard negative sample during the retrieval stage, with units of %. The best and bascline results are based on the paired t-test at the significance level 5%.
Hard Negative Type
R@ 100
N@ 100
R 50500
N@500
R R@100 1000
N @@ 1000
Baseline
0.4611000055
0.09911.0085
1.593:10.229
2.68510.188 0.241
0.35610..000
Prerank
0.575110.095
0.127:10.08
1.88310.170
0.29111.003
2.931-0.142 0.401
Coarse_ncg
0.555=0.066
0.121=0.021
1.729=#0152 0.267=0.033
2.758=0.169 0.376=0.035
Rank_neg
0.462 126 0.094+0.030
1.695+0.230
0.249=0.0043
2.733=0.221 0.359=0.042
Rank.pos
0.648=0.004
0.134+0.017
1.794+0.187
0.277+0.028
2.737=0.173
0.376=0.022 0.05
Rerank_neg
0.577=10.091
0.119:10.019
1.80440.208
0.274410033
2.724=1:242 0.006
Rerank_pos
0.687=00.887
0.144:100118
0.889-
0.225
2.892=:01005
0.401
Exposure neg
0.603--0.093 0.137++0.016
1.860 0.295+
2.902-002221
0.033
Table 3: Recall(R) and NDCG(N) results (mean + std) obtained by using a single different stage
sample as the cascade sample during the retrieval stage, with units of %. The best and baseline results are based on the paired (-test at the significance level 5%.
Cascade Type
R 100
N@ 100
5@0
500
R@ 1000
N@ 1000
Baseline
0.461110.085
0.099110.085
1.593:10.229
0.241:100045
2.685.10.186
0.356:10.040
Prerank_neg
0.677:10.061
0.167:10.01
2.268:10.129
0.367:10.08
3.446:10.111
0.492:10.042
Coarse_neg
0.665:40.120 120
0.163:10.045
2.253:10.052
3.371:10.090 00.037
0,479:10.038
Rank neg
0.704 150 0.173:0.049 2.282:10.250
0.373=100055
3,410=10.203
0,491
Rank pos
0.685110.094
0.1511100025 2.191=000885
0.340=0.023
3.346=10007
0.462:10019
Reeank_neg
0.707110083 0.163:10.224
2.273=:0.11
0.359=1.024
3.338110083
0.471:0.022
Rerank_pos
0.795++0.100 108
0.176++0.025 2.263+00.078
0.361=00011
3.394+-00088
0:480++0.016
Exposure_neg
0.692+0.071 0.156++0.028
2.150++..10 108 0,340+00003
3.266++0.18, 183 0.458+0.036
FS-LTTR
0.803++0.095 0.215++0.02 2.466++0.000 0.425=0.009 3.606=0.000
0.545++0.22
3.1.2 INTERPLAY BETWEEN RETRIEVAL AND SUBSEQUENT STAGES
The most important characteristic of industrial RS is the multi-stage. Every stage has its duty and mature paradigm. The goal of each stage is consistent, which is to fit the user's preference. Although models of all stages aim at fitting the user's preference, they not capture the user's preference perfectly, Few people focus on the interplay between stages. The academic researchers lack available datasets and the industrial engineers only devote effort the assigned stage. (Zheng al. 2024) has pointed out that there are two factors influencing the video's ' exposure and the user's 's feedback. First, the video. Second, it is the preference of the subsequent stage towards
it is the user's preference the video. For example, one video that the user likes is retrieved during the retrieving stage but is filtered out by the ranking model due to its imperfect preference estimation ability. This video is inefficient for the whole RS because it can not be exposed to the user at all. The optimal solution for the model of each stage is to select videos that satisfy the preference of the user and subsequent stages simultaneously. FS-LTR (Zheng et al., 2024) has proposed the Generalized Probability Ranking Principle (GPRP) to prove that the solution proposed above is optimal theoretically. We implement FS-LTR in this section to see its effectiveness. The user's preference can be learned from the positive feedback samples and randomly sampled negative samples. To learn the preference of subsequent stages, we introduce additional ranking loss, which forces the logits of samples from high-priority stages to be bigger than the logits of samples from low-priority stages. The priority of stages are (positive:6, exposure neg:5, rerank pos:4, rank pos:4, rerank neg:3, rank neg:3, corase neg:2, pre-rank neg: 1, random neg:0). Exposure neg represents the video that has been exposed to the user (realshow=1) but obtains negative feedback. This definition of priority applies throughout the paper. We always keep one positive sample with 200 negative samples. We first introduce the stage preference one stage a time by replacing random negatives with stage samples with BPR as Eq []::
N
LIS.LIR
Σ
BPR(o,, 0j)
T= jE(k:pa <p.)
where N equals 200, P((k) represents the priority level of sample ilk), and Pk < P; means the priority level of sample k is lower than sample
7

---

Published as a conference paper at ICLR 2025
We have the following findings from experiment results in Table (1) FS-LTR gains performance
enhancement when introducing the sample of each stage respectively compared to the baseline. (2) Under the same negative setting, FS-LTR can achieve better results compared with the results of hard negative mining in Table 2 (3) As the K of Recall K becomes smaller, the performance improvement becomes better. For example, when we add 1 pre-rank neeg as hard negative, the relative promotion of Recall @ 100, 500, 1000 are 46.8%, 42.4%, 28.3% respectively. NDCG@K K holds the same trends. We also try to introduce multiple samples from more stages gradually to investigate the effectiveness of modeling more subsequent stages' preferences in subsection A.5 of the Appendix.
3.2 COARSE RANKING
Coarse ranking receives favorable videos from the pre-ranking stage and filters less favorable videos to fulfill its duty. As the candidate videos are more similar and not easy distinguish, coarse ranking models take more feature fields as input and use a more complex neural network to ensure their modeling capacity. However, there are 3, 000 videos be scored in our scenario, the two-tower structure is still the best choice. We take DSSM (Huang et al. 2013) as the coarse ranking model. The Multi-layer Perceptron (MLP) of the user and video towers in DSSM are set to be [128, 64, 32]. Existing coarse ranking models are almost learned through the exposure of positive and negative samples. AUC (Area Under the Curve) on the testing exposure samples is employed to assess the algorithm's 's performance. Effective view is the learning signal. Following the retrieval experiment, data from the first 36 days for training and the last day's data for evaluation. The feature fields include user iid, device_id, age, gender, province, video author id, category levell.ne category level two, upload type, upload timestamp, request timessttt. The upload timestamp and request. timestamp are divided into the week, day, hour feature fields. Besides, add the user's s past 50 effective_ effectiveevideos videos as the behavior sequence. We process effective behavior sequences through mean pooling. We apply embedding for all the feature fields and set the embedding dimension to 8. The batch size is 024 and the learning rate is 2. Binary Cross Entropy is the loss, and Adam is used for optimization. We also utilize the stage samples to explore the auxiliary ranking task and user behavior sequence modeling in the coarse ranking model, both of which boost the AUC metric greatly. The methods, together with the experiment results and analysis, are in subsections A.6 and A.7 of the Appendix.
3.2.1 DATA DISTRIBUTION SHIFT
Data distribution shift is a longstanding problem in RS. Due to the absence of datasets containing stage samples, few works (Ma et al., 20186 Qin et al. 2022) focus on the problem in the coarse ranking stage. The coarse ranking model is trained based on the exposed samples which contains 6 videos at most but has to score 3, 000 videos in each request. The data distribution between training and testing exists huge inconsistency. What's worse, the AUC metric evaluated on the exposure space for guiding the offline algorithm's optimization is inconsistent with the online scenario (Song et al.) 2022, Zhang et al. 2023b). The collected stage samples make the evaluation space more consistent with the online situation. Following (Zhang et al.) (023b). we apply the Recall @ K metric, which is consistent with the effect of online business. Because we saved all the videos in the ranking stage on February 18, 2024, the candidate set for calculating the Recall @ K and NDCG@ K composed of the videos in the ranking stage together with videos of coarse_neg. We set K to 100, 200. We also report the classical AUC metric. We try to directly supplement the stage samples as extra negative samples into the training data. Although it's possible to introduce false negative videos, this can still largely reduce the difference in data distribution between training and testing. However, supplementing extra negative samples increases the machine overload. Thus, we show the relationship between the performance and the quantity of the additional negative samples.
We have the following conclusions from the result in Table 4 (1) Supplementing stage videos as extra negative samples can largely enhance the Recall and NDCG metric. The improvement can be attributed to the consistency of data distribution between training and testing.
(2) When increasing the quantity of extra negative samples, the improvement becomes greater. And introducing partial or all videos from all corresponding stages gains the best results. This indicates that the more consistent the data distribution between training and testing, the more improvement. (3) Introducing videos from rank pos and rerank pos gains light enhancement compared to coarse/rank/rerank neg. We think that there are false negative samples that mislead the model's learnin.. (4) The classical AUC metric has
8

---

Published as a conference paper at ICLR 2025
Table 4: The result (mean + std) of using different stages' samples extra negatives for Coarse Ranking. The best and baseline results are based on the paired t-test at the significance level 5%.
Neg Type #N
AUC
LogLoss
Recall 100
NDCG@ 100
Recall@20X 200 NDCG@200
Baseline
0.708
0.592-1..03
0.27110.027
0.059-10.027
0.535=0.009
0.096110.003
0.705
0.608
0.32110.012
0.072:10.027
0.597=0.038
0.11111.0003
Coarse_neg
0.633:10.016 10
0.773
0.392:10.012
0.088.10.004
0.668=0.007
0.126:10.003
1
0.704410.002 0.615=10.004
0.353-10.013
0.079-10.003
0.638=0.004
0.118:10.002
Rank Rank_neg
0.618:100016 10
0.825=0.027
0.454:10.011
0.102:+0.005
0.726=0.005
0. 0.0004
1
0.704=0.001 0.603=0.001
0.275=0.027
0.061=0.004
0.557=0.005
0.100=:0.001
Rank pos
0.623+00000 10
0.769=0.003
0.290++0.002
0.069+0.004
0.591=0.019
0.111+00.004
0.702:10.01 0.616=0.005
0.337+0.007
0.07610.001
0.605=0.002
0.113=000001
Rerank neg
0.608:10021
0.821
0.380=0.014
0.08410.003
0.673=0.004
0.125=:0.033
1
0.703110001 0.607-0.003
0.26410.011
0.06010.003
0.548=0.003
0.099110.002
Rerank pos
0.618:10.024
0.782:10.025
0.285-10.015
0.069100003
0.587=0.011
0.1111100033
1
0.662.10.006
0.704
0.3861000000
0.08411.001
0.676-0.008
0.125110.001
All
0.563.10.004
1.243.10.030
0.455 0.105110.001
0.728 0.004
0.144
Table 5: The result (mean + std) of interplay between Coarse Ranking and Subsequent Stages. The best and baseline results are based on the paired t-test the significance level 5%.
Method
AUC
LogLoss
Recall 100
NDCG@ 100
Recall 200
NDCG@200
Baseline
0.71811.001
0.592:10.003
0.271=
0.535:10.009 0.059:10.027
0.096.1
PositiveRank
0.5541000005
1.040:10.051
0.457=0.001
0.112:10.001
0.723:10.002
0.149:10.001
FS-LTR
0.473+0.013
1.253++0.071
0.475=0.002
0.119++0.001
0.734++0002
0.155++0.001
opposite trends. After adding extra negative samples, the gap between the training data distribution and the data distribution of exposure space for evaluating AUC enlarges. As we mentioned in the retrieval section, there exists a hardness level among different stage samples. Expanding negatives degrades the model's ' ability to distinguish hard negatives (exposed un-effective view samples) but enhances the capability of recognizing less hard negatives (videos from stages).
3.2.2 INTERPLAY BETWEEN COARSE RANKING AND SUBSEQUENT STAGES
FS-LTR is general principle and is applicable in the coarse ranking stage. We implement FS-LTR with samples of positive, exposure neg. rerank pos, rerank neg, rank pos, rank neg, coarse neg, which is the inference space of the coarse ranking model. order to apply the loss [ we aggregate samples of the same request into the same batch. We also add contrast experiment, PostiveRank, positive samples bigger than the logits of the other samples, The result in Table 5 shows that FS-LTR can achieve the best performance on the Recall/NDCG. It demonstrates the necessity of learning the preferences of both the user and the subsequent stages.
in which we just make the logits
3.3 RANKING
Ranking is nearly the most important stage in the industrial multi-stage RS and has been studied sufficiently. It determines the displayed items to the user. Its candidate video set is the output of the coarse ranking stage. Given the importance and difficulty of the task, ranking model has the most complex neural network structure and uses most feature fields. The time cost is acceptable because it only needs to score 500 videos. We utilize DIN (Zhou et al. 2018) as the ranking model. The architecture of the DIN's MLP is [128, 128, 32, 1J. Ranking model also learned on the exposure space and evaluates AUC on testing exposure samples. For the experiment settings, the ranking model remains the same as the coarse ranking model. We also use the stage samples to explore the auxiliary ranking task and user behavior sequence modeling in the ranking model, both of which improve the classical AUC greatly. Detail of methods and experiments are in subsection A.8 and A.9 of the Appendix.

---

Published as a conference paper at ICLR 2025
Table 6: The result (mean + std) of using different stages' samples as negatives for Ranking.
The best and bascline results are based on the paired t-test at the significance level 5%.
Neg Type
#N
AUC
LogLoss
@5call@50
NDCG@50
Recall @ 100 NDCG@ 100
Baseline
0.727-10.001
0.883.00.003
0.16910.0055
0.045110002
0.319100008 0.069110.002
1
0.7111100001
0.61010.008
0.223110.005
0.061110.02
0.395:10.007 0.0881100002
neg
10
0.645:10.003
0.8101100032
0.26441 0.012
0.074110.004
0.454:10.014
0.105:10.005
1
0.711100001 0.60410.008
0.176:10.005
0.047-10.001
0.327:10.010
0.072:10.002
Rank. Pank.pos
0.653+00.022 10
0.724:10022
0.185:+0.009
0.049+10.003
0.331+000015
0.073++0000
1
0.708=00.01 0.616=0.006
0.215=0.0003
0.05911.001
0.38010.006
0.085+0.001
Rerank neg
0.624+0.005 10
0.815+00028
0.232++0.018
0.064+00000
0.406++0.031
0.092+-0.007
1
0.71110.002 0.60810.006
0.17010.012
0.04511.003
0.31911.019
0.06910.004
Rerank pos
10
0.646:10.002
0.78211.033
0.183:10.016
0.04810.005
0.335:10.009
0.07310.003
1
0.675110.003 0.69710.010
0.23411.0055
0.064110.002
0.41110.0004
0.09310.002
All
0.602-10.005 10
1.07610.009
0.208
0.078 0.467 0.108.10.009
Table 7: The result (mean + std) of interplay between Ranking and Subsequent Stages. The best and baseline results are based the paired t-test at the significance level 5%.
Method
AUC
LogLoss
R@00
N@
R@ 100
N@ 100
Baseline
0.727+00001
0.583+-0003
0.169-+-0.05
0.045=0.002
0.319++0.008
0.069++0.002
PositiveRank
0.564+-0.003
1,466++0.13
0.309++0.016
0.093=0.006
0.506++0.014
0.125++0.006
FS-LTR
0.46110.005
1.21511.391
0.323:10.012
0.098 0.003
0.525.1.014
0.13110.000
3.3.1 DATA DISTRIBUTION SHIFT
The ranking model also suffers from the data distribution shift problem. In each request, there are at most 6 exposure samples for training but 500 videos to be scored. The data distribution gap between training and testing still exists. Fortunately, the inconsistency is not as serious as the coarse ranking model. The exploration experiment setting for alleviating the data distribution shift problem the same as the coarse ranking model, including motivation, method, and evaluation metrics. The difference is that samples of coarse neg are excluded from training and evaluation because they are not the ranking model's candidate video set. The result is shown in Table We can find that the more consistent the data distribution between training and testing. the more the Recall and NDCG improve. Other conclusions are the same as coarse ranking, and we don't 't repeat them here.
3.3.2 INTERPLAY BETWEEN RANKING AND SUBSEQUENT STAGES
We also conduct FS-LTR in the ranking stage. The experiment settings are mostly the same as coarse ranking except that training samples are from positive, exposure rank_neg, rank-pos, rerank_neg verank_pos, pos, which is the inference space of the ranking model. PositiveRank serves as the contrast purpose. The results in Table lead the same conclusions as coarse ranking.
4 CONCLUSIONS
In this paper, propose a new dataset called RecFlow which captures information across the entire pipeline of an industrial recommendation system. It will provide researchers with convenience for studying multi-stage recommendations. We also conduct extensive preliminary experiments using RecFlow in retrieval, coarse ranking, and ranking stages. The experimental results demonstrate that utilizing stage samples indeed enhances recommendations.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
The work was supported by grants from the National Natural Science Foundation of China (No. U24A20253).
REFERENCES
Nabiha Asghar. Yelp dataset challenge: Review rating prediction. arXiv preprint arXiv: 1605.0536..
2016.
Aijun Bai, Rolf Jagerman, Zhen Qin, Le Yan, Pratyush Kar, Bing-Ronp Lin, Xuanhui Wang. Michael Bendersky, and Mare Najork. Regression compatible listwise objectives for calibrated ranking with binary relevance. In Proceedings of the 32nd ACM International Conference Information and Knowledge Management, pp. 1-502-4000 2023.
Irwan Bello, Sayali Kulkarni, Sagar Jain, Craig Boutilier, Ed Chi, Elad Eban, Xiyang Luo, Alan Mackey, and Ofer Meshi. Seq2slate: Re-ranking and slate optimization with rnns. arXiv preprint arXiv:1810.020/1 2018.
Weijie Bian, Kailun Wu, Lejian Ren, Qi Pi, Yujing Zhang, Can Xiao, Xiang-Rong Sheng, Yong-Nan Zhu. Zhangming Chan, Na Mou, et al. Can: feature co-action network for click-through rate prediction. In Proceedings of the fifteenth ACM international conference web search and data mining, pp. 57-65, 2022.
Jianxin Chang, Chenbin Zhang, Zhiyi Fu, Xiaoxue Zang, Lin Guan, Jing Lu, Yiqun Hui, Dewei Leng, Yanan Niu, Yang Song, al. Twin: Two-stage interest network for lifelong user behavior modeling in ctr prediction at kuaishou. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 7785-3794, 2023.
Jiawei Chen, Hande Dong, Xiang Wang, Fuli Feng. Meng Wang, and Xiangnan He. Bias and debias in recommender system: A survey and future directions. ACM Transactions on Information Systems, 41(3):1-39, 2023.
Heng-Tze Cheng, Levent Koe, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, al. Wide & deep learning for recommender systems. In Proceedings of the 1st workshop on deep learning for recommender systems, pp. 7-10, 2016.
Paul Covington, Jay Adams, and Emre Sargin. Deep neural networks for youtube recommendations. In Proceedings of the 10th ACM conference recommender systems, pp. 191-198, 2016.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp.
Ieee, 2009.
Jingtao Ding, Yuhan Quan, Xiangnan He, Yong Li, and Depeng Jin. Reinforced negative sampling for recommendation with exposure data. In IJCAI, pp. 2230-2236. Macao, 2019.
Zhifang Fan, Dan Ou, Yulong Gu, Bairan Fu, Xiang Li, Wentian Bao, Xin-Yu Dai, Xiaoyi Zeng, Tao Zhuang, and Qingwen Liu. Modeling users' contextualized page-wise feedback for click- through rate prediction in e-commerce search. In Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining, pp. 262-270, 2022.
Chongming Gao, Shijun Li, Wenqiang Lei, Jiawei Chen, Biao Li, Peng Jiang, Xiangnan He, Jiaxin Mao, and Tat-Seng Chua. Kuairec: A fully-observed dataset and insights for evaluating recom- mender systems. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management, pp. 540-550, 2022a.
Chongming Gao, Shijun Li, Yuan Zhang, Jiawei Chen, Biao Li, Wenqiang Lei, Peng Jiang, and Xiangnan He. Kuairand: an unbiased sequential recommendation dataset with randomly exposed videos. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management, pp. 3953-3957, 2022b.
11

---

Published as a conference paper at ICLR 2025
Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He. Deepfi:: a factorization- machine based neural network for etr prediction. arXiv preprint 1703.04774 2017.
Maxwell Harper and Joseph Konstan. The movielens datasets: History and context. Acm transactions on interactive intelligent systems (fiis), 5(4):1-19, 2015.
Xinran He, Junfeng Pan, Ou Jin, Tianbing Xu, Bo Liu, Tao Xu, Yanxin Shi, Antoine Atallah, Ralf Herbrich, Stuart Bowers, et al. Practical lessons from predicting clicks on ads at facebook. In Proceedings of the eighth international workshop on data mining for online advertising, pp. 1 1-9, 2014.
Balázs Hidasi, Alexandros Karatzoglou, Linas Baltrunas, and Domonkos Tikk. Bession-based recommendations with recurrent neural networks. arXiv preprint arXiv: 151669999 2015.
Xuyang Hou, Zhe Wang, Qi Liu, Tan Qu, Jia Cheng, and Jun Lei. Deep context interest network for click- thhrouhh rate prediction. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, pp. 3948 01088,9922222 2023.
Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry Heck. Learning deep structured semantic models for web search using clickthrough data. In Proceedings of the 22nd ACM international conference on Information & Knowledge Management, pp. 2333-2333 2333,,33,, 2013.
Tongwen Huang, Zhiqi Zhang, and Junlin Zhang. Fibinet: combining feature importance and bilinear feature interaction for ttcc-through rate prediction. In Proceedings of the 13th ACM conference on recommender systems, pp. 169-177, 2019.
Sagar Imambi, Kolla Bhanu Prakash, and GR Kanagachidambaresaan Pytorch. Programming with TensorFlow: Solution for Edge Computing Applications, pp. 87-104, 2021.
W'agg-Chngg Kang and Julian McAuley. Self-attentive sequential recommendation. In 2018 IEEE international conference on data mining (ICDM), pp. 197-206. IEEE, 2018
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv: 6980, 2014.
Xiang Li, Shuwei Chen, Jian Dong, Jin Zhang. Yongkang Wang, Xingxing Wang, and Dong Wang. Decision-making context interaction network for click-through rate prediction. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 5195-5202, 2023.
Defu Lian, Qi Liu, and Enhong Chen. Personalized ranking with importance sampling. In Proceedings of The Web Conference 2020, pp. 1093- 1103, 2020.
Xiao Lin, Xiaokai Chen, Linfeng Song. Jingwei Liu, Biao Li, and Peng Jiang. Tree based progressive regression model for watch-time prediction in short-video recommendation. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 4497 4506, 2023.
Zhutian Lin, Junwei Pan, Shangyu Zhang, Ximei Wang, Xi Xiao, Shudong Huang, Lei Xiao, and Jie Jiang. Understanding the ranking loss for recommendation with sparse user feedback. arXiv preprint arXiv:2403.1 100.4444444 14144, 2024.
Qi Liu, Zhilong Zhou, Gangwei Jiang, Tiezheng Ge, and Defu Lian. Deep task-specific bottom repre- sentation network for multi-task recommendation. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, pp. 1637-1646, 2023.
Qi Liu, Xuyang Hou, Defu Lian, Zhe Wang, Haoran Jin, Jia Cheng, and Jun Lei. At4etr: Auxiliary match tasks for enhancing click-through rate prediction. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 8787-8795, 2024.
Jiazhen Lou, Hong Wen, Fuyu Lv, Jing Zhang, Tengfei Yuan, and Zhao Li. Re-weighting negative samples for model-agnostic matching. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1823-1827, 2022.
Jiaqi Ma. Zhe Zhao, Xinyang Yi, Jilin Chen, Lichan Hong, and Ed H Chi. Modeling task relationships in multi task learning with multi multi-gate mixture-of -fpertess. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 1930-199 1939, 2018a.
12

---

Published as a conference paper at ICLR 2025
Xiao Ma, Liqin Zhao, Guan Huang, Zhi Wang, Zelin Hu, Xiaoqiang Zhu, and Kun Gai. Entire space multi-task model: An effective approach for estimating post-click conversion rate. In The 4 International ACM SIGIR Conference on Research & Development in Information Retrieval, pp. 1137-1140, 2018b.
Jianmo Ni, Jiacheng Li, and Julian McAuley. Justifying recommendations using distantly- reviews and fine-graine ane-grained aspects. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNI.P-ICNIIP),, pp. 188- 1888-17,200 2019.
Changhua Pei, Yi Zhang, Yongfeng Zhang, Fei Sun, Xiao Lin, Hanxiao Sun, Jian Wu, Peng Jiang. Junfeng Ge, Wenwu Ou, et al. Personalized re-ranking for recommendation. In Proceedings of the 13th ACM conference on recommender systems, pp. 3-11, 2019.
Jiarui Qin, Jiachen Zhu, Bo Chen, Zhirong Liu, Weiwen Liu, Ruiming Tang, Rui Zhang, Yong Yu, and Weinan Zhang. Rankflow: Joint optimization of multi stage cascade ranking systems as flows. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 814 824, 2022.
Steffen Rendle and Christoph Freudenthaler. Improving pairwise learning for item recommendation from implicit feedback. In Proceedings of the 7th ACM international conference on Web search and data mining. pp. 273-2822 2014.
Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme. Bpr: Bayesian personalized ranking from implicit feedback. arXiv preprint arXiv: 220..6618, 2012.
Xiang-Rong Sheng, Jingyue Gao, Yueyao Cheng, Siran Yang, Shuguang Han, Hongbo Deng, Yuning Jiang. Jian Xu, and Bo Zheng. Joint optimization of ranking and calibration with contextualized hybrid model. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 4813-4822, 2023.
Bing-Chegg Shi, Yang Yu, Qing Da, Yong Chen, and An-Xiang Zeng. Virtual-taobao: Virtualizing real-world online retail environment for reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 4902-4909, 2019.
Jinbo Song, Ruoran Huang, Xinyang Wang, Wei Huang, Qian Yu, Mingming Chen, Yafei Yao, Chaosheng Fan, Changping Peng, Zhangang Lin, et al. Rethinking large-scale pre-ranking system: Entire-chain cross-domain models. In Proceedings of the 31 ACM International Conference on Information & Knowledge Management, pp. 4495-4499, 2022.
Zhongxiang Sun, Zihua Si, Xiaoxue Zang. Dewei Leng, Yanan Niu, Yang Song, Xiao Zhang, and Jun Xu. Kuaisar: A unified search and recommendation dataset. In Proceedings of the 32nd ACM International Conference Information and Knowledge Management, pp. 5407-5411, 2023.
Hongyan Tang, Junning Liu, Ming Zhao, and Xudong Gong. Progressive layered extraction (ple): A novel multi-task learning (mtl model for personalized recommendations. In Fourteenth ACM Conference on Recommender Systems, pp. 269-278, 2020.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding, arXiv preprint 10.10:1804.07441, 2018
Fangye Wang, Yingxu Wang, Dongsheng Li, Hansu Gu, Tun Lu, Peng Zhang, and Ning Gu. Enhanc- ing prediction with context-aware feature representation learning. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 343-352, 2022.
Zhe Wang, Liqin Zhao, Biye Jiang, Guorui Zhou, Xiaoqiang Zhu, and Kun Gai. Cold: Towards the next generation of pre-ranking system. arXiv preprint arXiv:2007. /6/22, 2020.
Jaanping Wei, Yujie Zhou, Zhengwei Wu, and Ziqi Liu. Enhancing pre-ranking performance: Tackling intermediary challenges in multi-stage cascading recommendation systems. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 5950-5958. 2024.
13

---

Published as a conference paper at ICLR 2025
Le Yan, Zhen Qin, Xuanhui Wang, Michael Bendersky, and Mare Najork. Scale calibration of deep ranking models. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 4300-4309, 2022.
Guanghu Yuan, Fajie Yuan, Yudong Li, Beibei Kong, Shujie Li, Lei Chen, Min Yang, Chenyun Yu, Bo Hu, Zang Li, et al. Tenrec: A large-scale multipurpose benchmark dataset for recommender systems. Advances in Neural Information Processing Systems, 35:1 1480-114999 2022.
Weinan Zhang, Tianqi Chen, Jun Wang, and Yong Yu. Optimizing top-n collaborative filtering via dynamic negative item sampling. In Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval, pp. 785-788, 2013.
Yuan Zhang, Xue Dong, Weijie Ding, Biao Li, Peng Jiang, and Kun Gai. Divide and conquer: Towards better embedding- baseed retrieval for recommender systems from multi-task task perspective. In Companion Proceedings of the ACM Web Conference 2023, pp. 366-370, 2023a.
Zhao-Yu Yu Zhang, Xiang-Rone Rong Sheng, Yujing Zhang, Biye Jiang, Shuguang Han, Hongbo Deng, and Bo Zheng. Towards understanding the overfitting phenomenon of deep click-through rate models. In Proceedings of the st ACM international conference on information & knowledge 2022.
management, pp.
Zhixuan Zhang, Yuheng Huang, Dan Ou, Sen Li, Longbin Li, Qingwen Liu, and Xiaoyi Zeng. Rethinking the role of pre-ranking in large-scale e-commerce searching system. arXiv preprint 2rXiv:230S.13647, 2023b.
Kesen Zhao, Shuchang Liu, Qingpeng Cai, Xiangyu Zhao, Ziru Liu, Dong Zheng, Peng Jiang, and Kun Gai. Kuaisim: A comprehensive simulator for recommender systems. Advances in Neural Information Processing Systems, 011444880448999 2023.
The Zhao, Lichan Hong, Li Wei, Jilin Chen, Aniruddh Nath, Shawn Andrews, Aditee Kumthekar, Maheswaran Sathiamoorthy, Xinyang Yi, and Ed Chi. Recommending what video to watch next: a multitask ranking system. In Proceedings of the 13th ACM Conference Recommender Systems, pp. 43-51, 2019.
Kai Zheng, Haijun Zhao, Rui Huang, Beichuan Zhang, Na Mou, Yanan Niu, Yang Song, Hongning Wang, and Kun Gai. Full stage learning rank: unified framework for multi-stage systems. In Proceedings of the ACM on Web Conference 2024, pp. 3621-3631, 2024.
Kaifu Zheng, Lu Wang, Yu Li, Xusong Chen, Hu Liu, Jing Lu, Xiwei Zhao, Changping Peng, Zhangang Lin, and Jingping Shao. Implicit user awareness modeling via candidate items for ctr prediction in search ads. In Proceedings of the ACM Web Conference 2022, pp. 246-255, 2022.
Guorui Zhou, Xiaoqiang Zhu, Chenru Song, Ying Fan. Han Zhu, Xiao Ma, Yanghui Yan, Junqi Jin, Han Li, and Kun Gai. Deep interest network for tlick-through rate prediction. In Proceedings of the 24th ACM SIGKDI) international conference knowledge discovery & data mining, pp. 1059-1068, 2018
Guorui Zhou, Na Mou, Ying Fan, Qi Pi, Weijie Bian, Chang Zhou, Xiaoqiang Zhu, and Kun Gai. Deep interest evolution network for click- thrkuhh rate prediction. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pp. 5941-5948, 2019.
Han Zhu, Xiang Li, Pengye Zhang, Guozheng Li, Jie He, Han Li, and Kun Gai. Learning tree-based deep model for recommender systems. In Proceedings of the 24th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, pp. 1079-1088, 2018.
14

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 FEATURE DESCRIPTION
The request id identifies each recommendation request and request request_ timestamp represents the time when the recommendation request arises. Every has unique ID named user_id ,id. device idd means the device that initiates the recommendation request. We also provide the user's 's profile information including age, gender, province. Age is grouped into buckets. video id identifies each video. author_ic id represents the one who uploads the video. We also record the video's attributes involving category Leve,onee, category Level_wo, upload_type, upload timesstanp duration. category level one, category level two are categories of the video, where category level one is the coarse-graings category (e.g., sports, history, K-pop, etc.) and category level two indicates the fine-grained category (e.g., UEFA Champions League, Ming Dynasty, BLACKPINK, etc). The upload_type and
stand for the type of the video (e.g., micro-video, long-video, picture, etc) and the time when the video was uploaded. duration is the video's lasting time. Next, we describe the fields identifying the stage information. The effectivv_ _vew and long_view are the binary features (0 and 1) defined according business interest. long-view is more strict than effective _view. like indicates whether the user clicks the button. follow means the user follows the video's author. forward represents the user sharing the video. comment stands for whether the user makes review about the video. Note that the feedback values of the unexposed video are all set to 0. The fields request id, user id, device id, agge gender, province, video id, author iid, category level category level two, upload type are all have been anonymized ensuring the privacy protection.
text
A.2 DATASET COMPARISON
Table 8: The characteristic comparison of different recommendation datasets.
Dataset
Stage Sample
feedbbesss #Users
#Interac- tions
True_neg Req_id
Moviel.ens-200 (Harper Konstan
138K
20M
X
2015
Amazon Ni al.
2019
X
2
/
233M
X
X
Yelp (Asghar 2016)
1
1.9M
8M
X
V
Taobao Zhu él al. 2018
4
987K
100M
TenRec-QKV Yuan et al.
2022
X
4
S.0M
142M
J
TenRec-QKA Yuan cf al.
2022
X
6
1,3M
46M
X
X
KuaiRec Gao ct al. 2022a
X
1
7K
12M
X
KuaiRand Gao
2022bb
6
27K
322M
KuaiSAR
Sun el 2023)
9
26K
RecFlow
V(199B)
7
42K
38M
A.3 LIMITATIONS
RecFlow, while valuable in lots of recommendation research problems, also has its own drawbacks. Understanding advantages and disadvantages is vital for ensuring accurate academic use. First, we collect data from only one recommendation scenario which causes RecFlow can be applied the multi/cross-doman recommendation. Second, RecFlow can't advance the research of multimodal recommendation because of lacking multimodal features such as text and image. On the other hand, it needs more hardware resource cost because RecFlow contains 1,924,337,70 704 instances.
A.4 RETRIEVAL: THE EFFECT OF NUMBER HARD NEGATIVES IN RETRIEVAL STAGE
The result in Table 9 is the result of varying the number of hard negative samples from each stage. We set the number to 2 and 10 for observation. (1) Increasing the number of hard negative videos from prerank_neg can further improve the performance but with diminishing marginal effect. (2) For coarse neg, rank pos, rerank neg. rerank pos, exposure neg, adding videos from them as hard negative samples degrades the performance. The closer the stage to the positive feedback, the more
15

---

Published as a conference paper at ICLR 2025
Table 9: RR) and NDCG(N) results obtained by using 2 10 different stage samples the hard negative sample during the retrieval stage, with units of %.
HN Type
#HN R@ 100
N@ 100
@ 500
N@500
R@ 1000
N 1000
Baseline
0.461
0.099
1.593
0.241
2.685
0.356
0.474
0.108
1.664
0.257
2.574
0.352
Prerank.neg
10
0.457
0.101
1.515
0.236
2.319
0.321
0.634
0.140
1.948
0.305
2.844
0.400
Coarse_neg
10
0.524
0.125
1.564
0.256
2.347
0.339
0.492
0.104
1.687
0.254
2.645
0.355
Rank_ne
10
0.323
0.069
1.321
0.193
2.231
0.289
0.589
0.126
1.846
0.284
2.722
0.376
Rank pos
10
0.544
0.109
1.544
0.235
2.310
0.315
0.606
0.120
1.849
0.277
2.831
0.381
Rerank_nee neg
10
0.336
0.070
1.186
0.176
2.032
0.265
0.428
0.091
1.584
0.236
2.551
0.338
Rerank-pos
10
0,219
0.043
0.954
0.135
1.819
0,226
0.576
0.138
1.854
0.300
2.866
0.407
exposure_ne neg
10
0.629
0.142
1.924
0.305
2.856
0.403
Table 10: Recall(R) and NDCG(N) results obtained by using different combinations of stage samples as cascade samples during the retrieval stage, with units of %. CN-PN represents the use of coarse neg and preran neg. R-CN-PN represents the of exposure.neg, coarse_neg, and prerank neg. ALL
represents the use all stage samples the current request.
Cascade Type
R@ 100
N@ 100
R@:500
N@500
R@ 1000
Nē 1000
Baseline
0.461
0.099
1.593
0.241
2.685
0.356
CN-PN
0.803
0.215
2.466
0.425
3.606
0.545
EN-CN-PN
0.771
0.198
2.481
0.413
3.648
0.536
All
0.663
0.150
1.972
0.315
3,018
0.425
degradation. The phenomenon demonstrates that there exists hardness level between videos from different stages. The closer to the positive feedback the stage, the more difficult for the retrieval model to distinguish. (3) When we increase the number videos from rank_neg from 1 to 2, the performance is somewhat boosted. However, it still suffers from severe performance drop when taking 10 rank_neg hard negative videos. As pointed out in (He et al. 2014 Zhang et al.) (023a)), the ratio between easy and hard negatives has critical influence performance. We guess that harder negatives need more easy negatives and leave the hardness and ratio hard negative samples for further research.
A.5 RETRIEVAL: MORE RESULT OF RETRIEVAL'S INTERPLAY EXPERIMENT
The result of Table 10 is the experiment of introducing samples from more different stages into the FS-LTR. We find that directly introducing samples from all stages is better than the baseline but is not the best. The setting of CN-PN achieves the best in our exploration, which indicates that ranking regularization between some priority levels may be unnecessary. We leave the in-depth exploration for future research.
A.6 COARSE RANKING: AUXILIARY RANKING TASK
Increasing the ranking ability of the model trained with pointwise loss function (e.g.. Click-through Rate prediction model) by adding an auxiliary ranking task has gained much attention recently (Yan
al. 2022
Bai et al. 2023 Sheng al. 2023 Liu al. 2024 Lin et al. 2024). The auxiliary
16

---

Published as a conference paper at ICLR 2025
Table 11: The result of the auxiliary ranking task for the coarse ranking stage.
Method
AUC
LogLoss
R@ 100
N@ 100
R@200
N@000
Baseline
|
0.718
0.592
0.271
0.059
0.535
0.096
w/ AuxLoss
0.721
0.588
0.287
0.061
0.541
0.096
Table 12: The result of competitive relation modeling in UBM during coarse ranking stage.
Method
AUC
LogLoss
R@ 100
N@ 100
R@ R00
N@000
Baseline
0.718
0.592
0.271
0.059
0.535
0.096
Competing Seq
0.722
0.588
0.293
0.064
0.574
0.103
ranking task forces the logits of positive samples to be bigger than negative samples within the same batch session through pairwise or listwise ranking loss. Inspired by these works, we propose new auxiliary ranking task by forcing the logits of positive samples bigger than the stage samples of the the negative samples. Note that the stage samples
same request. There is no ranking regularization are only for auxiliary loss. The total loss function is as (2)
N BCEWithLogitonn v1) ta* N
N Σ BPR(o).O,)) j=
L= N
where N is the batch size, N+ is the number of positive samples in the batch, ja represents the stage sample within the same request as j. K is the size of stage samples, and o is the logit output by DSSM. a is the weight of auxiliary ranking loss. In the experiment, we use all the stage samples from coarse neg, rank neg, rank vank,pos, rerank neg, rerank, pox stages. The result Table IT shows that the AUC increases by 0.002 and the Logloss decreases by 0.004, which is significant improvement (Guo ct al. 2017). Recall and NDCG also gain improvement. The designed auxiliary ranking task promotes both the classical and the newly proposed metrics, which demonstrates its effectiveness.
A.7 COARSE RANKING: USER BEHAVIOR SEQUENCE MODELING
Competitive relation modeling has been attracting attention in user behavior sequence modeling (UBM) recently (Zheng al. 2022; Hou al., 2023 Fan al., 2022 Li al. 2023). Its motiva- tion is that the user's 's feedback on items is also influenced by the displayed context. For example, if one user likes red T-shirts, he/she will click a pink T-shirt surrounded by items which he/she is not interested in, but he/she will click the red T-shirt surrounded by pink, blue, and yellow T-shirts. The competitive relation among displayed items has an impact on the user's feedback. stages. These
There also exists a competitive relation among the videos in the rerank/rank,pp videos compete for exposure to the user. Inspired by (Hou et al. 2023). we explore introducing the competitive information in the stage samples into the UBM. For the user's past effective_ videos S = [01, 02 ..., Uso]. we regard 10 videos in the rank-pos from the same request of each effective view video in S as the competitive information. We represent the competitive relation as C' [[ 1, ...,,..., v1,10]: [12,1, 22,2, 12,10 .... [U50,1, U50,2, ..., 1o]. We apply the hierarchi- cal attention algorithm model the competitive relation. First, we perform target attention between each effeettive view behavior i; and its competing context [Vg, Ui.2, ... 10]. We will obtain the refined competing behavior representation E [C1, C2, ..., c50]. Then, do mean pooling on E to the user's ' competitive relation aware interest competing.interes innterest. Table 12 shows the experiment results. Both AUC and Logloss are improved significantly by 0.004. What's 's more, Recall 100, and NDCG@ @1 10,200 also get better performance. The video competitive information in the rank pos a useful signal for UBM. The result indicates that there exists a method that can improve both the classical AUC/Loglos and the newly applied Recall/NDCG metric.
A.8 RANKING: AUXILIARY RANKING TASK
We also conduct the auxiliary ranking task in the ranking stage. The ranking loss is still Eq The stage samples used in the ranking loss come from rank neg, rank pos, rerank neg, rerank pos stages.
17

---

Published as a conference paper at ICLR 2025
Table 13: The result of the auxiliary ranking task for the ranking stage.
Method
AUC
LogLoss
R@:00
N@
R@100
N@ 100
Bascline
0.727
0.583
0.169
0.045
0.319
0.069
w/ AuxLoss
0.729
0.583
0.168
0.045
0.316
0.068
Table 14: The result of competitive relation modeling in UBM during ranking stage.
AUC
LogLoss
R@50
SO
R@ 100
N@ 100
Baseline
0.727
0.583
0.169
0.045
0.319
0.069
Competing Seq
0.732
0.578
0.168
0.045
0,313
0.068
The results are summarized in Table 13 We can draw findings the same with the auxiliary ranking task of coarse ranking, which verifies the broad effectiveness of the auxiliary ranking loss based on the stage samples.
A.9 RANKING: USER BEHAVIOR SEQUENCE MODELING
Competing modeling is also explored in the ranking stage. We still choose the videos in the rank_pos the competing context. We change the modeling method and make more suitable for DIN. After acquiring the refined competing behavior sequence representation E = [C1 C22 C50]. we perform target attention between target video Vtrrget and E. Finally, we obtain the user's competing-awe aware interest competing interest, towards the target video Starget- The result in Table [4 shows that both the AUC and Logloss are improved by 0.005 but the Recall and NDCG have no change. The result is not perfect as coarse ranking and it is worth exploring the modeling method continuously.
18


Reviews:
[{"review_id": "IB7OUjroCX", "reviewer": "Reviewer_UTjX", "summary": "This paper first proposes a full-flow recommendation dataset collected from the industrial video recommendation scenarios. The overall process includes retrieval, pre-ranking, coarse ranking, ranking, re-ranking, and edge ranking. The logs are collected from January 13 to February 18, 2024. The datasets can be accessed via a half-anonymized link that denotes the authors' institute.", "questions": "1. Regarding the ten merits presented in the introduction, it remains unclear which characteristics are unique to the RecFlow dataset compared to existing benchmarks.\n2. As in line 143, what is the rationale behind the number of videos selected for each stage?\n3. Also, can you explain why you chose 200 negative samples for each positive?\n4. Some typos in the paper; for example, in line 379, recall 100 happens twice. This error occurs lots of times.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "1. The proposed full-flow dataset provides a strong groundwork for follow-up research. For example, models can learn how to alleviate selection bias due to the discrepancy between the training and inference stages.\n2. The authors performed comprehensive experiments and presented the results of the experiments with means and variances.\n3. The complete datasets are available for further research.", "weaknesses": "1. The paper's current presentation lacks clarity and coherence, making it difficult to follow. Additionally, there are numerous minor grammatical and structural errors throughout the text.\n2. While the initial explosion stage involves large-scale data, the subsequent re-ranking and edge-ranking stages utilize significantly smaller datasets. This inconsistency undermines the paper's claim of working with large-scale industrial data.\n3. The paper's novelty is not effectively demonstrated through comparative analysis with existing work. Particularly in the introduction, while the authors enumerate the merits of the RecFlow dataset, they fail to provide meaningful comparisons with related work. The innovation of this research can only be discerned through prior knowledge of the field rather than through the authors' presentation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BxiKAKn2ww", "reviewer": "Reviewer_p5Ha", "summary": "This paper mainly focuses on introducing a dataset, RecFlow. This dataset contains full-flow recommendation data, including retrieval, pre-rank, coarse ranking, ranking, re-ranking, and edge ranking. Containing two periods, the datasets provide an opportunity to study full-stage recommendations in industry. The full-stage recommendation is widespread in industry recommendations and is supposed to be investigated. Some experiments are conducted to give examples of how to utilize this dataset.", "questions": "1. Actually, the samples in every stage are based on the filtered strategy in the previous stage. So, will this strategy bring bias? And if we use a different strategy, can the conclusion still hold? For example, in industry, from retrieval to pre-ranking usually consists of several strategies. How does this benchmark reflect this?\n\n2. Are the results from Table 4 to Table 7 reproducible?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. An essential and practical problem in real industry recommendation. The full-stage recommendation is widespread in the industry; this dataset really provides a new perspective on this problem.\n\n2. The collection strategy is provided, and privacy protection is carefully considered.\n\n3. Experiments are provided to show how to use this dataset.", "weaknesses": "1. Despite providing collection and analysis, the collection procedure should be provided in more detail to show that it is reasonable and correct. Moreover, the analysis is too simple, and more intuition about this dataset can be given.\n\n2. The experiments provided to show how to use this dataset are interesting. However, in line 079, the author argues that Recflow can provide merits of ten tasks. It should be supposed that the experiments on these tasks should be provided.\n\n3. There are some typos. For example, Line 314 Recall@100,500,100 should be 1000. The whole paper should be proofread.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zMHYrNilgg", "reviewer": "Reviewer_GxXY", "summary": "The paper presents RecFlow, an industrial-scale recommendation dataset that captures the full recommendation pipeline with multiple stages, featuring 38M user interactions and 1.9B samples collected from 42K users and 9M items over a span of 37 days. One of RecFlow’s innovations is its inclusion of unexposed items at each pipeline stage, allowing for important analysis of distribution shifts between training and serving environments. The dataset also supports multi-task recommendation and behavior modeling by capturing various user feedback signals.\n\nExperiments show that modeling stage-specific interactions and addressing distribution shift with RecFlow data improves recommendation performance, with some methods proving effective in real-world systems.", "questions": "1. What measures did you take to ensure the dataset is representative?\n2. How can the dataset help handling cold-start users/items better?\n3. Could you please provide more details on the online A/B testing setup and results?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1. The paper presents the first comprehensive large-scale dataset that captures the complete recommendation pipeline, filling a critical gap in the field where existing datasets only contain exposure data. It could enable further research into real-world problems that were previously difficult to study, eg: distribution shift, stage interaction effects.\n2. Good motivation is provided by clearly articulating the limitations of existing datasets and the importance of studying full recommendation pipelines.\n3. The dataset is well documented with clear descriptions of features, collection methodology and privacy protection measures. The privacy protection approach is also robust, using a combination of user consent, feature anonymization and careful data filtering.\n4. Experimental validation is thorough with multiple runs, standard deviation reporting and comprehensive ablation studies across different stages.", "weaknesses": "1. The paper doesn't adequately address the computational challenges of working with such a large dataset. Details about storage requirements and recommended sampling strategies would be valuable for practitioners.\n2. The multi-task learning potential of the dataset is mentioned but not thoroughly explored. Given the rich set of user feedback signals, this seems like a missed opportunity.\n3. While the authors mention online A/B testing validation, the details are sparse. More information about the production deployment and real-world performance would strengthen the paper's practical impact claims.\n4. Analysis of how the stage samples could help with cold-start recommendations problem could be a useful contribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SjwiCXORX4", "reviewer": "Reviewer_Y4hS", "summary": "This paper published a new dataset for multi-stage-funnel-based recommendation system, where the key difference from existing datasets is the inclusion of unexposed samples. Most existing datasets only contain samples that are exposed to users, and ranking and earlier-stage models are typically trained on exposed samples with user feedback. So inclusion of unexposed samples can facilitate research for many interesting problems in multi-stage recommendation, such as the distribution gap between training and serving, multi-stage consistency etc.", "questions": "what’s the rationale for partitioning the data collection into two periods? Understanding the rationale for partitioning the data collection into two periods would help readers assess the dataset's representativeness and potential use cases. Could you explain the reasoning behind this decision and discuss any differences between the two periods that researchers should be aware of?\n\ndo you also log content features for the items? content features such as text/image/video/etc. content description (e.g., metadata, or embedding representations etc.) can be very useful features for recommendation.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "As far as I know, this is a first contribution of benchmark datasets that includes multi-stage and unexposed samples. Could be useful for researching important problems in multi-stage recommendation systems.", "weaknesses": "The evaluation criterion for the quality of a benchmark dataset for industrial recommendation system should be fidelity to an actual online recommendation system. For example, if researchers come up with new algorithms with metrics improvement using this dataset, then when it’s deployed to a real online system, such improvement can be validated. So it would be great if the authors can demonstrate such fidelity to some extent, e.g., by running online A/B test to compare the online performance and offline metrics to see the correlation. \n\nThe rules for determining how many samples for each stage seem quite ad-hoc w/o explanation of considerations. Could you explain the considerations that went into determining the number of samples collected at each stage? Are these numbers representative of typical production systems?\n\nsome typos: \nline 239: datatse -> dataset\nline 256: quote in wrong direction \nline 296/399: 1e-1/1e-2 not well formatted \nline 308/361: randomly sampling -> randomly sampled", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)