SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Dataset description We construct our simulator KuaiSim on the KuaiRand [10] dataset, a large-scale unbiased sequential recommendation dataset collected from the Kuaishou<sup>4</sup> App. It elicits user preferences on the randomly exposed items to collect unbiased data. You can find detailed data collection process in Appendix A.1. The availability of unbiased sequential data allows us to conduct unbiased offline evaluations, which greatly aids in constructing our simulator. We specifically choose the KuaiRand-Pure version. By doing so, we maintain a focused dataset for constructing our simulator. The KuaiRand dataset encompasses 12 distinct feedback signals. We utilize six positive feedback signals (i.e., 'click', 'view time', 'like', 'comment', 'follow', and 'forward'). Additionally, we incorporate two negative feedback signals (i.e., 'hate' and 'leave'). The other feedback signals occur infrequently and are therefore excluded from our analysis. Furthermore, we broaden the scope of our simulator by incorporating the ML-1m<sup>5</sup> dataset. This dataset serves as a benchmark commonly used in RSs and includes ratings provided by 6,014 users for 3,417 movies. In our analysis of the ML-1m dataset, we consider movies with user ratings higher than 3 as positive samples, representing a 'like,' while the rest are deemed negative samples, representing a 'hate.'
Dataset process We standardize both datasets by preprocessing them into a cohesive format, ensuring that each record follows a sequential order of (user features, user history, exposed items, user feedback, and timestamp). The detailed statistics of the resulting dataset can be found in Table 2. To guarantee the data quality of the KuaiRand dataset, we apply 50-core filtering where videos with less than 50 occurrences are removed. To generate session data, we segment each day chronologically, treating each day as an individual session.
4https://www.kuaishou.com/cn
Figure 2: Data analysis on KuaiRand dataset. (a) User response distribution. (b) Same day request gap distribution. (c) Cross day request time gap distribution. (d) Return time distribution
Table 3: Benchmarks for the request level task of KuaiSim. Best values are in bold.
Algorithm	Average L-reward	Max L-reward	Coverage	ILD
CF	2.253 ± 0.024	4.039 ± 0.001	100.969 ± 7.193	0.543 ± 0.007
ListCVAE	2.075 ± 0.039	4.042 ± 0.001	446.100 ± 15.648	0.565 ± 0.004
PRM	2.174 ± 0.017	3.811 ± 0.003	27.520 ± 3.210	0.538 ± 0.004
Primary data analysis We present our primary data analysis on the KuaiRand-Pure dataset in Figure 2. A full analysis procedure and more detailed results can be found in Appendix A.2. In our observation: 1) Figure 2(a) displays the distribution of user feedback, showcasing the frequency of occurrences for the six positive and one negative ('hate')  $0/1$  feedback signals. Notably, the negative feedback signal, 'is hate,' appears the least frequently. Among the positive feedback signals, 'is click' exhibits the highest frequency, while 'is forward' has the lowest occurrence. 2) In Figure 2(b) and 2(c), we present the distribution of same day and cross day request gaps, respectively. These gaps represent the time intervals, in hours, between consecutive requests. The occurrences of different request time gaps follow a geometric distribution pattern. 3) By analyzing the frequency of session returns and plotting the return time distribution in Figure 2(d), we observe that the return time also adheres to a geometric distribution, i.e., given  $p_{\mathrm{ret}}$ , the  $d$ -th day return probability is  $(1 - p_{\mathrm{ret}})^{d-1}p_{\mathrm{ret}}$ . And note that the percentage of return time greater than 10 is nearly negligible.
4 Baselines and Evaluation Protocol
List-wise recommendation with request level simulator: For list-wise recommendation, we first consider a standard collaborative filtering baseline CF [17]. It adopts a point-wise approach, where the user-item scoring function is trained through binary cross entropy. We then include a generative approach ListCVAE [16], which captures the distribution of recommended lists using a conditional VAE. We also include a reranking approach PRM [36], which is built upon a CF-based initial ranker and utilizes a transformer-based re-ranker to encode the intermediate candidate set.
Sequential recommendation with whole-session simulator: With the standard MDP formulation, we include the following RL methods as comparisons: A2C [31] is a synchronized version of A3C [31] that applies the policy gradient algorithm on the actor and TD error learning on the critic. SA2C [50] combines a negative sampling strategy with supervised sequential learning to enhance reward signal. DDPG [26] is a deep deterministic policy gradient algorithm and the continuous action space is used to represent the recommendation list, for both the actors and critics. TD3 [9] is an extension of DDPG that improves the stability of training by incorporating the clipped double Q-learning. HAC [28] is also an extension of DDPG that decomposes the generation of item lists into a hyper-action(embedding) inference step and an effect-action(item list) selection step.
Retention optimization with cross-session simulator: The retention optimization task is essentially a hyper-parameter search problem. We first consider CEM [39] as a black-box optimization method. It iteratively samples and updates a population of candidate hyper-parameters based on their performance. We then include TD3 [9] method similar to that in whole-session level, but use return time as the reward. We also include RLUR [3] as the state-of-the-art method. It uses intrinsic rewards [34] to enhance policy learning and introduces a novel soft regularization method to address the trade-off between sample efficiency and stability.
Table 4: Benchmarks for the whole-session task of KuaiSim. Best values are in bold.
Algorithm	Depth	Average reward	Total reward	Coverage	ILD
TD3	14.63 ± 0.03	0.6476 ± 0.0028	9.4326 ± 0.0756	24.20 ± 2.55	0.9864 ± 0.0004
A2C	14.02 ± 0.02	0.5950 ± 0.0019	8.3905 ± 0.1026	27.41 ± 1.08	0.9870 ± 0.0002
SA2C	14.34 ± 0.02	0.6251 ± 0.0014	8.9547 ± 0.0241	27.14 ± 2.01	0.9872 ± 0.0002
DDPG	14.89 ± 0.04	0.6841 ± 0.0013	10.0873 ± 0.0571	20.95 ± 3.27	0.9850 ± 0.0006
HAC	14.98 ± 0.03	0.6895 ± 0.0017	10.1742 ± 0.0634	35.70 ± 1.22	0.9874 ± 0.0004
Table 5: Benchmarks for the cross-session task of KuaiSim. Best values are in bold.
Algorithm	Return day ↓	User retention ↑
CEM	3.573 ± 0.012	0.572 ± 0.002
TD3	3.556 ± 0.010	0.581 ± 0.001
RLUR	3.481 ± 0.010	0.607 ± 0.002
↑: the higher the better; ↓: the lower the better.
4.2 Evaluation Protocol
List-wise recommendation with request level simulator: List-wise reward (L-reward) is the average of item-wise immediate reward. We use both the average L-reward and the max L-reward across user requests in a mini-batch. Coverage describes the number of distinct items exposed in a mini-batch. Intra-list diversity (ILD) estimates the dissimilarity between items in each recommended list, based on item embedding.
Sequential recommendation with whole-session simulator: Besides coverage and ILD, we also use other metrics. Whole-session reward: total reward is the average sum of immediate rewards for each session. The average reward is the average of total reward for each request. Depth represents how many interactions before the user leaves.
Retention optimization with cross-session simulator: Return day is the average time gap (day) between the last request of session and the first request of session. User retention is the average ratio of visiting the system again.
5 Benchmark Results and Analysis
We provide benchmark results for all three tasks in this section. To ensure the reproducibility and reliability of results, we computer averages across five distinct sets of results. The  $\pm$  represents the standard deviation of five results. Detailed experiment implementation and hyper-parameter settings are provided in Appendix B.
List-wise recommendation with request level simulator: As shown in Table 3, among the evaluated models, ListCVAE demonstrates the best performance in terms of maximum reward and diversity. Its ability to generate diverse and high-reward recommendations makes it an effective choice for list-wise recommendation tasks. On the other hand, PRM exhibits the worst performance among the evaluated models. A significant challenge in this task lies in efficiently searching the extensive combinatorial space of list actions. A promising avenue for future research involves addressing this challenge by simultaneously improving the diversity of recommendation results and reducing the complexity of the combination space.
Sequential recommendation with whole-session simulator: As shown in Table 4, the HAC framework consistently demonstrates superior performance across long-term metrics, showcasing the effectiveness of its hyper-actions and the efficiency of its learning method. Notably, HAC outperforms other frameworks, indicating its high level of expressiveness and efficacy in recommendation tasks. On the other hand, A2C exhibits the poorest performance and appears to be the most unstable learning framework among the evaluated methods. SA2C outperforms A2C by capitalizing on an improved reward signal generated through negative sampling and supervised training. While slightly trailing behind HAC, the DDPG framework also delivers commendable results. Current methods in this task often overlook the importance of incorporating long-term feedback. Therefore, a promising research direction lies in exploring how to model complex inter-session relationships effectively.
Table 6: A comparison between KuaiSim and other simulators. Best values are in bold.
Simulators	Depth	Average reward	Total reward	AUC
RL4RS	14.39 ± 0.02	0.640 ± 0.015	9.235 ± 0.122	0.6929 ± 0.0019
Recogym	13.55 ± 0.01	0.535 ± 0.013	7.194 ± 0.109	0.6729 ± 0.0026
RecSim	14.05 ± 0.02	0.588 ± 0.006	9.347 ± 0.143	0.6842 ± 0.0031
VirtualTaobao	14.45 ± 0.02	0.646 ± 0.009	9.570 ± 0.077	0.6866 ± 0.0014
KuaiSim	14.86* ± 0.01	0.679* ± 0.011	10.081* ± 0.116	0.7234* ± 0.0021
“*” indicates the statistically significant improvements (i.e., two-sided t-test with  $p < 0.05$ ) over the best baseline.
Table 7: Benchmarks for the whole-session task on ML-1m datasets. Best values are in bold.
Algorithm	Depth	Average Reward	Total reward	Coverage	ILD
TD3	13.50 ± 0.01	0.5388 ± 0.007	7.4035 ± 0.0152	44.18 ± 2.19	0.9866 ± 0.0001
A2C	13.55 ± 0.01	0.5468 ± 0.002	7.4487 ± 0.0171	27.31 ± 1.44	0.9870 ± 0.0001
DDPG	13.64 ± 0.02	0.5476 ± 0.005	7.5333 ± 0.0273	61.03 ± 2.08	0.9871 ± 0.0001
HAC	13.70 ± 0.01	0.5482 ± 0.008	7.5791 ± 0.0206	29.85 ± 1.92	0.9872 ± 0.0001
Retention optimization with cross-session simulator: As shown in Table 5, TD3 demonstrates better performance than CEM in terms of both metrics, showcasing the effectiveness of reinforcement learning techniques. However, RLUR surpasses both TD3 and CEM significantly, indicating its superior performance in the evaluated tasks. Indeed, the exploration of this task is still in its early stages. While some research has attempted to model user retention on a day-by-day basis, capturing more enduring user feedback has considerable potential in exploring approaches.
Comparing with other simulators In section 2.4, we compare our KuaiSim with other simulators qualitatively in terms of datasets and tasks, showing that our KuaiSim is the only one that meets all requirements. In this section, we compare our KuaiSim with other simulators quantitatively. We reconstruct some competitive simulators on the KuaiRand dataset for the whole-session task to illustrate the effectiveness of our KuaiSim. In order to evaluate the simulator, we divide it into two planes of evaluation. On the one hand, how accurately the simulator fits the environment. Since some simulators have only one feedback signal, click, we compare the AUC predicted for this signal. On the other hand, the agent is trained using the simulator. We utilize three metrics proposed in the whole-session evaluation protocol, depth, average reward, and total reward. As shown in Table 6, we utilize the DDPG algorithm to train an agent with different simulators. Among these simulators, KuaiSim outperforms the others by a significant margin across all evaluation metrics. This outcome demonstrates that KuaiSim accurately aligns with the environment, resulting in superior agent training. Compared to rule-based simulators like RecSim and RecoGym, our simulator can be trained in a supervised manner to fit the real environment better. Additionally, compared to Virtual-Taobao and RL4RS, KuaiSim directly samples users from datasets without the need for fitting user states, avoiding potential errors. These distinctions highlight the strengths of our simulator, KuaiSim, which contributes to its enhanced accuracy and reliability in simulating user behavior and preferences.
Data migration analysis To showcase the data migration capabilities of our construction process, we implement KuaiSim on the ML-1m dataset and evaluate its performance on the whole session task. The benchmark results, as presented in Table 7, indicate that HAC continues to outperform other methods in all metrics, except for coverage. Notably, DDPG exhibits the highest coverage and achieves the best diversity in the recommended results. On the other hand, A2C outperforms TD3, positioning itself as a stronger-performing model compared to the latter. TD3 exhibits the poorest performance among the evaluated models. In a nutshell, these findings show that our KuaiSim can work well on ML-1m and emphasize the effectiveness of KuaiSim in adapting to different datasets.
Parameter analysis We further investigate the effect of some vital hyper-parameters for training the Algorithm TD3 with our KuaiSim. We examine the influence of max time step in the range of [5, 10, 15, 20, 25, 30], the results are presented in Figure 3 (a). Optimal model performance is achieved when the max time step is set to 20, and the return day fluctuates between 2.2 and 2.3 for other settings. Additionally, we delve into the effect of slate size, varying it across [5, 10, 15, 20,
Figure 3: Parameter sensitive analysis on KuaiRand dataset. (a) Max time step. (b) Slate size.
25, 30]. the results are shown in Figure 3 (b). The best model results are obtained when the slate size is set to 20. The performance exhibits a decline as the slate size deviated from this optimum point, either increasing or decreasing. This is because when the slate size is excessively small, the recommendation list fails to encompass items that align with the user's preferences. Conversely, when the slate size is overly large, the recommendation list contains too much noise that is not of interest to the user. This overload of information not only diverts the user's focus but also tests their patience. The model's outcomes demonstrate a consistent and stable pattern across different parameter settings, underscoring the robust stability of our KuaiSim training agent.
Limitations and potential direction for solutions There remain certain areas within our RecSim framework that warrant refinement. Uncovered recommendation problems: While we propose three levels of recommendation tasks to encompass a majority of recommendation challenges, numerous others exist, including explainable recommender systems [6, 33], diversity recommendations [13, 19], and fairness recommendations [47, 23]. Incorporating the interpretability, diversity, fairness, and other indicators of the recommendation results into the modeling of the simulator is a promising strategy to broaden our simulator's scope. Expanding to diverse domains: While both the KuaiRand and ML-1m datasets center around video recommendations, it's worth noting that the KuaiSim framework can be extended to encompass a variety of domains such as music, news, and e-commerce. The iterative refinement process we've undertaken for constructing and evaluating the simulator greatly facilitates such expansions.
Ethical considerations and potential societal impact We categorize our ethical considerations and potential societal impact into three key aspects. Safeguarding user privacy: While our approach involves training the simulator using real user logs, it's important to note that the datasets we employed have been stripped of any privacy-sensitive information. The user characteristics utilized in our study are comprehensively presented in Appendix A.1 Table 2. These characteristics reflect the user's interactions with the application, and each user is denoted by an anonymous uid. Thus, it's essential to emphasize that we do not leverage any private user information, effectively mitigating concerns related to privacy breaches. Unbiased KuaiRand dataset: Because online A/B test usually consumes much time and money, which makes it impractical for academic researchers to conduct the evaluation online. However, offline evaluation will cause bias because of massive missing data, i.e., the user-item pairs that have not occurred in the test set. One fundamental approach to address this issue in offline evaluation is to collect unbiased data, i.e., to elicit user preferences on the randomly exposed items. This unbiased dataset empowers us to conduct unbiased offline evaluations without compromising user privacy. The detailed data collection process can be found in Appendix A.1. Promoting field advancement: We provide various competitive algorithms for three task levels as benchmarks, coupled with a meticulously enhanced process for constructing and evaluating simulators. This framework not only facilitates the advancement of the field but also presents a promising avenue for research by extending these simulators to encompass a broader spectrum of tasks and fields.
In summary, KuaiSim stands as a comprehensive and sophisticated simulator that encompasses multiple task levels, establishing benchmarks and enabling thorough evaluations in the realm of recommendation systems. Through its refined construction and evaluation process, as well as its effectiveness in replicating user behaviors, KuaiSim contributes to the development and advancement of recommendation system techniques and methodologies.
ACKNOWLEDGEMENT
This research was supported by Research Impact Fund (No. R1015-23). It was also supported by Kuaishou. It was also partially supported by APRC - CityU New Research Initiatives (No.9610565, Start-up Grant for New Faculty of City University of Hong Kong), CityU - HKIDS Early Career Research Grant (No.9360163), Hong Kong ITC Innovation and Technology Fund Midstream Research Programme for Universities Project (No.ITS/034/22MS), Hong Kong Environmental and Conservation Fund (No. 88/2022), and SIRG - CityU Strategic Interdisciplinary Research Grant (No.7020046, No.7020074).
[1] M Mehdi Afsar, Trafford Crump, and Behrouz Far. 2022. Reinforcement learning based recommender systems: A survey. Comput. Surveys 55, 7 (2022), 1-38.
[2] Qingyao Ai, Keping Bi, Jiafeng Guo, and W Bruce Croft. 2018. Learning a deep listwise context model for ranking refinement. In The 41st international ACM SIGIR conference on research & development in information retrieval. 135-144.
[3] Qingpeng Cai, Shuchang Liu, Xueliang Wang, Tianyou Zuo, Wentao Xie, Bin Yang, Dong Zheng, Peng Jiang, and Kun Gai. 2023. Reinforcing User Retention in a Billion Scale Short Video Recommender System. In Companion Proceedings of the ACM Web Conference 2023. 421-426.
[4] Qingpeng Cai, Zhenghai Xue, Chi Zhang, Wanqi Xue, Shuchang Liu, Ruohan Zhan, Xueliang Wang, Tianyou Zuo, Wentao Xie, Dong Zheng, et al. 2023. Two-Stage Constrained Actor-Critic for Short Video Recommendation. In Proceedings of the ACM Web Conference 2023. 865–875.
[5] Xiaocong Chen, Lina Yao, Julian McAuley, Guanglin Zhou, and Xianzhi Wang. 2021. A survey of deep reinforcement learning in recommender systems: A systematic review and future directions. arXiv preprint arXiv:2109.03540 (2021).
[6] Xu Chen, Yongfeng Zhang, and Ji-Rong Wen. 2022. Measuring "why" in recommender systems: A comprehensive survey on the evaluation of explainable recommendation. arXiv preprint arXiv:2202.06466 (2022).
[7] Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, et al. 2016. Wide & deep learning for recommender systems. In Proceedings of the 1st workshop on deep learning for recommender systems. 7-10.
[8] Denise de Oliveira, Ana LC Bazzan, Bruno Castro da Silva, Eduardo W Basso, Luis Nunes, Rosaldo Rossetti, Eugenio de Oliveira, Roberto da Silva, and Luis Lamb. 2006. Reinforcement Learning based Control of Traffic Lights in Non-stationary Environments: A Case Study in a Microscopic Simulator.. In EUMAS.
[9] Scott Fujimoto, Herke Hoof, and David Meger. 2018. Addressing function approximation error in actor-critic methods. In International conference on machine learning. PMLR, 1587-1596.
[10] Chongming Gao, Shijun Li, Yuan Zhang, Jiawei Chen, Biao Li, Wenqiang Lei, Peng Jiang, and Xiangnan He. 2022. KuaiRand: An Unbiased Sequential Recommendation Dataset with Randomly Exposed Videos. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management. 3953-3957.
[11] Jin Huang, Harrie Oosterhuis, Maarten De Rijke, and Herke Van Hoof. 2020. Keeping dataset biases out of the simulation: A debiased simulator for reinforcement learning based recommender systems. In Proceedings of the 14th ACM Conference on Recommender Systems. 190-199.
[12] Liwei Huang, Mingsheng Fu, Fan Li, Hong Qu, Yangjun Liu, and Wenyu Chen. 2021. A deep reinforcement learning based long-term recommender system. Knowledge-Based Systems 213 (2021), 106706.
[13] Bei Hui, Lizong Zhang, Xue Zhou, Xiao Wen, and Yuhui Nian. 2022. Personalized recommendation system based on knowledge embedding and historical behavior. Applied Intelligence (2022), 1-13.
A Detailed Dataset Description and Analysis
A.1 Detailed Feature Description
Table 1: Detailed interaction information in KuaiRand dataset.
Interaction field	Feature type	Explanation
user id	int64	The unique identifier for the user.
video id	int64	The unique identifier for the video.
date	int64	The date when the interaction occurred.
hour min	int64	The time of the interaction in hours and minutes.
time ms	int64	The timestamp of the interaction in milliseconds.
is click	int64	The user feedback of click.
is like	int64	Indicating if the user liked the video.
is follow	int64	Indicating if the user followed the author.
is comment	int64	Indicating if the user wrote a comment.
is forward	int64	A binary signal indicating if the user forwarded the video.
is hate	int64	A binary signal indicating if the user disliked the video.
long view	int64	Indicating the completeness of the video.
Table 2: Rich user feature in KuaiRand dataset.
User feature	Feature type	Explanation
user id	int64	The unique identifier for the user.
user active degree	str	The level of user activity, classified as ‘high active’, ‘full active’, ‘middle active’, or ‘UNKNOWN’.
is live streamer	int64	Indicates whether the user is a live streamer.
is video author	int64	Indicates if the user has uploaded any videos.
follow user num range	str	The number range of followed users.
fans user num range	str	The number range of fans.
friend user num range	str	The number range of friends.
register days range	str	The range of the number of days since user registration.
one-hot features	int64	Encrypted features of the user.
Table 3: Rich video feature in KuaiRand dataset.
User feature	Feature type	Explanation
video id	int64	The unique identifier for the video.
author id	int64	The unique identifier for the author of the video.
video type	str	The type of the video, categorized as "NORMAL" or "AD".
upload type	str	The upload type of the video.
music type	int64	The background music type used in the video.
tag	str	A list of key categories or labels associated with the video.