SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Kesen Zhao $^{1}$ , Shuchang Liu $^{2*}$ , Qingpeng Cai $^{2\dagger}$ , Xiangyu Zhao $^{1\dagger}$ , Ziru Liu $^{1}$ , Dong Zheng $^{2}$ , Peng Jiang $^{2}$ , Kun Gai $^{3}$
$^{1}$ City University of Hong Kong,  $^{2}$ Kuaishou Technology,  $^{3}$ Unaffiliated {kesenzhao2-c, ziruliu2-c}@my.cityu.edu.hk
{liushuchang, caiqingpeng, zhengdong, jiangpeng}@kuaishou.com xianzhao@cityu.edu.hk, gai.kun@qq.com
Reinforcement Learning (RL)-based recommender systems (RSs) have garnered considerable attention due to their ability to learn optimal recommendation policies and maximize long-term user rewards. However, deploying RL models directly in online environments and generating authentic data through A/B tests can pose challenges and require substantial resources. Simulators offer an alternative approach by providing training and evaluation environments for RS models, reducing reliance on real-world data. Existing simulators have shown promising results but also have limitations such as simplified user feedback, lack of consistency with real-world data, the challenge of simulator evaluation, and difficulties in migration and expansion across RSs. To address these challenges, we propose KuaiSim, a comprehensive user environment that provides user feedback with multi-behavior and cross-session responses. The resulting simulator can support three levels of recommendation problems: the request level list-wise recommendation task, the whole-session level sequential recommendation task, and the cross-session level retention optimization task. For each task, KuaiSim also provides evaluation protocols and baseline recommendation algorithms that further serve as benchmarks for future research. We also restructure existing competitive simulators on the KuaiRand Dataset and compare them against KuaiSim to further assess their performance and behavioral differences. Furthermore, to showcase KuaiSim's flexibility in accommodating different datasets, we demonstrate its versatility and robustness when deploying it on the ML-1m dataset. The implementation code is available online to ease reproducibility<sup>3</sup>.
Reinforcement Learning (RL)-based recommender systems (RSs) have drawn considerable attention in both academia and industry [45, 49, 5, 3, 4, 52, 51, 55]. They regard the recommendation procedures as sequential interactions between users and agents, and learn an optimal recommendation policy that maximizes the long-term cumulative reward from users. While it is theoretically superior to standard learning-to-rank methods [2], it is usually sub-optimal to evaluate the RL model using offline data, because of the missing online exploration and the impossibility of counterfactual evaluation. As an alternative, one may directly deploy RL models in online environments and let them interact with real users. In practice, this turns out to be challenging and resource-intensive [58], since an untrained or premature recommendation model can adversely impact the user experience and lead to undesirable real-time data, subsequently affecting the model's training performance [20]. In order to
*The first two authors contributed equally to this work
<sup>3</sup>https://github.com/Applied-Machine-Learning-Lab/KuaiSim
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Bechmarks.
mitigate these challenges, the research of user simulators has emerged recently to serve as a pre-online verification method for RL models [8, 14, 11, 37, 44, 40, 57]. Intuitively, simulators offer a practical solution for simulating user responses, allowing for the training and evaluation of RS models that are challenging to assess using offline data alone. And it enables researchers to iteratively improve RS models without relying exclusively on real-time user interactions.
Despite the effectiveness of existing simulators in some specially designed recommendation scenarios [14, 44], they are still far from the real-world environment in several aspects, which limits their effectiveness in evaluating online learning models: 1) While the most standard setup simulates single immediate feedback, user responses in most web services are multi-behavior in essence, for example, in short-video recommendation platforms, users can click, like, forward, or download. 2) In addition to the aforementioned immediate responses, a user may leave the app and then come back, generating a leave signal and a retention signal, but existing simulators do not address these long-term or delayed behaviors. The retention signal, in particular, is closely related to some crucial evaluation metrics (i.e., the daily average users) of online systems [54, 27], which could be identified as an essential building block that is still seriously underestimated in the research field. 3) Many rule-based simulators [14] are specially constructed for certain recommendation scenarios that may restrict their accommodation towards other recommendation services. 4) More recent simulators [44, 41] addressed the previous issue through pretraining of simulators according to log data, but during online interactions, they need to sample a user according to a pretrained user generator in addition to the user response model. This may further amplify the inconsistency between the simulated environment and the real-world data distribution. 5) Finally, simulators are capable of evaluating recommendation models, but there is still limited research on how to evaluate a simulator. In other words, we can only be certain of the superiority of an RL method when it achieves better results on the simulator and the simulator is consistent with the real-world environment.
To meet this demand, we propose KuaiSim, a comprehensive simulator that provides a user response environment at three distinct task levels: the request level recommendation task addresses the multi-behavior feedback and list-wise evaluation, the whole-session level sequential recommendation task addresses the long-term reward optimization under the standard RL setting, and cross-session level recommendation task addresses the retention optimization problem. The resulting simulator consists of a user immediate response model that generates feedback for each recommendation, a user leave model that specifies the end of a session, and a user retention model that determines how long the user returns to the system and starts a new session. To ensure its consistency with the real-world environment, KuaiSim uses the log data to pretrain the user response models and engage user sampling during simulation. This also allows for flexible adaptation on other datasets as long as the required data format suffices. In summary, our contribution can be summarized as follows:
- We propose a comprehensive simulator, KuaiSim, that covers three levels of recommendation tasks to encompass a majority of recommendation challenges. We also provide comparisons of various competitive algorithms for each task as benchmarks to boost future research work.
- Our refined simulator construction and evaluation process is user-friendly and flexible, and we showcase the data migration capabilities of KuaiSim on both the original dataset KuaiRand and the widely available ML-1m benchmark dataset.
- Additionally, we conduct a comparative analysis of KuaiSim against existing simulators, including RecoGym [38], RecSim [14], RL4RS [44], and VirtualTaobao [41]. The results demonstrate that our simulator excels in its ability to approximate real-world environments.
2.1 List-wise Recommendation
The list-wise recommendation problem focuses on the interconnected nature of items within a presented list, as highlighted in our previous studies [56, 59, 60]. The main objective is to understand and learn from the distinctions between including or excluding specific items in the exposed list [2, 32]. Recent research has indicated that in multi-stage recommendation systems, re-ranking models can effectively capture item correlations due to the reduced candidate set size, enabling the utilization of powerful neural models [29]. In the past few years, there has been an ongoing discussion regarding the generative perspective of list-wise recommendation [16, 29]. To address the challenge of the vast combinatorial output space for lists, the generative approach directly models the distribution of recommended lists and generates entire lists using deep generative models. For instance, ListCVAE [16] employs Conditional Variational Autoencoders (CVAE) to capture positional biases and item interdependencies within the list distribution. However, our previous studies [29] have revealed that
Table 1: A comparison between KuaiSim and other simulators.
Simulators	Real dataset	Recommendation task
Request-level	Whole-session	Cross-session
RecoGym	✓
RecSim	✓
RL4RS	✓	✓	✓
Virtual-Taobao	✓	✓
KuaiSim	✓	✓	✓	✓
ListCVAE struggles with accuracy-diversity trade-offs. One limitation of these methods is that they fail to consider the broader impact of a single recommendation on the entire session.
2.2 Sequential Recommendation
The session-based recommendation (SBR) considers the beginning and end of an interaction session. Our ultimate objective is to optimize the overall future reward of the user session, which aligns more with the notion of partial SBR as defined in [44]. In our settings, the prevailing solution relies on the Markov Chain assumption to model the dynamic transitions in user-system interactions. The primary challenge in this line of research is how to construct a representative user state based on extensive historical data [22, 25]. Initial solutions to the recommendation problem utilized collaborative filtering techniques [7, 18, 24, 35], while subsequent approaches incorporated deep neural networks [48] such as Recurrent Neural Networks [15], Convolution Networks [43] and Self-Attention [42, 53, 21] to enhance the model's expressive power. These techniques aim to capture the rich and intricate information contained in user and item features, as well as their interaction histories. However, the common underlying principle of these methods is to accurately encode long-term histories without directly optimizing for long-term user rewards.
2.3 Retention Optimization
The objective of the next session recommendation is to suggest the next session, based on the analysis of inter-session dependencies within historical sessions [46]. In addition to considering dependencies within individual sessions, the inclusion of inter-session dependencies, such as user retention, has been proven to be crucial for improving the overall performance of recommendations by our previous works [54, 3]. Multi-level session data exhibits a hierarchical structure, encompassing multiple levels, including interaction and session levels. Within this framework, both the dependencies within each level and those across different levels play a pivotal role in influencing subsequent recommendations. For instance, the recommended results from the previous session can impact the time interval before the user initiates the next session (return time). Consequently, effectively capturing and accurately modeling inter-level dependencies, while optimizing user retention, poses a significant challenge [3]. Unfortunately, there is a scarcity of research addressing this area. In response to this need, our simulator offers a unique capability to predict the return time of users, providing a valuable tool for conducting in-depth research in this under-explored domain.
2.4 Existing Simulators
The efficacy of reinforcement learning in recommendation systems is attributable to its ability to learn from user feedback and adapt to evolving user preferences over extended periods [1, 12, 30, 61]. However, the application of reinforcement learning to real-world tasks can be challenging due to the large volume of data involved, which can result in high sampling costs [41]. To address this challenge, researchers have developed simulators for recommender systems using reinforcement learning (RL) algorithms. In this paper, we provide an overview of the existing four simulators.
RecoGym [38] RecoGym overcomes the issue of exploding variation and allows for the simulation of user reactions to arbitrary recommendation policies. It is based on a cycle of collecting performance data, developing a new version of the recommendation model, A/B testing, and rolling out, which shares similarities with the RL setup.
RecSim [14] Recsim is a configurable simulation platform, which supports sequential interaction with users and allows for easy configuration of environments with varying assumptions about user preferences, item familiarity, user latent state and dynamics, and choice models.
Figure 1: The workflow of simulator and a common MDP setting for user-system interaction.
RL4RS [44] Current research on RL-based recommender systems (RL-based RS) lacks a properly validated simulation environment, advanced evaluation methods, and a real dataset, leading to a reality gap. To address this, the RL4RS dataset is introduced, which aims to bridge the reality gap in current RL-based RS research. The dataset is collected from a popular game by NetEase Games and is anonymized using a three-phase anonymization procedure.
Virtual-Taobao [41] Virtual-Taobao is a simulator that is trained using historical customer behavior data. To enhance the simulation accuracy, it utilizes GAN-SD (GAN for Simulating Distributions) and MAIL (Multi-agent Adversarial Imitation Learning) for generating customer features with better distribution matching and more generalizable customer actions.
But all these simulators ignore the optimization of long-term feedback, such as user retention, and mostly support only a single recommended task. In Table 1, we summarize the characteristics of existing works and our KuaiSim in terms of datasets and tasks. It can be seen that our KuaiSim benchmark is the only one that meets all requirements.
This section begins by providing a comprehensive explanation of the fundamental definitions and settings pertaining to our three tasks: request level, whole-session level, and cross-session level. Subsequently, we delve into the intricacies of constructing our simulator, offering a detailed account of the process. Lastly, we present an introduction to the dataset we employ and conduct a thorough analysis of the KuaiRand dataset.
3.1 Multi-level Reinforcement Learning-based Recommendation
We consider the RL-based recommendation task as a sequential interaction between a recommender system (agent) and users (environment) and emphasize the multi-behavior and cross-session nature of this interaction. Specifically, users may provide short-term feedback in different formats such as clicks, views, and likes; and they may also provide delayed long-term feedback like the retention signal that can only be observed when users return to the system. We provide the overview of this comprehensive interaction paradigm as Figure 1, which is a cross-session multi-behavior user-system interaction flow. Formally, in the  $t$ -th interaction step in a specific session, the recommendation system receives a user request  $\mathcal{O}_t$  and generates a recommendation  $\mathcal{A}_t$  as action. Here we assume that each user request consists of a set of static user profile features  $\mathcal{U}$  and the most recent interaction history  $\mathcal{H}_{:t-1}$  that dynamically changes through time. We also assume a candidate item set  $\mathcal{C}$ , so a typical recommendation action has  $\mathcal{A}_t \in \mathcal{C}^K$ , which is a list of size  $K$ . And a user session typically refers to the continuous interactions of the user from the starting point when opening the app until the exit. On the other side of the flow, the user receives the recommendation and provides the user feedback  $\mathcal{Y}_t$  of three types: 1) The immediate feedback  $\mathcal{Y}_t^{(I)} \in \mathbb{R}^{b \times K}$  directly reveals the user's preference of the recommended items, and  $b$  represents the number of behavior signal types; 2) The leave signal  $\mathcal{Y}_t^{(L)} \in \{0,1\}$  specifies whether the user stops the current session and exits the app; 3) If the user leaves the current session, there is an additional return time (i.e. retention) signal  $\mathcal{Y}_t^{(R)}$  that indicates how long the user will come back and start a new session. Then, we can formulate the three-level learning tasks with different learning goals and each focuses on a specific research question, namely, the request level, the session-level, and the cross-session level.
Algorithm 1 Step — the detail workflow of KuaiSim.
Input Format: observation  $\mathcal{U}$ $\mathcal{H}_{t - 1}$  , and recommendation action  $\mathcal{A}_t$    
Output: immediate feedback  $\mathcal{V}_t^{(I)}$  , leave signal  $\mathcal{V}_t^{(L)}$  , and retention signal  $\mathcal{V}_t^{(R)}$  (cross-session) The user immediate response module:   
1: User history encoding  $\mathbf{h}_t\gets$  Transformer  $(\mathcal{U},\mathcal{H}_{:t - 1})$    
2: Ground truth user state  $\mathbf{s}_t\gets \mathbf{h}_t\oplus \mathcal{U}$    
3: Behavior attention  $w_{t}\leftarrow \mathrm{DNN}(\mathbf{s}_{t})$    
4: Behavior likelihood  $p(y|\mathbf{s}_t)\gets w_t\odot \mathcal{A}_t - \rho \times$  item_corrlation(At)   
5: Sample final immediate feedback  $\mathcal{V}_t^{(I)}\sim p(y|\mathbf{s}_t)$    
The user leave module:   
6: Immediate reward  $r_t\gets$  reward_func  $\mathcal{V}_t^{(I)})$    
7: User temper  $\leftarrow$  user temper - immediate reward   
8: Leave signal  $\mathcal{V}_t^{(L)}\gets 1$  if user temper  $\leq \mathbb{T};0$  otherwise   
The user retention module:   
9: Personal retention bias  $b_{u}\gets$  DNN(s  $t$    
10: Response retention bias  $b_{r}\gets \lambda_{1}r_{t}$    
11: Next day return probability  $p_{\mathrm{ret}}\gets b_u + b_r + \lambda_2b,$  where  $b$  is the global retention bias   
12: Return time  $\mathcal{V}_t^{(R)}\sim$  Geometric  $(p_{\mathrm{ret}})$  if  $\mathcal{V}_t^{(L)} = 1$  , otherwise  $\mathcal{V}_t^{(R)} = 0$  Post processing module:   
13: Update user history  $\mathcal{H}_{:t}\gets \mathcal{H}_{:t - 1}\oplus (\mathcal{A}_t,\mathcal{V}_t^{(I)},\mathcal{V}_t^{(L)},\mathcal{V}_t^{(R)})$    
14: If  $\mathcal{V}_t^{(L)} == 1$  , the user leave the current session   
15: Else if not reaching the max session number, then continue.   
16: Otherwise, sample a new user U, H from data and replace the current user.
Request level This problem focuses on single request optimization where the recommendation list could be optimized in a list-wise way. Different from the standard learning-to-rank paradigm where a point-wise model is trained to predict the ranking score of each individual item, the list-wise recommendation assumes that each recommendation action corresponds to a list of items that are exposed to the user in order, and it addresses the existence of item correlations within recommendation lists [29]. In this case, conventional point-wise evaluations like Recall and NDCG metrics become ineffective in evaluating these cross-item patterns, and simulators are introduced to fill in the gap.
Whole-session level When we look at each user session as a whole and want to optimize the overall recommendation performance of the entire session, cross-request influences are included in the picture and the problem corresponds to the most standard MDP setting for RL-based models: Each recommendation drives a user state transition and thus the long-term performance in the session, which could be effectively modeled as a Markov Decision Process [1].
Cross-session level Further expanding our view to the entire interaction sequences across sessions, an extra user return time signal becomes observable, which refers to the time interval between the last request of a session and the first request of the subsequent session. This introduces an additional learning goal of retention optimization, which aims to learn a policy that provides satisfactory recommendations that can reduce the user's return time. Note that this learning goal is directly related to daily average users (DAU) which is the core evaluation metric for many web services. However, its optimization is extremely challenging for its uncertainty and uncontrollable delay [3].
3.2 Simulator Building Blocks
As we would simulate the online environment and provide a training and evaluation platform for all aforementioned tasks, our simulator aims to provide the multi-behavior immediate responses, the user leave model, and the user retention model. Formally, the user simulator is a response generation function for recommendation actions  $S: \mathcal{O}_t, \mathcal{A}_t \to \mathcal{Y}_t$ . We provide a detailed workflow of this function in Algorithm 1 which consists of three feedback generation modules:
User immediate response module (UIRM) is responsible for generating the user's immediate feedback  $\mathcal{Y}_t^{(I)}$ . It first infers a ground-truth user state  $\mathbf{s}_t$  (assumed implicitly from RL models), then outputs the behavioral likelihood for each immediate feedback type. Specifically,  $\oplus$  represents concatenation and  $\odot$  represents dot product. The item_corr function is introduced to suppress the positive behaviors for items with higher correlation with other items in the same recommendation
Table 2: Statistics of datasets.
Datasets	Users	Items	Interactions	Sessions	Density
KuaiRand-Pure	27077	7551	1,436,609	246738	0.70%
ML-1m	6,400	3,706	1,000,208	16629	4.22%
list. This behavior simulates the user's demands for item diversity since lower item correlation induces a higher chance of positive interactions. In order to ensure the validity of the UIRM model, we require pretraining on the log data, and for each immediate feedback type, we can adapt the standard point-wise learning using binary cross-entropy. As a result, any datasets that provide logs of user-wise recommendation-feedback sequences would sufficiently support this module.
User leave module maintains a user temper/patience factor that directly determines the leave signal. We assume a maximum length of each user session, and initialize the user temper as equal to this maximum length. Then the user gradually loses the temper during the interactions and finally leaves the session when the temper is too low. In each step, the  $\mathcal{Y}^{(I)}$  inferred by the UIRM is reused to calculate the immediate reward that represents the user's satisfaction with the  $\mathcal{A}_t$ . We assume that less satisfactory recommendations make users lose their temper faster. The initial temper value, the decrease rate of temper, and the leave threshold are all adjustable hyperparameters.
User retention module is specifically designed for cross-session tasks, which predicts the user's return time. The user's return time typically follows a geometric distribution (as we will further discuss in section 3.3), so we only predict a next-day return probability  $p_{\mathrm{ret}}$  to simulate this behavior. Specifically,  $p_{\mathrm{ret}}$  combines the global retention bias, a personal retention bias, and a response retention bias. The personal retention bias reflects the differences in user's activity level, e.g., highly active users may use the system every day with next-day return probability, but lower-activity users may come back to the system after several weeks. The response retention bias assumes that better recommendations also increase the user's returning probability since users are more satisfactory to the system. We tune the global bias to fit the geometric distribution of the data and restrict the return day signal in  $\mathcal{Y}_t^{(R)} \in \{1, \dots, D\}$  after each session. Here, we set  $D = 10$  in our simulation because the percentage of requests returned after the tenth day is almost zero.
In order to maintain distributional consistency between the simulator and the real-world environment, we build KuaiSim with the support of log data. The data will be used to pretrain the UIRM and sample the user's initial observation during online interactions as the post-processing module in algorithm 1. In this section, we further describe the example datasets used in building up the simulators.