ResAct: Reinforcing Long-term Engagement in Sequential Recommendation with Residual Actor
Authors: Wanqi Xue
Qingpeng Cai
Ruohan Zhan
Dong Zheng
Peng Jiang
Kun Gai
Bo An
Venue: ICLR 2023 poster (2023)
Keywords: Sequential Recommendation
Long-term Engagement
Reinforcement Learning

Abstract: Long-term engagement is preferred over immediate engagement in sequential recommendation as it directly affects product operational metrics such as daily active users (DAUs) and dwell time. Meanwhile, reinforcement learning (RL) is widely regarded as a promising framework for optimizing long-term engagement in sequential recommendation. However, due to expensive online interactions, it is very difficult for RL algorithms to perform state-action value estimation, exploration and feature extraction when optimizing long-term engagement. In this paper, we propose ResAct which seeks a policy that is close to, but better than, the online-serving policy. In this way, we can collect sufficient data near the learned policy so that state-action values can be properly estimated, and there is no need to perform online exploration. ResAct optimizes the policy by first reconstructing the online behaviors and then improving it via a Residual Actor. To extract long-term information, ResAct utilizes two information-theoretical regularizers to confirm the expressiveness and conciseness of features. We conduct experiments on a benchmark dataset and a large-scale industrial dataset which consists of tens of millions of recommendation requests. Experimental results show that our method significantly outperforms the state-of-the-art baselines in various long-term engagement optimization tasks.

Published as a conference paper at ICLR 2023
RESACT: REINFORCING LONG-TERM ENGAGEMENT IN SEQUENTIAL RECOMMENDATION WITH RESIDUAL ACTOR
Wanqi Xue-.*, Qingpeng Cai', Ruohan Zhan Dong Zheng", Peng Jiang", Kun Gai', Bo An-
1 Nanyang Technological University. Kuaishou Technology, 'Hong Kong University of Science and Technology, *Unaffiliated wanqi001 @e @e.ntu. edu.s...gg. (caiqingpeng, zhengdong, mhengdong.jaangpeng kuaishou. rhzhan @ hk, gai.kun @ qq.com, boan @ntu.entuuuu edu.sg
ABSTRACT
Long-term engagement is preferred over immediate engagement in sequential rec- ommendation as it directly affects product operational metrics such as daily active users (DAUs) and dwell time. Meanwhile, reinforcement learning (RL) is widely regarded as a promising framework for optimizing long-term engagement in se- quential recommendation. However, due to expensive online interactions, it is very difficult for RL algorithms to perform state-action value estimation, explo- ration and feature extraction when optimizing long-term engagement. In this pa- per, we propose ResAct which seeks a policy that is close to, but better than, the online-serving policy. In this way, we can collect sufficient data near the learned policy so that state-action values can be properly estimated, and there is no need to perform online interaction. ResAct optimizes the policy by first reconstructing the online behaviors and then improving it via a Residual Actor. To extract long-term information. ResAct utilizes two information-theoreee the expressiveness and conciseness of features. We conduct experiments benchmark dataset and a large-scale industrial dataset which consists of tens of millions of recommendation requests. Experimental results show that our method significantly outperforms the state-of-the-art baselines in various long-term en- gagement optimization tasks.
regularizers confirm
a
INTRODUCTION
In recent years, sequential recommendation has achieved remarkable success in various fields such recommendation (Wu et al.) 2017: Zheng al. 2018 de Souza Pereira Moreira et al.) 2021]. digital entertainment (Donkers et al.
2017 Huang et al. 2018 Pereira et al. 2019). E- commerce (Chen et al. 2018 Tang & Wang 2018) and social media (Zhao et al. 2020b] Rappaz cet al. 2021). Real-life products, such as Tiktok and Kuaishou, have influenced the daily lives of billions of people with the support of sequential recommender systems. Different from traditional recommender systems which assume that the number of recommended items is fixed, sequen- tial recommender system keeps recommending items to a user until the user quits the current ser- vice/session (Wang et al. 2019 Hidasi et al. 2016). In sequential recommendation, as depicted in Figure | users have the option to browse endless items one session and can restart new session after they quit the old one (Zhao al.) 2020c). To this end, an ideal sequential recommender system would be expected to achieve i) low return time between sessions, i.e., high frequency user visits; and ii) large session length two characteristics, i.c., return time and session length, as long-term engagement, in contrast to im- mediate engagement which is conventionally measured by click-througe rates (Hidasi et al. 2016). Long-term engagement is preferred over immediate engagement in sequential recommendation as it directly affects product operational metrics such daily active users (DAUs) and dwell time.
that more items can be browsed in each session. We denote these
"The work was done during internship Kuaishou Technology.

---

Published as a conference paper at ICLR 2023
Despite great importance, unfortunately, how to effectively improve long-term engagement in quential recommendation remains largely uninvestigated. Relating the changes in long-term user engagement to single recommendation is tough problem (Wang al. 2022). Existing works on sequential recommendation have typically focused on estimating the probability of immediate engagement with various neural network architectures (Hidasi et al., 2016: [ang & >Wang] 2018)- However, they neglect to explicitly improve user stickiness such as increasing the frequency of visits or extending the average session length. There have been some recent efforts optimize long-term engagement in sequential recommendation. However, they are usually based on strong assumptions such as recommendation diversity will increase long-term engagement (Teo et al. 2016 Zou all] 2019). In fact, the relationship between recommendation diversity and long-term engagement is largely empirical, and how to measure diversity properly is also unclear (Zhao et al. 2020c).
Recently, reinforcement learning has achieved impressive advances in various sequential decision- making tasks, such as games (Silver et al. 2017, Schrittwieser et al. 2020), autonomous driving (Kiran et al.] [2021) and robotics (Levine et al., 2016). Reinforcement learning in
User
Besponse
Request 1
Session
general focuses on learning policies which maximize cu- mulative reward from long-term perspective (Sutton &
Request 2
Recommender
Request N
Barto 2018). To this end, it offers us a promising frame- work to optimize long- fong-term engagement in sequential rec- ommendation (Chen et al. 2019). We can formulate the recommender system as an agent, with users the envi- ronment, and assign rewards to the recommender system based on users' response, for example, the return time between two sessions. However, back to reality, there are significant challenges. First, the evolvement of user stickiness lasts for long period, usually days months, which makes the evaluation of state-action value difficult. Second, probing for rewards previously unexplored areas, i.c., exploration, requires live experiments and may hurt user experience. Third, rewards of long-term engagement only occur at the beginning or end of a session and are therefore sparse com- pared to immediate user responses. As result, representations of states may not contain sufficient information about long-term engagement.
System
Figure 1: Sequential recommendation.
To mitigate the aforementioned challenges, propose to learn recommendation policy that is close to, but better than, the online-serving policy. In this way, i) we can collect sufficient data near the learned policy so that state-action values can be properly estimated; and ii) there is no need to perform online interaction. However, directly learning such a policy is quite difficult since we need to perform optimization in the entire policy space. Instead, our method, ResAct, achieves by first reconstructing the online behaviors of previous recommendation models, and then improving upon the predictions via Residual Actor. The original optimization problem is decomposed into two sub-tasks which are easier to solve. Furthermore, to learn better representations, two information- theoretical regularizers are designed to confirm the expressiveness and conciseness of features. We conduct experiments on benchmark dataset and real-world dataset consisting of tens of millions of recommendation requests. The results show that ResAct significantly outperforms previous state- of-the-an art methods in various long term engagement optimization tasks.
2 PROBLEM STATEMENT
In sequential recommendation, users interact with the recommender system on session basis. A session starts when a user opens the App and ends when he/she leaves. As in Figure [ when a user starts a session, the recommendation agent begins to feed items to the user, one for each rec- ommendation request, until the session ends. For each request, the user can choose to consume the recommended item or quit the current session. A user may start new session after he/she exits the old one, and can consume an arbitrary number of items within a session. An ideal recommender system with a goal for long-term engagement would be expected to minimize the average return time between sessions while maximizing the average number of items consumed in session. Formally, we describe the sequential recommendation problem as a Markov Decision Process (MDP) which is defined by a tuple (S.A.7. P, R, 7). S S., x S is the continuous state space. S indicates the state of a user. Considering the bession-request structure in sequential recommendation, we de- compose S into two disjoint sub-spaces, i.e., S, and S,, which is used to represent session-level
2

---

Published as a conference paper at ICLR 2023
(high-level) features and request-level (low-level) features, respectively. is the continuous action space (Chandak et al. 2019; Zhao et al. 2020a), where a E A is a vector representing recom- IK is the transition function, where p( Q,) defines the state mended item. P : S A transition probability from the current state & to the next state St+1 after recommending an item dt- : S x A IR is the reward function, where r(st, at) is the immediate reward by recommending dz at state 8t. The reward function should be related to return time and/or session length; is the discount factor for future rewards.
Given a policy :[als): S x A - IR, we define a state-action value function Q"(ss, a) which outputs the expected cumulative reward (return) of taking an action at state and thereafter following ..
(" a+) (
a.) +
Σ (((--)). ae) e'=++1
The optimization objective is to seek a policy T(als) such that the return obtained by the recom- mendation agents is maximized, i.e., max; (1T) = Bssndy (110))))))) [Q" (s: 0.)]. Here di (-) denotes the state visitation frequency at step / under the policy æ.
3 REINFORCING LONG-TERM ENGAGEMENT WITH RESIDUAL ACTOR
To improve long-term engagement, we propose to leam a recommendation policy which is broadly consistent to, but better than, the online-serviin policy In this way, i) we have access to sufficient data near the learned policy so that state-action values can be properly estimated because the notori- ous extrapolation error is minimized (Fujimoto et al., (209);; and ii) the potential of harming the user experience is reduced as we can easily control the divergence between the learned new policy and the deployed policy (the online-serving policy) and there is no need to perform online interaction. Despite the advantages, directly learning such a policy is rather difficult because need to perform optimiza- tion throughout the entire huge policy space. Instead, we propose to achieve by first reconstructing the online- serving policy and then improving it. By doing so, the original optimization problem is decomposed into two sub-tasks which are more manageable.
Reconstruction
Selection
Don
A
Specifically, let *(ls) denote the policy we want learn; we decompose it into à = aon + A(s, aon) where Gon is sampled from the servnng policy Tons i.e. Gon ~ Ton (a)s), and A(s, onn) is the residual which is determined by deterministic actor. We expect that adding the residual will lead to higher expected return, i.e., [()) 2 I [TTo). As in Figure 2 our algorithm, ResAct, works in three phases:
Prediction
Figure 2: Workflow ResAct.
i) Reconstruction:
ResAct first reconstructs the naline-serving policy, i.e. oon (a|s) s Ton (a)s). ,. by supervised learning. Then ResAct samples n actions from the reconstructed policy, i.e., {ātm non (als)) [a|s))] as estimators of doni
ii) Prediction: For each estimator a ResAct predicts the residual and applies it to a i.e., ā' = ã' + A(s, a. We need to learn the residual actor to predict A(s, don) such that is better than aan in general;
iii) Selection: ResAct selects the best action from the {āt (a as the final output, i.e., arg max; (s, for à E {āt)
In sequential recommendation, state representations may not contain sufficient information about long-term engagement. To address this, we design two prove the expressiveness and conciseness of the extracted state features. The regularizers are maxi- mizing mutual information between state features and long-term engagement while minimizing the entropy of the state features in order to filter out redundant information. The overview of ResAct is depicted in Figure Jand we elaborate the details in the subsequent subsections. A formal description for ResAct algorithm is shown in Appendix A
regularizers to im-
'The online-serving policy is a historical policy or mixture of policies which generate logged data to approximate the MDP sequential recommendation.
3

---

Published as a conference paper at ICLR 2023
a
Data Flow
Gradient Flow
Low-level State Encoder
High-level Encoder
CVAE-Encoder
Residual Actor
Looo
Policy Gradient
L
Q a)
on
CVAE-Decoder
Residual Sub-actor
1|
State-action Networks
1. Rec
Figure 3: Schematics of our approach. The CVAE-Encoder generates an action embedding distri- bution, from which a latent vector is sampled for the "VAE-Decoder reconstruct the action. The reconstructed action Ton- together with state features extracted by the high-level and low-level state encoders, are fed the residual actor predict the residual A. After adding the residual, the action and the state are sent to the state-action value networks, from which policy gradient can be gener- ated. The framework can be trained in an end-to-end manner.
3.1 RECONSTRUCTING ONLINE BEHAVIORS
To reconstruct behaviors of the online-serving policy, we should learn a mapping non (als) from states to action distributions such that 7on (als) Ton (als) where Ton (als) is the online-serving policy. A naive approach is to use a model Dalss; ; 02) with parameters 02 to approximate Ton (a|s) and optimize Od by minimizing
E Es Tom (alss [((aass; 0.) - aon)
(2)
However, such deterministic action generation only allows for an instance of action and will cause huge deviation if the only estimator is not precise. To mitigate this, we propose to encode aon into a latent distribution conditioned on 8, and decode samples from the latent space to get estimators of generate multiple action estimators by sampling from the latent distribu- tion. The key idea is inspired by conditional variational auto-encoder (CVAE) (Kingma & Welling 2014]. We define the latent distribution C(s, aon) multivariate Gaussian whose parameters, i.e., mean and variance, are determined by an encoder E. Gon 0a) with parameters 6.. Then for each latent vector
Gon- By doing so,
decoder D(a 8s c; 92) with parameters 0a to map back to an action. To improve generalization ability, we apply a KI. regularizer which controls the devia- tion between C(s, aon) and its prior which is chosen as the multivariate normal distribution NO, 1). Formally, we can optimize Og and 0g by minimizing the following loss:
C(s, don),
Rec 02,00 F...am. [(aaass c; 0a) aan)2+ [L(C(ss Gon; (o, 1))]-
(3)
where Gon Tom (als) and C(s, don's O. When performing behavior reconstruction for an unknown state 8, we not know its aon and therefore cannot build C(s, don; 0z). As a mitigation, sample n latent vectors from the prior of C(s, don), i.c., {c² ~ NO 1) Then for each di, we can generate an estimator of don by using the decoder ã = ((als, c.;
3.2 LEARNING TO PREDICT THE OPTIMAL RESIDUAL
By learning the CVAE which consists of E(-1s, aon; On) and D(ass, 0a), we can easily reconstruct the online-serving policy and sample multiple estimators of aon by { = D(als, c 02), of ~ (0, 1- For each a a we should predict the residual A(s, azm) such that ã a + A(s a,n) is better than ãt. We use model f(A ss, a; 0y) with parameters 0, to approximate the resid- ual function A(s, a). Particularly, the residual actor f(Als, 0; consists of a state encoder and sub-actor, which are for extracting features from user state and predicting the residual based
²C(8, oon : 4e) is parameterized by O. hecause it is multivariate Gaussian whose mean and variance are the output of the encoder F[] aon; 0c).
4

---

Published as a conference paper at ICLR 2023
on the extracted features, respectively. Considering the bi-level quential recommendation, we design a hierarchical state encoder consisting of a high-level en- coder (sp 0%) and a low-level encoder fi(s;; 0t) for extracting features from session-level (high- level) state 8n and request-level (low-level) state 8t, respectively. To conclude, the residual actor f(A 8, ff) (fr, ft, fa] works as follows:
structure in
2/1 fn (sw: 0n), 2 fi(s;; 0)); = Concat(zn. 2);; A fa(z, 0a).
(4)
Where 2, and 2 are the extracted high-level and low-level features, respectively; 2 is the concate- nation of 2n and Rt, and fa(z, a; 0a) parameterized by 0₂ is the sub-actor. Here, 0 {0,, O₁ 0a)-
Given a state s and a sampled latent vector e ~ (0, 1), ResAct generates an action with a deter- ministic policy (a|ss, c) = ((aonlss, 0a) + f(A]s, aon 0,)) We want optimize the parameters {0s,0z, of i F[ass, c) so that the expected cumulative reward J (#) is maximized. Based on the De- terministic Policy Gradient (DPG) theorem (Silver al., 2014; [illicrap al. 2016), we derive the following performance gradients (a detailed derivation can be found in Appendix B
Vo, Es,c [V. (s, Ve, [(A|s, az 0f)la=((asscoo)))
(5)
Ion Be,c VoQ (8, () D(als, 0a)]
(6)
Here #[als, c) D(āon| |s. 0a) + f( Ton; 0y). p(.) is the probability function of a random variable, Q*(ss, a) the state-action value function for f.
To learn the state-action value function, referred to as critic. (s, a) in Eq. 5 and Eq. (6), we adopt Clipped Double Q-learning (Fujimoto all 2018) with two models Q1 (s, a; 0g and Q2(s, a: p to approximate it. For transitions (8t, az, ft, 8t+1 from logged data, we optimize onr and Ogn to minimize the following Temporal Difference (TD) loss:
LTD 1:Ba;
R(seaarrttte [(), on (e: 02)) v) .j (1,2);
== rt - min [e] A' (a4411844)) (, (2(5441, 4' (ac+1l
Where Q/ Q2 and A are target models whose parameters soft-updated to match the corre- sponding models (Fujimoto et al. (018)..
According to the DPG theorem, gain value improvement in I(t):
update the parameters 0y in the direction of Ve, (+t) to
of - OF J(F), = (On, 01, 0a)-
(8)
For 0a since it also needs to minimize 0. thus the updating direction is
0a + 02 + Vo, J(+) - Vo.
Rec
(9)
Based on (als, c), theoretically. we can obtain the policy (a(s) by marginalizing out the la- tent vector e: ((as)) = (ac)a e)de. This integral can be approximated 5
as f(als) s where fcr NO0, 1)}{-- However, given that we already have critic Q1 fg). we can alternatively use the critic to select the final output:
#(als) = talls,c");
(10)
c arg max (s, ((lls.)) CA NN,,
3.3 FACILITATING FEATURE EXTRACTION WITH INFORMATION-TIDORETCCAL REGULARIZERS
Good state representations always ease the learning of models NNielsnn 2015). Considering that session-level states 8). E S₁ contain rich information about long-term engagement, we design two
regularizers to facilitate the feature extraction. Generally, we expect the leamned features have Expressiveness and Conciseness. To leam features with the desired properties, we propose to encode session-level state 8μ into a stochastic embedding space instead of deterministic vector. Specifically, 8s is encoded into multivariate Gaussian distribution
5

---

Published as a conference paper at ICLR 2023
N(us [h) whose parameters 14, and on are predicted by the high-level encoder (s 0n). For- mally, (Hh, Tn) = (sn; 0h) and 2n ~ NNuu, σ4) 2 is the representation for session-level state 8,. Next, introduce how to achieve expressiveness and conciseness in A/e-
Expressiveness. We expect the extracted features to contain as much information as possible about long-term engagement rewards, suggesting intuitive approach to maximize the mutual information between 2n and r(s,a) However, estimating and maximizing mutual information
Le, (Znir) ff pea, (zn) (=))p(((=)) log p(r)
d is practically intractable. Instead, derive lower bound for the mutual information objective based on variational inference (Alemi et al. 2017)
// po, (zx)p(r(z))))) o(r|=x; 0z)
Ia, (Zhi r) 2
dæ
(r)
(11)
/ʃ pea (EN))(1))))) orr|zn: dzadr H(r),
where o(r|zn: Po) is a variational neural estimator of p(r(2)) with parameters 6. H(r) = - pr) log p(r)dr is the entropy of reward distribution. Since (r) only depends user sponses and stays fixed for the given environment, we can turn to maximize a lower bound of To, (zair) which leads the following expressiveness loss (the derivation is in Appendix C
Esp L 0, Be,sn (sulaa) [H(p(r)s)lo(((za) 0,))]
(12)
where is state, 8h is session-level state, po, (h) ((n,, and H(-II-) denotes the cross entropy between two distributions. By minimizing 00, we confirm expressiveness of Z/.
Conciseness. If maximizing r)( is the only objective, we could always ensure a maximally informative representation by taking the identity encoding of session-level state (z, 5n2) (Alemi et al. 2017); however, such an encoding is not useful. Thus, apart from expressiveness, we want 24 to be concise enough to filter out redundant information from 84. To achieve this goal, also want pe, (sk (ehl=s)dssxdd,,, pe,, (Rx) dsyddz, such that 2h is the minimal sufficient statistic of 8h for inferring r. Computing the marginal distribution of 2h. po, (Zn), is usu- ally intractable. So we introduce m m((za) variational approximation to pe, (zn), which is conven- tionally chosen as the multivariate normal distribution (0, 1). Since K [Po,, (2h) (mm))))) 2 O, can easily have the following upper bound:
to minimize La,, (ZA5 sh) fʃ pee (Za(sA) log
- /f p(s))po, (zn(s)) log Pa. (Zn (8h) dzp
Ies (zN: sn)
(13)
m(zn)
Minimizing this upper bound leads to the following conciseness loss:
Con = =
-DD( p(on)) If ppo. (Enl9T) log po, (2n18H) dzn dsns m(=x)
(14)
E, [KL (2 (2alss))lm(22)))
By minimizing L we achieve conciseness in Rh.
4 EXPERIMENT
We conduct experiments on a synthetic dataset MovieLensL-Im and real-world dataset RecL-25m demonstrate the effectiveness of ResAct. We are particularly interested in Whether ResAct able to achieve consistent improvements over previous state-of-the-ar methods? yes, why?
4.1 EXPERIMENTAL SETTINGS
Table I: Statistics of RecL-25m
Datasets. As there is no public dataset explicitly containing signals about long-term engagement, we synthesize a dataset named MovieLensL-Im based on MovieLens-Img (a popular benchmark for evaluating recommen-
Users
Sessions
Requests
6,126,583
25,921,753
Avg return time thi Avg session length
Aug of sessions
62.3277
75%
11.2794
4.8792
85
25%
4.3264
2.1358
30
dation algorithms) and collected large-scale industrial dataset Recl-255m from real-life streaming

---

Published as a conference paper at ICLR 2023
Table 2: Performance comparison on MovieLensL--Im. The "+" in-
Table 3: Performance comparison on Recl.-25m in various tasks. The "+" indicates 95%
confidence intervals.
dicates 95% confidence intervals.
Return Time
Session Length
Both
Return
DDPG TD3 TD3_BC
0.6375 +0.0059
0.3290
+0.0056
0.5908
+0.0092
DDPG
1,7429 20.054
0,6756 10.0133
0.4015
10.0073
0,5498
10.0103
TD3
1.7363 =0.056
0.6436 10.0059
0.3671
10.0037
0.5563
10.0050
TD3 BC
1.7135 =0.0541
BCQ
0.6837 +0.0061
0.3836
+0.0033
0.5915 +0.0049
BCQ
1.7898 =0.0320
IQL
0.6296 1:0.0944
0.3430
1.00057
0.5579
1000066
IQL
1.7360 =0.0546
TI.
0.6404 +-0.0058
0.3186
0.00032
0.5345
+-000488
IL
1,7485 =0.0310
CVAE
0.6410 110.0558
0,3178
10.0031
0,5346
10.0047
IL. CVAE
1.7344 =0.0316
ResAct (Ours)
0.7980 10.0067 0.5433 0.00045
0.6675 10.0053
ResAct (Ours)
18123 =0.0319
platform of short-form videos. MovieLensL--Im is constructed by assuming that long-term engage- ment is proportional to the movie ratings (5-star scale) in MovieLens-Ims RecL--25m is collected by tracking the behaviors of 99,899 users (randomly selected from the platform) for months and recording their long-term engagement indicators, i.e., return time and session length [ The statistics of RecL-25mm are provided in Table I where 25% and 75% denote the corresponding percentile. We did not count the average return time because there are users appearing only once whose return time may go infinity. The state of user contains information about gender, age, and historical inter- actions such as like rate and forward rate. The item to recommend is determined by comparing the inner product of action and the embedding of videos (Zhao al. (0220)). Rewards are designed to measure the relative influence of an item on long-term engagement (details are in Appendix D)
Evaluation Metric and Baselines. We adopt Normalised Capped Importance Sampling (NCIS) (Swaminathan & Joachims, (013). a standard offline evaluation method (Gilotte et al. 2018 Farajtabar al., al.] 2018). to assess the performance of different policies. Given that Tg is the behavior policy, I is the policy to assess, we evaluate the value by
Einanree Pr,ag (8, (isarree Pring (8,a)
P = min (a Cato)(a)
JACIS (T)
Σ SET
c
[
)x)(o)(a))
(15)
Here T is the testing set with usage trajectories, dn()) denotes a multivariate Gaussian distribution of which mean is given by T(s), is a clipping constant to stabilize the evaluation. We compare our method with various baselines, including classic reinforcement learning methods (DDPG, TD3), reinforcement learning with offline training (TD3 BC, BCQ, IQL), and imitation learning methods (IL., II. CVAE). Detailed introduction about the baselines are in Appendix E Our method empha- sises on the learning and execution paradigm, and is therefore orthogonal to those approaches which
focus on designing neural network architectures, e.g., GRU4Rec (Hidasi et al. 2016
4.2 OVERALL PERFORMANCE
MovieLensL-Im. We first evaluate our method on bench- mark dataset MovieLensL-Im which contains 1,000,209 anonymous ratings of approximately 3,900 movies made by 6,040 MovieLens users. We sample the data of 5000 users as the training set, and use the data of the remaining users as the test set (with 50 users as the validation set). As in Table 2 our method, ResAct, outperforms all the baselines, indi- cating its effectiveness. We also provide the learning curve in Figure 4 It can be found that ResAct learns faster and more stable than the baselines on MovieLensL-Inn
Retur
1.7
1K
2X
Figure 4: Learning curves of RL-
based methods on ens-e-en
RecL-25m. We test the performance of ResAct on RecL-25m in three modes: i) Return Time mode, where the reward signal r(6) is calculated by Eq. 21 ii) Session Length mode, where the reward signal r(n) is calculated by Eq. [2] and iii) Both, where reward signal is generated by convex combination of r()) and r(n) with weights of 0.7 and 0.3 respectively. The weights is determined by real-world demands on the operational metrics. We also perform sensitivity analysis on the reward weights in Appendix G Among the 99,899 users, we randomly selected 80% of the users
Data samples and codes can be found in https://wwwe wttp:://wwww.ddrpoxx.ooooo/ wttp://////...ooooooooooo///m//m
AADE
7

---

Published as a conference paper at ICLR 2023
the training set, of which 500 users were reserved for validation. The remaining 20% users constitute the test set. As shown in 3 our method significantly outperforms the baselines in all the settings. The classic reinforcement learning algorithms, c.g., DDPG and TD3, perform poorly in the tasks, which indicates that directly predicting an action is difficult. The decomposition of actions effectively facilitates the learning process. Another finding is that the offline reinforcement learning algorithms, c.g., IQL, also perform poorly, even though they are specifically designed to learn from logged data. By comparing with imitation learning, we find that the residual actor has successfully found policy to improve an action, because behavior reconstruction cannot achieve good performance alone. To compare the learning process, we provide learning curves for those RL- based algorithms. Returns are calculated on the validation set with approximately 30,000 sessions. As in F the performance of ResAct increases faster and more stable than the other methods, suggesting that it is easier and more efficient to predict the residual than to predict an action directly.
RasAct (ours)
IQL
TD3 BC
TDB
DDPG
Return Time
Session Length
Both
Test Betumm
0.7
0.
0.6
0.6
0.3
Text
0.5
0.2
0
5K
1.KK
15K Steps
20K
25K
30K
5K
1.0K
15K
2.0K
25K
30K
5K
1.0K
35K Steps
20K
25K
30K
Figure 5: Learning curves of RL-based methods on RecL-25m, averaged over 5 runs.
Session Length
Both
0.5
Recur
0.7
0.4
ResAct we CVAE
0.5
15K Steps
30K
10K
20K
25K
15K
30K
Steps
Steps
Figure 6: Learning curves for ResAct with CVAE and a deterministic reconstructor (w/o CVAE).
4.3 ANALYSES AND ABLATIONS
How does ResAct work? To understand the working process of ResAct, we plot t-SNE (Van der Maaten & Hinton| 2008) embedding of actions generated in the execution phase of ResAct. As in Figure 7 the recon- structed actions, denoted by the red dots, are located around the initial action (the red star), suggesting that ResAct successfully samples sev- eral online-behavior estimators. The blue dots are the t-SNE embedding of the improved actions, which are generated by imposing residuals on the reconstructed actions. The blue star denotes the executed action of ResAct. We can find that the blue dots are near the initial actions but cover wider area then the red dots.
INI BEST RECON IMPRD
*
Figure 7: The t-SNE vi- sualization of actions.
Effect of the CVAE. We design the CVAE for behivie reconstruction because of its ability generate multiple estimators. To explore the effect of the CVAE and whether de- terministic action reconstructor can achieve similar performance, we disable the CVAE in Re- sAct and replace it with feed-forward neural network. The feed-forward neural network is trained by using the loss in Equation 2 Since the feed-forward neural network is deterministic,
ResAct does not need to perform the selection phase as there is only one candidate action. We provide the learning curves of ResAct with and without the CVAE in Figure 6 As we can find, there a significant drop in improvement if we disable the CVAE. We deduce that this because a deterministic behavior reconstruc- tor can only generate estimator, and if the prediction is inaccurate, performance will be severely harmed.
Return Time
Session Length
Both
0.79
0.66
15
no
25
15 25
20
25
0=15
Figure Ablations for the number of online-
behavior estimators.
8

---

Published as a conference paper at ICLR 2023
Number of Online-hehavior Estimators. Knowing that generating only one action estimator might hurt performance, we want to further investigate how the number of estimators will affect the perfor- mance of ResAct. We first train ResAct and then change the number of online-behavior estimators to 5, 10, 15, 20 and 25. As in Figure 8 consistent improvement in performance can be observed across all the three tasks as we increase the number of estimators. The fact suggests that generating more action candidates will benefit the performance, in line with our intuition. We also perform analysis about how the quality of online-behavior estimators could affect the performance in Ap- pendix H Because the sampling of action estimators is independent, the parallelization of ResAct is not difficult implement and we can easily speed up the inference.
cetmmattion-thooettee Regular- izers. To explore the effect of the designed regularizers, we disable L00,00. o and both of them in ResAct, respectively. As shown in Table the removal of any of the regularizers results in significant drop in performance, suggesting that the regularizers facilitate the extraction of features and thus ease the learning process. An interesting finding is that removing both of the regularizers does not necessarily results in worse performance than removing only one. This suggests that we cannot simply expect either expressiveness conciseness of features, but rather the combination of both.
Table 4: Ablations for the information-theoreiinn regulariz- ers. The "+" indicates 95% confidence intervals. Scssion Length
Return Time
Both
ResAct w/o dixp
0.7980 +-000667
0.5433
+0.0045
0.6675 =0.0053
0.6610 +00000
0.3895
+0.0034
0.6074 +0.0052
w/o Lay
0.6944 +-00061
0.4542 +0,0038
0.6041 +0.0051
w/n Dog lig.
0.7368 +0.0064 0.3854 +0.0033
0.6348 +0.0049
RELATED WORK
Sequential Recommendation Sequential recommendation has been used model real-world rec- ommendation problems where the browse length is not fixed (Zhao et al., 2020c). Many existing works focused on encoding user previous records with various neural network architectures. For example, GRU4Rec (Hidasi et al., (016) utilizes Gated Recurrent Unit to exploit users' interac- tion histories; BERT4Rec (Sun al. al.1201)) employs deep bidirectional self-attention structure to learn sequential patterns. However, these works focus on optimizing immediate engagement like click-through rates. FeedRec (Zou et al., 2019) was proposed to improve long-term engagement in sequential recommendation. However, it is based on strong assumption that recommendation diversity will lead to improvement in user stickiness.
Reinforcement Learning in Recommender Systems. Reinforcement learning (RI.) has attracted much attention from the recommender system research community for its ability to capture po- tential future rewards (Zheng et al.) 2018 Zhao et al. [2118 Zou et al. 2019 Zhao et al. 2020b Chen et al. 202T: Cai et al. 2023b a). Shani et al. (2005) first proposed to treat recommendation as a Markov Decision Process (MDP), and designed model-based RL method for book recom- mendation. Dulac- et al. (2035) brought RL to MDPs with large discrete action spaces and demonstrated the effectiveness on various recommendation tasks with up to one million actions. Chen et al. (2019) scaled a batch RL algorithm, i.e., REINFORCE with off-policy correction to real- world products serving billions of users. Despite the success, previous works required RL agents to learn in the entire policy space. Considering the expensive online interactions and huge state-action spaces, learning the optimal policy in the entire MDP is quite difficult. Our method instead learns a policy near the online-serving policy to achieve local improvement (Kakade & Langford 2002 Achham al. 2017). which is much easier.
CONCLUSION
In this work, we propose ResAct to reinforce long-term engagement in sequential recommendation. ResAct works by first reconstructing behaviors of the online-serving policy, and then improving the reconstructed policy by imposing an action residual. By doing so, ResAct learns a policy which is close to, but better than, the deployed recommendation model. To facilitate the feature extraction, two nnarmation-thcoretiicn
regularizers are designed to make state representations both expressive and concise. We conduct extensive experiments on a benchmark dataset MovieLensL--I //m and a real- world dataset RecL-25m. Experimental results demonstrate the superiority of ResAct over previous state-of-the-ar algorithms in all the tasks.

---

Published as a conference paper at ICLR 2023
Ethics Statement. ResAct is designed to increase long-term user engagement, increasing the time and frequency that people use the product. Therefore, will inevitably cause addiction issues. To mitigate the problem, we may apply some features, e.g.. the age of users, control the strength of personalized recommendation. This may be helpful to avoid addiction to some extent.
Reproducibility Statement. We describe the implementation details of ResAct in Appendix and also provide our source code and data in the supplementary material and external link.
ACKNOWLEDGEMENT
This research is supported by the National Research Foundation, Singapore under its Industry Align- ment Fund- Pre-posiine ree-postiioning (IAF-PP) Funding Initiative. Any opinions, findings and conclusions or recommendations ex expressed in this material are those of the author(s) and do not reflect the views of National Research Foundation, Singapore.
REFERENCES
Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In International Conference Machine Learning, pp. 22-31. PMLR, 2017.
Alexander Alemi, Ian Fischer, Joshua Dillon, and Kevin Murphy. Deep variational information
bottleneck. ICLR, 2017.
Qingpeng Cai, Shuchang Liu, Xueliang Wang, Tianyou Zuo, Wentao Xie, Bin Yang, Dong Zheng, Peng Jiang, and Kun Gai. Reinforcing user retention in a billion scale short video recommender system. arXiv preprint arXiv:2302.01724. 2023a.
Qingpeng Cai, Zhenghai Xue, Chi Zhang, Wanqi Xue, Shuchang Liu, Ruohan Zhan, Xueliang Wang, Tianyou Zuo, Wentao Xie, Dong Zheng, et al. Two-stage constrained actor-critic fo short video recommendation. arXiv preprint arXivv2302.0168/ 2023b.
Yash Chandak, Georgios Theocharous, James Kostas, Scott Jordan, and Philip Thomas. Learn- ing action representations for reinforcement learning. In International Conference on Machine Learning, pp. 941-950. PMLR, 2019.
Minmin Chen, Alex Beutel, Paul Covington, Sagar Jain, Francois Belletti, and Ed Chi. Top- off-policy correction for a reinforce recommender system. In Proceedings of the 12th ACM International Conference on Web Search and Data Mining, pp. 456-464, 2019.
Minmin Chen, Yuyan Wang, Can Xu, Ya Le, Mohit Sharma, Lee Richardson, Su-Lin Wu, and Ed Chi. Values of user exploration in recommender systems. In Proceedings of the 15th ACM Conference on Recommender Systems, pp. 85-95, 2021.
Xu Chen, Hongteng Xu, Yongfeng Zhang, Jiaxi Tang, Yixin Cao, Zheng Qin, and Hongyuan Zha. Sequential recommendation with user memory networks. In Proceedings of the 1 11th ACM Inter- national Conference Web Search and Data Mining, pp. 108-116, 2018
Gabriel de Souza Pereira Moreira, Sara Rabhi, Jeong Min Lee, Ronay Ak, and Even Oldridge. Trans- formers4Rec: Bridging the gap between NLP and sequential/session based recommendation. In Proceedings of the 15th ACM Conference on Recommender Systems, pp. 143-153, 2021.
Tim Donkers, Benedikt Loepp, and Jürgen Ziegler. Sequential user-based recurrent neural network recommendations. Proceedings of the 11th ACM Conference on Recommender Systems, pp- 152-1600, 2017.
Gabriel Dulac-Arold, Richard Evans, Hado Hasselt, Peter Sunehag, Timothy Lillicrap. Jonathan Hunt, Timothy Mann, Theophane Weber, Thomas Degris, and Ben Coppin. Deep rein- forcement learning in large discrete action spaces. arXiv preprint 112.::151..07679, 2015.
Mehrdad Farajtabar, Yinlam Chow, and Mohammad Ghavamzadeh. More robust doubly robust off-policy evaluation. In ICML,, pp. 1447-1456, 2018.
10

---

Published as a conference paper at ICLR 2023
Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. NeurIPS, 34, 2021.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In ICML, pp. 1587-1596, 2018.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. ICML, pp. 2052-20622 2019.
Alexandre Gilotte, Clément Calauzènes, Thomas Nedelec, Alexandre Abraham, and Simon Dollé. Offline a/b testing for recommender systems. In Proceedings of the 11th ACM International Conference on Web Search and Data Mining, pp. 198-206, 2018.
Balázs Hidasi, Alexandros Karatzoglou, Linas Baltrunas, and Domonkos Tikk. Session-based rec- ommendations with recurrent neural networks. In ICLR, 2016.
Jin Huang, Wayne Xin Zhao, Hongjian Dou, Ji-Rong Wen, and Edward Y Chang. Improving quential recommendation with knowledge-enhanced memory networks. The 41st International ACM SIGIR Conference Research & Development in Information Retrieval, pp. 505-514, 2018.
Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In International Conference on Machine Learning, pp. 267-274. PMLR, 2002.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1:Xiv:1112.69000 2014.
Diederik Kingma and Max Welling. Auto-encoding variational bayes. In JCML, 2014.
B Ravi Kiran, Ibrahim Sobh, Victor Talpaert, Patrick Mannion, Ahmad A Al Sallab, Senthil Yoga- mani, and Patrick Pérez. Deep reinforcement learning for autonomous driving: A survey. IEEE Transactions Intelligent Transportation Systems, 2021.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with in-sample Q-Learning, In ICLR, 2022.
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuo- motor policies. Journal of Machine Learning Research, 17(1):1334-15 131113333--3,, 2016
Timothy Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Frez, Yuval Tassa, David Silver, and Daan Wierstra, Continuous control with deep reinforcement learning. ICLR, 2016.
Michael A Niclsen. Neural Networks and Deep Learning, volume 25. Determination Press San Francisco, CA, USA, 20155
Bruno Pereira, Alberto Ueda, Gustavo Penha, Rodrygo LT Santos, and Nivio Ziviani. Online learning rank for sequential music recommendation. In Proceedings of the 13th ACM Confer- ence on Recommender Systems, pp. 237-245, 2019.
Jérémie Rappaz, Julian McAuley, and Karl Aberer. Recommendation on live-streaming platforms: dynamic availability and repeat consumption. In Proceedings of the 15th ACM Conference on Recommender Systems, pp. 390-399, 2021.
Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering Atari, Go, chess and shogi by planning with learned model. Nature, 588(7839):604 609, 2020.
Guy Shani, David Heckerman, Ronen I Brafman, and Craig Boutilier. An MDP- based recommender system. Journal of Machine Learning Research, 6(9). 2005.
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In ICMI., pp. 387-395, 2014.
11

---

Published as a conference paper at ICLR 2023
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of Go without human knowledge. Nature, 550(7676):55-359) 2017.
Fei Sun, Jun Liu, Jian Wu, Changhua Pei, Xiao Lin, Wenwu Ou, and Peng Jiang. BERT4Rec: Sequential recommendation with bidirectional encoder representations from transformer. Pro- ceedings of the 28th ACM International Conference on Information and Knowledge Management, pp. 1441-1450, 2019.
Richard S Sutton and Andrew G Barto. Reinforcement Learning: An Introduction. MIT Press, 2018
Adith Swaminathan and Thorsten Joachims. The self self-normalized estimator for counterfactual learn- ing. NeurIPS, 28, 2015.
Jiaxi Tang and Ke Wang. Personalized top-n sequential recommendation via convolutional sequence embedding. Proceedings of the 11th ACM International Conference on Web Search and Data Mining, pp. 565-573, 2018.
Choon Hui Teo, Houssam Nassif, Daniel Hill, Sriram Srinivasan, Mitchell Goodman, Vijai Mohan, and SVN Vishwanathan. Adaptive, personalized diversity for visual discovery. In Proceedings of the 10th ACM Conference Recommender Systems, pp. 35-38, 2016.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE, Journal of Machine Learning Research, 9(11), 2008.
Shoujin Wang, Liang Hu, Yan Wang, Longbing Cao, Quan Z. Sheng, and Mehmet Orgun. Sequential recommender systems: Challenges, progress and prospects. In IJCAI, pp. 6332-6338, 2019.
Yuyan Wang, Mohit Sharma, Can Xu, Sriraj Badam, Qian Sun, Lee Richardson, Lisa Chung, Ed H Chi, and Minmin Chen. Surrogate for long-term user experience in recommender systems. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 4100-4109, 2022.
Qingyun Wu, Hongning Wang, Liangjie Hong, and Yue Shi. Returning is believing: Optimizing long-term user engagement in recommender systems. In Proceedings of the 2017 ACM on Con- ference on Information and Knowledge Management, pp. 1927-1936, 2017.
Dongyang Zhao, Liang Zhang, Bo Zhang, Lizhou Zheng, Yongjun Bao, and Weipeng Yan. Mahrl: Multi-goals abstraction based deep hierarchical reinforcement learning for recommendations. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 871 1800 2020a.
Xiangyu Zhao, Liang Zhang, Zhuoye Ding, Long Xia, Jiliang Tang, and Dawei Yin. Recommenda- tions with negative feedback via pairwise deep reinforcement learning. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 1040- 1048, 2018
Xiangyu Zhao, Xudong Zheng, Xiwang Yang, Xiaobing Liu, and Jiliang Tang. Jointly learning to recommend and advertise. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 3319-3327, 2020b.
Yifei Zhao, Yu-Hang Zhou, Mingdong Ou, Huan Xu, and Nan Li. Maximizing cumulative user engagement sequential recommendation: An online optimization perspective. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 2784-2792 2020c.
Guanjie Zheng, Fuzheng Zhang, Zihan Zheng, Yang Xiang, Nicholas Jing Yuan, Xing Xie, and Zhenhui Li. DRN: A deep reinforcement learning framework for news recommendation. In WWW, pp. 167-176, 2018.
Lixin Zou, Long Xia, Zhuoye Ding, Jiaxing Song, Weidong Liu, and Dawei Yin. Reinforcement learning to optimize long-term user engagement in recommender systems. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp- 2810-2818, 2019.
12

---

Published as a conference paper at ICLR 2023
A OVERALL ALGORITHM
We provide the learning process of ResAct in Algorithm Particularly, the CVAE trained to reconstruct the online-serving policy, the residual actor is trained for predicting the optimal residual for each reconstructed action, and the critic networks is trained to guide the optimization of the decoder in CVAE and the residual actor. For the target networks (line 2), {0a. tn. 8.. f.. fg. is for the target policy, and , ,,, is for the target critics. The execution process of ResAct is
summarized in Algorithm 2 Only the decoder of the CVAE, the residual actor and one of the crities are used during execution.
Algorithm 1: ResAct-LEARNING
Input: Logged data collected by the online-serving policy D (88, de, Tty 81+1))
Initialize the CVAE: [F(c), o; 02). Daas, c; (,)),, the residual actor:
{fr(z (s;: 0n), fi(2)]s(; 0t), fa (A(A,, a; (a)), the critic networks
{Qi a; ), Q2(s, a; 0 922)) and the variational estimator o(r|zn; 0o)
Set soft-update rate and initialize the target networks /' 0 for 8 E (0s. On 0i far og 0g for k= to K do
Sample batch of transitions (se, des Tz, 8141) from
0e 0g- Ve.Lbu.. 0a (Lg.
Rec
82 in Eq.
Update 0,2 according to Eq.
Update {0n, 0n) according Eq.
Oaz < 0q: Ve., La;
LTD
{1, 2] day
is in Eq.
On + On Ve, 8y,
Eap
Va, Con
10
Oo + 0, Vo. 0.
Ezp
(T.0,.0 is in Eq. 12 and Conon
Eq. 14
11
Update the target networks:
70 7)'0 for 0 (Od, On, Or, Onr Og 440
end
Algorithm 2: ResAct-EXECUTION
Input: State number of estimators Reconstruction
Generate n estimators of don (ā'n df NO 1)
// Prediction
for āon E Ho do Predict the residual A = f(A|s, āon; Oy) as in Eq.
Apply the residual: à don A end // Selection
arg max, Qı a; 0g)) E (
Output: Action a"
13

---

Published as a conference paper at ICLR 2023
B THE DERIVATION OF PERFORMANCE GRADIENTS
We begin by deriving the gradients of ( with respect to the parameters the residual actor.
/| p(c)p (l 4 s.0) a
Vo, ((t)
= |f plc)p (s) oat
(16)
= E VoQ* a) aa Va f( s, a; 0))lo=
The decoder D(a (, 0a) also affects the policy. The gradients of J(#) with respect to 0a) is derived similarly:
/|
Vo. Vo
Vos cldds
|| p(c)p" (s) Dllala daddess
(17)
E., [vQ (s, (s,a)l==taaaa) Vo,
C DERIVING THE EXPRESSIVENESS LOss
We expect the extracted features contain much information as possible about long-term gagement rewards, suggesting an intuitive approach to maximize the mutual information between 2h and r(s, The mutual information Le, (zhir) is defined according
Iea (znir) = /ʃ p r) log
p(zk,r)
dændr
p ((z)) p (r)
(18)
// po, (2m)(((()))) log p(r(zn) dz
(r)
However, estimating and maximizing mutual information is practically intractable. Inspired by vari- ational inference (Alemi et al. (2017), we derive tractable lower bound for the mutual information objective. Considering that K L(p(r (1)))l(())2)))) 2 O, by the definition of KL-divergence, we have p(r(2)) log p(rl 2n )dr 2 p(r 2n. log g(rl (r1 (2x)dr dr where (a)) is arbitrary distribution. Here, we introduce o(r i 0.) as a variational neural estimator with parameters 6, of p(r(2x). Then,
// pe, (2))p(r(zz)))) o[r|2n; 0o) dz (r)
Iea (zhrr) 2
(19)
// pe,, (=A))((())))) log a(rl= 0.)d + H(r)
where (r))= = J p(r log p(r)dr is the entropy of reward distribution. Since (r) only depends on user responses and stays fixed for the given environment, we turn maximize lower bound of Te, (zarr which leads to the following expressiveness loss:
/ʃ
Eap L
pe, (zx))((((z)) log o(r) 0,) 0.)d2
//] p(s)po, log o(r|zn 0o)dsdzzdr
[I] p(s p(s) (znls))) )p( log o AM: dddddd
(20)
Ex (enlsn)
|
p(rls) log o(r km: dr
Es (snlew) [H(p(rls)llo(r(zzs..
where s is state, 8n is session-level state, pe, (zalsh) (Nn,,, and H(--l-) denotes the cross entropy between two distributions. By minimizing T. we confirm expressiveness of z
14

---

Published as a conference paper at ICLR 2023
D DATASETS
D.I MovieLensL-Im
Moviel is synthesized from Moviel.ens-Im which is representative benchmark dataset for sequential recommendation. MovieLens-Im provides 1,000,209 anonymous ratings of approxi- mately 3,900 movies made by 6,040 MovieLens users. Ratings are made a 5-star scale. As MovieLens-Im does not contain any information about long-term user engagement, to generalize it to long-term engagement problem, we make an assumption that a user's long-term engagement is proportional to the movie ratings. Specifically, we assume that recommending a movie for which user rates 2-stars will not affect engagement, a movie with 3-stars, 4-stars and 5-stars will benefit the long-term engagement by 1, 2, and 3, respectively. Recommending movie with 1-star is harmful to engagement and will be given a negative reward, -1. The task in MovieLensL- Im is to maximize cumulative benefits on long-term engagement.
D.2 DESIGNING OF REWARDS IN Recl.-25m
The rewards of long-term engagement in RecL-25m are designed based on the statistics of the dataset. As general guideline, expect rewards to reflect the influence of recommending an item on a user. However, behaviors of users have large variance which makes the influence difficult measure. For example, if we simply make rewards proportional proportional to return time, the recommender system would focus on improving the experience of high activity users, because by doing so can obtain larger rewards. However, in reality, is equally if not more important to facilitate the conversion of low activity user to high activity user, which re- quires us to improve the experience of low activity users. To address this issue, we turn to measuring the relative influence of an item. Concretely, we calculate the average return time Sang and the av- erage session length Wyog for user u, and use these two statistics to quantify rewards. For user u, given a time duration 6" between two sessions, the corresponding reward is calculated by
session length, inversely
r(6") min(6u ang: 870%) 1) c,)p(0,55
(21)
8'h
where 875% is the 75th percentile of the average return time for all users, which is designed to differentiate active and inactive users. Rewards for the session length is calculated similarly as
r(7"*) (L
17 Tävg x x..s) clip(0,.))
(22)
where 17"h is the length of a session in the logged data of user u. Since 8 and can only be calculated at session-level, without loss of generality, we provide rewards at the end of each session, where rewards for return time is assigned to the previous session.
E BASELINES
Our method is compared with various baselines, including classic reinforcement learning meth- ods (DDPG, TD3), offline reinforcement learning algorithms (TD3_BC, BCQ, IQL., and imitation learning methods (IL, IL_CVAE)
DDPG (Lillierap al. 2016): An reinforcement learning algorithm which concurrently learns Q-function and a policy. It uses the Q-function to guide the optimization of the policy.
TD3 (Fujimoto et al. 2018): An off-policy reinforcement learning algorithm which applies clipped double-Q learning, delayed policy updates, and target policy smoothing.
TD3 (Fujimoto & Gu 202T]: : An reinforcement learning designed for offline training.
It adds a behavior cloning (BC) term to the policy update of TD3.
BCQ (Fujimoto et al., 2019): An off-policy algorithm which restricts the action space in order to force the agent towards behaving similar to on-policy.
IQL (Kostrikov et al. 2022): An offline reinforcement learning method which takes a state conditional upper expectile to estimate the value of the best actions in state.
15

---

Published as a conference paper at ICLR 2023
IL: Imitation learning treats the training set expert knowledge and learns mapping between observations and actions under demonstrations of the expert.
IVVC (Kingma & Welling 2014]: Imitation learning method with the policy con- trolled by a conditional variational auto-encoder.
F EXPERIMENTAL DETAILS
Across all methods and experiments, for fair comparison, each network generally uses the same ar- chitecture (3-layers MI.P with 256 neurons each hidden layer) and hyper-parameter We provide the speerpparmmetrs for ResAct in Table 5
All methods are implemented with Py PyTorch.
Table 5: Hyper-parameters of ResAct.
Hyper-parameter
Value
Optimizer
Adam (Kingma Ba. 2014
Actor Learning Rate
x 10
Critic Learning Rate
x 10
Batch Size
4096
Normalized Observations
Ture False
Gradient Clipping
Discount Factor
0.9
Number of Behavior Estimators
20
Weight of I, Eap
10 2
Weight of Con
x 10
Target Update Rate
x 10
Number of Epoch
5
G SENSITIVITY ANALYSIS FOR THE REWARD WEIGHTS
When setting the reward weights in the Both mode, we use some usual empirical values by following the real-world requirements for operational metrics. The reward occurs only at the end of each session, which makes it representative for sequential recommendations. If try other designs, only the value of the reward will change, not the frequency of learning signal. To justify that our algorithm robust to different reward weights, we perform sensitivity analysis for the weights (return time: session length) of rewards in the Both mode. As shown in Table 6 our algorithm consistently outperforms the baselines under different reward weights.
Table 6: Sensitivity Analysis for the Reward Weights the Both Mode.
(0.7: 0.3)
(0.5: 0.5)
(0.3: 0.7)
DDPG
0.5908 +-00092
0.5040 +-0.0733
0.4172 +=0.0059
TD3
0.5498 +.00133
0.4941 +.00086
0.4385 +
TD3 BC
0.5563 #0.0050
0.4978 +0.0033
0.4393 #0.0038
BCQ
0.5915 #0.0049
0.5261
#0.0042
0.4605
+0.0037
IQL
0.5579 #0.0067
0.4812 #0.0054
0.4046 #0.0044
II.
0.5345 +0.0048
0.4727
+0.0041
0.4111
+0.0036
CV
0.5346 +0.0047
0.4726
+0.0041
0.4107
+0.0036
ResAct (Ours)
0.6675 1..0053
0.5948
0.0045
0.5220
0.00039
16

---

Published as a conference paper at ICLR 2023
H QUALITY OF
ESTIMATORS
Despite the selection phase, the quality the unline-behavior estimators, or the action candidates, still significantly affects the performance. On the hand, the action candidate directly constitutes the final action. On the other hand, the sampled action candidates serve as inputs of the residual module and the selection module. It certain that sampling an infinite number of action candidates will cover the best action which is sampled by the CVAE. However, action candidates which are far from behavir policy may be incorrectly selected. The reason that the residual mod- ule and the selection module are unlikely to encounter such out-of-distributie therefore cannot make accurate predictions. The distribution of action candidates should be as close possible to the distribution of online services ensure that the output of the residual and selec- tion modules is reliable. We conduct experiments by uniformly sampling 20 action candidates and adding them to the action candidates reconstructed by the CVAE. As in Table there is significant decrease performance even though we increase the number of action candidates.
(OOD) actions and
Table 7: Performance comparison between ResAct and ResAct with uniformly augmented action candidates. The "4" indicates 95% confidence intervals.
Return Time
Session length
Both
ResAct
0.7980 +0.0067
0.5433 +0.0045
0.6675 +0.0053 0.4839 +0.0054
ResAct + 20 candidates (uniform)
0.5501 +:000688
0.3489 +0.0041
17


Reviews:
[{"review_id": "qdWuVpLFks", "reviewer": "Reviewer_VU33", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces a new framework ResAct for improving long-term user engagement in sequential recommendation. ResAct works in three phases: i) Reconstruction - Training a model to imitate the current recommendation policy and reconstruct recommendation actions; ii) Prediction - Predicting actor residuals that is likely to improve the reconstructed actions in phase one; iii) Selection – selecting the optimal action among all improved actions in phase two. The overall framework is interesting and reasonable. I think it can bring new thinking to the community about how to improve long-term engagement in recommendation and might lead to real-world applications. ", "summary_of_the_review": "Overall, the paper has good motivation and the method is novel and interesting. Experiments demonstrate the effectiveness of the method. I support to accept the paper.", "strength_and_weaknesses": "Strengths:\n1.\tThe motivation of the method is sound. Improving long-term engagement is a difficult problem and directly optimize it is challenging. ResAct decomposes the problem into several manageable sub-tasks which significantly eases the learning process. The idea of first reconstructing and then improving could inspire people to better solve the problem.\n2.\tThe techniques used in this paper is interesting, e.g., the two information-theoretical regularizers. I am impressed that improving expressiveness or conciseness alone will not lead to improvements and only optimizing them together will improve performance.\n3.\tThorough experiments are conducted on various of datasets, including a public benchmark dataset and a real-world industrial dataset. Results are compared with a wide range of baselines. The empirical analysis investigates the working process of the method.\n\nWeaknesses:\n1.\tSome figures can be better clarified, e.g., it is not clear what the dashed line stands for in fig.3\n2.\tAs discussed in section 4.3, the inference speed is one of the main limitations of the method.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "or_mTJEhZ6", "reviewer": "Reviewer_2DQo", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper investigates how to use reinforcement learning (RL) for sequential recommendations. The authors present several innovations for how to address challenges in recommender systems. For one, exploration with actual users can be expensive as they can lead to bad user experience. Also, due to sparse feedback at long time intervals, it is challenging to perform feature selection to remove noisy inputs. To address the first challenge, the authors developed ResAct, which uses supervised learning to learn the production policy and sample additional actions from that policy to select the best one. This eliminates the need to directly explore with users. To address feature selection, they use information theoretic principles to select features that highly predict the reward without retaining redundant information about the state. They demonstrate gains from their approach against several strong baselines using open-source datasets. They also perform ablation experiments to show what design choices lead to their gains.", "summary_of_the_review": "I am inclined to accept this paper for publication. The problem of how to apply reinforcement learning to sequential recommendations is an important and unsolved real-world problem. The authors make good progress on this problem and present multiple innovations that improve performance on this problem. They present ablation studies which identify how their proposed algorithms lead to improvements in their offline experiments. Finally, the code and data samples are promised to be open-sourced after publication. I have only a mild concern that the two areas for innovation (ResAct and information theoretic regularization) seem to be orthogonal improvements, but I still think the community will benefit from reading this manuscript.", "strength_and_weaknesses": "Strengths - \n1. Detailed offline experiments with thorough ablation studies, showing how and why their approach improves over existing work.\n2. How to wield RL for recommender systems is an important yet unsolved problem with large real-world impact. \n3. Experiments are open-source and open-data (pending paper acceptance).\n\nWeaknesses - \n1. The ResAct algorithm and the feature selection regularization are a bit orthogonal to each other, and could potentially be presented in separate publications.\n2. It's preferable to use a temporal holdout rather than a population holdout for evaluating a recommender system. In the real world, we train in the past and deploy on the future.\n3. The experiments on the MovieLens dataset use simulated data inspired by the real-world open source dataset, so may not reflect actual user behavior.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "XpyN1wfn1nz", "reviewer": "Reviewer_EPzq", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors are interested in the long-term engagement of users in recommender systems, which is less studied than immediate engagement. To optimize this long-term engagement in sequential recommendation, they resort reinforcement learning (RL). However, directly applying RL methods to recommender systems can be expensive, and may hurt user experience. To alleviate these issues, the authors introduce ResAct. ResAct is composed of two main modules: (i) the first which reconstructs the online-serving policy, (ii) the second which residually modify the recommendation in order to improve long-term engagement. By doing this, ResAct is able to learn a policy which improves long-term engagement, but stays close to the online-serving policy (limiting the risk of harming user experience). \nMoreover, the authors introduce two information-theoretical regularizers (expressiveness and conciseness) which allows to extract meaningful features from sparse reward signals. ", "summary_of_the_review": "This paper is very pleasant to read. The content of the paper is original and scientifically solid.", "strength_and_weaknesses": "The paper is very pleasant to read even if it combines ideas from different fields (reinforcement learning and Bayesian inference). All the different parts are presented concisely and well illustrated, making the paper accessible for the whole community. Integrating residual improvement in this framework seems like a good idea, which makes a good trade off between exploitation and exploration in recommender systems. The paper is scientifically solid, and the experiments are convincing.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Ch21jK-xFur", "reviewer": "Reviewer_pqv6", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper works on the sequential recommendation for long-term engagement. This topic is interesting and very important in either academia or industrial. \nIn this work, the authors argued that though reinforcement learning was widely accepted as a standard method for long-term engagement optimization, it is not easy to apply RL in practice due to the requirement of online interaction/serving and model update (feature extraction). The authors proposed a new policy that learns the state-action values without the requirement of exploration. The framework first reconstructed the online behaviors based on the offline data, and then improves the model with residual actor. Further, the framework is enhanced with two information theoretical regularizers which improves the expressiveness and conciseness of the extracted features. Comprehensive experiments are conducted on one two datasets, and the performance is improved compared to the existing baselines.", "summary_of_the_review": "Overall, the paper works on an interesting problem and proposed an empirically good framework for solving the long-term engagement recommendation problem. However, the details of the framework, especially the motivation and intuition of the adapted techniques, are lacking. For the current version of the paper, the proposed framework is like a combination of existing technique with strong assumption of the data. Therefore, I would suggest a borderline reject for this paper.", "strength_and_weaknesses": "This paper focuses on an interesting problem in reinforcement learning, where the online exploration is expensive to perform. And the paper is well-organized, the logic and the descriptions of the technique details are clear and easy to follow. The proposed model is kind of interesting, e.g., adding the information related regularizations. The authors also performed comprehensive experiments to support the proposed solution.\n\nHowever, my concern of the paper lies in the following aspects.\nFirst, the novelty of the proposed solution is somehow limited. The proposed solution performs an offline policy learning, and combines the residual learning and regularization. The method is somehow straightforward and there's not much technique parts really related to the long-term action modeling.\n\nSecond, the authors claims that no exploration was needed for the policy. This is based on their assumption of the access to the sufficient data (correct me if I am wrong). This assumption is not valid especially for real-world scenarios. Meanwhile, for reinforcement learning with online exploration, the insufficient data is the key problem to solve. If sufficient data is accessible, why we need a complicated model to do the predication?\n\nBesides, I am not very clear about the motivation/intuition of adoption a residual actor in the framework. Why and how the reward is decomposed into two parts. If we need the residual modeling, does it mean the first reconstruction step is inaccurate, which seems contradict to the assumption that sufficient data is accessible.\n\nFor the experiment part, the authors also compared the model with classical reinforcement learning solutions, are they performing any exploration in the experiment? If not, is the comparison valid?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster