Deep Evidential Reinforcement Learning for Dynamic Recommendations
Authors: Dingrong Wang
Krishna Prasad Neupane
Ervine Zheng
Qi Yu
Venue: Submitted to ICLR 2023 (2023)
Keywords: recommender system
exploration
actor-critic

Abstract: Reinforcement learning (RL) has been applied to build recommender systems (RS) to capture users' evolving preferences and continuously improve the quality of recommendations. In this paper, we propose a novel deep evidential reinforcement learning (DERL) framework that learns a more effective recommendation policy by integrating both the expected reward and evidence-based uncertainty. In particular, DERL conducts evidence-aware exploration to locate items that a user will most likely take interest in the future. Two central components of DERL include a customized recurrent neural network (RNN) and an evidential-actor-critic (EAC) module. The former module is responsible for generating the current state of the environment by aggregating historical information and a sliding window that contains the current user interactions as well as newly recommended items  that may encode future interest. The latter module performs evidence-based exploration by maximizing a uniquely designed evidential Q-value to derive a policy giving preference to items with good predicted ratings while remaining largely unknown to the system (due to lack of evidence). These two components are jointly trained by supervised learning and reinforcement learning. Experiments on multiple real-world dynamic datasets demonstrate the state-of-the-art performance of DERL and its capability to capture long-term user interests.

Under review as a conference paper at ICLR 2023
DEEP EVIDENTIAL REINFORCEMENT LEARNING FOR DYNAMIC RECOMMENDATIONS
Anonymous authors Paper under double-blind review
ABSTRACT
Reinforcement learning (RL) has been applied to build recommender systems (RS) to capture users' evolving preferences and continuously improve the quality recommendations.
this paper, we propose a novel deep evidential reinforce- ment learning (DERL) framework that learns more effective recommendation policy by integrating both the expected reward and evidence-baser based uncertainty. particular, DERL conducts evidence-awee will most likely take interest in the future. Two central components of DERL in- clude customized recurrent neural network (RNN) and evidential-actor-ciin (EAC) module. The former module is responsible for generating the current state of the environment by aggregating historical information and a sliding window that contains the current user interactions as well as newly recommended items that may encode future interest. The latter module performs evidence- based ex- ploration by maximizing a uniquely designed evidential Q-value to derive policy giving preference items with good predicted ratings while remaining largely un- known to the system (due to lack of evidence). These two components are jointly trained by supervised learning and reinforcement learning. Experiments on mul- tiple real-world dynamic datasets demonstrate the state-of-the-arr performance of DERL and its capability to capture long-term user interests.
exploration to locate items that user
INTRODUCTION
Recommender systems (RS) have been widely used for providing personalized recommendations in diverse fields such media, entertainment, and e-commerce by effectively improving user ex- perience (Su Khoshgoftaar, 2009; Sun et al., 2014; Xie et al., 2018). Various methods have been introduced to tackle the recommendation problem. Traditional methods include: collaborative filtering, which captures user preferences using information of similar users (Koren, 2008), content- based, where extra information is used for better latent preference and item representation (Mooney & Roy, 2000), and hybrid, which integrates both collaborative and content based methods for a more effective recommendation (Burke, 2002). Deep learning (DL) has also been increasingly used to build RS due to its ability to model complex and non-linear user-item relationships (Cheng et al., 2016; Guo et al., 2017).
Most RS methods mentioned above consider recommendation as a static process, which fails to con- sider users' evolving preferences. Some efforts have been devoted to capture users' evolving pref- erences by shifting the user latent preference over time (Koren, 2009; Charlin et al., 2015; Gultekin & Paisley, 2014). Similarly, sequential recommendation methods (Kang & McAuley, 2018; Tang & Wang. 2018) attempt to incorporate users' dynamic behavior by leveraging previously interacted items. However, both static and dynamic recommendation methods primarily focus on maximizing the immediate (short-term) reward when making recommendations. As result, they fail to take into account whether these recommended items will lead long-term returns the future, which essential to maintain stable user base for the system in the long run.
Several recent works have adapted reinforcement learning (RL) in the RS (Chen et al., 2019b; Zhao et al., 2017). RL has already gained huge success in diverse fields, such robotics (Kober et al., 2013) and games (Silver et al., 2017). The core idea of RL is to learn an optimal policy to maximize the total expected reward in the long run. RL methods consider recommendation procedure as sequential interactions between users and RL agents to learn the optimal recommendation policies effectively. Although RI. approaches show promising results in RS (Chen et al., 2019b; Zheng al.,

---

Under review as a conference paper at ICLR 2023
0.6
2014
30
Cumulative Rewwrdd
DERL ---n c---redddy
Positive Count
Cenre Count
20
8.4
10
2
Stops
10
Famance
Stana
corredy Ovtler wher
Steps
10
(a)
Positive Count
(b) Genre Count
(c) Cumulative Reward
Figure 1: Different recommendation behavior between an existing RL model and DERL
2018), they primarily rely on standard exploration strategies (e-greedy), which less effective in large item space with sparse reward signals given the limited interactions for most users. Therefore, they may not learn the optimal policy that provides the most informative recommendations to capture effective user preferences and achieve maximum expected reward in the long run.
Figure further illustrates the limitation of ex- isting RL methods using a standard e-greedy strategy for exploration. The RL agent primar- ily focuses on highly-rated items in early steps as shown in Figure la. Most of these items come from the same genre shown in Fig- ure Ib, which is further verified by the detailed recommendation list given by Table 1. Such recommendation behavior leads to a lower cu- mulative reward in the later steps shown in Figure Ic. As Table shows, e-greedy mostly focuses Drama movies based the user's current preference. only captures one novel genre (i.e., Musical, bold in the table) that matches the user's long-term interest. It is clear that more sys- tematic exploration is essential to discover users' long-term interests to maximize the future reward.
Table 1: Examples of recommended movies
Model
Movies
Movie Genre
Sound of Music (1965)
Musical
Casino (1995)
Drama
DERL
Ben-Hur (1959)
Action,Adventure
The Bug' 's Life (1998)
Animation, Comedy
Babe (1995)
Children's ', s,Comedy
Pocahontas (1995)
Musical
Wizard of Oz (1939)
Drama
e-greedy
Christmas Story (1983)
Drama
Frin Brockovich (2000)
Drama
Restoration (1995)
Drama
To address the above key challenges, we propose a novel deep evidential reinforcement learning (DERL) method that utilizes balanced exploitation (with high predicted ratings) and exploration (with evidence-basedd uncertainty) strategy for effective recommendations. We formulate an eviden- tial RL framework that augments the maximum reward RL objective with wrisence-based uncer- tainty maximization. More importantly, the evidence-based uncertainty formulation substantially improves exploration and robustness by acquiring diverse behaviors that are indicative of a user's long-term interest. As shown in Figure 1b, DERL devotes a strong focus more diverse genres (denoted by 'others' the figure) and many these capture the long-term interest from the user as verified by the detailed recommendation list in Table 1. In this case, we refer to user's long-term interest as the genres of movies frequantly watched by the user in the later phase of interactions (i.e., after time step in the given example). These genres are Musical, Action, Adventure, Comedy, and Animation, which match almost perfectly with what DERL recommends in Table 1. DERL seam- lessly integrates
major components: customized RNN and an Evidence-Actor- Critic module. The former primarily focuses on generating the current state of the environment by aggregating the previous state, current items captured by a sliding window, and future recommended items from the RL agent. This provides effective means of dynamic state representation for better future recom- mendations. Meanwhile, the EAC module leverages evidence- based uncertainty to most effectively explore the item space to identify items that potentially align with the user's 's long-term interest. encourages learning the optimal policy by maximizing a novel evidential Q-value to achieve a maximum long-term cumulative reward.
The main contribution of this paper is fourfold:
A novel recommendation model that integrates reinforcemmet learning with evidential learning to
provide uncertainty-aware recommendations.
Evidence-based uncertainty maximization enable stability and effective exploration.
An off-policy formulation to effectively promote the reuse of previously collected data while stabilizing model training, which is important to address data scarcity in recommender systems. Seamless integration of a customized RNN, an actor-critic network, and an evidential network to provide end-to-end integrated training process.
2

---

Under review as a conference paper at ICLR 2023
We conduct extensive experiments over four real-world datasets and compare with state-of of- ff-the-art baselines to demonstrate the effectiveness of the proposed model.
RELATED WORK
Static models. Matrix Factorization (MF) leverages user and item latent factors to infer user pref- erences (Koren et al., 2009; Funk, 2006; Koren, 2008). MF is further extended with Bayesian Per- sonalized Ranking (BPR) (Rendle et al., 2012) and Factorization Machine (FM) (Rendle, 2010). Recently, deep learning-based recommender systems (Cheng et al., 2016; Guo al., 2017) have achieved impressive performance. DeepFM (Guo et al., 2017) integrates traditional FM and deep learning to learn low- and high-order feature interactions. Both wide and deep networks are jointly trained in (Cheng et al., 2016) for better memorization and generalization. In graph-based methods (Berg et al.. 2017), users and items are represented as a bipartite graph and links are predicted to provide recommendations. Similarly, Neural Graph Collaborative Filtering (Wang et al., 2019) ex- plicitly encodes the collaborative signal via high-order connectivities in the user-item bipartite graph via embedding propagation.
Dynamic and sequential models. Dynamic model shifts latent preference over time in- corporate temporal information. TimeSVD++ (Koren, 2009) considers time-specific factors, which uses additive bias to model user and item related temporal changes. Gaussian state-space models have been used to introduce time-evolving factors with one-way Kalman filter (Gultekin & Pais- ley, 2014). To process implicit data, Sahoo al. extended the hidden Markov model (Sahoo al., 2012), and Charlin et al. (Charlin et al., 2015) further augmented it with the Poisson emission. How- ever, these models capture user evolving preference, and they are less aware future interactions and provide recommendations based on fixed strategies. Similarly, sequential models utilize users' historical interactions to capture users' preferences over time. Tang et al. utilized a CNN architec- ture to capture union level and point level contributions (Tang & Wang, 2018). Also. Kang et al. leveraged transformer-baeee user representation to better capture their interest (Kang & McAuley, 2018) and Sun et al. utilized bidirectional encoder for sequential recommendation (Sun al., 2019). Sequential models neglect long-term users' preferences. The proposed DERL model aims to fill this critical gap by performing evidence guided exploration and maximizing total expected reward.
RL-based models. RL-based RS models aim to learn an effective policy to maximize the total expected reward in the long run. The on-policy learning with contextual bandit (Li et al., 2010) and Markov Decision Process (MDP) (Zheng et al., 2018) exploits by interacting with real customers in an online environment. A collaborative contextual bandit algorithm called CoLin (Wu et al., 2016) utilizes graph structure in collaborative manner. On the other hand, off-policy utilizes Monte Carlo (MC) and lefforel-difference (TD) methods to achieve stable and efficient learning with users' history (Farajtabar et al., 2018) 8). Similarly, model-based RI. models user-agent interaction via a generative adversarial network (Bai al., 2019). Pseudo Dyna-Q (Zou et al., 2020) further integrates both direct and indirect RL approaches in a single unified framework without requiring real customer interactions. However, the above methods utilize random exploration strategies, which are less effective at capturing users' long-term preferences. In contrast, our DERL utilizes evidence- based uncertainty to systematically explore the item space to maximize the long-term reward.
PRELIMINARIES
We first introduce the standard RS setup in RI. and provide an overview of evidential theory. Recommendation Formulation with RL. We formulate recommendation tasks in a RL setting, where a RL agent interacts with the environment (users and items) to recommend the next items to a user over time in sequential order to maximize the cumulative reward. We design this problem as the MDP, which includes a sequence of states, actions, and rewards. More formally, a tuple (S, A. p, r) is defined as:
State space (S): A state St = RNN(-Is--1,,, uz) E S is generated by a customized RNN that utilizes previous state s and current user u, embedding which is generated from the concate- nation of M recently interacted items provided by a sliding window (see details later) and an RL-agent.
Action space (A): An action a, E A is represented as a continuous parameter vector that recom- mends top-N items for user based on the current state s, at time
3

---

Under review as a conference paper at ICLR 2023
Transition probability (p): The transition probability p(st+1|s;, a.) quantifies the probability from state S; to St+1 with an action a,.
Reward (r): The environment provides immediate reward as a feedback based on items rec- ommended (actions a,) to the user in state St-
Uncertainty and the Evidential Theory. Theory of evidence is a generalization of Bayesian the- ory to subjective probabilities (Dempster, 1968). We briefly introduce subjective logic (SL) (Jsang, 2016) and discuss uncertainty estimation based on SL. SL is probabilistic logic that is built upon probability theory and belief theory. It represents uncertainty by introducing vacuity of evidence in its opinion, which is a multinomial random variable in domain Y = {I...., K.. This opinion can be equivalently represented by a K K-dimensional Dirichlet distribution Dir(pla) where is a strength over K classes and evidence is the measure of the number of supportive observations from data for each class. It has fixed relationship with the concentration parameter given non-informative prior. Let ex be the evidence for class k. SL measures different types of second-order uncertainty through evi- dences, including vacuity, dissonance, and a few others (Josang et al., 2018). In particular, vacuity corresponds to the uncertainty mass of a subjective opinion wt
(P1,..., Pk) governs categorical distribution over Y. The term
K S
vac(w) = 1
S
(Ex + +1
k=1
Since vacuity is defined by lack of evidence in the data sample, it provides natural way to facilitate the exploration of an RL agent, which will be detailed next.
4 DEEP EVIDENTIAL REINFORCEMENT LEARNING FOR DYNAMIC RECOMMENDATION
Overview. We propose deep eviden- tial reinforcement learning model to per- form dynamic recommendations as shown in Figure The model includes recur- rent neural network (RNN) to maintain dy- namic state space, and an evidential-actor- critic (EAC) module to explore the item space by introducing the evidence-batt based uncertainty (vacuity) into new eviden- tial RL setting. By incorporating previous state information, recent items captured by a sliding window, and the recommended items from the RL agent, the RNN module generates the current state St. This state is further passed the action network that predicts the mean and variance to form a Gaussian policy distribution. We sample a current action from the policy distribu- tion that corresponds to the latent preference of the user that simultaneously captures the past (via previous state current (through a sliding window) and future interest (through RL exploration). By leveraging the current action and item embeddings, the evidence network provides the evidence that be used to form the rating prediction for exploitation while estimating the uncertainty for effective exploration. The Q-network (critic) generates an evidential Q-value for evidential policy updates of the action network. Table 4 in Appendix A summarizes the major notations.
Policy
Item Pool
Pa
Weighted rating uncertainty
Action network
Current action
Evidence network
Recommendation
Embedding
Concotonation
Critic network
O value: Gi
Reward
top N Recommender Items
Enrowisiing
Rocal
Bants
Figure 2: Overview of the DERL framework
4.1 ENVIRONMENT SETUP
We start by describing the environment of the proposed evidential RL agent. The environment consists of user-interacted items (an item pool Z) from this user's interaction history H., embedding matrix E to generate the user embedding, the RNN for dynamic state generation, and an evidential reward process (ERP) that specifies incentive mechanism each action of the agent. Our reward process encourages a balance between exploitation (based on predicted ratings) and exploration (based evidence-hased based uncertainty) when making recommendations users. In particular,
4

---

Under review as a conference paper at ICLR 2023
recommended list consists of limited number of items. The proposed ERP ranks candidate items according to an evidential score that integrates the predicted rating and evidence-basee uncertainty:
a, rating..: XWdy (i|s:, a:)
(2)
where À balances the rating and the uncertainty, and rating.. is the predicted rating, Given K possible rating classes, the evidence network (introduced later in this section) outputs an evidence vector (Ei ..., (ikk)) cik) for each item i. This will allow us evaluate ratingu,i as > Pik x k where Pia: is rating probability given by (9). Meanwhile, uncertainty Uz (i]s,, ar) for item can be evaluated through (1). Based on the evidential score, an RL agent will choose the top-N items to form list N₂, and recommend them to the user. As the feedback to the agent, the user provides the actual rating for each recommended items. Consequently, the evidential reward is
1 x (st, ar) = N
Σ (rating.., - -) + X24 (()s),a))
(3)
VEN.
where rating..; the user assigned ground truth rating and T is a threshold chosen based on the rating mechanism (T = 3 for a - 5 rating system). Given the evidential reward, we introduce an evidential Q-value, which can be computed by repeatedly applying the Bellman operator (B*):
B"Q" (st: a.) 2r (8t, ar) + 1=
(4)
where (s{) = Ea a.).. The evidential Q-value will be used for the update of EAC module, which is introduced later in this section.
4.2 THE CUSTOMIZED RNN FOR LATENT STATE GENERATION
specially designed RNN is used to maintain the state space of a dynamic RS environment. In particular, a state s, is generated by aggregating three pieces of information: the previous state 8;-1- items interacted by the user in the the current step, and newly recommended items. Here, an item is also an embedding vector which encodes item entity information. By aggregating all this informa- tion, the current state can evolve from the previous state by effectively capturing the past preference and future predicted preference of the user. In particular, newly interacted items extracted from the user's s interaction history H,, using a sliding window and the currently recommended items are obtained by invoking the a,-1- Assume that total M items obtained with half from the slid- ing window and the rest from the action. These M items then go through an embedding matrix to produce user embedding u, for time step t. Then, S; is formed by
(5) To train the customized RNN, we collect additional data tuples [S Ur] into the replay buffer. We then sample batches from the buffer and send u, and 8,-1 to the RNN module that generates the current state Sg. After that, we send s; to action network that samples a, from the action distribution. Action a will then go through the evidence network predict the evidence vector for cach candidate item. Finally, we compute evidential loss JEvi as defined in (10) and conduct backpropagation with respect RNN parameter
S, RNN(s:-1,
(w) = Ees
In this way, the computing graph is maintained even in the offline setting and the RNN can be trained as in the standard supervised setting.
4.3 EVIDENTIAL ACTOR CRITIC (EAC)
Training goal. A standard RL model maximizes the expected sum of rewards. We consider a generalized evidential reward function r defined in (3), which augments the standard RL objective with the average evidence-based uncertainty of the recommended items to encourage exploration of the item space. We achieve our training goal by updating the evidential actor network that finds the optimal policy to maximize the expected cumulative evidential reward as:
Jn = Σ
1=
where is the distribution of (st, a.) from the data or the replay buffer and T is the total number of time steps the episode. A novel benefit of the new objective is to allow the agent to interact with more informative items for more effective exploration of large item space. EAC consists of three key networks: action network, evidence network, and critic network, which will be detailed next.
5

---

Under review as a conference paper at ICLR 2023
Action network. The action network (or policy network) utilizes the current state S from the offline replay buffer and outputs a policy distribution T ((]]st), which is modeled as Gaussian. From this distribution, sample an action a, that is used in the evidence and the critic networks to provide recommendations and direct the policy update, respectively. For action network update, we use backward update signals from the critic network:
Vod+(o) aa, (s;, a,)) x T-
(8)
This gradient extends the DDPG style policy update (Lillicrap et al., 2015) by utilizing the chain rule the Q-network that updates the action network.
Evidence network. The evidence network predicts Dirichlet distribution of class probabilities, which can be considered as an evidence collection process. The learned evidence is informative to quantify the predictive uncertainty of recommended items. The network takes action a, from the replay buffer and item pool I to provide class level evidence. Then, the probability of rating l is
(Cak + 1)
Pik 5
where esk is the evidence collected for rating k for item i. To train the evidence network, we define a standard evidential loss by utilizing the MSE loss between rating class probability Pit and the one-hot ground truth label ys, in which Y/sk if ls is the correct rating, otherwise Vik 0:
Pia Pik)
(B))(e) = "(me Pix)2 +
(10)
S +
We update the network by backpropagating the evidential loss VEv((b) with its parameters vb.
Critic network. The critic network is designed to approximate evidential Q value utilizing the current state s; and action a, in fully connected neural network Qo(st, ac)) This Q-value judges whether the agent generated actions matches the current state S, requirements. We derived an update formulation for the critic network following the recent double DQN-based method (Mnih et al., 2015) that utilizes two critic networks to stabilize training process, achieve faster convergence, and provide a better Q-value as:
(s,, a.) BB (st, a.) + ( a++1), (
(11)
where (S4411, a(41) a target network, which updated slowly to stabilize the training process.
The evidential Q-function parameters are trained by minimizing the temporal difference (TD) emror:
1 (st, ar) (s,, (s,, 2]
JQ(0) 2e (f0.
(12)
2
where D is the distribution of (s:, ar, Sc+1 a rf (St, ac)) in offline buffer.
Furthermore, the Q-network optimized with stochastic gradient decent. The overall recommenda- tion algorithm is shown in Algorithm 1 of Appendix C.
4.4 DERIVATION OF EVIDENTIAL POLICY ITERATION
We derive evidential policy iteration general method for learning optimal uncertainty policies by alternating between evidential policy evaluation and evidential policy improvement in the maximum uncertainty framework. We compute the value of policy according to the maximum uncertainty objective of Eq. (7). DERL expresses a policy as a Gaussian distribution with mean and covariance of action neural network. With the above settings, show that the evidential policy iteration can achieve the optimal policy at convergence.
Lemma (Evidential Policy Evaluation). Given the Bellman operator D Eq. (4) and Q 1 B Q",, the Q-value will converge to the evidential Q-value of policy π + Lemma 2 (Evidential Policy Improvement), Given a new policy Tnew that is updated via Eq (8), then Q _ (St, a.) 2 (st. a;) for all ar).
Taew Theorem 3 (Evidential Policy Iteration). Alternating between evidential policy evaluation and evi- dential policy improvement for any policy
converges an optimum evidential policy n" such that Qt (st, ar) Q (st, ar) for all (st,ai) ar).

---

Under review as a conference paper at ICLR 2023
Table 2: Performance of Recommendation (average P@N and nDCG@N)
Category
Model
Moviel SerieeLens-lie IM
Moviel.ns enn-100K
Netfix
Yahoo! Music
P@5
nDCG@s
Pg5
nDCG@S
P@5
nDCG@5
P@5
nDCG@5
timeSVD++
0.5341
0.4328
0.5034
0.4145
0.5234
0.4220
0.5267
0,4190
Dynamic
CKF
0.5567
0.4481
0.5285
0.4322
0.5456
0.4344
0.5344
0.4216
CASER
D.5762
0.4613
0.5745
0.4365
Sequential
SASRec
0.6058
0.4862
0,5624
0,4515
0.5958
0.4621
0.5826
0.4422
BERT<Roc
0.6122
0.4957
0.5834
0,4855
0.5996
0.4667
0.5901
0.4522
e-gredyy
0.5977
0,4834
0.5580
0,4356
0.5850
0.4765
0.5909
0,4812
Reinforce
DRN
0.6057
0.5199
0.6154
0,5268
0.5826
0.4720
0.6085
0.5121
LIRD
0.6238
0,5332
0.5214
0.6193
0.5238
CoLin
0.6162
0.5216
0,6247
0.5285
0.5869
0.4782
0.6112
0.5194
Proposed
DERI.
0.6313
0.5365
0.6379
0.5386
0.6336
0.5372
0.6232
0.5330
Please refer to Appendices B.I, B.2 and B.3 for proofs.
The novel of use of vacuity, which is an evidence- based second-order uncertainty, for exploration in RL., can effectively identify uncertain and informative items (from large item space), indicative of users' long-term interest. In particular, the proposed evidential reward encourages the RL agent to recommend items that the model has the least knowledge (as indicated by a high vacu- ity). After collecting the user feedback, the RL agent can most effectively gain the knowledge on the user preference to make better recommendations the long run. should be noted that maximum untropy-based exploration, such soft-actor-critte (SAC) (Haarnoja et al., 2018), may not reach an optimum policy. It has been shown that high entropy may imply either high vacuity (lack of evidence) or high dissonance (conflict of strong evidence) (Shi al., 2020). However, dissonance is not effective for exploration in RS due to its focus on confusing items mostly derived based on the users' current interest. Lemma 2 shows that the evidential reward results the evidential Q-value that is optimal for the policy improvement. We have also experimentally shown this in qualitative study by demonstrating better recommendation performance than SAC based exploration.
Remark:
5 EXPERIMENTS
We conduct extensive experiments on four real-world datasets that contain explicit ratings: Movielens-IM, Movielens-100K, Netflix, and Yahoo! Music. For baseline comparisons, we use dynamic models: timeSVD++ (Koren, 2009), CKF (Gultekin & Paisley, 2014); sequential models: CASER (Tang & Wang, 2018), SASRec (Kang & McAuley, 2018), BERT4Rec (Sun et al., 2019); and RL-bbased models: e-greedy (Zhao et al., 2013), DRN (Zheng et al., 2018), LIRD (Zhao al., 2017), CoL (Wu et al., 2016). We evaluate rewards based on available ground-truth ratings, which avoids the model from learning from simulated rewards for the non-interacted items that may lead to ineffective recommendations. Further details about datasets, experimental setting, and baseline models are provided in Appendices D, E, and F.
Evaluation metrics. We use two standard metrics to measure the recommendation performance. We also use cumulative rewards for the RL-based methods.
Precision @:: It is the fraction of the top-N items recommended in each step of the episode that are positive (rating > 7) the user. We average over all users as the final precision.
nDCG@N: Normalized Discounted Cumulative Gain (nDCG) measures ranking quality, consid-
ering the relevant items within the top-N of the ranking list in each step of the RL episode. Cumulative Reward: It measures average reward considering rewards of top-N recommended items in each step for the RI. episode.
5.1 RECOMMENDATION PERFORMANCE COMPARISON
Table 2 summarizes the recommendation performance from all models. The proposed model bene- fits from both the RNN module and EAC module so that it provides better results all datasets. The dynamic and sequential models achieve less idcal performance due to their focus short-term user interest and inability to provide long-run or future preference. RL methods have shown a clear ad- vantage due their focus maximizing expected long-term rewards. Thanks to the evidence- based uncertainty exploration, DERL achieves the best performance among all DL based models.
We further show the step-wise performance of both precision @5 (P@5) and nDCG@5 metrics sidering top-5 recommended items in all datasets as shown in Figure 3. We show the average pre- cision and nDCG of the test users over each step after the model is fully trained to demonstrate the
7

---

Under review as a conference paper at ICLR 2023
1.0
1,8
1.0
14
OKE BEKT4ROC
CKF
TKE
CAF BERTORIC
0.8
LIDD
0.8
BERT4RcC
LID
08
BERT4ROC
LID
0.8
LID
0.0
$0.6
$0.6
$0.
0.4
0A4
6
10
12
14
10
12 19
Moviclens-
(b) Moviclens- 100K
30695
(c) Netflix
Stops (d) Yahoo! Music
0.0
+--C
DXT
0.6
DCT
0.0
---
C&T LRD
800.
UID
00.6
LDD
LAD
20.4
0.2
0.2
0,2
20
(c) Movielens- IM
(f) Moviclens- 100K
(g)
Netflx
(h) Yahoo! Music
Figure 3: Performance comparison in each time step: (a)-(d): P@S: (e)-(h); nDCG@5
Movietens:1M 1M
Movielens, 1.00:s100000 100K
Nettix
Yohoo
SESL
Compaatte Neworr
Camgatre
20
55
80
100
00
20
90
90
100
90
100
DG
20
45
100
(a) Movielens- IM
(b) Movielens 100K
(c)
felllxx
(d) Yahoo! Music
Musicc
Figure 4: Average cumulative reward for DERL and other RL-baselines in different training stages
Hovie MM
Movislens-1lae
Nutflic
Yahuo
1.00
0.35
0.25
0.75
0.25
0.25
302
0.00
10e
0.00
100
0.00
40
0.
(a) Moviclens- 1M
(b) Moviclens- -00K
(c) Netilix
(d) Yahoo! Music
Figure 5:
Average cumulative reward for DERL for different À
effectiveness of the dynamic recommendation. We fixed the step size to 16, 10, 20, and 10 for the Movielens- Movielens-100K, Netflix, and Yahoo! Music datasets based on their average num- ber of user-item interactions, respectively. At the initial steps, both precision and nDCG are low for all models (we choose the best model from each category as shown in Table 2). This is as expected due to lack of user interactions. All the models start to improve after the initial stage. Dynamic models and sequential models still have poor performance compared to the RI-based methods. The proposed DERL. model provides consistently better performance over the entire process. However, has a smaller advantage at the beginning due to its strong focus on exploration. It is also worth to note that the difference between DERL and other models appears be smaller on the plots because of the wide range of the y-values (0.2 - in most cases) to cover the entire recommendation cycle.
5.2 ABLATION STUDY
First, we provide a comparison of the average cumulative reward to demonstrate how the proposed model achieves higher cumulative reward than other RL-based models. Second, we analyze the impact of hyperparameter (A) that balances exploitation and exploration in the proposed model.
Cumulative reward for RL--based methods. We consider cumulative reward to measure test users' recommendation performance. We plot the average cumulative rewards for the proposed DERL. model and baseline RL-models in Figure to show their average reward over different training epochs in all four datasets. As can be seen, the cumulative rewards for DERL and RL-based model in the initial epochs are quite close. But in later epochs, DERL clearly outperforms the other baselines. This is because the model explores more effectively during the training process to enhance the knowledge of the model. Further, reward gains for those baselines are largely similar. But for DERL, it has shown significant improvement in comparison with those baselines.
8

---

Under review as a conference paper at ICLR 2023
Genre Count
Pesitive Count
Cumulatin Lalvvvive Reward
20
2
10
D
C
2
6 8 10
12
0.0
Genre
Steps
Steps
Figure 6: Genre count, Positive count, cumulative reward for a given user
Impact of hyperparameter (A). The hyperparameter (A) plays a critical role in recommending the top-N items and generating the evidential reward. We test three different settings: À = 0.1, = 0.5, and gradually reducing À from 0.5 to 0.1. As can be seen from Figure 5, dynamically adjusting À achieves consistently better performance on all datasets. This supports the intuition that in the early steps, a large À allows the model to conduct sufficient exploration. Once the model gains sufficient knowledge from the environment and is able to make accurate recommendation, reducing will allow the model to exploit its knowledge to provide effective recommendations.
5.3 QUALITATIVE STUDY
Table 3: Recommend movies for UserlD: 4967
We conduct a qualitative analysis to show the advantage of using evidence-based uncertainty (vacu- ity) for RI. exploration than other two competitive baselines: entropy- guided exploration as in the soft actor-critic (SAC) (Haarnoja et al., 2018) and contextual bandit algo- rithm (CoLin) (Wu et al., 2016). We select random test user (ID:4967) from the Movielens- IM dataset and show the genre counts, positive counts of recommended items, and cumulative reward in each step as shown in Figure 6. At the initial few steps, SAC has more positive counts but is less effective in exploration. DERL is able to explore more informative items (evidenced by more diverse genres), where different genres are denoted as Romance (R), Drama (D), Comedy (C), Thriller (T), Others (O) in the left plot. In later steps, DERL consistently outperforms both competitive model due to the better utilization of evidence- based uncertainty to discover more informative future items which could reflect the user's long-term preference. Table 3 shows the predicted vacuity for each recom- mended item. The overall higher vacuity scores indicate that DERL recommends more items that are currently unknown to the users, which is instrumental to explore their long-term interests. It also explores more diverse genres (5 vs. 3) of items than the baselines. The results show a consistent trend: DERI. focuses on exploration in the earlier phase by recommending more diverse items (left plot) and both competitive methods achieve better performance (more positive counts) during this phase (middle plot). Due to better exploration, DERL eventually achieves much better cumulative reward the later phase (right plot). More specifically, as Table shows, early steps, DERL identifies four out of five important items (genre types in bold) that are recommended based on (estimated) future long-term interest. However, SAC finds two important movies based on genre and three by the CoLin. SAC selects three comedy movies, which only reflects the user's current preference from the current and past interactions. Similarly, CoLinn in is also more focused on comedy rather than exploring diverse movies.
Model
Movies
Movic Geure
Vacuily
Round (195)
Adventury, Fantasy
Postino, (The Postman) (1994)
Romance
0.11
DERL
How to Make an American Quilt (1905)
Drama
0.14
Pocahontas (1995)
Musical
0.12
Three Lives and Only One Death (1996)
Comedy
0.22
Benjamin (1980)
Comedy
Return of the Pink Panther. The (1974)
Camedy
0.12
SAC
Lawnnower Man 2: Beyond Cybanpace (1996)
Sci-FiThriller
0.09
Ruling. Cliss, The (1972)
Comedy
0.11
Love Bloom (1935)
Romance
0.09
Young Sherlock Holies (1985)
Adventure
Karate Kid, Part The (1986)
Fantasy
0.07
CoLinn
Mighty for Young (1998)
Adventure 0.08
Christmas Vacation (1989)
Comedy 0.10
Futher the Bride Part II (1995)
Comedy 0.12
CONCLUSION
In this paper, we propose a novel deep evidential reinforcement learning framework for dynamic rec- ommendations. The proposed DERL framework learns more effective recommendation policy by integrating both the expected reward and evidence-based uncertainty. DERL integrates a customized RNN to generate the current state that accurately captures user interest and an boddennial-actor-rriie module to perform evidence-basee value. We theoretically prove the convergence behavior of the proposed evidential policy integration strategy. Experimental results on real-world data and comparison with the state-of oftte-of-th-arr the--at competi- tive models demonstrate the effectiveness of the proposed model.
exploration to optimize policy by improving an evidential Q-

---

Under review as a conference paper at ICLR 2023
REFERENCES
Xueying Bai, Jian Guan, and Hongning Wang. Model-based reinforcement learning with adversarial training for online recommendation. arXiv preprint arXiv:1911.0384. 2019.
James Bennett, Stan Lanning, al. The netflix prize. In Proceedings of KDD cup and workshop, volume 2007, pp. 35. New York, NY, USA., 2007.
Rianne van den Berg, Thomas N Kipf, and Max Welling. Graph convolutional matrix completion. arXiv preprint arXiv:
2017.
Robin Burke. Hybrid recommender systems: Survey and experiments. User modeling and user- adapted interaction, 2002.
Laurent Charlin, Rajesh Ranganath, James MeInerny, and David M Blei. Dynamic poisson fac- torization. In Proceedings of the 9th ACM Conference Recommender Systems, pp- 155-162, 2015.
Minmin Chen, Alex Beutel, Paul Covington, Sagar Jain, Francois Belletti, and Ed H. Chi. Top-k off-policy correction for reinforce recommender system. In Proceedings of the Twelfih ACM International Conference on Web Search and Data Mining, WSDM 19, pp. 456 464, New York, NY. USA. 2019a. Association for Computing Machinery. ISBN 9781450359405. doi: 10.1145/ $289600.3290999 URL htwps::tp....
3290999.
Minmin Chen, Alex Beutel, Paul Covington, Sagar Jain, Francois Belletti, and Ed H Chi. Top-k off-policy correction for reinforce recommender system. In Proc. of WSDM, 2019b.
Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, et al. Wide & deep learning for recom- mender systems. Proceedings of the Ist workshop on deep learning for recommender systems, 2016.
Arthur P Dempster. A generalization of bayesian inference. Journal of the Royal Statistical Society: Series (Methodological), 1968.
Gideon Dror, Noam Koenigstein, Yehuda Koren, and Markus Weimer. The yahoo! music dataset and kdd-cup' 11. In Proceedings of KDD Cup 2011, pp. 3-18. PMLR, 2012.
Mehrdad Farajtabar, Yinlam Chow, and Mohammad Ghavamzadeh. More robust doubly robust off-policy evaluation. In Proc. of ICML, 2018
Mingsheng Fu, Anubha Agrawal, Athirai A Irissappane, Jie Zhang, Liwei Huang, and Hong Qu Deep reinforcement learning framework for category-based item recommendation. IEEE Trans- actions on Cybernetics, 2021.
Simon Funk. Incremental svd method, 2006. URL https://ssitter.orrg// /olrna1/20061211.1 html.
San Gultekin and John Paisley. A collaborative kalman filter for time-evolving dyadic processes. In Proc. of ICDM, 2014.
Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He. Deepfm: a factorization- machine based neural network for ctr prediction. arXiv preprint 1rXiv:1703.04247 2017.
Tuomas Haarooja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- on machine learning, 2018.
Xiangnan He, Kuan Deng, Xiang Wang. Yan Li, Yongdong Zhang, and Meng Wang. Lightgen: Simplifying and powering graph convolution network for recommendation. Proceedings of the 43rd International ACM SIGIR conference research and development in Information Retrieval, pp. 639-648, 2020.
Audun Josang, Jin-Hee Cho, and Feng Chen. Uncertainty characteristics of subjective opinions. In 2018 st International Conference on Information Fusion (FUSION). 2018.
10

---

Under review as a conference paper at ICLR 2023
Audun Jsang. Subjective logic. In Subjective Logic, 2016.
Kang-Cheng Kang and Julian McAuley. Self-attentive sequential recommendation. In Proc. of ICDM, 2018.
Jens Kober, J Andrew Bagnell, and Jan Peters. Reinforcement learning in robotics: A survey. The International Journal of Robotics Research, 2013.
Yehuda Koren. Factorization meets the neighborhood: multifaceted collaborative filtering model. In Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 426-434, 2008.
Yehuda Koren. Collaborative filtering with temporal dynamics. In Proc. of KDD, 2009.
Yehuda Koren, Robert Bell, and Chris Volinsky. Matrix factorization techniques for recommender systems. Computer, 42(8):30-37, 2009.
Lihong Li, Wei Chu, John Langford, and Robert E Schapire. A centextual-bandee approach to personalized news article recommendation. Proceedings of the 19th international conference on World wide web, 2010.
Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv: (ttp:::099099999 2015.
Feng Liu, Ruiming Tang, Xutao Li, Weinan Zhang, Yunming Ye, Haokun Chen, Huifeng Guo, Yuzhou Zhang, and Xiuqiang He. State representation modeling for deep reinforcement learning based recommendation. Knowledge-Bard -Based Systems, 105:106170, 2020.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski. et al. Human-level control through deep reinforcement learning. nature, 51 8(7540):529-333 2015.
Raymond Mooney and Loriene Roy. Content-based book recommending using learning for text categorization. In Proceedings of the fifth ACM conference on Digital libraries, 2000.
Steffen Rendle. Factorization machines. In Proc. of ICDM, 2010.
Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme. Bpr: Bayesian personalized ranking from implicit feedback. arXiv preprint arXiv:/205. 220.22255222.22 2012.
Nachiketa Sahoo, Param Vir Singh, and Tridas Mukhopadhyay. A hidden markov model for collab- orative filtering. Mis Quarterly, pp. 1329-1356, 2012.
Weishi Shi, Xujiang Zhao, Feng Chen, and Qi Yu. Multifaceted uncertainty estimation for label- efficient deep learning. Advances in Neural Information Processing Systems, 2020.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 2017.
Xiaoyuan and Taghi M Khoshgoftaar. survey of collaborative filtering techniques. Advances in artificial intelligence, 2009.
Fei Sun, Jun Liu, Jian Wu, Changhua Pei, Xiao Lin, Wenwu Ou, and Peng Jiang. Bert4rec: Sequen- tial recommendation with bidirectional encoder representations from transformer. In Proceedings of the 28th ACM international conference information and knowledge management, pp. 1441- 1450, 2019.
John Sun, Dhruv Parthasarathy, and Kush R Varshney. Collaborative kalman filtering for dynamic matrix factorization. IEEE Transactions on Signal Processing. 2014.
Richard Sutton, Andrew Barto, et al. Reinforcement learning. Journal of Cognitive Neuro- science, 1999.
11

---

Under review as a conference paper at ICLR 2023
Jiaxi Tang and Ke Wang. Personalized top-n sequential recommendation via convolutional sequence embedding. In Proc. of WSDM, 2018.
Ruoxi Wang, Rakesh Shivanna, Derek Cheng, Sagar Jain, Dong Lin, Lichan Hong, and Ed Chi. Den v2: Improved deep & cross network and practical lessons for web-scale learning to rank systems. In Proceedings of the Web Conference 202/, pp. 1785-1797, 2021.
Xiang Wang, Xiangnan He, Meng Wang, Fuli Feng, and Tat-Seng Chua. Neural graph collaborative filtering. In Proceedings of the 42nd international ACM SIGIR conference Research and development in Information Retrieval, 2019.
Qingyun Wu, Huazheng Wang, Quanquan Gu, and Hongning Wang. Contextual bandits in collabo- rative environment, In Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval, pp. 529-538, 2016.
Fenfang Xie, Liang Chen, Yongjian Ye, Zibin Zheng, and Xiaola Lin. Factorization machine based service recommendation on heterogeneous information networks. In 2018 IEEE International Conference on Web Services (ICWS), 2018.
Xiangyu Zhao, Liang Zhang, Long Xia, Zhuoye Ding, Dawei Yin, and Jiliang Tang. Deep rein- forcement learning for list-wise recommendations. arXiv preprint arXiv:1801.00200 2017.
Xiaoxue Zhao, Weinan Zhang, and Jun Wang. Interactive collaborative filtering. In Proceedings of the 22nd ACM international conference on Information and Knowledge Management, 2013.
Guanjie Zheng, Fuzheng Zhang, Zihan Zheng, Yang Xiang, Nicholas Jing Yuan, Xing Xie, and Zhenhui Li. Drn: A deep reinforcement learning framework for news recommendation. In Proc. of WWW, 2018.
Lixin Zou, Long Xia, Pan Du, Zhuo Zhang, Ting Bai, Weidong Liu, Jian-Yun Nie, and Dawei Yin. Pseudo dyna-q: A reinforcement learning framework for interactive recommendation. In Proceedings of the 13th International Conference Web Search and Data Mining, 2020.
12

---

Under review as a conference paper at ICLR 2023
Appendix
Organization of Appendix. In this Appendix, we first summarize the major mathematical nota- tions in Appendix A. We then present the proofs of lemmas and theorems in Appendix B. We show the detailed DERL algorithm in Appendix C. We present the details of the dataset in Appendix D, experimental setting in Appendix E, and baseline models in Appendix F. Further we include some additional comparison results in Appendix G. The link to the source code is given in Appendix H.
A SUMMARY OF NOTATIONS
Table 4: Summary of Notations
Notation
Description
step length of episode user (cpisode) and item indices embedding matrix for item embedding
U,
E
user u's embedding time step and action time evidence and evidence-based probability on rating class item
St, Be
csi; Pik
one hot rating label on item i parameters action and evidence networks parameters critic and recurrent networks
o. eb
0,
t,, Q(5e, ac)
recommendation policy or action network, Q value function or critic network
rating., rating.,i
predicted and actual rating for user a on item evidential score for user item i rating threshold and balance syper-parameter exploitation and exploration
score... 7,
(r(ss, ax) r (s,, ac), (*), K,M M
item i's evidence-hased uncertainty evidential reward and value the number of rating class, the number of interacted items in each time step
No. H.. I
top-N recommended items list for user u user i's interaction history and item pool
B PROOFS OF THEORETICAL RESULTS
In this section, we provide proofs of all lemmas and the theorem.
B. PROOF LEMMA
Given the evidential reward defined as (s ar) (ErEN, (rating -) + XU, ai)) the update rule for evidential Q- Q-value can written as:
(sc, ar) E
/ (Str,
ar) (St,ar)
(st. + 7. (2 ac+1)
(13)
Then based on the evaluation convergence rule (Sutton et al., 1999) with finite action space, is guaranteed that the Q-value will converge to the evidential Q-value of policy n.
B.2 PROOF OF LEMMA 2
The policy can be updated towards the new Q-value function. Consider the updated policy Nnw as the optimizer of the maximization problem.
Tnew arg max J (0) arg max D. 19 a:)]
(14)
13

---

Under review as a conference paper at ICLR 2023
Denote the old policy as wold- Using the update rule specified in Eq (8) with sufficiently small step size, we get an updated policy nnew that satisfies
Ea, Tnes [Q (8z, at)] 2 Baarnooee 1atooa (Sc. ax)]
(15)
Given Eq (15), we have the following inequality
Q Futa (st, ar)
(s ar) + 7 Bess waner 1 Told (51+1,
Sr" <r"(s,, ar) + BBs, 1 mnow ( 18++1,, ac+1]]
1 ($142, a(+2)l
(16)
-Q - (Sc, as)
where r" r"(s;, ar) is a evidential reward in step t. Therefore, we show that the new policy Tnew ensures @. (St, ar) 2 Ou (St, ar) for all az).
B.3 PROOF OF THEOREM 1
Let my denote the policy at iteration i. We already show that the sequence Q (8; a,) is mono- tonically increasing. Since Q= (s,, ar) bounded above, the sequence converges to some π². At convergence, it must be the case that J/ (I* (.(s:) S J (T(((s)) for π * x². Based on Lamma 2, we have - ar) > Q (st, a.) for all (st, a.). ) In other words, the evidence value of any other policy и is lower than that of the converged policy x². Therefore, it guarantees convergency to an optimal policy " such that:
Q5,(s;, 2 (s..
(17)
C DEEP EVIDENTIAL REINFORCEMENT LEARNING ALGORITHM
Algorithm Deep Evidential Reinforcement Learning
Require: Hyperparameters: or 3, A, and time step: T
Initialize RNN: w, action network: @, evidence network: si and critic network: 0 initial state: So and initial user embedding:
with M items from interactino history H.
2: for each epoch do for each user as episode do 4: for t do
3:
5:
Compute state: S. with (5).
6:
Compute action: a, ~ (-(ss)
7:
Compute evidential score using (2)
8:
Recommend top-N items based on computed evidential score.
9:
Compute rewards for top-N items utilizing Equation 3. Add (u,, 8;-1, r (st, 2 (-Ise, a.), done) into replay buffer
10:
11:
M Take 2 items from the sliding window and other 2
A items from RL-agent recom-
mended items
12:
for each gradient step do
13:
Sample batched data from replay buffer and update:
14:
Update action network with (8)
Update evidence network with (10)
16:
Update critic network with (12)
Update RNN network with (6)
end for 19: end for
18:
20: end for
21:
end for
D DESCRIPTION OF THE DATASETS
We evaluated DERL four public benchmark datasets that contain explicit ratings:
14

---

Under review as a conference paper at ICLR 2023
Movielens- 1M This dataset includes 1M explicit feedback (ratings) made by 6,040 anonymous users on 3,900 distinct movies from 04/2000 to 02/2003.
Movielens-100K:: This dataset contains 100,000 explicit ratings on a scale of (1-5) from 943 users on 1,682 movies. Each user at least rated 20 movies from September 19, 1997 through April 22, 1998.
Netflix (Bennett et al., 2007): This dataset has around 100 million interactions, 480,000 users, and nearly 18,000 movies rated between 1998 to 2005. We pre-processed the dataset and selected 6,042 users with user-item interactions from 01/2002 to 12/2005.
Yahoo! Music rating (Dror et al., 2012): The dataset includes approximately 300,000 user-supplied ratings, and exactly 54,000 ratings for randomly selected songs. The ratings for randomly selected songs were collected between August 22, 2006 and September 7, 2006.
E EXPERIMENTAL SETTING
We consider each user an episode for the RL setting and split users into 70% as training users and 30% as test users. For each user, we select the first M 10 interacted items to represent an initial state So- In the next state, we utilize previous state representation and concatenate five items embedding from sliding window and other five items embedding from RI agent to generate current state 5; by passing through the RNN module. Then, the action network generates mean and covariance for Gaussian policy from which action is sampled. This action is further passed to the evidence network, which utilizes the embeddings of user interacted items to produce corresponding evidence for each item. We use the setting of classification, where explicit ratings used as class labels. With that evidence, we compute the evidential score by evaluating evidence-based rating and uncertainty rank those items, which provides list of top-N final recommendations. We then evaluate the evidential reward. We put 10 interactions into each period and set = 3.
F COMPARISON BASELINES
We compare with dynamic, sequential, and reinforcement learning models:
Dynamic models include standard dynamic matrix factorization model timeSVD+ (Koren, 2009) as the time-evolving latent factorization model and collaborative Kalman filtering (CKF) (Gultekin Paisley, 2014).
Sequential models include Sequential Recommendation via Convolutional Sequence Embed- ding (Caser) (Tang & Wang, 2018), attention batentioo-bbsed sequential recommendation model (SAS- Rec) (Kang & McAuley, 2018), and sequential recommendation with bidirectional encoder (BERT4Rec) (Sun et al., 2019).
Reinforcement learning-based models include e-greedy (Zhao al., 2013), deep Q-network based news recommendation (DRN) (Zheng et al., 2018), and actor-critic based list-wise recom- mendation (LIRD) (Zhao al., 2017), and contextual bandit based method CoLin (Wu et al., 2016).
G ADDITIONAL EXPERIMENTS AND COMPARISON RESULTS
In this section, we present additional experiments, with focus on comparing different types of basclines.
G.1 COMPARISON WITH CONTEXTUAL BANDIT BASED METHODS
In the main paper, we compared with state-of-ther art collaborative contextual bandit based recom- mendation method, Colin (Wu et al., 2016). Here, we include two additional bandit based models 3:
Phttps: Phttps://gt thub. com/ Http//CCMM/Baddd tliib
htttpp:://rroppeere www.lensss.rrr.gg...e.oo. http:///ttg/aatttt///e//a///ae ensee/ 100k/
15

---

Under review as a conference paper at ICLR 2023
1.0
DEAL RENPORICE
DESL RETNFONCE
0.6
DEAL
DEAL FUII CICE
0.5
0.6
REINFONCE
0.5
1.1.0
0.3
0.4
0.4
6.2
0.2
0.
10
Stops
Staps
Stops
Steps
Figure 8: Performance comparison each time step
LinUCB (Lin) and Hybrid-LinUCB (HLin), to show more complete comparison. We report the cumulative reward. DERL shows a clear advantage over all three bandit methods, which further justifies its better exploration capability capture users' long-tenm Long-term interest.
Moviclens: 1M
Novicless 100K
Yahoo
Netflix
cururatt- Dhaura
0.5
30
0
20
0.0
Figure 7: Comparison between bandit-based models and ours in four different datasets.
G.2 COMPARISON WITH OTHER BASELINES
In this section, we include two recent models for comparison: DeepFM (Guo et al., 2017) and DCNv2 (Wang et al., 2021). DeepFM integrates traditional factorization machine and deep learn- ing to learn low- and high-order feature interactions. Similarly, DCNv2 is more expressive to learn feature interactions and also more cost-efficient. We also include classical RL-based method called REINFORCE (Chen et al., 2019a), which applies off-policy learning to handle data bias. The test performance metric P@5 and nDCG@5 among the proposed model DERL and above three baselines in two datasets Movielens- 1M and Movielens- 100K are shown in Table 5 and Figure 8. Although these two deep learning-basse based recommender models achieve reasonable recommendation performance, they mainly lack to handle the temporal preference of the users and hence perform worse than the proposed DERL method. Furthermore, REINFORCE has limited exploration power and cannot effectively capture long-term user preference in the future, hence its performance is also lower than DERL. We further add a static recommendation model, LightGCN (He et al., 2020), which computes user and item embeddings via linear aggregation of its neighbors. Its perfor- mance is lower than DERL in both datasets because it is ineffective handling highly sparse user interactions in dynamic setting. In addition,
also compare with recent RL-based methods: DRR (Fu et al., 2021) and DHRC (Liu et al., 2020). The significant advantage of exploration over these baselines further confirms its outstanding recommendation performance achieved by DERL.
Table 5: Comparison of Recommendation Performance (average P@ N and nDCG@N)
Model
MovieLens- IM
MovieLens- -nnKK
P@5
nDC@@5
P@S
nDCG@5
DeepFM
0.5647
0.4625
0.5428
0.4514
LightGCN
0,5668
0.4884
0.5714
0.4852
DRR
0.5370
0.5112
0,5890
0.5225
DHCRS
0.4980
0.5080
0.5214
0.4746
REINFORCE
0.6074
0.5116
0.5926
0,5149
DCNv2
0.6152
0.5187
0.6158
0.5166
DERL
0.6313--0.008
0.066
0..084
16

---

Under review as a conference paper at ICLR 2023
DEAL Umvorrarnty
Guruuabiv Rerverd
6.6
DEAL Uncertarnty SSA
Guruaatv
0.4
0.2
0.0
Stops
Stops
Figure 9: Comparisons exploration strategies in Movielens- 1M (left) and Moviclens- 100K (right).
G.3 IMPACT OF VACUITY FOR EXPLORATION
Uncertainty is commonly used to support exploration of RL. agents in different applications. Eviden- tial deep learning leverages second-order uncertainty (i.e., vacuity) to perform effective exploration. The proposed evidential reward encourage the RL agent recommend items that the model has the least knowledge (as indicated by high vacuity). After collecting the user feedback, RL agent most effectively gain the knowledge on the user preference to make better recommendations in the long run. The effectiveness vacuity guided exploration has been demonstrated in both the motivating example in the introduction and also our empirical evaluations.
We further investigate the effect vacuity in our proposed model by comparing DERL with an alternative design without vacuity. Furthermore, also compare exploration using the first order uncertainty, which is employed by soft-actor-critic (SAC). We show the comparison results on two datasets in Figure It can be seen that without uncertainty guided exploration, the model collects the least cumulative reward in long run. SAC utilizes entropy based exploration and achieves better cumulative reward than without uncertainty guided exploration. This provides evidence that role of the exploration is crucial in RL-based recommendation. However, it performs worse than the vacuity based DERL method. This is because vacuity guided exploration allows our model focus its exploration on the most informative items that help the model gain the most knowledge to form an optimal policy. The advantage over entropy- exploration also consistent with our earlier discussion in Section 4.4 of of the main paper.
H LINK FOR THE SOURCE CODE
https: /ttp:s/////mmmous..m// 4open. oacar ion-ndDD///// README .d
17


Reviews:
[{"review_id": "lvxgvPck7", "reviewer": "Reviewer_h31X", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new approach to exploration in RL for recommender systems. The authors propose using the concept of vacuity to model uncertainty about an item's rating, and include it as an exploration bonus into the reward function.\nOffline results through multiple datasets show that the proposed method, DERL, not only is able to achieve higher top-5 precision compared to other methods, but produces more varied results thanks to the exploration component.", "summary_of_the_review": "The paper proposes a novel approach to exploration for RL in the context of recommender systems. \nThe authors ran extensive tests in which the results align with the claim made. Most importantly, the qualitative show by inspection that DERL is able to produce more varied results, instead of biasing recommendations to one particular genre.\n\nThe paper is well written, but I would ask the authors to clarify a few things in the paper with more details:\n\n1 - In eq 9, what precisely is \"e_k\" in this case? What constitutes evidence for class k?\n2 - Eq 10., why use MSE as a loss of the first component when you have \"p_ik\" and \"y_ik\"? Isn't this akin to a classification problem, and you could use cross-entropy?\n3 - For the uncertainty \"U(i | s, a)\", can you clarify how eq. 1 is used in the evaluation. As defined, Eq. 1 is K / sum_k^K (e_k + 1), which means it is not a function of any particular class (K is number of classes, and the bottom is a sum over all classes). \nSo how can U( i | s ,a ) be evaluated for 1 particular item? Wouldn't  i be considered 1 class over all possible classes? ", "strength_and_weaknesses": "Strength:\n- The paper introduces a novel idea for exploration, as far as I am aware. Maintaining varied recommendations, while still showing relevant items to the user is a difficult problem, and the introduction of vacuity seems to be doing a good job at addressing that problem.\n- The authors evaluated in a large number of datasets over multiple competing benchmarks, which strengthen confidence in the propose method.\n\n\n\nWeakness:\n- There are a few key components of the paper that were not completely clear (see summary of review). For example, in eq 1, what constitutes evidence \"e_k\"? It should take some minor editing to clarify some of these questions.\n- I understand it's difficult to do, but since the paper focus is on exploration, it would have strengthened the work quite a bit to have an online experiment. Exploration means producing results that have not been produced before, and these counterfactuals cannot be evaluated from historical data. \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "aNcdwIXnrKH", "reviewer": "Reviewer_JmPi", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a exploration method, called  deep evidential reinforcement learning (DERL)  to improve the exploration ability of RLRS. The DERL algorithm uses recurrent neural network to represent the dynamic feature of the user state, and conducts evidential-actor-critic module to enable better exploration. Experimental results validates its effectiveness over dynamic models, sequential models, and previous RL for RS methods. Finally, some ablation studies and qualitative studies are conducted.", "summary_of_the_review": "Overall, the paper study an important problem for RL in RS. The paper is well written, and does experiments on several public datasets.\nHowever, the novelty of the paper is limited, and does not discuss and compare current exploration methods. Also, the experiments are not sound. Thus, I choose to reject this paper at this version.", "strength_and_weaknesses": "Strength:\n\n1.Exploration in RL for RS is an important problem.\n\n2.The paper is generally well written and easy to follow.\n\n3.The experiments are on several public datasets.\n\nWeakness:\n\n1.The technical quality of this paper is not enough, and it seems like a direct combination with Evidential Theory and Reinforcement Learning.\n\n2.The paper is not sound as there are many exploration methods in RL literature, such as count-based methods and intrinsic motivations(RND,ICM). But the paper does not discuss and compare these methods.\n\n3.The theoretical analysis is not novel, as it is a direct result of RL theory.\n\n4.The update rule of the critic network does not follow Double DQN, but follow the clipped double q-learning in the well known TD3 algorithm.\n\n5.The paper does not provide a specification of the experimental setup. Did the authors bulid simulator? If not, how to evaluate the performance of each policy in the offline setting?\n\n6.Why not compare SAC in Table 2 as SAC is compared in Figure 6?\n\n7.How to verify that the performance improvement over pervious RL methods indeed comes from the evidential reward? As we can see, you choose some advanced techniques like Eq (11), and it is not deployed in previous baselines.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "H0lKLaJJ6u", "reviewer": "Reviewer_Wp6W", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a deep evidential reinforcement learning framework, which aims to learn a more effective recommendation policy by integrating both the expected reward and evidence-based uncertainty. Specifically, a recurrent neural network (RNN) is used to exploit dynamic state space, which generates the current state by incorporating past (previous state information), current (sliding window interactions), and future (recommended items from the RL agent) interests. Besides, an EAC module (evidential-action-critic network) is used to perform evidence-based exploration by maximizing a uniquely designed evidential Q-value to derive an optimal policy. Experimental results on real-world data have been performed to demonstrate the effectiveness of the proposed model. ", "summary_of_the_review": "This paper provides a novel reinforcement learning-based recommendation framework. However, the experimental analysis seems not convincing enough. The baselines used in the experiments are not strong enough. ", "strength_and_weaknesses": "Strengths\n1) The authors propose a new recommendation framework, i.e., DERL, which integrates a customized RNN and an EAC (evidence-actor-critic) module to exploit dynamic state space and explore item space, respectively.\n2) The proposed method leverages vacuity, an second-order uncertainty, to perform effective exploration. \n3) The authors have performed extensive experiments on real datasets to demonstrate the effectiveness of the proposed method.\n \nWeaknesses\n1) In Section 1, the third contribution claimed by the authors is \"An off-policy formulation to effectively promote the reuse of previously collected data while stabilizing model training, which is important to address data scarcity in recommender system\". However, in this work, there is no experimental analysis demonstrating the proposed method can address the data scarcity challenge. \n2) In Figure 1 and Table 1, the examples show that existing methods fail to capture the user's dynamic preference to obtain the maximum reward. However, the description about the user's long-term preference is missing in these cases. \n3) In Section 4.3, for the evidence network, the authors introduce that \"e_{ik} is the evidence collected for rating k for item i\". However, it is not clear how to collect the learned evidence e_{ik} from the current action a_t and the item pool I using the evidence network. \n4) The description about the experimental datasets are not clear enough. For example, the number of interactions in the pre-processed Netflix dataset, the number of users and the number of items on Yahoo dataset. \n5) Even though this paper includes many baselines, some classical top-N item recommendation methods should be considered as baselines, e.g., BPRMF and LightGCN. For dynamic recommendation models, the authors need to consider the dynamic top-N recommendation methods as baselines, for example, Dynamic poisson factorization, RecSys 2015. \n6) The RL-based recommendation baselines, i.e., DRN, LIRD, and CoLin, are published in 2018 or earlier. The authors need to consider state-of-the-art RL-based recommendation methods as baselines. More RL-based recommendation methods can be found in these surveys: 1) Reinforcement learning based recommender systems: A survey, ACM Computing Surveys 2022, 2) A Survey on Reinforcement Learning for Recommender Systems, arXiv:2109.10665.\n7) Does the input feature of the model contain category information (e.g., movie genre)? What factors can help improve the diversity of genre in the evidence-based exploration? \n8) The link to the source code is expired. \n\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject