Looking into User’s Long-term Interests through the Lens of Conservative Evidential Learning
Authors: Dingrong Wang
Krishna Prasad Neupane
Ervine Zheng
Qi Yu
Venue: ICLR 2025 Poster (2025)
Keywords: recommender systems
evidence-aware exploration
evidential learning
Primary area: reinforcement learning
TL;DR: We propose a novel evidential conservative Q-learning framework (ECQL) that learns an effective and conservative recommendation policy by integrating evidence-based uncertainty and conservative learning

Abstract: Reinforcement learning (RL) provides an effective means to capture users' evolving preferences, leading to improved recommendation performance over time. However, existing RL approaches primarily rely on standard exploration strategies, which are less effective for a large item space with sparse reward signals given the limited interactions for most users. Therefore, they may not be able to learn the optimal policy that effectively captures user's evolving preferences and achieves the maximum expected reward over the long term. In this paper, we propose a novel evidential conservative Q-learning framework (ECQL) that learns an effective and conservative recommendation policy by integrating evidence-based uncertainty and conservative learning. ECQL conducts evidence-aware explorations to discover items that are located beyond current observations but reflect users' long-term interests. It offers an uncertainty-aware conservative view on policy evaluation to discourage deviating too much from users' current interests. Two central components of ECQL include a uniquely designed sequential state encoder and a novel conservative evidential-actor-critic (CEAC) module. The former generates the current state of the environment by aggregating historical information and a sliding window that contains the current user interactions as well as newly recommended items from RL exploration that may represent short and long-term interests respectively. The latter performs an evidence-based rating prediction by maximizing the conservative evidential Q-value and leverages an uncertainty-aware ranking score to explore the item space for a more diverse and valuable recommendation. Experiments on multiple real-world dynamic datasets demonstrate the state-of-the-art performance of ECQL and its capability to capture users' long-term interests.

Published as a conference paper at ICLR 2025
LOOKING INTO USER'S LONG-TERM INTERESTS THROUGH THE LENS OF CONSERVATIVE EVIDENTIAL LEARNING
Dingrong Wang', Krishna Prasad Neupanc", Ervine Zheng", Qi Yu Roohesser Institute of Technology, 2 *Amazon, Samsung Research (dw7445, kpn3569, mxz5733, qi. (..y)rrit.edu grr
ABSTRACT
Reinforcement learning (RL) provides an effective means to capture users' evolving preferences, leading to improved recommendation performance over time. How- ever, existing RL approaches primarily rely standard exploration strategies, which are less effective for a large item space with sparse reward signals given the limited interactions for most users. Therefore, they may not be able to learn the optimal policy that effectively captures user's s evolving preferences and achieves the maximum expected reward over the long term. In this paper, we propose novel evidential conservative Q-learning framework (ECQL) that learns an effective and conservative recommendation policy by integrating evidence-bard based uncertainty and conservative learning. ECQL conducts evidence-awand explorations to discover items that are located beyond current observations but reflect users' long-term interests. It offers an uncertainty- discourage deviating too much from users' current interests. Two central compo- nents of ECQL include uniquely designed sequential state encoder and a novel conservative evidential-actor--rii
conservative view on policy evaluation to
(CEAC) module. The former generates the current state of the environment by aggregating historical information and sliding window that contains the current user interactions as well as newly recommended items from RL exploration that may represent short and long-term interests respec- tively. The latter performs an evidence-based rating prediction by maximizing the conservative evidential Q-value and leverages an uncertainty-awer to explore the item space for a more diverse and valuable recommendation. Exper- iments on multiple real-world dynamic datasets demonstrate the state-of-the-arr performance of ECQL and its capability to capture users' long-term interests.
ranking score
INTRODUCTION
Recommender systems (RS) have been widely used for providing personalized recommendations in diverse fields, such as media, entertainment, and e-commerce by effectively improving user experience
(Su & Khoshgoftaar 2009 Sun et al. 2014 Xie et al. 2018). Increasing efforts have been devoted to capturing users' evolving preferences by shifting the latent user preference over time (Koren) 2009)]
Charlin et al. 2015 Gultekin & Paisley 2014). Similarly, sequential recommendation methods (Kng & McAuley| 2018 TTang & Wang 2018)) attempt to incorporate users' dynamic behavior by leveraging previously interacted items. However, existing dynamic recommendation methods primarily focus on maximizing the immediate (i.e., short-term) reward when making recommendations. As result, they fail to take into account whether these recommended items will lead to long-term returns in the future, which is essential to maintaining stable user base for the system in the long run.
Several recent works have adapted reinforcement learning (RL) in the RS context (Chen et al. 2019 Zhao et al. 2017). RL has gained huge success in diverse fields, such as robotics (Kober et al. 2013 and games Silver et al. 2011) 7). The core idea of RI. to learn an optimal policy to maximize the total expected reward in the long run. RI. methods consider recommendation procedure as sequential interactions between users and RI. agents to learn the optimal recommendation policies effectively.
Although RL approaches show promising results in RS (Chen et al. 2019 Zheng et al.) 2018), they
"Corresponding author.

---

Published as a conference paper at ICLR 2025
primarily rely standard exploration strategies (e.g., e-greedy). which are less effective for large item space with sparse reward signals given the limited interactions for most users. Therefore, they may not be able to learn the optimal policy that effectively captures user's evolving preferences and achieves the maximum expected reward over the long term.
Figure I further illustrates the limitation of ex-
ploration using standard c-greedy. In particu- lar, we compare a RS model trained using greedy and one trained using evidential uncer- tainty, which are denoted as e-greedy and ECQL, respectively. In each step, both models recom- mend five items, and we identify their ground- truth ratings are actually positive or not, by com- paring with a positive threshold. The count of positive items in each step defines the Positive Count. Also, we collect the genre type infor- mation of these items across 12 steps. In the end, we have 60 total items of different genres and use them generate the Genre Count plot. As can be seen, the existing RI. agent primarily focuses on highly-rated items in early steps, as shown in Figure la Most of these items come from a narrower set of genres as shown in Figure Ib] where different genres are denoted as Adventure (A), Drama (D), Comedy (C), Thriller (T), and Others (O). These only represent the user's temporary inter- est as in the later steps the positive count (See definition in Section of the same genre is decreased. Such recommendation behavior leads to lower test return (average rating of recommendation) in the later steps. To further show the advantage of exploration with evidential uncertainty, we define genres (bold) with top-5 average ground-truth ratings as user's long-term interests. Then we capture the last step's 's recommended items from one test user to see if they match these genres representing user's long-term interests. As Table shows, e-greedy mostly focuses on Drama movies based on the user's short-term interests analyzed from the initial user profile. It only captures one novel genre (i.e., Musical, bold in the table) that matches the user's long-term interest, while ECQL captures four items with more diverse genres that reflect the user's true interests. It clearly indicates that more systematic exploration is essential to discover users' long-term interests to maximize future return.
Positive Count
ECQL
30-
ECOL .grendy
25
20
15:
12
Steps
Genre
(a) Positive Count
(b) Genre Count
Figure 1: Different recommendation behavior for representative test user with e-greedy based ex- ploration and our model ECQL. with evidential uncertainty (i.e,, vacuity).
To address the above key challenges, we con- duct novel evidential conservative Q-learning (ECQL) that utilizes balanced exploitation (with high predicted ratings) and exploration (with evidential uncertainty) strategy for effec- tive recommendations. We formulate an evi- dential RL framework that augments the reward based RL objective with evidential uncertainty to facilitate the exploration unknown items. The evidential uncertainty formulation substan- e-greedy tially improves exploration and robustness by acquiring diverse behaviors that indicative of a user's long-term interest. As shown in Figure T (b), ECQL devotes a strong focus on more
Table 1: Examples of recommended movies
Model
Important Items (Movies)
Movie Genre
Sound of Music (1965)
Musical
Casino (1995)
Drama
ECQL
Ben-Hur (1959)
Action
The Rug's Life (1998)
Comedy
Bube (1995)
Comedy
Pocahontas (1995)
Musical
Wizard of Oz (1939)
Drama
Christmas Story (1983)
Drama
Erin Brockovich (2000)
Drama
Restoration (1995)
Drama
diverse genres (i.e., 'others' in the figure), and many of these capture the long-term interest from the user as verified by the detailed recommendation list in Table Additionally, we encourage the model explore items that do not significantly deviate from users" current interests given the sparse interactions. Such gradual exploration is guided by conservative evidential Q-value that prevents recommending totally irrelevant items, causing user frustrations. We theoretically prove that the conservative evidential Q-value provides an uncertainty-an behavior policy that represents users' current interests. Also, we demonstrate that such Q-value estimation produces a lower bound on the actual value of the target policy, providing a conservative view for safe recommendations. We further show that such a conservative view can be incorporated into a policy learning procedure with theoretically guaranteed policy improvement.
adjustment of optimism towards the
The proposed ECQL seamlessly integrates two major components: a sequential state encoder and a Conservative Evidential Actor-Critte (CEAC) module. The former primarily focuses generating
2

---

Published as a conference paper at ICLR 2025
the current state of the environment by aggregating the previous state, the current items captured by a sliding window, and the future items from the recommendation. It provides an effective means of dynamic state representation for better future recommendations. The CEAC module leverages evidential uncertainty to effectively explore the item space to recommend items that potentially align with the user's long-term interest. It encourages learning the optimal policy by maximizing novel conservative evidential Q-value to make more diverse recommendations that may reflect long-term interest while keeping a conservative view that does not deviate too much from current interests. The main contribution of this paper is five-fold:
a novel recommendation model that integrates reinforcement learning with evidential learning to provide uncertainty-awart aware diverse recommendations that may reflect users' long-term interests, evidential uncertainty guided exploitation to maximize information gain inform model learning, conservative off-policy formulation to avoid over-estimation of policy value that leads to low- quality recommendations due to overfitting to sparse training data from limited user interactions, a thorough theoretical analysis to justify the convergence behavior and avoid risky (or overly optimistic) recommendations.
seamless integration of sequential encoder, actor- network, and an evidence network to provide an end-to-end integrated training process.
We conduct extensive experiments over four real-world datasets and compare with state-of-the-ar baselines to demonstrate the effectiveness of the proposed model.
2 RELATED WORK
Sequential recommendation models. Recently developed sequential recommendation models utilize historical interactions to capture users' preferences over time through various deep architectures, including CNN (Tang & Wang 2018). transformer (Kang & McAuley 2018), and bidirectional encoder (Sun et al. 2019). 5 Rec (Zhou et al. 2020) considers the intrinsic data correlation with mutual information maximization to derive a self-supervised signal to enhance the data representation. CL4SRec (Xie et al. 2022) employs contrastive learning to learn the self-supervised signal from the user behavioral data, which helps extract more meaningful patterns for user representation. However, sequential models are inadequate to capture the users' long-term preferences. The proposed ECQL. model aims to fill this critical gap by performing sridence-guided exploration to maximize the total expected reward over the long run.
RL-based models. RL-based RS models aim to learn an effective policy to capture a user's latent preference. The on-policy learning with contextual bandit (Li et al., (010b) and Markov Decision Process (MDP) (Zheng et al., 2018) exploits by interacting with real customers in an online en- vironment. A collaborative contextual bandit algorithm called CoLin (Wu et al., 2016) utilizes graph structure in a collaborative manner. On the other hand, off-policy RL utilizes Monte Carlo (MC) and -hrporraldiiffrreee (TD) methods to achieve stable and efficient learning with users' history (Farajtabar et al., 2018). A Q-network with a hierarchical LSTM has been developed to optimize long-term user engagement (Zou et al. 2019). Similarly, model-based RL models user-agent interactions via generative adversarial network (Bai et al. 2001)). Pseudo Dyna-Q further integrates both direct and indirect RL approaches in single unified framework without requiring real customer interactions (Zou et al. 2020). More recently. SAR leverages an actor-critic network, where the action is generated as an adaptive sequence length to better represent the user's 's sequential (preference) pattern (Antaris & Rataildis 2021). ResAct utilizes a residual actor network to reconstruct a policy that is close to but more efficient than online policy in sequential recommendations (Xue et al.) 2022). However, most existing RI. based models rely on random exploration strategies, which are less effective at capturing users' long-term preferences, especially from a large item space with sparse user interactions. In contrast, our ECQL utilizes evidence-based uncertainty to systematically explore the item space to maximize the long-term reward.
3 PRELIMINARIES
Recommendation Formulation with RL. We formulate recommendation tasks in an RL setting, where an RL agent interacts with the environment (i.e., users and items) to recommend the next items user. We design this problem rewards, denoted as (S, A, P. R). state s, = SSE(-|s,-1, u.) E S is generated by a sequential state encoder that utilizes previous state - and current user embedding u, generated from the
MDP, which includes sequence of states, actions, and
3

---

Published as a conference paper at ICLR 2025
Gaussian
Pollcy
(l)
offiiing Scale
Previous state (s
Current state (6,)
Action network
Current action (a.)
Evidence network
Score Calculation
Rank
Embedding
Rating
Conservative Evidential Q-value: Q (s_taa
Concatenation
Critic
Evidential RL Reward
User Mask Based
THE
Top selection
Bon 1
Rom
Item 6
Nomm 10
Item
Ieem 15
twm
Interaction history (H,)
Figure 2: Overview of the ECQL framework
concatenation of N items provided by sliding window and an RL-agent. An action E A is vector that represents the user's current and potential preferences that is used to recommend top-N items for a user given the current state The transition probability P(s |St, as) quantifies the probability from state 8; to 8 with an action a, The RS environment provides a reward r; E R. based on the recommendation quality in the current time step.
Uncertainty and Evidential Theory. Subjective Logic (SL) is a probabilistic logic that is built upon probability theory and belief theory (Jsang 2016). It represents uncertainty by introducing vacuity of evidence in its opinion, which is multinomial random variable in K-simplex domain Y. This opinion can be equivalently represented by -idimensional Dirichlet distribution Dir(pla) where a is a strength vector over K classes and p = (Pi)...PPk) governs a categorical distribution over Y. The evidence e (ei,... .k) measures the number supportive observations from data for each class. It has fixed relationship e₁ Og informative prior. Let ex be the evidence for a class k. SL measures different types second-order uncertainty through evidence, including vacuity, dissonance, and a few others (Josang et al. 2018). In particular, vacuity corresponds to the uncertainty mass of a subjective opinion wt
with the Dirichlet strength a given non-
K s
K
= (ac(w))[((11))
S
(ex +1)
Since vacuity measures lack of evidence (knowledge), it provides a natural means for exploration.
4 EVIDENTIAL CONSERVATIVE Q-LEARNING
Overview. We propose an Evidential Conservative Q-learning RS model to perform dynamic recommendations as shown in Figure? 2 The model includes sequential state encoder (SSE) to maintain a dynamic state space with a sTiding window W; which moves along the user's interaction history H., over time input new data into the SSE, and conservative exidential--ctoor-eritt (CEAC) module which functions as an RL agent to explore the item space by introducing the evidence-bard uncertainty (i.e,, vacuity) into new off off-policy evidential RL setting. By incorporating previous state information, recent items captured by sliding window, and the recommended items from the RL agent, the sequential encoder generates the current state s,. This state further passed to the action network that predicts the mean and variance to form a Gaussian policy distribution. We sample current action a, from the policy distribution that corresponds to the latent preference of the user that simultaneously captures the past (via a previous state), current (through sliding window) and future interest (through RL exploration). By leveraging the current action and total item embeddings from the Item Pool (I), the evidence network provides the evidence that can be used to form the rating prediction for exploitation while estimating the uncertainty for better exploration. The Q-network (critic) generates conservative evidential Q-value for conservative policy updates of the action network. ili Appendix summarizes the major notations.
4.1 ENVIRONMENT SETUP
We start by deseribing the environment of the proposed evidential RL agent. The environment consists of buffer containing all the user-interacted item embeddings with ground-truth ratings (i.e.,
4

---

Published as a conference paper at ICLR 2025
the user's interaction history Hu), total item embeddings (i.e., item pool Σ), a sliding window W, that moves along the interaction history to generate new data input each time into the SSE for dynamic state maintenance, and recommendation mechanism that specifies score function to rank items and select top-N of them to recommend. The score function encourages a balance between exploitation (based on predicted ratings) and exploration (based on evidence-based uncertainty) and is defined as:
X score ratinng.. log(h; W. 1)
Un (-le.)
where E H, > W with h; being the original appearance position of a recommended item i in the user's interaction history and W, indexes the observation, i.e. the last experienced item appearance position reached by the current sliding window, which separates the observed items from the unknown ones. Parameter X E (0, 1) gradually balances the rating and the uncertainty along the training epochs and the test steps. rating,,i is the predicted rating for exploitation. Given K possible rating classes, the evidence network (introduced later in this section) outputs evidence vector e; = (ea..... exx) for each item This will allow us to evaluate rating as Σk= Pa where Psk is rating probability. Meanwhile, vacuity Ut (ile,) for item can be evaluated through (T) for exploration. We aim to recommend unknown items that reflect the user's 's future interests while avoiding too risky recommendations far ahead of the current observation. To this end, we consider the original appearance position h; and down-weight the information value (captured by vacuity (f)ce) of items that are interacted deep into the future. During testing, no down- weinh- is applied as Hu, h2 are unavailable. Based the ranking score, RL agent will choose the top-N items to form a list N₁, and recommend them to the user. As feedback to the agent, the user provides the actual rating for each recommended item. Consequently, the evidential reward is
(St, at)
(rating g..i + Σ (x(oss)) KEN.
(3)
N
K.N
R
where rating..; is the user assigned rating, is the threshold to identify if a user provided rating is positive. Evidential reward aggregates the recommended items' rating as a traditional reward r balanced with their vacuity predictions as measure of information gain, denoted as uncertainty regularizer R. During testing, for item i' not appearing in user u's interaction history Hy, a neutral rating rating..:' = will be assigned to give neutral feedback.
Remark: The novel use of vacuity, which is an evidence-based second-order uncertainty, for exploration in RL, can effectively identify uncertain and informative items (from a large item space), indicative of users' long-term interest. In particular, the proposed evidential reward encourages the RI. agent to recommend items that the model has the least knowledge (as indicated by a high vacuity). After collecting the user feedback, the RL agent can most effectively gain knowledge of the user preference to make better recommendations in the long run. It should be noted that maximum entropy-based exploration, such as soft-actor-critic (SAC) (Haarnoja et al., 2018), may not reach an optimum policy. It has been shown that a high entropy may imply either high vacuity (lack of evidence) or high dissonance (conffict of strong evidence) (Shi et al. 2020). However, dissonance is not effective for exploration in RS due to its focus on confusing items mostly derived based on the users' current interests. We have also empirically shown this in qualitative study by demonstrating better recommendation performance than SAC-based exploration in the experiments.
4.2 SEQUENTIAL STATE ENCODER
specially designed sequential state encoder (SSE) used maintain the state space of a dynamic RS environment. A state s; is generated by aggregating the previous state s,-1, items newly interacted by the user as the sliding window moves, and newly recommended items. By aggregating all this information, the current state can evolve from the previous state by concatenated user embedding u, which effectively captures the past preference and future predicted preference of the user. In particular, sliding window W, of length 2N starts from the beginning of the user's interaction history (H₂) at time step 0, then it moves forward along the interaction history N items each time as the newly observed items. The first N already observed items in the moved window W141 will be replaced by the top-N recommendation list at last time step t, the rest will be the newly observed items in time step + 1. Here, item means embedding vector that is generated by a pre-trained
5

---

Published as a conference paper at ICLR 2025
Word2Vec network to encode item information from raw text descriptions. Then, is formed by
S = SSE(s Uy)-
We train the SSE by optimizing action a, to maximize the conservative evidential Q-value in the critic network, which gives Equation (). where 5 (0, v) is the loss objective of the action network.
VuJesn(w) = (
4.3 CONSERVATIVE EVIDENTIAL ACTOR CRITIC (CEAC)
CEAC performs conservative off-policy training using three key networks: action network, critic network, and evidence network, which will be detailed next.
Conservative off- formulation. RS models aim to leverage offline interactions to predict users' future interests. Since the interaction data is inherently sparse, it may train an RL model that overfits the limited training data, leading to overestimated Q-values of previously unseen interactions. To address the data scarcity and sparse rewards in a typical RS environment, we apply an off-policy learning scheme to promote the reuse of previously collected data and stabilize the training. To further prevent overestimation the policy value, we extend the conservative Q-learning strategy developed for off-line RL settings (Kumar al. (2220) and leverage Q-value estimation to avoid over-optimistic predictions on unknown items' ratings and information value as defined in 33. Meanwhile, we increase the Q-value estimate of the behavior policy vg that encodes the current knowledge of user interests to avoid the target policy from deviating too much from previous knowledge. These conservative regularizers are balanced with the traditional Bellman training objective using a RS environment T time steps for one user u as one RL episode. In each time step i, the RL. model collects training tuples (st, a2, 1t++)) into replay buffer D. We iterate M such RL episodes and then conduct a conservative evidential Q-value Q evaluation of the currently learned policy by minimizing its Q-value while maximizing the Q value of the behavior policy, as given in (6), where is the parameter of critic network to minimize the loss objective Jqq:
penalize the target updated policy w's
In particular, we run the behavior policy in the
1))+
Jag (f) = E( S
(Q(s., a.) - ETT (ss a,))² + (Em-DDa Q(s{, a:)] 2
(ernDDaarraalla)) a()))
(6)
After conservative policy evaluation, conduct conservative policy improvement by optimizing the policy towards the optimal conservative evidential Q-value Q: objective as detailed in the action network. After policy improvement, we get a newly learned policy T and again alternate between the policy evaluation and improvement steps until convergence, Once we get a converged policy, we update the previous behavior policy Ty with the newly learned stable policy and begin the next M RL episodes by collecting new training tuples into the replay buffer with the updated behavior policy. After iterating among all users in the training set, we call an RL epoch. The detailed conservative off-policy formulation is shown in Algorithm Dof Appendix
Action network. The action network (or policy network) utilizes the current state S, from the offline replay buffer and outputs a policy distribution (-([ss), which is modeled Gaussian. From this distribution, we sample an action a that is used in the evidence and the critic networks to provide recommendations or direct the policy update. According to our off offlcc formulation, we use two separate action networks to represent the currently updated policy and previous behavior policy, respectively. The training of action network given by
VoJ=(e) V., (s:, a:)) x VoT
where @ is the parameter of the action network to minimize the loss objective J=
Critic network. The critic network is designed to approximate the conservative evidential Q-value by utilizing the current state s, and action a, in a fully connected neural network (s,s, a.). This Q-value judges whether the agent-generated actions match our training requirements. We derive an update formulation (6) for the critic network following the double DQN (Hasseli) 2010) that utilizes two critic networks to stabilize the training process, achieve faster convergence, and provide a better Q-value. Furthermore, the Q-network is optimized with stochastic gradient descent which back pook-propsgatts the action network as well as the sequential encoder in an end-to-end fashion.

---

Published as a conference paper at ICLR 2025
Evidence network. The evidence network predicts Dirichlet distribution of class probabilities, which can be considered as an evidence-collection process. The learned evidence e; = (es1,... εxk) is informative to quantify the predictive uncertainty of recommended items. The network takes action a, from the replay buffer and item pool I to provide class-level evidence. Then, the probability of rating & is Pit (esk + 1)/S,. To train the evidence network, we define standard evidential loss (8) by utilizing the MSE loss between rating class probability Pit and the one-hot ground truth label
Yz, in which yik 1 if is the correct rating, otherwise yik
J (e)
Σ Σ Yik Pst)2
Put(1 Pik)
(8)
S +
iEH,
We update the network by proo-paapggg the evidential loss (EE((V) with its parameters th.
4.4 CONSERVATIVE EVIDENTIAL POLICY ITERATION
We first highlight some important concept differences resulting from our novel evidential conservative setting: 1) is actually an evidential policy governed by an evidential reward r, and 2) T is the true behavior policy directly available from the off off-policy setting without any simulation from offline data. Resulting of these two key symbol differences, the empirical Bellman operator 8Tt should also be an evidential Bellman operator, where is an updated evidential policy. However, to leverage the existing theorems in relevant RL literature which are derived under the traditional RL concepts, make the evidential terms explicit by restoring: 1) the evidential reward rf to R, where R is the vacuity term in our case, and 2) evidential policy π to non-evidential target policy т. As a result, the empirical Bellman operator returns to its non-evidential version 8 and T return back to non-evidence guided behavior policy 7g. Based on these newly defined concepts, we first prove conservative evidential policy evaluation with Q-value updated in (6). In Appendix C.2) we further show that such policy evaluation leads to Ç-safe policy improvement over the behavior policy. Then, we disclose its intrinsic relationship with an uncertainty-awant adjustment of optimism towards the previous behavior policy.
Lemma 1 (Conservative Evidential Policy Evaluation). Given a policy मे and its conservative evidential Q value estimation Qr updated using (6) the expected conservative state value estimation V (s) lower-bounds the actual state value v (s) for any state s when the balancing factor
(*(s)
Cr.T,S
1
( (a(as)) - #T(aas))²
2
max BED
1 1-7
D(s)
(a(s)
Remark: Due to the interaction between the evidence-basee uncertainty in the evidential reward defined in (2), it leads to more strict constraint (through the balancing parameter a) in conservative policy evaluation to avoid risky recommendations when performing evidence based exploration. This one critical difference from standard conservative Q-learning (Kumar al. 2020), which does not consider evidential exploration. The detailed notation definition and proof are in Appendix C.I
Theorem 2 (Uncertainty Aware Optimism Adjustment). The conservative evidential Q-value update in (6) has an intrinsic relationship with an umcertainty-avare adjustment of optimism towards the behavior policy by representing the evidential reward r with traditional reward added by an uncertainty regularizer R, thus leading to a normal conservative Q-value update:
((*(a,))) (B* 2
Qu+1 f- min FRarDDD G²
(10)
+ E BBn
Q* (a,s)) - (EA-D (b) (1b) (1+ (a)*aaa)))) + const
The theorem manifests another novel interplay between the evidential uncertainty and the conservative policy update. From (10), we observe that the uncertainty regularizer IR serves as an importance weight for the Q-value estimation of the behavior policy Rg. This observation provides us with elegant interpretation of the proposed ECQL algorithm: by adding the evidence-busee based uncertainty measure, we are actually adjusting the optimism of the previously learned behavior policy Te according to the information gain (quantified by the vacuity) of a chosen action. If an action generated by a behavior policy leads to low information gain in the recommendation list (i.e, low average vacuity), then the importance weight for the Q-value of such behavior policy is lower and the ECQL algorithm will discourage the RL agent to keep imitating the behavior policy. We leave the detailed derivation in Appendix
Remark:
7

---

Published as a conference paper at ICLR 2025
Table 2: Recommendation performance comparison (average precision P@N, N, discounted cumulative gain nDCG@N N and test return R R@T)
Category
Model
Moviel
P85
Movial TMK
BIRGGS
P85
Necllix BIRGES
RC16
P95
Valool Music
RDOGeS
CASER
0.3762
0.4h13
MA
0.5435
0/4428
N/A
0.5633
0/4522
N5A
0.5745
0.4.15
NIA
SASKec
0.6058
024882
N/A
0.5024
0.4515
N/A
0,5958
0.4021
N7A
0.5826
0.4422
NAA
Sequental
ADFM
0.5749
0,4642
N7A
0.5387
0.467
NIA
0.5688
0.4412
N7A
0.5541
0/4218
N/A
0.5877
0.463
0.5739
0.44%
NIA
0.414
HPMN
0.5962
0.5550
N/A
0.5862
0,4574
NiA
0.5755
0,4244
BEXL4Rcc
0.6122
0.4957
NA
0,282
D-4001
N5A
0,5901
0,4522
Seg25gg
0.5818
0.7522
NIA
05572
0.618
0.5644
0-455
NIA
0.5762
0,3322
S*-Rce
0.610K
0.4926
NIA
015792
[1#20]
V/V
0:5884
1-4000
NIA
0.5786
0.4358
N/A
CL4SR
0,6135
0,4952
N/A
0,5811
A/A
0.
0.5000
0/4688
NIA
0.5841
0.4423
N/A
Dynamic
CKF
0.534
0.5567
0.4481
MW
MA
0,322
0.115
N/A
N/
0/228
NIA
0,5078
0A155
NIA
Deep Lecaning
DoepFM
DXNv2
0.6152
0.3647
0,4625
0.3167
AMA
MM
0.1158
05593
05166
115538
N/
N/A
03335
0.7744
0:4594
0.400
5000
NMA
VIA
0,5718
0:217
0.479
0,224
N/A
N/A
0.6191
0.5205
0.521
38.13
0:544
0.5849
0.8866
33.81
Bandit
0.6088
0.5205
0,612
0,5270
0.5819
0.5813
0,455
CoLin
0.5236
58.87
06227
0,5295
35.49
055355
0.509
58.13
0.5875
0,894
ggroody
0.3977
02434
33.15
0.330
1.2356
$2274
0.5530
014763
55.14
0.999
0/3122
31.37
SAC
0.6105
0.5215
61.63
0,6288
0.5277
35.78
0.5942
0.4839
57.82
0.0074
0.5075
35.44
DRN
0.6057
0.5199
55.26
0.6154
0.5368
$2.78
0.5836
0.4770
57.58
0,4855
0.5121
33.15
LIRD
0.6238
0,5332
61.85
0.6137
0.5222
35.88
0.6124
0.5214
63.59
0.6193
0.5238
38.19
Reinforee
SAR
0.5938
0.495
60.95
0,5957
0.498
35.05
0.5892
0.5022
62.84
0.5014
37.26
REINGOCE
0.6104
0.5138
0.5294
64.25
0.6174
0.5238
ResAct
0.6158
0.5215
67.34
0.6195
0.5234
38.96
06675
0.5319
64.67
0,615
0.5388
39.94
ILAC
0.5989
0.5005
61.28
0,6021
0.5098
38.55
0.5966
0.5138
63.14
0.6009
0.5051
38.18
SAS-SQN
0,5682
0.4755
57.09
0.571
0.4775
34,89
0.5704
0.4852
00.32
0.5765
0.4355
34,96
ECQL
0.6313
0,5365
68.54
0.6379
0.5386
42.31
0.6336
0,5372
70.19
0.6232
0.5330
43.59
EXPERIMENTS
We conduct experiments on multiple real-world datasets: Movielens- Movielens--00R Netflix,
and Yahoo! Music. For baselines, we dynamic models: timeSVD++ (Koren 2009). CKF
(Gultekin & Paisley 2014); sequential models: CASER (Tang & Wang (0118). SASRec (Kang
McAuley| (0188) ADFM (Li al., SD2M) (Cao et al.] 2022).HPMMM (Ren et al. 2019)
BERT4Rec (Sun et al. 2019). Seq2Seq (Ma et al. 2020), 5 (Zhou et al. 2020). CLASRec (Xie
ct al. 2022;; Deep Learning models: DeepFM (Guo et al. (017). DCNv2 (Wang et al. 2021):
Bandit models: Lin (Li et al. 2010a)). HLin (Wang et al. 2016), CoLin (Wu et al. 2016]; and
RL-bbased models: e-greedy (Zhao et al., 2013), SAC (Haamojj et al. 2018), DRN (Zheng et al.
2018), LIRD (Zhao et al.] 2017). SAR (Antaris & Rafailidis 2021). REINforCE (Chen et al. 2019
ResAct (Xue et al. (022). HAC (Liu et al., 2023). SAS-SQN (Silva et al. 2024). Further details
about datasets, metrics, settings, and baselines are in Appendix
5.1 EVALUATION METRICS
We use three standard metrics: Precision@) N and nDCG @ N to measure the average recommendation performance across all time steps. We also use test return R @T the sum of the average rating of all recommended items until time step T for the RL-based methods comparison. For any particular test user, Positive Count, and Genre Count are directly applied measure the qualitative performance. We also consider a standard offline evaluation metric Normalized Capped Importance Sampling (NCIS) (Swaminathan & Joachims| 2015) for RL comparison.
Precision@ @:: It is the fraction of the top-N items recommended in each step of the episode that are positive (i.e,, rating > 7) to the user. We average overall test users as the final precision. nDCG@) :N: Normalized Discounted Cumulative Gain (nDCG @ N) measures ranking quality, considering the top-N recommended items' information gain normalized by top-N items of the ideal ranking list based on the ordered ground truth rating in each step of the RL episode. We average overall test users as the final precision.
Test Return: We evaluate test rewards based on the average ground-truth ratings of top-N recommended items in each step within an RL episode. Test Return considers the sum of the test
rewards before any step within an RL. episode and averages across all test users.
Positive Count: It is the total count of the top-N items recommended in each step of the episode
that are positive (i.e., rating > 7) a particular test user.
* Genre Count: It is the total genre count of the top-N items recommended in each step of the episode to a particular test user.
NCIS: It is variant of average cumulative reward, weighted by the normalized capped importance sampling score based on the policy probability, formulated in E
8

---

Published as a conference paper at ICLR 2025
SEINEUCE
Test Return
Test Retturn
Return
60
Test Return
30
44
$0
20
10
10
10 12 14 16
10
10
10
Steps
Stops
Steps
Stops
(a) Movielens- IM
(b) Movielens- 100K
(c) Netflix
(d) Yahoo! Music
Figure 3: Test return for ECQL and other RL-baselines across different time steps
Count 5 1
SAC
Positive Count
P
Canatt-Cull
SAC
Genre Count
ECOL
Postuvvv Count
ECOL
COL
u G
(a) Genre Count
Genra
(b) Positive Count
Steps
(c) Genre Count
(d) Positive Count
Steps
Figure 4: Genre and Positive Count comparisons with RL models using bandit-based (CoLin) and
entropy-bane -based (SAC) explorations (a-b) well EQL without conservative learning (c-d)
5.2 RECOMMENDATION PERFORMANCE COMPARISON
Table 2 summarizes the recommendation performance from all models. The proposed model benefits from both the sequential state encoder and CEAC module so that it provides better results in all datasets. Sequential models achieve less ideal performance due to their focus on short-term user preference, which less effective in capturing long-term interest. RL methods have shown clear advantage due to their focus on maximizing expected long-term rewards. By leveraging the vacuity guided exploration, ECQL achieves the best performance among all RL based models.
To further demonstrate ECQL's superior performance, we con- duct an RI. baseline comparison on an offline evaluation metric NCIS, which measures the nor- malized capped importance sam- pling score weighted cumulative reward along the whole trajec- tory across all test users. As Ta- ECQL achieves sig- nificant better performance, i.e., higher normalized average cumu-
Table 3: RL comparison on offline evaluation metric Normalized Capped Importance Sampling (NCIS).
Model MovieLens-IM
MovieLens-100X
Netlix
Yahoo! Music
e-greedy
3.088
3,029
3.154
3.135
SAC
3.248
3,165
3.324
3.258
DRN
3.035
2,983
3.102
3.084
LIRD
3.545
3,478
3.629
3.569
SAR
3.835
3.765
3.928
3.843
REINforCE
3,926
3.845
4.024
3.939
ResAct
4.035
3.980
4.126
4,083
HAC
3.852
3,784
3.946
3.891
SAS-SQN
3.659
3.598
3.754
3.690
ECQL
4.124
4.112
4,223
4.178
lative reward than all the RL competitors, As can be seen, the test return for ECQL and the RL-based baselines, including DRN, LIDR, and CoLin (other models will be discussed in the ablation study), are quite close in the initial steps. But later steps, ECQL clearly outperforms the other baselines. This is because the model is able to explore more effectively during the training process enhance the knowledge the model. The effectiveness vacuity guided exploration will further investigated in our ablation study as detailed next.
5.3 ABLATION STUDY
We first investigate the effect of exploration of ECQL by comparing with two alternative designs: (1) w/o vacuity (denoted as Epsilon) and (2) w/o conservative learning guided exploration (denoted as EQL). We also compare exploration using the first-order uncertainty, employed by soft-actor-crins (SAC). 3 shows that without eetertainty-guidee exploration, Epsilon collects the least test return in the long run. SAC utilizes entropy- based exploration and achieves better test return than without uncertainty-guided exploration, almost equal to the performance of strongest reinforced baselines LIRD, REINfOrCE, and ResAct, across four different data sets. This provides evidence that the role of the exploration is crucial RL-based recommendations. However, its performance is worse than the vacuity- based ECQL method. This is because vacuity guided exploration allows our model to explore the most informative items so that can gain the most knowledge to form an optimal policy. However, given the sparse rewards, it is essential that the recommendation does not

---

Published as a conference paper at ICLR 2025
Table 4: Recommended movies for an example user
Model
Movies
Movie Genre
Vacuity
Kids of the Round Table (1995)
Adventure. Fantayy
0.12
Postino, (The Postmun) (1994)
Adventure, Romance
0.11
ECQL
How to Make an American Quilt (1995)
Drama
0.14
Pocahontas (1995)
Musical
0.12
Three Lives and Only One Death (1996)
Comedy
0.22
Karate Kid, Part II. The (1986)
Fantasy
0.07
Retum the Pink Pumther, The (1974)
Comedy
0.12
SAC
I Lawnmower Man 2: Beyond Cyherspace (1996)
Sci-Fi,Trriller
0.09
Young Sherlock Holmes (1985)
Adventure
0.11
Love in Bloom (1935)
Romance
Ruling Class, The (1972)
Tragody
0.14
Private Benjamin (1980)
Comedy
0.13
EQL
Mighty Joe Young (1998)
Adventure
0.08
Christmas Vacation (1989)
Action
0,14
Father of the Bride Part (1995)
Sci-Fi
0.14
deviate too much from user's interest. As shown, EQL performs worse than ECQL by clear margin, especially in the later steps when the model gets richer knowledge about the user through interactions.
Table 5: Impact of each module
We further evaluate the impact of the two key modules of ECQL: SSE and CEAC. We report P@5 and nDCG@5 on Movielens- 1M in Table [5] The integration of the two modules significantly outperforms each individual module. Additional ablation studies are presented in the Appendix.
SSE
CEAC
P@5
nDCG@5
0.4899
0.4175
X
0.6044
0,5108
0.6313
0.5365
5.4 QUALITATIVE STUDY
Impact of vacuity for exploration. We conduct a qualitative analysis to show the advantage of using evidence- based uncertainty (i.e., vacuity) for RL exploration when compared with other two competitive baselines: entropy-guides exploration as in the soft actor-critic (SAC) and contextual bandit algorithm (CoLin). We select random test user from the Movielens- 1M dataset and show the genre counts and positive counts of recommended items in Figure (() and (b). At the initial few steps, SAC has more positive counts but less effective in exploration. The proposed ECQL is able to explore more informative items (evidenced by more diverse genres). In later steps, ECQL consistently outperforms both SAC and CoL in positive counts due the better utilization of evidence-basee uncertainty to discover more informative future items that could reflect the user's long-term preference, as shown in Figure (b). Table shows the predicted vacuity for each recommended item in step 10. The overall higher vacuity scores indicate that ECQL recommends more items that are currently unknown to the users, which is instrumental in exploring their long-term interests. ECQL also explores more diverse genres and identifies four out of five important items (genre types in bold) that are positive in ground-truth ratings. Benefiting from better exploration, ECQL eventually achieves a much better test return compared to SAC and CoLin shown in Figure 3 More detailed discussions about how to control the exploration through hyper- hyper-parameter parameter included in Appendix E.4
Impact of conservative learning. The last two plots in Figure + show that comparing to EQL, which lacks conservative constraint, our proposed model achieves much higher positive counts across different steps while exploring diverse genres. Because of the sparse reward space and limited training data, making reasonable and safer recommendations based current confirmed knowledge is important to maintain a user base. As shown in Table EQL explores a wide range of different types of movies, but only one of them is positive reflecting user's true interest.
CONCLUSION
In this paper, we propose a novel evidential conservative Q-learning framework for dynamic recom- mendations. The proposed ECQL framework learns an effective and safe recommendation policy by integrating both the evidence- based exploration and conservative learning. ECQL. integrates a customized SSE to generate the current state that accurately captures module which functions as an RL agent to perform evidence-based exploration and trained through conservative evidential Q-learning in an off-policy formulation. We theoretically prove that the conservative evidential Qavalue provides an uncertainty- towards the behavior policy and lower- lower-bounds the actual value of the target policy.
interest and a conservative
cvidential-actor-e
adjustment of optimism
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Stefanos Antaris and Dimitrios Rafailidis. Sequence adaptation via reinforcement learning in recommender systems. In Proceedings of the 15th ACM Conference on Recommender Systems, pp. 714-718, 2021.
Peter Auer and Ronald Ortner. Logarithmic online regret bounds for undiscounted re- inforcement learning. In B. Schölkopf. J. Platt, and Hoffman (eds.). Advances in Neural Information Processing Systems. volume 19. MIT Press, 2006. URL httpps:/prrceeddini ngs wwwrinu.riiccmm www.tps.cc..pppprrrr......eeeee. filesseer_ele//eeeeeeeeeeeeeeeer files/ c1b7 1ll1:o0055aaS00eeaaaccaaaaaaaae. htt.pppppprr.lleeeepp////////// paper/2006/f1le.
pdf
Peter Auer, Thomas Jaksch, and Ronald Ortner. Near-optimal regret bounds for re- inforcement learning.
D. Koller, Schuurmans, Y. Bengio, and Bottou (eds.), Advances in Neural Information Processing Systems, volume 21. Curran Asso- ciates, Inc., 2008, URL httpss//ppooceedggg. http://g.........../ppp///////// e26622cdb33337500000000838888820
Paper. pdf
Xueying Bai, Jian Guan, and Hongning Wang. Model-based reinforcement learning with adversarial training for online recommendation. arXiv preprint arXiv:1911.03855 2019.
James Bennett, Stan Lanning, et al. The netflix prize. In Proceedings of KDD cup and workshop, volume 2007, pp. 35. New York, NY, USA., 2007.
Rianne van den Berg, Thomas N Kipf, and Max Welling. Graph convolutional matrix completion. arXiv preprint
2017.
Jesus Bobadilla, Abraham Gutierrez, Fernando Ortega, and Bo Zhu. Reliability quality measures for recommender systems. Information Sciences, 442:145-157, 2018.
Djallel Bouneffouf, Amel Bouzeghoub, and Alda Lopes Gançarski. Risk-aware recommender
systems. In International Conference on Neural Information Processing, 2013. URL https
//api sem.ntttttiii. www.hhoaa..rrrgoooooppoo
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprini 1rXi::1810./284/ 2018.
R. Burke. Hybrid recommender systems: Survey and experiments. User Modeling and User- Adapted Interaction, 12:331-370, 2002. URL httph:t/////// //api. www.snt.cschhlar.ooooo
CorpusID:3973
Yue Cao, Xiaojiang Zhou, Jiaqi Feng, Peihao Huang, Yao Xiao, Dayao Chen, and Sheng Chen. Sampling all you need on modeling long-term user behaviors for etr prediction. In Pro- ceedings of the 31st ACM International Conference on Information & Knowledge Management, CIKM '22, pp. 1974-2983 New York, NY, USA, 2022. Association for Computing Machinery. ISBN 450392333 doi: 10. 10.1145/35111088335555 3557082. URUhttps::/dol.or////
3511808. 3557082
Laurent Charlin, Rajesh Ranganath, James McInerney, and David M Blei. Dynamic poisson fac- torization. In Proceedings of the 9th ACM Conference on Recommender Systems, pp. 155-162, 2015.
Minmin Chen, Alex Beutel, Paul Covington, Sagar Jain, Francois Belletti, and Ed Chi. Top-k off-policy correction for reinforce recommender system. In Proc. of WSDM, 2019.
Heng-Tze Cheng, Levent Koe, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, al. Wide & deep learning for recommender systems. In Proceedings of the 1st workshop deep learning for recommender systems, 2016.
Sergio Taggao, Juan M. Fermández-Lune Juan F. Huete, and Nava Tintarev. Being confident about the quality of the predictions in recommender systems. In Pavel Serdyukov, Pavel Braslavski, Sergei O. Kuznetsov. Jaap Kamps, Stefan Rüger, Eugene Agichtein, llya Segalovich, and Emine Yilmaz (eds.). Advances in Information Retrieval, pp. 411 411-422 1-42,, Berlin, Heidelberg, 2013. Springer Berlin Heidelberg. ISBN 978-3-642-36973-5.
11

---

Published as a conference paper at ICLR 2025
Arthur F. da Costa, Marcelo G. Manzato, and Ricardo J. G. B. Campello. Corec: co-training approach for recommender systems. SAC '18, pp. 696-703, New York, NY, USA, 2018. Associa- tion for Computing Machinery. ISBN 19814. doi: 10.1145/1667132.3167729 URL
ht.tps://dot wtt.:/..rgg11..1 114. 145 31 6 32 31 67209
Gideon Dror, Noam Koenigstein, Yehuda Koren, and Markus Weimer. The yahoo! music dataset and kdd-cup' 11. In Proceedings of KDD Cup 2011, pp. 3-18. PMLR, 2012.
Mehrdad Farajtabar, Yinlam Chow, and Mohammad Ghavamzadeh. More robust doubly robust off-policy evaluation. In Proc. of ICML, 2018
Simon Funk. Incremental svd method. online, 2006. URL http::://llferr.rggg/// Jouzna1/20051211.2
San Gultekin and John Paisley. A collaborative kalman filter for time-evolving dyadic processes. In Proc. of ICDM, 2014.
Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He. Deepfm: a factorization- machine based neural network for etr prediction. arXiv preprint TrXhv:1703.04247 2017.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, 2018.
Hado Hasselt. Double q-learning, In Proc. of NeurlPS, 2010.
Audun Josang, Jin-Hee Cho, and Feng Chen. Uncertainty characteristics of subjective opinions. In 2018 1st International Conference on Information Fusion (FUSION), 2018.
Audun Jsang. Subjective logic. In Subjective Logic, 2016.
Wang-Cheng Kang and Julian McAuley. Self-attentive sequential recommendation. In Proc. of ICDM, 2018.
Jens Kober, J Andrew Bagnell, and Jan Peters. Reinforcement learning in robotics: A survey. The International Journal of Robotics Research, 2013.
Yehuda Koren. Factorization meets the neighborhood: multifaceted collaborative filtering model. In Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 426-434, 2008
Yehuda Koren. Collaborative filtering with temporal dynamics. Proc. of KDD, 2009.
Yehuda Koren, Robert Bell, and Chris Volinsky. Matrix factorization techniques for recommender systems. Computer, 42(8):30-37, 2009.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances Neural Information Processing Systems, 33:1179-1199 2020.
Lihong Li, Wei Chu, John Langford, and Robert E. Schapire. contextual- bandit approach to
personalized news article recommendation. CoRR, abs/1003.0146 2010a. http:t/://arxi
org/ hot//bb//a 1003.00000 0146
Lihong Li. Wei Chu, John Langford, and Robert E Schapire. A contextual-bandit approach to personalized news article recommendation. In Proceedings of the 19th international conference on World wide web, 2010b.
Xiaochen Li, Jian Liang, Xialong Liu, and Yu Zhang. Adversarial filtering modeling on long- term user behavior sequences for thick-through rate prediction. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 222 pp. 1969-1973, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 478145038 doi: 10.114553477445533337788 3477495. 3531788
ht https: http///....//0.11......./1
12

---

Published as a conference paper at ICLR 2025
Shuchang Liu, Qingpeng Cai, Bowen Sun, Yuhao Wang, Ji Jiang, Dong Zheng, Peng Jiang, Kun Gai, Xiangyu Zhao, and Yongfeng Zhang. Exploration and regularization of the latent action space in recommendation. In Proceedings of the ACM Web Conference 2023, pp. 833-844--40000008008888188888810 844, 2023.
Jianxin Ma, Chang Zhou, Hongxia Yang, Peng Cui, Xin Wang, and Wenwu Zhu. Disentangled self-supervision in sequential recommenders. Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2020.
Maciej A. Mazurowski. Estimating confidence of individual rating predictions in collabora- tive filtering recommender systems. Expert Systems with Applications, 40(10):3847-3857 2013. ISSN 0957-4174. doi: http://o..rrr/01..0.oo.s.ss..... / www.wwww.............ww.w.www www.ssseeeceiiiccc rect. com
URL https
Sean M. McNee, Shyong K. Lam, Catherine Guetzlaff, Joseph A. Konstan, and John Riedl. Con- fidence displays and training in recommender systems. In IFIP TC13 International Conference Human- Caamann-Cmmpter Interaction, 2003. URL Itttt::////// wtw..p..settntcccc..aa.a..am
CorpusID:179533
Fernando Ortega, Raúl Lara- Cabarra Ángel González-Prieto, and Jesús Bobadilla. Providing relia- bility in recommender systems through bermoulli matrix factorization. Information Sciences, 553: 110-128, 2021. ISSN 0020-0255. doi: htt.://cw......o..oo.o......../a
2000.........222200...22000 12.001. htthttsss:
7/ 7/www. www.sscieneeireee.oo www.eetttcocmeeee
lan Oshand, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 29, Curran Asso- ciates, Inc., 2016. 2016/fi1e/ 2000///1//6611aa1166/77135111100
fi www.:eeurr......peeeeeeeec pdd
Brendan O' Donoghue. Variational bayesian reinforcement learning with regret bounds. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Ad- vances in Neural Information Processing Systems, volume 34, pp. 28208-28221. Curran Asso- ciates, Inc., 2021. /ttph//tppsssppppppeessssssrrrrr 20277//1110441111663232222222220
wwwppprrrrcppppprppppppppepe..ee fi lellesppprr
pdf
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International conference machine learning, pp. 2778-2787.
PMLR, 2017.
Kan Ren, Jiarui Qin, Yuchen Fang. Weinan Zhang, Lei Zheng, Weijie Bian, Guorui Zhou, Jian Xu, Yong Yu, Xiaoqiang Zhu, and Kun Gai. Lifelong sequential modeling with personalized memorization for user response prediction. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR' 19, pp. 565-574, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 450366179999 11..0..11433333333 3331223
doi:
00.1145/3311184.333111
/RL/attss://ao..oo wtt.:/o0o/1...1155333333
Steffen Rendle. Factorization machines. In Proc. of ICDM, 2010.
Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme. Bpr: Bayesian
personalized ranking from implicit feedback. arXiv preprint arXiv: 12X.xv:205.221111 2012.
Nachiketa Sahoo, Param Vir Singh, and Tridas Mukhopadhyay. A hidden markov model for collabo- rative filtering. Mis Quarterly, pp. 1329-1356, 2012.
Weishi Shi, Xujiang Zhao, Feng Chen, and Qi Yu. Multifaceted uncertainty estimation for label- efficient deep learning. Advances in Neural Information Processing Systems, 2020.
Álvaro Labarea Silva, Denis Parra, and Rodrigo Toro Icarte. On the unexpected effectiveness of reinforcement learning for sequential recommendation. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 45432 45450. PMLR, 21-27 Jul 2024. URL https:// proceedi ngs..... ml mlr. www.spressss333 s va24b html
13

---

Published as a conference paper at ICLR 2025
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou. Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 2017.
Xiaoyuan Su and Taghi M Khoshgoftaar, A survey of collaborative filtering techniques. Advances in artificial intelligence, 2009.
Fei Sun, Jun Liu, Jian Wu, Changhua Pei, Xiao Lin, Wenwu Ou, and Peng Jiang. Bert4rec: Sequential recommendation with bidirectional encoder representations from transformer. In Proceedings of the 28th ACM international conference information and knowledge management, pp. 1441-1450, 2019.
John Z Sun, Dhruv Parthasarathy, and Kush R Varshney. Collaborative kalman filtering for dynamic matrix factorization. IEEE Transactions on Signal Processing. 2014.
Adith Swaminathan and Thorsten Joachims. The self-normalized estimator for counterfac- tual learning. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015. URL httpss://prooeedings.so neurips. cc/paper fil www.fee/pppeee/211/55.. Ei F1le:3300070ae11133cc00007777710 paw.err--apppppp.ppe
Jiaxi Tang and Ke Wang. Personalized top-n sequential recommendation via convolutional sequence embedding. In Proc. of WSDM, 2018.
Huazheng Wang, Qingyun Wu, and Hongning Wang. Learning hidden features for contextual bandits. In Proceedings of the 25th ACM International on Conference on Information and Knowledge Management, CIKM '16, pp. 1633-1642, New York, NY, USA, 2016. Association for Computing Machinery. ISBN 9781450340731. doi: 011145//983323.2983844 URL
Ruoxi Wang, Rakesh Shivanna, Derek Cheng, Sagar Jain, Dong Lin, Lichan Hong, and Ed Chi. Den v2: Improved deep & cross network and practical lessons for web-scale learning to rank systems. In Proceedings of the Web Conference 2021, pp. 1785-1797, 2021.
Xiang Wang, Xiangnan He, Meng Wang, Fuli Feng, and Tat-Seng Chua. Neural graph collaborative filtering. In Proceedings of the 42nd international ACM SIGIR conference on Research and development in Information Retrieval, pp. 165- 174, 2019.
Qingyun Wu, Huazheng Wang, Quanquan Gu, and Hongning Wang. Contextual bandits in a collaborative environment. In Proceedings of the 39th International ACM SIGIR conference Research and Development in Information Retrieval, pp. 02955529-555 508,5388280 2016.
Fenfang Xie, Liang Chen, Yongjian Ye, Zibin Zheng, and Xiaola Lin. Factorization machine based service recommendation on heterogeneous information networks. In 2018 IEEE International Conference on Web Services (ICWS), 2018
Xu Xie, Fei Sun, Zhaoyang Liu, Shiwen Wu, Jinyang Gao, Jiandong Zhang, Bolin Ding, and Bin Cui. Contrastive learning for sequential recommendation. In 2022 IEEE 38th international conference data engineering (ICDE), pp. 1259-1273 IEEE, 2022.
Wanqi Xue, Qingpeng Cai, Ruohan Zhan, Dong Zheng, Peng Jiang, Kun Gai, and Bo An. Resact: Reinforcing long-term engagement in sequential recommendation with residual actor. arXiv preprint 2006 22220222.000020000 2022.
Xiangyu Zhao, Liang Zhang, Long Xia, Zhuoye Ding, Dawei Yin, and Jiliang Tang. Deep reinforce- ment learning for list-wise recommendations. arXiv preprint arXiv: 1801000209 2017.
Xiaoxue Zhao, Weinan Zhang, and Jun Wang. Interactive collaborative filtering. In Proceedings of the 22nd ACM international conference on Information and Knowledge Management, 2013.
Guanjie Zheng, Fuzheng Zhang, Zihan Zheng, Yang Xiang, Nicholas Jing Yuan, Xing Xie, and Zhenhui Li. Dr:: deep reinforcement learning framework for news recommendation. In Proc. of www, 2018
14

---

Published as a conference paper at ICLR 2025
Kun Zhou, Hui Wang, Wayne Xin Zhao, Yutao Zhu, Sirui Wang, Fuzheng Zhang, Zhongyuan Wang, and JinRong Wen. S3-rec: Self-supervised learning for sequential recommendation with mutual information maximization. In Proceedings of the 29th ACM international conference information and knowledge management, pp. 1893-1902, 2020.
Lixin Zou, Long Xia, Zhuoye Ding, Jiaxing Song, Weidong Liu, and Dawei Yin. Reinforcement learning to optimize long-term user engagement recommender systems. Proc. of KDD, 2019.
Lixin Zou, Long Xia, Pan Du, Zhuo Zhang, Ting Bai, Weidong Liu, Jian- Yun Nie, and Dawei Yin. Pseudo dyna-q: A reinforcement learning framework for interactive recommendation. In Proceedings of the 13th International Conference on Web Search and Data Mining, 2020.
15

---

Published as a conference paper at ICLR 2025
Appendix
Table of Contents
A Summary of Notations
B Additional Related Work
C Proofs of Theoretical Results
C.I. Proof of Lemma I
Proof overview.
Notations.
Proof.
C.2 Proof of Conservative Policy Improvement
20
Proof.
C.3 Proof of Theorem
21
D Evidential Conservative Q-Learning Algorithm
E Detailed Experimental Setup and Additional Results
E.1
Detailed Experimental Sctup
E.2
Training and Test Setting
E.2.
Experimental Setting
E.2.2
Comparison Baselines
E.3
Additional Results
E.4
Additional Ablation Study
Additional Qualitative Analysis
E.6
Training Efficiency
28
F
Limitations, Future Works, and Broader Impact
G Source Code
29
16

---

Published as a conference paper at ICLR 2025
Organization of Appendix. In this Appendix, first summarize the major mathematical notations in Appendix [ We give an overview representative static recommendation models in Appendix B We then present the proofs of lemmas and theorems in Appendix [ We show the detailed ECQL
algorithm in Appendix D We present the details of the datasets, experimental setting and baseline models in Appendix [E] Further,
include some additional comparison results in Appendix [E.3] and
conduct an ablation study as well as a qualitative analysis in [ppendix[E.4]aar and E.S Limitation and
future extension of our work are discussed in Appendix F The link the source code is given in
Appendix G
A SUMMARY OF NOTATIONS
Table 6: Summary of notations
Symbol Group
Notation
Description
T, .
step size or length of episode and episode size for ECQL training
u, i, h,
user (episode), item and item's position indices user u's emhedding at time step state and action at time evidence and evidence-based probability rating class item
U1
Cakes Pis
ECQL
Yi
one-hot rating label on item i
o. xb
parameters of action and evidence networks parameters of critic and SSE networks
8, ((s,, as)
m,
recommendation policy action network, value func- tion or critic network
rating rating
predicted and actual rating for user on item evidential score for user on item
score...
A
rating threshold and balance hyper-parameter for ploitation and exploration
Y
RL discount factor for test return item i'ss evidence-based uncertainty evidential reward and Q value
(((()))))
(= a.), 1 as)
P(s (s,))
state transition probability the number of rating class
K
Environment
W W
number interacted items sliding window and recommended items in the recommendation list cach time step
No.
top-N recommended items list for user u's interaction history and item pool
II
Cv,r,5, Cr,s, Cr.5
constants dependent on the concentration properties (variance) of evidential reward r s) and/or state tran- sition matrix 7 T's''s.. |s, a)
Theoretical Results
a, 6,5
hyper-parameters controlling conservatism, high proba- bility and safe policy improvement guarantec. empirical MDP,
J(R, M), (T,
discounted return a policy M and actual MDP, M.
[D], A),
magnitude of state action space, state distribution from empirical MDP (offline dataset)
DECOL( R8)
ECQL regularizer detailed in C.I
B ADDITIONAL RELATED WORK
To provide a complete review of the existing literature on recommender systems, this section gives an overview of representative static and dynamic recommendation models that complement the sequential and RL-based models discussed in the main paper. We also discuss some other RL exploration strategies and of uncertainty in recommender systems.
17

---

Published as a conference paper at ICLR 2025
Static recommendation models. Matrix Factorization (MF) leverages user and item latent factors
to infer user preferences (Koren et al. 20099; Funk| [0066] Koren 2008). MF is further extended
with Bayesian Personalized Ranking (BPR) (Rendle et al. 2012) and Factorization Machine (FM) (Rendle 2010). Recently, deep learning-based recommender systems (Cheng et al. 2016 Guo et al.
(017) have achieved impressive performance. DeepFM (Guo al. 2017) integrates traditional FM and deep learning to learn low- and high-order feature interactions. The wide and deep networks are jointly trained in (Cheng et al. 2016) for better memorization and generalization. In graph-baser methods (Berg et al. 2017), users and items are represented as bipartite graph and links are predicted to provide recommendations. Similarly, Neural Graph Collaborative Filtering (Wang et al. 2019) explicitly encodes the collaborative signal via high-order connectivities in the user-item bipartite graph via embedding propagation.
Dynamic recommendation models. Dynamic recommendation models shift users' latent preferences over time to incorporate temporal information. TimeSVD++ (Koren 2009) considers time-specific fac- tors, which uses additive bias to model user and item-related temporal changes. Gaussian state-space models have been used to introduce time-evolving factors with a one-way Kalman filter (Gultekin & Paisley 2014). Hidden Markov model (Sahoo et al. 2012) and Poisson emission (Charlin et al., 2015) have also been leveraged to model temporal dependencies. However, these models primarily capture users' evolving preferences from the past that are less effective to predict future interactions.
Additional RL exploration strategy. ICM Pathak et al. (017)) and RND Burda et al. (2018) are designed for complex deep reinforcement learning (DRL) environments (e.g., Atari games) with sparse reward signals, where exploring novel states is critical to uncover rare rewards, penalizing events, or turning points. ICM uses curiosity facilitate exploration, measured by the error in predicting the next state from the current state-action pair, leveraging a forward dynamic model trained on prior interactions. Additionally, employs an inverse dynamic model to filter out irrelevant state representations. Similarly, RND employs fixed, randomly initialized network (random distillation network) as a reference model. predictor neural network trained to replicate the outputs of the reference model, and the prediction error for any new state serves as novelty score. However, directly applying these exploration strategies to dynamic recommendation settings may be less effective, as they are primarily designed for nigh-dimmesional DRL environments, such Atari games. These methods are not tailored to accurately quantify rating prediction uncertainty, which is crucial for encouraging the exploration of novel items in recommendation systems.
Use of uncertainty in recommendation models. Quantifying the uncertainty of a recommender system (RS) Fleger--Tamayo al. (2013); McNee et al. (2003) is important because it helps to reveal which rating predictions or recommended items are likely to be wrong Bobadilla et al. (2018).
Uncertainty estimates improve RS in three main ways:
1. Convey recommendation confidence Cleger-Tamayo et al. (013)) ) McNee et al. (2003). Display- ing uncertainty alongside recommendations can enhance the trustworthiness and credibility of RS. For instance, when the RS is uncertain about a recommendation, providing an explanation can help users understand the rationale behind the suggestion. This approach reduces the risk of confusing or untrustworthy recommendations and improves the overall user experience.
2. Guide which items to recommend Mazurowski (2013); Ortega et al. (2021) y: Bouneffout et al. (2013). Typically, a recommender system (RS) suggests items it predicts to be most relevant to the user. By incorporating uncertainty estimates, the RS can avoid recommending items with high predicted relevance but significant uncertainty, thereby reducing the risk of poor recommendations. Alternatively, the RS could strategically include uncertain recommendations to encourage user discovery of new items and gather valuable feedback for improving future predictions.
3. Optimize recommendation process Burke (2002); da Costa et al. (2018). Uncertainty can also assist recommender system (RS) in optimizing system leverages uncertainty to determine which predictions to include in its training data. More broadly, uncertainty estimates can guide the RS in selecting between different models or strategies. They are also valuable in active learning scenarios, where the system solicits user feedback on items it uncertain about, thereby improving the quality of future recommendations.
internal processes. For example, the CoRec
18

---

Published as a conference paper at ICLR 2025
C PROOFS OF THEORETICAL RESULTS
In this section, we provide detailed proofs of Lemmal] Theorem 2 and Ç-safe policy improvement guarantees over behavior policy by iterating conservative policy evaluation (6 and update 7
C.1 PROOF OF LEMMA 1
Proof overview. We first introduce some important notations that will be used in the proof of the Lemma. Our proof shows that conservative Q-value estimation updated step by step using (12) leads to a lower-bounded state-marginal value estimation compared to its previous iterate. Then, we compute the fixed point of such iteration to conclude that the converged state value estimation V*((s) lower-bounds the actual state value / V*(s). We first conduct the whole derivation tabular Q-learning setting where state transition is finite and precise (Auer & Ortner 2006), then incorporate the sampling error to make it a complete RL process.
Notations. Let k E N denote Q-iteration of policy evaluation. In an iteration k the objective in (6) is optimized using the Bellman backup (i.e., B ( the target value which is given as Equation (IT) by following a double DQN training fashion, where R. is the evidential regularizer, is the discounted factor, Q is the target Q-network representing a stable evaluation and Q is the Q-network representing the current learned policy's Q-value. Let Q* denote the true, tabular Q-function iterate in the actual MDP M, and Q* is estimated using the empirical Bellman operator B as: Q* + 1 Q* (for policy evaluation). We redefine Q: in the main paper as Q* when - oc (i.e., the k-tth Q-function iterate reaches convergence). We emphasize that T is the actual behavior policy directly available from last policy update, a is the
to guarantee conservatism, and T denotes the state value estimation, which is the expectation of corresponding Q-value under a given policy #: Vk = (Q"
B (st, ax) = -B
((r(s) a.) + x min ( (aa-(Q/---Sss (S1+1, ar+ Q1 (S a++1)
(II)
Assumptions. Following (Osband et al.
2016 O' Donoghue 2021 Auer et al. 2008), we use
concentration properties of 8 to control the sampling error. Formally, for all s. E D. with probability 2 1, there exists constant C,,7.8 dependent on the concentration properties (variance)
of reward r(s, a) and state transition matrix T''s'ss, |s, a) that satisfies 18* - (s, a) S V ((s.a)
C.
where 8 E (0, D(s,a))
denotes a vector of size |S||A|| containing square root inverse counts
for each state-action pair, except when D(s, a) 0, in which case the corresponding entry is very large but finite value 8 2Rmas.
Proof. In the tabular setting, and compute the Q-function update induced in the exact, tabular setting, thus giving 212 as one optimization step (this assumes 8= = =B
set the derivative of the modified objective in 66) to 0,
T (aass) #p (a(s)
Vs. (a, Ba 1²02 Q R
(12)
Where R is the expectation the evidential regularizer under the off-policy state-action distribution in fact adding a positive (a(a))) To the Q-function obtained, a point-wise lower bound is not guaranteed, i.e.,
D. Note that for state-action pairs (a, s) such that, ([ass) (als),
quantity, 1 #(ass)
Vs, a, *. 8) < (a, s). However, we show that on the other hand the expected state value of the estimated Q D-fuuction, i.e., Ç*k always lower bounds its previous iterate, since:
(s)(s) Ea-a((as)) (2*(a,)) -B a.
*s(aas) (1
#(als)
R a
(13)
19

---

Published as a conference paper at ICLR 2025
The value of the policy V is underestimated comparing to p=-11, since show that the evi- dential term DECQL(S) = >. $ (als))
is always positive under the condition Fp(a|s) - (1 R
X(als)
that a 2 Ex, aa
(#(a|s) (a(als)-Rpttl -p(n|م) |_1 which is the evidential constraint on conservative parameter a to (a()))
guarantee underestimation in each value update. To note this, we present the following derivation:
((als) *a(a|s)
(1 + R )
DECQL(S) == Σ
Tt Taass)
a
#(aas)
(1 + )
Σ (a(ass) - *s(a|s) + (a(ss)
Ta(a)
T(als) (a|s)
T i(als)
R -)
Σ (a(a))) - (
+ Σ ne (a|s)
+
(a|s)
(a|s)
Σ (T(a|s) (a)s)))
R
0, (R is an uncertainty regularity between 0 to I)
(a(s)
20
which requires a
(14)
(a(als) (mis))?
Σ,
(a(a(s) (a(a)))) Fa(a)s)
(a(a)))
The above derivation implies that each value iterate incurs some underestimation (*((s) S (s) when conservative
a satisfies some constraints. Now we compute the fixed point of
value iteration (13) as the equivalency to v**(s) when k - oc and get the following estimated policy value V* which shows a clear lower bound to actual state value function V (s), given that the
simulation error (T 1 > Er # can be avoided because of our off-policy setting:
TB - (1+ R
T
(s) = V* (s) a &
(15)
Then, by further incorporating the sampling error
(1 - 7 Co Renax 1-7/ DI
(s). we get the follow-
ing full evidential constraint for a, which is more strict constraint compared CQL:
Cr.Tr6o
1
(m(l {a(as))) TT((a())
max SED Vi((s)
C.2 PROOF OF CONSERVATIVE POLICY IMPROVEMENT
Lemma 3 (Conservative Policy Improvement). Given an optimal policy n. that is the fixed point under action optimization using Equation (7). then the policy x² (als) is Ç-safe policy improvement over behavior policy Tg in the actual MDP M. The expected discounted return attained by a policy
n² in the actual underlying MDP M, i.e., J(x", M), ), is guaranteed be higher than that attained by its behavior policy gg with a lowest Ç bound. Formally, we represent it as J(T", M,, 2 J(zg, M) - with high probability
8 where Ç given as:
Cr. +
RBmar Cr,6
VIA
V (F T2))) +1
2=
B Bandy 2, (s)
1 1-9
(1-
DDD(s)
1
E B (s) (DFQU ;,)(())
a
where 8 is a high probability control related to Ç. di is the state distribution from the empirical
MDP AZ. |A], [D] is the magnitude (norm) of action and state spaces. Cr.5 and Cr.5 are constants dependent on the concentration properties (variance) of evidential reward r (a, s) and state transition matrix T(s' |s, a). respectively. 7 is the discounted factor. D DECOL is an evidential conservative factor ensuring the conservatism in policy evaluation.
20

---

Published as a conference paper at ICLR 2025
Proof. In this section, we first show that this policy improvement procedure defined in Equation actually optimizes a penalized RI. objective [TT, M) a -1 Ra-d', (s) DEcQL (ā, (s)( using Lemma D.3.1 following (Kumar et al., 2020). where [TT, .M) is the empirical discounted return of policy æ in empirical MDP M, DECQr. is the conservatism term given in (1), and then we relate the performance of n= (aass) updated with this penalized RL objective, to the performance of itself and its behavior policy a in the actual MDP M under the non-evidential settings so that we can leverage existing Theorem D.4 proposed by Kumar et al. (Kumar et al. 2020), thus gives:
Onu
+ BBmax (1 7)2
TAI
((T,,M) 2 J( -
E Bandd (8)
DRoQu(=":Ta)) (T)(s) +
1- 1
+ a
Ex s- (s) (DEeE as)
(16)
Note that all symbols here are in their non-evidential definitions which follow the same constraint on conservatism parameprr a defined in (14). The explicitly expressed evidential term is contained in DECQL(#* (s)) The proof for this statement is divided into two parts. The
first part involves relating the return of nT in the empirical MDP M with the return of n3 in M. Since, π (als) optimizes the penalized RL objective, it is the best policy under empirical MDP M. and is guaranteed to behave better than the behavior policy R3 in lowest bound governed by a B (s) "EEQULL" [s)) shown in the last term in Equation (6). The next step
involves using concentration inequalities to upper and lower bound J [ M) and J(x M) and the corresponding difference for the behavior policy. According to Kumar et al. (Kumar et al. 2020), they apply Lemma D.4.1 to control such difference with y, Rmax: and Cr, Cr
C.3 PROOF OF THEOREM 2
We introduce the detailed derivative of Theorem 2 as given below:
JQ: (0) min (Ea-n,,a>s (Qaa.s)) Eg-D
((). (s) + EE min a (Fgw (a. ar
1 2
+ = E Bsarr.s (Q(a,s) - (s) + min (1)
2
= -mina
(Bar))- Bearg
Q
()a,s) -
rating
un Σ
Un(eee)) +7
N
N
SEN
K.N.
min a (BEw [Q(a, - (1)0))))
-r (1 a²) Q* (s', (s', a')))
]]
2 Dt
We use 8et Q² (a,s) to replace + y min(""(s', Q* (s', a")} and by some mathematical
reshaping, we get:
2 Ea - 8) ) + aEE t s)]
Joz (0) min Q+
a (B
+
[ 2 + R B (**2((a,s)) (17)
where C is constant relating to the Q-value iterate.
21

---

Published as a conference paper at ICLR 2025
D EVIDENTIAL CONSERVATIVE Q-LEARNING ALGORITHM
We provided detail training procedure for the ECQL method in Algorithm
Algorithm Eyidential Conservative Q-Learning
Require: Hyperparameters: o, X, r, epísode size M and step size
1: Initialize SSE: w, action network: é, evidence network: 1), and critic network: 0 initial state: So and initial user embedding: with W = 2N items in sliding window Wo from interaction history H. and Item Pool
2: for each epoch do for each user as an episode do for t ' do
3:
4:
5:
Compute state: 8, with
6:
Compute action:
(-(ss) Compute evidential score using
7:
8:
Recommend top-N items based computed evidential score to form recommen-
dation list Nu.
9:
Compute rewards based recommendation li, utilizing (3)
10:
Add (Sr, a, r (Sr, ax), S1+1, done) into replay buffer
Move sliding window and take W newly interacted items from H, and replace the
11:
other W items with the top-N items from the recommendation list N..-
12:
end for
13:
if episode index number reaches M then
14:
Sample batched data from replay buffer and forward into the networks.
15:
repeat
16:
Update critic network with (6)
Update action network with (7
18:
Update SSE network with (
Update evidence network with
20:
until converged
end if
22:
end for
23:
end for
E DETAILED EXPERIMENTAL SETUP AND ADDITIONAL RESULTS
In this section, we provide additional details on the experiments, including datasets, setting, evaluation metrics, and comparison baselines. We also report more results, including comparison with additional baselines on two larger scale datasets, ablation study and qualitative analysis.
E. DETAILED EXPERIMENTAL. SETUP
Description of datasets. Besides the four dynamic recommendation datasets as introduced in the main paper, include additional dataset Movielens- 10M evaluate how ECQL performs large-scale recommendation problems. Details of all the datasets are summarized below:
Movielens- M T
This dataset includes 1M ratings provided by 6,040 anonymous users 3,900
distinct movies from 04/2000 to 02/2003. Movielens-100% This dataset contains 100,000 ratings from 943 on 1,682 movies. Each
user at least rated 20 movies from September 19, 1997 through April 22, 1998.
Netflix (Bennett al. 2007): : This dataset has around 100 million interactions, 480,000 users. and nearly 18,000 movies rated between 1998 to 2005. We processed the dataset and selected 6,042 users with user-item interactions from 01/2002 12/2005.
22

---

Published as a conference paper at ICLR 2025
Yahoo! Music rating (Dror et al. 2012): ) The dataset includes approximately 300,000 user-
supplied ratings, and exactly 54,000 ratings for randomly selected songs. The ratings for randomly selected songs were collected between August 22, 2006 and September 7, 2006.
Movielens- 1ippnns--MMP 10M This dataset contains 10,000,054 ratings applied to 10,681 movies by 71,567 users, which released in January 2009. All users selected had rated at least 20 movies.
E.2 TRAINING AND TEST SETTING
For training, given a user interaction history Hu. we continuously capture most recent N items after current time step into a sliding window W,, which starts from the beginning of the interaction history. Then, we input the currently interacted N items into the model along with the recommended N items. These top-N N newly recommended items are selected from the item pool I by their predicted ranking score given by 2 Note that the ranking score masks out all non-interacted, previously interacted or recommended items to avoid lack of reward supervision or potential data leakage. Given above training setting, we further clarify how we conduct our testing. First, we split the user set into training and test users. Training users are used to train our ECQL model with their interaction history. In testing, for each test user, we assume prior knowledge of interaction history and run ECQL each user directly for different step size based on the minimum session length limit in each dataset. In each step, the user embedding is updated through the agent recommended items only. Then, report the average recommendation performance with three metrics Precision @ N, nDCG @ N and Test Return. As the agent continues collect more user interactions, the recommendation performance will continue to improve as shown in Figure
E.2. EXPERIMENTAL SETTING
We consider each user an episode for the RL setting and split users into 70% as training users and 30% as test users. For each user, we select the first W = 10 interacted items from history H,, to represent an initial state 8g- the next state, we utilize previous state representation and concatenate with five item embeddings from the sliding window and the other five item embeddings from RL based recommendation to generate current state S; by passing through the SSE module. Then, the action network generates mean and covariance for a Gaussian policy from which an action is sampled. This action further passed to the evidence network, which utilizes the item embeddings of all user interacted items (Item Pool I) produce corresponding evidence for each item. We use the setting of classification, where explicit ground-truit ratings are used as class labels. With that evidence, we compute the evidential score by evaluating evidence-based rating and uncertainty to rank those items, which provides a list of top-N (N = 5) final recommendations. We then evaluate the evidential reward based on their ground-truth ratings and information gain, functioning as the trade-off between exploitation and exploration. We set discounted factor y 1 and set 3 as a threshold to identify an item positive, i.e., whether its ground-trun rating larger than or equal to the threshold (ratingu.: 2 7). In testing. the agent may recommend items not interacted by the user. In such cases, we assign a neutral rating for those non-interacted items, where we set 7 = 3. For training, we conduct RL epochs, each with full training of all training users (episodes). Each epoch is equipped with an annealing ranging from 1 to 0.1 to adjust the emphasis from exploration to exploitation the knowledge of training users increases. For testing, we conduct only one RL epoch containing all test users, and we use an annealing À ranging from 0.5 to 0.1 across variable step sizes in four different (minimum session length) data sets. We report the corresponding recommendation performance step-wise averaged over all test users. We implement the experiments based on the PyTorch framework with two A-100 GPUs.
E.2.2 COMPARISON BASELINES
We compare with dynamic, sequential, deep learning, bandit and reinforcement learning based models. More specifically,
Dynamic models include standard dynamic matrix factorization model timeSVD++ (Koren 2009) as the time-evolving latent factorization model and collaborative Kalman filtering (CKF) (Gultekin)
& Paisley 2014).
httes///groopeesssrrr///ssssssss 0m/
23

---

Published as a conference paper at ICLR 2025
1.0
CEP
DUP
0.8
0.8
0.8
00.6
UID
80.6
LID
8 20.6
LIND
8.06
UID
0.4
0.4
0.4
10
14
0.2
10
Stops
Steps
Steps
Steps
(a) Movielens-Iie 1M
(b) Movielens-
(c) Netflix
(d) Yahoo! Music
1.0
1.0
0.8
OCT
CLiSitec
0.8
DKF
OUT
Coll
Colle
Colle
CoLin
0.6
0.6
0.6
0.6
E0.4
0.4
0.4
F..4
0.2
0.2
0.2
10 14
10 14
Stops
Staps
Steps
Steps
(c) Movielens-IM
(f) Movielens- 1O0K
(g) Netflix
(h) Yahoo! Music
Figure 5: Performance comparison in each time step: (a)-(d); P@S; (e)-(h): nDCG@5
Sequential models include Sequential Recommendation via Convolutional Sequence Embed- ding (Caser) (Tang & Wang 2018), attention-based sequential recommendation model (SASRec) [Kang & McAuley 2018), Adversarial Filtering Modeling( (ADM)) Sampling-based Deep Interest Modeling (SDIM), Hierarchical Periodic Memory Network (HPMN) and sequential recommen- dation with bidirectional encoder (BERT4Rec) (Sun et al. 2019) for learning users' sequential preferences. For latest baselines, we have self supervised S Rec Zhou et al. 22200), contrastive learning based CL4SRec (Xie al. 2022]. and Seq2Seq (Ma et al. (2020) which reconstructs the representation of the future sequence as a whole and disentangle the intentions behind the given sequence of behavior.
Bandit models include upper confidence bound (UCB-based) HLin [Wang et al. 2016), contextual bandit based method Lin (Li et al. 2010a) and CoLin (Wu et al. 2016) for news recommendation or collaborative environments respectively.
Reinforcement learning- based models include e-greedy (Zhao al. 2013), soft actor critic
(SAC) (Haarnoja et al.] 2018). 1). deep Q-network based news recommendation (DRN) ZZengg
jt al., 2018), actor-critic based list-wise recommendation (LIRD) (Zhao et al., 2017). Smart
Adaptive Recommendations (SAR) (Antaris & Rafailidis] (021). Top-K Off-Policy Correction Recommendations (REINforCE) (Chen et al. 2019), Sequential Recommendation with Residual Actor for online exploration (ResAct) (Xue et al.] (022). Hyper-Actor Critic (HAC) (Liu et al.
2023), and Supervised Advantage Actor-Critic SAS-SQN) (Silva et al. 2024).
E.3 ADDITIONAL RESULTS
In this section, we present additional experiments and compare with different types of baselines.
Step-wise recommendation performance. We further show the step-wise performance of both precision @5 (P@5) and nDCG@5 metries considering top-5 recommended items in all datasets as shown in Figure 5 We show the average precision and nDCG of all the test users over each step after the model fully trained to demonstrate the effectiveness of the dynamic recommendation. We fixed the step size to 16, 10, 20, and 10 for the Movielens- IIM, Movielens- 100K, Netflix, and Yahoo! Music datasets based on their minimum number of user-item session length, respectively. At the initial steps, both precision and nDCG are low for all models (we choose the best model from each category as shown in Table 2). This is as expected due to lack of user interacted item observations. All the models start to improve after the initial stage. Dynamic models and sequential models still have poor performance compared to the RL-based methods. The proposed ECQL model provides consistently better performance over the entire process. However, it has smaller advantage the beginning due to its strong focus on exploration. After several step's observation, it quickly grasp user's interests and outperforms all its competitors by a clear margin.
24

---

Published as a conference paper at ICLR 2025
Statistical testing results. We have conducted statistical tests by rumning model along with most competitive baselines from each category three times and collecting the corresponding mean and standard deviation of P@5 ( and nDCG @ performance datasets. The results are included in Table 7 It can be seen that the proposed model is more effective in performing effective exploration and provides more accurate recommendations, resulting in higher P@S and nDCG@ 5 on both datasets considering the mean and variance in multiple runs. We further conduct a significance test compare ECQL with the second best performing baseline LIRD. We obtain a p-value of 0.04, which confirms the performance advantage of ECQL over LIRD statistically significant.
MovieLens- IM and MovieLens- 100K
Table 7: Statistical testing results
MovieLens-IBe 1M
MovieLens--100%
Model
P@5
nDCG@5
P@5
nDCG@5
CL4SRec
0.61351100019
0.4952.10.014
0.581310.018
0.478110.012
CKF
0.5567100.08
0.448110.005
0.5285=0.012
0.4322:10001
DCNv2
0.6152+0.016
0.5187+0.013
0.6158+00.014
0.5166+00012
LIRD
0.6238-00.025
0.5332+00018
0.6137+0.023
0.5222+:0018
CoLin
0.6212:10.022
0.523610.016
0.6227-10022
0.5295100016
ECQL
0.6313+ 0.028
0.5365+ 0.021
0.6379+ 0.022
0.5386 0.016
Comparison with RL-based sequential methods larger datasets. In the main paper, we show that ECQL achieves a clear advantage over the baseline models on the four benchmark datasets. Now focus evaluating how ECQL performs on larger scale recommendation problems. We select the strongest baselines from each category for comparison with proposed ECQL on a subset selected from the large data set MovieLens- 10M, denoted as ML-10M, with the minimum user interaction length larger than 300. From Table g it is clear that our method is taking advantage of systematic exploration and providing much better performance in larger datasets. In comparison CoLin, LIRD has better performance due to its capability to leverage a complex reinforcement learning process to strengthen its user representation capability. However, also fails perform enough exploration and achieves lower performance as compared with our method.
Table 8: Comparison with recent RI. sequential baselines on larger dataset ML-10M
Model
ML--10M
P@5
nDCG@5
CL4SRec
0.5138
0,4598
CKF
0,4952
0.4325
DCNv2
0.5374
0,4762
CoLin
0.5574
0,4516
LIRD
0.5624
0,4642
ECQL
0.6425
0,5518
Combining ECQL with other RL4Rec methods. Our proposed ECQL could be generalized into any RL4Rec methods and helps to improve their recommendation performance. In Table we use recent RL4Rec sequential model CL4SRec as a test platform to show the performance improvement achieved by our ECQL comparing to other RL exploration strategies. We report the P@5 5 and nDCG@5 on a larger Movielens- 10M data set to show the results.
E.4 ADDITIONAL ABLATION STUDY
Impact of hyperparameter (A). The hyperparameter (A) plays a critical role in recommending the top-N items through generating corresponding ranking scores in test phase. We test three different settings: = 0.1, = 0.5, and gradually reducing À from 0.5 to 0.1. As can be seen from Figure 6, dynamically adjusting À achieves consistently better test returns on all datasets. This supports the intuition that in the early steps, a large À allows the model to conduct sufficient exploration. Once the model gains sufficient knowledge from the user preference and is able make accurate
25

---

Published as a conference paper at ICLR 2025
Table 9: Comparison between ECQL and other RL exploration strategies on another RL4Rec framework CLASRec
Model
M0V
P@5
nDCG@5
SAC
0,5923
0,4917
EQL
0.6075
0.5028
CoLin
0.6159
0.5133
é-greedy
0.5138
0.4598
ICM
0.5568
0.4689
RND
0.5348
0.4496
ECQL
0.6315
0.5318
A-005-0.11
1=0.1
Rest Return
30
0.0
A- 0.5
Test Raturn
60
--110-1
1-0.5-0..1
Test Return
40
1=0.1
0.0.5-0.1
40
20
20
20
10
Movielens-IN IM
Movielens-1006 100K
(c) Netflix
(d) Yahoo! Music
Figure 6: Test return of ECQL. for different À
recommendations, reducing À will allow the model to exploit knowledge to provide effective recommendations.
Impact of sequence encoders: this section, we investigate the impact of different sequence encoders, including RNN, LSTM, and GRU and report the result Movielens- IM. Table shows that our model's performance remains largely robust to different types of sequence encoders.
Table 10: Impact of sequential encoders
Sequential encoder
P@5
nDCG@5
RNN + CEAC
0,6313
0,5365
LSTM + CEAC
0.6312
0.5364
GRU + CEAC
0.6315
0.5366
Impact of exploration strategies: In the main paper, we show the test return from different exploration strategies demonstrate the effectiveness vacuity-guided exploration of ECQL. In this section, we further report the P@5 and nDCG@S on MovieLens IM to complement the test return result:
Epsilon (ie., c-greedy) is basic exploration strategy, which actor-critic to conduct Q-learning. Since we only set hyper-parameter c = 0.1, which our best practice after grid-search, there's no need combine conservative learning in this exploration direction.
SAC also leverages actor-critic to conduct Q-learning. The difference comparing c-greedy is that uses first-order uncertainty "Entropy" in the rating prediction for score- score-based based exploration and the reward design.
EQL is framework with traditional Q-learning in stead of conservative Q-learning, the exploration is purely guided by uncertainty without any constraints.
CoLin is collaborative contextual bandit based method which explicitly models the underly- ing dependency among users/bandits with the help of weighted adjacency graph, comparing to other bandit exploration technique like upper confidence bound based (UCB-basee HL factorUCB, it achieves best performance in multiple recommendation datasets.
26

---

Published as a conference paper at ICLR 2025
Table 12: Recommended Movies for random user
Model
Recommended Important Movies
Movie Genre
Vacuity
1995,Star Trek: Voyager: Season
Sci-Fi,Action
0.10
2005,7 Seconds
Action,Adventure
0.12
ECQL
1994.mmmorta Beloved
Romance
0.14
N996,No Way Back
Action
0.11
1creamrss
Sci-fi, Action
0.09
Planet
Adventure,,antern
0.16
1982,Nature: Antarctica
Documentary
0.15
EQL
1997,Sick Paula Abdul's 's Get Up 199r,Character
Romance, Drama
0.14
Educational
0.16
Romance,Drama
0.15
1979, Winter Kills
Action
0.08
1991 1991,Anm Antarctica:: IMAX
Documentary
0.16
é-greedy
1951,The Frogmen
Drama
0.13
1983,Silkwoodl 2002, The Powerpuff Girls Movie
Documentary
0.15
Animation
0.15
1951, The Lemon Drop Kid
Comedy
0.13
20st.Mostly Martha
Romance
0.13
SAC
1989,A Fishy Story
Romance
0.14
2004,Spartan 1997, 1997,The Game
Action, Military
0.11
Action
0.09
1965, The Great Race
Adventure,Action
N/A
S002,Obsessed Magoola: Bonus Material
Comance,Drama
N/A
CoLin
Military
N/A
The Battle of Algiers: Bonus Material
Military
N/A
1972,Sccta Aur Geeta
Romance
N/A
ICM uses curiosity to facilitate exploration, measured by the error in predicting the next state from the current state-action pair, leveraging forward dynamic model trained on prior interactions. Additionally, it employs an inverse dynamic model to filter out irrelevant state representations.
RND employs fixed, randomly initialized network (random distillation network) reference model. A predictor neural network is trained to replicate the outputs of the reference model, and the prediction error for any new state serves novelty score.
Table 11: Impact of exploration strategies
Model P@5
nDCG@5
Epsilon
0.5977
0.4834
SAC
0.6105
0.5215
EQL
0.6234
0.5331
CoLin
0.6212
0.5236
ICM
0.6057
0.5088
RND
0.6012
0,5024
ECQL
0.6313
0.5365
E.5 ADDITIONAL QUALITATIVE ANALYSIS
conduct an additional qualitative analysis of a particular user in the Netflix dataset by comparing the top-N recommended items in time step 16 of proposed method ECQL with other models using different exploration strategies, e.g.. é-greedy, SAC, and CoLin. We also include EQL to further validate the necessity of conservative Q-learning. As shown in Table [2] has five most frequently watched movie categories. By order, they are Action, Sci-Fi, Adventure, Drama, and Romance, which form top-5 frequency list and bold in the table. From the table,
with ID 254775
27

---

Published as a conference paper at ICLR 2025
Table 13: Training Efficiency Comparison (averaged training convergence time in hours)
Category
Model
GFLOPS
MovieLens-IM
MovieLens-1008s
Sequential
CL4SRec
5.48
12.9
4.5
Dynamic
CKF
5.15
11.0
3.8
Deep Learning
DCNv2
5.52
13.3
4.6
Bandit
CoLin
5,85
13.9
5.0
Reinforee
LIRD
5.87
14.0
5.0
Proposed
ECQL
5.08
12.0
4.2
we observe that ECQL successfully captures user's long-term interests by recommending important movies that all belong to the top-5 frequency list. Compared to ECQL, EQL recommends some irrelevant movies Documentary and Educational Genres that deteriorate the performance since is not constrained by conservative view in the Q-learning process. Frequently recommending totally irrelevant movies may pose a higher risk of losing the user. For different exploration strategies, e-greedy brings the least performance by recommending only two important movies that lie in the top-5 frequency list as it does not leverage a systematic exploration way to explore the user's potential interests. In comparison, bandit-based CoLin and entropy- -basedd SAC leverage bandit theory and first-order entropy to measure the uncertainty and both lead to an improved recommendation performance by recommending at least three important high frequent movies out of five that reflect the user's s interests. However, we emphasize that they only capture the short-time interest of the user, as demonstrated by their recommended important items' relative order in the top-5 frequency list.
The above analysis is from a random test user in the Netflix data set. For more comprehensive analysis, we calculate the Gini index of the diversity for all the recommended results. First, we test 1,800 users from Netflix test set and then collect 16 (number of time steps) 1,800 total recommendations. For each recommendation, we calculate its Gini index by separating results into different categories and then calculate
Gini = 1 - Σ P(c)2
(18)
where C' is the number of categories and P(c), [1, is the probability for each category. Finally, we average the calculated Gini index for 1,800 recommendations and get the averaged Gini index as 0.71, which is close to 1. This indicates that the recommendation of our model is quite diverse and contains objects from different categories. Comparing to SAC and CoLin which have averaged Gini indexes 0.65 and 0.68 respectively, ECQL achieves the most diverse recommendation thanks to the novel evidence- hidenceebbsed exploration, which also verified by Table [2
E.6 TRAINING EFFICIENCY
We report the training efficiency (averaged training convergence time measured in hours) comparison of the ECQL model on MovieLens- 100K and MovieLens- 1M datasets with respective strongest baseline in each category Table [3] For fair comparison. in all settings, the training resource is two A-100 GPUs of memory 80G and all baselines are implemented with PyTorch framework. Besides, our model's s computation per top-5 recommendation step reaches an average 5,08 GFL.OS..
F LIMITATIONS, FUTURE WORKS, AND BROADER IMPACT
In our evaluation, we applied the proposed ECQL method to the simulated off-policy setting using the offline user interaction datasets to mimic real-world scenarios. It could be more interesting to implement this ECQL method in the online scenarios which could better reflect the model's real-time performance. We are planning to further extend this work to apply to those real-world time sequence data in different fields such as health and autonomous vehicle to see its scalability, generalizability. and adaptability future work. And we will also conduet plenty of ablation and case analysis in these sparse and safety-critich domains to evaluate the effectiveness our unique evidence- based exploration balanced with conservative learning design.
28

---

Published as a conference paper at ICLR 2025
The proposed ECQL can be generally applied to safety-critical applications, which are common in many domains such as health, autonomous vehicle, cybersecurity, military operations, and more. We provide conservative evidential exploration which doesn't let learned policy deviate far from the behavioral policy and supports the gradual exploration of complex environment with sparse reward signals by leveraging fine-grained second-order uncertainties. The principled exploration can ensure high information gain with much reduced data annotations, which can benefit many domains where data annotation is costly.
G
SOURCE CODE
The source code and processed datasets can be accessed here. #itminingglab/B001
https://github.com
29


Reviews:
[{"review_id": "wWIPJbjsd0", "reviewer": "Reviewer_mvAf", "summary": "This paper aims to tackle the exploration problem for RL4RS algorithms. The paper propose a evidential conservative Q-learning framework (ECQL) that models the uncertainty of samples by an evidential network. The paper also controls the degree of exploration by a conservative critic update. Experiments on 4 datasets validates that ECQL outperforms dynamic models, sequential models, deep-learning models, bandit models, and stoa RL-based models.", "questions": "See the weaknesses.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality: The paper studies the exploration problem for RL-based RS, which is novel. Also, the application of the evidential network to quantify the exploration bonus is novel.\nQuality: The paper does detailed and sufficient experiments in both recommendation metrics and rl-based metrics such as NCIS. Recent RL-based methods are compared. The paper also provides case studies on ECQL and SAC, about the relevance and the exploration ability.\nClarity: The paper is well written and easy to follow.\nSignificance: The paper proposes a new algorithm to tackle the exploration problem for RL-based RS. I like the idea of controlling exploration by a conservative Q-learning updating mechanism.", "weaknesses": "1.The paper does not open-source their code.\n2.The paper does not discuss or compare other exploration methods that are widely used in RL, such as ICM and RND.\n3.Why do online RL methods perform well in the offline learning setting, such as SAC,HAC in Table 2?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3V4MYn8G5k", "reviewer": "Reviewer_JFxC", "summary": "This paper presents a approach to recommenders which aims to capture users' long-term interests through reinforcement learning and evidential uncertainty. The authors propose ECQL to address limitations in existing recommender systems, particularly in capturing evolving user preferences and long-term interests. The framework integrates evidence-based uncertainty and conservative learning to develop a conservative recommendation policy.\nECQL employs a typical sequential state encoder that generates the current state of the environment by aggregating,\nA sliding window containing current user interactions, older actions and a newly recommended items from RL exploration\nThis approach allows the model to represent both short-term and potential long-term user interests.\nAnother module seems to perform rating prediction by maximizing the conservative evidential Q-value", "questions": "I suggest the authors try and simplify both the method and the presentation of the work.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The experimental results seem to somewhat validate the approach", "weaknesses": "The paper is extremely dense written and difficult to follow, moreover the approach is over-engineered (see figure) . It is unclear what and by how much each component is contributing to the model. It is unclear if all these components are needed. Moreover the datasets used to not seem to have strong sequential behavioural patterns and the reliance on ratings makes the method somewhat irrelevant to modern recommender systems.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "U50C90yf23", "reviewer": "Reviewer_bGtM", "summary": "This paper proposes a framework called Conservative Evidential Q-Learning (ECQL), which aims to learn effective and robust recommendation strategies by combining evidential uncertainty with conservative learning. ECQL leverages evidence-based exploration to discover items that may lie beyond the current observation range but reflect users' long-term interests. By evaluating strategies from an uncertainty-driven conservative perspective, ECQL reduces the risk of recommending items that deviate excessively from users' current interests.The core components include a Sequential State Encoder, which combines users' historical information with a sliding window that encompasses both current interactions and new items recommended through reinforcement learning exploration, and a Conservative Evidential Actor-Critic Module, which maximizes conservative evidential Q-values to predict scores based on evidence and explores the item space using an uncertainty-based ranking score.Experimental results demonstrate that ECQL excels at capturing users' long-term interests, progressively recommending items that are not significantly different from users' current preferences. This maintains recommendation relevance even with sparse interactions.", "questions": "1. The motivation for introducing an uncertainty-aware exploration strategy is not clearly explained. The correspondence between the solution and the problem is unclear, and it doesn’t explain why uncertainty-aware exploration can effectively capture user’s evolving preferences and achieve the maximum expected reward over the long term, while existing methods cannot.\n\n2. There is generally a balance between exploration diversity and accuracy performance. Further explanation is needed on how the proposed uncertainty-based exploration strategy reflects this balance.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper provides a highly detailed and well-structured presentation of its content. The problem statement is thoroughly explained, supported by experimental results that clearly validate the proposed framework's effectiveness. Each experimental result is precise and comprehensive. The theoretical derivations are carefully detailed, making it easier for readers to follow the logical flow and understand the underlying principles. In addition, the extensive experimentation reinforces the paper’s claims, offering solid evidence for the model’s advantages in capturing long-term user interests and providing stable recommendations. Overall, the combination of clear theoretical foundations with rigorous experimental validation makes this paper a valuable contribution to the field.", "weaknesses": "1. The abstract does not mention the motivation behind the proposed method, which might leave readers somewhat confused.\n\n2. Introducing uncertainty into exploration strategies in reinforcement learning has already been extensively studied, so the novelty of this paper is not particularly outstanding.\n\n3. The paper could benefit from adding some related work on the use of uncertainty in recommendation systems.\n\n4. It would be helpful to experimentally verify whether the evidence network’s evidential score can serve as a plug-and-play component in an ε-greedy strategy to improve the performance of other RL4Rec methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bXMZs0dwuD", "reviewer": "Reviewer_dEvt", "summary": "This paper proposes a model based on evidential conservative Q-learning for sequential recommendation. It integrates the evidential uncertainty into the rating scores to encourage exploration in reinforcement learning. Meanwhile, conservative regularization terms are added to avoid the over-optimistic estimation of the Q-values. Some theoretical analyses are given to justify proposed methods. Experiments on several datasets empirically validate the superior performance with compared baselines on certain metics.", "questions": "All my questions are listed in the weakness part.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe studied problem is important. Building a model to pursue long-term reward while making efficient exploration and avoiding user dissatisfaction is a valuable and challenging topic for both academic and industry.\n2.\tModeling the uncertainty by introducing vacuity of evidence is interesting and novel.\n3.\tTheoretical proofs are given to justify proposed methods, building connections between the evidential uncertainty and the conservative policy update.\n4.\tEmpirical experimental results are promising, revealing superior performance over several baselines.", "weaknesses": "1.\tBased on the definition in Eq. (2), it appears that the candidate item pool throughout the training iterations is restricted to items with which the user has interacted. This limitation may introduce inconsistencies between training and inference and lead to bias issues.\n2.\tIn Eq. (2), it says that 'Wt indexes the time step reached by the current sliding window.' I’m unclear about the definition of 'time step' and its correlation with the original appearance position of a recommended item. Furthermore, it appears that this down-weight term is absent from both their theoretical and experimental analyses, raising questions about its necessity and rationale.\n3.\tIt says that “During testing, for item i′ not appearing in user u’s interaction history Hu, a neutral rating ratingu, i′ = τ will be assigned to give neutral feedback”.  This will lead to complete overlook of all missing user-item interaction values, which can make the evaluation results highly biased on metrics except for NCIS.\n4.\tIt's somewhat puzzling that a model optimized for long-term rewards can outperform state-of-the-art baselines, which are optimized for immediate rewards, on one-step metrics such as Precision and NDCG.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)