Looking into User’s Long-term Interests through the Lens of Conservative Evidential Learning

Authors: Dingrong Wang, Krishna Prasad Neupane, Ervine Zheng, Qi Yu
Primary area: reinforcement learning
Keywords: recommender systems, evidence-aware exploration, evidential learning

Abstract:
Reinforcement learning (RL) provides an effective means to capture users' evolving preferences, leading to improved recommendation performance over time. However, existing RL approaches primarily rely on standard exploration strategies, which are less effective for a large item space with sparse reward signals given the limited interactions for most users. Therefore, they may not be able to learn the optimal policy that effectively captures user's evolving preferences and achieves the maximum expected reward over the long term. In this paper, we propose a novel evidential conservative Q-learning framework (ECQL) that learns an effective and conservative recommendation policy by integrating evidence-based uncertainty and conservative learning. ECQL conducts evidence-aware explorations to discover items that are located beyond current observations but reflect users' long-term interests. It offers an uncertainty-aware conservative view on policy evaluation to discourage deviating too much from users' current interests. Two central components of ECQL include a uniquely designed sequential state encoder and a novel conservative evidential-actor-critic (CEAC) module. The former generates the current state of the environment by aggregating historical information and a sliding window that contains the current user interactions as well as newly recommended items from RL exploration that may represent short and long-term interests respectively. The latter performs an evidence-based rating prediction by maximizing the conservative evidential Q-value and leverages an uncertainty-aware ranking score to explore the item space for a more diverse and valuable recommendation. Experiments on multiple real-world dynamic datasets demonstrate the state-of-the-art performance of ECQL and its capability to capture users' long-term interests.

Decision: accept (poster)
Reviewer scores: 8, 3, 6, 8

Reviews and discussion:
<thread>
creation date:
1729541727056

thread_type:
official_review

<thread_object>
creator: reviewer_devt

<content>
summary:
This paper proposes a model based on evidential conservative Q-learning for sequential recommendation. It integrates the evidential uncertainty into the rating scores to encourage exploration in reinforcement learning. Meanwhile, conservative regularization terms are added to avoid the over-optimistic estimation of the Q-values. Some theoretical analyses are given to justify proposed methods. Experiments on several datasets empirically validate the superior performance with compared baselines on certain metics.

strengths:
1.	The studied problem is important. Building a model to pursue long-term reward while making efficient exploration and avoiding user dissatisfaction is a valuable and challenging topic for both academic and industry.
2.	Modeling the uncertainty by introducing vacuity of evidence is interesting and novel.
3.	Theoretical proofs are given to justify proposed methods, building connections between the evidential uncertainty and the conservative policy update.
4.	Empirical experimental results are promising, revealing superior performance over several baselines.

weaknesses:
1.	Based on the definition in Eq. (2), it appears that the candidate item pool throughout the training iterations is restricted to items with which the user has interacted. This limitation may introduce inconsistencies between training and inference and lead to bias issues.
2.	In Eq. (2), it says that 'Wt indexes the time step reached by the current sliding window.' I’m unclear about the definition of 'time step' and its correlation with the original appearance position of a recommended item. Furthermore, it appears that this down-weight term is absent from both their theoretical and experimental analyses, raising questions about its necessity and rationale.
3.	It says that “During testing, for item i′ not appearing in user u’s interaction history Hu, a neutral rating ratingu, i′ = τ will be assigned to give neutral feedback”.  This will lead to complete overlook of all missing user-item interaction values, which can make the evaluation results highly biased on metrics except for NCIS.
4.	It's somewhat puzzling that a model optimized for long-term rewards can outperform state-of-the-art baselines, which are optimized for immediate rewards, on one-step metrics such as Precision and NDCG.

questions:
All my questions are listed in the weakness part.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730519148270

thread_type:
official_review

<thread_object>
creator: reviewer_bgtm

<content>
summary:
This paper proposes a framework called Conservative Evidential Q-Learning (ECQL), which aims to learn effective and robust recommendation strategies by combining evidential uncertainty with conservative learning. ECQL leverages evidence-based exploration to discover items that may lie beyond the current observation range but reflect users' long-term interests. By evaluating strategies from an uncertainty-driven conservative perspective, ECQL reduces the risk of recommending items that deviate excessively from users' current interests.The core components include a Sequential State Encoder, which combines users' historical information with a sliding window that encompasses both current interactions and new items recommended through reinforcement learning exploration, and a Conservative Evidential Actor-Critic Module, which maximizes conservative evidential Q-values to predict scores based on evidence and explores the item space using an uncertainty-based ranking score.Experimental results demonstrate that ECQL excels at capturing users' long-term interests, progressively recommending items that are not significantly different from users' current preferences. This maintains recommendation relevance even with sparse interactions.

strengths:
This paper provides a highly detailed and well-structured presentation of its content. The problem statement is thoroughly explained, supported by experimental results that clearly validate the proposed framework's effectiveness. Each experimental result is precise and comprehensive. The theoretical derivations are carefully detailed, making it easier for readers to follow the logical flow and understand the underlying principles. In addition, the extensive experimentation reinforces the paper’s claims, offering solid evidence for the model’s advantages in capturing long-term user interests and providing stable recommendations. Overall, the combination of clear theoretical foundations with rigorous experimental validation makes this paper a valuable contribution to the field.

weaknesses:
1. The abstract does not mention the motivation behind the proposed method, which might leave readers somewhat confused.

2. Introducing uncertainty into exploration strategies in reinforcement learning has already been extensively studied, so the novelty of this paper is not particularly outstanding.

3. The paper could benefit from adding some related work on the use of uncertainty in recommendation systems.

4. It would be helpful to experimentally verify whether the evidence network’s evidential score can serve as a plug-and-play component in an ε-greedy strategy to improve the performance of other RL4Rec methods.

questions:
1. The motivation for introducing an uncertainty-aware exploration strategy is not clearly explained. The correspondence between the solution and the problem is unclear, and it doesn’t explain why uncertainty-aware exploration can effectively capture user’s evolving preferences and achieve the maximum expected reward over the long term, while existing methods cannot.

2. There is generally a balance between exploration diversity and accuracy performance. Further explanation is needed on how the proposed uncertainty-based exploration strategy reflects this balance.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730630645489

thread_type:
official_review

<thread_object>
creator: reviewer_jfxc

<content>
summary:
This paper presents a approach to recommenders which aims to capture users' long-term interests through reinforcement learning and evidential uncertainty. The authors propose ECQL to address limitations in existing recommender systems, particularly in capturing evolving user preferences and long-term interests. The framework integrates evidence-based uncertainty and conservative learning to develop a conservative recommendation policy.
ECQL employs a typical sequential state encoder that generates the current state of the environment by aggregating,
A sliding window containing current user interactions, older actions and a newly recommended items from RL exploration
This approach allows the model to represent both short-term and potential long-term user interests.
Another module seems to perform rating prediction by maximizing the conservative evidential Q-value

strengths:
The experimental results seem to somewhat validate the approach

weaknesses:
The paper is extremely dense written and difficult to follow, moreover the approach is over-engineered (see figure) . It is unclear what and by how much each component is contributing to the model. It is unclear if all these components are needed. Moreover the datasets used to not seem to have strong sequential behavioural patterns and the reliance on ratings makes the method somewhat irrelevant to modern recommender systems.

questions:
I suggest the authors try and simplify both the method and the presentation of the work.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730730671320

thread_type:
official_review

<thread_object>
creator: reviewer_mvaf

<content>
summary:
This paper aims to tackle the exploration problem for RL4RS algorithms. The paper propose a evidential conservative Q-learning framework (ECQL) that models the uncertainty of samples by an evidential network. The paper also controls the degree of exploration by a conservative critic update. Experiments on 4 datasets validates that ECQL outperforms dynamic models, sequential models, deep-learning models, bandit models, and stoa RL-based models.

strengths:
Originality: The paper studies the exploration problem for RL-based RS, which is novel. Also, the application of the evidential network to quantify the exploration bonus is novel.
Quality: The paper does detailed and sufficient experiments in both recommendation metrics and rl-based metrics such as NCIS. Recent RL-based methods are compared. The paper also provides case studies on ECQL and SAC, about the relevance and the exploration ability.
Clarity: The paper is well written and easy to follow.
Significance: The paper proposes a new algorithm to tackle the exploration problem for RL-based RS. I like the idea of controlling exploration by a conservative Q-learning updating mechanism.

weaknesses:
1.The paper does not open-source their code.
2.The paper does not discuss or compare other exploration methods that are widely used in RL, such as ICM and RND.
3.Why do online RL methods perform well in the offline learning setting, such as SAC,HAC in Table 2?

questions:
See the weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732366549380

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General response

comment:
We thank all reviewers for reviewing our paper and providing constructive comments and suggestions. Below, we summarize some of our major responses (MR):

---
**MR1: Motivation of proposed methodology.**

>Most existing methods treat recommendation as a static process, which prevents them from effectively accounting for users’ evolving preferences. Sequential recommendation methods address this limitation to some extent by leveraging previously interacted items to capture users’ dynamic behavior. However, prior RL-based recommender models largely rely on standard exploration strategies, such as $\epsilon$-greedy, which are less effective in scenarios with a large item space and sparse reward signals due to limited user interactions. As a result, these methods may struggle to learn an optimal policy that adequately captures users’ evolving preferences and achieves the maximum expected reward over the long term. The qualitative results presented by Figure 1 and Table 1 in the Introduction Section of the paper illustrate the limitations of existing approaches and further highlight the need for a systematic, uncertainty-aware exploration strategy.

>The proposed ECQL framework is designed to address the key limitations of existing RL based recommender systems. The novel evidential conservative Q-learning (ECQL) framework utilizes a balanced exploitation (with high predicted ratings) and exploration (with evidential uncertainty) strategy for effective recommendations. ECQL augments the reward based RL objective with **evidential uncertainty** to facilitate the exploration of unknown items. The evidential uncertainty formulation substantially improves exploration and robustness by acquiring diverse behaviors that are indicative of a user's long-term interest. Additionally, it encourages the model to explore items that do not significantly deviate from users' current interests given the sparse interactions. Such gradual exploration is guided by a conservative **evidential Q-value** that prevents recommending totally irrelevant items, causing user frustrations. 

**MR2: Novelty of introducing evidence-based uncertainty.**

>Compared to existing uncertainty-based exploration strategies, we would like to clarify that the proposed Evidential Conservative Q-Learning (ECQL) framework is specifically designed to address a set of novel challenges unique to recommender systems, including:

1. Limited training data due to scarce user interactions,

2.  Sparse and noisy reward signals, and

3. Complex dynamics in user behavior, where long-term interests are intertwined with temporary preferences.

>Our framework introduces a novel approach by employing a more precise second-order uncertainty measure—vacuity—to reflect the lack of knowledge about candidate items in the item pool based on the current model analysis. This vacuity-based measure is leveraged to discover novel items that have the potential to align with users’ long-term interests. While effective exploration is crucial to accurately capturing users’ evolving preferences, ineffective exploration can lead to irrelevant recommendations that quickly erode user trust. To address these long-standing challenges, we propose a newly designed evidential off-policy reinforcement learning model. This model learns an effective and safe recommendation policy through a **non-trivial integration of evidence-based exploration and conservative learning**, providing a balanced approach to exploration and exploitation in recommender systems. Our comprehensive evaluation and through ablation study show that ECQL outperforms existing RL methods with uncertainty-based exploration strategies with a large margin, which further justifies the effectiveness of our novel exploration strategy. 

**MR3: Discussion and comparison with other exploration methods.**

>As suggested by the reviewers, we have included a detailed discussion of other exploration methods, including ICM and RND. Additionally, we conducted further experiments to adapt these exploration strategies to the recommender system setting. The comparison results demonstrate that our proposed approach outperforms both ICM and RND in effectively capturing user preferences and guiding recommendations. These findings further validate the effectiveness of our method in addressing the challenges of sparse rewards and user interest shifts in recommender systems. 

---
We have provided detailed responses to each individual comment from all reviewers and incorporated the suggested changes into the revised paper, with the updates highlighted in blue. We sincerely appreciate the insightful questions and feedback, which have helped us further refine and strengthen our work. We look forward to any additional feedback based on our responses. Thank you again for your valuable input!

Best, 

Authors
</content>
</thread_object>

</thread>
<thread>
creation date:
1732898360252

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of our responses and reminder of the reviewer-author discussion deadline [December 2nd]

comment:
Dear Reviewers,

Thank you once again for taking the time to review our paper and provide thoughtful comments and suggestions. As the reviewer-author discussion deadline (December 2nd) approaches, we would like to summarize our key responses to your feedback:

- **Reviewer mvAf**:

  - Included comparisons with other RL exploration methods (ICM and RND) as suggested.
  - Clarified the settings for baseline models, including SAC and HAC.

- **Reviewer JFxC**:

  - Emphasized that ECQL includes only essential RL components, avoiding unnecessary complexity, while highlighting its key novelties.
  - Referenced our ablation study results to illustrate the contribution of each component.
  - Clarified the experimental settings.

- **Reviewer bGtM**:

  - Expanded on the motivation and novelty of our work.
  - Highlighted the versatility of our model as a plug-and-play component.
 
- **Reviewer dEvt**:

  - Provided a clear definition of "time step" in Equation 2.
  - Justified the fairness of our experimental setup.

We believe that these revisions have significantly improved our paper, and we deeply value your constructive feedback. Please let us know if there are any additional questions or concerns.

Sincerely,

Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
The submission proposes a novel Evidential Conservative Q-Learning (ECQL) framework aimed at addressing the exploration-exploitation tradeoff in reinforcement learning-based recommender systems. The ECQL integrates evidence-based uncertainty and conservative learning to optimize recommendations for long-term user interests while ensuring robustness and relevance. Two key components underpin this work: a sequential state encoder that models dynamic user behavior and a Conservative Evidential Actor-Critic (CEAC) module that balances exploration with user interest consistency. The extensive experimentation demonstrates ECQL's superiority across various benchmarks, achieving state-of-the-art results and effectively capturing long-term user preferences.

Strengths of the paper include its originality in applying evidential uncertainty to recommender systems, detailed experimental validation, and strong theoretical contributions, including the justification of the connection between evidential uncertainty and conservative policy updates. The reviewers generally appreciated the problem significance, methodological rigor, and comprehensive ablation studies that validated the contributions of individual components.

However, the paper has certain weaknesses. Some reviewers noted that introducing uncertainty into reinforcement learning is not novel, requiring the authors to clarify their distinct contribution to recommender systems specifically. Other issues raised include limited sequential behavioral patterns in the datasets, concerns about over-engineering, and questions about whether the model's components are all essential. These concerns were addressed in the rebuttal with additional experimental results, detailed clarifications, and revisions to the paper.

The most important reason for recommending acceptance is the paper’s solid contributions to addressing a critical challenge in RL-based recommender systems—effectively capturing user preferences over the long term. The authors successfully defended the novelty and utility of their approach through thoughtful responses, additional experiments, and improved clarity in their rebuttal. While some concerns about novelty and generalizability were raised, the majority of the reviewers agreed that the work represents a meaningful advancement in this domain.