Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a deep evidential reinforcement learning framework, which aims to learn a more effective recommendation policy by integrating both the expected reward and evidence-based uncertainty. Specifically, a recurrent neural network (RNN) is used to exploit dynamic state space, which generates the current state by incorporating past (previous state information), current (sliding window interactions), and future (recommended items from the RL agent) interests. Besides, an EAC module (evidential-action-critic network) is used to perform evidence-based exploration by maximizing a uniquely designed evidential Q-value to derive an optimal policy. Experimental results on real-world data have been performed to demonstrate the effectiveness of the proposed model. 
strength_and_weaknesses: Strengths
1) The authors propose a new recommendation framework, i.e., DERL, which integrates a customized RNN and an EAC (evidence-actor-critic) module to exploit dynamic state space and explore item space, respectively.
2) The proposed method leverages vacuity, an second-order uncertainty, to perform effective exploration. 
3) The authors have performed extensive experiments on real datasets to demonstrate the effectiveness of the proposed method.
 
Weaknesses
1) In Section 1, the third contribution claimed by the authors is "An off-policy formulation to effectively promote the reuse of previously collected data while stabilizing model training, which is important to address data scarcity in recommender system". However, in this work, there is no experimental analysis demonstrating the proposed method can address the data scarcity challenge. 
2) In Figure 1 and Table 1, the examples show that existing methods fail to capture the user's dynamic preference to obtain the maximum reward. However, the description about the user's long-term preference is missing in these cases. 
3) In Section 4.3, for the evidence network, the authors introduce that "e_{ik} is the evidence collected for rating k for item i". However, it is not clear how to collect the learned evidence e_{ik} from the current action a_t and the item pool I using the evidence network. 
4) The description about the experimental datasets are not clear enough. For example, the number of interactions in the pre-processed Netflix dataset, the number of users and the number of items on Yahoo dataset. 
5) Even though this paper includes many baselines, some classical top-N item recommendation methods should be considered as baselines, e.g., BPRMF and LightGCN. For dynamic recommendation models, the authors need to consider the dynamic top-N recommendation methods as baselines, for example, Dynamic poisson factorization, RecSys 2015. 
6) The RL-based recommendation baselines, i.e., DRN, LIRD, and CoLin, are published in 2018 or earlier. The authors need to consider state-of-the-art RL-based recommendation methods as baselines. More RL-based recommendation methods can be found in these surveys: 1) Reinforcement learning based recommender systems: A survey, ACM Computing Surveys 2022, 2) A Survey on Reinforcement Learning for Recommender Systems, arXiv:2109.10665.
7) Does the input feature of the model contain category information (e.g., movie genre)? What factors can help improve the diversity of genre in the evidence-based exploration? 
8) The link to the source code is expired. 




clarity,_quality,_novelty_and_reproducibility: Clarity: The paper is clear, however, some detailed descriptions of networks need to be improved. 
Quality: The paper is technically sound, and some claims are supported by the experiments.  
Novelty: This paper provides a novel idea that integrates reinforcement learning with evidential learning to perform exploration.
Reproducibility: The repository of the source code is expired. 

summary_of_the_review: This paper provides a novel reinforcement learning-based recommendation framework. However, the experimental analysis seems not convincing enough. The baselines used in the experiments are not strong enough.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a new approach to exploration in RL for recommender systems. The authors propose using the concept of vacuity to model uncertainty about an item's rating, and include it as an exploration bonus into the reward function.
Offline results through multiple datasets show that the proposed method, DERL, not only is able to achieve higher top-5 precision compared to other methods, but produces more varied results thanks to the exploration component.
strength_and_weaknesses: Strength:
- The paper introduces a novel idea for exploration, as far as I am aware. Maintaining varied recommendations, while still showing relevant items to the user is a difficult problem, and the introduction of vacuity seems to be doing a good job at addressing that problem.
- The authors evaluated in a large number of datasets over multiple competing benchmarks, which strengthen confidence in the propose method.



Weakness:
- There are a few key components of the paper that were not completely clear (see summary of review). For example, in eq 1, what constitutes evidence "e_k"? It should take some minor editing to clarify some of these questions.
- I understand it's difficult to do, but since the paper focus is on exploration, it would have strengthened the work quite a bit to have an online experiment. Exploration means producing results that have not been produced before, and these counterfactuals cannot be evaluated from historical data. 


clarity,_quality,_novelty_and_reproducibility: The paper is well-written and easy to read. Some important pieces could be explained in more detail to avoid the reader having to guess how things were implemented, but overall I had little trouble understanding the proposed method.
In my opinion, this approach to exploration is novel and aims to address a common issue in recommender systems. I have not seen the concept of vacuity being used as an exploration bonus for RL in recommenders.
There are enough details in the paper to reimplement the architecture, so I believe that reproducibility should not be a major issue.
summary_of_the_review: The paper proposes a novel approach to exploration for RL in the context of recommender systems. 
The authors ran extensive tests in which the results align with the claim made. Most importantly, the qualitative show by inspection that DERL is able to produce more varied results, instead of biasing recommendations to one particular genre.

The paper is well written, but I would ask the authors to clarify a few things in the paper with more details:

1 - In eq 9, what precisely is "e_k" in this case? What constitutes evidence for class k?
2 - Eq 10., why use MSE as a loss of the first component when you have "p_ik" and "y_ik"? Isn't this akin to a classification problem, and you could use cross-entropy?
3 - For the uncertainty "U(i | s, a)", can you clarify how eq. 1 is used in the evaluation. As defined, Eq. 1 is K / sum_k^K (e_k + 1), which means it is not a function of any particular class (K is number of classes, and the bottom is a sum over all classes). 
So how can U( i | s ,a ) be evaluated for 1 particular item? Wouldn't  i be considered 1 class over all possible classes?
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a exploration method, called  deep evidential reinforcement learning (DERL)  to improve the exploration ability of RLRS. The DERL algorithm uses recurrent neural network to represent the dynamic feature of the user state, and conducts evidential-actor-critic module to enable better exploration. Experimental results validates its effectiveness over dynamic models, sequential models, and previous RL for RS methods. Finally, some ablation studies and qualitative studies are conducted.
strength_and_weaknesses: Strength:

1.Exploration in RL for RS is an important problem.

2.The paper is generally well written and easy to follow.

3.The experiments are on several public datasets.

Weakness:

1.The technical quality of this paper is not enough, and it seems like a direct combination with Evidential Theory and Reinforcement Learning.

2.The paper is not sound as there are many exploration methods in RL literature, such as count-based methods and intrinsic motivations(RND,ICM). But the paper does not discuss and compare these methods.

3.The theoretical analysis is not novel, as it is a direct result of RL theory.

4.The update rule of the critic network does not follow Double DQN, but follow the clipped double q-learning in the well known TD3 algorithm.

5.The paper does not provide a specification of the experimental setup. Did the authors bulid simulator? If not, how to evaluate the performance of each policy in the offline setting?

6.Why not compare SAC in Table 2 as SAC is compared in Figure 6?

7.How to verify that the performance improvement over pervious RL methods indeed comes from the evidential reward? As we can see, you choose some advanced techniques like Eq (11), and it is not deployed in previous baselines.
clarity,_quality,_novelty_and_reproducibility: 1.The paper is well written.

2.The paper provides the code link, and test algorithms in public datasets.

3.The novelty is limited as it is a direct combination of Evidential Theory and Reinforcement Learning.

4.The paper is not sound in aspects of baselines, ablation study, and the experimental setup, as pointed out in Strength And Weaknesses part.
summary_of_the_review: Overall, the paper study an important problem for RL in RS. The paper is well written, and does experiments on several public datasets.
However, the novelty of the paper is limited, and does not discuss and compare current exploration methods. Also, the experiments are not sound. Thus, I choose to reject this paper at this version.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Summary: This paper presents a deep evidential reinforcement learning (DERL) approach to sequential recommendation. The basic idea is to add evidential uncertainty around the expected reward to encourage exploration (similar to UCB). On top the standard actor-critic framework with an actor network and a critic network, an evidence network is also incorporated to compute a so-called evidential Q-value and DERL optimizes policy based of it. Experimental results on public benchmark datasets demonstrate the effectiveness of the proposed approach.

Strength: All the reviewers agree that exploration in recommender systems is an important task. The idea of using evidential uncertainty to encourage exploration is interesting. Extensive experiments on real datasets demonstrate the effectiveness of the proposed method.

Weaknesses: The reviewers have some general concerns about the choice of baselines and empirical studies. Specifically, reviewer JmPi argues that more sophisticated exploration strategies should be compared against. Moreover, even though the authors included some additional RL baselines during the discussions phase, none of them have exploration in it. There are also questions regarding the technical contribution of the work given its more applied nature, which I think is a fair point. Finally, the writing can be improved: for example, evidence is huge part of the paper yet 2 out of the 3 reviewers had trouble understanding how exactly the evidence is computed. 

One reviewer gave a high score but also acknowledged the issues pointed out by the other reviewers and wouldn't necessarily champion the paper. 
justification_for_why_not_higher_score: Based on the reviews and the subsequent discussions, I think it's fair to say that this paper is not ready for publication yet given there are still concerns about the methodological contribution as well as the choice of baselines.
Decision: Reject