Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper works on the sequential recommendation for long-term engagement. This topic is interesting and very important in either academia or industrial. 
In this work, the authors argued that though reinforcement learning was widely accepted as a standard method for long-term engagement optimization, it is not easy to apply RL in practice due to the requirement of online interaction/serving and model update (feature extraction). The authors proposed a new policy that learns the state-action values without the requirement of exploration. The framework first reconstructed the online behaviors based on the offline data, and then improves the model with residual actor. Further, the framework is enhanced with two information theoretical regularizers which improves the expressiveness and conciseness of the extracted features. Comprehensive experiments are conducted on one two datasets, and the performance is improved compared to the existing baselines.
strength_and_weaknesses: This paper focuses on an interesting problem in reinforcement learning, where the online exploration is expensive to perform. And the paper is well-organized, the logic and the descriptions of the technique details are clear and easy to follow. The proposed model is kind of interesting, e.g., adding the information related regularizations. The authors also performed comprehensive experiments to support the proposed solution.

However, my concern of the paper lies in the following aspects.
First, the novelty of the proposed solution is somehow limited. The proposed solution performs an offline policy learning, and combines the residual learning and regularization. The method is somehow straightforward and there's not much technique parts really related to the long-term action modeling.

Second, the authors claims that no exploration was needed for the policy. This is based on their assumption of the access to the sufficient data (correct me if I am wrong). This assumption is not valid especially for real-world scenarios. Meanwhile, for reinforcement learning with online exploration, the insufficient data is the key problem to solve. If sufficient data is accessible, why we need a complicated model to do the predication?

Besides, I am not very clear about the motivation/intuition of adoption a residual actor in the framework. Why and how the reward is decomposed into two parts. If we need the residual modeling, does it mean the first reconstruction step is inaccurate, which seems contradict to the assumption that sufficient data is accessible.

For the experiment part, the authors also compared the model with classical reinforcement learning solutions, are they performing any exploration in the experiment? If not, is the comparison valid?
clarity,_quality,_novelty_and_reproducibility: The paper is well-organized and logic is clear. But for the details of the proposed solution, more explanation would be appreciated to make the paper more clear. For the proposed solution, the novelty is somehow limited. Without the motivation/intuition of the adapted techniques, the framework looks like a combination of several methods.
summary_of_the_review: Overall, the paper works on an interesting problem and proposed an empirically good framework for solving the long-term engagement recommendation problem. However, the details of the framework, especially the motivation and intuition of the adapted techniques, are lacking. For the current version of the paper, the proposed framework is like a combination of existing technique with strong assumption of the data. Therefore, I would suggest a borderline reject for this paper.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors are interested in the long-term engagement of users in recommender systems, which is less studied than immediate engagement. To optimize this long-term engagement in sequential recommendation, they resort reinforcement learning (RL). However, directly applying RL methods to recommender systems can be expensive, and may hurt user experience. To alleviate these issues, the authors introduce ResAct. ResAct is composed of two main modules: (i) the first which reconstructs the online-serving policy, (ii) the second which residually modify the recommendation in order to improve long-term engagement. By doing this, ResAct is able to learn a policy which improves long-term engagement, but stays close to the online-serving policy (limiting the risk of harming user experience). 
Moreover, the authors introduce two information-theoretical regularizers (expressiveness and conciseness) which allows to extract meaningful features from sparse reward signals. 
strength_and_weaknesses: The paper is very pleasant to read even if it combines ideas from different fields (reinforcement learning and Bayesian inference). All the different parts are presented concisely and well illustrated, making the paper accessible for the whole community. Integrating residual improvement in this framework seems like a good idea, which makes a good trade off between exploitation and exploration in recommender systems. The paper is scientifically solid, and the experiments are convincing.
clarity,_quality,_novelty_and_reproducibility: Clarity. The paper is very clear and well written. All notions (from RL and Bayesian inference) are well explained. 

Quality. The paper is scientifically solid, and the experiments are convincing.

Novelty. This work is original. Integrating residual actor seems to be a good way to learn the policy. 

Reproducibility. Experiments seem to be reproducible. 

summary_of_the_review: This paper is very pleasant to read. The content of the paper is original and scientifically solid.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper investigates how to use reinforcement learning (RL) for sequential recommendations. The authors present several innovations for how to address challenges in recommender systems. For one, exploration with actual users can be expensive as they can lead to bad user experience. Also, due to sparse feedback at long time intervals, it is challenging to perform feature selection to remove noisy inputs. To address the first challenge, the authors developed ResAct, which uses supervised learning to learn the production policy and sample additional actions from that policy to select the best one. This eliminates the need to directly explore with users. To address feature selection, they use information theoretic principles to select features that highly predict the reward without retaining redundant information about the state. They demonstrate gains from their approach against several strong baselines using open-source datasets. They also perform ablation experiments to show what design choices lead to their gains.
strength_and_weaknesses: Strengths - 
1. Detailed offline experiments with thorough ablation studies, showing how and why their approach improves over existing work.
2. How to wield RL for recommender systems is an important yet unsolved problem with large real-world impact. 
3. Experiments are open-source and open-data (pending paper acceptance).

Weaknesses - 
1. The ResAct algorithm and the feature selection regularization are a bit orthogonal to each other, and could potentially be presented in separate publications.
2. It's preferable to use a temporal holdout rather than a population holdout for evaluating a recommender system. In the real world, we train in the past and deploy on the future.
3. The experiments on the MovieLens dataset use simulated data inspired by the real-world open source dataset, so may not reflect actual user behavior.
clarity,_quality,_novelty_and_reproducibility: Clarity- Paper is well written and understandable by someone familiar with popular RL techniques and recommender systems.
Quality - Experiments appear to be performed and evaluated competently.
Novelty- Authors propose multiple innovations and demonstrate the value of each.
Reproducibility - Work should be highly reproducible, though code and data will only be open-sourced pending acceptance.
summary_of_the_review: I am inclined to accept this paper for publication. The problem of how to apply reinforcement learning to sequential recommendations is an important and unsolved real-world problem. The authors make good progress on this problem and present multiple innovations that improve performance on this problem. They present ablation studies which identify how their proposed algorithms lead to improvements in their offline experiments. Finally, the code and data samples are promised to be open-sourced after publication. I have only a mild concern that the two areas for innovation (ResAct and information theoretic regularization) seem to be orthogonal improvements, but I still think the community will benefit from reading this manuscript.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper introduces a new framework ResAct for improving long-term user engagement in sequential recommendation. ResAct works in three phases: i) Reconstruction - Training a model to imitate the current recommendation policy and reconstruct recommendation actions; ii) Prediction - Predicting actor residuals that is likely to improve the reconstructed actions in phase one; iii) Selection – selecting the optimal action among all improved actions in phase two. The overall framework is interesting and reasonable. I think it can bring new thinking to the community about how to improve long-term engagement in recommendation and might lead to real-world applications. 
strength_and_weaknesses: Strengths:
1.	The motivation of the method is sound. Improving long-term engagement is a difficult problem and directly optimize it is challenging. ResAct decomposes the problem into several manageable sub-tasks which significantly eases the learning process. The idea of first reconstructing and then improving could inspire people to better solve the problem.
2.	The techniques used in this paper is interesting, e.g., the two information-theoretical regularizers. I am impressed that improving expressiveness or conciseness alone will not lead to improvements and only optimizing them together will improve performance.
3.	Thorough experiments are conducted on various of datasets, including a public benchmark dataset and a real-world industrial dataset. Results are compared with a wide range of baselines. The empirical analysis investigates the working process of the method.

Weaknesses:
1.	Some figures can be better clarified, e.g., it is not clear what the dashed line stands for in fig.3
2.	As discussed in section 4.3, the inference speed is one of the main limitations of the method.
clarity,_quality,_novelty_and_reproducibility: The paper is well-written and easy to follow. Complete codes and data samples are provided.

summary_of_the_review: Overall, the paper has good motivation and the method is novel and interesting. Experiments demonstrate the effectiveness of the method. I support to accept the paper.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Summary: This paper presents ResAct for sequential recommendation. The basic idea is to use offline reinforcement learning (RL) to optimize long-term engagement. As custom in many offline RL literature, ResAct starts by imitating the online serving policy and subsequently adding a residual (through a residual actor) to arrive at a policy that is close to but better than the online serving policy. ResAct also utilizes two information-theoretical regularizers to improve the expressiveness and conciseness of the extracted state features. Experimental results demonstrate that ResAct outperforms other RL approaches at optimizing long-term rewards.  

Strengths: All the reviewers agree that this paper is addressing an important and challenging problem, which is to optimize for long-term engagement in sequential recommender systems. The proposed ResAct follows the standard offline RL approaches, that is to stay close to the logging policy and make a policy improvement. The paper is overall well-written (with minor issues below) and the experimental results are largely convincing (if with some minor issues, as pointed out by reviewer 2DQo). 

Weaknesses: There are some concerns from the reviewers about the novelty of the proposed approach, which is a fair observation. However, I think this is overall a more applied work with promising empirical studies, which makes up for it. Another concern is around the two contributions of the paper (ResAct and the information-theoretical regularization) seemingly orthogonal to each other, which I agree, but even the reviewer who pointed it out doesn't think this is a major issue. 

I also read the paper myself and have communicated my comments with the authors. To summarize, the authors take an unusual approach to use a continuous action space to represent items in a discrete set (which is also what makes adding a residual possible). This deserves much more elaboration (the authors did have a reasonable rationale for it). Furthermore, the paper claims "no need to explore" which is rather misleading (also pointed out by reviewer pqv6) and the authors agree to change the phrasing. Finally, this work draws close parallel to policy improvement work dating back to Kakade & Langford 2002 (with many followup work) which at least deserves some mentions. 
justification_for_why_not_higher_score: This paper presents a solid piece of work but its novelty is slightly limited to be appealing to the general ICLR audience. 
justification_for_why_not_lower_score: I think it is basically unanimously agreed that the recommender systems research community would benefit from seeing this work. It has some flaws but not to the extent of being rejected.
Decision: Accept: poster