Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors argue that it is important to 1) estimate the impact of recommendation systems of user preferences, 2) evaluate if the shifts would be undesirable, and 3) optimize to avoid undesirable shifts.  The authors propose a method to do this and rely on simulations to evaluate it.
main_review: I do not wish to reiterate the strengths and weaknesses as described by the other reviewers.  Instead, I will pose a set of points for the authors to consider as they revise the paper.

- Is it true that consumer preferences shift?  In support of this claim, the authors cite models that assume that consumer preferences shift.  There could be many reasons for shifting preferences; for example novelty-seeking or learning one's own preferences are two distinct cases.  In other situations, preferences might not shift at all.  Citing work outside of computer science might be the best for supporting this point [e.g., 1].
- More subtly, there is a difference between consumer preferences and consumer behavior [2]. This paper does not distinguish between the two.
- Even when consumer preferences shift, do RSs really influence users?  Prior work with Amazon data suggests that at least 75% of observed activity would likely occur in the absence of recommendations [3].
- I am fully supportive of simulation methods; this is an active area of research, however, and best practices are still being established (see the "Workshop on Simulation Methods for Recommender Systems," https://simurec.piret.info/).  Figure 1 needs to point to the assumptions used to generate it (otherwise it might as well be a cartoon).  I strongly encourage the release of simulation source code to ensure that the work is reproducible and that other researchers can explore performance under modified simulation assumptions.  Additionally, showing similar results under different simulation assumptions would go a long way to dispel concerns about the use of simulation methods.
- There are many LaTeX formatting issues that need to be fixed.  E.g., tilde over 30% and 45% on page 2, backwards quotes all over the place, etc.

[1] Dzyabura, Daria and Hauser, John R., Recommending Products When Consumers Learn Their Preferences (February 15, 2017). Available at SSRN: https://ssrn.com/abstract=2202904 or http://dx.doi.org/10.2139/ssrn.2202904

[2] Friese, Malte, Michaela Wänke, and Henning Plessner. "Implicit consumer preferences and their influence on product choice." Psychology & Marketing 23.9 (2006): 727-740.

[3] Sharma, Amit, Jake M. Hofman, and Duncan J. Watts. "Estimating the causal impact of recommendation systems from observational data." Proceedings of the Sixteenth ACM Conference on Economics and Computation. 2015.




summary_of_the_review: I think this paper could be modified to 1) better motivate the work and 2) consider alternative simulation assumptions to show the method is robust.  Otherwise, it addresses an interesting problem in a new way.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper studies an interesting problem of estimating preference shifts induced by recommender systems and proposes a distance metric for the shifted preferences based on the so-called "safe shifts". The proposed metric can help penalize undesired preference shifts, which can be easily used in training and evaluation. Experiments via the simulated recommendation environment and simulated human behavior verify the effectiveness of the proposed approach.

main_review: 
## Strengths
1. The studied problem is interesting. Estimating the induced preferences are quite important for recommender systems, especially for real-world applications, where users indeed interact with recommendation algorithms.
2. The proposed method makes sense. It is well motivated.

## Weaknesses

- Why don't the authors deploy some wide-used simulators, such as Virtual-Taobao [1]. The literature survey about the simulator in recommender systems should be considered.
- It is always noisy if the experiments are based on the simulation. 
- The experimental setup is overall too ideal. For example, in real-world applications, the recommendation engine/system may do not allow preference manipulation.
- There are a lot of choices that are selected biasedly. For example, the authors use a BERT model (Sun et al., 2019) as the human model, which leaves me a question: how about the other choices? Is this model the best choice? 
- The authors should re-think the value of this work. If there is no solid evaluation manner, a proper solution is to plug this method into some commonly-accepted recommendation pipelines. For example, you can first use the estimated preferences shifts to improve the backbone recommendation models and then evaluate based on traditional evaluation manners of recommender systems. In the current version of this work, it is still unclear whether the method work or not.
- There is no competitive baseline method in the results (Table 1 and Table 2). I suggest the authors at least add some heuristic methods as baselines.
- The complexity of the proposed approach is unclear. Will it be a burden for the recommendation model?

[1] Shi, Jing-Cheng, et al. "Virtual-taobao: Virtualizing real-world online retail environment for reinforcement learning." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 33. No. 01. 2019.

## After Rebuttal
The authors have carefully summarized my concerns into five aspects. I agree with the reply to the 1st, 2nd, and 4th, while I partly agree or disagree with the 3rd and 5th. Thus I have raised my score.
summary_of_the_review: Overall speaking, this paper studies an interesting problem of estimating preference shifts induced by recommender systems. Nevertheless, the proposed method and experimental setup are based on too many assumptions, which may do not hold in the real world. Therefore, I recommend rejection.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper discusses an interesting but important problem in recommender systems, i.e., the users’ preference shift under the influence of recommender systems. Currently, both RL-based and myopic solutions cannot avoid influencing preferences in undesirable ways. Instead, authors provide a simulation framework that enables system designers to estimate the induced preference shifts an RS; evaluate its influence before deployment; and actively optimize to avoid such shifts. The basic idea is two-fold. The first design is to estimate user preference dynamics by training a user predictive model from historical user interaction data, which supports both future preference estimation and counterfactual estimation. The second one is to evaluate and optimize the current recommendation policy by measuring whether its influence is close to users’ natural preference shift. 
By simulation study, authors show that recommender systems optimizing for staying in the above trust-region can avoid manipulative behaviors (e.g., changing preferences in ways that make users more predictable).

main_review: Strengths:
1.	This paper focuses on an important problem that becomes increasingly important for recommender systems nowadays.
2.	The simulation framework, though with simplification and assumptions, is elaborately designed and seems promising for handling the preference shift problem.
3.	Idea of introducing users’ natural preference shift and penalizing the recommendation policy to stay in the trust-region is interesting and novel.

Weakness:
1.	Many technical details are missing, making it like a preliminary report.
-	In Sec. 4, when introducing estimating users’ preferences, the inference process is demonstrated in detail. On the contrary, descriptions about training a user predictive model given the historical data are vague, which is equally important. 
-	For example, how to obtain $P(z^{\\pi '}_H | z^{\\pi }_\text{T+1})$ in Sec.4.1 (Estimation under known internal state dynamics)? 
-	Eq. 3 seems to have a typo, "=" and "$\approx$" might swap their positions?
-	Authors have not released their experiment code, which makes it difficult for other colleagues to follow.
2.	Experiment details are also not clear. For example,
-	In ground truth human dynamics, given $b_t^H(s)$ and $u_t$, how is the distribution over users’ next timestamp preferences generated?
-	Does the counterfactual preference estimation used in evaluating the possible influence of recommendation policies?
-	What is the myopic method used in experiments?
-	What is the physical meaning of the “SUM” (of engagement)? Authors claim in the abstract that their framework can avoid manipulative behaviors but still generate engagement. What is the meaning of generating engagement? It seems to me that this framework can improve engagement under the initial or naturally shifted user preferences according to Table 1/2.
3. Presentation quality can be improved by polishing up the wording and rearranging the figures and tables in Sec. 6&7.

summary_of_the_review: Although an interesting study, the paper has limitations (please see "weaknesses" section above). I would say that the current version of the paper is marginally below the acceptance threshold, but I am looking forward to the authors addressing my concerns above in their rebuttal.

## After Author Response
The authors' response has addressed most of my concerns and I choose to raise the score to 6. 

Also, I expect the authors to format their released experiment code, such that followers can better understand the model and simulation.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors address the challenge of preference shifts, a potentially undesired effect of recommendation systems (RSs). A preference shift is a byproduct of any RL system, in which the systems drive users to places (i.e., in their particular preference representation) where they are easily satisfiable. This can happen, e.g., by positive feedback loops. The main challenge of tacking this undesired effect is that we lack the “natural preferences” or a robust way to predict preference shift given a recommendation policy. The authors suggest building a system, which is partially data-driven and partially based on behavioral assumptions to simulate recommendation policies and find safe shifts --- shifts ensuring that users’ preferences will not be influenced too much by the specific policy. They explain the ingredients of the system and experimentally test it.
main_review: Reasons to accept:
*The ideas presented in this paper are somewhat taboo, and commercial companies are not likely to do this type of research. This is another reason to encourage studying it.

*The methods the authors suggest are practical, and their results lay the foundations for “neutral” recommendation policies.

Reasons to reject:

*The results rely on modeling assumptions, for instance, their choice modeling. Different assumptions can change the conclusions entirely. 

*The work has many limitations (despite that the authors address most of them). For instance, I am not sure that random recommendation is a good benchmark to test for preference shifts; the authors did not motivate this selection well enough.

I think the topic of this paper should be a core one at the ethical ML research direction, and yet it does not receive enough attention (especially not from industry). The counterfactual arguments and tools the authors propose are essential for conducting a what-if analysis like the one they propose, so the behavioral assumptions are somewhat inevitable. Overall, I think the community can benefit from the ideas presented in the paper, so I vote for (weakly) acceptance. 

There are several typos in the paper; here are some of them:
-page 2: the \tilde sign is presented incorrectly with the percentages. 
-page 2: "we e active", e -> take?
-page 18 "the the" inside the footnote.
summary_of_the_review: The paper addresses a real-world problem and provides tools to deal with it. I vote for weak acceptance.
review_title: null

Meta-review: This paper studies the influence of recommender systems on users' preferences. The authors propose a method for estimating preference shifts, evaluating their desirability, and avoiding such shifts (when needed).

After the initial review and discussion period, a fourth reviewer with significant recsys experience and a very good knowledge of this sub area was invited to provide an additional review of the paper. This is reviewer vNt7. Their review was positive overall but did highlight some limitations and potential ways to improve the paper's grounding in the recsys literature.

Overall, the main strengths of this paper were that it studies an interesting and practically motivated question. The reviewers also found the proposed solution reasonable. 

The main limitations are twofold. One, the results use a single set of simulation assumptions. Showing similar results under different simulation assumptions would be helpful to better understand the robustness and potential limitations of the approach. Two, there is a certain disconnect with the simulation literature. See comments from reviewers vNt7 and kWQ2 (although I found your reply to Virtual-Taobao convincing).

Overall and given the final reviewer recommendations (three marginally above and one marginally below), this is a very borderline paper. However, the consensus view of the committee is that it would benefit from additional work before publication.

I am sorry that I cannot recommend acceptance at this stage. I do believe that some of the suggestions from the reviewers highlighted above (more diverse simulation, better grounding in current recsys simulation literature and in the field) will be useful in preparing the next version of this work.
Decision: Reject