Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary: 
This paper studies an interesting problem and proposes a novel recommendation framework to utilize user uncertainty over different item dimensions and the impact of display policy over users. To achieve the maximized expected posterior utility, it also provides a technically sound solution to derive the approximately optimal policy based on ADMM, and achieve provides insights for the commercial recommendation based on the experiments with real-world data.

Reasons for score: 
I like the idea of taking into consideration user uncertainty and display policy. My major concern is about experiments (see cons below). Hope the authors can address my concern in the rebuttal period.
 
Pros: 
1. The paper studies the most important problem for recommendation platforms: commercial benefits. The problem itself will have great impacts on real-world scenarios.
2. The proposed recommendation framework is novel for taking into consideration user uncertainty and display policy for maximizing commercial benefits. Deriving the problem into maximizing the posterior utility, this paper also proposes the ADMM-based solution to derive the approximately optimal policy. 
3. Experiments on the real-world dataset provide some practical insights about how to achieve commercial benefits for the recommendation platform.
 
Cons: 
1. Although several insightful experiments are provided, I still have several suggestions on the experiments to enhance the quality of the paper: 
(1) Although the data is from a real-world scenario, there are only results on expected utility. It might be valuable to investigate the real-world beneficial increases (if it is sensitive for companies to share the exact value, maybe it can provide the value of the increasing percentage).
(2) There is no baseline. It might because this paper utilizes expected posterior utility to evaluate the effectiveness,  this metric may not be suitable for other recommendation baselines. Thus, is it possible to select another metric as a supplement and have several comparison experiments?
2. In the introduction, it would be better to provide more details about “item dimensions” and “user uncertainty”, which seems not very clear to me. 

Questions during the rebuttal period: 
Please address and clarify the cons above 
review_title: Interesting topic but there remains some concerns on experiments

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary:

This paper proposes a recommendation approach by optimising expected utility over item display policies. The objective also accounts for the uncertainty of item representations which is particularly interesting. The ADMM method is adopted to search for an approximately optimal item display policy for both linear and nonlinear utility functions. Experiments are conducted and results presented.


Strengths:

The proposed approach incorporates the uncertainty of item representations in recommendation systems which is very interesting in my opinion.

The paper is well organised and it would be even better if lemma 1 and the proof of corollary 2 were moved to Appendix.


Concerns:

I was unable to find any empirical evaluation on test data for the proposed approach, in addition, lacking experiments that compare the proposed method with competing methods is another major concern.

How efficient is the proposed method on large item set (e.g., millions of items)? It seems very challenging to optimise the constrained quadratic program (Eq 21-22) in that case.


Minor comments:

Sec 1 first sentence: 
TicTok -> TikTok

In Eq (1) and many other equations: 
What is the "display" random variable?

Assumption 1: 
the probability of being x -> the density of being x

Page 5 first sentence: 
"Corollary 2 reveals that the posterior utility can be effectively calculated within constant time …"
I disagree with this.

Sec 4.1 second paragraph: 
"we also incorporate an entropy-based regularization term into our model."
Could this be precise?


Updates: I would like to thank the authors for their response and the updated draft. Unfortunately, I believe the above major concerns are still valid and therefore retain my original rating.

review_title: Interesting approach but lacking sufficient empirical evaluation

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper studies the problem of system utility optimization by explicitly exploiting user uncertainty regarding a specific item’s properties (e.g., certain dimensions of item representations) and the impact of the display policy. The basic idea is that when a user is uncertain about some properties of a particular item (e.g., quality of the item) or he/she might be more (or less) influenced by the system’s past recommendations, the system would have room to recommend items that maximize system utility. The idea sounds interesting and feasible to me, but the presented technical solution is unfortunately opaque and hard to follow. I do have a few of questions regarding the two assumptions and main results in this work.

First, what is the meaning of $p_{u,i}(x)$? Based on the description, it should be the representation of item $i$ from user $u$’s perspective. However, the way to estimate the parameters of this distribution is based on the system learnt embeddings in user groups and item categories. Why would those learnt embeddings or such an estimation method correspond to a particular user’s assessment on an item? For example, the user group is constructed by age and gender, does it mean all users with the same gender and age would have the same uncertainty on items, disregarding to their specific experiences of those items? I believe this item uncertainty could be better modeled with respect to individual user’s interaction history with the item (e.g., non-durable consumer goods or service might be a better target for this study). 

The second assumption is even harder to understand. Should this $p_{u,i}(display|x;\pi)$ be understood as a user’s belief of why item $x$ is recommended to him/her? But how would this estimate on the user side affect his/her behavior? The higher the probability, the more likely the user would click on the result? Based on Eq (5), it seems to be the case; but why would it be in practice? For example, when the user realizes the system is taking advantage of him/her for profit, would the user go against the recommendations? This actually leads to a bigger question regarding the problem setup in this paper: throughout the paper, the tone is the system takes advantage of users to maximize its “commercial benefits”, which is very shortsighted. This is based on a static assumption about users’ behaviors and their trust on the system, as required by Eq (5). However, once the user realizes the system’s strategy and loses his/her trust on the system, the system will lose further opportunities for making recommendations (or gaining its utilities). 

The reported experiment results are also not easy to follow. For example, Figure 1 and 2 showed the obtained utility from the learnt policy, which illustrate the learning is converging. But how good is the learnt policy? There should be some baselines for comparison purpose, e.g., at least against a random policy? Figure 3’s result isn’t a surprise: the utility function is a sigmoid function, which is a generalized linear function, over the item representation and user preference. As a result, how the learnt policy reacts to different user preference can be suggested by the linear case’s analysis. Moreover, I do not quite follow the definition of $v_0$: “record the learned expected value (w.r.t. the learned policy) of the elements with dimension 0 of the mean vectors, which is denoted as $v_0$.”   

By the way, I could not derive Eq (5) from Eq (4), and this is what I have: 
$$
p_{u,i}(x|display,\pi)=\frac{ p_{u,i}(x,display,\pi)}{ p_{u,i}(display,\pi)} =\frac{ p_{u,i}(x) p_{u,i}(display,\pi|x)}{ p_{u,i}(display,\pi)}
$$

**Acknowledgement of author rebuttal** I appreciate the detailed discussions that the authors have provided. However, my concerns still remain, and I am not convinced to improve my overall evaluation. 
review_title: Interesting angle to look at a recommendation problem; but the provided technical details and experiment results are hard to follow

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper aims to model the posterior utility of showing an item given a static display policy, where the utility function captures both: (a) uncertainty over item dimensions from the user perspective, and (b) influence of the policy on the user. It is motivated by the fact that most recommender systems don't take into account that user may be highly uncertain about value/utility in certain dimensions (e.g., color of a product) while more certain about others. The platform can use this information explicitly while optimizing for what to show.

A question that needs empirical evidence is whether uncertainty in certain dimensions have affect on recommendation outcomes. A priori it seems unclear which type of an effect this has. For instance, if a user is more uncertain about some dimensions for item 1 vs item 2, how can the platform exploit it? 


 - Isn't the below incorrect as in the very next line you point out works that capture this:

> may affect the users’ behaviors, which is crucial but always ignored.

 - The following seems untrue. There are several papers at RecSys (and workshops that focus on explicit modeling of user behavior and their response to displays, see https://recsys.acm.org/recsys20/workshops/ )

>  However, most existing works in recommendation area do not explicitly model the impact of the display policy
on the user, which might be self-limiting and incomplete.



In eq (1),  the decision variable seems to be a prob. mass function over the items. How is the recommendation performed is unclear at this point.

It is unclear what x represents in Assumption 1/eq (2). The papers say its the representation of the item. So is x a realization of item attributes from the user's perspective? Can they be interpreted as utilities/values per unit?

The normality assumptions in eq (2) and (3): what is the price of reality deviation from these?

It says v_u is the expected item representation given a display policy \pi, but unclear how this can be computed. Assumption 1 does not have any influence of display policy on it.

In eq (3), what is the random variable 'display' indicating? Is it just user imagining that the item i will be displayed? This is very unclear.

The dependence of the utility on display policy is through the vector v_u which is unfortunately not explained fully.

Lemma 1, Corollary 1 and 2 seem to follow in a straightforward manner from standard properties of Gaussians. 
The use of importance sampling to deal with a nonlinear f (which is still unclear how we get this in the first place? domain knowledge?) seems straightforward as well.

The use of ADMM procedure, while good, would also be considered standard at this point.

Overall, the paper is well written, although the ideas are relatively less novel.


 - [Minor] Rephrase below:

> .. serving as the role of user aggregation
review_title: The paper aims to model the posterior utility of showing an item given a static display policy, where the utility function captures both: (a) uncertainty over item dimensions from the user perspective, and (b) influence of the policy on the user.

Meta-review: This paper received borderline negative scores. The reviewers all agree that the proposed approach is interesting. However, there are also common concerns around the clarity of the paper, as well as lacking sufficient empirical evaluation. One reviewer also argues that technical contribution is relatively limited. The author responses were taken into account but it didn't manage to swing the reviews. Therefore, I recommend reject and wish the authors can incorporate the feedback in the revision. 
Decision: Concerns raised (can publish with adjustment)