Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors introduce a generalized feedback model in order to use delayed labels. This is very relevant in e-commerce platforms where a specific action we care about (for example buying a product) can happen after some time from the first interactions with the platform. Generally this is an important problem in e-commerce platforms. Is related basically, to attribution mechanism which is very important when one considers to build a training dataset from click-stream data. The authors compare their approach with other similar approaches on to two datasets. The authors observe improvements using their approach.
strengths_and_weaknesses: This is very important problem. The way is tackled by the authors is interesting and they try to clarify all the points of their approach. By adding and modeling the intermediate actions they can capture and propagate the delayed feedback.

There is novelty in the approach as it considers temporal distributions and is the main one.

I have some doubts of whether such an approach can be easily implemented in a real use case. No such on-line experiments are described in the paper and essentially the paper builds on such a use-case. By also checking the exact results in the supplemental material improvements there is some relatively small improvements.
 
questions: - I am not sure I got on how many actions you consider in your experiments. In on-line platforms we can have quite large number of different actions. How you deal with this?
- Did not get in the architecture when you say that you concatenate one-hot encoded y. For y \in {0, 1} what exactly you concatenate? Could you clarify?
- I see in the experiments you use a quite large batch size. How you came up with this value?
limitations: Yes the authors discuss briefly limitations and giving some idea of overcoming it, but they do not really elaborate on the type of the solution.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: In this paper, based on the assumption that post-click user behaviors are informative to conversion rate prediction and can be used to improve timeliness, the authors propose a generalized delayed feedback model (GDFM) that unifies both post-click behaviors and early conversions as stochastic post-click information. Based on GDFM, they further establish a novel perspective that the performance gap introduced by delayed feedback can be attributed to temporal and sampling gaps. They measure the quality of post-click information with a combination of temporal distance and sample complexity. The training objective is re-weighted accordingly to highlight informative and timely signals. The experimental performance results show their model’s effectiveness.
strengths_and_weaknesses: *Positive points*
1. Recommendation is one of the most important applications of machine learning.
2. The proposed method achieves good performance. 
3. The method is proposed based on a thorough analysis.

*Negative points*
1. The proposed method is not novel enough. The authors assert that previous literature concentrates on utilizing early conversions to mitigate delayed feedback problem and propose a generalized delayed feedback model (GDFM) that unifies both post-click behaviors and early conversions as stochastic post-click information. Indeed, what they tend to address can be defined as heterogeneous feedback problem and there are some relationships between different feedback types. Besides, there have already been some works such as EHCF that addressed this problem via multi-task learning and linear transformation, whose solution is very similar to this work.

[1] Chen et al. Efficient Heterogeneous Collaborative Filtering without Negative Sampling for Recommendation. AAAI 20.

2. The baselines are not enough. It is necessary to compare with some important and highly related heterogeneous collaborative filtering recommendation models such as EHCF.

3. The presentation should also be improved. For example, there is not model framework in their methodology part as their proposed method is with multi-target. I strongly suggest a figure that can illustrate their overall design. 

questions: Please answer my questions corresponding to the negative points: (1) Model novelty; (2) Missing baselines.
See the detailed comments above.
limitations: Some limitations are mentioned and I think such this algorithm-driven paper for improving user experience in recommendation system does not have negative societal impact.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper aims to alleviate the delayed feedback problem in conversion rate prediction (CVR) via proposing a new method: generalized delayed feedback model (GDFM). After clicking the item url, it might take months for a user to really purchase the item. Hence, the positive sample might be treated as negative sample in the training and this problem could bias the model.

Instead of only relying on observed labels, their key idea is to introduce other user behaviors or actions like put the item into shopping cart as an auxiliary information or proxy to better predict the conversion.

According to entropy between the conversion label and an action, they design a weight to reflect how informative the action is. Besides, they also consider the time gap and sample complexity into designing the weight for the action.

They conduct experiments over two public datasets and propose detailed evaluations.
strengths_and_weaknesses: Strengths:

(1) Since years ago, papers of recommender systems have become some sort of LEGO game -- like how to put a CNN over a DNN etc., Unlike those papers, this one really analyze the impact of user actions and clearly present the generation process in a probability graph way, which looks reasonable to me. I really enjoy this kind of papers.

(2) Using entropy between user action and conversion label to measure the information carried by the action seems reasonable. In addition, they also consider time gap and sample complexity into designing the weight for the action, which shows the comprehensiveness of their thinking. My question (also presented in the next section) is that are these factors firstly considered by this paper?

Weakness:

(1) The experiments could be further enhanced:

(a) Is there any online experiments?

(b) What is the architecture used in the experiments? there are no clues in the paper.

(2) Will algorithm 2 significantly slower the training? Where is the time complexity analysis? 


questions: (1) Using entropy between user action and conversion label to measure the information carried by the action seems reasonable. In addition, they also consider time gap and sample complexity into designing the weight for the action, which shows the comprehensiveness of their thinking. My question (also presented in the next section) is that are these factors firstly considered by this paper? did any related work consider these factors?

(2) In 167: "Since rank(M) < n requires that some rows of M are linear combination of the others,  which rarely happens in real-world problems". So you are saying rank(M) = n holds for most of real-world problems, may I ask why? 

In this specific situation, what is the physical meaning of rank(M) = n? does that mean user action and conversion has a very strong relationship? like a user must purchase the item in the shopping cart?

in 160, "We can recover p(y|x) from p(a|x) if and only if rank(Mx) = n." My opinion is that rank(Mx) = n is a very strong condition and also we don't need to 100% recover p(y|x) from p(a|x), does I understand right? 

(3) In section 3.2.2, I understand that how much information an action carry about the conversion can be measured by the entropy between them and I see H (y|a) in Equation (5). However, in 195: "Since we are estimating p(y|x) through a proxy distribution p(a|y)". May I ask where does p(a|y) come from? could you show the relation of p(y|x), p(a|y) and p(a|x) in some sort of equations, I am a bit confusing here.


limitations: In Section 2, they said: "However, the ground-truth conversion labels are available only after a long delay: A user may purchase several months later after a click event x."

Well, several months? that is a bit against my life experience. 

Does this delay of several months a common situation?

I guess it would be better to show the delay distribution from a real-world CVR system so that we know how serious this delay problem can be and how important this paper's contribution can be.

But I understand that it may not be appropriate to show that due to company's policy.

Expected to see that though!
review_title: null

Meta-review: metareview: The paper presents an approach for dealing with delayed feedback in online learning settings such as large scale recommender systems, where the delays may be significant as in the case of predicting conversion rate for online shopping where a user may spend days or weeks deciding to finally click "purchase" after first viewing listings or putting items in a shopping cart.

There were quite a range of viewpoints on this paper, including a clear rejection from RiVF.  After reading through the paper myself and the responses from the authors and other reviewers to the points raised by this reviewer, I have determined that the key argument made by this reviewer -- that the paper is not novel, due to prior work on Efficient Heterogeneous Collaborative Filtering -- is not sufficient to warrant rejection of the paper.  The GDFM and EHCF settings are fundamentally different, and while one could imagine using methods from EHCF as part of a solution in this problem space, the current paper focuses clearly on the inherent difficulty of delayed feedback.  Furthermore, as the other two reviewers note, the problem itself is highly important, difficult to solve, and the current paper puts forward interesting and effective methods that are reasonably evaluated on publicly available static datasets.  For these reasons, I am discounting the rejection and recommending acceptance.

I do agree with reviewers who suggest that online ("real world") experiments would strengthen the paper significantly.  I understand that the nature of production level / deployed industrial settings can make it difficult to exhaustively report results, but even a paragraph of anecdotal evidence or experience here would be helpful.  I believe that the paper is still worthy of acceptance, but am marking "less certain" because of this factor.


confidence: Less certain
Decision: Accept