Generalized Delayed Feedback Model with Post-Click Information in Recommender Systems
Authors: Jia-Qi Yang
De-Chuan Zhan
Venue: NeurIPS 2022 Accept (2022)
Keywords: recommender systems
delayed feedback
conversion rate prediction

Abstract: Predicting conversion rate (e.g., the probability that a user will purchase an item) is a fundamental problem in machine learning based recommender systems. However, accurate conversion labels are revealed after a long delay, which harms the timeliness of recommender systems. Previous literature concentrates on utilizing early conversions to mitigate such a delayed feedback problem. In this paper, we show that post-click user behaviors are also informative to conversion rate prediction and can be used to improve timeliness. We propose a generalized delayed feedback model (GDFM) that unifies both post-click behaviors and early conversions as stochastic post-click information, which could be utilized to train GDFM in a streaming manner efficiently. Based on GDFM, we further establish a novel perspective that the performance gap introduced by delayed feedback can be attributed to a temporal gap and a sampling gap. Inspired by our analysis, we propose to measure the quality of post-click information with a combination of temporal distance and sample complexity. The training objective is re-weighted accordingly to highlight informative and timely signals. We validate our analysis on public datasets, and experimental performance confirms the effectiveness of our method.

Generalized Delayed Feedback Model with Post-Click Information in Recommender Systems
Jia-Qi Yang De-Chuan Zhan'
State Key Laboratory for Novel Software Technology Nanjing University, Nanjing, 210023, China rangja@lamda. .jj.. edu..uu..uu....e....m.. zhandcōn edu.
Abstract
Predicting conversion rate (c.g., the probability that user will purchase an item) is fundamental problem in machine learning based recommender systems. How- ever, accurate conversion labels are revealed after long delay, which harms the timeliness of recommender systems. Previous literature concentrates utilizing early conversions to mitigate such a delayed feedback problem. In this paper, we show that post-click user behaviors are also informative conversion rate prediction and can be used to improve timeliness. We propose generalized de- layed feedback model (GDFM) that unifies both post-click behaviors and early conversions stochastic post-click information, which could be utilized to train GDFM in a streaming manner efficiently. Based on GDFM, we further establish a novel perspective that the performance gap introduced by delayed feedback can be attributed temporal gap and a sampling gap. Inspired by our analysis, we propose to measure the quality of post-click information with a combination of temporal distance and sample complexity. The training objective is weighted accordingly to highlight informative and timely signals. We validate our analysis on public datasets, and experimental performance confirms effectiveness of our method.
Introduction
Conversion rate (CVR) prediction has become a core problem in display advertising with the prevalence of the cost-per-conversion (CPA) payment mI]e With CPA, advertisers bid for predefined user behaviors such as purchases or downloads. Similar demands also arise from online retailers that aim to improve sales volume which need to recommend items with high conversion rates to users. However, conversions (e.g.. purchases) may happen after long delay [5], which leads to delay in conversion labels. The adverse impact of such a delayed feedback problem becomes increasingly significant with rapidly changing market, where the features of users and items are updated every second. Thus, how to timely update the CVR prediction model with delayed feedback attracts much attention in recent years [.II.
Since conversions are gradually revealed after click events, previous literature concentrates on utilizing available conversion labels that reveal earlier. Inserting early conversions into the training data stream will also introduce many fake negative samples (6), which will eventually convert but have not converted yet. To mitigate the adverse impact of fake negatives, Chapelle [5] proposes a delayed feedback model (DFM) to model conversion rate along with expected conversion delay, then the model is trained to maximize the likelihood of observed labels. The delayed feedback model further improved by introducing kernel distribution estimation method
[12] and more expressive
neural networks [10 13 [4] However, the need to maintain a long time scale offline dataset impedes
Chuan Zhan is the corresponding author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

training efficiency of DFM based methods in large scale recommender systems [6]. To enable efficient training in a streaming manner, Ktena et al. [6] proposes to insert a negative sample once click sample arrives and re-insert a duplicated positive sample once its conversion is observed. The fake negative samples are supposed to be corrected by an importance sampling [5] method. Importance sampling based methods are further improved by adopting different sampling strategies 779 16.
Besides the conversion labels, many events related with conversion exist in real-world recommender systems 17419. For example, after clicking through an item, a user may decide to add this item to shopping cart, Statistical data reveals that such post-click behaviors have strong relationship with conversions: About 12% of items in a shopping cart will finally be bought, while the proportion less than 2% without entering a shopping [7]. Intuitively. utilizing post-click behaviors can potentially mitigate the delayed feedback problem in CVR prediction since the time delay of post-click behaviors is usually much shorter than conversions. However, the corelations between post-click behaviors and conversions are far more complex than only considering conversions, and the problem is further complicated by the streaming nature of training with delayed feedback.
In this work, we propose Generalized Delayed Feedback Model (GDFM) which generalizes the delayed feedback model (DFM) [ unify post-click information and early conversions. Based on GDFM, we establish a novel view on learning with delayed feedback. We argue that training using post-click information in the delayed feedback problem intrinsically different from traditional learning problems from three perspectives: (i) Estimating conversion rates via post-click actions requires more samples than using conversion labels directly, which highlights the importance of sample complexity. (ii) The post-click actions bring information of past distributions, which incurs a temporal gap. (iii) The signals provided by post-click actions are highly stochastic and lead to large variance on the target model during training, which may instead hinder the performance of the conversion rate prediction. Inspired by our analysis, we propose to (i) measure the information carried by a post-click action with conditional entropy, which we empirically show is related to the sample complexity of estimating the conversion rate; (ii) measure the temporal distribution gap by time delay; (iii) stabilize streaming training with a regularizer trained on past data distribution. We conduct various experiments on public datasets with delayed feedback under a streaming training and evaluation protocol. Experimental results validate the effectiveness of our method. To summarize,
We propose a generalized delayed feedback model (GDFM) support various user behaviors beyond conversion labels and arbitrary revealing times of user actions.
From novel perspective, attribute the difficulty of efficiently utilizing user behaviors to a sampling gap and a temporal gap in delayed feedback problems.
Based on our analysis, we propose a re-weighting method that could selectively use the user actions that are most informative to improve performance. Our method achieves stable improvements compared to baselines.
2 Background
In recommender systems, the data stream is collected from user behavior sequence. Without loss of generality, we take an E-Commerce search engine as an example, while a similar procedure also holds for display advertising[2 2 3 and personalized recommender(4). When user searches for a keyword, the search engine will provide several items for this user. Here, a (user-keyword-itmm) tuple called an imprexsion, which corresponds to a sample x. After viewing an item's detail page, the user may purchase it. The probability that a user will buy after clicking is called conversion rate (CVR). Conversion can be any desired user behavior, such as registering an account or downloading game. Without loss of generality, we only consider one type of conversion in the rest of this paper, while our method can be applied to multi-class case[20] without modification. If an impression sample is finally converted, it will be labeled y 1 and y of x corresponds to p(y = =|xx).
0 otherwise. The conversion rate
The distribution p(x) and p(y[x) changes rapidly in real-world recommender systems. For example, when a promotion starts, the conversion rate may increase steeply for items with a large discount, and such promotion happens every day. The CVR model has to be updated timely to capture such distribution change. However, the ground-truth conversion labels are available only after a long delay: Many users purchase several days after a click event x(5). Usually, sample will be
2

---

Weight
[ 1|5.y = 1,6,/ 1
Revraled actions
g 1.6))
in heads for in accion
sn cevealing morks
Cbj
Fay
Prodicted probabillries
90000
e(a, 0)
4(0
Embodding of [x,,]
Input
(a) Proxy feedback loss
(b) Network architecture
Figure 1: (a) The proxy feedback loss on the jth action ay. (b) The neural network architecture of GDFM.
labeled as negative after a long enough waiting time, c.g., 30 days[5], this delay time of conversion label is denoted as by. Besides conversion labels, post-click actions (behaviors) are denoted as ay.J E {I,... m). ). Each action aj is paired with revealing time 6,. which means the value of aj determined after óy delay. For example, put an item into the shopping cart 10 minutes (6)) after clicking. true, this action is labeled as a, 1, and a, 0 otherwise. There be several revealing times for single type of action, e.g.. we may choose to reveal the shopping cart information at 10 minutes, 30 minutes and hour, they treated different actions since the revealing time is different. It is noteworthy that revealing the conversion labels is also treated as a type of action. To capture the characteristic that the data distribution changes along with time, the data distribution is denoted by px(, p²(()'' (y|x) time t. Our goal is to predict p (ylx) at time by training model qa (y)x). where & denotes model parameters. Without data distribution shift, it straightforward optimize likelihood by sampling from pt. However, in the delayed feedback problem, the latest available labels have a Sy delay. Thus, simply training using available labeled data will lead to qa (y xx) = p2-6y (y (y|x) instead.
query the database to whether a user has
3 Generalized Delayed Feedback Model
We propose the following generalized delayed feedback model describe the relationship between data features x, conversion labels y, post-click actions revealing time of actions 5, and data distribution at time
(x 0, (ylx)p(a(xx y. 8)p(()p' (x)
Our formulation unified previous delayed feedback models 12 while enable more general feedback and revealing times. p² (y|x) ) is the target CVR distribution, we assume that p (ylx) is independent to user actions a, which enables us to extend the existing CVR prediction framework seamlessly. To model the post-actions, we introduce post-action distribution p(a|x,) y,), which 8),
depends on sample features x, the conversion label y, and the revealing time We assume the post- action distribution p(a]x,) y, fixed or changes much slower than p (ylx), so that the post-action distribution does not depend on time t. This assumption is critical for learning under delayed feed- back since the relationship between post-actions and conversion should be stable so that post-actions can be informative. Previous literature also relies on similar assumptions implicitly, for example, a predictable delay time of conversion or loss function that depends on stable post-action distribution in our formulation (79)) Our formulation decouples the post-action distribution and conversion distribution, which enables us to formulate this assumption explicitly and conduct further analysis based on this assumption. An alternative but more intuitive definition is to define pd (xx y, a, 5) = ('(a (, o) ()(y) x, a, (x))(x),), which requires to sum out a and 8 to make a pre- diction. We discuss this approach in the supplementary material. We further introduce revealing

---

time distribution of post-actions p()), which is independent of other variables. is also known as the elapsed time in previous literature [ g The revealing time enables us to inject action information at controllable time 8, which is typically less than b, so that the model can be updated much earlier. We introduce a streaming training method of GDFM in the next section.
3.1 Training GDFM
We introduce an action prediction model (a(a (. 6) to estimate the action distribution p(a(x, y. 6), the CVR prediction model is denoted as (y(x), which will be used to predict the conversion rate y' (y|x). In the data stream, not all the information in GDFM defined in Eq. ( is available at that same time, so we propose a streaming training method for GDFM that can utilize available information in the data stream. According to the availability of y, there are several different cases.
We assume the current timestamp
When x, is available The action prediction model gg(aa x, 5) can be trained with following loss when a, x and y are available.
Laction E, log qo (a,//x, y,
Note that with the assumption that p(a, y, 5,) is invariant with respect to i, loss Eq. will be minimized when p(a, x, y, v, go (a, x, y. 6,).
When y, x is available When we have the ground-truth label of y, we are able to update the CVR model directly by minimizing cross entropy.
Ls, -E,- log qar-Ey (y)
Note that this loss is minimized when p² 6y (y) x) ast (y (x).
When a, x available Since the delay times of user actions (8,) are smaller than the conversion label delay 6g. we can update the CVR model once a user action is observed. When we observe the jth user action a, at revealing time 5j. we can update the prediction model (y (x) using following proxy feedback loss:
C/, = EE, log Logg(a, 8,) = -E, , log Σ
(y(xx)
which is depicted in Figure. | (a). However, it is unclear whether minimizing Eq. learning p' pxx))).
(4) will help
3.2 Narrowing the delayed feedback gap with post-actions
The training objective of the proxy feedback loss can be viewed as a multi-task training problem with different actions, which is also explored by Hou al. [10] and Li al. [. However, learning with delayed feedback is different from general multi-task learning problems without delayed feedback. In learning with delayed feedback, since we are training on different data distribution at different
times, the tasks are intrinsically conflicted, which may be harmful to the target [10] 21 its effect the following sections.
We analyze
3.2.1 Post-actions can be informative
Since in GDFM we are using post-actions from t 5, instead of t 8y, we have a chance to do better. But will optimizing Eq. (4) improve the performance of q(y xx)? Without loss of generality, we consider an action {a ). and p(a(x) Ey p(y x)y( a a|x,y) y) (condition d omitted), estimate p(a)x) accurately and k: 2 n, have following results:
y {y1,--. -.-, yn Assuming
Lemma 3.1. Use a matrix Mg E Ø denote the conditional probability p(a|x, y). where (Mx);; = p(a = aj y = i). We can recover p(y(x) from p(a(x) if and only rank (Ma)) =
Proof sketch: p(aax) is linear transformation from p(ylx) and the transform matrix is given by Mg. By construction, solution exists and rank(Mx) guarantees that the solution is unique.

---

Lemma. 3.1 highlights the benefits of utilizing post-actions: if the relationship between post- action and the target is predictable (we can train model g(a)x, y) to approximate p(a|x. y)) and informative (rank(M) (M) n1) we can recover the target distribution even without the ground-truth label y. Since rank(M) < n requires that some rows of M are linear combination of the others, which rarely happens in real-world problems, we assume the assumption holds true in the next section. We discuss the case that this assumption fails in Section. [.2.4, However, even if rank (M) = n, recovering p(y|x) is still challenging. Because we are estimating p(a(x) using samples from p(a[x), and the difficulty of estimating p(y|x via estimating [aaxx) also depends on properties of p X, y).
3.2.2 Measuring information carried by actions
Not all actions carry the same amount of information about p(y|x). For example, action may carry zero information about y if it is constant and irrelevant to y; it may carry full information if always equals y. The general relationship between a and y is far more complex, which lies between non-informative and fully informative. We want to estimate the information quantity of each action to utilize actions more efficiently. To this end, propose to conditional entropy to measure information carried by actions. The conditional entropy of y given a can by calculated by
H(yla) -E p(y, a) log p(a)
(5)
p(y,a)
a.y
Conditional entropy measures the amount of information needed to describe y given the value of a. When a is non- -nnimmformetiv to y. that is p(a. y) = p(a)p(y), we have Hylaa) = Hy). which is the maximum value of H (y la); when a determines value of y. that is, there is function f(a).
we have (y]a) = O. Since the scale of (y a) only depends on p(y), this information metric is comparable among different a, which is desired property.
Empirically, we found that conditional entropy also related to the sample complexity of estimating p(y| x) via p(a(x)
Ey p(a] x, y y))(((x) (6 omitted) as in Eq. 44). In learning with delayed feedback, we are sampling from different dis- tribution p' at different time stamp f, which indicates that to perform well, we should be able to learn fast. Thus, analy- sis that is based on large amount of i.i.d samples from a fixed distribution can't t capture the difficulty of learning with delayed feedback. Property testing [22) considers sample com- plexity of learning distributions. Specifically, distinguishing between two discrete distributions p and with probability at least 1 - requires 52( (v.g) log 2) samples from and p.
10
10-3
10
Mirls)
Where Drv is the total variance distance which is defined by -
Figure 2: Conditional entropy and transformed distance.
Div(p, q) = Σ, (p(i)-q(i), Since we are estimating p(y (x) through proxy distribution p)aly), the sample complexity depends on how close the transformed distribution p p(alx) is.
Empirically, found that using p(a(x) will make p(y(x) more difficult to learn, that is, the distance of two distribution Pr (y x) and p2 (y(x) will decrease when they are transformed to p1 (a (a)x) and Pz(alx) x) correspondingly. So the difficulty of learning p(y) x) via p(a xx) depends the change of distribution distance incurred by the stochastic transformation p(aly) y)) We empirically investigate the relationship between the change of distribution distance and conditional entropy by Monte Carlo sampling, the results in Figure. 2 indicate that as the conditional entropy increases, the transformed distribution distance decreases exponentially, which motivates the following information weights
Winto (p(to y)) (yrla))
where a is a arper-parameter. Eq. (6) roughly measures the reciprocal of sample complexity of 63 learning -6 (y xx) via ( x).. ). so that actions are weighted based on their effective information
carried by each sample.
3.2.3 Measuring temporal gap
Besides the sample complexity of estimating -² (y/xx) via action distribution, the intrinsic temporal gap between p' and p'- also influence the predicting performance. Even if we have unlimited
5

---

samples from p'=63 we are only able to recover p² (ylx) instead of p' (y)x). So we also need to take the revealing time új into consideration. However, we can't estimate the gap hetween p'=6, and pf accurately since the exact distribution is unknown and changes along with time. Empirically (Figure. 3) we found that the gap measured by KL divergence tends to increase as the time gap increases, So we propose the following temporal weight
Wrime (ōz)
which measures the gap between p' and p²=03_ B a apeer-parameter that reflects the expected changing speed of data distribution along with time, larger value of 3 indicates that we discard more information from stale data.
Overall, we introduce a weight on loss Eq. 4 as follows:
W (p(paa y), dj) (pnto W)m (6,)
For simplicity, our w does not depend on x, the extension to x dependent weights is straightforward. The weight w, can be decomposed into two components, the first component Winfo measures how informative action ay is to the target y. The second component Wime measures the gap between p and p 4; The procedure of calculating weight vector w E R" is summarized in Algorithm. 2
Algorithm Streaming training of GDFM
Algorithm 2 Estimating wj
1: for {x.y y, a, 8) in data stream do
Input:
Dataset {y,a} Parameter: a, B.
2: for yx y, a} that y is unlabeled do
3: Update ( (yyee (x) with Eq. (10)
4: end for
1: Estimate p(a. y) by counting.
5: for yx, a) that y is labeled do
Calculate Hy a) with Eq. 5
6: Update qo (a|x, y, 8) with Eq. 2
3: Calculate Winfo with Eq. (6).
7: Update 4g- sy (y(x) with Eq.
Calculate Wime with Eq. 7
8: end for
5: Calculate weights w with Eq. 8
9:
end for
Normalize w have mean (w)
3.2.4 Reducing variance by delayed regularizer
Estimating probability with sampling will incur instable variance during stream training, which may harm performance. Since we are estimating using a stochastic proxy p(alx), the variance will be even higher. To reduce variance and stabilize training, we introduce a regularizer loss Eq. (9) that constraints update step during training, where grr-,, (y(x) trained with loss Eq. 3
Lreg KL 6, (y x)] go (y(x))
Besides reducing variance during training, another critical function of Creg is to make GDFM safe to introduce more actions. The analysis in Section. 3.2.1 shows that if an action is informative, we can learn p(y) x) through p(a x). However, if rank(M < n (typically because
n) or the information carried by p(a(x) is too weak to recover p(ylx) as analyzed in Section. 3.2.2 GDFM may fail to learn p(y x) because of interference from action signals. The regularizer Ioss Lreg introduces the ground-truth truth labels of y from p¹ p' which guarantees that the performance learned by GDFM will not be much worse than p - (the Vanilla method in experiments). In this perspective, frey make it safer to introduce various post-click information into our training pipeline without worrying about sudden performance drop, which is critical in production environments. The overall loss function with revealing time 5, is
Laj f2:wwyk wyC's + XLreg
(10)
The streaming training procedure of GDFM is summarized in Algorithm,
4 Experiments
4.1
Datasets and data analysis
We use two large-scale real-world datasets: 1) Criteo Conversion Logs is collected from an online display advertising service within 60 days and consists of about 16 million samples with conversion
hattps://latte criteo. wttpa//ww...o////////////////or.

---

2) Taobao User Behavior is a subset of user behaviors on Taobao collected within 9 days and consists of more than 70 million samples and 1 million users Taobao dataset provides user behaviors within (page-view, buy, cart, favorite).
labels and timestamps(3] SI.
Day
10
20
30
Cart/Pay
Fav/Pay
0.06
0.020
0.3
0.05
divergeen Bergende
0.20
0.015
A(Y)As)
0.04
0.15
0.010
0.03
0.10
0.1-
0.02
0.005
0.05
0.0
0.01
0.00
0.000
10
20
30
Day
Figure 3: (Left) Conditional entropy (orange) and temporal gap (blue) changes along with time in Criteo dataset. (Right) In Taobao dataset, the conditional distribution p(aly) (bar) is stabler than conversion rate (line).
We conduct data analysis on the datasets to validate our assumption on GDFM. The results are depicted in Figure. First, we investigate the change of conversion rate distribution p(y[x). Since exact p(y xx) is not available, we train a CVR model on the first 30 days to estimate the conversion rate at the 30th day, which is denoted as pd (y x). Then we fine tune this model day by day to estimate the conversion rate at p" (y) here denotes the tth day. In this way, we can estimate the temporal gap between pd (ylx) and p' x) by their Kullback-LLeible divergence (x Il''). Figure. (Left) we can see that the temporal gap grows along with time, which necessitate the analysis of time dependent conversion distribution p¹ p'(y() in Section. 3 Secondly, we investigate the information quality of actions with conditional entropy observed conversion at revealing time as post-action As, and vary the revealing time from 0 to 30 days. We can infer from Figure. 3 (Left) that the information carried by this action steadily increase (the conditional entropy decrease) along with time, which is intuitive that the latter actions are more informative. Thirdly, investigate the relationship between post-click actions and conversions. We plot the conversion rate and the proportion cart and favorite actions in converted samples (which corresponds to p(a 1/y 1)) in the Taobao dataset in Figure. (Right). We can see that the action distribution p(a = y = 1) is stabler comparing to conversion rate, which conforms our assumption of stable action distribution in Section. 3
proposed in Section. [.2.2] Specifically, use the
4.2 Evaluation
Streaming evaluation We follow a streaming evaluation protocol proposed by Yang et al. [8] Specifically, the datasets are split into pretraining and streaming datasets. An initial conversion rate prediction model as well as auxiliary models (e.g., the importance weighting model ESDFM and action distribution model 4s in GDFM) are trained on this pretraining dataset to simulate stable state after long time of streaming training in real-world recommender systems. Then the models are evaluated and updated hour by hour in the streaming dataset. Each method is trained with the available information at the corresponding timestamps. Following practice in 1, [6), report the receiver operating characteristic curve (AUC), precision-recall curve (PR-AUC) and negative log- ligeleklohood (NLL). The metrics are calculated within each hour. We report the average performance on the streaming dataset.
Baseline methods evaluate the performance of GDFM, we compare with the following methods: 1) Pretrain: The CVR model is trained on the pretraining dataset, then fixed in streaming evaluation. The following methods start with this pre-trained model at the beginning of the streaming evaluation. 2) Vanilla: Wait for ā, time, then use conversion labels to train the model. 3) Oracle: Use conversion labels to train without delay. The oracle method corresponds to the upper bound of performance
Id= Id=11a1geee
"code available at htttttpp///gtttuu...///tt//tt//t wwwpggggg/dmmmmmmmg _ipp222

---

with feedback delay. We compare two representative importance sampling methods, 4) FNW A sample labeled as negative on arrival, and a duplicate inserted once its conversion observed. The fake negative weighted (FNW) loss adjusts the loss function; 5) ES-DFM[8]: After waiting for a predefined time, sample is labeled as negative conversion has not been observed. If sample converts after the waiting time, duplicate is inserted. The loss function is adjusted by the ES-DFM loss. We also compare with multi-task learning method based on DFM. 6) MM-DFM Treating predicting the observed conversion label multi-task learning problem, the tasks are optimized jointly using streaming data.
Table 1: Performance of compared methods on Criteo and Taobao dataset. The Pretrain method corresponds 0%, and the Oracle method corresponds 100% their absolute performance is in parentheses. We also report the standard deviation of each method with 5 different random seeds.
Criteo
Taobuo
Method
AUC
PR-AUC
NLL
AUC
PR-AUC
NLL
Pretrain
0.05((0.815)
0.0%())
0.0%())414)
(.073)
0.0%((0.054)
(0008))
Vanilla
25.2:10.2%
25.11..22%
24.6.1..1%
58.9110.8%
61.7+11.7%
46.0%11.9%
F
62.0:10.3%
03.1+
1..3+
39.6:10.8%
-3.75=
-3%1=12%
ES-DFM 8
71.4-0.5%
1.13
1..2%
62.6:10.7%
19.6:13.2%
-.14=:122%
MM-DFM[10]
6..7
39.2:16.7%
54.211..6%
59.8113.6%
62.1
10.5%
GDFM(ours)
74.910.7%
68.11:166%
72.4100.6%
7.441055%
80.711.9%
49.61.3.1%
Oracle
10095(0.841)
100%((0.64))
(0.399
(0074))
100%)(0.063)
(000((0
Implementation Following g 91. we discretize and treat numerical features the same as categorical features in the Criteo dataset. Since the Taobao dataset does not provide user features,
the last 5 user behaviors as features 23). Inspired by Weinherger et al. [24], we hash user ID) and item ID into bins and an embedding to represent each bin. We the same architecture for all the methods to ensure a fair comparison. All the methods are carefully tuned. We use a = 2, B = 1, X = 0.01, Ir
10 for GDFM. The network structure and procedure calculate the proxy feedback loss Eq. (4) used by GDFM is depicted in Figure. (b).
The performance is reported in Table. ① Following 9 16], we report the relative improvement of each method to the performance gap between the Pre-trained model and the Oracle model. From the results, we can infer that, 1) GDFM performs significantly better than compared methods. 2) It is noteworthy that FNW and ES-DFM brings a significant negative impact to NLL on the Taobao dataset, which is caused by the fake negative samples introduced by them; 3) The performance of FNW and MM-DFM has a more considerable variance compared with ES-DFM and GDFM (especially NLL Taobao), since ES-DFM uses fixed weighting model to re- weight its loss, which has similar effect to GDFM's explicit regularizer that can stabilize training; 4) On the Criteo dataset, GDFM outperforms other methods without using post-click information other than conversion labels, which indicates that our information measure is effective for early conversions; 5) On the Taobao dataset, GDFM also utilizes post-click information such cart and favorite. The results indicate that introducing rich post-click information into GDFM can improve the performance of CVR prediction.
4.3 Experimental analysis of GDFM
To further investigate the source of performance gain of GDFM, we construct several experiments First, we remove the effect of information weights and the regularizer loss by setting the coefficients to zeros correspondingly. The results reported in Table. (2) indicate that the performance of GDFM a combined effect different components. Without a and X, the performance drops significantly, which indicates that sample complexity and training variance are influential factors in the Criteo dataset. Dropping 3 has relatively little influence to performance, which indicates that the temporal gap in the Criteo dataset has weaker influence.
Secondly, to validate the capacity of GDFM to deal with more actions, we construct an additional user action in the Criteo dataset. This action is set to the same as with probability p. and set - y with probability p. this way, we can investigate how the information mea- sure influences the performance of GDFM. We report the results with different p in Table. (3)
8

---

Table 2: Effect of
on Criteo.
Table 3: Adding more actions.
Method
AUC
PR-AUC
NLL
Entropy
AUC
PR-AUC
NLL
w/o
0,8333
0.6244
0.3983
0.500
0.529
0.835
0,631
0.396
w/o 3
0.8343
0.6314
0.3964
0.800
0.394
0.836
0.634
0.394
w/o A
0.8339
0.6226
0.3988
0.900
0.264
0.839
0.638
0.392
GDFM(full)
0.8349
0.6311
0.3960
0.950
0.166
0.840
0.641
0.391
When p = 0.5, the newly inserted action does not contain information about y, and the perfor- mance does not decrease, which indicates that GDFM can extend to more actions safely; When >> 0.5, the action becomes more informative, and the performance of GDFM steadily improves, which indicates that GDFM can utilize the information carried by actions efficiently; When p 0.95, the action highly informative y. the performance approaches Oracle, which indicates that GDFM can efficiently extract information from high-quality action features.
0.8
weights
Waime
Thirdly, we plot the weigths Wintos Wime and w correspond- ing to different revealing time 6, with in other experiments. We can infer from the Figure. 4 that the information carried by actions increases as the revealing time increases; 2) the temporal gap entarges as the revealing time increases; 3) the combined informa- tion weights w reaches its maximum value on the 7th day, which indicates that the information revealed on the 7th day is most informative to conversion prediction. Informa- tion revealed other times still has negligiie weights that influence training.
Winb
2, =
0.4
10-1
102
10a
107
107
6y (hours)
Figure 4: Information weights on the Criteo dataset.
5 Related work and discussion
deal with the delayed feedback problem. The following methods predict the probability of whether a user converts before a predefined waiting time However, how to define the strategy of duplicating training samples varies among previous literatures 1 16]. On the contrary, the revealing time distribution explicitly defined in GDFM, which indicates that each sample should be duplicated once at a revealing time. The information weight w in GDFM is effectively adjusting the revealing time with the aid of GDFM's information measure.
Delayed feedback model (DFM) S firstly introduces survival analysis 23 26
To the best of our knowledge, we are the first to analyze the property of learning with delayed feedback with a time-varying distribution. Existing analysis is stable (unlimited samples from 5). or the action distribution p(a[x) can be estimated accurately [ 9 [6]. Since p'(x. y) indeed varies along with time (otherwise streaming training is unnecessary), ), assuming a static p(x. can not capture the property of the delayed feedback problem; The action distribution p(a(x) Ey p(a|x, ) x) also changes following y (y (x). Thus, GDFM can be better model for understanding and analyzing the delayed feedback problem.
T6 assumes the distribution
Delayed feedback problem has also attracted attention from bandit learning communities 27 22] where the objective is to minimize regret in decision problem. Vernade et al. [27 introduces similar idea of using intermediate observations before getting rewards and shows that such intermediate observations improve regret, which agrees with our analysis from different perspective.
limitation of GDFM that the training cost grows linearly with the number of different revealing time oj. which may be a potential bottleneck in large-scale streaming training.

---

Conclusion
We present an analysis of the delayed feedback problem based on the assumption that the relationships between post-click behaviors and conversions are relatively stable. Our results indicate that to improve the performance of learning under delayed feedback, we should utilize post-click information as complements. Therefore, we propose generalized delayed feedback model to incorporate general user behaviors and a re-weighting method to utilize behavior information efficiently. Experiments on public datasets validate the effectiveness of our method.
Acknowledgements
This work is supported by NSFC (61921006).
References
[1] Olivier Chapelle, Eren Manavoglu, and Rómer Rosales. Simple and scalable response prediction for display advertising. ACM Trans. Intell. Syst. Technol., 5(4):61:1-61:) 2014.
[2] Quan Lu, Shengjun Pan, Liang Wang. Junwei Pan, Fengdan Wan, and Hongxia Yang. A practical framework of conversion rate prediction for online display advertising. In ADKDD, pages 9:1 9:1-9:9. ACM, 2017.
[3] Kuang-chii Lee, Burkay Orten, Ali Dasdan, and Wentong Li. Estimating conversion rate in display advertising from past erformance data. In KDD, pages 768-776. ACM, 2012.
[4] Dietmar Jannach and Michael Jugovac. Measuring the business value of recommender systems. ACM Trans. Manag. Inf. Syst., 10(4):16:1-11:33, 2019.
[5] Olivier Chapelle. Modeling delayed feedback in display advertising. Sofus A. Macskassy, Claudia Perlich, Jure Leskovec, Wei Wang, and Rayid Ghani, editors, KDD, pages 1097-1105.
ACM, 2014.
[6] Sofia Ira Ktena, Alykhan Tejani, Lucas Theis, Pranay Kumar Myana, Deepak Dilipkumar, Ferenc Huszár, Steven Yoo, and Wenzhe Shi. Addressing delayed feedback for continuous training with neural networks in CTR prediction. In RecSys, pages 187-195. ACM, 2019.
[7] Shota Yasui, Gota Morishita, Komei Fujita, and Masashi Shibata. A feedback shift correction in predicting conversion rates under delayed feedback. In WWW, pages 2740-2746. ACM / IW3C2, 2020.
[8] Jia-Qi Yang, Xiang Li, Shuguang Han, Tao Zhuang, De-Chuan Zhan, Xiaoyi Zeng, and Bin Tong. Capturing delayed feedback in conversion rate prediction via elapsed- time sampling. In AAAI, pages 4582-4589. AAAI Press, 2021.
[9] Siyu Gu, Xiang-Ron Rong Sheng, Ying Fan, Guorui Zhou, and Xiaoqiang Zhu. Real negatives matter: Continuous training with real negatives for delayed feedback modeling. In KDD, pages 2890-2898. ACM, 2021.
[10] Yilin Hou, Guangming Zhao, Chuanren Liu, Zhonglin Zu, and Xiaoqiang Zhu. Conversion prediction with delayed feedback: A multi-task learning approach. In ICDM, pages 191-199. IEEE, 2021.
[11) Haoming Li, Feiyang Pan, Xiang Ao, Zhao Yang, Min Lu, Junwei Pan, Dapeng Liu, Lei Xiao, and Qing He. Follow the prophet: Accurate online conversion rate prediction in the face of delayed feedback. In SIGIR, pages 1915-1919. ACM, 2021.
[12] Yuya Yoshikawa and Yusaku Imai. A nonparametric delayed feedback model for conversion rate prediction. CoRR, abs/1882.00055, 2018.
[13] Yanshi Wang, Jie Zhang, Qing Da, and Anxiang Zeng. Delayed feedback modeling for the entire space conversion rate prediction. CoRR, abs/2011.11826, 2020.
10

---

[14] Yumin Su. Liang Zhang, Quanyu Dai, Bo Zhang, Jinyao Yan, Dan Wang, Yongjun Bao, Sulong Xu, Yang He, and Weipeng Yan. An attention- based model for conversion rate prediction with delayed feedback via post-click calibration. In IJCAI, pages 3522-3528. ijcai.org, 2020.
[15] Surya T Tokdar and Robert E Kass. Importance sampling: a review. Wiley Interdisciplinary Reviews: Computational Statistics, 2(1):54 2010.
[16] Yu Chen, Jiaqi Jin, Hui Zhao, Pengjie Wang, Guojun Liu, Jian Xu, and Bo Zheng. Asymptoti- cally unbiased estimation for delayed feedback modeling via label correction. In WWW, pages 369-379, ACM, 2022.
[17] Hong Wen, Jing Zhang, Yuan Wang, Fuyu Lv, Wentian Bao, Quan Lin, and Keping Yang. Entire space multi-task modeling via post-click behavior decomposition for conversion rate prediction. In SIGIR, pages 2377-2386 ACM, 2020.
[18] Hong Wen, Jing Zhang, Fuyu Lv. Wentian Bao, Tianyi Wang, and Zulong Chen. Hierarchically modeling micro and macro behaviors via multi-task learning for conversion rate prediction. In SIGIR, pages 2187-2191 ACM, 2021.
[19] Han Zhu, Xiang Li, Pengye Zhang, Guozheng Li, Jie He, Han Li, and Kun Gai. Learning tree-based deep model for recommender systems. In KDD, pages 1079-1088. ACM, 2018.
[20] Junwei Pan, Yizhi Mao, Alfonso Lobos Ruiz, Yu Sun, and Aaron Flores. Predicting different types of conversions with multi-task learning in online advertising. In KDD, pages 2689-2697. ACM, 2019.
[21] Michael Crawshaw. Multi-task learning with deep neural networks: A survey, CoRR, hbss/200.09799 2020.
[22] Clément L Canonne. A survey on distribution testing: Your data is big. but is blue? Theory of Computing, pages 1-100, 2020.
[23] Han Zhu, Daqing Chang, Ziru Xu, Pengye Zhang, Xiang Li, Jie He, Han Li, Jian Xu, and Kun Gai. Joint optimization of tree-based index and deep model for recommender systems. In NeurlPS, pages 3973-3983 2019.
[24] Kilian Q. Weinberger, Anirban Dasgupta, John Langford, Alexander J. Smola, and Josh At- tenberg. Feature hashing for large scale multitask learning. In ICML, volume 382, pages 1113-1120. ACM, 2009.
[25] Ping Wang, Yan Li, and Chandan K. Reddy. Machine learning for survival analysis: A survey. 2019.
ACM Comput. Surv.,
[26] Nicola Barbieri, Fabrizio Silvestri, and Mounia Lalmas. Improving post-click user engagement on native ads via survival analysis. In WWW, pages 761-770. ACM, 2016.
[27] Claire Vernade, András György, and Timothy A. Mann. Non-stationary delayed bandits with intermediate observations. In ICML, volume 119, pages 9722-9732. PMLR, 2020.
[28] András György and Pooria Joulani. Adapting to delays and data in adversarial multi-armed bandits. In ICML, volume 139, pages 3988 3997. PMLR, 2021.
[29] Ciara Pike-Burke, Shipra Agrawal, Csaba Szepesvári, and Steffen Grünewälder. Bandits with delayed, aggregated anonymous feedback. In ICML, volume 80, pages 4102-4110, PMLR, 2018.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] see Section.
11

---

(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes] See supplementary,
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or as a URL)? [Yes] Code and detailed information in supplementary. Data urls at footnote 3
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Details in supplementary.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes] Details in supplementary.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes] Licenses are in the urls.
(c) Did you include any new assets either in the supplemental material as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
12


Reviews:
[{"review_id": "ylXW3eGgez_", "reviewer": "Reviewer_RiVF", "summary": "In this paper, based on the assumption that post-click user behaviors are informative to conversion rate prediction and can be used to improve timeliness, the authors propose a generalized delayed feedback model (GDFM) that unifies both post-click behaviors and early conversions as stochastic post-click information. Based on GDFM, they further establish a novel perspective that the performance gap introduced by delayed feedback can be attributed to temporal and sampling gaps. They measure the quality of post-click information with a combination of temporal distance and sample complexity. The training objective is re-weighted accordingly to highlight informative and timely signals. The experimental performance results show their model’s effectiveness.", "questions": "Please answer my questions corresponding to the negative points: (1) Model novelty; (2) Missing baselines.\nSee the detailed comments above.", "limitations": "Some limitations are mentioned and I think such this algorithm-driven paper for improving user experience in recommendation system does not have negative societal impact.", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "*Positive points*\n1. Recommendation is one of the most important applications of machine learning.\n2. The proposed method achieves good performance. \n3. The method is proposed based on a thorough analysis.\n\n*Negative points*\n1. The proposed method is not novel enough. The authors assert that previous literature concentrates on utilizing early conversions to mitigate delayed feedback problem and propose a generalized delayed feedback model (GDFM) that unifies both post-click behaviors and early conversions as stochastic post-click information. Indeed, what they tend to address can be defined as heterogeneous feedback problem and there are some relationships between different feedback types. Besides, there have already been some works such as EHCF that addressed this problem via multi-task learning and linear transformation, whose solution is very similar to this work.\n\n[1] Chen et al. Efficient Heterogeneous Collaborative Filtering without Negative Sampling for Recommendation. AAAI 20.\n\n2. The baselines are not enough. It is necessary to compare with some important and highly related heterogeneous collaborative filtering recommendation models such as EHCF.\n\n3. The presentation should also be improved. For example, there is not model framework in their methodology part as their proposed method is with multi-target. I strongly suggest a figure that can illustrate their overall design. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "I3GqaYfjSr_", "reviewer": "Reviewer_fM2F", "summary": "This paper aims to alleviate the delayed feedback problem in conversion rate prediction (CVR) via proposing a new method: generalized delayed feedback model (GDFM). After clicking the item url, it might take months for a user to really purchase the item. Hence, the positive sample might be treated as negative sample in the training and this problem could bias the model.\n\nInstead of only relying on observed labels, their key idea is to introduce other user behaviors or actions like put the item into shopping cart as an auxiliary information or proxy to better predict the conversion.\n\nAccording to entropy between the conversion label and an action, they design a weight to reflect how informative the action is. Besides, they also consider the time gap and sample complexity into designing the weight for the action.\n\nThey conduct experiments over two public datasets and propose detailed evaluations.", "questions": "(1) Using entropy between user action and conversion label to measure the information carried by the action seems reasonable. In addition, they also consider time gap and sample complexity into designing the weight for the action, which shows the comprehensiveness of their thinking. My question (also presented in the next section) is that are these factors firstly considered by this paper? did any related work consider these factors?\n\n(2) In 167: \"Since rank(M) < n requires that some rows of M are linear combination of the others,  which rarely happens in real-world problems\". So you are saying rank(M) = n holds for most of real-world problems, may I ask why? \n\nIn this specific situation, what is the physical meaning of rank(M) = n? does that mean user action and conversion has a very strong relationship? like a user must purchase the item in the shopping cart?\n\nin 160, \"We can recover p(y|x) from p(a|x) if and only if rank(Mx) = n.\" My opinion is that rank(Mx) = n is a very strong condition and also we don't need to 100% recover p(y|x) from p(a|x), does I understand right? \n\n(3) In section 3.2.2, I understand that how much information an action carry about the conversion can be measured by the entropy between them and I see H (y|a) in Equation (5). However, in 195: \"Since we are estimating p(y|x) through a proxy distribution p(a|y)\". May I ask where does p(a|y) come from? could you show the relation of p(y|x), p(a|y) and p(a|x) in some sort of equations, I am a bit confusing here.\n\n", "limitations": "In Section 2, they said: \"However, the ground-truth conversion labels are available only after a long delay: A user may purchase several months later after a click event x.\"\n\nWell, several months? that is a bit against my life experience. \n\nDoes this delay of several months a common situation?\n\nI guess it would be better to show the delay distribution from a real-world CVR system so that we know how serious this delay problem can be and how important this paper's contribution can be.\n\nBut I understand that it may not be appropriate to show that due to company's policy.\n\nExpected to see that though!", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n(1) Since years ago, papers of recommender systems have become some sort of LEGO game -- like how to put a CNN over a DNN etc., Unlike those papers, this one really analyze the impact of user actions and clearly present the generation process in a probability graph way, which looks reasonable to me. I really enjoy this kind of papers.\n\n(2) Using entropy between user action and conversion label to measure the information carried by the action seems reasonable. In addition, they also consider time gap and sample complexity into designing the weight for the action, which shows the comprehensiveness of their thinking. My question (also presented in the next section) is that are these factors firstly considered by this paper?\n\nWeakness:\n\n(1) The experiments could be further enhanced:\n\n(a) Is there any online experiments?\n\n(b) What is the architecture used in the experiments? there are no clues in the paper.\n\n(2) Will algorithm 2 significantly slower the training? Where is the time complexity analysis? \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "5HOS7HEzO3B", "reviewer": "Reviewer_fuKP", "summary": "The authors introduce a generalized feedback model in order to use delayed labels. This is very relevant in e-commerce platforms where a specific action we care about (for example buying a product) can happen after some time from the first interactions with the platform. Generally this is an important problem in e-commerce platforms. Is related basically, to attribution mechanism which is very important when one considers to build a training dataset from click-stream data. The authors compare their approach with other similar approaches on to two datasets. The authors observe improvements using their approach.", "questions": "- I am not sure I got on how many actions you consider in your experiments. In on-line platforms we can have quite large number of different actions. How you deal with this?\n- Did not get in the architecture when you say that you concatenate one-hot encoded y. For y \\in {0, 1} what exactly you concatenate? Could you clarify?\n- I see in the experiments you use a quite large batch size. How you came up with this value?", "limitations": "Yes the authors discuss briefly limitations and giving some idea of overcoming it, but they do not really elaborate on the type of the solution.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This is very important problem. The way is tackled by the authors is interesting and they try to clarify all the points of their approach. By adding and modeling the intermediate actions they can capture and propagate the delayed feedback.\n\nThere is novelty in the approach as it considers temporal distributions and is the main one.\n\nI have some doubts of whether such an approach can be easily implemented in a real use case. No such on-line experiments are described in the paper and essentially the paper builds on such a use-case. By also checking the exact results in the supplemental material improvements there is some relatively small improvements.\n ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept