A Feedback Shift Correctionin Predicting Conversion Rates
under Delayed Feedback
Shota Yasui, Gota Morishita,
Komei Fujita, Masashi Shibata
The Web Conference 2020

Introduction andProblem Setting2

Conversion Prediction
Predict Conversion-Rate(CVR) for each request.

DSP
bidrequest

Useruse Apps

AD Auction

Predicting CVR is important to decide the bid price3

Ideal loss function
The following loss should be minimized.

The ideal parameters are as follow

features
Conversion

modelThis is not possible!
Because we do not observe c due to the delayed feedback.
4

Delayed Feedback5Delayed Feedback
timestamp of click and cv for certain user

time
timestamp ofClicktimestamp of CVdelay●user takes sometimes to purchase items after clicked the ad.
6

The problem of Delayed Feedback
timestamp of click and cv for certain user

training
beginsUnobserved

time
timestamp ofClicktimestamp of CV

included in training data●

we can not observe CV for this user●

this sample is recognized as negative label! (mislabeled)7

The relation between Y and C

C=1

correctlylabeled
S=1

Y=1
Prob of correctly labeledmislabeled
S=0

Y=0
C=0

true label

Prob of mislabel

observable label

8Bias in standard supervised approach

ideal loss

actual loss(ERM)

Inconsistent!

9Our Solution
Importance Weight Approach10

Importance Weight(FSIW) approach
We propose consistent loss based on the Importance Weight(Propensity Score)ideal-loss

Unbiased-loss(consistent?)

Importance Weight11

Importance Weight(FSIW) approach
Our empirical loss

Importance Weight

The basic idea is to weight each sample
by the conditional density ratio.
12How to estimate FSIW

We estimate these probability from data oldenough to observe S and C.

13

Counterfactual Dead Linetraining data
week 1week 2week 3

discard14

Counterfactual Dead Linetraining data
week 1

Train models forweek 2week 3

discard15

Counterfactual Dead Linetraining data
week 1week 2week 3

discardTrain models fortraining data
Importance weightweek 1week 2Train the CVR model

week 316

features of our proposed method
It is just a importance weight○

can be used for any CVR model○

can fit the delay nonparametrically○does not increase the time complexity ofCVR models

17Experiment

18Conversion Logs Dataset●Open data provided by Criteo（Link）●30days of click and CV log●

Used in Chapelle(2014)●observation period is 30days

19Experiment procedure
iterate for 7days

day = 22train(3 weeks)test
day = 23train(3 weeks)test
day = 24train(3 weeks)

test

averaging these results

day = 28train(3 weeks)

test

time20Result 1
Pure-LogisticRegression●Chapelle(2014)

Proposed Method

Normalized-logloss(NLL) is the most important metrics○

we use prediction probability for bidding○logloss(LL) is sensitive to the base CVR21

Dynalyst Data
●DSP in Cyberagent.inc●2 experiments○○

the same procedure as the first experiment■focus on three campaigns■

baseline model is FFM (Juan 2017)

Online A/B test22

Three Campaigns●

Observational period is different by campaings○ S: 1days○ M: 3days○ L: 7days

23

Result 2

Only Campaign L shows the improvement.
24

Follow Up Online Experiment@Campaign-L●

Improved cost consumption and CV.

●CPA does not change or slightly decreased.
25

Conclusion● We proposed a consistent loss to predict CVR under DelayedFeedback.

● Our method performs better in two offline and one online
experiment.

Thank you for listening! 26

appendix

cumulative distribution of delay28

effect of counterfactual deadline29