Calibration Matters: Tackling Maximization Bias in Large-scale Advertising Recommendation Systems
Authors: Yewen Fan
Nian Si
Kun Zhang
Venue: ICLR 2023 poster (2023)
Keywords: Maximization bias
calibration
distribution shifts
neural networks
recommendation system
computational advertisement

Abstract: Calibration is defined as the ratio of the average predicted click rate to the true click rate. The optimization of calibration is essential to many online advertising recommendation systems because it directly affects the downstream bids in ads auctions and the amount of money charged to advertisers. Despite its importance, calibration often suffers from a problem called “maximization bias”. Maximization bias refers to the phenomenon that the maximum of predicted values overestimates the true maximum. The problem is introduced because the calibration is computed on the set selected by the prediction model itself. It persists even if unbiased predictions are achieved on every datapoint and worsens when covariate shifts exist between the training and test sets. To mitigate this problem, we quantify maximization bias and propose a variance-adjusting debiasing (VAD) meta-algorithm in this paper. The algorithm is efficient, robust, and practical as it is able to mitigate maximization bias problem under covariate shifts, without incurring additional online serving costs or compromising the ranking performance. We demonstrate the effectiveness of the proposed algorithm  using a state-of-the-art recommendation neural network model on a large-scale real-world dataset.

Published as a conference paper at ICLR 2023
CALIBRATION MATTERS: TACKLING MAXIMIZATION BIAS IN LARGE-SCALE ADVERTISING RECOMMENDA- TION SYSTEMS
Yewen Fan Nian Si Kun Zhang
Carnegie Mellon University
2 Mohamed bin Zayed University of Artificial Intelligence
3 University of Chicago Booth School of Business
(* Equal contribution)
ABSTRACT
Calibration is defined as the ratio of the average predicted click rate the true click rate. The optimization of calibration is essential to many online advertising recommendation systems because directly affects the downstream bids in ads auctions and the amount of money charged to advertisers. Despite its importance, calibration often suffers from problem called "maximization bias". Maximization bias refers to the phenomenon that the maximum of predicted values overestimates the true maximum. The problem is introduced because the calibration computed on the set selected by the prediction model itself. It persists even if unbiased predictions are achieved on every datapoint and worsens when covariate shifts exist between the training and test sets. To mitigate this problem, we quantify maxi- mization bias and propose auriacc-adjjuting debiasing (VAD) meta-algorithm this paper. The algorithm is efficient, robust, and practical as it is able mitigate maximization bias problem under covariate shifts, without incurring additional online serving costs or compromising the ranking performance. We demonstrate the effectiveness of the proposed algorithm using a state-of-the-ane recommendation neural network model on large-scale real-world dataset.
INTRODUCTION
The online advertising industry has grown exponentially in the past few decades. According to Statista (2022), the total value the global internet advertising market was worth USD 566 billion in 2020 and is expected to reach USD 700 billion by 2025.
In the online advertising industry, to help advertisers reach the target customers, demand-side platforms (DSPs) try to bid for available ad slots in an ad exchange. A DSP serves many advertisers simultaneously, and ads provided by those advertisers form the DSP's ads candidate pool. From DSP's perspective, the advertising campaign pipeline executes as follows:
(1) The DSP uses data to build machine learning (ML) models for advertisement value estimation. An advertisement's value is often measured by the click-through rate (CTR) conversion rate.
(2) When the exchange sends requests in the form of online bidding auctions for some specific ad slots to DSP, the DSP uses the ML models to predict values for ads in its ads candidate pool.
(3) For the bidding requests, the DSP needs to choose the most suitable ads from its ads candidate pool. Therefore, based on the estimated values, the DSP chooses the ad candidates with the highest values and submits corresponding bids to the auctions in the ad exchange.
(4) For each auction, ad with the highest bid would win the auction, and would be displayed (i.e., recommended) in this specific ad slot. The ad exchange would charge the winning DSP a certain amount of money based on the submitted bid and the auction mechanism.
For the machine learning models in Step (2), besides learning the ranking (i.e. which ads sent to ad exchange), DSPs also need to accurately estimate the value of the chosen ads, because in

---

Published as a conference paper at ICLR 2023
Step (3), DSPs bid based on the estimated value obtained from Step (2). Thus, DSPs try to avoid underbidding or overbidding, the latter of which may result in over-charging advertisers. We measure the estimation accuracy by calibration. which is the ratio of the average estimated value (e.g.. estimated click-through rate) to the average empirical value (e.g., whether user click or not).
Calibration is essential to the success of online ads bidding methods, well-calibrated predictions Calibration
are critical to the efficiency of ads auctions (He et al. 2014] McMMaan et al. (203)).
is also crucial in applications such as weather forecasting (Murphy & Winkler 1977: DeGroot &
Fienberg 1983, Gneiting& & Raftery 2005), personalized medicine (Jing et al. 2012) and natural
language processing (Nguyen & O' O'Conoo 2015; Card & Smith 2018). There is rich literature
for model calibration methods (Zadrozny & Elkan 20011 Menon et al., 2012 Deng et al.
2020: Naeini al., 2015| Kumar et al., 2019: Platt et al. 1999; Guo et al., [201]] Kull et et. 2017
al.1 2019). These existing methods focus on calibration for model bias. However, those methods do not explicitly consider the selection procedures in Step (3) of the aforementioned recommendation
system pipeline. In this case, even if unbiased predictions are obtained for each ad, the calibration may perform poorly on the selection set due to maximization bias. Maximization bias occurs when maximization is performed on random estimated values rather than deterministic true values. We will provide a concrete example to illustrate the difference between maximization bias and model bias.
Example 1. Assume there are two different ads with the same "true" CTR 0.5. Now we consider an ML model that learns the CTR of the two ads from data independently. We assume that the ML model predicts the CTR of either one of the ads as 0.6 or 0.4 with equal probabilities. Note that the estimation unbiased and thus having zero model bias. After both advertisements are submitted to the auction system, the ad with the highest estimated CTR will be selected. In this case, the probability of the system selecting an ad with an estimated CTR 0.6 is 75% and an ad with an estimated CTR 0.4 is 25%. Therefore, in this example, the model has maximization bias because it overestimates the true value of the ads (3/4 x 0.6 1/4 x 0.4 = 0.55 > 0.5). This example explains why there may be maximization bias in model after selection even if the model has zero model bias.
Hypothetically, if the DSP submits all the ads with their corresponding bids to an ad exchange, the maximization bias is analogous to the so-called winner's curse, even in the absence of selection and maximization procedures during Step (3). In auction theory, the winner's 's curse means that in common value auctions. the winners tend to overbid if they receive noisy private signals. Consequently, this calibration issue arises wider context.
What makes calibration even harder is the covariate shifts between training and test data (Shen et al., 2021: Wang et al.] (02T). The training data only consists of the previous winning and displayed ads, but during testing, DSPs need to select from a much larger ads candidate set. Therefore, the test set contains many ads that are underrepresented in the training set since those types of ads have never been recommended before. These covariate shifts will invalidate aforementioned calibration methods that reduce bias using labeled validation sets.
In this paper, we propose a practical meta-algorithm to tackle the maximization bias in calibration, which could be in tandem with other calibration methods. Our algorithm neither compromises the ranking performance nor increases online serving overhead (e.g., inference cost and memory cost). Our contributions are summarized below:
(1) We theoretically quantify the maximization bias in generalized linear models with Gaussian distributions. We show that the calibration error mainly depends on the variances of the predictor and the test distribution rather than number of items selected.
(2) We propose an efficient, robust, and practical meta-algorithm called variance-adjustin debias (VAD) method" that can apply to any machine learning method and any existing calibration methods. This algorithm is mostly executed offline without any additional online serving costs. Furthermore, the algorithm robust covariate shifts that are common in modern recommendation systems.
(3) We conduct extensive numerical experiments to demonstrate the effectiveness of the proposed alga--algri in both synthetic datasets using logistic regression model and large-scale real- world dataset using state-of-the-an recommendation neural network. In particular, applying VAD in tandem with other calibration methods always improve the calibration performance compared with applying other calibration methods alone.
code available in http://amoyymaassoooon.cccccccee
2

---

Published as a conference paper at ICLR 2023
2 RELATED WORK
There is a long line of work regarding calibration methods. Broadly speaking, existing methods could be classified into two groups: non paraammetriicc and parametric methods. farthdss(Kweon) et al., (021) On one hand, non-parametric methods utilizes binning ideas, which include histogram binning (Zadrozny &
Elkan 200T), isotonic regression (Zadrozny & Eliann 2002), smoothed isotonic regression (Deng
al. 2020). Bayesian binning (Nacini et al., 2015), and gealing-binning calibrator (Kumar et al. 2019) On the other hand, parametric methods explicitly learn a parametric function mapping from the model original scores to calibrated probabilities. Example methods include Platt scaling (Platt et al., 1999), temperature scaling (Guo et al.) 2017). Beta calibration (Kull et al.) [017]. and Dirichlet calibration (Kull et al. 2019). We refer the readers to Kween et al. (2021) for a comprehensive survey on various calibration methods. As discussed in Introduction, those methods are not designed for correcting maximization bias.
Maximization bias appears in many different domains, ranging from economics (Van den Steen 2004] Capen et al. I97I). decision analysis (Smith & Winkler 2006), to statistics, which includes model selection (Varma& & Simon 2006), over-fitting (Cawley & Talbot 2010), selection bias (Heckman 1979). and feature selection (Ambroise & McLachlan 2002). Maximization bias is especially well-documented in reinforcement learning literature (see, [Sutton & Barto (2018) Section 6.7)). In reinforcement learning, estimating value function is a fundamental task, where value function is typically the maximum of many expected values of different actions. To reduce maximization bias, double learning or cross-validation estimators (Van Hasselt 2010 2011: 2013 Van Hasselt et al. 2016) are used and demonstrate strong empirical performances. The basic idea is to train two separate models: one model selects while the other predicts the probability. However, this type of methods is not applicable to large-scale ads recommendation systems since it will double online serving costs, and online serving efficiency is essential to recommendation system performances.
Calibration for maximization bias also closely related to estimating the maximum mean of several random variables in operations research and machine learning. Various estimators were proposed: Chen & Dudewiez (1976) develop a two-stage procedure to provide a confidence interval of the highest mean; Lesnevski al. (2007) further integrate their method with screening ideas, variance- reduction, and common random numbers techniques; Liu et al. (2019) propose an upper confidence bound (UCB) approach: Chang al. (205) incorporate similar UCB components into the Monte Carlo tree search; and Eramo et al. 2016) use weighted average of the sample means, where the weights are computed by Gaussian approximations. However, those methods are either simulation- based or have access to multiple i.i.d. copies of random variables; thus, they cannot be directly applied to supervised learning settings in recommendation systems.
3 PRELIMINARIES AND PROBLEM SETTING
Consider a supervised learning setting. For each data point, we have high-dimensional feature X E X' C Rd and label Y E y & {0, 1}. In this paper, we focus on the binary label Y that represents whether the user clicks or not. Our method can be easily extended to continuous labels. Suppose we have access to a labeled training set (X, Y) ~ Dirain and an unlabeled test validation set X ~ D which has the same distribution as the X margin of the real test set Drest Note that there are often covariate shifts between Dirain and Drest since the training set consists of the historical recommended items, while the test set contains all possible candidates, many of which may have never been seen by users before. However, we can reasonably assume that there are no concept drifts between training and test distributions in Assumption 3.1
Assumption 3.1 (No concept drift). The conditional distribution Y given X is the same between the training distribution and the test distribution.
Then, the recommendation system pipeline is summarized in Flowchart Specifically, in the first step, the predictor f -> [0, 1] is prediction of PP(Y = =1|X); in the second step, we rank all items by f(x) and pick the top (unknown apriori) proportion, where the selected set is denoted by De in the third step, we consider two different calibration metrics, including calibration error E
He et al. (2014) and expected calibration error (ECE) and we provide additional results for maximum
calibration error (MCE) in the Appendix (Naeini et al. 2015 Kweon et al. 2021)
3

---

Published as a conference paper at ICLR 2023
The calibration enror & and ECE are defined by
E EiEDiuR f(x,) EieDe Yt
M
Bml
EkeBm yk
EkeB f((x)
1, ECE A
(1)
N
Bmll
Bml
m=
where we partition all items in Diet into M equi-spaced bins by their values, and Bm is m-thh bin and N [Piest] the number of samples.
Train a prediction model f on Pirain and p
Select top a proportion of candidates on Prest ranked by f(x).
Compute calibration metrics in Equation (1).
Figure 1: Flowchart visualization of the procedure.
In Section 6.2, execute the pipeline on a real-world ads recommendation system dataset using state-of-the-art neural network and observe that calibration errors are consistently larger than 3% as shown Table 2 if not performing any debiasing methods. Therefore, the goal of this paper is to find predictor that minimizes calibration errors in the selected set without compromising the ranking performance or incurring additional online serving costs. The serving costs refer to the costs of executing the method on the test sets.
We note that the unlabeled validation set D is easily obtained and offline available because use the candidate sets from previous online requests. Furthermore, is usually small (but unknown) in practice since DSPs usually have large ads candidate set.
Remark: maximization bias v.s. model bias In this paper, we tackle maximization bias, which is different from model bias in machine learning. Model bias is the difference between the expected model prediction and true value for given ad. Note that model predictions have randomness due to random data and stochastic optimization algorithms. However, maximization bias in our paper is a different type of bias, orthogonal model bias. Maximization bias exists because of the selection (maximization) step. Even models with zero model bias may have maximization bias. We refer the readers back Example to see the difference between model bias and maximization bias.
4 QUANTIFYING MAXIMIZATION BIAS IN GENERALIZED LINEAR MODELS
Generalized linear models are unifying framework of linear regression, logistic regression, and Poisson regression (Nelder & Wedderburn 1972). In particular, neural networks (NNs) are categorized as generalized linear model if we view the neurons from the second-tt coond-to-last last layer as features and use a sigmoid function as the final activation function. In this section, we provide rigorous quantification for the maximization bias in generalized linear models with Gaussian features.
By a slight abuse of notation, Drest. represents the population distribution of (X,Y) in the test set. We need to select the top-a percent from the test set. We further assume the underlying true model is generalized lincar in both training and test sets with the same conditional distributions, ie.. Y|X Ber (Ø(3. X)), where o6-) is positive, continuous differentiable, and monotonically increasing function. Let 3. be the parameter learned from the N -asample training set sampled from D which is independent of Drest. We do not need assume marginal distributions of X in the training data Derain and test data Drast have the same distribution.
We let Q1 a (Z) be the 1 quantile of distribution Z, i.e., P(Z 2 (9-a(()) a.
By the monotonicity of Φ(-), the estimated average probability on the selection set is
ED Deroin @ ( 1 91 (3
the actual average probability on the selection set is
E Eprnat Dersan
(8. x) 1 2 91 - Bw)
(3)
Note that these expectations taken with respect to the randomness of both X and 8x. To abbreviate the notion, we drop the subscripts Dtest; Dirain when there is no confusion. We quantify the maximization bias for this generalized linear model with covariate shifts in Theorem 4.
4

---

Published as a conference paper at ICLR 2023
Theorem 4.1. Suppose X N (p, Σ) and <(-) is positive, Lipschitz continuous, twice differen- tiable, and monotonically increasing function. If By is a maximum likelihood estimator, we have the estimated average probability on the selection set
E
(3xx)
20 = (3 VDTEENZ 122 2224-0 (2()
and the maximization bias on the selection set
E
(x
1
2 41 11 (
E
④ (5. x) 18x (
VANTAN
=E
(t)d)tt
+0
(4)
N
where Z ((001) 1) and (1) A E[d (3y + 1Z) 2 01 91 (Z)] The O(L/N) term only depends on dz Ba, and Σ, but does not depend on a.
Remark: 1. The Gaussianity assumptions of the feature X are not necessary. One only needs B and B X to be jointly Gaussian conditional on fy and the proof would still go through. Figure in Appendix B.2 shows that X X is indeed very close to the Gaussian distribution.
X
2. Note that do not assume that the training and test distributions the same. Theorem true under arbitrary covariate shifts.
In the setting of Theorem 4.1 Var(āx|X) is heterogeneous across X, and we allow to choose any proportion of the test set. Furthermore, if is small and is small, (t(t) has the same order as E Z Z q1-a which large, and the maximization bias mainly depends on AN E,B and T 13.. The formal statement in Lemma A.2 in Appendix A
To reduce the maximization bias, we discount NN X by X. Corollary discounted estimator.
4.2 studies the bias for this
Corollary 4.2. Suppose the same assumptions in Theorem 4.1 are imposed. If we change B to
X + (1 u X E [0, 11, the bias becomes
E ④
[B x + -))33m))
BIX o1 (3T [3N]
AVbLEa
-E 0
(3. x) x 21-11-a6 ( =E
h'(t)dt
- 0 N
(5)
AlSo VANEAN
The transformation T - -BT + (1-
-1 J is linear; thus, it does not alter the item rankings.
Despite its simplicity, we are able to find a À such that the leading term of (5) is approximately zero in Section 5 i.e.,
AAE l((t)dt
R0 O.
Va
5 VARIANCE-ADIUSTING DEBIASING META-ALGORITHE
Based on the theory developed in Section
the goal is to find À such that
ANDA.
IN
> ~
DI
BT IT
The denominator EN is easy to estimate BNNBN Var (3T To estimate the nominator 3 B,,, we first observe the decomposition that
B [ [ + E VarDunin (5xX-3x117)
(6)
5

---

Published as a conference paper at ICLR 2023
provided that B[3x] Bx; this equality is proved in Appendix A Therefore, MN could be approximated by EN ED
VVarD.am
( 3xx[))]
where the variance of 3N could be estimated by bootstrapping retraining the model using different random seeds. Note that this approximation is relatively accurate if the feature dimension is large and the correlations between dimensions are small. This is because in this case, / BBN and 3 IB, concentrate on their means with an error O (1(V/d))
1 (
ER = E OD
and EEES. - Op
In the context of generalized linear models, we observe that x ( Therefore, in general cases, define 1(x:) / (fxx)) In practice, Mxx) be obtained either by inverting or extracting the last layer of the NN if the neural network is used as the prediction model with ø as the activation function of the last layer. Then, based on these analysis, we propose a variance- adjusting debiasing (VAD) meta-algorithm in Algorithm D Since the À estimation procedure is fully non-pane non-parametric and depends only on some means, variances, and conditional variances, our meta-algorithm applies to any machine learning algorithm.
Algorithm 1 Variaacce-ddjusting debiasing (VAD) method
1: Input: Training dataset Derain the unlabeled test validation set Dval-test,X a link function Φ IR + [0, 1J. and the number of replications S.
2: Output: A variance adjusting debiased predictor fVAD-
3: Train model the training set and obtain the predictor f1-
4: Bootstrap (i.e. random sampling with replacement) the dataset S - times or retrain the model
S - 1 times using different random seeds and obtain the predictor .....
Ss. :
5: Let (x)) o (f((z)) for i E [S]. Then compute the means Y = Ex.a ten. [x() for
i E [S] the test variance (0% Dar x14 (X)].
2
6: Compute the expected conditional variance
(4(x) ((x)))))) );
(61) 2 = E -ral-
8-1 1(x) -Y 1
(0/ 8: Output fVAAD.) (1/( + ()-))
(81) (0t)/(()
7: Compute A =
Since the debiased predictor SVAD is monotonic transformation of the original predictor fi in Algorithm!] | the prediction rankings remain the same. Further, À and Y are computed purely offline as long as we have unlabeled validation set that has the same distribution as the test set. Thus, no additional serving costs are added. Furthermore, since we only need to estimate means and variances, we only need samples from the unlabeled candidate set of reasonable size. We can sample recent data points in the large (and potentially non-stationary) candidate set to get sufficiently accurate estimates.
For the link function <6-) in logistic regression and NNs with final sigmoid activation function, ((x) = (1 + exp(-x)) and is the logit (the last layer in NNs) of the predictor Si.
<(-) could also chosen as the identity mapping ((x) x. We note that this choice of xx() has similar performance to the choice of d(x) = (1 + (xxp(=2))-1 We report additional results about the behavior the identity link function in Appendix B
On Line 4 in Algorithm 1 we recommend bootstrapping (Bfron & Tibshirani 1994] if the base training model lacks intrinsic randomness, e.g., logistic regression, which is an efficiently solvable convex optimization problem. However, if the base model is highly non-convex with multiple local optima, e.g., neural networks, we recommend random initializations and random data orders because many empirical studies (Nixon et al. 2020
Lakshminarayanan et al. 2016 Lee et al. 2015) show that

---

Published as a conference paper at ICLR 2023
bootstrapping may hurt the performance in deep neural networks and the estimation of the conditional variances would benefit from the algorithmic randomness (Jiang et al. (022).
In our method, only need to choose one fact, S = 2 would be sufficient and results in lower training cost. Therefore, our method doubles the training cost, which is usually acceptable in practice. More importantly,
S and we do not need specify In
method does not incur any additional online serving costs. All results reported in Section 6 usee S = 2. We report additional results about different choices in Appendix B
EXPERIMENT RESULTS
In this section, demonstrate the performance of our method in both synthetic data and real-world ads recommendation dataset. We use calibration errors, ECE, and MCE to evaluate performances. Note that evaluation metrics are calculated the selection set, i.e., choose top proportion of test data points using model predictions and compute the evaluation metrics in the top-a selection set. We provide additional numerical results for the Avazu dataset in Appendix B.3
6.1 SYNTHETIC DATA
Data and Model We consider logistic regression model. We assume the response Y follows the Bernoulli distribution with probability (1 exp(- BT x)) for 3, eR.
In the experiments, we consider 20, 3 = 11, _] x (ptrain Evaain in the training set generate Nerain 3000
Dtrain. and X (Prest Eeest) in the test set Deest In the training Li.d. training samples to train a logistic regression model. For our proposed method, we bootstrap 2 times. In the test set, choose top proportion of Ntsst 30000 data points using the model predictions and compute the evaluation metrics in the top-a selection set. We generate Nval-test 30000 data points as the test validation dataset.
Covariate Shift We assume there is covariate shift and no concept drifts, where we assume
Mtrrin = (0.05, 0.05,.... 0.05, and /eest = 0.05, -0.05..... -..0,"",,
and Etrain Etrst 0.12 Iaxa- The positive sample rate is 72.3% in the training set, which is higher than the positive sample ratio in the test set (27.7%). This is consistent with real-world recommendation systems since the training data the previously recommended sets, in which items should have better performance than items in the whole candidate sets (i.c. test sets).
Performance We replicate the experiments 100 times and report averages and standard errors of calibration errors and ECE (with number of bins M = 10) for a E {2%, 10%} in Table I (MCE in Appendix B.I), where Vanilla stands for original predictions. We plot average calibration errors and average ECE for a E [2%, 10%] in Appendix B.1 We also report Log Loss improvement Appendix B. indicating our method improves prediction quality as well.
Table 1: Average and standard errors of calibration errors and ECE for synthetic data
2%
10%
Method
Calibration eтrоr
ECE
Calibration error
ECE
Vanilla
8.55%=0.68%
0,0656=0.0019
7.34%=0075%% 0.62 4.73%%
0.0425+10,001 0.0334+10.0015
VAD 0.76%
0.0572-0.0013
Note that we do not compare with other calibration methods under logistic regression model since logistic regression produces well-calibrated predictions. Applying other calibration methods to logistic regression model will not improve the performance. (Niculescu-Mizil & Caruana 2005).
Table shows that the vanilla model without debiasing has a calibration error more than 7%, which is mainly due to maximization bias logistic regression produces well-calibrated predictions. After applying VAD, the calibration error is sufficiently close to zero. All the superiority statistically significant at the 1% significance level. Additional experiment results reported in Appendix B.
7

---

Published as a conference paper at ICLR 2023
6.2 REAL-WORLD DATA
Dataset We the Criteo Ad Kaggle dataset | to demonstrate our method's performance. The Criteo Ad Kaggle dataset is a common benchmark dataset for CTR predictions. It consists of week's worth of data, approximately 45 million samples in total. Each data point contains a binary label, which indicates whether the user clicks or not, along with 13 continuous, 26 categorical features. The positive label accounts for 25.3% of all data. The categorical features consist of 1.3 million categories on average, with feature having more than 10 million categories, 5 features having more than million categories. Due to computational constraints in our experiments, we use the first 15 million samples, shuffle the dataset randomly, and split the whole dataset into 85% train Dirain. 1.5% saidattion-rain D vaa-ttrain 1.5% validation-test Dval-test, and 12% test Diest datasets.
Base Model We use the state-of- ofetth-art deep learning recommendation model (DLRM) (Naumov ct al. 2019) open-sourced by Meta as our baseline model. DLRM employs a standard architecture for ranking tasks, with embeddings to handle categorical features, Multilayer perceptrons (MLPs) to handle continuous features and the interactions of categorical features and continuous features. Throughout our experiments, we use the default parameters and a SGD optimizer. Note that our method is model-agnostic, so it can be directly applied to other models (e.g. support vector machines, boosted trees, nearest neighbors, etc Friedman et al. (20TT)
Baseline Calibration Methods We compare our method with various classic calibration methods. For parametric methods, we compare against Platt scaling (Platt al. [999). For non-parametric methods, we compare against histogram binning (Zadrozny & Elkan 2002 200T). isotonic regression (Menon et al. 2012). and scaling- binning calibrator (Kumar et al. 2019). We use the labeled training validation dataset to do calibration for above methods.
Note that none of the existing work explicitly considers maximization bias and thus fails to perform well in our setting. VAD can be combined with all existing calibration methods to achieve better per- formance by making a small change to the original VAD algorithm. VAD takes predictions calibrated by other calibration methods as inputs. Additionally, instead of directly using À calculated from Dval-test.X. we first calculate Aval-test. from D x using original predictions and Aval-train from Dral-train, (the unlabeled validation set which has the same distribution as the X margin of the training set) using original predictions, and then use À = Aval-test Aval- train to adjust predictions calibrated by other methods. This change is due to the fact that other calibration methods already compensate for maximization bias in training distribution to some extent.
VAD Parameters The last layer of the DLRM network uses the sigmoid activation function. In our method, by using the link function ((x) = (1 + exp(-x)) we compute the means Y variance (dt)), and expected conditional variance (o))² of the last layer's 's neuron. To compute the expected prediction variance (at)², ; we keep the training data unchanged and modify the random initialization and data orders since the optimizer itself incurs sufficient randomness. In the experiment, we train
S times for our method.
Covariate Shift Since the underlying true data-generating process is unknown, we employ a different strategy construct -uttof-disttibuuton test data than how we generate synthetic data: we train another DLRM model using 85% x 15 million samples different from the original dataset; we randomly keep each data point in the original test set with probability 1 p. where p is the newly-trained DLRM model prediction for the data point. The training set remains the same. After this shift, the positive samples account for 20. 1% of all test data. By doing this, we ensure that the distributional change is only a covariate shift, and the positive sample ratio in the test data is lower than the positive sample ratio in the training data, which is consistent with the real-world recommendation systems.
Performance We replicate the experiments 40 times and report averages and standard errors of calibration errors and ECE (with number of bins M = 50) for a E (2%, 10%) in Tables 2 and 3 (MCE is in Appendix B.2). where Origina ginallllllll stands for using calibration methods solely and VAD+ + represents the tandem combination of the calibration methods and VAD. We plot average calibration errors and average ECE for [2%, 10%] in Figure [ We also report Log Loss improvement in
Appendix B.2, indicating our method also improves prediction quality.
8

---

Published as a conference paper at ICLR 2023
Table 2: Average calibration errors on the Criteo Ad Kaggle dataset
a 2%
10%
Method
Original
VAD+
Original
VAD+
Vanilla
3.23%=10.419%
3.94%:10663%
-0.4%%=0.6664
Histogram Binning
2.16%=10.06%
1.44%:10.06%
1.87%:40.04%
0.98%=00.04%
Platt Scaling
1.60%:t..13%
0.86%+0.12%
1.80%+0.07%
0.91%:t0.07%
2.30%:10..33 1.62%:10..6%%
1.56%=0.11%%
1.99%=0.07%
1.11%+00.0%%
Isotonic Regression
0.87%:10.07%
1.75%:10.04%%
0.85%:10.04%
Table 3: Average ECE on the Criteo Ad Kaggle dataset
a= 2%
10%
Method
Original
VAD+
Original
VAD+
Vanilla
0.0287=10.0044
0.0225+000016
0.0271#0.0029
0.0208=0.0022
Histogram Binning
0.0213:+0.0003
0.0185+0.0003
0.0132::0.0022
0.0105=0.0002
Platt Scaling
0.0179:10.0006
0.015810.0004
0.01214:00003
0.0092:10.0002
Scaling-Binnning
0.0217++0.0077
0.0188+
0.0131+00.003
0.0101+000002
Isotonic Regression
0.0193=10.0004
0.0175=000003
0.0126=t0.0002
0.0101#0.0002
e ...Van Vanill
.e. Vanilla
VAD
VAD
3%
Platt Scaling
Platt Scaling
Platt Scaling VAD
- Platt Scaling +VAD
Colibration
2%
0.02
0.01
-196
296
44%
6%
8%
10%
2%
4%
6%
8%
10%
0x
(a) Calibration Eroor
(h) BCE
Figure 2: Average calibration errors and ECE for the Criteo Ad Kaggle dataset
Table and Figure 2(a) show that the vanilla model without debiasing has calibration error more than 3%. After debiased by existing calibration methods, there are still 1.5% 2.0% over-calibrations, which are largely due to maximization bias. Among all methods, the sole VAD method (i.e., not functioning in tandem with any other calibration methods) perform the best, but has a large variance. Moreover, from Tables 2 3 and Figure 2 We find that method (VAD) outperforms the vanilla method. Particularly, all the calibration methods in tandem with VAD achieve better performance than the sole calibration methods alone. All the superiority is statistically significant at the 1% significance level. Additional experiment results are reported in Appendix B.2
CONCLUSION
We proposed a theory-certified meta-algorithm variance-ane adjuusing debiasing (VAD) to tackle maxi- mization bias in recommendation systems. The meta-algorithm is easy-to-implement by adding only a few lines, scalable to large-scale systems with no additional serving costs, applicable to any machine learning methods, and robust to covariate shifts between training and test sets. Empirical results show its significant superiority over other methods. Our method can be directly used industry with minor modifications, e.g. doing VAD separately for each group of data instead of doing VAD globally. Interesting follow-ups include combining VAD and other calibration methods better way and further reducing the training cost. We leave these for future work.

---

Published as a conference paper at ICLR 2023
8 REPRODUCIBILITY STATEMENT
We open-soureed our implementation at
ACKNOWLEDGEMENTS
This project was partially supported by the National Institutes of Health (NIH) under Contract 1559855, by the NSF-Convergence Accelerator Track-D award #2134901, by grant from Apple Inc., a grant from KDDI Research Inc, and generous gifts from Salesforce Inc., Microsoft Research, and Amazon Research.
REFERENCES
Christophe Ambroise and Geoffrey J McLachlan. Selection bias in gene extraction on the basis of microarray expreexprnoooon data. Proceedings of the national academy of sciences, 99(10): 6562-6566, 2002.
Edward C Capen, Robert V Clapp, and William M Camphell, Competitive bidding in high-risk situations. Journal of petroleum technology, 23(06):6441-5) 1971.
Dallas Card and Noah A Smith. The importance of calibration for estimating proportions from annotations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long Papers), pp. 1636-1646, 2018.
Gavin C Cawley and Nicola LC Talbot. On over-fittingg in model selection and subsequent selection bias in performance evaluation. The Journal of Machine Learning Research, 1 11:2079-2107, 2010.
Hyeong Soo Chang, Michael C Fu, Jiaqiao Hu, and Steven Marcus. An adaptive sampling algorithm for solving markov decision processes. Operations Research, 53(1): 2005.
Hubert Chen and Edward Dudewicz. Procedures for fixed-width interval estimation of the largest normal mean. Journal of the American Statistical Association, 71 71(355):522-56, 1976.
David R Cox and E Joyce Snell. A general definition of residuals. Journal of the Royal Statistical Society: Series B (Methodological), 30(2):248 265, 1968.
Morris H DeGroot and Stephen E Fienberg. The comparison and evaluation of forecasters. Journal of the Royal Statistical Society: Series D (The Statistician), 32(1-2): 1983.
Chao Deng, Hao Wang, Qing Tan, Jian Xu, and Kun Gai. Calibrating user response predictions in online advertising. In Joint European Conference Machine Learning and Knowledge Discovery in Databases, pp. 208-223. Springer, 2020.
Carlo D'Eramo, Marcello Restelli, and Alessandro Nuara. Estimating maximum expected value through gaussian approximation. In International Conference Machine Learning, pp. 1032- 1040. PMLR, 2016.
Bradley Efron and Robert J Tibshirani. An introduction to the bootstrap. CRC press, 1994.
Jerome Friedman, Trevor Hastie. Robert Tibshirani, et al. The elements of statistical learning. volume 1. Springer series in statistics New York, 2001.
Tilmann Gneiting and Adrian E Raftery. Weather forecasting with ensemble methods. Science, 310 (7746)2448244) 2005.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Weinberger. On calibration of modern neural networks. In International Conference on Machine Learning, pp. 1321-1330. PMLR, 2017.
Xinran He. Junfeng Pan, Ou Jin, Tianbing Xu, Bo Liu, Tao Xu, Yanxin Shi, Antoine Atallah, Ralf Herbrich, Stuart Bowers, et al. Practical lessons from predicting clicks on ads at facebook. In Proceedings of the Eighth International Workshop on Data Mining for Online Advertising, pp. 1-9, 2014.
10

---

Published as a conference paper at ICLR 2023
James J Heckman. Sample selection bias specification error. Econometrica: Journal of the econometric society, pp. 153-161, 1979.
Xiaoqian Jiang, Melanie Osl, Jihoon Kim, and Lucila Ohno-Machado. Calibrating predictive model estimates support personalized medicine. Journal of the American Medical Informaties Association, 19(2):263-274, 2012.
Yiding Jiang, Vaishnavh Nagarajan, Christina Baek, and Zico Kolter. Assessing generalization of sgd via disagreement. arXiv preprint arXiv:2106. 13799, 2021.
Meelis Kull, Telmo Silva Filho, and Peter Flach. Beta calibration: well-founded and easily implemented improvement on logistic calibration for binary classifiers. In Artificial Intelligence and Statistics, pp. 623-631. PMLR, 2017.
Meelis Kull, Miquel Perello Nieto, Markus Kängsepp, Telmo Silva Filho, Hao Song, and Peter Flach. Beyond temperature scaling: Obtaining well-calibrated multi-class probabilities with dirichlet calibration. Advances in neural information processing systems, 32, 2019.
Ananya Kumar, Percy Liang, and Tengyu Ma. Verified uncertainty calibration. arXiv preprint arXiv:1909.00155 2019.
Wonbin Kweon, SeongKu Kang, and Hwanjo Yu. Obtaining calibrated probabilities with personalized ranking models. arXiv preprint arXiv:2112.07428, 2021.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. arXiv preprint 1rXiv:16/2.01144, 2016.
Stefan Lee, Senthil Purushwalkam, Michael Cogswell, David Crandall, and Dhruv Batra. Why m heads are better than one: Training a diverse ensemble of deep networks. arXiv preprint arXiv:1511.06314. 2015.
Vadim Lesnevski, Barry L Nelson, and Jeremy Staum. Simulation of coherent risk measures based on generalized scenarios. Management Science, 53(11):1756-176) 2007.
Jianxun Lian, Xiaohuan Zhou, Fuzheng Zhang, Zhongxia Chen, Xing Xie, and Guangzhong Sun. xdeepfm: Combining explicit and implicit feature interactions for recommender systems. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 1754-1763, 2018
Guangwu Liu, Wen Shi, and Kun Zhang. An upper confidence bound approach to estimating coherent risk measures. In 2019 Winter Simulation Conference (WSC), pp. 914-925, IEEE, 2019.
H Brendan McMahan, Gary Holt, David Sculley, Michael Young. Dietmar Ebner, Julian Grady, Lan Nic, Todd Phillips, Eugene Davydov, Daniel Golovin, et al. Ad click prediction: a view from the trenches. In Proceedings of the 19th ACM SIGKDD international conference Knowledge 2013.
discovery and data mining, pp.
Aditya Krishna Menon, Xiaoqian J Jiang, Shankar Vembu, Charles Elkan, and Lucila Ohno-Machado. Predicting accurate probabilities with a ranking loss. In Proceedings of the... International Conference on Machine Learning, International Conference on Machine Learning, volume 2012, pp. 703. NIH Public Access, 2012.
Allan H Murphy and Robert L Winkler. Reliability of subjective probability forecasts of precipitation and temperature. Journal of the Royal Statistical Society: Series C (Applied Statistics). 26(1):
41 41-47, 1977.
Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated prob- abilities using bayesian binning. In Twenty-Ninth AAAI Conference on Artificial Intelligence, 2015.
Maxim Naumov, Dheevatsa Mudigere, Hao-Jun Michael Shi, Jianyu Huang, Narayanan Sundaraman, Jongsoo Park, Xiaodong Wang. Udit Gupta, Carole-Jean Wu, Alisson G Azzolini, al. Deep learning recommendation model for personalization and recommendation systems. arXiv preprint arXiv: 10990000009 2019.
11

---

Published as a conference paper at ICLR 2023
John Ashworth Nelder and Robert WM Wedderburn. Generalized linear models. Journal of the Royal Statistical Society: Series (General), 135(3):370-33) 384, 1972.
Khanh Nguyen and Brendan O'Connor. Posterior calibration and exploratory analysis for natural language processing models. arXiv preprint wrXiv:1588.05154 2015.
Alexandru Niculescu-Mizil and Rich Caruana. Predicting good probabilities with supervised learning. In Proceedings of the 22nd international conference Machine learning, pp. 625-632, 2005.
Jeremy Nixon, Balaji Lakshminarayanan, and Dustin Tran. Why are bootstrapped deep ensembles not better? In "I Can't I Believe It's 's Not Better! Better!"NeurIP" "NeurIPS 2020 workshop, 2020.
John Platt al. Probabilistic outputs for support vector machines and comparisons regularized likelihood methods. Advances in large margin classifiers, 10(3):1--7,, 1999.
Weichen Shen. Deepetr: Easy-to--ss,mmduuar and extendible package of deep-learning based ctr models. https: http://www.....eeeeeeeeeeeeeeeea
2017.
Zheyan Shen, Jiashuo Liu, Yue He, Xingxuan Zhang, Renzhe Xu, Han Yu, and Peng Cui. Towards out-of-distribution generalization: A survey, arXiv preprint arXiv:2108.1366 13624, 2021.
James E Smith and Robert L Winkler. The optimizer's curse: Skepticism and postdecision surprise in decision analysis. Management Science, 52(3):311-322, 2006.
Statista. U.s. digital advertising industry statistics & facts, 2022.
URL hetps://wwi www
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Eric Van den Steen. Rational overoptimism (and other biases). American Economic Review, 94(4):
1141-1151, 2004.
Aad W Van der Vaart. Asymptotic statistics, volume Cambridge university press, 2000.
Hado Van Hasselt. Double q-learning. Advances in neural information processing systems, 23: 2613-2621, 2010.
Hado Van Hasselt. Estimating the maximum expected value: an analysis of (nested) cross validation and the maximum sample average. arXiv preprint arXiv: 1302.7177 2013.
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q- learning. In Proceedings of the AAAI conference on artificial intelligence, volume 30, 2016.
Hado Philip Van Hasselt. Insights in reinforcement rearning: formal analysis and empirical evaluation of temporul-differene learning algorithms. Utrecht University, 2011.
Sudhir Varma and Richard Simon. Bias in error estimation when using cross-validation for model selection. BMC bioinformatics, 7(1): 2006.
Jindong Wang, Cuiling Lan, Chang Liu, Yidong Ouyang, Wenjun Zeng, and Tao Qin. Generalizing to unseen domains: A survey domain generalization. arXiv preprint arXiv:2003.03097, 2021.
Bianca Zadrozny and Charles Elkan. Obtaining calibrated probability estimates from decision trees and naive bayesian classifiers. In loml, volume 1, pp. 609-616. Citeseer, 2001.
Bianca Zadrozny and Charles Elkan. Transforming classifier scores into accurate multiclass probabil- ity estimates. In Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 694-699, 2002.
12

---

Published as a conference paper at ICLR 2023
A PROOFS
Before the proof of Theorem 4.1 we first collect some useful results from the standard MLE theory.
Lemma A.1. 3 follows the central limit theorem:
VN (3x (BN- -.) B.) = N(
where I is the Fisher information matrix defined as
02 (Y (n (8.
(). x)) (1- L)) (8, x3.x
Iju: = EDDania
08,0,R
Furthermore, the bias of B of order 1/N, L.e, E[3N - =.1=
Proof. The first claim follows from the Lipschitzness of the link function & and Theorem 5.39 in Van der Vaart (2000). The second claim follows from formula 20 in Cox & Snell (168)).
Let Z (0 1), then conditional 3%. have 25X N N (B. μ. EB.) the standard normal distribution. Then, we have for the estimated average proba- bility on the selection set conditional on Ax.
and 3_ X
+ (3xx) 1 (3x =E [ (3% ++ v 1 1z 2 (1),
EDicst
Note that conditional on N. COVDene 3T = B3..G. Therefore, we have
B 3xx² x5x)
x x
x +
B,
BNEBN
BREEN
where
33
8.
TNX
1 3NX
3N, B..
EBN
Note that
(BNI3.)
22
8. x
B
BN, 3. N
3T. S.E.
EB,
3 BBN
BN ENEBB
then, we have
(BEES.) 2
BNEB.
B x d
Z +
ΣTES.
Z2 + B H.
where Z2 (O, 1) independent to Z and d probability can be reformulated as
means equal in distribution. Thus, the actual average
E
[0 1
x
(3T,3.)
BTEB
EE
7 +
3T ETE.
Zo2 + 3 1
Z 11 (2),88
/ IBN
13

---

Published as a conference paper at ICLR 2023
Note that 3N only depends on the training set. Therefore, BN. Z, Zz are mutually independent. By the Taylor expansion, we have
(3N23,)
EN. Z E,
E
V /
BN EBN
Z/ B2 p
Z
BEE33
7 13
7
V
(3TE3)) 2
BEES.
+ Exicain
6 VBNIBB
Z + 3I
BT MB,
z
E
1Z
+
0
BI.BBN
N
= EDeratn
Z + B.T
Z +0 N
By Lemma A.1
have
)1
Exeaai
B BT
BN-I3N
(B VONINNZ Z
BI VBNEAN
Finally by taking the Taylor expansion of Φ
around
+ 33 ,4, we
have
1 7 33/14
3EE V
BNEB,
Eperatt
Φ
7 + B.T 1
Z
Derrann
Φ
2 +0 NV.
VBNIBN
Therefore, the actual average probability is
E ④ (3Tx)
13x
( BN) =
Φ
Z + SKH1
22 2
Let
h(t) E [o + 22 (2),
and
(1()) E [o' (BT 12) 2 ạ1 (Z)
Then, by using the Taylor expansion again, the maximization bias is
E
(xxx)
1 2 -1 (37x((3111
E (3. X)
1 (3xx
AN + V 122 (2)
E
B3 2 + BrTH)
+ 2 22 (a
VANIAN ATRe
+0 (1( VATAAA
E
d((t)dt
N
14

---

Published as a conference paper at ICLR 2023
Conditional on PN. we have + ~ N (oKu, 13
Proof of Corollary
4.2
Then, have
D
e (A + 1) 1 2 -1-a (,))
Φ (Brut X BN 2 2 91-a (2), BN
E
The remaining proof follows similar lines with the proof of Theorem 4.1
Proof of Equation
o Note that IE [3x] = :.: we have
E = [B + E [BN B.)1
3| + B 3.) E(BN - B.)]-
Then, the second term on the right hand size is equivalent to
E| B B.) I(3N 3.)]
El [ - Ba) T (X- (x-x)(X- - (0T( B.31
((X--M) - T B.))²
E
By taking expectation conditional on X, we have
[IIX- -5-) ([(X-p)T(C) [Bx - |x = Var (Ix-1)T
1) ( Bwlx).
I De
which is because E [N = Br. By the tower property, we have the desired result.
Lemma A.2. We assume
1. o' d'xx) S for E R and d'(x) 2 co > 0for x E [1,r].
2. P (B E (rr, 2 ci and S μ + t(91-a S pr + 20,91-a(2) S for some
t tr.
Then, we have for E [4,, [r] and a 5 0.2,
0o01
E 7 7 2 - (Z)) & h(())
< CE Z (Z))
2
Proof. The upper bound is immediate as ø' (BN + tZ) < C and
h'(t)
E [0' (3 + +Z)= Z 2 (2)
< CE 2 (Z)
Now, we focus on the lower bound.
A'(t)
> E Jo' (BNP + tZ) Z [p 44 111l 2 (Z)
E [ (Bhm+ + 12) 2TT E [1,, 7 [41-a 2 2 (z)]
Note that when
3NP (.,, [t,, E (2) (2))))2)(((2)
we have
o' d3 tZ) >0.
Then, (t(t) is lower bounded by
(1())
2 E. ZII E [44, pr], 2 (9 (2), 2 (2)
coc E. [ZT{Z (1) ( >
(Z)]
15

---

Published as a conference paper at ICLR 2023
Since 0.2, have
E3 12112 E (0l-o(2)221-a(22)) g1-a
E [z|z 2 --
(Z)], (
which yields the desired lower bound.
B NUMERICAL RESULTS
B.1 SYNTHETIC DATA
In this section, we report additional results on synthetic dataset. We plot average calibration errors,
average ECE and average MCE for [2%, 10%) in Figure 3 MCE is defined as
ΣKEB.. Yk
ΣKEB.,, f(xx)
MCE
max mE(
[Bm]
[Bm]
We test methods with different hyperparameters. Specifically, we test our method with S = 3. and with identity mapping ((x) = x (denoted as VAD(p)). The results are summarized in Tables 6 We find that for the VAD method, S outperforms S = 2, with the cost of more training resources needed. VAD(p) and VAD have similar performances.
Vanille VAD
0,07
Veniaa VAD
0.16
Vanilla
Collaation Enver
8%
0.06
0.14
VAD
8%
0.12
456
0.05
256
0.04
0.00
-256
0.03
0.06
2%
0%
65%
10%
2%
43
8%
10%
2%
5%
6%
8%
10%
(a) Calibration Error
(b) ECE
MCE
Figure 3: Average calibration errors, ECE, and MCE on the synthetic data
Tables reports the Log Loss reduction on the selection set. Log Loss defined as
LogLoss(Dr tests /)
Σ Vi log(((()))) + (1 - ys) log f(x)))
(ost
DE
Then, the Log Loss reduction is defined by
LogLoss (D. fVAD)
LogLoss(Dp Pesst fvauilia)
Negative Log Loss reduction means that we achieve lower loss. We find that after applying our method, we achieve lower Log Loss, meaning that we improve the prediction quality.
Table 6: Average and standard errors of MCE on synthetic data
Vanilla
VAD
VAD(p)
VAD (S=3)
0= 2%
0.1502+-0.041
0.1333+0.0035
0.1327+0.0035
0.1300+0..034
0=
0.0041
0.1130+000031
0.1118+0.0032
0.1119+000031
Q= %
0.1098110.0033
0.0955+0.0028
0.09441000028
0.0965+0.0028
a=59 %
0.1080-10.0033
0.09261:000026
0.001510.0026
0.0916+0.0025
a=6%
0.0972++00033.
0.0815+0.0025
0.0818+0.0024
0.0810+0.0024
a=7%
0.0927+00.0029
0.0789+0.0025
0.078410.0024
0.0793+0.0024
8=
0.0082
0.0033+0.0023
0.0031
0.0743-10.0024
0= %%
0.0030-00.0000
0.0706+10.0023
0.00944
0.0093
10%
0.0836+1000277
0.070410.0022
0.0698+0.0022
0.0690-L0.002
16

---

Published as a conference paper at ICLR 2023
Table 4: Average and standard errors of calibration errors on synthetic data
Vanilla
VAD
VAD(p)
VAD (S=3)
a=
8.55%:10.68%
0.06%:10.72
1.00%=0.70%
0.34%==0.63%
a=3%
8.05%=0.70%%
-0.5%%:1.699%
0.28%=0.68%
-0.5%%+0.64%%
Q=4%
8.09%:10.70%%
-0.60%:10.67%
0.40%=00.5%
-0.23%=10.67%
a=5%
8.19%=00.9%%
-0.13%=00.1%%
0.95%=0.69%
-0.10%=+0.66%
a=6%
8.09%:10669%
0.26%:1:0.1%%
0.75%=0.71%
-0.10%+00.67%
a=7%
7.85%%
-0.08%
1.11%=0.715
+0.70%
0=
7.69%%
0.20%
L.12%=0.669
-0.24%+00.67%
a=9%
7.74%%
-0.1%%1..71%
0.94%=0.71%
-0.1%=10.71%
10%
7.35%
0.62%=0..3%%
0.70%=0.72%
-.08%%10.70%%
Table 5: Average and standard errors of ECE on synthetic data
Vanilla
VAD
VAD(p)
VAD (S=3)
0=2%
0.0019
0.0572+-000013
0.0566++0.0013
0.0055++0.0013
a=3%
0.0566=1000000
0,0478=0.0013
0.047420.0013
0.046310.0013
Q=4%
0.0526110.0221
0.0414:100014
0.040910.0013
0.0416::00012
a=5%
0.0510::0.000
0,041410.0013
0.0410=0.0013
0.040410.0012
a=6%
0.0472=+0.0021
0.0385+0.0014
0.0384+0.0013
0.0069+0.0013
a=7 %
0.0459=:0.0021
0.0367=000014
0.0363=0.0014
0.0358+0000113
a=
0.045110.0020
0.0340+1.00012
0.0001
0=9%
0.0437+-0.001
0.0338
0.0338
a= 10%
0.0425+0.0021
0.033410.0015
0.033140.0014
0.0326-10.0013
Table 7: Log loss reduction synthetic data by applying VAD
2%
3%
4%
5%
6%
7%
8%
9%
10%
-0.45%
-0.42%
-0.42%
-0.37%
-0.35%
-0.33%
-0.37%
-0.32%
-0.30%
B.2 CRITEO KAGGLE DATASET
In this section, we report additional results on the Criteo Ad Kaggle dataset. Similarly with Appendix B.I we test more hyperparameters. Note that for the VAD method with 3, we replicate 26 times. The results are summarized in Tables 8 and 10 We plot average MCE for E [2%, 10%] in Figure 4 We find that for the VAD method, S outperforms S = 2 slightly, with the cost of more training resources needed. VAD(p) and VAD have similar performances. In addition. Table reports the Log Loss reduction. We find that after applying our method, we achieve lower Log Loss uniformly, meaning that improve the prediction quality.
17

---

Published as a conference paper at ICLR 2023
0.08
-Vaannllla
VAD
Platt Scaling
Platt Scaling
0.06
-............................... -
MCE
0.04
0.02
2%
4%
6%
8%
10%
a
Figure 4: MCE the Criteo Ad Kaggle dataset
Table 8: Average and standard errors calibration errors on Criteo Ad Kaggle dataset
Ct
Methods
Original
VAD+
VAD(p)+
VAD+ (S=3)
Vanilla
3.53%=
-0.38%-10.50 50%%
-0.54%110.46
-0.7%%110.61% %
Histogram Binning
2.19%:E0.06%
1.40%=0.06%
1.26%=10066
1.40%
3%
Platt Scaling
1.77%=
0.98%=10.1 %
0.81%=0.11%%
1.00%:10.13%
Scaling-Binning
2.27%
1.48%= %
1.30%=0.11%
1.50%=10.14%
Isotonic Regression
1.86%=+0.05%
1.06%=0.05%
0.89%=0.06%
1.06%+10.05%
Vanilla
3.82%%00.52%
-0.6%%50.56%
-0.1%=+0.53%
-0.76%=10669%
Histogram Binning
2.00%%++..06%
1.19%-+00.55%
1.10%+0.06%
1.18%++0.06%
5%
Platt Scaling
1.88%++O. 10%%
1.03%%+0..0%
0.95%+0.10%
1.06%++0.12%
Scaling- Biinnng
2.22% 10%
1.37%:1..10%
1.28%
1.39%-10.12%
Isotonic Regression
1.70%
0.94%+0.05%
0.85%
0.95%
Vanilla
3.90%=10.57%
-0.4%%-0.61%
0.04%
-0.0%%11.74%%
Histogram Binning
1.92%
1.05%:10.04%
1.02%
1.03%-10.05%
7%
Platt Scaling
1.86%=00.08%
0.99%+0.08%
0.95%=0.08%
0.97%=10.10%
Scaling Binnnin
2.19%
1.26%=00.0%%
1.22%=0008%%
1.24%=+0.10%
Isotonic Regression
1.70%
0.84%%+0.04%
0.81%+0.04%
0.81%-+0.05%
Vanilla
3.93%-+0.61 %
00446%+0.64%
0.16%+0.614 %
+1.00%
Histogram Binning
1.80%%+
1.00%+0.04%
1.01%+0.04%
0.95%++0.05%
9%
Platt Scaling
1.82%%
0.93%
0.93%=0.08%
0.89%%+0.10%
Scaling-Binning
2.00%%
1.15%:10.079%
1.15%
1.11%-10.10%
Isotonic Regression
1.74%
0.84%
0.80%%
0.79%-10.05%
18

---

Published as a conference paper at ICLR 2023
Table 9; Average and standard emrors of ECE on Critco Ad Kaggle dataset
Methods
Original
VAD+
VAD(p)+
VAD+ (S=3)
Vanilla
0.029010.0026
0.0218=0.0018
0.021910.0015
0.022310.0020
Histogram Binning
0.0190=100003
0.0160=0.0003
0.015410.0003
0.015920.0003
3%
Platt Scaling
0.0169=0.0005
0.014410.0004
0.0042+000003
0.0145::0.0044
Scaling- Scaling-Binning
0.0192:#0.006
0.0161=0.0005
0.0153+0.0004
0.0160=0.0006
Isotonic Regression
0.0001=0.0003
0.0157=0.0003
0.0155+0.0003
0.0155#0.0003
Vanilla
0.0287:100028
0.0213=0.0020
0.0211
0.0217=0.0024
Histogram Binning
0.0003+000003
0.0133+00000
0.0128+0.0002
0.0132++00003
5%
Platt Scaling
0.0051+00.005
0.0121+0.0000
0.0120+0.0003
0.0119+0.0004
Sealing-Binning
0.0166=0.0005
0.0032=0.0004
0.012810.0004
0.013310.0005
Isotonic Regression
0.0156+1000003
0.0130=0.0003
0.0129+0.0003
0.0130+0.0003
Vanilla
0.0201100.029
0.0211#0.0021
0.0009
0.0216:000025
Histogram Binning
0.0146:10.0002
0.01171000002
0.011410.0002
0.01171:00003
7%
Platt Scaling
0.0137=000004
0.0108=0.0003
0.0107#0.0003
0.0107#0.0003
Scaling- Binnnin
0.0149=0.0004
0.0116=0.0003
0.011410.0003
0.01181:00003
Isotonic Regression
0.0002
0.0012
0.0113+0.0002
0.0112+000003
Vanilla
0.0274+00.0029
0.0208+0.0022
0.0205+000020
0.0212+000006
Histogram Binning
0.0135++0.0002
0.0007
0.0105+0.0002
0.0106+000002
9% Platt Scaling
0.0125=10.0044
0.0096=0.0002
0.00961:000003
0.009510.0003
Scaling-Binning
0.0136:10.0004
0.010410.0002
0.010310.0003
0.010510.0003
Isotonic Regression
0.0129=10.0022
0.010410.0002
0.010410.0002
0.010410.0003
Table 10: Average and standard errors of MCE on Criteo Ad Kaggle dataset
Methods
Original
VAD+
VAD(p)+
VAD+ (S=3)
Vanilla
0.0655:0.0037
0.0551
0.056110.0026
0.05471100035
Histogram Binning
0.0524:10.011
0.046711.0001
0.0459110.0011
0.0448:10.0012
3%
Platt Scaling
0.0510=0.0015
0.046010.0013
0.0457:0.0013
0.045410.0017
Scaling- BiinningBinnig
0.0513=0.0013
0.046411.0012
0.0451#0.0011
0.0462+t0.0013
Isotonic Regression
0.0572-10.0166
0.0015
0.0510:0.0016
0.051010.0018
Vanilla
0.0575=0.0035
0.0477+1.0027
0.0486:10.0024
0.0469:10.0334
Histogram Binning
0.0475=0.0011
0.0420+0.0011
0.0412+0.0012
0.0417+0.0012
5%
Platt Scaling
0.0424=0.0010
0.0367+0.0010
0.0367+10.000
0.0363+10.002
Baaling-Binning
0.005110.0013
0.0093
0.0083
0.0396:10.0016
Isotonic Regression
0.047411.0016
0.0416:0.0005
0.041510.0015
0.040410.0012
Vanilla
0.0532:10.0035
0.044
0.0460-10.002
0.0451+00.031
Histogram Binning
0.0402++000099
0.0351 0.00009
0.0040
0.0344:10.0012
7%
Platt Scaling
0.0378=0.0010
0.0326+10.000
0.0327+0.0010
0.0325+0.0012
0.0405=0.0001
0.035410.0011
0.0330-10.0010
0.036140.0016
Isotonic Regression
0.0420=00.012
0.0062+0.0012
0.0364+0.0011
0.0357+0.0011
Vanilla
0.0001=0.0002
0.0421
0.0432+000022
0.0422+000029
Histogram Binning
0.0363=000008
0.0310+0.0007
0.0309+10.0008
0.0296+0000088
9%
Platt Scaling
0.0349-10.0008
0.0295-10.0009
0.0296110.0008
0.0296:10.0009
Scaling-Binn Binninn
0.0076-10.0010
0.0027
0.0312+10.0009
0.0333-40.0012
Isotonic Regression
0.0386:000010
0.0035-4..0110
0.0032-L0.0009
0.0322+40.0009
19

---

Published as a conference paper at ICLR 2023
Table 11: Log loss reduction Critco Ad Kaggle dataset by applying VAD
Methods
a=
a==%
a=4%
a=5%
a=6%
ax7%
a=
a=9%
a= 10%
Vanilla
-0.27%
-0.26%
-0.25%
-0.23%
-0.21%
-0.20%
-0. 0.19%
-0.18%
-0.17%
Histogram Binning
-0.08%
-0.07%
-0.06%
-0.05%
-0.04%
-0.04%
-0.04%
-0.03%
-0.03%
Platt Scaling
-0.05%
-0.05%
-0.05%
-0.04%
-0.04%
-0.04%
-0.03%
-0.03%
-0.03%
Scaling-Binning
-0.09%
-0.07%
-0.06%
-0.06%
-0.05%
-0.04%
-0.04%
-0.04%
-0.04%
Isotonic Regression
-0.05%
-0.05%
-0.04%
+0.04%
-0.04%
-0.03%
-0.03%
-0.03%
-0.03%
20

---

Published as a conference paper at ICLR 2023
We further check the performance using different bin numbers M {30, 40, 60, 70, in Tables 12 to We observe that regardless of the number of bins we choose, our method always outperforms.
15
Table 12: Average and standard errors of ECE on Criteo Ad Kaggle dataset (M = 30,40) 40)
Original (M=30)
VAD+ (M=30)
Original (M=40)
VAD+ (M=40)
Vanilla
0.0279=1000211
0.0204=0.0019
0.0285=0.0026
0.0211
Histogram Binning
0.017410.0003
0.0139=000003
0.0181#0.0003
0.0149#0.00%
3%
Platt Scaling Scaling- Baannng
0.0152++0.0000
0.0122++0.004
0.0159+0.0005
0.0132++0.000
0.0179++0.0006
0.0143+000005
0.0185++0.0006
0.0152++0.000.
Isotonic Regression
0.0161+0.0003
0.0133+000003
0.017110.0003
0.0145
Vanilla
0.0283+10.029
0.0203+L0.002
0.028510.0009
0.0209+0.0020
Histogram Binning
0.0150:10:0003
0.0115+100003
0.015810.0003
0.0126 0.0002
5%
Platt Scaling
0.0138+1000005
0.0103
0.014410.0005
0.01 00.0000
Scaling- Scaling-BBinnng
0,0156+10.0006
0.0117=0.0004
0.0163=000005
0.0127 .0004
Isotonic Regression
0.0143-000003
0.01144000003
0.014910.0003
0.0122 0.002
Vanilla
0.0276:10.0030
0.0203#0.0022
0.0277+0.0030
0.0207
Histogram Binning
0.0136++000002
0.0103+0.0002
0.0140+000002
0.0111+0.0002
7%
Platt Scaling
0.0127++0.000
0.0092+00.0003
0.0132+0.0000
0.0100++00000
Scaling- Binning
0.0142+10.0004
0.0104
0.0044
0.01
Isotonic Regression
0.012811.0002
0.010110.0002
0.0132+100002
0.0106+10.0002
Vanilla
0.0269100.0300
0.0202+10.0022
0.0272+00.030
0.0205
Histogram Binning
0.0126:10.0002
0.0094+00.0002
0.012
0.0099+0.0000
9%
Platt Scaling
0.0117=000004
0.008
0.0121
0.0090
Scaling- Scaling-BBining
0.0128+10.000. 0.0119++0.0002
0.0092+000003
0.0131
0.0096=10.0003
Isotonic Regression
0.0090++0.0002
0.0124+0.0002
0.0098
Table 13: Average and standard errors of ECE Criteo Ad Kaggle dataset (M 60, 70)
Original (M=60)
VAD+ (M=60)
Original (M=70)
VAD+ (M=70)
Vanilla
0.0293--00006
0.0226++0.0017
0.0300++0.005
0.0232+00.001
Histogram Binning
0.0196110.0005
0.0168+0.0003
0.020410.0003
0.0177+000000
3%
Platt Scaling
0.01771000000
0.015410.0003
0.0185:10.0005
0.0163:1.0000:
Scaling-Binning
0.0199=10.0008
0.0169=10.004
0.0206+10.0005
0.0177+000004
Isotonic Regression
0.0188++0.003
0.0166+10.0003
0.0195+:0.0003
0.0173+0.0003
Vanilla
0.0292+0.0028
0.0219+0.0019
0.029510.0028
0.0224+0.0019
Histogram Binning
0.016
0.0140=10.0002
0.017410.0003
0.0147+0.0002
5%
Platt Scaling
0.0158+
0.0131+0.0003
0.0163+000004
0.0138 +0.0003
Scaling BaalingBiinnig
0.0172-+0.0005
0.0140+000004
0.0176+000005
0.0146+0.0003
Isotonic Regression
0.0163+10.0022
0.013810.0002
0.017110.0002
0.014940.0002
Vanilla
0.0283-10.0021
0.021410.0020
0.02861000029
0.0218
Histogram Binning
0.015110.0000
0.012410.0002
0.0157+000002
0.0131 0.00
7%
Platt Scaling
0.004140.0004
0.0112+00.002
0.0147+000004
0.012040.0002
Scaling -innning
0.0155+100004
0.0125+0.0002
0.0159+0.0004
0.0130+0.0000
Isotonic Regression
0.0146+0.00022
0.0123+0.0002
0.0149+0.0002
0.0127+0.0002
Vanilla
0.0276+10.0029
0.0212+0.0021
0.0277+0.0029
0.0213+000021
Histogram Binning
0.0139++0.0002
0.0113+0.0002
0.0042+0000022
0.0116+0.0001
9% Platt Scaling
0.0129++0.0003
0.0102+000002
0.0133+-00000
0.0106++0.0002
Scaling-Binning Binning
0.013910.0004
0.0109+0000002
0.014310.0003
0.011440.0002
Isotonic Regression
0.01341100002
0.0110-10.0002
0.0138+000002
0.0115
21

---

Published as a conference paper at ICLR 2023
Table 14: Average and standard errors MCE on Criteo Ad Kaggle dataset (M 30, 40)
Original (M=30)
VAD+ (M=30)
Original (M=40)
VAD+ (M=40)
Vanilla
0.0542:10.0035
0.0442::000266
0.058910.0033
0.0494:00.004
Histogram Binning
0.043410.0010
0.0380=0.0010
0.0482+100011
0.0427:0.0011
3%
Platt Scaling
0.0398=10.0011
0.0341=0.0011
0.04451:000013
0.0391=0.0012
0.0415=+0.001
0.0364=0.0013
0.0483=0.0014
0.0431+0.0014
Isotonic Regression
0.0465=+0.0014
0.0406=0.0013
0.0531
0.0473++00014
Vanilla
0,0485=000035
0.0396=0.0025
0.0538+0.0037
0.0442++0.002
Histogram Binning
0.0379+1000100
0.0000+000010
0.0434+0.0010
0.0082++0.001
5%
Platt Scaling
0.0343++000009
0.0286+000008
0.0300+000010
0.0337++00000
Scaling- Binnnig
0.0369=1000011
0.0317+0.0011
0.0403+0.0013
0.0351=0.0013
Isotonic Regression
0.0389-10.0012
0.0336+0.0011
0.042410.0012
0.0368+00.001
Vanilla
0.0456-1000033
0.0005+0.0025
0.04921100034
0.0406
Histogram Binning
0.0341+0.0007
0.0007
0.03641000008
0.0311 0000
7%
Platt Scaling
0.0302
0.0248=t0.0008
0.0349+000009
0.0285
Scaling Binnning
0.0340=10.0009
0.0290=000009
0.0367
0.0316 0,009
Isotonic Regression
0.0349-10.0009
0.0000
0.0395+000009
0.0337
Vanilla
0.0434++0.0031
0.0353++0.004
0.0468+
0.0394+0.0000
Histogram Binning
0.0307++0.0006
0.0005+000006
0.0346+000008
+..93+000008
9%
Platt Scaling Bealing-Binning
0.0273++0.0007
0.0009+0.0007
0.0308
0.0255+0.0007
0.0010+100010 0.0331+0.0009
0.0267+000010
0.033810.0010
0.0288
Isotonic Regression
0.027410.0009
0.035810.0008
0.0303+100008
Table 15: Average and standard errors of MCE on Criteo Ad Kaggle dataset (M 60, 70)
Original (M=60)
VAD+ (M=60)
Original (M=70)
VAD+ (M=70)
Vanilla
0.0595+0.0027 0.0695+00.033
0.0752+000033
0.0644+0.0028
Histogram Binning
0.0568110.0112
0.0513:10.012
0.0610+1.0012
0.0560:10.001
3%
Platt Scaling
0.00481100012
0.049410.0011
0.060010.0012
0.054940.0011
Scaling- Biinng--Binningggg
0.0550=10.0012
0.0500=10.0011
0.0598:0.0015
0.0552:t0.0014
Isotonic Regression
0.0606+000014
0.0552+t0.001
0.065810.0016
0.0611+0.0015
Vanilla
0.0631+0.0035
0.0540+1.0022
0.06341:00033
0.0543+10.024
Histogram Binning
0.0001++0.001
0.0455+0.0011
0.0540++0.011
0.0485++0.0110
5%
Platt Scaling Scaling- Binnnin
0.0473++0.0010
0.0019+000010
0.0489+0000100
0.0436+0.0010
0.0484-100001
0.0433+0.0013
0.0511+0.0014
0.0461+0.0013
Isotonic Regression
0.0492:10.0155
0.0440-10.0014
0.0533+000015
0.0479+1000014
Vanilla
0.0559-10.0032
0.0478=10.0024
0.0605110.0032
0.0517+0.0026
Histogram Binning
0.0439+10.0010
0.0387+100000
0.0458110.00
0.0407+000000
7%
Platt Scaling
0.0404+0.0011
0.0349+0.0010
0.0446+10.000
0.0395+0.0007
Scaling Scalingg-Bining
0.0442:10.0012
0.0393+0.0011
0.0462+0.0012
0.0411#0.0012
Isotonic Regression
0.0454+00.0000
0.0090+0.0010
0.0482+000001
0.0427+0.0011
Vanilla
0.0533-10.032
0.044740.0023
0.055310.0032
0.0478+0.0026
Histogram Binning
0.0398+000000
0.0345+0.0009
0.0424+0.0010
0.0372+0.0010
9%
Platt Scaling
0.0372++0.0008
0.0318+0.0007
0.0397+0.0010
0.0346+0000009
Scaling-Binning
0.0401110.0010
0.0349+000009
0.04251100010
0.0373
Isotonic Regression
0.0416:10.0010
0.036410.0010
0.0452+0.0012
0.0399-10.0012
We then check the performance using different S E (4, 5, 6) in Tables to 18
We find that different
Ss have similar performance, except that for the Vanilla method, VAD+ Vanilla with 5 better.
is indeed
22

---

Published as a conference paper at ICLR 2023
Table 16: Average and standard errors of calibration errors Critco Ad Kaggle dataset (S 4,5,6)
Original
VAD+ (S=4)
VAD+ (S=5)
VAD+ (S=6)
Vanilla
3.53%: 2.19%:t0.06%
-0.8%:10.700
0.40%=1.70%% 0.20%=0.76%%
Histogram Binning
1.42%=10.08%
1.45%=0.099%
1.46%=0.10%
3%
Platt Scaling
1.77%
1.12%=+..15%
1.17%+0.17%
1.12%10.18%
Scaling- Biinngggiiningggg
2.27%:+t..12
1.62%=
1.68%
1.61%=0.19%
Isotonic Regression
1.86%++0.05%
1.08%+0.06%
1.08%+0.07%
1.12%+0.08%
Vanilla
3.82%%0.552%
-0.96%+0.77%
0.21%==0.85%
0.39%+0.79%%
Histogram Binning
2.03%-+.0.00
1.17%-+-0.06
1.20%%
+..00%
5%
Platt Scaling
1.88%-+-0. 100
1.16%-+..11%
1.24%
+..15%
Sealing-Binning
2.22%:
1.49%
1.57%=0.15%
1.4%%=0.16%
Isotonic Regression
1.79%=+0.00%%
0.91%=10006%
0.95%
0.89%+0.00%%
Vanilla
3.90%
-1.13%=0..82%
-0.1%==0911%
0.27%
Histogram Binning
1.92%=10.03%
0.99%:10.05%
1.01%:10.06%
0.90%%
7%
Platt Scaling
1.86%++0.08%
1.04%+0.11%
1.11%+0.12%
1.06%=0.13%
Scaling- Binnnin
2.13%+00.0%%
1.31%+#.111%
1.38%+0.12%
1.32%=0.13%%
Isotonic Regression
1.70%
0.77%
0.79%
0.77%
Vanilla
3.93%-+-0.11%
-1.26%
-0.48% 0.96%
0.17%
Histogram Binning
1.89%%+-..3%%
+..0%%
0.95% +.06%
0.96%
9% Platt Scaling
1.82%=00008%
0.94%
1.01%=1.11%%
0.97%=0.12%%
Scaling- Binnnng
2.04%:10.08%
1.17%=10.10%
1.24%
1.20%
Isotonic Regression
1.74%:
0.75%
0.80%+10.0%%
0.79%
Table 17: Average and standard errors of ECE on Criteo Ad Kaggle dataset (S =4,5,)
Original
VAD+ (S=4)
VAD+ (S=5)
VAD+ (S=6)
Vanilla
0.029010.0026
0.02241000025
0.021410.0027
0.019310.0021
Histogram Binning
0.0190+10.0003
0.01611000004
0.0162+0.0005
0.016110.0005
3%
Platt Scaling
0.016910.0005
0.014710.0005
0.015110.0006
0.014810.0005
Scaling- Scaling-Binning
0.0192+100006
0.016510.0007
0.0165+000008
0.015910.0008
Isotonic Regression
0.000
0.0152=1.0003
0.015310.0004
0.015110.0004
Vanilla
0.0287+0.0028
0.021310.0028
0.02031000031
0.01841000025
Histogram Binning
0.0163:10.000
0.0133=000003
0.0132+0.0003
0.0130+10.004
5%
Platt Scaling
0.0051
0.0120+0.0005
0.0124+0.0005
0.0119+:00000
Sealing-Binning
0.0166:10.0005
0.0037+0.0006
0.013810.0007
0.01321:000088
Isotonic Regression
0.015610.0005
0.013010.0004
0.0000
0.013010.0005
Vanilla
0.0081
0.0213+0.0028
0.0202+0.0032
0.018110.0026
Histogram Binning
0.0146:10.0002
0.0003
0.00144100003
0.0113+-00003
7%
Platt Scaling
0.0137-000004
0.010810.0004
0.011240.0000
0.010810.0005
Scaling- Biinnig
0.0149=000004
0.0120-000004
0.0121+:00005
0.01181:0.005
Isotonic Regression
0.0038+0.0002
0.0013+0.0003
0.0112+0.0004
0.0110+0.0004
Vanilla
0.027410.0029
0.0208+0.0029
0.0198+0.0033
0.0177+000027
Histogram Binning
0.0135++0.002
0.0105+0.0002
0.0105+0.0003
0.0103+0.0000
9%
Platt Scaling
0.0125-00.004
0.0004
0.009
0.00941:0.0004
BaalingBBinning
0.0034
0.0004
0.010
0.0105110.0004
Isotonic Regression
0.0020
0.0003
0.0103-10.0003
0.01011:0.0044
23

---

Published as a conference paper at ICLR 2023
Table 18: Average and standard errors of MCE Criteo Ad Kaggle dataset (S = 4,5,6)
Original
VAD+ (S=4)
VAD+ (S=5)
VAD+ (S=6)
Vanilla
0.0035+00.037 0.052410.0011
0.0541=0.0033
0..0044
0.0511+0.0037
Histogram Binning
0.0457=0.0012
0.0453+0.0015
0.0460=0.0017
3%
Platt Scaling
0.0510=1.0015
0.0452=100019
0.046310.0021
0.0451#0.0019
Scaling-Binning
0.0013=1.0013
0.0472:100012
0.046910.0003
0.046010.0011
Isotonic Regression
0.0572+10.016
0.0495=0.0022
0.0500+0.0027
0.0488+:0.007
Vanilla
0.0575=100033
0.0455=0.0035
0.0442+0.0000
0.0413::00031
Histogram Binning
0.0475=100011
0.0417=0.0014
0.0420=100006
0.0419=:.0018
5%
Platt Scaling Baaling-Binning
0.00241000010
0.0360=0.0014
0.006310.0016
0.0352:10.0017
0.0451+0.0013
0.009
0.0390+0.0020
0.0388+0.0024
Isotonic Regression
0.047410.0016
0.0403+0.0015
0.0412+0.0017
0.0407+0.0017
Vanilla
0.0032+0.0035
0.0444+0.0034
0.00399000099
0.0418+000034
Histogram Binning
0.0409+1.0009
0.0347=0.0014
0.00311:.0004
0.033310.0016
7%
Platt Scaling
0.00701000010
0.032810.0015
0.033210.0018
0.032910.0022
Scaling- Scaling-Binning
0.0005
0.0370=0.0017
0.0357+0.0014
0.0349=000006
Isotonic Regression
0.0420+0.0012
0.0359=0.0013
0.0357+0.0015
0.0355#0.0018
Vanilla
0.0501
0.041810.0031
0.041710.0037
0.0401+0.0033
Histogram Binning
0.0363++000088
0.0289
0.020110.0010
0.028610.0011
Platt Scaling
0.0049+000008
0.030410.0011
0.0309+0.0011
0.0307+0.0013
0.0076+000010
0.0033
0.0336+0.0016
0.0319+0.0014
Isotonic Regression
0.0386+10.0110
0.0322+0.0012
0.031610.0014
0.0317+0.0016
24

---

Published as a conference paper at ICLR 2023
Finally, we present histograms of the last-layer neuron (logit) in the neural networks to justify Gaussiamity assumptions in Theorem [41] Pigure[3]plots the histograms of model predictions from models using three different random seeds, where the black line is the estimated Gaussian density. By the Kolmogorov-Smima test, we cannot reject the null hypothesis that the empirical distribution is Gaussian at the 5% significance level.
0.3
0.3
0.3
0.2
0.2
0.2
0.1
0.1
0.1
0
-2
-4
-4
(a)
(b)
(c)
Figure 5: Histograms of the last-layer neuron (logit) of the neural networks
B.3 AVAZU DATASET
Dataset We the Avazu CTR Prediction dataset demonstrate method's performance. The Avazu CTR Prediction dataset is common benchmark dataset for CTR predictions. It consists 10 days of click-through data, ordered chronologically. Due to computational constraints in our experiments, we use the first 10 million samples, shuffle the dataset randomly, and split the whole dataset into 77% train D Dirain 1.5% validation-tres Dral-train 1.5% validation-test - and 20% test Deest datasets.
Base Model We use the xDeepFM model (Lian et al. 2018) open-sourced in (hhen 2017).
Covariate Shift We train another xDeepFM model. We randomly keep each data point in the original test set with probability
p p 0.2, probability 0.1 if > 0.3 and probability 2.2 7 p 0.2 < p 0.3, where is the newly-trained xDeepFM model prediction for the data point. The training remains the same. By doing this, we ensure that the distributional change is only covariate shift, and the positive sample ratio in the test data is lower than the positive sample ratio in training data, which consistent with the real-world recommendation systems.
We report results for the calibration error, ECF, and MCE in Tables 19 20 and 21 respectively. We observe that all the calibration methods tandem with VAD achieve better performance than the sole calibration methods alone, which is consistent with results in our paper.
25

---

Published as a conference paper at ICLR 2023
Table 19: Average and standard emrors of calibration the Avazu dataset
Original
VAD+
Histogram Binning
3.65%:10.21%
1.32%:10.222%
Platt Scaling
3.77%:+0.24%
1.41%=+0.24%
3%
Scaling-Binning
3.78%:10.24%
1.42%:+0.24%
Isotonic Regression
3.77%:10.22%
1.42%:L1023%%
Histogram Binning
5.10%-10.26%
2.76%
Platt Scaling
5.19%
2.82%
5%
5.27%:10.27%
2.91%=10.27%
Isotonic Regression
5.17%:10.27%
2.80%:10.27%
Histogram Binning
5.79%=10.28%
3.47%:10.28%%
Platt Scaling
5.73%:10.28%
3.39%:1:.289%
7%
5.76%+10.2%%
3.42%++0.28
Isotonic Regression
5.67%:+0.27%
3.32%:+0.2899
Histogram Binning
6.07%:10.27%
3.78%+O0.27%
Platt Scaling
5.97%.++..9%%
3.65%
9%
Scaling-
5.98%++0.29%
3.66%%
Isotonic Regression
5.91%:10.27%%
3.59%:10.27%%
Table 20: Average and standard errors of ECE on the Avazu dataset
Original
VAD+
Histogram Binning
0.02091000007 0.0218::00007
0.018610.0005
Platt Scaling
0,0190=0.0007
3%
Boaling-Binningg
0.0212+10.0066
0.018410.0006
Isotonic Regression
0.0215+000007
0.0189+0.0005
Histogram Binning
0.0208++000077
0.0173+000005
Platt Scaling
0.0211+:0.0009
0.0172++0.0077
5%
Scaling-Binning
0.0213+10.000S
0.017310.0007
Isotonic Regression
0.0210+10.0007
0.017110.0005
Histogram Binning Platt Scaling
0.0202+10.000
0.0163-10.0066
0.0201+:0.0008
0.0157++0.000
7%
Scaling-Binning
0.0200+1000008
0.0157+0.0007
Isotonic Regression
0.0199+000007
0.005
Histogram Binning
0.0194+0.0006
0.0153+0.0005
Platt Scaling
0.0193++0.000
0.0148+000007
9%
Scaling-Binning
0.0191+0.0008
0.0147+000007
Isotonic Regression 0.0190-10.0006
0.0147:10.0005
26

---

Published as a conference paper at ICLR 2023
Table 21: Average and standard errors of MCE the Avazu dataset
Original
VAD+
Histogram Binning
0.063611.0002 0.061110.0017
0.0571#0.0022
Platt Scaling
0,0566:000018
3%
Scaling-Binning
0.0611+:00014
0.055410.0017
Isotonic Regression
0.0663++0.002
0.0599+0.0021
Histogram Binning
0.0544:1.0014
0.0486:1.0014
Platt Scaling
0.0524:10.001
0.047110.0019
5%
Scaling-Binning
0.0015-+0.0013
0.0463+000015
Isotonic Regression
0.0527+0.0013
0.0403:0.0005
Histogram Binning
0.0085
0.0420+0.0013
Platt Scaling
0.0489:1000016
0.0432+10.0016
7%
Scaling-Binnin
0.0457+0.0012
0.0409+0.0015
Isotonic Regression
0.0480+:0.0014
0.0428+0.0016
Histogram Binning
0.0469+0.0014
0.,0010
Platt Scaling
0.0425+0.0014
0.0081
9%
Bealing-Binning
0.00055
0.0005
Isotonic Regression 0.0454:10.0014
0.0389+0.0014
27


Reviews:
[{"review_id": "n3ufQxgC0D", "reviewer": "Reviewer_1fZs", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors study the problem of correcting maximization bias in advertising recommendation systems. The authors identify maximizing bias from the operation of ranking and select top a% ads and it presents even the model is well-calibrated on the entire distribution. The authors proposed a simple correction algorithm motivated via rigorous analysis of GLM with mild assumptions. Experiments on both synthetic and real-world datasets are carried out with comparison to other calibration methods.", "summary_of_the_review": "The paper studied a new source of bias in Ads ranking and proposed a simple and just-justified solution. However, there is some concern in empirical evaluation and practicality in real-world application.", "strength_and_weaknesses": "Strength\n1. The paper studied an important and practical problem in real-world Ads ranking systems.\n2. The proposed method is well-justified under mild conditions from GLM.\n3. The algorithm is simple to implement without additional inference cost. \n\nWeakness\n1. The results in Theorem 4.1 and Corollary 4.2 seem to suggest \\alpha does not play a role in deciding the \\lambda parameter in the algorithm. Consider the extreme case \\alpha=100% which is the original distribution; if there is no model bias, the algorithm should be a no-op and not change the prediction at all.\n2. Many real-world Ads ranking systems utilize online learning of model parameters. It would be interesting to see how the method can be generalized to online learning settings for more practical applications.\n3. It would be better if the authors could test the proposed methods on several other Ads ranking datasets. It would also help to experiment on a few different model structures. Moreover, an empirical validation of the Gaussian assumption would be informative.\n4. For the empirical evaluation, it would be better to have the results on \\alpha=100% to measure just the model bias of vanilla and other methods.\n5. For the evaluation results, why does the CE and ECE have different ordering and different performance by varying \\alpha. Intuitively, as \\alpha becomes bigger, the maximization bias becomes smaller and the comparing algorithm should have better performance. Also, though the proposed method does change accuracy, it would be better to provide the metrics like log loss and AUC as a reference for model performance.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "S59Ajw_mWL", "reviewer": "Reviewer_hQai", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Calibration is important for models used for predictions in online advertising systems. It can be challenging because there is a form of adverse selection, where any over-predictions are more likely to result in these ads being shown. There is also a challenge because the training data is typically the ads that were shown and so is not a random selected subset of ads.\n\nThis work proposes a method for a calibration step making use of an unlabelled test set that is unbiased (containing all possible candidates). Then it creates multiple estimators (e.g. through bootstrapping) and estimates the variance of their predictions on the test set. This is used to calibrate the predictions. There is both theoretical analysis and (offline) experiments of the approach.\n", "summary_of_the_review": "A good paper on an interesting topic. It may be too specialized for the ICLR audience to appreciate. I have some concerns about the practicality of the approach and it would be more compelling to see this tried online.\n", "strength_and_weaknesses": "Strengths:\n1. This is an important, practical problem.\n\n2. The paper is fairly well-explained although there are areas for improvement (particularly the jump from the theory with alpha and the practical algorithm).\n\n3. The offline results are compelling including on real datasets (both decreasing bias and improve accuracy).\n\nWeaknesses:\n1. This may be of more interest in a venue focused on online advertising.\n\n2. It would be more compelling to try an online experiment (but it may not be possible for the authors).\n\n3. The assumption that you can practically obtain all candidates may not be realistic in many industry settings  where that set is very large and constantly changing.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "DVjUHL7-X3", "reviewer": "Reviewer_imJk", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this work, the authors theoretically quantify the maximization bias in generalized linear models with Gaussian distributions. In practice, the maximization bias often manifests itself in the calibration of online advertising recommendation system. The authors propose a variance-adjusting de-biasing meta-algorithm that is able to mitigate the maximization bias problem, and is robust to covariate shifts between training and test sets that are common in modern recommendation systems. The algorithm can be used in tandem with other calibration methods without compromising the ranking performance nor increasing online serving overhead. The authors demonstrate the effectiveness of the proposed algorithm in both synthetic datasets using a logistic regression model and a large-scale real-world dataset using a state-of-the-art recommendation neural network model.", "summary_of_the_review": "Overall, this is a high-quality paper written with clarity. Its main contribution includes theoretically quantifying the maximization bias in a supervised learning settings in recommendation systems; the development of a novel, robust, and practical algorithm that optimizes calibration; the demonstration of the effectiveness of the meta-algorithm. Their results can be strengthened by showing comparison results against existing calibration methods that explicitly considers the maximization bias such as double learning or cross-validation estimators.", "strength_and_weaknesses": "Strength:\nThe paper is well motivated and clearly written, with several highlights:\n1.\tThey theoretically quantify the maximization bias in a supervised learning settings in recommendation systems. They prove theorems and corollaries that offer insights on the source of calibration error and the development of their novel algorithm.\n2.\tTheir proposed algorithm is theoretically supported and efficient, which makes it practical for any machine learning methods and can be used in tandem with any existing calibration methods. They also prove that it is robust to covariate shifts between training and test sets, which is common in modern recommendation systems.\n3.\tThe authors conducted extensive numerical experiments to demonstrate the effectiveness of the proposed meta-algorithm.\n\nWeaknesses:\n1.\tAs the authors mentioned, none of the baseline calibration methods they chose to benchmark against explicitly considers the maximization bias and thus fails to perform well in their setting. It would make their results more convincing if they can show some comparison results against existing calibration methods that explicitly considers the maximization bias such as double learning or cross-validation estimators. Even though these alternative estimators may not be practical in a large-scale ads recommendation system, they could still serve as valuable baselines and offer additional insights. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "5JphPmxVhq", "reviewer": "Reviewer_MXBz", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors focus on the problem of calibration in the online advertising setting, where the ML model has to predict the probability that an ad will be clicked. These probabilities are then used in the ad exchange to inform the ranking and selection of ads, but also to estimate the value of the chosen ad to charge the advertisers.\n\nSpecifically, the paper focuses on tackling maximization bias when calibrating the model. This maximization bias occurs in this setting because the same dataset is used to determine the maximizing action and to estimate its value.\n\nThe first contribution of the authors is a theoretical analysis of the problem in the case of generalized linear models with Gaussian features. This analysis leads to a debiasing term to adjust for this bias. This debiasing term is learned by retraining the model on bootstrap samples (or using different random seeds.)\n\nThe debiasing logic is then extrapolated to a more general algorithm that can be applied to any ML model.\n\nThe experimental section encompasses a synthetic study of a logistic regression model with Gaussian features (to emulate the analysis of the GLM case), as well as a deep learning recommendation model (DLRM) on the Criteo Ad dataset.", "summary_of_the_review": "I do see some novelty and appreciate the formal analysis (even though it only applies to GLM with Gaussian features). \n\nMy recommendation is a weak accept.", "strength_and_weaknesses": "Pros:\n- Analysis for the GLM case is sound. The analysis relies on the fact that the ad selection is so that only the top-scorer ads really matter.\n- Algorithm is simple and can be applied to any ML model.\n\nCons: \n- Requires multiple training runs (even though the authors suggest a single extra run might suffice)\n- The experiments rely on adding covariate shift (both for the synthetic and Criteo dataset), so it is unclear what the benefits of using this approach are in the case where there's little-to-none covariate shift.\n- The debiasing algorithm relies on learning the X margin in the test set.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster