Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors focus on the problem of calibration in the online advertising setting, where the ML model has to predict the probability that an ad will be clicked. These probabilities are then used in the ad exchange to inform the ranking and selection of ads, but also to estimate the value of the chosen ad to charge the advertisers.

Specifically, the paper focuses on tackling maximization bias when calibrating the model. This maximization bias occurs in this setting because the same dataset is used to determine the maximizing action and to estimate its value.

The first contribution of the authors is a theoretical analysis of the problem in the case of generalized linear models with Gaussian features. This analysis leads to a debiasing term to adjust for this bias. This debiasing term is learned by retraining the model on bootstrap samples (or using different random seeds.)

The debiasing logic is then extrapolated to a more general algorithm that can be applied to any ML model.

The experimental section encompasses a synthetic study of a logistic regression model with Gaussian features (to emulate the analysis of the GLM case), as well as a deep learning recommendation model (DLRM) on the Criteo Ad dataset.
strength_and_weaknesses: Pros:
- Analysis for the GLM case is sound. The analysis relies on the fact that the ad selection is so that only the top-scorer ads really matter.
- Algorithm is simple and can be applied to any ML model.

Cons: 
- Requires multiple training runs (even though the authors suggest a single extra run might suffice)
- The experiments rely on adding covariate shift (both for the synthetic and Criteo dataset), so it is unclear what the benefits of using this approach are in the case where there's little-to-none covariate shift.
- The debiasing algorithm relies on learning the X margin in the test set.

clarity,_quality,_novelty_and_reproducibility: The paper presentation can be improved in my opinion:
- The "Example 1" outlining a case of maximization bias isn't very clear. Should be rephrased/clarified or just replaced with a better example.
- Figure 1 flowchart is unclear and unhelpful.
- MCE: isn't used in main body, please move to Appendix.
- The claim that "h'(t) has the same order as ..." should be clarified and formalized in a proof in the Appendix.
- Algorithm 1: notation with f_i^\ell and f_i is quite confusing. I would suggest to define the predictors to be \psi(f_i(x)) and clarify that f_i can be obtained either by inverting \psi or looking at the last layer of the NN.
- Can you give some intuitive explanation on \sigma_f / \sigma_Y ? When is this ratio close to 1 or close to 0? Also why can't it be more than 1?
summary_of_the_review: I do see some novelty and appreciate the formal analysis (even though it only applies to GLM with Gaussian features). 

My recommendation is a weak accept.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Calibration is important for models used for predictions in online advertising systems. It can be challenging because there is a form of adverse selection, where any over-predictions are more likely to result in these ads being shown. There is also a challenge because the training data is typically the ads that were shown and so is not a random selected subset of ads.

This work proposes a method for a calibration step making use of an unlabelled test set that is unbiased (containing all possible candidates). Then it creates multiple estimators (e.g. through bootstrapping) and estimates the variance of their predictions on the test set. This is used to calibrate the predictions. There is both theoretical analysis and (offline) experiments of the approach.

strength_and_weaknesses: Strengths:
1. This is an important, practical problem.

2. The paper is fairly well-explained although there are areas for improvement (particularly the jump from the theory with alpha and the practical algorithm).

3. The offline results are compelling including on real datasets (both decreasing bias and improve accuracy).

Weaknesses:
1. This may be of more interest in a venue focused on online advertising.

2. It would be more compelling to try an online experiment (but it may not be possible for the authors).

3. The assumption that you can practically obtain all candidates may not be realistic in many industry settings  where that set is very large and constantly changing.
clarity,_quality,_novelty_and_reproducibility: Clarity:
The paper is fairly well-written and explained.

Quality: good quality.

Novelty: I believe this method is novel, including the theoretical analysis. It is a compelling direction to consider how to make use of unlabeled data in a practical problem.

Reproducibility: The code is made available by the authors (I did not run it) and the datasets are publically available so it should be reproducible.

summary_of_the_review: A good paper on an interesting topic. It may be too specialized for the ICLR audience to appreciate. I have some concerns about the practicality of the approach and it would be more compelling to see this tried online.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors study the problem of correcting maximization bias in advertising recommendation systems. The authors identify maximizing bias from the operation of ranking and select top a% ads and it presents even the model is well-calibrated on the entire distribution. The authors proposed a simple correction algorithm motivated via rigorous analysis of GLM with mild assumptions. Experiments on both synthetic and real-world datasets are carried out with comparison to other calibration methods.
strength_and_weaknesses: Strength
1. The paper studied an important and practical problem in real-world Ads ranking systems.
2. The proposed method is well-justified under mild conditions from GLM.
3. The algorithm is simple to implement without additional inference cost. 

Weakness
1. The results in Theorem 4.1 and Corollary 4.2 seem to suggest \alpha does not play a role in deciding the \lambda parameter in the algorithm. Consider the extreme case \alpha=100% which is the original distribution; if there is no model bias, the algorithm should be a no-op and not change the prediction at all.
2. Many real-world Ads ranking systems utilize online learning of model parameters. It would be interesting to see how the method can be generalized to online learning settings for more practical applications.
3. It would be better if the authors could test the proposed methods on several other Ads ranking datasets. It would also help to experiment on a few different model structures. Moreover, an empirical validation of the Gaussian assumption would be informative.
4. For the empirical evaluation, it would be better to have the results on \alpha=100% to measure just the model bias of vanilla and other methods.
5. For the evaluation results, why does the CE and ECE have different ordering and different performance by varying \alpha. Intuitively, as \alpha becomes bigger, the maximization bias becomes smaller and the comparing algorithm should have better performance. Also, though the proposed method does change accuracy, it would be better to provide the metrics like log loss and AUC as a reference for model performance.

clarity,_quality,_novelty_and_reproducibility: Overall, the paper is well written and easy to follow. The authors studied a new source of bias in Ads ranking. 
summary_of_the_review: The paper studied a new source of bias in Ads ranking and proposed a simple and just-justified solution. However, there is some concern in empirical evaluation and practicality in real-world application.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: In this work, the authors theoretically quantify the maximization bias in generalized linear models with Gaussian distributions. In practice, the maximization bias often manifests itself in the calibration of online advertising recommendation system. The authors propose a variance-adjusting de-biasing meta-algorithm that is able to mitigate the maximization bias problem, and is robust to covariate shifts between training and test sets that are common in modern recommendation systems. The algorithm can be used in tandem with other calibration methods without compromising the ranking performance nor increasing online serving overhead. The authors demonstrate the effectiveness of the proposed algorithm in both synthetic datasets using a logistic regression model and a large-scale real-world dataset using a state-of-the-art recommendation neural network model.
strength_and_weaknesses: Strength:
The paper is well motivated and clearly written, with several highlights:
1.	They theoretically quantify the maximization bias in a supervised learning settings in recommendation systems. They prove theorems and corollaries that offer insights on the source of calibration error and the development of their novel algorithm.
2.	Their proposed algorithm is theoretically supported and efficient, which makes it practical for any machine learning methods and can be used in tandem with any existing calibration methods. They also prove that it is robust to covariate shifts between training and test sets, which is common in modern recommendation systems.
3.	The authors conducted extensive numerical experiments to demonstrate the effectiveness of the proposed meta-algorithm.

Weaknesses:
1.	As the authors mentioned, none of the baseline calibration methods they chose to benchmark against explicitly considers the maximization bias and thus fails to perform well in their setting. It would make their results more convincing if they can show some comparison results against existing calibration methods that explicitly considers the maximization bias such as double learning or cross-validation estimators. Even though these alternative estimators may not be practical in a large-scale ads recommendation system, they could still serve as valuable baselines and offer additional insights. 

clarity,_quality,_novelty_and_reproducibility: Clarity: this paper is very well motivated and written. It is easy to follow and a good read.

Quality: this work is of high quality. The problem is well motivated and illustrated. The authors theoretically quantify the maximization bias and provide a theoretically proven algorithmic solution to tackle it. 

Novelty: this paper provides theoretical insights and proposes a novel meta-algorithm to address an important problem in large-scale ads recommendation system. 

Reproducibility: the authors demonstrate the effectiveness of the proposed algorithm using a state-of-the-art recommendation neural network model on a large-scale real-world dataset.

summary_of_the_review: Overall, this is a high-quality paper written with clarity. Its main contribution includes theoretically quantifying the maximization bias in a supervised learning settings in recommendation systems; the development of a novel, robust, and practical algorithm that optimizes calibration; the demonstration of the effectiveness of the meta-algorithm. Their results can be strengthened by showing comparison results against existing calibration methods that explicitly considers the maximization bias such as double learning or cross-validation estimators.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: All reviewers are positive about this paper, and the authors have addressed suggestions for improvement well.

As the AC, I have a few concerns or suggestions that I hope the authors will address in the final version.

1. Maximization bias is essentially the same thing as the so-called winner's curse, which arises in auctions in general. To make the paper more exciting and useful to a broader audience, discuss this connection.

2. The authors recommend S=2 at the top of page 7. This makes the VAD algorithm reminiscent of the double learning (DL) that is well known in RL. Please discuss.

3. The paper says that DL is "not applicable to large-scale ads recommendation systems since it will double online serving costs, and online serving efficiency is essential to recommendation system performances." But the second model in DL is needed just once, to estimate the value of the action chosen using the first model. In contrast, the first model must be invoked many times, on each candidate action, so DL does not double online serving costs. DL seems simple and affordable in practice, and in industry simplicity is of high value. One of the authors' responses shows that DL works very well in practice.

4. I am not convinced that the authors have really absorbed correctly and cited correctly the references. Consider the 2012 paper "Predicting accurate probabilities with a ranking loss." The current submission cites this paper with context "accurately estimate the value of the chosen ads" and "isotonic regression". Both these topics are mentioned in the 2012 paper, but it is definitely not the first or best reference for either topic, and in fact *choosing* ads is not mentioned at all.
justification_for_why_not_higher_score: This paper is not really about learning representations.
justification_for_why_not_lower_score: All reviewers and the AC agree that the problem tackled is important, and the reviewers believe that the main theorem is correct.
Decision: Accept: poster