Post-Calibration Techniques: Balancing Calibration and Score Distribution Alignment

Authors: agathe  fernandes machado
arthur  charpentier
emmanuel  flachaire
ewen  gallic
françois  hu
Venue: NeurIPS_cc_2024_Workshop_BDU
Published: 2024-09-03 22:22:11

Abstract: A binary scoring classifier can appear well-calibrated according to standard calibration metrics, even when the distribution of scores does not align with the distribution of the true events. In this paper, we investigate the impact of post-processing calibration on the score distribution (sometimes named "recalibration"). Using simulated data, where the true probability is known, followed by real-world datasets with prior knowledge on event distributions, we compare the performance of an XGBoost model before and after applying calibration techniques. The results show that while applying methods such as Platt scaling or isotonic regression can improve the model's calibration, they may also lead to an increase in the divergence between the score distribution and the underlying event probability distribution.

Main paper: 
 introduction: 
 When estimating a probabilistic scoring classifier, the model must not only discriminate between observations according to their class but also return scores that can be interpreted as probabilities. The distribution of scores produced by the classifier should align with the underlying event distribution. To assess whether classifiers return probabilistic scores, one must evaluate the model's calibration [4, 24, 8] . While some models, such as logistic regression when correctly specified, are known to be well-calibrated [22] , others, including ensemble methods like Random Forests (RF) [13, 3] and XGBoost [10] , are not inherently calibrated [28] . To assess a model's ability to provide probabilistic scores, the literature recommends evaluating its calibration using metrics like the Brier Score (BS, [4] ) or the Integrated Calibration Index (ICI, [1] ). When a model is not well-calibrated, post-processing calibration methods including Platt scaling [29] , Beta calibration [17] , or isotonic regression [37] are often applied to adjust the scores [21, 11, 17] . After applying calibration techniques, these metrics generally indicate an improvement in the model's calibration relative to its initial state.Since the true underlying probability distribution of the data is typically unobserved in practice, calibration metrics are assessed solely on the classifier's output range. Fernandes Machado et al. [9] demonstrated with simulated data that methods such as RF and XGBoost, can appear wellcalibrated according to standard calibration metrics and exhibit strong discrimination based on performance metrics, yet still fail to align the score distribution with the true event distribution. This discrepancy can arise when predicted scores from those algorithms lack the heterogeneity present in the underlying data distribution. They demonstrate this misalignment by comparing the selection of model hyperparameters based on Kullback-Leibler (KL) divergence with the selection based on performance or calibration metrics, knowing the true event distribution in the case of simulated data. For real data, where the true distribution is unknown, the approach involves prior information about the underlying data distribution to better align it with the predicted score distribution. Their analysis only considers model evaluation to accurately interpret predicted scores as probabilities, typically through calibration metrics. However, many practitioners employ post-calibration techniques to ensure that output scores represent probabilistic estimates. In this paper, we examine how post-calibration techniques affect the variability of score distribution in XGBoost binary classifiers, comparing it to the true underlying data distribution using KL divergence. We find that these post-processing methods often reduce score heterogeneity. Additionally, the misalignment between tree-based models optimized for KL divergence and those optimized for calibration or performance metrics persists and may even worsen after calibration, indicating that score alignment can decrease following postcalibration. Using simulated data with known true probabilities, followed by real-world datasets with prior knowledge of event distributions, we evaluate the abilities of XGBoost-predicted scores as probabilistic estimates before and after applying calibration techniques, with an emphasis on score distribution rather than solely on their calibration. 
calibration: 
 We focus on the context of a binary scoring classifier. Let Y ∈ Y = {0, 1} be a binary response variable, and let X ∈ X = R d denote features. The goal is to predict s(X) = P(Y = 1|X), using a sample of n i.i.d. observations (x i , y i ) n i=1 . We estimate this probability ŝ(x i ) ∈ [0, 1] using an XGBoost classifier, which produces a distribution of estimated scores ŝ(X). If the score distribution is poorly calibrated, these scores cannot be interpreted as the "true underlying probabilities" [33, 19, 16] . A model ŝ is well-calibrated for a binary variable Y when [31] :EQUATIONi.e., equivalently,E[Y | ŝ(X) = p] = p, ∀p ∈ [0, 1].To measure calibration, the literature suggests various metrics. Here, we focus on two of them: BS and ICI. The former [12, 17, 29, 30] , often used to assess a model's calibration, is a proper scoring rule that also accounts for refinement loss [18] . It writes: BS= n -1 n i=1 ŝ(x i ) -y i 2 [4]. Morerecently, Austin and Steyerberg [1] introduced the ICI, a metric that relies on the calibration curve.In the binary case, the calibration curve writes g :[0, 1] → [0, 1], p → g(p) := E[Y | ŝ(X) = p].For a well-calibrated model, the calibration curve corresponds to the identity function, g(p) = p, where the predicted score p equals the true likelihood of the event. Graphically, this is represented by the calibration curve aligning with the 45-degree diagonal. While the calibration curve is usually estimated using bins [35, 20, 27] , the ICI relies on a smoother version, based on splines. The empirical version writes ICI = n -1 n i=1 ŝ(x i ) -ĝ ŝ(x i ) , which corresponds to computing the average of the absolute difference between the estimated calibration curve and the identity function, the latter representing perfect calibration.When using scores generated by a model estimating the probability of a binary event, the literature advocates calibrating the model by applying the calibration curve g-which serves as a transformation function-on the scores [29, 37, 17, 20] . In this paper, we focus on three calibration methods: Platt scaling, isotonic regression, and Beta calibration.Platt Scaling This parametric approach consists of fitting a logistic regression to the binary response variable using predicted scores of a binary classifier as the unique feature [29] . The obtained calibrated probabilities are g(ŝ(x)) = 1 + exp -1 s (ŝ(x) -µ ) -1, where µ and s (s > 0 for a non-decreasing calibration map g) are estimated on a calibration set. It should be noted that Platt scaling is unable to learn the identity function g if the predicted scores are already calibrated [17] .The scores returned by a binary classifier are in range [0, 1]. Beta calibration [17] builds on this feature and assumes that the score within each class of the target variable y are distributed according to a Beta distribution. By contrast, Platt scaling assumes the scores follow a Normal distribution within each class. The calibration map writes g(ŝ(x)) = (1 + exp {-a log ŝ(x) + b log(1 -ŝ(x)) -c}) -1, where a, b, and c are the three parameters that need to be estimated on a calibration set. Unlike Platt scaling, Beta calibration can learn the identity function g (with a = b = 1, c = 0), making it suitable for already well-calibrated models. By restricting a, b > 0, the calibration map is monotone.This solution arises from a constrained optimization problem [37] , solved using the Pool-Adjacent-Violators Algorithm, ensuring that corrected predicted scores remain monotonic: min β1,...,βnn i=1 (y (i) -β i ) 2 , s.t. β 1 ≤ . . . ≤ β n , where y (i) corresponds to the value in {y 1 , • • • , y n } associated with the i-th largest predicted score {ŝ(x 1 ), • • • , ŝ(x n )}. Isotonic regression will lead to g(ŝ(x i )) = β ⋆i where β ⋆ i solve the optimization problem. 
score heterogeneity: 
 To accurately interpret predicted scores from a binary classifier as probabilistic estimates, since the true underlying probability s(X) is usually unobservable, calibration metrics rely solely on the predicted score range. When a binary classification model is well-calibrated, the distribution of its scores ŝ(X), as defined by Eq. 1, should align with the actual probability of the event in the vicinity of score values. Therefore, calibration metrics cannot fully capture discrepancies between the score distribution and the true probability distribution of the response variable Y when the predicted score variability does not accurately reflect the latter.Kullback-Leibler divergence Fernandes Machado et al. [9] demonstrated through simulated data that scores from ensemble methods may exhibit less variability compared to the true underlying probabilities when selecting hyperparameters based on calibration (ICI) or performance (AUC) metrics. This reduced heterogeneity makes calibration metrics less reliable for interpreting output scores as probabilities of event occurrence. Instead of evaluating discrepancies solely on predicted score values, the authors emphasize evaluating the model's probabilistic estimates using KL divergence between the overall score distribution, ŝ(X), and the available information on the "true" distribution, s(X). Additionally, the flexibility of tree-based methods like XGBoost enables the selection of model hyperparameters based on KL divergence instead of traditional performance metrics, ensuring the availability of a model whose predicted score distribution closely aligns with prior knowledge.Bayesian Framework When working with simulated data, the distribution s(X) is fully known, allowing for the direct computation of KL divergence with ŝ(X). However, with real data, the KL divergence can only be computed by relying on a prior belief about the distribution of s(X), potentially informed by expert opinion, and thus assuming a prior distribution B. In the following, as in Fernandes Machado et al. [9] , we take s(X) ∼ B = Beta(α, β) as the assumed prior distribution where each probability p i of the i-th observation is a sample from B. We observe a sequence of n independent (as the features X i are considered n i.i.d. random variables) but non-identically distributed binary random variables Y i where Y i |s(X i ) = p i ∼ Bernoulli(p i ). In this case, instead of selecting the model with hyperparameters that minimize the empirical mean of the KL divergence across individual distributions, we directly minimize the distance between the prior distribution B and the overall distribution of ŝ(X).We extend the work of Fernandes Machado et al. [9] by investigating how score heterogeneity predicted by certain XGBoost algorithms is affected after applying postcalibration techniques such as Platt scaling, Beta calibration, or isotonic regression. These methods can potentially reduce score heterogeneity; for instance, isotonic regression applies a stepwise function g. Additionally, with Platt scaling, the range of calibrated predicted scores is always narrower than the range of the initial scores when the parameter s ≥ 1 4 (see Appendix A.1). And, due to the concavity of the sigmoid function over [0, +∞], this post-calibration method tends to reduce the range of predicted scores more significantly when the initial scores are highly concentrated. 
numerical experiments 4.1 simulated data: 
 We use the simulated data from Fernandes Machado et al. [9] . We consider four data-generating processes (DGPs), all of which use a logistic link function. The first three are from Ojeda et al. [26] , the fourth adds interaction terms (see Appendix B.1). For each DGP, we generate data that include more or less noise variables: 0, 10, 50 or 100. We split the data into four samples: the train and validation samples used to train an XGBoost model and select the set of hyperparameters, the calibration sample to train a calibrator using the selected model, and lastly, a test sample to assess the performance of models. We select the model's hyperparameters (number of boosting iterations and maximum tree depth) to optimize either one of three different criteria on the validation set: maximizing AUC (AUC*), minimizing KL divergence (KL*), and, for illustrative purposes, producing a model that is poorly calibrated based on the ICI metric (High ICI). Once the hyperparameters are selected, a calibration technique is applied to the scores. This allows for a comparison of models on the test set, both before and after calibration, according to the chosen optimization criterion. We run the simulations on 100 replications for each configuration. The results for DGP 1 are shown in Fig. 1 (see Fig. C17 for full results and Table C1 for numerical values). The x-axis represents calibration, measured by the ICI, where lower ICI values indicate better calibration. The y-axis shows the KL divergence between the model's predicted score distribution and the true probabilities, with lower values indicating closer alignment between the two distributions. A model is preferable when it achieves better calibration and closer alignment between score distributions and true probabilities. Shapes represent models before calibration, while arrows show their performance after applying post-hoc calibration. Ideally, post-hoc calibration improves both metrics for uncalibrated models, resulting in arrows pointing down and to the left on the graph.When the model is selected to optimize AUC (AUC*), calibration is generally fairly good across all DGPs, regardless of the number of noise variables. Applying a post-hoc calibration technique typically reduces the ICI, further improving model calibration. However, Platt scaling (green solid arrows) often fails, as the logistic function lacks the identity mapping. The score distributions from models optimized for AUC, however, are poorly aligned with the true probability distributions. For noise-free datasets, the KL divergence is approximately 2.5 times larger compared to models optimized for KL divergence. This gap widens as the number of noise variables increases. When post-calibration techniques are applied to AUC-optimized models, KL divergence increases with Platt scaling and isotonic regression but decreases with Beta calibration. However, even with Beta calibration, the KL divergence remains higher than that of models optimized for KL divergence. For initially miscalibrated models (High ICI), post-calibration generally improves calibration, with improvements seemingly unaffected by the number of noise variables. However, the impact on KL divergence is more mixed, with no systematic improvement observed, particularly for DGPs 2 and 4.Overall, while post-calibration improves model calibration, it does not consistently align score distributions with true probabilities and may even exacerbate misalignment, highlighting trade-offs between calibration and distribution alignment. The 10 datasets from the UCI ML Repository used in Fernandes Machado et al. [9] are used here (see details in Appendix B.2). For each dataset, we apply the method outlined in Section 4.1, this time calculating the KL divergence between the predicted score distribution and the prior distribution described in Section 3. 1 The results across the 10 datasets are shown in Fig. 2 Platt scaling learns parameters, µ and s (s > 0 for a non-decreasing calibration map g) on a calibration set. The obtained calibrated probabilities are:g ŝ(x) = 1 1 + exp -1 s ŝ(x) -µ .With Platt scaling, the range of calibrated predicted scores is always narrower than the range of the initial scores when the parameter s ≥ 1 4 . Indeed, since ρ = 1 4 is the minimum value for which σ(x) = 1 1+exp -x remains ρ-Lipschitz on R, for x 1 < x 2 ∈ R, we have:σ x 2 -µ s -σ x 1 -µ s ≤ 1 4 x 2 -µ s - x 1 -µ s ≤ 1 4s |x 2 -x 1 | with s > 0.As a result, if s ≥ 1 4 , the range of (ŝ(x i )) n i=1 is larger than the range of the calibrated scores with Platt scaling (g(ŝ(x i ))) n i=1 . Let ŝm (resp. ŝM ) denote the minimum (resp. the maximum) value of (ŝ(x i )) n i=1 . If s ≥ 1 4 , we have:g(ŝ M ) -g(ŝ m ) ŝM -ŝm = σ ŝM -µ s -σ ŝm-µ s ŝM -ŝm ≤ 1.And, due to the concavity of the sigmoid function over [0, +∞], this post-calibration method tends to reduce the range of predicted scores more significantly when the initial scores are highly concentrated.To simulate data, we consider the DGPs from Fernandes Machado et al. [9] . The first three are from Ojeda et al. [26] . In the fourth, an interaction term between two predictors is added. Each scenario uses a logistic model to generate the outcome. Let Y i be a binary variable following a Bernoulli distribution:Y i ∼ B(p i ), where p i is the probability of observing Y i = 1. The probability p i is defined by:p i = P(Y = 1 | x i ) = 1 + exp(-η i ) -1 . (B.2)For the second DGP, to introduce non-linearities, p 3 is used as true probabilities instead of p.For all DGPs, η i = x ⊤ i β, where x i is a vector of covariates and β is a vector of arbitrary scalars. The covariate vector includes two continuous predictors for DGPs 1 and 2. For DGP 3, it includes five continuous and five categorical predictors. For DGP 4, it contains three continuous variables, the square of the first variable, and an interaction term between the second and third variables. Specifically,η i = β 1 x 1,i + β 2 x 2,i + β 3 x 3,i + β 4 x 2 1,i + β 5 x 2,i × x 3,i .Continuous predictors are drawn from N (0, 1). Categorical predictors consist of two variables with two categories, one with three categories, and one with five categories, all uniformly distributed. The values of coefficients β are reported in Table B1 .For each DGP, we generate data considering four scenarios with varying numbers of noise variables: 0, 10, 50, or 100 variables drawn from N (0, 1).For the fourth DGP, to achieve a similar probability distribution to DGP 1, we perform resampling using a rejection algorithm (the algorithm is detailed in [9] ). The datasets are split into four parts: a training sample, a validation sample, a calibration sample, and a test sample, each containing 10,000 observations. The empirical distribution of samples of from each DGP are shown in Fig. B1 .The main characteristics of the datasets are summarized in Table B2 .Most of the datasets used are associated with classification tasks. If not, they contain a binary variable suitable for classification or a variable that can be converted into a binary variable. The target variables for each dataset are as follows: • abalone: gender of abalones (1 for male, 0 for female); originally used to predict the size of abalones.• adult: high income (1 if income ≥ 50k per year).• bank: subscription to a term deposit (1 if yes, 0 otherwise).• default: default payment (1 if default, 0 otherwise).• drybean: type of dry bean (1 if dermason, 0 otherwise); originally a multi-class variable.• coupon: acceptance of a recommended coupon in different driving scenarios (1 if accepted, 0 otherwise).• mushroom: mushroom classification (1 if edible, 0 otherwise).• occupancy: prediction of room occupancy (1 if occupied, 0 otherwise); originally aimed at predicting the age of occupancy from physical measurements.• winequality: quality of wine (1 if quality ≥ 6, 0 otherwise); originally a scale from 0 to 10, with 0 being bad quality and 10 being good quality.• spambase: email classification (1 if spam, 0 otherwise).For each of the four DGPs (see Section B.1) and each configuration of the number of noise variables (0, 10, 50, or 100), we generate 100 sample replications. For each sample, we train an XGBoost model on 10,000 observation using the xgb.train function from the R package xgboost. The learning rate is set to 0.3. The tree depth (argument max_depth) varies according to the following values: 2, 4, 6. The number of boosting iterations (argument nrounds) ranges from 1 to 400. All variables (predictors and, if applicable, noise variables) are included in the model without transformation.For each model configuration, we select the hyperparameters based on different criteria using the validation set results. Specifically, we make three model choices:• AUC*: hyperparameters are selected to maximize the AUC.• KL*: hyperparameters are chosen to minimize the Kullback-Leibler divergence between the scores on the validation set and the true probability distribution (observable here in the context of simulated data).• High ICI: hyperparameters are selected to produce relatively poor calibration, as measured by the ICI. Specifically, we select the model with the smallest ICI among those with an ICI at least one standard deviation above the mean ICI obtained during grid search.Once the hyperparameters are selected, we apply a recalibration method on an independent calibration set: either Platt scaling, Beta calibration, or isotonic regression.The model performance is then evaluated on a test set, allowing for comparison based on: (i) the metric used to select the hyperparameters, and (ii) whether or not calibration techniques were applied to the scores. Table C1 reports the average values of metrics calculated on the test set over 100 replications for each DGP and each number of noise variables in the training data. The values are presented for models selected by optimizing, on the validation set, either AUC (AUC*) or KL divergence (KL*), as well as for a model with poor calibration (High ICI). The metrics are calculated before applying any calibration method (column "None"), after applying Platt scaling, Beta calibration and isotonic regression calibration. Fig. C17 shows the performance of the models, measured by the KL divergence between the test set score distribution (x-axis) and the true probability distribution (y-axis), before and after applying calibration methods. The values represent the average of these two metrics over 100 replications for each DGP (rows), based on the number of noise variables in the training set (columns). The point corresponds to the model whose hyperparameters (number of boosting iterations and tree depth) are selected to maximize AUC on the validation set. The square represents the model selected by minimizing the Kullback-Leibler divergence between the score distribution on the validation set and the true probability distribution. The triangle denotes a model with poor calibration on the test set. Solid green arrows illustrate the change in metrics after applying Platt scaling calibration, dotted orange arrows show changes after applying Beta calibration, and dashed purple arrows indicate changes after applying isotonic regression calibration.We train XGBoost models on the 10 datasets presented in Section B.2. Unlike Section C.1, the true probabilities underlying the binary events are not observable. Here, we assume that we have prior knowledge about the probability distribution, which can be considered as expert opinion. To simulate this expert opinion, we assume that the true probabilities follow a Beta distribution. The parameters of this distribution, specific to each dataset, are estimated via MLE using the scores from a GAMSEL model [6] .Using these prior distributions, it is possible to replicate the estimation procedure previously applied to the simulated data. Each dataset is split into two parts: 80% of the observations are used to train an XGBoost model (on a training set comprising 70% of these observations, with hyperparameters selected based on metrics calculated on the remaining 20% validation set), and the remaining 30% are used for model calibration (with 60% of this subset forming the calibration set) and for testing model performance on unseen data (the remaining 40%).Table C2 presents the metrics calculated on the test set for models selected based on their validation performance, according to AUC (AUC*), KL divergence between the score distribution and the prior distribution (KL*), or to intentionally obtain poor calibration (High ICI), both before and after applying calibration methods. For real datasets, High ICI refers to the model with hyperparameters yielding the highest AUC among models with an ICI at least one standard deviation above the mean ICI observed during grid search. This table complements Fig. 2 from the main part of the article. 0.04 0.08 0.12 0.04 0.08 0.12 0.04 0.08 0.12 0.04 0.08 0.12 0.04 0.08 0.12 0.04 0.08 0.12 0.04 0.08 0.12 0.04 0.08 0.12 0.04 0.08 0.12 0.04 0.08 0.12 0.04 0.08 0.12 0.04 0.08 0.12 0.04 0.08 0.12 0.04 0.08 0.12 0.04 0.08 0.12 0.04 0.08 0.12 0.0 0For illustration purposes, the parameters of the prior distribution B are estimated via maximum likelihood using scores from a GAMSEL model[6], where the event is regressed on the variables generating the data. 
 
 Appendix:  


Decision: This paper has borderline scores. The reviewers are mainly worried about presentation quality and not enough aspects being explained. At least part of this stems from the short four-page workshop format, and possibly missing information in the appendix that could be added in the camera-ready. I think most of this can be addressed in the camera-ready, and therefore recommend acceptance.

Reviews: review: title: title: The paper discusses post-processing calibration methods Platt scaling and isotonic regression to align the true and predicted probability distribution of data in a binary scoring classifier model. It compares the performance of an XGBoost model before and after applying these calibration techniques using simulated data with known true probabilities and real-world datasets with prior event distribution knowledge.
main_review: main_review: Summary

The paper discusses post-processing calibration methods Platt scaling and isotonic regression to align the true and predicted probability distribution of data in a binary scoring classifier model. It compares the performance of an XGBoost model before and after applying these calibration techniques using simulated data with known true probabilities and real-world datasets with prior event distribution knowledge.

Strengths

1 - Good introduction, I like how the first paragraph is structured.

2 - Isotonic Regression - I like the fact that the paper didn't use acronyms and specified the algorithm name (Pool-Adjacent-Violators Algorithm)

3 - Score Heterogeneity - I like how the paper frames and describes the problem of calibration metrics not being able to fully capture the discrepancies.

4 - Kullback-Leibler divergence - The paragraph is well explained and gives an adequate amount of information needed.

5 - Simulated Data - Good experiment design. I like that the calibration technique is applied both before and after comparing the models. 

Weakness

Major Weakness

1 - Section 2.1 - Insufficient information about BS. There's more explanation about ICI only and not BS. 

2 - Section 2.1 - Insufficient explanation about the 45-degree diagonal line. I was not able to understand the relevance of this and why specifically this.

3 - Section 2.2 - What are mu and s? There's no clear explanation of these 2 terms. Is 's' the same as mentioned in 1st paragraph of section 2? Only the last line of this paragraph is relevant and the rest should have been explained in a better way. 

4 - Section 2.2 - What is beta? Why is beta_1<=...<=beta_n? I was unable to understand the equation because of the beta term, a better explanation should have been provided. Again, only the last line in this paragraph is what I feel is relevant.

5 - Section 4 - This section does not provide any explanation of the results obtained. No explanation has been provided on how to analyze the graphs/plots provided. There should have been some information about how to interpret the graphs/plots and how to infer from them.

6 - Appendix - A lot of graphs/plots are provided for various noise levels without any explanation about them and how to infer a conclusion from them. I was not able to analyze the graphs at all as it is very difficult to infer from them without any explanation or direction. There are also a lot of tables that lack clear explanations on what to infer from them. 

7 - No validation of results - It doesn't mention any means to validate the results provided.

Minor Weakness

1 - Section 1 - Citation issue in introduction 2nd paragraph. The author's name should be mentioned.

2 - Section 3 - Kullback-Leibler divergence - citation format issue. The author's name should be mentioned.

3 - Section 3 - Bayesian Framework - Improper citation format - In this paragraph the paper by Fernandes et al has been cited twice, rephrase the sentence.

4 - Section 3 - Bayesian Framework - The formulae provided are not necessary, since this is not the method being followed. Potentially save some space.

5 - Section 3 - Calibration techniques - Improper citation format for [9]

6 - Section 4.1 - Simulated Data - Improper citation format for both [9] and [25]

7 - Section 4.1 - Simulated Data - This paragraph mentions optimizing based on 3 criteria in this sentence:- "We select the model’s hyperparameters (number of boosting iterations and maximum tree depth) to optimize three different criteria on the validation set: maximizing AUC (AUC*), minimizing KL divergence (KL*), or, for illustrative purposes, producing a model that is poorly calibrated based on the ICI metric (High ICI)." Even though this is grammatically correct it could be clear in its structure. Use 'and' instead of 'or'.

Remarks

If there is not enough space to briefly describe something, I would suggest citing it in the main paper body and adding a section in the appendix, that explains it clearly. Otherwise, an incomplete amount of information creates more confusion.

Conclusion

Overall, the paper provides valuable information about post-processing calibration methods for aligning probability distributions in binary classifiers. The introduction and problem framing is strong, and the experimental design effectively evaluates calibration techniques before and after model application. However, the paper lacks detailed explanations for key metrics and concepts, which hampers reader understandability. Additionally, the experimental results are presented without adequate analysis or interpretation, leaving the findings underexplored and difficult to validate. The paper would benefit from clearer explanations of the methodologies, better contextualization of results, and a more thorough discussion of their implications. 
I recommend rejecting the paper in its current form and suggest that the authors provide more detailed explanations, particularly regarding the results, to improve the clarity and impact of their work.
title: title: Review for Post-Calibration Techniques: Balancing Calibration and Score Distribution Alignment
main_review: main_review: 1. Summary of the Paper
The paper presents an exploration of post-processing calibration techniques, specifically Platt scaling and isotonic regression, and their impact on both model calibration and the alignment between score distributions and true event distributions. The authors employ simulated and real-world datasets to demonstrate that while these techniques improve traditional calibration metrics (like Brier Score or Integrated Calibration Index), they often lead to divergence between score distributions and actual event probabilities. The work primarily focuses on XGBoost binary classifiers.
2. Strengths
• The paper addresses a significant problem in model calibration, highlighting the potential risks of relying solely on calibration metrics without considering score distribution alignment.
• The experiments are well-designed, comparing different calibration techniques and showcasing their strengths and limitations through rigorous analysis using both simulated and real-world data.
• The paper’s discussion on the limitations of current calibration methods and the proposal to optimize model parameters based on Kullback-Leibler (KL) divergence offers a fresh perspective on the topic.
3. Weaknesses
• The explanation of score heterogeneity in the paper is insufficiently formalized and lacks a coherent structure, making it challenging to understand its exact meaning and significance in the study. Furthermore, the specific definition and implications of score heterogeneity are underdeveloped, resulting in an unclear presentation of its role.
• The paper lacks deeper theoretical grounding for the observed effects of calibration techniques on score heterogeneity. While empirical results are provided, the reasons behind the divergence between calibration and score distribution alignment need more rigorous explanation.
• The work only evaluates two common calibration methods (Platt scaling and isotonic regression). The inclusion of more modern techniques (e.g., beta calibration or spline-based methods) could have enriched the comparison.
4. Minor Comments
• The sentence “Isotonic regression, like Platt scaling, assumes …” appears to contain an error.
• Figures and tables could be enhanced to make the results more interpretable, especially for those unfamiliar with calibration and distribution alignment.