SlowGuess/NeurIPS_2023_OCR huggingface dataset:
To compute this calibrated score, we need to estimate the distributions  $\mathbb{P}\left(\hat{P}, D \mid \hat{Y} = Y\right)$  and  $\mathbb{P}\left(\hat{P}, D \mid \hat{Y} \neq Y\right)$ , and the class ratio  $\frac{\mathbb{P}\left(\hat{Y} \neq Y\right)}{\mathbb{P}\left(\hat{Y} = Y\right)}$ .
To estimate the probability density functions  $\mathbb{P}\left(\hat{P}, D \mid \hat{Y} = Y\right)$  and  $\mathbb{P}\left(\hat{P}, D \mid \hat{Y} \neq Y\right)$ , various density estimation methods can be used, such as parametric methods like Gaussian mixture models or non-parametric methods like kernel density estimation (KDE)[31]. We choose KDE because it is flexible and robust, making no assumptions about the underlying distribution (see Appendix C for specific implementation details). Specifically, we split samples into two groups based on whether they are correctly classified and then use KDE to estimate the two densities. To obtain the class ratio  $\frac{\mathbb{P}(\hat{Y} \neq Y)}{\mathbb{P}(\hat{Y} = Y)}$ , we simply use the ratio of the number of correctly classified samples and the number of misclassified samples in the validation set. The pseudocode for inference and training can be found in Appendix 2 and 1.
5.2 Discrete Confidence: Bin Mean-Shift
The Bin-Mean-Shift approach aims to first use 2-dimensional binning to estimate the joint distribution of proximity  $D$  and input confidence  $\hat{P}$  and then estimate  $\mathbb{P}\left(\hat{Y} = Y \mid \hat{P}, D\right)$ . Considering test samples  $X$  with proximity  $D = D(X)$  and uncalibrated confidence score  $\hat{P}$ , we first group samples into 2-dimensional equal-size bins based on their  $D$  and  $\hat{P}$  (other binning schemes can also be used; we choose quantile for simplicity). Next, for each bin  $B_{mh}$ , we calculate its accuracy  $\mathcal{A}(B_{mh})$  and mean confidence  $\mathcal{F}(B_{mh})$ . Then, the confidence scores of samples within the bin are adjusted as:
$$
\hat {P} _ {\text {o u r s}} = \hat {P} + \lambda \cdot \left(\mathcal {A} \left(B _ {m h}\right) - \mathcal {F} \left(B _ {m h}\right)\right), \tag {5}
$$
where the shrinkage coefficient  $\lambda \in (0,1]$  is a hyper-parameter, controlling the bias-variance trade-off. Ideally, setting  $\lambda = 1$  would ideally achieve our goal. However, in practice, we often encounter bins with a smaller number of samples, whose estimate of  $\mathcal{A}(B_{mh}) - \mathcal{F}(B_{mh})$  will have high variance and therefore inaccurate. To reduce variance in these scenarios, we can set a smaller  $\lambda$ . In practice, we choose  $\lambda = 0.5$  as a reasonable default for all our experiments, which we find offers consistent performance across various settings.
Note that our approach (i.e. applying a mean-shift in each bin) differs from the typical histogram binning method (replacing the confidence scores with its mean accuracy in each bin). Rather than completely replacing the input confidence scores  $\hat{P}$  (which are often reasonably well-calibrated), our approach better utilizes these scores by only adjusting them by the minimal mean-shift needed to correct for proximity bias in each of the 2-dimensional bins.
5.3 Theoretical Guarantee
Here we present that our method, Bin-Mean-Shift, can consistently achieve a smaller Brier Score given a sufficient amount of data in the context of binary classification. The Brier Score [5] is a strictly proper score function that measures both calibration and accuracy aspects [22], with a smaller value indicating better performance. As illustrated below, our algorithm's Brier Score is asymptotically bounded by the original Brier Score, augmented by a non-negative term.
Theorem 5.1 (Brier Score after Bin-Mean-Shift is asymptotically bounded by Brier Score before calibration). Given a joint data distribution  $\pi(X, Y)$  and a binary classifier  $f$ , for any calibration algorithm  $h$  that outputs score  $h(\hat{P})$  based on model confidence  $\hat{P}$ , we apply Bin-Mean-Shift to derive calibrated score  $\tilde{h}(\hat{P})$  as defined in Equation (5). Let  $h_c(\hat{P}) = h(\hat{P}) \times \mathbb{1}\{\hat{Y} = 1\} + (1 - h(\hat{P})) \times \mathbb{1}\{\hat{Y} = 0\}$  denote the probability assigned to class 1 by  $h(\hat{P})$ , and define  $\tilde{h}_c(\hat{P})$  similarly. Then, the Brier Score before calibration can be decomposed as follows:
$$
\underbrace {\mathbb {E} _ {\pi (X , Y)} \left[ \left(h _ {c} (\hat {P}) - Y\right) ^ {2} \right]} _ {\text {B r i e r S c o r e b e f o r e C a l i b r a t i o n}} = \underbrace {\mathbb {E} _ {\pi (X , Y)} \left[ \left(\tilde {h} _ {c} (\hat {P}) - Y\right) ^ {2} \right]} _ {\text {B r i e r S c o r e a f t e r C a l i b r a t i o n}} + \underbrace {\mathbb {E} _ {B \sim \mathbb {P} (B)} \left[ \left(\hat {A} (B) - \hat {\mathcal {F}} (B)\right) ^ {2} \right]} _ {\geq 0} + o (1),
$$
where  $\mathbb{P}(B)$  is determined by the binning mechanism used in Bin-Mean-Shift.
Table 1: Calibration performance of ImageNet pretrained ResNet50 on long-tail dataset iNaturalist 2021. * denotes significant improvement (p-value  $< {0.1}$  ). 'Base' refers to existing calibration methods, 'Ours' to our method applied to calibration. Note that 'Conf+Ours' shows the result of our method applied directly to model confidence. Calibration error is given by  $\times  {10}^{-2}$  .
Method	ECE ↓	ACE ↓	MCE ↓	PIECE ↓
base	+ours	base	+ours	base	+ours	base	+ours
Conf	4.85	0.78*	4.86	0.76*	0.55	0.18*	4.91	1.51*
TS	2.03	0.70*	2.02	0.78*	0.30	0.14*	2.34	1.43*
ETS	1.12	0.66*	1.15	0.77*	0.18	0.13*	1.79	1.38*
PTS	4.86	0.71*	4.90	0.86*	2.96	0.12	7.04	1.44*
PTSK	2.97	0.65*	3.01	0.82*	0.56	0.11*	4.66	1.41*
MIR	1.05	0.98	1.09	1.06	0.18	0.19	1.63	1.64
Remark. Note that when the calibration algorithm  $h$  is an identity mapping, it demonstrates that Bin-Mean-Shift achieves better model calibration performance than the original model confidence, given a sufficient amount of data. The detailed proof is relegated to Appendix A.
In this section, we aim to answer the following questions:
- Performance across different datasets and model architectures: How does PROCAL perform on datasets with balanced distribution, long-tail distribution, and distribution shift, as well as on different model architectures?
- Inference efficiency: How efficient is our PROCAL? (see Appendix E.1)
- Hyperparameter sensitivity: How sensitive is PROCAL to different hyperparameters, e.g., neighbor size  $K$ ? (See Appendix F.2)
- Ablation study: What is the difference between Density-Ratio and Bin-Mean-shift on calibration? How should the choice between these techniques be determined? (See Appendix F.1)
Evaluation Metrics. Following [10], we adopt 3 commonly used metrics to evaluate the confidence calibration: Expected Calibration Error (ECE), Adaptive Calibration Error (ACE [29]), Maximum Calibration Error (MCE) and our proposed PIECE to evaluate the bias mitigation performance. More detailed introduction of these metrics can be found in Appendix C.
Datasets. We evaluate the effectiveness of our approach across large-scale datasets of three types of data characteristics (balanced, long-tail and distribution-shifted) in image and text domains: (1) Dataset with balanced class distribution (i.e. each class has an equal size of samples) on vision dataset ImageNet [7] and two text datasets including Yahoo Answers Topics [49] and MultiNLI-match [44]; (2) Datasets with long-tail class distribution on two image datasets, including iNaturalist 2021 [3] and ImageNet-LT [27]; (3) Dataset with distribution-shift on three datasets, including ImageNet-C [15], MultiNLI-Mismatch [44] and ImageNet-Sketch [42].
Comparison methods. We compare our method to existing calibration algorithms: base confidence score (Conf) [16], scaling-based methods such as Temperature Scaling (TS) [10], Ensemble Temperature Scaling (ETS) [48], Parameterized Temperature Scaling (PTS) [39], Parameterized Temperature Scaling with K Nearest Neighbors (PTSK), and binning based methods such as Histogram Binning (HB), Isotonic Regression (IR) and Multi-Isotonic Regression (MIR) [48]. Throughout the experiment section, we apply Density-Ratio Calibration to Conf, TS, ETS, PTS, and PTSK and apply Bin-Mean-Shift to binning-based methods IR, HB, and MIR. HB and IR are removed from the long-tail setting due to its instability when the class sample size is very small.
More details on baseline algorithms, datasets, pretrained models, hyperparameters, and implementation details can be found in Appendix C.
Figure 3: Calibration errors on ImageNet across 504 timm models. Each point represents the calibration result of applying a calibration method to the model confidence. Marker colors indicate different calibration algorithms used. Among all calibration algorithms, our method consistently appears at the bottom of the plot. See Appendix E Figure 11 for high resolution figures.
Table 2: We use RoBERTa models [26] fine-tuned on Yahoo and MultiNLI Match, respectively, as their models. 'Base' refers to existing calibration methods and 'Ours' refers to our method applied to existing calibration methods. Calibration error is given by  $\times  {10}^{-2}$  .
Method	ECE ↓	ACE ↓	MCE ↓	PIECE ↓
base	+ours	base	+ours	base	+ours	base	+ours
Conf	4.56	0.51	4.56	0.54	0.84	0.09	4.73	1.32
TS	0.50	0.42	0.46	0.46	0.09	0.07	2.22	1.34
ETS	0.62	0.43	0.59	0.45	0.09	0.07	2.22	1.37
PTS	0.55	0.42	0.52	0.42	0.10	0.09	2.03	1.41
PTSK	0.61	0.47	0.51	0.51	0.13	0.09	2.15	1.38
HB	3.28	1.99	4.88	2.09	1.68	0.67	5.33	2.92
IR	0.64	0.84	0.64	0.77	0.13	0.19	2.08	1.75
MIR	0.63	0.72	0.54	0.61	0.11	0.18	2.08	1.65
(a) Yahoo Answer Topics
Method	ECE ↓	ACE ↓	MCE ↓	PIECE ↓
base	+ours	base	+ours	base	+ours	base	+ours
Conf	2.47	1.45	2.62	1.46	0.85	0.37	3.54	2.78
TS	1.70	1.22	1.76	1.35	0.41	0.40	3.03	2.78
ETS	1.58	1.24	1.56	1.26	0.66	0.39	3.04	2.77
PTS	7.53	2.42	7.50	2.46	4.21	0.93	7.78	3.56
PTSK	10.14	4.26	10.14	4.40	7.40	2.83	10.33	5.15
HB	1.11	1.05	1.17	1.50	0.36	0.25	3.76	2.51
IR	0.88	1.43	1.07	1.23	0.36	0.34	2.46	2.61
MIR	0.71	1.03	1.07	1.36	0.26	0.40	2.58	2.35
(b) MultiNLI Mismatch
6.2 Effectiveness
Datasets with the balanced class distribution. The results on ImageNet of 504 models from timm [43] are depicted in Figure 3, where our method (red color markers) consistently appears at the bottom, achieving the lowest calibration error across all four evaluation metrics in general. This indicates that our method consistently outperforms other approaches in eliminating proximity bias and improving confidence calibration. We also select four popular models from these 504 models, specifically BeiT [2], MLP Mixer [38], ResNet50 [12] and ViT [8]. A summary of their results is presented in Table 4 of Appendix E. Additionally, Table 2a and Table 5 present the calibration results for the text classification task on Yahoo Answers Topics and the text understanding task on MultiNLI, where our method consistently improves the calibration of existing methods and model confidence. SeeAppendix E for more details.
Datasets with the long-tail class distribution. Table 1 shows the results on the long-tail image dataset iNaturalist 2021. Our method ('ours') consistently improves upon existing algorithms ('base') regarding reducing confidence calibration errors (ECE, ACE, and MCE) and mitigating proximity bias (PIECE). Note that even when used independently ('Conf+ours') without combining with existing algorithms, our method achieves the best performance across all metrics. This result suggests that our algorithm can make the model more calibrated in the long-tail setting by effectively mitigating the bias towards low proximity samples (i.e. tail classes), highlighting its practicality in real-world scenarios where data is often imbalanced and long-tailed. ImageNet-LT results in Table 6 of Appendix E.3 show similar improvement.
Datasets with distribution shift. Table 2b shows our method's calibration performance when trained on an in-distribution validation set (MultiNLI Match) and applied to a cross-domain test set (MultiNLI Mismatch). The results suggest that our method can improve upon most existing methods on ECE, ACE and MCE, and gain consistent improvement on PIECE, indicating its effectiveness in mitigating proximity bias. Moreover, empirical results on ImageNet-C (Figure 12) and ImageNet-Sketch (Table 7) also demonstrate consistent improvements of our method over baselines. Besides, compared to Bin-Mean-Shift, Density-Ratio exhibits more stable performance on enhancing the existing baselines. More analysis on their comparison can be found in Appendix F.1.
In this paper, we focus on the problem of proximity bias in model calibration, a phenomenon wherein deep models tend to be more overconfident on data of low proximity (i.e. lying in the sparse region of data distribution) and thus suffer from miscalibration. We study this phenomenon on 504 public models across a wide variety of model architectures and sizes on ImageNet and find that the bias persists even after applying the existing calibration methods, which drives us to propose PROCAL for tackling proximity bias. To further evaluate the miscalibration due to proximity bias, we propose a proximity-informed expected calibration error (PIECE) with theoretical analysis. Extensive empirical studies on balanced, long-tail, and distribution-shifted datasets under four metrics support our findings and showcase the effectiveness of our method.
Potential Impact, Limitations and Future Work We uncover the proximity bias phenomenon and show its prevalence through large-scale analysis, highlighting its negative impact on the safe deployment of deep models, e.g. unfair decisions on minority populations and false diagnoses for underrepresented patients. We also provide PROCAL as a starting point to mitigate the proximity bias, which we believe has the potential to inspire more subsequent works, serve as a useful guidance in the literature and ultimately lead to improved and fairer decision-making in real-world applications, especially for underrepresented populations and safety-critical scenarios. However, our study also has several limitations. First, our PROCAL maintains a held-out validation set during inference for computing proximity. While we have shown that the cost can be marginal for large models (see Inference Efficiency in Appendix E.1), it may be challenging if applied to small devices where memory is limited. Future research can investigate the underlying mechanisms of proximity bias and explore various options to replace the existing approach of local density estimation. Additionally, we only focus on the closed-set multi-class classification problem; future work can generalize this to multi-label, open-set or generative settings.
The author extends gratitude to Yao Shu and Zhongxiang Dai for the insightful discussions during the review response period, and to Yifei Li for providing constructive feedback on the manuscript draft. This research is supported by the National Research Foundation Singapore under its AI Singapore Programme (Award Number: [AISG2-TC-2021-002]).
[1] Alaaeldin Ali, Hugo Touvron, Mathilde Caron, Piotr Bojanowski, Matthijs Douze, Armand Joulin, Ivan Laptev, Natalia Neverova, Gabriel Synnaeve, Jakob Verbeek, et al. Xcit: Cross-covariance image transformers. Advances in neural information processing systems, 34:20014-20027, 2021.
[2] Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image transformers, 2022.
[3] Sara Beery, Arushi Agarwal, Elijah Cole, and Vighnesh Birodkar. The iwildcam 2021 competition dataset. arXiv preprint arXiv:2105.03494, 2021.
[4] Carlo Bonferroni. Teoria statistica delle classi e calcolo delle probabilita. *Pubblicazioni del R Istituto Superiore di Scienze Economiche e Commerciali di Firenze*, 8:3-62, 1936.
[5] Glenn W Brier et al. Verification of forecasts expressed in terms of probability. Monthly weather review, 78(1):1-3, 1950.
[6] Rich Caruana, Yin Lou, Johannes Gehrke, Paul Koch, Marc Sturm, and Noemie Elhadad. Intelligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission. In Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining, pages 1721-1730, 2015.
$^{2}$ Our codes are available at: https://github.com/MiaoXiong2320/ProximityBias-Calibration.git
[41] Grant Van Horn, Elijah Cole, Sara Beery, Kimberly Wilber, Serge Belongie, and Oisin Mac Aodha. Benchmarking representation learning for natural world image collections. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12884-12893, 2021.
[42] Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. In Advances in Neural Information Processing Systems, pages 10506-10518, 2019.
[43] Ross Wightman. Pytorch image models. https://github.com/rwrightman/pytorch-image-models, 2019.
[44] Adina Williams, Nikita Nangia, and Samuel R Bowman. A broad-coverage challenge corpus for sentence understanding through inference. arXiv preprint arXiv:1704.05426, 2017.
[45] Miao Xiong, Shen Li, Wenjie Feng, Ailin Deng, Jihai Zhang, and Bryan Hooi. Birds of a feather trust together: Knowing when to trust a classifier via adaptive neighborhood aggregation. arXiv preprint arXiv:2211.16466, 2022.
[46] Bianca Zadrozny and Charles Elkan. Obtaining calibrated probability estimates from decision trees and naive bayesian classifiers. In *Icml*, volume 1, pages 609–616, 2001.
[47] Bianca Zadrozny and Charles Elkan. Transforming classifier scores into accurate multiclass probability estimates. In Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, pages 694-699, 2002.
[48] Jize Zhang, Bhavya Kailkhura, and T Yong-Jin Han. Mix-n-match: Ensemble and compositional methods for uncertainty calibration in deep learning. In International conference on machine learning, pages 11117-11128. PMLR, 2020.
[49] Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. Advances in neural information processing systems, 28, 2015.
A Proof of Theorem 5.1
Notation. Let  $\pi(X, Y)$  denote the true underlying distribution of the sample  $X$  and its label  $Y$ . The empirical dataset used for Bin-Mean-Shift is denoted by  $D_{n} = \{(\mathbf{x}_{1}, y_{1}), (\mathbf{x}_{2}, y_{2}), \ldots, (\mathbf{x}_{n}, y_{n})\}$ , where  $n$  is the number of samples in the dataset, and each datapoint  $(\mathbf{x}_{i}, y_{i})$  is independently and identically sampled from  $\pi(X, Y)$ .
A classifier  $f$  is defined as  $f(X) = (\hat{Y},\hat{P})$ , where  $\hat{Y}$  represents the predicted label and  $\hat{P}$  represents the model's confidence, i.e., the estimate of the probability of correctness [10].
We use  $h(\hat{P})$  to denote the output confidence score for class prediction  $\hat{Y}$  calibrated by any calibration algorithm  $h$ . For example, in the simplest case,  $h$  can be the identity mapping, in which case  $h(\hat{P})$  represents the original model confidence.
To simplify notation, we use  $B = B(X)$  to denote the bucket to which  $X$  belongs. Given a bucket  $B$ , we compute the empirical accuracy  $\hat{\mathcal{A}}_n(B)$  and mean confidence score  $\hat{\mathcal{F}}_n(B)$  for samples in the bucket, based on the empirical dataset  $D_n$ . Given the classifier  $f$  and calibrator  $h$ , they are computed as follows:
$$
\hat {\mathcal {A}} _ {n} (B) = \frac {1}{| B |} \sum_ {(\mathbf {x}, y) \in D _ {n}} \mathbb {1} \{(y = \hat {y}) \wedge (B (\mathbf {x}) = B) \} \tag {6}
$$
$$
\hat {\mathcal {F}} _ {n} (B) = \frac {1}{| B |} \sum_ {(\mathbf {x}, y) \in D _ {n} \wedge (\mathbf {x} \in B)} h (\hat {P}) \tag {7}
$$
Additionally, we use  $\mathcal{A}(B)$  and  $\mathcal{F}(B)$  to denote the expected confidence and actual accuracy of samples from the underlying data distribution  $\pi(X,Y)$  and belonging to the bucket  $B$ :
$$
\mathcal {A} (B) = \mathbb {E} _ {\pi (X, Y)} \left[ \mathbb {1} \{Y = \hat {Y} \} \mid B (X) = B \right] \tag {8}
$$
$$
\mathcal {F} (B) = \mathbb {E} _ {X} \left[ h (\hat {P}) \mid B (X) = B \right] \tag {9}
$$
With these definitions, our proposed Bin-Mean-Shift algorithm can be expressed as:
$$
\tilde {h} (\hat {P}) = h (\hat {P}) + \hat {\mathcal {A}} _ {n} (B) - \hat {\mathcal {F}} _ {n} (B), \tag {10}
$$
where  $B$  is the bucket that the input sample  $X$  belongs to, and  $\tilde{h}(\hat{P})$  is the score calibrated using our Bin-Mean-Shift algorithm.
First we revisit the definition of Brier Score. The Brier Score [5] is a strictly proper score function that measures both calibration and accuracy aspects [22], with a smaller value indicating better performance. The Brier Score is defined as mean square loss as follows for binary classification:
$$
\text {B r i e r S c o r e} = \mathbb {E} _ {\pi (X, Y)} \left[ \left(h _ {c} (\hat {P}) - Y\right) ^ {2} \right], \tag {11}
$$
where  $h_c(\hat{P})$  is the probability assigned to class 1 by the calibration algorithm  $h$ :
$$
h _ {c} (\hat {P}) = \left\{ \begin{array}{l l} h (\hat {P}) & \text {w h e n} \hat {Y} = 1 \\ 1 - h (\hat {P}) & \text {w h e n} \hat {Y} = 0 \end{array} \right. \tag {12}
$$
In short, this can be represented as  $h_c(\hat{P}) = h(\hat{P}) \times \mathbb{1}\{\hat{Y} = 1\} + (1 - h(\hat{P})) \times \mathbb{1}\{\hat{Y} = 0\}$ .
Theorem A.1 (Brier Score after Bin-Mean-Shift is asymptotically bounded by Brier Score before calibration). Given a joint data distribution  $\pi(X, Y)$  and a binary classifier  $f$ , for any calibration algorithm  $h$  that outputs a score  $h(\hat{P})$  based on model confidence  $\hat{P}$ , we apply Bin-Mean-Shift to derive the calibrated score  $\tilde{h}(\hat{P})$  as defined in Equation (10). Let  $h_c(\hat{P}) = h(\hat{P}) \times \mathbb{1}\{\hat{Y} = 1\} + (1 - h(\hat{P})) \times \mathbb{1}\{\hat{Y} = 0\}$  denote the probability assigned to class 1 by  $h(\hat{P})$ , and define  $\tilde{h}_c(\hat{P})$  similarly. Then, the Brier Score before calibration can be decomposed as follows:
$$
\underbrace {\mathbb {E} _ {\pi (X , Y)} \left[ \left(h _ {c} (\hat {P}) - Y\right) ^ {2} \right]} _ {\text {B r i e r S c o r e b e f o r e C a l i b r a t i o n}} = \underbrace {\mathbb {E} _ {\pi (X , Y)} \left[ \left(\tilde {h} _ {c} (\hat {P}) - Y\right) ^ {2} \right]} _ {\text {B r i e r S c o r e a f t e r C a l i b r a t i o n}} + \underbrace {\mathbb {E} _ {B \sim \mathbb {P} (B)} \left[ \left(\hat {\mathcal {A}} _ {n} (B) - \hat {\mathcal {F}} _ {n} (B)\right) ^ {2} \right]} _ {\geq 0} + o (1),
$$
Proof. First we prove the following equality which re-expresses the Brier score in an equivalent form:
$$
\mathbb {E} _ {\pi (X, Y)} \left[ \left(h _ {c} (\hat {P}) - Y\right) ^ {2} \right] = \mathbb {E} _ {\pi (X, Y)} \left[ \left(h (\hat {P}) - \tilde {Y}\right) ^ {2} \right], \tag {13}
$$
where  $\hat{Y} = \mathbb{1}\{Y = \hat{Y}\}$  is a binary variable representing whether the model's prediction is correct:
$$
\tilde {Y} = \mathbb {1} \{Y = \hat {Y} \} = \left\{ \begin{array}{l l} \mathbb {1} \{Y = 0 \} = 1 - Y, & \text {w h e n} \quad \hat {Y} = 0 \\ \mathbb {1} \{Y = 1 \} = Y, & \text {w h e n} \quad \hat {Y} = 1 \end{array} \right. \tag {14}
$$
Then we consider  $h_c(\hat{P}) - Y$ . By Eq. (12):
$$
\begin{array}{l} \left(h _ {c} (\hat {P}) - Y\right) ^ {2} = \left\{ \begin{array}{l l} (1 - h (\hat {P}) - Y) ^ {2}, & \text {w h e n} \quad \hat {Y} = 0 \\ \left(h (\hat {P}) - Y\right) ^ {2}, & \text {w h e n} \quad \hat {Y} = 1 \end{array} \right. (15) \\ = \left\{ \begin{array}{l l} (\tilde {Y} - h (\hat {P})) ^ {2}, & \text {w h e n} \quad \hat {Y} = 0 \\ (h (\hat {P}) - Y) ^ {2}, & \text {w h e n} \quad \hat {Y} = 1 \end{array} \right. (16) \\ \end{array}
$$
So we have  $(h_c(\hat{P}) - Y)^2 = (h(\hat{P}) - \tilde{Y})^2$  which completes the proof of the equality in Eq. (13). Using this, we rewrite the Brier score as follows:
$$
\text {B r i e r S c o r e} = \mathbb {E} _ {\pi (X, Y)} \left[ \left(h (\hat {P}) - \tilde {Y}\right) ^ {2} \right], \tag {17}
$$
Second, we decompose the Brier score: