SlowGuess/NeurIPS_2023_OCR huggingface dataset:
To search for such a robust statistic that is suitable for our purpose, we turn to Average Thresholded Confidence (ATC) [13] for inspiration. ATC tremendously improves the performance of AC precisely by offering such protection against the overconfident outlier predictions (shown in supp material) by returning the percentile above the threshold rather than the mean. In a similar vein, we safeguard COT against outlier costs by introducing COT with Thresholding, COTT.
Specifically, to compute the threshold  $t \in [0,1]$ , we first sample a validation set  $\{(x_{\mathrm{val}}^{(i)},y_{\mathrm{val}}^{(i)})\}_{i = 1}^{n} \sim P_S(x,y)$ . Let  $\hat{P}_{\mathrm{val}}$  denote the empirical validation distribution. We compute the optimal transport plan  $\mathbf{P}_{\mathrm{val}}^* = \arg \min_{\mathbf{P} \in \Pi(\vec{f}_{\#}\hat{P}_{\mathrm{val}}(\vec{c}),\hat{P}_S(\vec{y}))} \langle \mathbf{P},\mathbf{C} \rangle$ , where  $\mathbf{C}$  is the cost matrix determined by the L-infinity distance. Note that  $\mathbf{P}_{\mathrm{val}}^* \in \{0,1\}^{n\times n}$  is a permutation matrix due to the equal number of confidence vectors and one-hot labels (Proposition 2.1 in [32]). Then the threshold  $t$  is set such that the validation of error the classifier equals the fraction of samples with transport cost higher than  $t$ , i.e.
$$
\epsilon_ {\mathrm {v a l}} = \frac {1}{n} | \{\mathbf {C} _ {i j} \geq t | \mathbf {P} _ {\mathrm {v a l} i j} ^ {*} = 1 \} |
$$
With the threshold  $t$  learned from the validation set, we can compute the COTT target error estimate. Let  $\hat{P}_S(\vec{y})$  denote the empirical source label distribution and  $\vec{f}_{\#}\hat{P}_T(\vec{c})$  denote the empirical distribution of confidence vectors of samples from the target distribution  $P_T(x)$ . We compute the optimal transport plan  $\mathbf{P}^{*} = \arg \min_{\mathbf{P}\in \Pi (\vec{f}_{\#}\hat{P}_T(\vec{c}),\hat{P}_S(\vec{y}))}\langle \mathbf{P},\mathbf{C}\rangle$ . Our COTT estimate is then defined as
$$
\hat {\epsilon} _ {\mathrm {C O T T}} = \frac {1}{n} | \left\{\mathbf {C} _ {i j} \geq t \mid \mathbf {P} ^ {*} _ {i j} = 1 \right\} |
$$
Note on Implementation: While solving the linear program of optimal transport for COT and COTT is rather straightforward with existing solvers such as POT [12], its time complexity  $\mathcal{O}(n^3)$  prohibits large-scale inputs. We bypass this problem by breaking the test dataset into small batches with a maximum size of 10,000. Concretely, given  $n$  test samples, we sample with replacement for  $[n / 10,000]$  batches and compute an error estimate for each batch. We get the final error estimation of COT and COTT by taking an average of batch estimates.
In this section, we empirically compare COT and COTT with existing methods on various benchmark datasets. For all experiments, we trained the model on in-distribution data and froze the model after convergence. To predict the model's performance on the target domain, we only used unlabeled data from the target domain. When we have a test set size greater than 10,000, we show the results of utilizing the batched version of COT and COTT detailed in Section 3.2. For context, solving the OT problem of size 10,000 only takes around 10s, thus adding only negligible computation overhead.
4.1 Datasets and Nature of Shift
In our comprehensive evaluation, we consider more than 10 benchmark datasets across multiple modalities, including vision and language, with a variety of distribution shifts. Each type of shift is introduced in the following paragraphs:
Synthetic Shift: First, we consider distribution shifts caused by common visual corruptions, such as brightness, defocusing, and blurriness, which are common in real-world settings. We used the corrupted versions of CIFAR10, CIFAR100, and ImageNet proposed in [19], which includes 19 types of common visual corruptions across 5 levels of severity.
Novel Subpopulation Shift: Next, we consider novel subpopulation shifts, where the subpopulations in the train and test sets differ. For example, models might have only observed golden retrievers for the dog class but not huskies. In our experiments, we used the BREEDS benchmark [36], which leveraged the ImageNet[8] class hierarchy to create 4 datasets, Living-17, Nonliving-26, Entity13, and Entity-30. With this benchmark, we aim to understand if error estimation methods could capture challenges in an increasingly diverse world.
Natural Shift: Finally, we consider non-simulated shifts in which the distribution shifts are induced through differences in the data collection process, such as ImageNet-V2 and CIFAR10-V2 proposed in [34]. We also include ImageNet-Sketch [40], which consists of sketched images of the original ImageNet classes. Additionally, we consider distribution shifts faced in the wild, such as ones curated in the WILDS benchmark [24]. We consider four WILDS datasets, two for language tasks (Amazon-WILDS, CivilComments-WILDS) and two for vision tasks (Camelyon17, RxRx1). These datasets reflect various discrepancies between data sources happening in the real world. For example, hospitals might use different stain colors for pathological images. Different groups might follow distinct standards assigning star ratings for their reviews.
4.2 Architectures and Evaluations
For vision tasks, We trained ResNet18 (CIFAR10, CIFAR100) and ResNet50 [18] (ImageNet, Living17, Nonliving26, Entity13, Entity30, Camelyon17-WILDS, RxRx1-WILDS); for language tasks (Amazon-WILDS, CivilComments-WILDS), we fine-tuned DistilBERT-base-uncased [35]. We followed training setups from previous works [13] and provided the full details in the supplemental material. After training, we calibrated models using Temperature Scaling (TS) [17] on the in-distribution validation data, effectively adjusting the output probabilities of the neural network to match the actual correctness likelihood. This approach has previously demonstrated that TS consistently improves error estimation performance for all methods [13]. To evaluate different methods, we utilized the mean absolute difference between their predicted errors and the true errors, which are obtained using ground truth labels. We refer to this metric as Mean Absolute Error (MAE).
We consider an array of baselines to compare against our methods: COT and COTT.
Table 1: Mean Absolute Error (MAE) between the estimated error and ground truth error to compare different methods. The "shift" column denotes the nature of distribution shifts for each dataset. For vision datasets, we reported results for ResNet18 and ResNet50; for language datasets, we reported results for DistilBERT-base-uncased. The results are averaged over 3 random seeds. We highlight the best-performing method. We defer the full table with std to the supplemental material.
Dataset	Shift	AC	DoC	Baselines	Ours
IM	GDE	ATC-MC	ATC-NE	COT	COTT
CIFAR10	Natural	5.97	5.38	5.87	5.9	3.38	3.15	5.41	3.33
Synthetic	9.1	8.53	9.26	8.84	4.2	3.37	2.17	1.7
CIFAR100	Synthetic	10.83	8.76	12.07	11.36	6.8	6.63	2.09	2.59
ImageNet	Natural	8.5	7.43	8.62	5.62	3.57	2.6	3.88	2.41
Synthetic	10.34	9.28	12.87	6.54	1.59	3.41	3.24	1.42
Entity13	Same	19.63	19.2	17.5	15.37	8.09	7.23	8.47	2.61
Novel	29.61	29.18	27.22	24.48	14.54	9.49	15.9	5.46
Entity30	Same	16.97	16.21	13.56	13.98	8.19	9.08	5.9	2.46
Novel	27.57	26.81	23.96	23.4	13.46	8.57	15.11	5.94
Living17	Same	14.84	14.67	11.22	9.94	4.88	5.43	6.25	2.94
Novel	29.61	29.18	27.22	24.48	14.54	9.49	15.9	5.53
Nonliving26	Same	19.25	18.43	16.6	12.77	11.18	9.69	7.06	3.34
Novel	31.37	30.54	28.79	23.37	19.93	16.56	17.8	10.46
Camelyon17-WILDS	Natural	9.44	9.44	10.24	5.19	7.73	7.73	7.27	5.71
RxRx1-WILDS	Natural	5.21	8.44	8.09	7.48	6.53	6.86	3.25	5.82
Amazon-WILDS	Natural	2.62	2.35	2.34	17.04	1.63	1.54	2.43	2.01
CivilCom.-WILDS	Natural	1.54	0.96	0.86	8.7	2.3	2.3	1.23	4.68
Average Confidence (AC) estimates target error by taking the average of one minus the maximum softmax confidence of target data.  $\hat{\epsilon}_{\mathrm{AC}} = \mathbb{E}_{x\sim \hat{P}_T}[1 - \max_{j\in \mathcal{Y}}\vec{f}_j(x)]$
Difference of Confidence (DoC a.k.a. DOC-Feat) [16] estimates target error through the difference between the confidence of source data and the confidence of target data.  $\hat{\epsilon}_{\mathrm{DoC}} = \mathbb{E}_{x\sim \hat{P}_S}[\mathbb{1}[\arg \max_{j\in \mathcal{Y}}\vec{f}_j(x)\neq y]] + \mathbb{E}_{x\sim \hat{P}_T}[1 - \max_{j\in \mathcal{Y}}\vec{f}_j(x)] - \mathbb{E}_{x\sim \hat{P}_S}[1 - \max_{j\in \mathcal{Y}}\vec{f}_j(x)].$
Importance Re-weighting (IM) estimates target error as a re-weighted source error. The weights are calculated as the ratio between the number of data points in each bin in target data and source data. This is equivalent to [6] using one slice based on the underlying classifier confidence.
Generalized Disagreement Equality (GDE) [22] estimates target error as the disagreement ratio of predictions on the target data using two independently trained models  $\vec{f}(x)$  and  $\vec{f}'(x)$ .  $\hat{\epsilon}_{\mathrm{GDE}} = \mathbb{E}_{x \sim \hat{P}_T}[\mathbb{1}[\arg \max_{j \in \mathcal{Y}} \vec{f}_j(x) \neq \arg \max_{j \in \mathcal{Y}} \vec{f}_j'(x)]]$ .
Average Thresholded Confidence (ATC) [13] first identifies a threshold  $t$  such that the fraction of source data points that have scores below the threshold matches the source error on in-distribution validation data. Target error is estimated as the expected number of target data points that fall below the identified threshold.  $\hat{\epsilon}_{\mathrm{ATC}}(s) = \mathbb{E}_{x\sim \hat{P}_T}[\mathbb{1}[s(\vec{f} (x)) < t]]$ , where  $s$  is the score function mapping the softmax vector to a scalar. Two different score functions are used, Maximum Confidence (ATC-MC) and Negative Entropy (ATC-NE).
In addition, we also compare our method to ProjNorm [41]. ProjNorm cannot provide a direct estimate, instead, the authors demonstrated their metric has the strongest linear correlation to true target error compared to existing baselines. In this case, we followed their setup and performed a correlation analysis to draw a direct comparison. We included results in the supplemental material and showed that our method consistently outperforms ProjNorm.
Figure 4: Qualitative results for AC, ATC, COT, and COTT, comparing error estimates vs. ground truth target error. Accurate estimates should be close to  $y = x$  (dashed black line). Notably, COT and COTT remain accurate even when the shifts are large. By contrast, AC and ATC often severely underestimate the error, which is particularly evident in the Nonliving-26 dataset.
4.4 Results
In Table 1, we report the MAE results grouped by datasets and nature of shifts. Across all benchmarks, we observe that COT always obtains lower estimation error than AC, supporting our theoretical analysis that COT additionally leverages pseudo-label shift to fight miscalibration. On synthetic shift benchmarks (CIFAR10-Synthetic, CIFAR100-Synthetic, ImageNet-Synthetic BREEDS-same), COT is  $2 - 4 \times$  better than AC, drastically reducing the estimation error. On novel subpopulation shift (BREEDS-novel), COT cuts about half of the AC error. On natural shift, COT also improves upon AC. On ImageNet natural shift datasets, COT reduces the error from 8.5 to 3.88 compared to AC. COTT, presents the best overall results, surpassing the best current method (ATC) by a notable margin. On synthetic shift benchmarks, COTT is  $2 - 3 \times$  better than ATC-NE, the stronger version of ATC that uses a negative entropy score function. On novel subpopulation shift benchmarks, COTT is at least 4 absolute percent better than ATC-NE. On natural shift benchmarks, however, we observed mixed results. On ImageNet-Natural, COTT is better than ATC-NE while on CIFAR10-Natural, COTT is marginally worse. On WILDS benchmarks, no single method dominantly outperforms others. Note that the WILDS benchmark datasets contain label shift, meaning  $P_{S}(y) \neq P_{T}(y)$ . This leads COTT to overestimate the error on the CivilComments-WILDS. Nonetheless, COTT has the smallest worst-case error of 5.82 compared to ATC-NE's 7.73, demonstrating its robustness even on distribution shifts faced in the wild.
In Figure 4, we use scatterplots to visualize estimations given by different methods, notably AC, ATC, COT, and COTT, where we plot the predicted error against the true error. Ideally, these scattered points should demonstrate a strong linear correlation and closely follow the  $y = x$  line. While this is true for COT and COTT, AC and ATC often underestimate, sometimes by a large margin. In Nonliving-26, we can see data points representing shifts whose ATC predicted errors are around 0.1 while true errors are close to 1. These observations corroborate the necessity of leveraging pseudo-label shifts to guard against such catastrophic failures of existing confidence-based methods. We include the scatterplots for all datasets in the supplemental material, where we show that COT and COTT successfully prevent severe underestimation seen in other methods.
In this work, we proposed COT and COTT, simple yet effective methods leveraging the optimal transport framework to estimate a classifier's performance on out-of-distribution data with only unlabeled samples. Our method is motivated by the observation that existing methods failed to take into account the fact that classifiers often demonstrate pseudo-label shifts when performance drops on an unseen target domain. We formalized this observation by connecting AC and COT under the Wasserstein metric and showing that COT provably provides more conservative estimates by assuming a target marginal distribution. Combining COT with a thresholding strategy that has shown extensive empirical success, we further introduced COTT. On all eleven benchmark tasks, COT and COTT manage to avoid severe underestimation of the model's error seen in current methods. In particular, COTT achieves significantly lower prediction errors than existing methods for most models and datasets (up to 3x better), establishing new state-of-the-art results. We believe our work makes
an important step forward in making performance estimation methods more reliable and theoretically grounded and thus more applicable in real-world applications.
Limitations: Despite the strong empirical performance, our method does suffer from several limitations. First, our method is currently only applicable to multi-class classification problems under the assumption that the marginal label distribution must remain constant between the train and test distributions, though as we will show in the supplemental material, our method is still quite robust when the difference between source and target marginal is small. In addition, while we have derived a lower bound for error estimates, we have not currently formulated an upper bound; however, we have not observed overestimating the error to be a common problem empirically. Lastly, the relationship between miscalibration and pseudo-label shift is currently only empirical in nature, not theoretical.
[1] David Alvarez-Melis and Nicolo Fusi. Geometric dataset distances via optimal transport. Advances in Neural Information Processing Systems, 33:21428-21439, 2020.
[2] Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International conference on machine learning, pages 214-223. PMLR, 2017.
[3] Christina Baek, Yiding Jiang, Aditi Raghunathan, and Zico Kolter. Agreement-on-the-line: Predicting the performance of neural networks under distribution shift, 2023.
[4] Yogesh Balaji, Rama Chellappa, and Soheil Feizi. Normalized wasserstein for mixture distributions with applications in adversarial learning and domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6500-6508, 2019.
[5] Krishnakumar Balasubramanian, Pinar Donmez, and Guy Lebanon. Unsupervised supervised learning ii: Margin-based classification without labels. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, pages 137-145. JMLR Workshop and Conference Proceedings, 2011.
[6] Mayee Chen, Karan Goel, Nimit S Sohoni, Fait Poms, Kayvon Fatahalian, and Christopher Ré. Mandoline: Model evaluation under distribution shift. In International Conference on Machine Learning, pages 1617-1629. PMLR, 2021.
[7] Shai Ben David, Tyler Lu, Teresa Luu, and David Pál. Impossibility theorems for domain adaptation. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pages 129–136. JMLR Workshop and Conference Proceedings, 2010.
[8] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. IEEE, 2009.
[9] Weijian Deng and Liang Zheng. Are labels always necessary for classifier accuracy evaluation? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15069-15078, 2021.
[10] Weijian Deng, Stephen Gould, and Liang Zheng. What does rotation prediction tell us about classifier accuracy under varying testing environments? In International Conference on Machine Learning, pages 2579-2589. PMLR, 2021.
[11] Pinar Donmez, Guy Lebanon, and Krishnakumar Balasubramanian. Unsupervised supervised learning i: Estimating classification and regression errors without labels. Journal of Machine Learning Research, 11(4), 2010.
[12] Rémi Flamary, Nicolas Courty, Alexandre Gramfort, Mokhtar Z. Alaya, Aurélie Boisbunon, Stanislas Chambon, Laetitia Chapel, Adrien Corenflós, Kilian Fatras, Nemo Fournier, Léo Gautheron, Nathalie T.H. Gayraud, Hicham Janati, Alain Rakotomamonjy, Ievgen Redko, Antoine Rolet, Antony Schutz, Vivien Seguy, Danica J. Sutherland, Romain Tavenard, Alexander Tong, and Titouan Vayer. Pot: Python optimal transport. Journal of Machine Learning Research, 22(78):1-8, 2021. URL http://jmlr.org/papers/v22/20-451.html.
[30] Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. Advances in neural information processing systems, 32, 2019.
[31] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
[32] Gabriel Peyré, Marco Cuturi, et al. Computational optimal transport: With applications to data science. Foundations and Trends® in Machine Learning, 11(5-6):355–607, 2019.
[33] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do cifar-10 classifiers generalize to CIFar-10? arXiv preprint arXiv:1806.00451, 2018.
[34] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize toImagenet? In International conference on machine learning, pages 5389-5400. PMLR, 2019.
[35] Victor Sanh, Lysandre Debut, Julien Chaumont, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108, 2019.
[36] Shibani Santurkar, Dimitris Tsipras, and Aleksander Madry. Breeds: Benchmarks for subpopulation shift. arXiv preprint arXiv:2008.04859, 2020.
[37] Hidetoshi Shimodaira. Improving predictive inference under covariate shift by weighting the log-likelihood function. Journal of statistical planning and inference, 90(2):227-244, 2000.
[38] Simon Stepputtis, Joseph Campbell, Mariano Phielipp, Stefan Lee, Chitta Baral, and Henri Ben Amor. Language-conditioned imitation learning for robot manipulation tasks. Advances in Neural Information Processing Systems, 33:13139-13150, 2020.
[39] Cédric Villani et al. Optimal transport: old and new, volume 338. Springer, 2009.
[40] Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. In Advances in Neural Information Processing Systems, pages 10506-10518, 2019.
[41] Yaodong Yu, Zitong Yang, Alexander Wei, Yi Ma, and Jacob Steinhardt. Predicting out-of-distribution error with the projection norm. arXiv preprint arXiv:2202.05834, 2022.
For readers' convenience, we review the statements of the propositions and corollaries and provide the full proofs below.
Proof. We first show the following equality. Let  $j^{*} = \arg \max_{j}\vec{f}_{j}(x)$
$$
1 - \vec {f} _ {j ^ {*}} (x) = \| \vec {y} - \vec {f} (x) \| _ {\infty} \tag {1}
$$
where  $\vec{y}_j = \mathbb{1}[j = j^*]$ . Let  $\vec{f}_{-j^*}(x)$  denote the vector  $\vec{f}(x)$  with  $j^*$ -th element removed. Since for a confidence vector  $\| \vec{f}(x) \|_1 = 1$ ,
$$
1 - \vec {f} _ {j ^ {*}} (x) = \| \vec {f} _ {- j ^ {*}} (x) \| _ {1} \geq \| \vec {f} _ {- j ^ {*}} (x) \| _ {\infty}
$$
Therefore, we have obtained the desired Equality 1:
$$
\| \vec {y} - \vec {f} _ {j} (x) \| _ {\infty} = \max  \{\| \vec {f} _ {- j ^ {*}} (x) \| _ {\infty}, 1 - \vec {f} _ {j ^ {*}} (x) \} = 1 - \vec {f} _ {j ^ {*}} (x)
$$
Next, we consider the optimal transport plan between  $\vec{f}_{\#}P(\vec{c})$  and  $P_{\mathrm{pseudo}}(\vec{y})$ . Namely, we show all confidence vectors  $\vec{f}(x^{(i)})$  are coupled with their one-hot pseudo-labels  $\vec{y}^{(i)}$ . This can be observed by the fact that the one-hot pseudo-label is the one-hot label that achieves the lowest L-infinity cost, i.e.
$$
\| \vec {f} (x ^ {(i)}) - \vec {y} ^ {(i)} \| _ {\infty} \leq \| \vec {f} (x ^ {(i)}) - \vec {y} ^ {\prime} \| _ {\infty}, \forall \vec {y} ^ {\prime} \in \{0, 1 \} ^ {k} \cap \Delta^ {k - 1}
$$
Suppose there exist confidence vectors that are not coupled with their one-hot pseudo-labels, then all individual costs are suboptimal and the total cost is suboptimal as well, contradicting the assumption that the transport plan is optimal. Therefore,
$$
W _ {\infty} (\vec {f} _ {\#} P (\vec {c}), P _ {\text {p s e u d o}} (\vec {y})) = \frac {1}{n} \sum_ {i = 1} ^ {n} \| \vec {f} (x ^ {(i)}) - \vec {y} ^ {(i)} \| _ {\infty} \tag {2}
$$
Combining Equality 1 and Equality 2, we obtain the desired relationship between AC error estimate and  $W_{\infty}$  distance
$$
\hat {\epsilon} _ {\mathrm {A C}} = \frac {1}{n} \sum_ {i = 1} ^ {n} \left(1 - \max  _ {j} \vec {f} _ {j} (x ^ {(i)})\right) = \frac {1}{n} \sum_ {i = 1} ^ {n} \| \vec {f} (x ^ {(i)}) - \vec {y} ^ {(i)} \| _ {\infty} = W _ {\infty} (\vec {f} _ {\#} P (\vec {c}), P _ {\text {p s e u d o}} (\vec {y}))
$$
Proof. We first show the following equality, which establishes the relationship between AC accuracy estimate with  $W_{\infty}$  distance:
$$
1 - \hat {\epsilon} _ {\mathrm {A C}} = W _ {\infty} (\vec {f} _ {\#} P (\vec {c}), \delta_ {0}) \tag {3}
$$
Since  $W_{\infty}(\vec{f}_{\#}P(\vec{c}), \delta_0)$  transports  $\vec{f}_{\#}P(\vec{c})$  to  $\delta_0$ , the optimal transport plan couples every element in  $\vec{f}_{\#}P(\vec{c})$  to 0. For each  $x^{(i)}$ , its confidence vector  $\vec{f}_{\#}P(\vec{c})$  has a transport cost  $\| \vec{f}(x^{(i)}) - 0 \|_{\infty}$ . Hence,
$$
1 - \hat {\epsilon} _ {\mathrm {A C}} = \frac {1}{n} \sum_ {i = 1} ^ {n} \max  _ {j} \vec {f} _ {j} (x ^ {(i)}) = \frac {1}{n} \sum_ {i = 1} ^ {n} \| \vec {f} (x ^ {(i)}) - 0 \| _ {\infty} = W _ {\infty} (\vec {f} _ {\#} P (\vec {c}), \delta_ {0})
$$
With this, our inequality is simply the Triangle Inequality in  $(\mathcal{P}(\mathbb{R}^k),W_{\infty})$
$$
W _ {\infty} (\vec {f} _ {\#} P (\vec {c}), \delta_ {0}) + W _ {\infty} (\vec {f} _ {\#} P (\vec {c}), P ^ {\prime} (\vec {y})) \geq W _ {\infty} (P ^ {\prime} (\vec {y}), \delta_ {0}) = 1
$$
Combined with Equation 3, we obtain the desired inequality
$$
W _ {\infty} (\vec {f} _ {\#} P (\vec {c}), P _ {\mathrm {p s e u d o}} (\vec {y})) = 1 - W _ {\infty} (\vec {f} _ {\#} P (\vec {c}), \delta_ {0}) \leq W _ {\infty} (\vec {f} _ {\#} P (\vec {c}), P ^ {\prime} (\vec {y}))
$$