SlowGuess/NeurIPS_2023_OCR huggingface dataset:
For an algorithm that decides if  $p \neq q$ , the Type I error [14] occurs when it incorrectly decides  $p \neq q$  and the corresponding probability of Type I error is named the significant level. In contrast, the Type II error [14] occurs when it incorrectly decides  $p = q$ , and the corresponding probability of not making this error is called the test power [34]. In the experiments, the samples of two given datasets  $\widehat{p}$  and  $\widehat{q}$  are drawn from different distributions, i.e., the ground truth is  $p \neq q$ . Both the significant level and the test power are sensitive to the two probability distributions. Accordingly, to measure the performance of a learning algorithm, we calculate its test power when a certain significant level  $\alpha \in [0, 1]$  can be guaranteed. A higher test power indicates better performance when  $p \neq q$ .
The significant level can be guaranteed with a permutation test [17]. For the given datasets  $\widehat{p}$  and  $\widehat{q}$ , the permutation test uniformly randomly swaps samples between  $\widehat{p}$  and  $\widehat{q}$  to generate  $Z$  dataset pairs  $\{(\widehat{p}^z,\widehat{q}^z)\}_{z = 1}^Z$ . For  $\widehat{p}$  and  $\widehat{q}$ , R-Div outputs the empirical estimator  $\widehat{\mathcal{D}}_{\mathrm{R}}(\widehat{p}\| \widehat{q})$ . Note that if  $p = q$ , swapping samples between  $\widehat{p}$  and  $\widehat{q}$  cannot change the distribution, which indicates that the permutation test can guarantee a certain significant level. If the samples of  $\widehat{p}$  and  $\widehat{q}$  are drawn from different distributions, the test power is 1 if the algorithm can output  $p\neq q$ . Otherwise, the test power is 0. Specifically, to guarantee the  $\alpha$  significant level, the algorithm output  $p\neq q$  if  $\widehat{\mathcal{D}}_{\mathrm{R}}(\widehat{p}\| \widehat{q})$  is in the top  $\alpha$ -quantile among  $\mathcal{G} = \{\widehat{\mathcal{D}}_{\mathrm{R}}(\widehat{p}^{z}\| \widehat{q}^{z})\}_{z = 1}^{Z}$ . We perform the test for  $K$  times using different randomly-selected dataset pairs  $(\widehat{p},\widehat{q})$  and average the output values as the test power of the algorithm. If not specified, we set  $\alpha = 0.05$ ,  $Z = 100$  and  $K = 100$ . The procedure of calculating the average test power of R-Div is summarized in Algorithm 2 (see Appendix B.2).
5.2 Unsupervised Learning Tasks
Benchmark Datasets: Following the experimental setups as [41] and [62], we adopt four benchmark datasets, namely Blob, HDGM, HIGGS, and MNIST. We consider learning unsupervised models on these datasets, and the considered hypothesis spaces are for generative models. Specifically, MNIST adopts the variational autoencoders (VAE) [33], while the other low dimensional data adopt kernel density estimation (KDE) [27]. For fair comparisons, we follow the same setup as [62] for all methods. Because all the compared methods have hyper-parameters, we evenly split each dataset into a training set and a validation set to tune hyper-parameters and compute the final test power, respectively. The average test powers for a certain significant level on HDGM [41], Blob [41], MNIST [38] and HIGGS [1] are presented in Figure 3, Figure 4, Table 8 (see Appendix D.1) and Table 2, respectively. Overall, our proposed R-Div achieves the best test power on all datasets. On HDGM, R-Div obtains the best performance for different dimensions and achieves the perfect test power with fewer samples. Specifically, the test power of R-Div decreases gracefully as the dimension increases on HDGM. On Blob with different significant levels, R-Div achieves the same test power as the second-best method with a lower variance. On MNIST, both H-Div and R-Div achieve the perfect test power on all the sample sizes. On HIGGS, R-Div achieves the perfect test power even on the dataset with the smallest scale.
The discrepancy as measured by R-Div is distinct for two different distributions, even with a small sample size. This advantage stems from mitigating overfitting of H-Div by learning the minimum hypothesis and evaluating its empirical risks across varying datasets. R-Div learns a minimum hypothesis based on the mixed data and yields consistent empirical risks for both datasets when samples are sourced from the same distribution. Conversely, if samples from the two datasets come from different distributions, the empirical risks differ. This is because the training dataset for the minimum hypothesis does not precisely match either of the individual datasets.
To better demonstrate the advantages of R-Div, we measure the amount of overfitting by calculating empirical risks on  $p$  and  $q$ . The results presented in Table 1 indicate that R-Div achieves a notably clearer discrepancy, attributable to the significant differences in empirical risks on  $p$  and  $q$ . This is because R-Div mitigates overfitting by optimizing a minimum hypothesis on a mixed dataset
Table 1: The empirical risks on  $p$  and  $q$  for L-Div, H-Div, and R-Div on MNIST when  $N = {200}$  .
Methods	Hypothesis	εp(hp)	εq(hq)	|εp(hp) - εq(hq)|
L-Div	hp = hq ∈ H	713.0658	713.1922	0.1264
H-Div	hp ∈ arg minh∈H εp(h)
hq ∈ arg minh∈H εq(h)	111.7207	118.9020	7.1813
R-Div	hp = hq ∈ arg minh∈H εp±q/2(h)	144.0869	99.6166	44.4703
Table 2: The average test power  $\pm$  standard error at the significant level  $\alpha  = {0.05}$  on HIGGS.  $N$  represents the number of samples in each given dataset, and boldface values represent the relatively better discrepancy estimation.
N	1000	2000	3000	5000	8000	10000	Avg.
ME	0.120±0.007	0.165±0.019	0.197±0.012	0.410±0.041	0.691±0.691	0.786±0.041	0.395
SCF	0.095±0.022	0.130±0.026	0.142±0.025	0.261±0.044	0.467±0.038	0.603±0.066	0.283
C2STS-S	0.082±0.015	0.183±0.032	0.257±0.049	0.592±0.037	0.892±0.029	0.974±0.070	0.497
C2ST-L	0.097±0.014	0.232±0.017	0.399±0.058	0.447±0.045	0.878±0.020	0.985±0.050	0.506
MMD-O	0.132±0.050	0.291±0.012	0.376±0.022	0.659±0.018	0.923±0.013	1.000±0.000	0.564
MMD-D	0.113±0.013	0.304±0.035	0.403±0.050	0.699±0.047	0.952±0.024	1.000±0.000	0.579
H-Div	0.240±0.020	0.380±0.040	0.685±0.015	0.930±0.010	1.000±0.000	1.000±0.000	0.847
R-Div	1.000±0.000	1.000±0.000	1.000±0.000	1.000±0.000	1.000±0.000	1.000±0.000	1.000
and assessing its empirical risks across diverse datasets. Consequently, R-Div can yield a more pronounced discrepancy for two markedly different distributions.
5.3 Supervised Learning Tasks
Multi-domain Datasets: We adopt the PACS dataset [40] consisting of samples from four different domains, including photo (P), art painting (A), cartoon (C), and sketch (S). Following [40] and [57], we use AlexNet [37] as the backbone. Accordingly, samples from different domains can be treated as samples from different distributions. We consider the combinations of two domains and estimate their discrepancies. The results in Figure 5 (see Appendix D.2) represent that the test power of R-Div significantly improves as the number of samples increases. In addition, R-Div obtains the best performance on different data scales. The results in Table 3 show that R-Div achieves the superior test power across all the combinations of two domains. This is because the compared H-Div method suffers from the overfitting issue because it searches for a minimum hypothesis and evaluates its empirical risk on the same dataset. This causes empirical risks on different datasets to be minimal, even if they are significantly different. R-Div improves the performance of estimating the discrepancy by addressing this overfitting issue. Specifically, given two datasets, R-Div searches a minimum hypothesis on their mixed data and evaluates its empirical risks on each individual dataset. R-Div obtains stable empirical risks because the datasets used to search and evaluate the minimum hypothesis are not the same and partially overlap.
Spurious Correlations Datasets: We evaluate the probability distribution discrepancy between MNIST and Colored-MNIST (with spurious correlation) [3]. The adopted model for the two datasets is fully-connected and has two hidden layers of 128 ReLU units, and the parameters are learned using Adam [32]. The results in Table 4 reveal that the proposed R-Div achieves the best performance, which indicates that R-Div can explore the spurious correlation features by learning a minimum hypothesis on the mixed datasets.
Corrupted Datasets: We assess the probability distribution discrepancy between a given dataset and its corrupted variants. Adhering to the construction methods outlined by Sun et al.[55] and the framework of H-Div[62], we conduct experiments using H-Div and R-Div. We examine partial corruption types across four corruption levels on both CIFAR10 [36] and ImageNet [15], employing ResNet18 [22] as the backbone. Assessing discrepancies in datasets with subtle corruptions is particularly challenging due to the minor variations in their corresponding probability distributions. Therefore, we employ large sample sizes,  $N = 5000$  for CIFAR10 and  $N = 1000$  for ImageNet,
Table 3: The average test power  $\pm$  standard error at the significant level  $\alpha = 0.05$  on different domain combinations of PACS. P, A, C and S represent photo, art painting, cartoon, and sketch domains, respectively. The distribution discrepancies for six combinations of two domains are estimated. For example,  $A + C$  represents the combination of art painting and cartoon domains. Boldface values represent the relatively better discrepancy estimation.
Datasets	A+C	A+P	A+S	C+P	C+S	P+S	Ave.
H-Div	0.846±0.118	0.668±0.064	0.964±0.066	0.448±0.135	0.482±0.147	0.876±0.090	0.714
R-Div	1.000±0.000	1.000±0.000	1.000±0.000	0.527±0.097	0.589±0.160	0.980±0.040	0.849
Table 4: The average test power on MNIST and Colored-MNIST at the significant level  $\alpha  = {0.05}$  .
ME	SCF	C2STS-S	C2ST-L	MMD-O	MMD-D	H-Div	R-Div
0.383	0.204	0.189	0.217	0.311	0.631	0.951	1.000
and observe that methods yield a test power of 0 when the corruption level is mild. Consistently across different corruption types, R-Div demonstrates superior performance at higher corruption levels and outperforms H-Div across all types and levels of corruption. Moreover, our results on ImageNet substantiate that the proposed R-Div effectively estimates distribution discrepancies in large-resolution datasets.
Advanced Network Architectures: For supervised learning tasks, R-Div selects the network architecture for the specific model. To verify the effect of R-div, we perform experiments on a wide range of architectures. We compare R-div with H-div because both of them are model-oriented and the other comparison methods have specific network architectures. Specifically, we consider the classification task on CIFAR10 and Corrupted-CIFAR10 and adopt four advanced network architectures, including ResNet18 [22], SENet [24], ConvNeXt [42], ViT [16]. The results are summarized in Table 6. For large-scale datasets  $(N = 10000)$ , both H-Div and R-Div achieve perfect test power with different network architectures because more samples are applied to estimate the probability distribution discrepancy precisely. However, R-Div has the better test power for more challenging scenarios where the datasets are small  $(N = 1000, 5000)$ . This may be because R-Div addresses the overfitting issue of H-Div by optimizing the minimum hypothesis and evaluating its empirical risk on different datasets. Furthermore, for different network architectures, CNN-based networks achieve better performance than the transformer-based models (ViT). According to the analysis of Park and Kim [49] and Li et al. [39], this may be because multi-head attentions (MHA) are low-pass filters with a shape bias, while convolutions are high-pass filters with a texture bias. As a result, ViT more likely focuses on the invariant shapes between CIFAR10 and the corrupted-CIFAR10 to make the two datasets less discriminative.
5.4 Case Study: Learning with Noisy Labels
R-Div is a useful tool to quantify the discrepancy between two probability distributions. We demonstrate how it can learn robust networks on data with noisy labels by distinguishing samples with noisy labels from clean samples in a batch. Furthermore, we explain how the discrepancy between clean and noisy samples affects the classification generalization ability [58].
5.4.1 Setup
We adopt the benchmark dataset CIFAR10 [36]. As CIFAR10 contains clean samples, following the setup [51], we corrupt its training samples manually, i.e., flipping partial clean labels to noisy labels. Specifically, we consider symmetry flipping [56] and pair flipping [21]. For corrupting a label, symmetry flipping transfers it to another random label, and pair flipping transfers it to a specific label. Following the setup [29], we adopt 0.2 noise rate, i.e., the labels of  $20\%$  samples will be flipped. The classification accuracy is evaluated on clean test samples.
We apply R-Div to train a robust DNN on a corrupted dataset. The training strategy separates clean and noisy samples by maximizing the discrepancy estimated by R-Div. The ground truth labels
Table 5: The average test power on a dataset and its corrupted variants at the significant level  $\alpha = 0.05$ .
Dataset	Corruption	0.5%	1%	2%	5%
H-Div/ R-Div
CIFAR10 (N=1000)	Gauss	0.000 / 0.000	0.195 / 0.219	0.368 / 0.411	0.710 / 0.748
Snow	0.000 / 0.000	0.096 / 0.189	0.251 / 0.277	0.366 / 0.416
Speckle	0.000 / 0.000	0.275 / 0.340	0.458 / 0.513	0.685 / 0.771
Impulse	0.000 / 0.000	0.127 / 0.185	0.277 / 0.326	0.629 / 0.652
ImageNet (N=5000)	Gauss	0.000 / 0.000	0.680 / 0.722	1.000 / 1.000	1.000 / 1.000
Snow	0.000 / 0.000	0.774 / 0.816	0.785 / 0.836	1.000 / 1.000
Speckle	0.000 / 0.000	0.586 / 0.624	0.762 / 0.804	1.000 / 1.000
Impulse	0.000 / 0.000	0.339 / 0.485	0.545 / 0.635	1.000 / 1.000
Table 6: The average test power on advanced network architectures.
N	ResNet18	SENet	ConvNeXt	ViT
H-Div / R-Div
1000	0.116 / 0.252	0.201 / 0.296	0.186 / 0.322	0.119 / 0.211
5000	0.622 / 0.748	0.478 / 0.790	0.778 / 0.844	0.492 / 0.651
10000	1.000 / 1.000	1.000 / 1.000	1.000 / 1.000	1.000 / 1.000
for noisy samples are unreliable. Therefore, these labels can be treated as complementary labels representing the classes that the samples do not belong to. Accordingly, the training strategy optimizes the losses of ground truth and complementary labels for clean and noisy samples, respectively.
We assume that clean and noisy samples satisfy different distributions, i.e., clean and noisy distributions. Specifically, we apply R-Div to detect clean and noisy samples from a batch. Accordingly, we separate the batch samples  $\mathcal{B}$  into predicted clean samples  $\mathcal{B}_c$  and predicted noisy samples  $\mathcal{B}_n$  by maximizing R-Div. The batch and the whole dataset contain IID samples from an unknown mixture distribution about clean and noisy distributions. Both the minimum hypotheses on the batch and the whole dataset can be treated to approximate the optimal hypothesis of the mixture distribution. To prevent learning a minimum hypothesis for each given batch, we pretrain a network on the whole corrupted dataset rather than a batch and apply this pretrained network (minimum hypothesis) to separate clean and noisy samples in a batch.
Recall that R-Div calculates the empirical risk difference between two datasets under the same minimum hypothesis of their mixed dataset. Hence, to maximize the discrepancy of R-Div, we calculate the empirical risk of each sample in the batch by the pretrained network, sort the batch samples according to the empirical risks, and treat the samples with low and high empirical risks as predicted clean and noisy samples, respectively. Specifically, we separate noisy and clean samples according to the estimated noise rate  $\gamma \in [0,1]$ . For clean samples with trusted ground truth labels, we optimize a network by minimizing the traditional cross-entropy loss. In contrast, the labels for noisy samples are untrusted. However, their noisy labels indicate the classes they do not belong to. Accordingly, we can treat their untrusted ground truth labels as complementary labels and optimize a network by minimizing the negative traditional cross-entropy loss on noisy samples. Thus, we retrain a robust network by optimizing the objective function  $\min_{h\in \mathcal{H}}\widehat{\epsilon}_{\mathcal{B}_c}(h) - \widehat{\epsilon}_{\mathcal{B}_n}(h)$  for each batch, where the loss function  $l$  is the traditional cross-entropy loss. In this work, we adopt ResNet18 [22] as the backbone. In the pretraining and retraining processes, the learning rate [60] starts at 0.1 and is divided by 10 after 100 and 150 epochs. The batch size is 128, and the number of epochs is 200. The hyper-parameter  $\gamma$  varies from 0.1 to 0.7 with a 0.1 step size.
(b) Detection
Figure 2: Results on CIFAR10 with symmetry flipping. All values are averaged over five trials. Left: Classification accuracy of pretrained and retrained networks. Middle: Precision and recall rates of detecting clean and noisy samples. Right: Discrepancy between predicted clean and noisy samples.
(c) Discrepancy
5.4.2 Experimental Results
We evaluate the classification accuracy on clean test samples, the precision and recall rates of detecting clean and noisy samples, and the discrepancy between predicted clean and noisy samples. The results are reported in Figure 2 and Figure 6 (see Appendix D.3). Recall that the noise rate is 0.2. Figure 2(a) and Figure 6(a) show that the network retrained with R-Div improves the classification performance over the pretrained network when the estimated noise rate  $\gamma \in [0.2, 0.5]$ . The retrained network achieves the best performance when the estimated noise rate is nearly twice as much as the real noise rate. To the best of our knowledge, how to best estimate the noise rate remains an open problem and severely limits the practical application of the existing algorithms. However, our method merely requires an approximate estimation which can be twice as much as the real noise rate.
Further, Figure 2(b) and Figure 6(b) show that there is a point of intersection between the precision and recall curves at  $\gamma = 0.5$ . The aforementioned results indicate that  $\gamma \leq 0.5$  can lead to improved performance. Accordingly, we know that the detection precision is more essential than its recall rate. Specifically, a larger estimated noise rate ( $\gamma \geq 0.5$ ) indicates that more clean samples are incorrectly recognized as noisy samples. In contrast, an extremely lower estimated noise rate which is lower than the real one indicates that more noisy samples are incorrectly recognized as clean samples. Both situations can mislead the network in learning to classify clean samples.
In addition, Figure 2(c) and Figure 6(c) show that a larger  $\gamma$  leads to a smaller discrepancy. Because the majority of training samples are clean, most of the clean samples have small test losses. A larger  $\gamma$  causes more clean samples to be incorrectly recognized as noisy samples, which decreases the test loss of the set of predicted noisy samples. Thus, the empirical estimator will decrease. Based on the results reported in Figure 2(a) and Figure 6(a), the largest discrepancy cannot lead to improved classification performance because most of the noisy samples are incorrectly recognized as clean samples. In addition, we observe that the retrained network obtains improved classification performance when the estimated discrepancy approximates the mean value  $1.5 \times 10^{-3}$ , which can be used as a standard to select an estimated noise rate  $\gamma$  for retraining a robust network.
We tackle the crucial question of whether two probability distributions are identically distributed for a specific learning model using a novel and effective metric, R-divergence. Unlike existing divergence measures, R-divergence identifies a minimum hypothesis from a dataset that combines the two given datasets, and then uses this to compute the empirical risk difference between the individual datasets. Theoretical analysis confirms that R-divergence, serving as an empirical estimator, uniformly converges and is bounded by the discrepancies between the two distributions. Empirical evaluations reveal that R-divergence outperforms existing methods across both supervised and unsupervised learning paradigms. Its efficacy in handling learning with noisy labels further underscores its utility in training robust networks. If the samples from the two datasets are deemed identically distributed, a subsequent question to be investigated is whether these samples are also dependent.
This work was supported in part by the Australian Research Council Discovery under Grant D-P190101079 and in part by the Future Fellowship under Grant FT190100734.
[1] Claire Adam-Boundarios, Glen Cowan, Cécile Germain, Isabelle Guyon, Balázs Kégl, and David Rousseau. How machine learning won the higgs boson challenge. In *ESANN*, pages 1–6, 2016.
[2] Rohit Agrawal and Thibaut Horel. Optimal bounds between f-divergences and integral probability metrics. J. Mach. Learn. Res., 22:1-128, 2021.
[3] Martín Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. CoRR, pages 1-31, 2019.
[4] Peter L. Bartlett and Shahar Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results. J. Mach. Learn. Res., 3:463-482, 2002.
[5] Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. A theory of learning from different domains. Mach. Learn., 79(1-2):151-175, 2010.
[6] Abhijit Bendale and Terrance E. Boult. Towards open set deep networks. In CVPR, pages 1563-1572, 2016.
[7] Christopher M Bishop and Nasser M Nasrabadi. Pattern recognition and machine learning, volume 4. Springer, 2006.
[8] Longbing Cao. Beyond i.i.d.: Non-iid thinking, informatics, and learning. IEEE Intell. Syst., 37(4):5-17, 2022.
[9] Longbing Cao. Non-iid learning. IEEE Intell. Syst., 37(4):3-4, 2022.
[10] Xiuyuan Cheng and Alexander Cloninger. Classification logit two-sample testing by neural networks for differentiating near manifold densities. IEEE Trans. Inf. Theory, 68(10):6631-6662, 2022.
[11] Kacper Chwialkowski, Aaditya Ramdas, Dino Sejdinovic, and Arthur Gretton. Fast two-sample testing with analytic representations of probability measures. In NeurIPS, pages 1981-1989, 2015.
[12] Imre Csiszár. Eine informationstheoretische Ungleichung und ihre anwendung auf beweis der ergodizitaet von markoffschen ketten. Magyer Tud. Akad. Mat. Kutato Int. Koezl., 8:85-108, 1964.
[13] Imre Csiszár. I-divergence geometry of probability distributions and minimization problems. The annals of probability, pages 146-158, 1975.
[14] Frederik Michel Dekking, Cornelis Kraaikamp, Hendrik Paul Lopuhaa, and Ludolf Erwin Meester. A Modern Introduction to Probability and Statistics: Understanding why and how, volume 488. Springer, 2005.
[15] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, pages 248–255, 2009.
[16] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, pages 1-22, 2021.
[17] Michael D Ernst. Permutation methods: a basis for exact inference. Statistical Science, 19(4):676-685, 2004.
[18] Noah Golowich, Alexander Rakhlin, and Ohad Shamir. Size-independent sample complexity. In  $COLT$ , volume 75, pages 297-299, 2018.
[19] Mehmet Gönen and Ethem Alpaydin. Multiple kernel learning algorithms. J. Mach. Learn. Res., 12:2211-2268, 2011.
Section A.1 presents the lemmas used to prove the main results. Section A.2 presents the main results and their detailed proofs.
Lemma A.1 ([53], Theorem 26.5). With probability of at least  $1 - \delta$ , for all  $h \in \mathcal{H}$
$$
\left| \epsilon_ {u} (h) - \widehat {\epsilon} _ {\widehat {u}} (h) \right| \leq 2 \mathcal {R} _ {H} ^ {l} (\widehat {u}) + 4 c \sqrt {\frac {\ln (4 / \delta)}{N}}.
$$
Lemma A.2 ([53], Theorem 26.5). With probability of at least  $1 - \delta$ , for all  $h \in \mathcal{H}$