SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zhilin Zhao Longbing Cao
Data Science Lab, School of Computing & DataX Research Centre
Macquarie University, Sydney, NSW 2109, Australia
zhaozh17@hotmail.com, longbing.cao@mq.edu.au
Real-life data are often non-IID due to complex distributions and interactions, and the sensitivity to the distribution of samples can differ among learning models. Accordingly, a key question for any supervised or unsupervised model is whether the probability distributions of two given datasets can be considered identical. To address this question, we introduce R-divergence, designed to assess model-oriented distribution discrepancies. The core insight is that two distributions are likely identical if their optimal hypothesis yields the same expected risk for each distribution. To estimate the distribution discrepancy between two datasets, R-divergence learns a minimum hypothesis on the mixed data and then gauges the empirical risk difference between them. We evaluate the test power across various unsupervised and supervised tasks and find that R-divergence achieves state-of-the-art performance. To demonstrate the practicality of R-divergence, we employ R-divergence to train robust neural networks on samples with noisy labels.
Most machine learning methods rely on the basic independent and identically distributed (IID) assumption [7], implying that variables are drawn independently from the same distribution. However, real-life data and machine learning models often deviate from this IID assumption due to various complexities, such as mixture distributions and interactions [9]. For a given supervised or unsupervised task [45], a learning model derives an optimal hypothesis from a hypothesis space by optimizing the corresponding expected risk [53]. However, this optimal hypothesis may not be valid if the distributions of training and test samples differ [54], resulting in distributional vulnerability [63]. In fact, an increasing body of research addresses complex non-IID scenarios [8], including open-world problems [6], out-of-distribution detection [23, 63], domain adaptation [28], and learning with noisy labels [46]. These scenarios raise a crucial yet challenging question: how to assess the discrepancy between two probability distributions for a specific learning model?
Empirically, the discrepancy between two probability distributions can be assessed by the divergence between their respective datasets. However, the actual underlying distributions are often unknown, with only a limited number of samples available for observation. Whether the samples from two datasets come from the same distribution is contingent upon the specific learning model being used. This is because different learning models possess unique hypothesis spaces, loss functions, target functions, and optimization processes, which result in varying sensitivities to distribution discrepancies. For instance, in binary classification [26], samples from two datasets might be treated as positive and negative, respectively, indicating that they stem from different distributions. Conversely, a model designed to focus on invariant features is more likely to treat all samples as if they are drawn from the same distribution. For example, in one-class classification [52], all samples are considered positive, even though they may come from different components of a mixture distribution.
In cases where the samples from each dataset originate from multiple distributions, they can be viewed as drawn from a complex mixture distribution. We can then estimate the discrepancy between these two complex mixture distributions. This suggests that, in practice, it is both meaningful and necessary to evaluate the distribution discrepancy for a specific learning model, addressing the issue of model-oriented distribution discrepancy evaluation.
Estimating the discrepancy between two probability distributions has been a foundational and challenging issue. Various metrics for this task have been proposed, including F-divergence [12], integral probability metrics (IPM)[2], and H-divergence[62]. F-divergence metrics, like Kullback-Leibler (KL) divergence [13] and Jensen Shannon divergence [47], assume that two distributions are identical if they possess the same likelihood at every point. Importantly, F-divergence is not tied to a specific learning model, as it directly measures discrepancy by computing the statistical distance between the two distributions. IPM metrics, including the Wasserstein distance [48], maximum mean discrepancy (MMD)[25], and L-divergence[5, 31], assess discrepancy based on function outputs. They postulate that any function should yield the same expectation under both distributions if they are indeed identical, and vice versa. In this vein, L-divergence explores the hypothesis space for a given model and regards the largest gap in expected risks between the two distributions as their discrepancy. This necessitates evaluating all hypotheses within the hypothesis space, a task which can be computationally demanding. Moreover, hypotheses unrelated to the two distributions could result in inaccurate or biased evaluations. On the other hand, H-divergence [62] contends that two distributions are distinct if the optimal decision loss is greater when computed on their mixed distribution compared to each individual one. It calculates this optimal decision loss in terms of training loss, meaning it trains a minimal hypothesis and evaluates its empirical risk on the identical dataset. A network with limited training data is susceptible to overfitting [50], which can lead to underestimated discrepancies when using H-divergence, even if the two distributions are significantly different.
To effectively gauge the model-oriented discrepancy between two probability distributions, we introduce R-divergence. This novel metric is built on the notion that two distributions are identical if the optimal hypothesis for their mixture distribution yields the same expected risk on each individual distribution. In line with this concept, R-divergence evaluates the discrepancy by employing an empirical estimator tailored to the given datasets and learning model. Initially, a minimum hypothesis is derived from the mixed data of the two datasets to approximate the optimal hypothesis. Subsequently, empirical risks are computed by applying this minimum hypothesis to each of the two individual datasets. Finally, the difference in these empirical risks serves as the empirical estimator for the discrepancy between the two probability distributions. The framework of R-divergence is illustrated in Figure 1.
2 Model-oriented Two-sample Test
We measure the discrepancy between two probability distributions  $p$  and  $q$  for a specific supervised or unsupervised learning method. Let  $\mathcal{X}$  and  $\mathcal{Y}$  be the spaces of inputs and labels, respectively. Assume we observe two sets of  $N$  IID samples from the two distributions, i.e.,  $\widehat{p} = \{x_i\}_{i=1}^N \sim p$ , and  $\widehat{q} = \{x_i'\}_{i=1}^N \sim q$ , respectively. We assume the mixture distribution  $u = (p + q)/2$  and its corresponding mixed data  $\widehat{u} = \widehat{p} \cup \widehat{q}$ .
For a learning model  $\mathcal{T}$ , we assume  $\mathcal{H}$  is its hypothesis space and  $l$  is its  $L$ -Lipschitz bounded loss function, i.e.,  $|l(\cdot ,\cdot)|\leq c$ . For an input  $x\in \mathcal{X}$ , we suppose  $\| x\| \leq B$  and  $a\in \mathcal{A}$  is its corresponding target function for the model. For example,  $a(x) = y\in \mathcal{V}$  for supervised classification and  $a(x) = x\in \mathcal{X}$  for unsupervised input reconstruction. In general, a loss function can be defined as  $l(h(x),a(x))$  for any hypothesis  $h\in \mathcal{H}$  and input  $x\in \mathcal{X}$ .  $h(x)$  represents the hypothesis output for  $x$ . For example,  $h(x)$  produces a predicted label by a classifier or the reconstructed input by unsupervised input reconstruction. For the mixture distribution  $u$  and its corresponding dataset  $\widehat{u}$ , we define the expected risk  $\epsilon_u(h)$  and empirical risk  $\widehat{\epsilon}_{\widehat{u}}(h)$  for any  $h\in \mathcal{H}$  [61]:
$$
\epsilon_ {u} (h) = \mathbb {E} _ {x \sim u} l (h (x), a (x)), \widehat {\epsilon} _ {\widehat {u}} (h) = \frac {1}{2 N} \sum_ {x \in \widehat {u}} l (h (x), a (x)). \tag {1}
$$
Then, we define the optimal hypothesis  $h_u^*$  and the minimum hypothesis  $\widehat{h}_u$ , respectively,
$$
h _ {u} ^ {*} \in \arg \min  _ {h \in \mathcal {H}} \epsilon_ {u} (h), \widehat {h} _ {u} \in \arg \min  _ {h \in \mathcal {H}} \widehat {\epsilon} _ {\widehat {u}} (h). \tag {2}
$$
Figure 1: R-divergence estimates the distribution discrepancy on datasets  $\widehat{p}$  and  $\widehat{q}$ .  $\widehat{h}_{\widehat{u}}$  represents the minimum hypothesis learned on the mixed data  $\widehat{u} = \widehat{p} \cup \widehat{q}$ .  $\widehat{\epsilon}_{\widehat{p}}$  and  $\widehat{\epsilon}_{\widehat{q}}$  represent the empirical risks of the minimum hypothesis on data  $\widehat{p}$  and  $\widehat{q}$ , respectively. R-divergence  $\widehat{\mathcal{D}}_{\mathrm{R}}$  estimates the distribution discrepancy by the empirical risk difference between the two datasets.
A learning model of the task aims to learn the minimum hypothesis  $\widehat{h}$  on observed data to approximate the optimal hypothesis  $h^*$ . This aim can be converted to the model-oriented two-sample test: For datasets  $\widehat{p}$  and  $\widehat{q}$  and their learning task with hypothesis space  $\mathcal{H}$  and loss function  $l$ , can we determine whether their probability distributions are identical (e.g.,  $p = q$ ) for the specific model? This research question is equivalent to: whether the samples of the two datasets can be treated as sampled from the same distribution for the model?
3 R-divergence for Distribution Discrepancy
To fulfill the model-oriented two-sample test, we develop R-divergence to estimate the discrepancy between two probability distributions  $p$  and  $q$  for a model  $\mathcal{T}$ . R-divergence regards two distributions as identical if the optimal hypothesis on their mixture distribution has the same expected risk on each individual distribution. Accordingly, with the corresponding datasets  $\widehat{p}$  and  $\widehat{q}$ , R-divergence estimates the optimal hypothesis  $h_u^*$  on the mixture distribution  $u$  by learning the minimum hypothesis  $\widehat{h_{\widehat{u}}}$  on the mixed dataset  $\widehat{u}$ . Then, R-divergence estimates the expected risks  $\epsilon_p(h_u^*)$  and  $\epsilon_q(h_u^*)$  on the two distributions by evaluating the empirical risks  $\widehat{\epsilon_{\widehat{p}}}(\widehat{h_{\widehat{u}}})$  and  $\widehat{\epsilon_{\widehat{q}}}(\widehat{h_{\widehat{u}}})$  on their data samples. R-divergence measures the empirical risk difference between  $\widehat{\epsilon_{\widehat{p}}}(\widehat{h_{\widehat{u}}})$  and  $\widehat{\epsilon_{\widehat{q}}}(\widehat{h_{\widehat{u}}})$  to estimate the distribution discrepancy.
Accordingly, for a supervised or unsupervised model with hypothesis space  $\mathcal{H}$  and loss function  $l$ , R-divergence evaluates the discrepancy between distributions  $p$  and  $q$  by,
$$
\mathcal {D} _ {\mathbb {R}} (p \| q) = d \left(\epsilon_ {p} \left(h _ {u} ^ {*}\right), \epsilon_ {q} \left(h _ {u} ^ {*}\right)\right) = \left| \epsilon_ {p} \left(h _ {u} ^ {*}\right) - \epsilon_ {q} \left(h _ {u} ^ {*}\right) \right|. \tag {3}
$$
$d(\cdot, \cdot)$  represents the absolute difference. Intuitively, if  $p = q$ , then  $u = p = q$  and  $h_u^*$  is the optimal hypothesis for both  $p$  and  $q$ . Thus, the expected risks on the two distributions are the same, which leads to a zero discrepancy. In contrast, if  $p \neq q$ ,  $h_u^*$  is the optimal hypothesis for  $u$ , and it performs differently on  $p$  and  $q$ . Then, the expected risk difference can evaluate the discrepancy between the two distributions for the specific model. Different from F-divergence without considering specific learning models, R-divergence involves the property of the specific model in evaluating the model-oriented distribution discrepancy conditional on the optimal hypothesis. Specifically, R-divergence infers the discrepancy by comparing the distance between the learning results on two distributions. The learning result is sensitive to the learning model, i.e., its hypothesis space and loss function.
Further,  $\mathcal{D}_{\mathrm{R}}(p\| q)$  can be estimated on datasets  $\widehat{p}$  and  $\widehat{q}$ . R-divergence estimates the discrepancy by searching a minimum hypothesis  $\widehat{h}_u$  on the mixed dataset  $\widehat{u} = \widehat{p} \cup \widehat{q}$ , and calculates the empirical risks  $\widehat{\epsilon_{\widehat{p}}}(\widehat{h}_u)$  and  $\widehat{\epsilon_{\widehat{q}}}(\widehat{h}_u)$  on each individual dataset. Then,  $\mathcal{D}_{\mathrm{R}}(p\| q)$  can be estimated by the empirical risk difference  $\widehat{\mathcal{D}}_{\mathrm{R}}(p\| q)$ :
$$
\widehat {\mathcal {D}} _ {\mathrm {R}} (\widehat {p} \| \widehat {q}) = d \left(\widehat {\epsilon_ {\widehat {p}}} \left(\widehat {h _ {\widehat {u}}}\right), \widehat {\epsilon_ {\widehat {q}}} \left(\widehat {h _ {\widehat {u}}}\right)\right) = \left| \widehat {\epsilon_ {\widehat {p}}} \left(\widehat {h _ {\widehat {u}}}\right) - \widehat {\epsilon_ {\widehat {q}}} \left(\widehat {h _ {\widehat {u}}}\right) \right|. \tag {4}
$$
$\widehat{\mathcal{D}}_{\mathbb{R}}(\widehat{p}\| \widehat{q})$  can be treated as an empirical estimator for the distribution discrepancy  $\mathcal{D}_{\mathbb{R}}(p\| q)$
Because  $\widehat{h}_u$  is the minimum hypothesis on the mixed dataset  $\widehat{u}$ ,  $\widehat{h}_u$  will overfit neither  $\widehat{p}$  nor  $\widehat{q}$ . Specifically, both  $\widehat{\epsilon}_{\widehat{p}}(\widehat{h}_u)$  and  $\widehat{\epsilon}_{\widehat{q}}(\widehat{h}_u)$  will not tend towards zero when  $\widehat{p}$  and  $\widehat{q}$  are different. This property ensures that the empirical estimator will not equal zero for different given datasets. Further, as  $\widehat{u}$  consists of  $\widehat{p}$  and  $\widehat{q}$ , both  $\widehat{\epsilon}_{\widehat{p}}(\widehat{h}_u)$  and  $\widehat{\epsilon}_{\widehat{q}}(\widehat{h}_u)$  will not tend towards extremely large values, which ensures a stable empirical estimator. In general, due to the property that the minimum hypothesis learned on the mixed data is applied to evaluate the empirical risk on each individual dataset, R-divergence can address the overfitting issue in H-divergence [62]. The procedure of estimating the model-oriented discrepancy between two probability distributions is summarized in Algorithm 1 (see Appendix B.1).
R-divergence diverges substantially from L-divergence [31] and H-divergence [62]. While R-divergences and L-divergences apply to different hypotheses and scenarios, R-divergences and H-divergences differ in their assumptions on discrepancy evaluation and risk calculation. Notably, R-divergence enhances performance by mitigating the overfitting issues seen with H-divergence. H-divergence assumes that two distributions are different if the optimal decision loss is higher on their mixture than on individual distributions. As a result, H-divergence calculates low empirical risks for different datasets, leading to underestimated discrepancies. In contrast, R-divergence tackles this by assuming that two distributions are likely identical if the optimal hypothesis yields the same expected risk on each. Thus, R-divergence focuses on a minimum hypothesis based on mixed data and assesses its empirical risks on the two individual datasets. It treats the empirical risk gap as the discrepancy, ensuring both small and large gaps for similar and different datasets, respectively. Technically, R-divergence looks at a minimum hypothesis using mixed data, while L-divergence explores the entire hypothesis space. Consequently, R-divergence is model-oriented, accounting for their hypothesis space, loss function, target function, and optimization process.
Accordingly, there is no need to select hypothesis spaces and loss functions for R-divergence, as it estimates the discrepancy tailored to a specific learning task. Hence, it is reasonable to anticipate different results for varying learning tasks. Whether samples from two datasets are drawn from the same distribution is dependent on the particular learning model in use, and different models exhibit varying sensitivities to distribution discrepancies. Therefore, the distributions of two given datasets may be deemed identical for some models, yet significantly different for others.
Here, we present the convergence results of the proposed R-divergence method, i.e., the bound on the difference between the empirical estimator  $\hat{\mathcal{D}}_{\mathrm{R}}(\hat{p}\| \hat{q})$  and the discrepancy  $\mathcal{D}_{\mathrm{R}}(p\| q)$ . We define the L-divergence [35]  $\mathcal{D}_{\mathrm{L}}(p\| q)$  as
$$
\mathcal {D} _ {\mathrm {L}} (p \| q) = \sup  _ {h \in \mathcal {H}} \left(\left| \epsilon_ {p} (h) - \epsilon_ {q} (h) \right|\right). \tag {5}
$$
Further, we define the Rademacher complexity  $\mathcal{R}_{\mathcal{H}}^l (\widehat{u})$  [4] as
$$
\mathcal {R} _ {\mathcal {H}} ^ {l} (\hat {u}) \equiv \frac {1}{2 N} \mathbb {E} _ {\sigma \sim \{\pm 1 \} ^ {2 N}} \left[ \sup  _ {h \in \mathcal {H}} \sum_ {x \in \hat {u}} \sigma l (h (x), a (x)) \right]. \tag {6}
$$
According to the generalization bounds in terms of L-divergence and Rademacher complexity, we can bound the difference between  $\widehat{\mathcal{D}}_{\mathrm{R}}(\widehat{p}\| \widehat{q})$  and  $\mathcal{D}_{\mathrm{R}}(p\| q)$
Theorem 4.1. For any  $x \in \mathcal{X}$ ,  $h \in \mathcal{H}$  and  $a \in \mathcal{A}$ , assume we have  $|l(h(x), a(x))| \leq c$ . Then, with probability at least  $1 - \delta$ , we can derive the following general bound,
$$
\left| \mathcal {D} _ {R} (p \| q) - \widehat {\mathcal {D}} _ {R} (\widehat {p} \| \widehat {q}) \right| \leq 2 \mathcal {D} _ {\mathrm {L}} (p \| q) + 2 \mathcal {R} _ {\mathcal {H}} ^ {l} (\widehat {p}) + 2 \mathcal {R} _ {\mathcal {H}} ^ {l} (\widehat {q}) + 1 2 c \sqrt {\frac {\ln (8 / \delta)}{N}}.
$$
Corollary 4.2. Following the conditions of Theorem 4.1, the upper bound of  $\sqrt{\operatorname{Var}\left[\widehat{\mathcal{D}}_R(\widehat{p}||\widehat{q})\right]}$  is
$$
\frac {3 4 c}{\sqrt {N}} + \sqrt {2 \pi} \left(2 \mathcal {D} _ {\mathrm {L}} (p \| q) + 2 \mathcal {R} _ {\mathcal {H}} ^ {l} (\widehat {p}) + 2 \mathcal {R} _ {\mathcal {H}} ^ {l} (\widehat {q})\right).
$$
The detailed proof is given in Section A. Note that both the convergence and variance of  $\widehat{\mathcal{D}}_{\mathrm{R}}(\widehat{p} \| \widehat{q})$  depend on the L-divergence, Rademacher complexity, and the number of samples from each dataset. Recall that L-divergence is based on the whole hypothesis space, but R-divergence is based on an optimal hypothesis. The above results reveal the relations between these two measures: a smaller L-divergence between two probability distributions leads to a tighter bound on R-divergence. As the Rademacher complexity measures the hypothesis complexity, we specifically consider the hypothesis complexity of deep neural networks (DNNs). According to Talagrand's contraction lemma [44] and the sample complexity of DNNs [18, 59], we can obtain the following bound for DNNs.
Proposition 4.3. Based on the conditions of Theorem 4.1, we assume  $\mathcal{H}$  is the class of real-valued networks of depth  $D$  over the domain  $\mathcal{X}$ . Let the Frobenius norm of the weight matrices be at most  $M_1,\ldots ,M_D$ , the activation function be 1-Lipschitz, positive-homogeneous and applied element-wise (such as the ReLU). Then, with probability at least  $1 - \delta$ , we have,
$$
| \mathcal {D} _ {\mathrm {R}} (p \| q) - \widehat {\mathcal {D}} _ {\mathrm {R}} (\widehat {p} \| \widehat {q}) | \leq 2 \mathcal {D} _ {\mathrm {L}} (p \| q) + \frac {4 L B (\sqrt {2 D \ln 2} + 1) \prod_ {i = 1} ^ {D} M _ {i}}{\sqrt {N}} + 1 2 c \sqrt {\frac {\ln (8 / \delta)}{N}}.
$$
Accordingly, we can obtain a tighter bound if the L-divergence  $D_{\mathrm{L}}(p\| q)$  meets a certain condition.
Corollary 4.4. Following the conditions of Proposition 4.3, with probability at least  $1 - \delta$ , we have the following conditional bounds if  $\mathcal{D}_{\mathrm{L}}(p\| q)\leq \left|\epsilon_u(h_u^*) - \epsilon_u(\widehat{h}_u)\right|$
$$
\left| \mathcal {D} _ {\mathrm {R}} (p \| q) - \widehat {\mathcal {D}} _ {\mathrm {R}} (\widehat {p} \| \widehat {q}) \right| \leq \frac {8 L B (\sqrt {2 D \ln 2} + 1) \prod_ {i = 1} ^ {D} M _ {i}}{\sqrt {N}} + 2 2 c \sqrt {\frac {\ln (1 6 / \delta)}{N}}.
$$
Then, the empirical estimator  $\widehat{\mathcal{D}}_{\mathrm{R}}(\widehat{p}\| \widehat{q})$  converge uniformly to the discrepancy  $\mathcal{D}_{\mathrm{R}}(p\| q)$  as the number of samples  $N$  increases. Then, based on the result of Proposition 4.3, we can obtain a general upper bound of  $\widehat{\mathcal{D}}_{\mathrm{R}}(\widehat{p}\| \widehat{q})$ .
Corollary 4.5. Following the conditions of Proposition 4.3, as  $N\to \infty$  , we have,
$$
\widehat {\mathcal {D}} _ {\mathrm {R}} (\widehat {p} \| \widehat {q}) \leq 2 \mathcal {D} _ {\mathrm {L}} (p \| q) + \mathcal {D} _ {\mathrm {R}} (p \| q).
$$
The empirical estimator will converge to the sum of L-divergence  $\mathcal{D}_{\mathrm{L}}(p||q)$  and R-divergence  $\mathcal{D}_{\mathrm{R}}(p||q)$  as the sample size grows, as indicated in Corollary 4.5. As highlighted by Theorem 4.1, a large L-divergence can influence the convergence of the empirical estimator. However, a large L-divergence is also useful for determining whether samples from two datasets are drawn from different distributions, as it signals significant differences between the two. Therefore, both L-divergence and R-divergence quantify the discrepancy between two given probability distributions within a learning model. Specifically, L-divergence measures this discrepancy across the entire hypothesis space, while R-divergence does so in terms of the optimal hypothesis of the mixture distribution. Hence, a large L-divergence implies that the empirical estimator captures a significant difference between the two distributions. Conversely, a small L-divergence suggests that the empirical estimator will quickly converge to a minor discrepancy as the sample size increases, as evidenced in Corollary 4.4. Thus, the empirical estimator also reflects a minimal distribution discrepancy when samples from both datasets are drawn from the same distribution.
We evaluate the discrepancy between two probability distributions w.r.t. R-divergence (R-Div)<sup>1</sup> for both unsupervised and supervised learning models. Furthermore, we illustrate how R-Div is applied to learn robust DNNs from data corrupted by noisy labels. We compare R-Div with seven state-of-the-art methods that estimate the discrepancy between two probability distributions. The details of the comparison methods are presented in Appendix C.
<sup>1</sup>The source code is publicly available at: https://github.com/Lawliet-zzl/R-div.
The discrepancy between two probability distributions for a learning model is estimated on their given datasets IID drawn from the distributions, respectively. We decide whether two sets of samples are drawn from the same distribution. A model estimates the discrepancy on their corresponding datasets  $\widehat{p} = \{x_i\}_{i=1}^N \sim p$  and  $\widehat{q} = \{x_i'\}_{i=1}^N \sim q$ .  $\widehat{\mathcal{D}}_{\mathrm{R}}(\widehat{p} \| \widehat{q})$  serves as an empirical estimator for the discrepancy  $\mathcal{D}_{\mathrm{R}}(p \| q)$  to decide whether  $p \neq q$  if its output value exceeds a predefined threshold.