Title: R-divergence for Estimating Model-oriented Distribution Discrepancy
Venue: NeurIPS 2023
Accepted: yes
Keywords: non-IID

Distribution Discrepancy

Data Divergence

Two-sample Test
Abstract: Real-life data are often non-IID due to complex distributions and interactions, and the sensitivity to the distribution of samples can differ among learning models. Accordingly, a key question for any supervised or unsupervised model is whether the probability distributions of two given datasets can be considered identical. To address this question, we introduce R-divergence, designed to assess model-oriented distribution discrepancies. The core insight is that two distributions are likely identical if their optimal hypothesis yields the same expected risk for each distribution. To estimate the distribution discrepancy between two datasets, R-divergence learns a minimum hypothesis on the mixed data and then gauges the empirical risk difference between them. We evaluate the test power across various unsupervised and supervised tasks and find that R-divergence achieves state-of-the-art performance. To demonstrate the practicality of R-divergence, we employ R-divergence to train robust neural networks on samples with noisy labels.

R-divergence for Estimating Model-oriented Distribution Discrepancy

Abstract

Real-life data are often non-IID due to complex distributions and interactions, and the sensitivity to the distribution of samples can differ among learning models. Accordingly, a key question for any supervised or unsupervised model is whether the probability distributions of two given datasets can be considered identical. To address this question, we introduce R-divergence, designed to assess modeloriented distribution discrepancies. The core insight is that two distributions are likely identical if their optimal hypothesis yields the same expected risk for each distribution. To estimate the distribution discrepancy between two datasets, Rdivergence learns a minimum hypothesis on the mixed data and then gauges the empirical risk difference between them. We evaluate the test power across various unsupervised and supervised tasks and find that R-divergence achieves state-ofthe-art performance. To demonstrate the practicality of R-divergence, we employ R-divergence to train robust neural networks on samples with noisy labels.

1 Introduction

Most machine learning methods rely on the basic independent and identically distributed (IID) assumption [7], implying that variables are drawn independently from the same distribution. However, real-life data and machine learning models often deviate from this IID assumption due to various complexities, such as mixture distributions and interactions [9]. For a given supervised or unsupervised task [45], a learning model derives an optimal hypothesis from a hypothesis space by optimizing the corresponding expected risk [53]. However, this optimal hypothesis may not be valid if the distributions of training and test samples differ [54], resulting in distributional vulnerability [63]. In fact, an increasing body of research addresses complex non-IID scenarios [8], including open-world problems [6], out-of-distribution detection [23, 63], domain adaptation [28], and learning with noisy labels [46]. These scenarios raise a crucial yet challenging question: how to assess the discrepancy between two probability distributions for a specific learning model?

Empirically, the discrepancy between two probability distributions can be assessed by the divergence between their respective datasets. However, the actual underlying distributions are often unknown, with only a limited number of samples available for observation. Whether the samples from two datasets come from the same distribution is contingent upon the specific learning model being used. This is because different learning models possess unique hypothesis spaces, loss functions, target functions, and optimization processes, which result in varying sensitivities to distribution discrepancies. For instance, in binary classification [26], samples from two datasets might be treated as positive and negative, respectively, indicating that they stem from different distributions. Conversely, a model designed to focus on invariant features is more likely to treat all samples as if they are drawn from the same distribution. For example, in one-class classification [52], all samples are considered positive, even though they may come from different components of a mixture distribution.

In cases where the samples from each dataset originate from multiple distributions, they can be viewed as drawn from a complex mixture distribution. We can then estimate the discrepancy between these two complex mixture distributions. This suggests that, in practice, it is both meaningful and necessary to evaluate the distribution discrepancy for a specific learning model, addressing the issue of model-oriented distribution discrepancy evaluation .

Estimating the discrepancy between two probability distributions has been a foundational and challenging issue. Various metrics for this task have been proposed, including F-divergence [12], integral probability metrics (IPM)[2], and H-divergence[62]. F-divergence metrics, like KullbackLeibler (KL) divergence [13] and Jensen Shannon divergence [47], assume that two distributions are identical if they possess the same likelihood at every point. Importantly, F-divergence is not tied to a specific learning model, as it directly measures discrepancy by computing the statistical distance between the two distributions. IPM metrics, including the Wasserstein distance [48], maximum mean discrepancy (MMD)[25], and L-divergence[5, 31], assess discrepancy based on function outputs. They postulate that any function should yield the same expectation under both distributions if they are indeed identical, and vice versa. In this vein, L-divergence explores the hypothesis space for a given model and regards the largest gap in expected risks between the two distributions as their discrepancy. This necessitates evaluating all hypotheses within the hypothesis space, a task which can be computationally demanding. Moreover, hypotheses unrelated to the two distributions could result in inaccurate or biased evaluations. On the other hand, H-divergence [62] contends that two distributions are distinct if the optimal decision loss is greater when computed on their mixed distribution compared to each individual one. It calculates this optimal decision loss in terms of training loss, meaning it trains a minimal hypothesis and evaluates its empirical risk on the identical dataset. A network with limited training data is susceptible to overfitting [50], which can lead to underestimated discrepancies when using H-divergence, even if the two distributions are significantly different.

To effectively gauge the model-oriented discrepancy between two probability distributions, we introduce R-divergence. This novel metric is built on the notion that two distributions are identical if the optimal hypothesis for their mixture distribution yields the same expected risk on each individual distribution . In line with this concept, R-divergence evaluates the discrepancy by employing an empirical estimator tailored to the given datasets and learning model. Initially, a minimum hypothesis is derived from the mixed data of the two datasets to approximate the optimal hypothesis. Subsequently, empirical risks are computed by applying this minimum hypothesis to each of the two individual datasets. Finally, the difference in these empirical risks serves as the empirical estimator for the discrepancy between the two probability distributions. The framework of R-divergence is illustrated in Figure 1.

2 Model-oriented Two-sample Test

We measure the discrepancy between two probability distributions p and q for a specific supervised or unsupervised learning method. Let X and Y be the spaces of inputs and labels, respectively. Assume we observe two sets of N IID samples from the two distributions, i.e., ̂ p = { x i } N i =1 ∼ p , and ̂ q = { x ′ i } N i =1 ∼ q , respectively. We assume the mixture distribution u = ( p + q ) / 2 and its corresponding mixed data ̂ u = ̂ p ∪ ̂ q .

For a learning model T , we assume H is its hypothesis space and l is its L -Lipschitz bounded loss function, i.e., | l ( · , · ) | ≤ c . For an input x ∈ X , we suppose ‖ x ‖ ≤ B and a ∈ A is its corresponding target function for the model. For example, a ( x ) = y ∈ Y for supervised classification and a ( x ) = x ∈ X for unsupervised input reconstruction. In general, a loss function can be defined as l ( h ( x ) , a ( x )) for any hypothesis h ∈ H and input x ∈ X . h ( x ) represents the hypothesis output for x . For example, h ( x ) produces a predicted label by a classifier or the reconstructed input by unsupervised input reconstruction. For the mixture distribution u and its corresponding dataset ̂ u , we define the expected risk glyph[epsilon1] u ( h ) and empirical risk ̂ glyph[epsilon1] ̂ u ( h ) for any h ∈ H [61]:

Then, we define the optimal hypothesis h ∗ u and the minimum hypothesis ̂ h u , respectively,

Alearning model of the task aims to learn the minimum hypothesis ̂ h on observed data to approximate the optimal hypothesis h ∗ . This aim can be converted to the model-oriented two-sample test : For datasets ̂ p and ̂ q and their learning task with hypothesis space H and loss function l , can we determine whether their probability distributions are identical (e.g., p = q ) for the specific model? This research question is equivalent to: whether the samples of the two datasets can be treated as sampled from the same distribution for the model?

3 R-divergence for Distribution Discrepancy

To fulfill the model-oriented two-sample test, we develop R-divergence to estimate the discrepancy between two probability distributions p and q for a model T . R-divergence regards two distributions as identical if the optimal hypothesis on their mixture distribution has the same expected risk on each individual distribution. Accordingly, with the corresponding datasets ̂ p and ̂ q , R-divergence estimates the optimal hypothesis h ∗ u on the mixture distribution u by learning the minimum hypothesis ̂ h ̂ u on the mixed dataset ̂ u . Then, R-divergence estimates the expected risks glyph[epsilon1] p ( h ∗ u ) and glyph[epsilon1] q ( h ∗ u ) on the two distributions by evaluating the empirical risks ̂ glyph[epsilon1] ̂ p ( ̂ h ̂ u ) and ̂ glyph[epsilon1] ̂ q ( ̂ h ̂ u ) on their data samples. R-divergence measures the empirical risk difference between ̂ glyph[epsilon1] ̂ p ( ̂ h ̂ u ) and ̂ glyph[epsilon1] ̂ q ( ̂ h ̂ u ) to estimate the distribution discrepancy.

Accordingly, for a supervised or unsupervised model with hypothesis space H and loss function l , R-divergence evaluates the discrepancy between distributions p and q by,

glyph[negationslash]

d ( · , · ) represents the absolute difference. Intuitively, if p = q , then u = p = q and h ∗ u is the optimal hypothesis for both p and q . Thus, the expected risks on the two distributions are the same, which leads to a zero discrepancy. In contrast, if p = q , h ∗ u is the optimal hypothesis for u , and it performs differently on p and q . Then, the expected risk difference can evaluate the discrepancy between the two distributions for the specific model. Different from F-divergence without considering specific learning models, R-divergence involves the property of the specific model in evaluating the modeloriented distribution discrepancy conditional on the optimal hypothesis. Specifically, R-divergence infers the discrepancy by comparing the distance between the learning results on two distributions. The learning result is sensitive to the learning model, i.e., its hypothesis space and loss function.

Further, D R ( p ‖ q ) can be estimated on datasets ̂ p and ̂ q . R-divergence estimates the discrepancy by searching a minimum hypothesis ̂ h u on the mixed dataset ̂ u = ̂ p ∪ ̂ q , and calculates the empirical risks ̂ glyph[epsilon1] ̂ p ( ̂ h u ) and ̂ glyph[epsilon1] ̂ q ( ̂ h u ) on each individual dataset. Then, D R ( p ‖ q ) can be estimated by the empirical risk difference ̂ D R ( p ‖ q ) :

̂ D R ( ̂ p ‖ ̂ q ) can be treated as an empirical estimator for the distribution discrepancy D R ( p ‖ q ) .

Because ̂ h u is the minimum hypothesis on the mixed dataset ̂ u , ̂ h u will overfit neither ̂ p nor ̂ q . Specifically, both ̂ glyph[epsilon1] ̂ p ( ̂ h u ) and ̂ glyph[epsilon1] ̂ q ( ̂ h u ) will not tend towards zero when ̂ p and ̂ q are different. This property ensures that the empirical estimator will not equal zero for different given datasets. Further, as ̂ u consists of ̂ p and ̂ q , both ̂ glyph[epsilon1] ̂ p ( ̂ h u ) and ̂ glyph[epsilon1] ̂ q ( ̂ h u ) will not tend towards extremely large values, which ensures a stable empirical estimator. In general, due to the property that the minimum hypothesis learned on the mixed data is applied to evaluate the empirical risk on each individual dataset, Rdivergence can address the overfitting issue in H-divergence [62]. The procedure of estimating the model-oriented discrepancy between two probability distributions is summarized in Algorithm 1 (see Appendix B.1).

R-divergence diverges substantially from L-divergence [31] and H-divergence [62]. While Rdivergences and L-divergences apply to different hypotheses and scenarios, R-divergences and H-divergences differ in their assumptions on discrepancy evaluation and risk calculation. Notably, R-divergence enhances performance by mitigating the overfitting issues seen with H-divergence. H-divergence assumes that two distributions are different if the optimal decision loss is higher on their mixture than on individual distributions. As a result, H-divergence calculates low empirical risks for different datasets, leading to underestimated discrepancies. In contrast, R-divergence tackles this by assuming that two distributions are likely identical if the optimal hypothesis yields the same expected risk on each. Thus, R-divergence focuses on a minimum hypothesis based on mixed data and assesses its empirical risks on the two individual datasets. It treats the empirical risk gap as the discrepancy, ensuring both small and large gaps for similar and different datasets, respectively. Technically, R-divergence looks at a minimum hypothesis using mixed data, while L-divergence explores the entire hypothesis space. Consequently, R-divergence is model-oriented, accounting for their hypothesis space, loss function, target function, and optimization process.

Accordingly, there is no need to select hypothesis spaces and loss functions for R-divergence, as it estimates the discrepancy tailored to a specific learning task. Hence, it is reasonable to anticipate different results for varying learning tasks. Whether samples from two datasets are drawn from the same distribution is dependent on the particular learning model in use, and different models exhibit varying sensitivities to distribution discrepancies. Therefore, the distributions of two given datasets may be deemed identical for some models, yet significantly different for others.

4 Theoretical Analysis

Here, we present the convergence results of the proposed R-divergence method, i.e., the bound on the difference between the empirical estimator ̂ D R ( ̂ p ‖ ̂ q ) and the discrepancy D R ( p ‖ q ) . We define the L-divergence [35] D L ( p ‖ q ) as

Further, we define the Rademacher complexity R l H ( ̂ u ) [4] as

According to the generalization bounds in terms of L-divergence and Rademacher complexity, we can bound the difference between ̂ D R ( ̂ p ‖ ̂ q ) and D R ( p ‖ q ) .

Theorem 4.1. For any x ∈ X , h ∈ H and a ∈ A , assume we have | l ( h ( x ) , a ( x )) | ≤ c . Then, with probability at least 1 -δ , we can derive the following general bound,

Corollary 4.2. Following the conditions of Theorem 4.1, the upper bound of √ Var [ ̂ D R ( ̂ p ‖ ̂ q ) ] is

The detailed proof is given in Section A. Note that both the convergence and variance of ̂ D R ( ̂ p ‖ ̂ q ) depend on the L-divergence, Rademacher complexity, and the number of samples from each dataset. Recall that L-divergence is based on the whole hypothesis space, but R-divergence is based on an optimal hypothesis. The above results reveal the relations between these two measures: a smaller L-divergence between two probability distributions leads to a tighter bound on R-divergence. As the Rademacher complexity measures the hypothesis complexity, we specifically consider the hypothesis complexity of deep neural networks (DNNs). According to Talagrand's contraction lemma [44] and the sample complexity of DNNs [18, 59], we can obtain the following bound for DNNs.

Proposition 4.3. Based on the conditions of Theorem 4.1, we assume H is the class of real-valued networks of depth D over the domain X . Let the Frobenius norm of the weight matrices be at most M 1 , . . . , M D , the activation function be 1-Lipschitz, positive-homogeneous and applied element-wise (such as the ReLU). Then, with probability at least 1 -δ , we have,

Accordingly, we can obtain a tighter bound if the L-divergence D L ( p ‖ q ) meets a certain condition.

Corollary 4.4. Following the conditions of Proposition 4.3, with probability at least 1 -δ , we have the following conditional bounds if D L ( p ‖ q ) ≤ ∣ ∣ ∣ glyph[epsilon1] u ( h ∗ u ) -glyph[epsilon1] u ( ̂ h u ) ∣ ∣ ∣ ,

Then, the empirical estimator ̂ D R ( ̂ p ‖ ̂ q ) converge uniformly to the discrepancy D R ( p ‖ q ) as the number of samples N increases. Then, based on the result of Proposition 4.3, we can obtain a general upper bound of ̂ D R ( ̂ p ‖ ̂ q ) .

Corollary 4.5. Following the conditions of Proposition 4.3, as N →∞ , we have,

The empirical estimator will converge to the sum of L-divergence D L ( p || q ) and R-divergence D R ( p ‖ q ) as the sample size grows, as indicated in Corollary 4.5. As highlighted by Theorem 4.1, a large L-divergence can influence the convergence of the empirical estimator. However, a large L-divergence is also useful for determining whether samples from two datasets are drawn from different distributions, as it signals significant differences between the two. Therefore, both L-divergence and R-divergence quantify the discrepancy between two given probability distributions within a learning model. Specifically, L-divergence measures this discrepancy across the entire hypothesis space, while R-divergence does so in terms of the optimal hypothesis of the mixture distribution. Hence, a large L-divergence implies that the empirical estimator captures a significant difference between the two distributions. Conversely, a small L-divergence suggests that the empirical estimator will quickly converge to a minor discrepancy as the sample size increases, as evidenced in Corollary 4.4. Thus, the empirical estimator also reflects a minimal distribution discrepancy when samples from both datasets are drawn from the same distribution.

5 Experiments

We evaluate the discrepancy between two probability distributions w.r.t. R-divergence (R-Div) 1 for both unsupervised and supervised learning models. Furthermore, we illustrate how R-Div is applied to learn robust DNNs from data corrupted by noisy labels. We compare R-Div with seven state-of-the-art methods that estimate the discrepancy between two probability distributions. The details of the comparison methods are presented in Appendix C.

5.1 Evaluation Metrics

The discrepancy between two probability distributions for a learning model is estimated on their given datasets IID drawn from the distributions, respectively. We decide whether two sets of samples are drawn from the same distribution. A model estimates the discrepancy on their corresponding datasets ̂ p = { x i } N i =1 ∼ p and ̂ q = { x ′ i } N i =1 ∼ q . ̂ D R ( ̂ p ‖ ̂ q ) serves as an empirical estimator for the discrepancy D R ( p ‖ q ) to decide whether p = q if its output value exceeds a predefined threshold.

glyph[negationslash]

glyph[negationslash]

glyph[negationslash]

For an algorithm that decides if p = q , the Type I error [14] occurs when it incorrectly decides p = q , and the corresponding probability of Type I error is named the significant level . In contrast, the Type II error [14] occurs when it incorrectly decides p = q , and the corresponding probability of not making this error is called the test power [34]. In the experiments, the samples of two given datasets ̂ p and ̂ q are drawn from different distributions, i.e., the ground truth is p = q . Both the significant level and the test power are sensitive to the two probability distributions. Accordingly, to measure the performance of a learning algorithm, we calculate its test power when a certain significant level α ∈ [0 , 1] can be guaranteed. A higher test power indicates better performance when p = q .

glyph[negationslash]

glyph[negationslash]

The significant level can be guaranteed with a permutation test [17]. For the given datasets ̂ p and ̂ q , the permutation test uniformly randomly swaps samples between ̂ p and ̂ q to generate Z dataset pairs { ( ̂ p z , ̂ q z ) } Z z =1 . For ̂ p and ̂ q , R-Div outputs the empirical estimator ̂ D R ( ̂ p ‖ ̂ q ) . Note that if p = q , swapping samples between ̂ p and ̂ q cannot change the distribution, which indicates that the permutation test can guarantee a certain significant level. If the samples of ̂ p and ̂ q are drawn from different distributions, the test power is 1 if the algorithm can output p = q Otherwise, the test power is 0 . Specifically, to guarantee the α significant level, the algorithm output p = q if ̂ D R ( ̂ p ‖ ̂ q ) is in the top α -quantile among G = { ̂ D R ( ̂ p z ‖ ̂ q z ) } Z z =1 . We perform the test for K times using different randomly-selected dataset pairs ( ̂ p, ̂ q ) and average the output values as the test power of the algorithm. If not specified, we set α = 0 . 05 , Z = 100 and K = 100 . The procedure of calculating the average test power of R-Div is summarized in Algorithm 2 (see Appendix B.2).

5.2 Unsupervised Learning Tasks

Benchmark Datasets: Following the experimental setups as [41] and [62], we adopt four benchmark datasets, namely Blob, HDGM, HIGGS, and MNIST. We consider learning unsupervised models on these datasets, and the considered hypothesis spaces are for generative models. Specifically, MNIST adopts the variational autoencoders (V AE) [33], while the other low dimensional data adopt kernel density estimation (KDE) [27]. For fair comparisons, we follow the same setup as [62] for all methods. Because all the compared methods have hyper-parameters, we evenly split each dataset into a training set and a validation set to tune hyper-parameters and compute the final test power, respectively. The average test powers for a certain significant level on HDGM [41], Blob [41], MNIST [38] and HIGGS [1] are presented in Figure 3 , Figure 4 , Table 8 (see Appendix D.1) and Table 2, respectively. Overall, our proposed R-Div achieves the best test power on all datasets. On HDGM, R-Div obtains the best performance for different dimensions and achieves the perfect test power with fewer samples. Specifically, the test power of R-Div decreases gracefully as the dimension increases on HDGM. On Blob with different significant levels, R-Div achieves the same test power as the second-best method with a lower variance. On MNIST, both H-Div and R-Div achieve the perfect test power on all the sample sizes. On HIGGS, R-Div achieves the perfect test power even on the dataset with the smallest scale.

The discrepancy as measured by R-Div is distinct for two different distributions, even with a small sample size. This advantage stems from mitigating overfitting of H-Div by learning the minimum hypothesis and evaluating its empirical risks across varying datasets. R-Div learns a minimum hypothesis based on the mixed data and yields consistent empirical risks for both datasets when samples are sourced from the same distribution. Conversely, if samples from the two datasets come from different distributions, the empirical risks differ. This is because the training dataset for the minimum hypothesis does not precisely match either of the individual datasets.

To better demonstrate the advantages of R-Div, we measure the amount of overfitting by calculating empirical risks on p and q . The results presented in Table 1 indicate that R-Div achieves a notably clearer discrepancy, attributable to the significant differences in empirical risks on p and q . This is because R-Div mitigates overfitting by optimizing a minimum hypothesis on a mixed dataset glyph[negationslash]

glyph[negationslash]

and assessing its empirical risks across diverse datasets. Consequently, R-Div can yield a more pronounced discrepancy for two markedly different distributions.

5.3 Supervised Learning Tasks

Multi-domain Datasets: We adopt the PACS dataset [40] consisting of samples from four different domains, including photo (P), art painting (A), cartoon (C), and sketch (S). Following [40] and [57], we use AlexNet [37] as the backbone. Accordingly, samples from different domains can be treated as samples from different distributions. We consider the combinations of two domains and estimate their discrepancies. The results in Figure 5 (see Appendix D.2) represent that the test power of R-Div significantly improves as the number of samples increases. In addition, R-Div obtains the best performance on different data scales. The results in Table 3 show that R-Div achieves the superior test power across all the combinations of two domains. This is because the compared H-Div method suffers from the overfitting issue because it searches for a minimum hypothesis and evaluates its empirical risk on the same dataset. This causes empirical risks on different datasets to be minimal, even if they are significantly different. R-Div improves the performance of estimating the discrepancy by addressing this overfitting issue. Specifically, given two datasets, R-Div searches a minimum hypothesis on their mixed data and evaluates its empirical risks on each individual dataset. R-Div obtains stable empirical risks because the datasets used to search and evaluate the minimum hypothesis are not the same and partially overlap.

Spurious Correlations Datasets: We evaluate the probability distribution discrepancy between MNIST and Colored-MNIST (with spurious correlation) [3]. The adopted model for the two datasets is fully-connected and has two hidden layers of 128 ReLU units, and the parameters are learned using Adam [32]. The results in Table 4 reveal that the proposed R-Div achieves the best performance, which indicates that R-Div can explore the spurious correlation features by learning a minimum hypothesis on the mixed datasets.

Corrupted Datasets: We assess the probability distribution discrepancy between a given dataset and its corrupted variants. Adhering to the construction methods outlined by Sun et al.[55] and the framework of H-Div[62], we conduct experiments using H-Div and R-Div. We examine partial corruption types across four corruption levels on both CIFAR10 [36] and ImageNet [15], employing ResNet18 [22] as the backbone. Assessing discrepancies in datasets with subtle corruptions is particularly challenging due to the minor variations in their corresponding probability distributions. Therefore, we employ large sample sizes, N = 5000 for CIFAR10 and N = 1000 for ImageNet,

and observe that methods yield a test power of 0 when the corruption level is mild. Consistently across different corruption types, R-Div demonstrates superior performance at higher corruption levels and outperforms H-Div across all types and levels of corruption. Moreover, our results on ImageNet substantiate that the proposed R-Div effectively estimates distribution discrepancies in large-resolution datasets.

Advanced Network Architectures: For supervised learning tasks, R-Div selects the network architecture for the specific model. To verify the effect of R-div, we perform experiments on a wide range of architectures. We compare R-div with H-div because both of them are model-oriented and the other comparison methods have specific network architectures. Specifically, we consider the classification task on CIFAR10 and Corrupted-CIFAR10 and adopt four advanced network architectures, including ResNet18 [22], SENet [24], ConvNeXt [42], ViT [16]. The results are summarized in Table 6. For large-scale datasets (N = 10000), both H-Div and R-Div achieve perfect test power with different network architectures because more samples are applied to estimate the probability distribution discrepancy precisely. However, R-Div has the better test power for more challenging scenarios where the datasets are small (N = 1000, 5000). This may be because R-Div addresses the overfitting issue of H-Div by optimizing the minimum hypothesis and evaluating its empirical risk on different datasets. Furthermore, for different network architectures, CNN-based networks achieve better performance than the transformer-based models (ViT). According to the analysis of Park and Kim [49] and Li et al. [39], this may be because multi-head attentions (MHA) are low-pass filters with a shape bias, while convolutions are high-pass filters with a texture bias. As a result, ViT more likely focuses on the invariant shapes between CIFAR10 and the corrupted-CIFAR10 to make the two datasets less discriminative.

5.4 Case Study: Learning with Noisy Labels

R-Div is a useful tool to quantify the discrepancy between two probability distributions. We demonstrate how it can learn robust networks on data with noisy labels by distinguishing samples with noisy labels from clean samples in a batch. Furthermore, we explain how the discrepancy between clean and noisy samples affects the classification generalization ability [58].

5.4.1 Setup

We adopt the benchmark dataset CIFAR10 [36]. As CIFAR10 contains clean samples, following the setup [51], we corrupt its training samples manually, i.e., flipping partial clean labels to noisy labels. Specifically, we consider symmetry flipping [56] and pair flipping [21]. For corrupting a label, symmetry flipping transfers it to another random label, and pair flipping transfers it to a specific label. Following the setup [29], we adopt 0 . 2 noise rate, i.e., the labels of 20% samples will be flipped. The classification accuracy is evaluated on clean test samples.

We apply R-Div to train a robust DNN on a corrupted dataset. The training strategy separates clean and noisy samples by maximizing the discrepancy estimated by R-Div. The ground truth labels

for noisy samples are unreliable. Therefore, these labels can be treated as complementary labels representing the classes that the samples do not belong to. Accordingly, the training strategy optimizes the losses of ground truth and complementary labels for clean and noisy samples, respectively.

We assume that clean and noisy samples satisfy different distributions, i.e., clean and noisy distributions. Specifically, we apply R-Div to detect clean and noisy samples from a batch. Accordingly, we separate the batch samples B into predicted clean samples B c and predicted noisy samples B n by maximizing R-Div. The batch and the whole dataset contain IID samples from an unknown mixture distribution about clean and noisy distributions. Both the minimum hypotheses on the batch and the whole dataset can be treated to approximate the optimal hypothesis of the mixture distribution. To prevent learning a minimum hypothesis for each given batch, we pretrain a network on the whole corrupted dataset rather than a batch and apply this pretrained network (minimum hypothesis) to separate clean and noisy samples in a batch.

Recall that R-Div calculates the empirical risk difference between two datasets under the same minimum hypothesis of their mixed dataset. Hence, to maximize the discrepancy of R-Div, we calculate the empirical risk of each sample in the batch by the pretrained network, sort the batch samples according to the empirical risks, and treat the samples with low and high empirical risks as predicted clean and noisy samples, respectively. Specifically, we separate noisy and clean samples according to the estimated noise rate γ ∈ [0 , 1] . For clean samples with trusted ground truth labels, we optimize a network by minimizing the traditional cross-entropy loss. In contrast, the labels for noisy samples are untrusted. However, their noisy labels indicate the classes they do not belong to. Accordingly, we can treat their untrusted ground truth labels as complementary labels and optimize a network by minimizing the negative traditional cross-entropy loss on noisy samples. Thus, we retrain a robust network by optimizing the objective function min h ∈H ̂ glyph[epsilon1] B c ( h ) -̂ glyph[epsilon1] B n ( h ) for each batch, where the loss function l is the traditional cross-entropy loss. In this work, we adopt ResNet18 [22] as the backbone. In the pretraining and retraining processes, the learning rate [60] starts at 0.1 and is divided by 10 after 100 and 150 epochs. The batch size is 128 , and the number of epochs is 200 . The hyper-parameter γ varies from 0 . 1 to 0 . 7 with a 0 . 1 step size.

(a) Accuracy

(b) Detection

(c) Discrepancy

CIFAR10(Symmetric)

CIFAR10(Symmetric)

CIFAR10(Symmetric)

70

80

3.5×10-3

3

60

65

Discrepancy

2.5

40

2

60

1.5

Retrained(R-Div)

Pretrained

20

-Precision

Recall

1

55

0

0.5

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.1

0.2

0.3

0.4

0.5

0.6

0.7

5.4.2 Experimental Results

Weevaluate the classification accuracy on clean test samples, the precision and recall rates of detecting clean and noisy samples, and the discrepancy between predicted clean and noisy samples. The results are reported in Figure 2 and Figure 6 (see Appendix D.3). Recall that the noise rate is 0 . 2 . Figure 2(a) and Figure 6(a) show that the network retrained with R-Div improves the classification performance over the pretrained network when the estimated noise rate γ ∈ [0 . 2 , 0 . 5] . The retrained network achieves the best performance when the estimated noise rate is nearly twice as much as the real noise rate. To the best of our knowledge, how to best estimate the noise rate remains an open problem and severely limits the practical application of the existing algorithms. However, our method merely requires an approximate estimation which can be twice as much as the real noise rate.

Further, Figure 2(b) and Figure 6(b) show that there is a point of intersection between the precision and recall curves at γ = 0 . 5 . The aforementioned results indicate that γ ≤ 0 . 5 can lead to improved performance. Accordingly, we know that the detection precision is more essential than its recall rate. Specifically, a larger estimated noise rate ( γ ≥ 0 . 5 ) indicates that more clean samples are incorrectly recognized as noisy samples. In contrast, an extremely lower estimated noise rate which is lower than the real one indicates that more noisy samples are incorrectly recognized as clean samples. Both situations can mislead the network in learning to classify clean samples.

In addition, Figure 2(c) and Figure 6(c) show that a larger γ leads to a smaller discrepancy. Because the majority of training samples are clean, most of the clean samples have small test losses. A larger γ causes more clean samples to be incorrectly recognized as noisy samples, which decreases the test loss of the set of predicted noisy samples. Thus, the empirical estimator will decrease. Based on the results reported in Figure 2(a) and Figure 6(a), the largest discrepancy cannot lead to improved classification performance because most of the noisy samples are incorrectly recognized as clean samples. In addition, we observe that the retrained network obtains improved classification performance when the estimated discrepancy approximates the mean value 1 . 5 × 10 -3 , which can be used as a standard to select an estimated noise rate γ for retraining a robust network.

6 Conclusions

We tackle the crucial question of whether two probability distributions are identically distributed for a specific learning model using a novel and effective metric, R-divergence. Unlike existing divergence measures, R-divergence identifies a minimum hypothesis from a dataset that combines the two given datasets, and then uses this to compute the empirical risk difference between the individual datasets. Theoretical analysis confirms that R-divergence, serving as an empirical estimator, uniformly converges and is bounded by the discrepancies between the two distributions. Empirical evaluations reveal that R-divergence outperforms existing methods across both supervised and unsupervised learning paradigms. Its efficacy in handling learning with noisy labels further underscores its utility in training robust networks. If the samples from the two datasets are deemed identically distributed, a subsequent question to be investigated is whether these samples are also dependent.

Acknowledgments and Disclosure of Funding

This work was supported in part by the Australian Research Council Discovery under Grant DP190101079 and in part by the Future Fellowship under Grant FT190100734.

References

Claire Adam-Bourdarios, Glen Cowan, Cécile Germain, Isabelle Guyon, Balázs Kégl, and David Rousseau. How machine learning won the higgs boson challenge. In ESANN , pages 1-6, 2016.

Rohit Agrawal and Thibaut Horel. Optimal bounds between f-divergences and integral probability metrics. J. Mach. Learn. Res. , 22:1-128, 2021.

Martín Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. CoRR , pages 1-31, 2019.

Peter L. Bartlett and Shahar Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results. J. Mach. Learn. Res. , 3:463-482, 2002.

Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. A theory of learning from different domains. Mach. Learn. , 79(12):151-175, 2010.

Abhijit Bendale and Terrance E. Boult. Towards open set deep networks. In CVPR , pages 1563-1572, 2016.

Christopher M Bishop and Nasser M Nasrabadi. Pattern recognition and machine learning , volume 4. Springer, 2006.

Longbing Cao. Beyond i.i.d.: Non-iid thinking, informatics, and learning. IEEE Intell. Syst. , 37(4):5-17, 2022.

Longbing Cao. Non-iid learning. IEEE Intell. Syst. , 37(4):3-4, 2022.

Xiuyuan Cheng and Alexander Cloninger. Classification logit two-sample testing by neural networks for differentiating near manifold densities. IEEE Trans. Inf. Theory , 68(10):6631-6662, 2022.

Kacper Chwialkowski, Aaditya Ramdas, Dino Sejdinovic, and Arthur Gretton. Fast two-sample testing with analytic representations of probability measures. In NeurIPS , pages 1981-1989, 2015.

Imre Csiszár. Eine informationstheoretische ungleichung und ihre anwendung auf beweis der ergodizitaet von markoffschen ketten. Magyer Tud. Akad. Mat. Kutato Int. Koezl. , 8:85-108, 1964.

Imre Csiszár. I-divergence geometry of probability distributions and minimization problems. The annals of probability , pages 146-158, 1975.

Frederik Michel Dekking, Cornelis Kraaikamp, Hendrik Paul Lopuhaä, and Ludolf Erwin Meester. A Modern Introduction to Probability and Statistics: Understanding why and how , volume 488. Springer, 2005.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR , pages 248-255, 2009.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR , pages 1-22, 2021.

Michael D Ernst. Permutation methods: a basis for exact inference. Statistical Science , 19(4):676-685, 2004.

Noah Golowich, Alexander Rakhlin, and Ohad Shamir. Size-independent sample complexity. In COLT , volume 75, pages 297-299, 2018.

Mehmet Gönen and Ethem Alpaydin. Multiple kernel learning algorithms. J. Mach. Learn. Res. , 12:2211-2268, 2011.

Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Schölkopf, and Alexander J. Smola. A kernel two-sample test. J. Mach. Learn. Res. , 13:723-773, 2012.

Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor W. Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In NeurIPS , pages 8536-8546, 2018.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR , pages 770-778, 2016.

Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In ICLR , pages 1-12, 2017.

Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excitation networks. In CVPR , pages 7132-7141, 2018.

Arun Shankar Iyer, J. Saketha Nath, and Sunita Sarawagi. Maximum mean discrepancy for class ratio estimation: Convergence bounds and kernel selection. In ICML , volume 32, pages 530-538, 2014.

Hamid Jalalzai, Stéphan Clémençon, and Anne Sabourin. On binary classification in extreme regions. In NeurIPS , pages 3096-3104, 2018.

Heinrich Jiang. Uniform convergence rates for kernel density estimation. In ICML , volume 70, pages 1694-1703, 2017.

Junguang Jiang, Yifei Ji, Ximei Wang, Yufeng Liu, Jianmin Wang, and Mingsheng Long. Regressive domain adaptation for unsupervised keypoint detection. In CVPR , pages 6780-6789, 2021.

Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentornet: Learning datadriven curriculum for very deep neural networks on corrupted labels. ICML , pages 2309-2318, 2018.

Wittawat Jitkrittum, Zoltán Szabó, Kacper P. Chwialkowski, and Arthur Gretton. Interpretable distribution features with maximum testing power. In NeurIPS , pages 181-189, 2016.

Daniel Kifer, Shai Ben-David, and Johannes Gehrke. Detecting change in data streams. In VLDB , pages 180-191, 2004.

Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR , pages 1-13, 2015.

Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In ICLR , pages 1-14, 2014.

Matthias Kirchler, Shahryar Khorasani, Marius Kloft, and Christoph Lippert. Two-sample testing using deep learning. In AISTATS , pages 1387-1398, 2020.

Nikola Konstantinov and Christoph Lampert. Robust learning from untrusted sources. In ICML , pages 3488-3498, 2019.

Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.

Alex Krizhevsky. One weird trick for parallelizing convolutional neural networks. In CoRR , pages 1-7, 2014.

Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. In Proc. IEEE , volume 86, pages 2278-2324, 1998.

Bo Li, Jingkang Yang, Jiawei Ren, Yezhen Wang, and Ziwei Liu. Sparse fusion mixture-ofexperts are domain generalizable learners. In ICLR , pages 1-44, 2023.

Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M. Hospedales. Deeper, broader and artier domain generalization. In ICCV , pages 5543-5551, 2017.

Feng Liu, Wenkai Xu, Jie Lu, Guangquan Zhang, Arthur Gretton, and Danica J. Sutherland. Learning deep kernels for non-parametric two-sample tests. In ICML , pages 6316-6326, 2020.

Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In CVPR , pages 11966-11976, 2022.

David Lopez-Paz and Maxime Oquab. Revisiting classifier two-sample tests. In ICLR , pages 1-15, 2017.

Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of Machine Learning . MIT Press, 2018.

Kevin P. Murphy. Machine learning - a probabilistic perspective . MIT Press, 2012.

Nagarajan Natarajan, Inderjit S. Dhillon, Pradeep Ravikumar, and Ambuj Tewari. Learning with noisy labels. In NeurIPS , pages 1196-1204, 2013.

Frank Nielsen. On the jensen-shannon symmetrization of distances relying on abstract means. Entropy , 21(5):485, 2019.

Felix Otto and Michael Westdickenberg. Eulerian calculus for the contraction in the wasserstein distance. SIAM J. Math. Anal. , 37(4):1227-1255, 2005.

Namuk Park and Songkuk Kim. How do vision transformers work? In ICLR , pages 1-26, 2022.

Gabriel Pereyra, George Tucker, Jan Chorowski, Lukasz Kaiser, and Geoffrey E. Hinton. Regularizing neural networks by penalizing confident output distributions. In ICLR , pages 1-11, 2017.

Scott E. Reed, Honglak Lee, Dragomir Anguelov, Christian Szegedy, Dumitru Erhan, and Andrew Rabinovich. Training deep neuralnetworks on noisy labels with bootstrapping. In ICLR , 2015.

Lukas Ruff, Nico Görnitz, Lucas Deecke, Shoaib Ahmed Siddiqui, Robert A. Vandermeulen, Alexander Binder, Emmanuel Müller, and Marius Kloft. Deep one-class classification. In ICML , pages 4390-4399, 2018.

Shai Shalev-Shwartz and Shai Ben-David. Understanding Machine Learning From Theory to Algorithms . Cambridge University Press, 2014.

Masashi Sugiyama, Shinichi Nakajima, Hisashi Kashima, Paul von Bünau, and Motoaki Kawanabe. Direct importance estimation with model selection and its application to covariate shift adaptation. In NeurIPS , pages 1433-1440, 2007.

Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei A. Efros, and Moritz Hardt. Test-time training with self-supervision for generalization under distribution shifts. In ICML , pages 9229-9248, 2020.

Brendan van Rooyen, Aditya Krishna Menon, and Robert C. Williamson. Learning with symmetric label noise: The importance of being unhinged. In NeurIPS , pages 10-18, 2015.

Haohan Wang, Songwei Ge, Zachary C. Lipton, and Eric P. Xing. Learning robust global representationsby penalizing local predictive power. In NeurIPS , 2019.

Jiaheng Wei, Hangyu Liu, Tongliang Liu, Gang Niu, Masashi Sugiyama, and Yang Liu. To smooth or not? when label smoothing meets noisy labels. In ICML , pages 23589-23614, 2022.

Xiaobo Xia, Tongliang Liu, Nannan Wang, Bo Han, Chen Gong, Gang Niu, and Masashi Sugiyama. Are anchor points really indispensable in label-noise learning? NeurIPS , pages 6835-6846, 2019.

Hongyi Zhang, Moustapha Cissé, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In ICLR , pages 1-13, 2018.

Lijun Zhang, Tianbao Yang, and Rong Jin. Empirical risk minimization for stochastic convex optimization: $o(1/n)$-and $o(1/nˆ2)$-type of risk bounds. In COLT , volume 65, pages 1954-1979, 2017.

Shengjia Zhao, Abhishek Sinha, Yutong He, Aidan Perreault, Jiaming Song, and Stefano Ermon. Comparing distributions by measuring differences that affect decision making. In ICLR , pages 1-20, 2022.

Zhilin Zhao, Longbing Cao, and Kun-Yu Lin. Revealing the distributional vulnerability of discriminators by implicit generators. IEEE Trans. Pattern Anal. Mach. Intell. , 45(7):8888-8901, 2023.

A Proofs

Section A.1 presents the lemmas used to prove the main results. Section A.2 presents the main results and their detailed proofs.

A.1 Preliminaries

Lemma A.1 ([53], Theorem 26.5) . With probability of at least 1 -δ , for all h ∈ H ,

Lemma A.2 ([53], Theorem 26.5) . With probability of at least 1 -δ , for all h ∈ H ,

Lemma A.3 ([44], Talagrand's contraction lemma) . For any L-Lipschitz loss function l ( · , · ) , we obtain,

Lemma A.4 ([18], Theorem 1) . Let H be the class of real-valued networks of depth D over the domain X . Assume the Frobenius norm of the weight matrices are at most M 1 , . . . , M d . Let the activation function be 1-Lipschitz, positive-homogeneous, and applied element-wise (such as the ReLU). Then,

A.2 Main Results

Theorem 4.1. For any x ∈ X , h ∈ H and a ∈ A , assume we have | l ( h ( x ) , a ( x )) | ≤ c . Then, with probability at least 1 -δ , we can derive the following general bound,

Proof. We observe that

The first two inequalities are owing to the triangle inequality, and the third inequality is due to the definition of L-divergence Eq.(5). We complete the proof by applying Lemma A.1 to bound ∣ ∣ ∣̂ glyph[epsilon1] ̂ p ( ̂ h u ) -glyph[epsilon1] p ( ̂ h u ) ∣ ∣ ∣ and ∣ ∣ ∣ glyph[epsilon1] q ( ̂ h u ) -̂ glyph[epsilon1] ̂ q ( ̂ h u ) ∣ ∣ ∣ in Eq.(7).

Corollary 4.2. Following the conditions of Theorem 4.1, the upper bound of √ Var [ ̂ D R ( ̂ p ‖ ̂ q ) ] is

Proof. For convenience, we assume C = 2 D L ( p ‖ q ) + 2 R l H ( ̂ p ) + 2 R l H ( ̂ q ) . Following to proof of Corollary 1 in [62], we have

where the third inequality is due to

obtained from the results of Theorem 4.1.

Proposition 4.3. Based on the conditions of Theorem 4.1, we assume H is the class of real-valued networks of depth D over the domain X . Let the Frobenius norm of the weight matrices be at most M 1 , . . . , M D , the activation function be 1-Lipschitz, positive-homogeneous and applied element-wise (such as the ReLU). Then, with probability at least 1 -δ , we have,

Proof. We complete the proof by applying Lemma A.3 and Lemma A.4 to bound the Rademacher complexity of deep neural networks in Theorem 4.1.

Corollary 4.4. Following the conditions of Proposition 4.3, with probability at least 1 -δ , we have the following conditional bounds if D L ( p ‖ q ) ≤ ∣ ∣ ∣ glyph[epsilon1] u ( h ∗ u ) -glyph[epsilon1] u ( ̂ h u ) ∣ ∣ ∣ ,

Proof. Following the proof of Theorem 4.1, we have

The first two inequalities are owing to the triangle inequality, and the third inequality is due to the given condition D L ( p ‖ q ) ≤ ∣ ∣ ∣ glyph[epsilon1] u ( h ∗ u ) -glyph[epsilon1] u ( ̂ h u ) ∣ ∣ ∣ . We complete the proof by applying Lemma A.2 to bound ∣ ∣ ∣ glyph[epsilon1] u ( h ∗ ) -glyph[epsilon1] u ( ̂ h ) ∣ ∣ ∣ and Lemma A.1 to bound ∣ ∣ ∣̂ glyph[epsilon1] ̂ p ( ̂ h u ) -glyph[epsilon1] p ( ̂ h u ) ∣ ∣ ∣ and ∣ ∣ ∣ glyph[epsilon1] q ( ̂ h u ) -̂ glyph[epsilon1] ̂ q ( ̂ h u ) ∣ ∣ ∣ in Eq.(7).

Corollary 4.5. Following the conditions of Proposition 4.3, as N →∞ , we have,

Proof. Based on the result on Proposition 4.3, for any δ ∈ (0 , 1) , we know that

when N →∞

. We complete the proof by applying the triangle inequality.

B Algorithm Procedure

B.1 Training Procedure of R-Div

Algorithm 1 Estimate model-oriented distribution discrepancy by R-divergence

Input: two datasets ̂ p and ̂ q and a learning model T with hypothesis space H and loss function l Generate the merged dataset: u = p ∪ q

̂ ̂ ̂ Learn the minimum hypothesis on the mixed data:

Evaluate the empirical risks: glyph[epsilon1] p ( ̂ h u ) and glyph[epsilon1] q ( ̂ h u )

Estimate the R-divergence as the discrepancy:

̂ ̂ ̂ ̂ ̂ ̂

Output: empirical estimator ̂ D R ( ̂ p ‖ ̂ q )

B.2 Calculation Procedure of the Average Test Power

C Compared Methods

Mean embedding (ME) [11] and smooth characteristic functions (SCF) [30] are the state-of-the-art methods using differences in Gaussian mean embeddings at a set of optimized points and frequencies, respectively. Classifier two-sample tests, including C2STS-S [43] and C2ST-L [10], apply the classification accuracy of a binary classifier to distinguish between the two distributions. The binary classifier treats samples from one dataset as positive and the other dataset as negative. These two methods assume that the binary classifier cannot distinguish these two kinds of samples if their distributions are identical. Differently, C2STS-S and C2ST-L apply the test error and the test error gap to evaluate the discrepancy, respectively. MMD-O [20] measures the maximum mean discrepancy (MMD) with a Gaussian Kernel [19], and MMD-D [41] improves the performance of MMD-O by replacing the Gaussian Kernel with a learnable deep kernel. H-Divergence (H-Div) [62] learns optimal hypotheses for the mixture distribution and each individual distribution for the specific model, assuming that the expected risk of training data on the mixture distribution is higher than that on each individual distribution if the two distributions are identical. The equations of these compared methods are presented in Figure 7.

D Additional Experimental Results

D.1 Benchmark Dataset

FME

+SCF

C2STS-S

C2ST-L

MMD-O-MMD-D

R-Div

HDGM

HDGM

1

1

TestPower

Power

Test

0.5

0.5

Average

Average

0

5

10

15

20

0

2000

4000

6000

8000

10000

Dimension ofSamples

NumberofSamples

ME-→SCF-C2STS-S

C2ST-L--MMD-O→-MMD-D--H-Div-

R-Div

Blob (α = 0.05)

1

TestPower

TestPower

0.5

0.5

Average

Average

0

0

0

20

40

60

80

100

0

20

40

60

80

100

NumberofSamples

NumberofSamples

D.2 PACS Dataset

PACS (A + C)

Power

0.8

0.6

e

0.4

Averag

g

0.2

H-Div

R-Div

0

10

15

20

25

30

35

Number of Samples

40

D.3 Learning with Noisy Labels

(a) Accuracy

(b) Detection

(c) Discrepancy

CIFAR10 (Pairflip)

CIFAR10 (Pairflip)

CIFAR10 (Pairflip)

70

80

3×10-3

68

60

2.5

66

Discrepancy

64

40

2

62

60

Retrained (R-Div)

Pretrained

20

Precision

Recall

1.5

58

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.1

0

0.2

0.3

0.4

0.5

0.6

0.7

1

0.1

0.2

0.3

0.4

0.5

0.6

0.7

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nA new divergence metric, called R-divergence, is proposed in the paper. This metric states that two distributions are identical if the optimal hypothesis on the mixture distribution has the same expected risk on each individual distribution.\nIf two distributions are the same, then the optimal hypothesis on the mixture distribution is the same as the one for the individual distributions. This means that the expected risk is similar for each individual distribution. However, if the distributions are different, then the optimal hypothesis on the mixture distribution would behave differently for the individual distributions. This would result in different expected risks for the individual distributions.\nThe main difference between the proposed R-divergence and the existing H-divergence is that R-divergence calculates the minimum hypothesis and the empirical risk on different datasets. This prevents R-divergence from the overfitting issue that H-divergence suffers from.\n\nThe paper also utilizes R-divergence to learn with noisy labels, whereby R-divergence is utilized per batch to estimate the clean and noisy samples, and then the network is retrained with minimizing the cross entropy over the clean batch while maximizing it over the corrupted samples. \n\nThe paper conducts two-sample tests over different datasets and architecture and achieves superior numbers than H-divergence.\n\nSTRENGTHS:\n1. The paper is very well written and is easy to follow. The difference with H-divergence is well highlighted too in the paper. \n\n2. The paper achieves strong results over the baseline divergence metrics over different datasets and with different model architectures.  \n\n3. I find the comparison between ViT and CNN architectures particularly interesting, whereby ViT architectures achieve worse test power than CNN ones, indicating that they are learning shape-biased features which makes it hard to discriminate between CIFAR-10 and corrupted CIFAR-10. \n\n4. Using R-divergence to improve the performance of models under corrupted labels is a nice example of how these divergence metrics could be useful.\n\nWEAKNESSES:\n1. The comparison with CIFAR-10 and corrupted CIFAR-10 is conducted with only one type of corruption when H-divergence considers a variety of corruptions. Can the authors do more detailed experimentation with different corruptions as done in H-divergence.\n\n2. Can the authors also try to do some experiments with ImageNet and its corrupted variants to more justify their proposed method. Since the experiment needs to be done with a limited number of samples, this experiments shouldn't be too computationally difficult.\n\nQUESTIONS:\nI have already mentioned the questions in the weakness section.\n\nLIMITATIONS:\nNone", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes R-divergence as a measure of discrepancy between two distribution. The R-divergence is the difference between the empirical risks computed on two datasets using a hypothesis trained on a mixture of both datasets. The authors use the R-divergence and the permutation test to identify if two datasets are from different distributions and evaluate this method on a number of benchmarks.\n\nSTRENGTHS:\nThe proposed R-divergence is surprisingly simple to compute: it is the difference in the empirical losses on two datasets. The method uses a trained model to define a discrepancy measure between distributions which could also include classifiers or generative models. The authors show that R-divergence can successfully separate data from different distributions and performs favorably when compared to other methods.\n\nWEAKNESSES:\n**Why is R-divergence a good measure of discrepancy?** Methods like the H-divergence [1] are rooted in principled definitions of a *probability divergence* and it satisfies properties like the triangle inequality. On the other hand R-divergence seems to be more of a heuristic and it is unclear why it is a good measure of discrepancy between two probability distributions. The authors argue that it achieves SoTA results on some benchmarks when paired with the permutation test but that does not necessarily qualify it to be a good definition of a divergence.\n\n**Problems with the definition**: The definition seems to be reliant on the choice of the loss function and the hypothesis class. If we consider a very rich hypothesis class, we can find a hypothesis with 0 loss; this can only happen if the loss is 0 for both the distributions. Hence, the R-divergence between any two distributions will be 0 as long as we select a very rich hypothesis class. This can be problematic since the discrepancy can change based on the loss function or hypothesis class.\n\nI worry that many results in the paper heavily depend on such choices. For example, the result on MNIST makes use of a 2-layer fully-connected network (Line 239) and I wonder if the same results will continue to hold if we consider a deep convolutional network. \n\n**Limited insight from theory**: The main theoretical result is that the empirical R-divergence converges to the true R-divergence in the limit of infinite samples. This follows from the uniform convergence of the empirical losses to the true losses . However it is unclear why they converge quickly as claimed in the introduction (Line 73). The theory also offers no insight into why the R-divergence is better than other measures of discrepancy.\n\n**Simple failure case of R-divergence**: Consider two distributions\n$$p_1(x, y) = \\frac{1}{2}\\left( \\delta_{(-1, -1)} + \\delta_{(1, 1)} \\right) \\qquad \\text{and} \\qquad \\frac{1}{2}\\left( \\delta_{(-100, -1)} + \\delta_{(100, 1)} \\right),$$\ndefined on $\\mathbb{R} \\times$ { $-1, 1$}. \nThe two distributions (defined on point masses) can be perfectly separated by a linear hypothesis at 0. Hence, their R-divergence is 0 if we use a linear hypothesis class, even if the two distributions are clearly different. \n\n**Understanding when R-divergence can fail**: The R-divergence requires that the losses evaluated on the two distributions are different which is a heuristic that can only work in certain scenarios. This can occur if there is input or label noise in one of the distributions (like in Table 4, Figure 2). I worry that the authors have over-engineered to a specific set of benchmarks while ignoring many other important scenarios.\n\nFor example one could consider one distribution to be classes 0,1 from MINST and the other distribution to be classes 2,3. Can R-divergence separate the two distributions using a permutation test? Since all samples achieve a similar loss, I would expect the R-divergence to be close to 0.\n\n\n[1] Zhao, Shengjia, et al. \"Comparing distributions by measuring differences that affect decision making.\" International Conference on Learning Representations. 2021.\n\nQUESTIONS:\nMany of my questions have been included in a previous section but are summarized again below:\n\n1. Why is R-divergence a good measure of discrepancy?\n2. Do the results on MNIST hold if we consider a deep convolutional network?\n3. Can we separate out two distributions where one contains classes 0,1 and the other contains classes 2,3 from MNIST (or CIFAR10)?\n4. Do these results rely on the datasets having different scales for the losses and is this common across many datasets?\n\nLIMITATIONS:\nThe authors can expand upon some of the limitations of their method. For example, the authors can include a discussion about how R-divergence depends on the choice of the loss function or the hypothesis class.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes R-divergence to measure the distributional discrepancy of two datasets. The general idea is to learn a minimum hypothesis on the mixed data and then define R-divergence as the difference of the empirical risk of the learned hypothesis between two datasets. The proposed divergence leverages the merits of both H-divergence and L-divergence. Sufficient experiments are performed to demonstrate the superior performance. The case study on learning with label noise is also interesting.\n\nSTRENGTHS:\n1. The method is very easy to follow. Authors also provide valid theoretical analysis (although some results rely heavily on previous literature). \n2. Sufficient experiments in different learning scenarios (e.g., multiple domains, spurious correlations, and noise labels) are performed. \n3. The case study on learning with label noises is interesting. However, a suitable comparison with state-of-the-art methods in this category can enhance the quality of this section.\n\nWEAKNESSES:\n1. It seems to me that R-divergence simply combines the merits of L-divergence and the recently proposed H-divergence. The original L-divergence is intractable since it requires evaluation of all hypotheses in the hypothesis space. By contrast, R-divergence just learns a minimum hypothesis on the mixed data. However, the general idea of defining divergence as the difference of risks on the mixed data and individual datasets is investigated in the H-divergence. \n2. Since R-divergence is model-oriented. How to select hypothesis space and loss functions in practice?\n3. The divergence or distance of distributions is an intrinsic property of two datasets. I can understand that a learning model-based approach provides a new solution. However, the examples in second paragraph of Introduction are hard to convince me why it is more meaningful to measuring divergence by considering a specific model; or why we really need a model-oriented discrepancy measure.\n\nQUESTIONS:\nPlease see weaknesses points 2 and 3. Additionally, according to Table 1, it seems R-div is much better than others in case of limited samples. Can you give more analysis on the reason?\n\nLIMITATIONS:\nAuthors did not discuss potential limitations or potential negative societal impacts. It would be better if authors can discuss How to select hypothesis space and loss functions in practice?", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nA new distance for comparing two samples of two distributuions p and q is proposed that is based on learning a model on the mixtre distribution of p and q. More precisely, the distance is the difference between errors of the learned model evaluated on the individual distributions. Theoretical statements on the estimation quality are given together with empirical evaluations.\n\nSTRENGTHS:\n- Simple method.\n- Empirical results are very good.\n- I could not detect any errors while checking the proofs.\n\nWEAKNESSES:\n- Theoretical upper bound on estimation quality depends on L-divergence between distributions. That is, if $\\mathcal{D}_L(p||q)$ is large, the estimate of the porposed distance might have a very low quality even if the sample size goes to infinity.\n- Citations are sometimes confusing, which makes the main argumentation hard to follow. For example, I cannot find a discussion of an \"overfitting problem of H-divergence\" in [50], as it is noted in the introduction. I also cannot find the reference [63] in [50], although [63] is given as reference of the H-divergence in the paper under review.\n- Theoretical notation/assumptions could be extended, e.g.: What is $\\mathcal{A}$? Does it need to have specific properties so that Theorem 1 is satisfied? What are the properties of the input/output spaces so that the thoerems hold? How do you exactly specify a learning model $\\mathcal{T}$?\n\nQUESTIONS:\n- Do the authors have an idea about the convergence of the empirical estimator towards the true distance for increasing sample size, in the case when the L-distance is large? E.g. is there a lower bound on the left-hand side of Corollary 4.4?\n- What is the effect of regularization of the trained neural networks (e.g., weight decay) on the porposed distance in the experiments?\n- Concerning the main motivation of improving overfitting, is there an empirical evaluation comparing the \"amount of overfitting\" (however this is measured) of H-divergence and L-divergence?\n\nLIMITATIONS:\n- What is a possible effect in the test error, when the upper bounds in the theoretical statements are large, as e.g. in the case of large L-divergence?", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-19"}