Title: Revealing Distribution Discrepancy by Sampling Transfer in Unlabeled Data
Venue: NeurIPS 2024
Accepted: yes
Keywords: Non-IID Data; Distribution Discrepancy; Density Ratio; Likelihood Ratio; Generalization
Abstract: There are increasing cases where the class labels of test samples are unavailable, creating a significant need and challenge in measuring the discrepancy between training and test distributions. This distribution discrepancy complicates the assessment of whether the hypothesis selected by an algorithm on training samples remains applicable to test samples. We present a novel approach called Importance Divergence (I-Div) to address the challenge of test label unavailability, enabling distribution discrepancy evaluation using only training samples. I-Div transfers the sampling patterns from the test distribution to the training distribution by estimating density and likelihood ratios. Specifically, the density ratio, informed by the selected hypothesis, is obtained by minimizing the Kullback-Leibler divergence between the actual and estimated input distributions. Simultaneously, the likelihood ratio is adjusted according to the density ratio by reducing the generalization error of the distribution discrepancy as transformed through the two ratios. Experimentally, I-Div accurately quantifies the distribution discrepancy, as evidenced by a wide range of complex data scenarios and tasks.

Revealing Distribution Discrepancy by Sampling Transfer in Unlabeled Data

Abstract

There are increasing cases where the class labels of test samples are unavailable, creating a significant need and challenge in measuring the discrepancy between training and test distributions. This distribution discrepancy complicates the assessment of whether the hypothesis selected by an algorithm on training samples remains applicable to test samples. We present a novel approach called Importance Divergence (I-Div) to address the challenge of test label unavailability, enabling distribution discrepancy evaluation using only training samples. I-Div transfers the sampling patterns from the test distribution to the training distribution by estimating density and likelihood ratios. Specifically, the density ratio, informed by the selected hypothesis, is obtained by minimizing the Kullback-Leibler divergence between the actual and estimated input distributions. Simultaneously, the likelihood ratio is adjusted according to the density ratio by reducing the generalization error of the distribution discrepancy as transformed through the two ratios. Experimentally, I-Div accurately quantifies the distribution discrepancy, as evidenced by a wide range of complex data scenarios and tasks.

1 Introduction

The assumption that data are independently and identically distributed (IID) is staple in statistical machine learning. It suggests that a hypothesis selected by an algorithm, after observing several training samples, should perform effectively on test samples from the same unknown distribution. However, this assumption often oversimplifies the intricate and diverse nature of real-world data, particularly in non-IID scenarios [1, 2]. Thus, if training samples are considered in-distribution (ID), there is a risk that test samples may deviate from this distribution, characterized as out-of-distribution (OOD) [3]. This distribution discrepancy between training and test distributions poses a critical and challenging non-IID learning [2] question: How to quantify the applicability of a hypothesis derived from training samples to test samples? . This question is generally applicable to areas like OOD generalization [4], OOD detection [5, 6], domain adaptation [7, 8], transfer learning [9], semi-supervised learning [10], robust learning [11], and adversarial learning [12].

The applicability of a hypothesis can be determined by the distribution discrepancy between training and test distributions. When the two distributions align, meaning both training and test samples are ID, the hypothesis derived from training samples can be seamlessly applied to test samples. However, in reality, particularly when test samples fall OOD, this scenario rarely holds. Under such circumstances, decisions must be based on the extent of the distribution discrepancy. This may require enhancing the hypothesis generalization capability towards test samples or rejecting these

samples outright. These actions are fundamental to the principles of OOD generalization and OOD detection, respectively [13].

Evaluating the distribution discrepancy between training and test distributions for a selected hypothesis presents a significant challenge, as training samples are typically labeled, whereas test samples often are not [14]. This discrepancy means that conventional statistical distances, such as F-divergences [15], integral probability metrics [16], and total variation [17], are not suitable for this situation. Furthermore, density ratio methods [18, 19] offer a partial solution by disregarding label information and estimating the ratio between input distributions. Moreover, more performance prediction techniques [20] aim to navigate this challenge by examining the uncertain outcomes of the hypothesis, potentially leading to unreliable results. A detailed discussion of related work is provided in Appendix A.

To tackle the issue of unavailable test labels, we introduce the concept of Importance Divergence (I-Div), which measures the training-test distribution discrepancy w.r.t. the difference between the expected risks of the selected hypothesis on training and test distributions. To estimate the expected risk on test distributions without label access, the core strategy involves importance sampling to transfer the sampling patterns from the test distribution to the training distribution. This process requires the estimation of density and likelihood ratios. Specifically, the density ratio, informed by the selected hypothesis, is obtained by minimizing the Kullback-Leibler divergence between the actual and estimated input distributions. Simultaneously, the likelihood ratio is adjusted according to the density ratio by reducing the generalization error related to the distribution discrepancy as transformed through the two ratios. As a result, I-Div leverages the estimated density and likelihood ratios to quantitatively measure the distribution discrepancy between training and test distributions, eliminating the need for test class labels, and thus quantifying the applicability of the hypothesis across different datasets.

2 Preliminaries

Let X denote the input space, and Y represent the set of labels. The joint probability distributions are represented by P for training samples and Q for test samples. Assume we observe a labeled training dataset ̂ P and a unlabeled test dataset ̂ Q containing N IID samples from P and Q , respectively. Q X denotes the marginal distribution of Q over X . ̂ P and ̂ Q are described as

where the label space Y encompasses all labels of samples from both P and Q . We also define H as the hypothesis space and L ( · , · ) ∈ (0 , B L ) as the bounded loss function. The expected and empirical risks [21] for a hypothesis h ∈ H : X → Y on distribution P and the dataset ̂ P are defined as glyph[epsilon1] P ( h ) and ̂ glyph[epsilon1] ̂ P ( h ) , respectively, i.e.,

where E and ̂ E represent the expectation with respect to a data distribution and the sample average over a dataset, respectively. Accordingly, an algorithm A aims to select the empirical risk minimizer ̂ h ̂ P after observing the samples from the training dataset ̂ P by

to approximate the optimal hypothesis h ∗ P selected from the distribution P through h ∗ P ∈ arg min h ∈H glyph[epsilon1] P ( h ) .

Our central research question is formulated as follows: How can we quantify the applicability of the minimizer ̂ h ̂ P , originated from the training samples ̂ P , to the unlabeled test samples ̂ Q ? To address this question quantitatively, we delve into methodologies for assessing the distribution discrepancy between training and test distributions, particularly focusing on doing so without the need to access class labels from test samples.

3 Importance divergence

For a hypothesis chosen by an algorithm after observing training samples, to assess its applicability to test samples in the absence of ground truth labels, we introduce the concept of Importance Divergence (I-Div). I-Div estimates the distribution discrepancy between training and test distributions for the given hypothesis. To address the issue of unavailable ground truth labels, it leverages importance sampling, density ratios, and likelihood ratios, facilitating the sampling transfer in the test distribution back to the training distribution.

In this section, we first present the distribution discrepancy with importance sampling, which transfers the data sampling from the test distribution to the training distribution. Following this, we discuss the methodologies for estimating the hypothesis-oriented density and adaptive likelihood ratios, essential components of this discrepancy, to facilitate this sampling transfer. The hypothesis-oriented density ratio is specifically tailored to each hypothesis, as it assesses the suitability of a specific hypothesis based on the resulting distribution discrepancy. The adaptive likelihood ratio is adjusted according to the density ratio to expedite the convergence of the distribution discrepancy. Lastly, we utilize an empirical estimator of the distribution discrepancy to evaluate the applicability of a hypothesis selected from training samples to test samples.

3.1 Distribution discrepancy with importance sampling

For the hypothesis ̂ h ̂ P , I-Div evaluates the distribution discrepancy between training and test distributions without requiring the ground truth labels of the test samples. A smaller discrepancy implies that the training and test samples could be considered as drawn from the same distribution with respect to the given hypothesis, and vice versa. By using variational divergence, we can express this discrepancy as the difference between the expected risks of the hypothesis on training and test distributions by

If the training and test distributions are aligned, i.e., P = Q , then the expected risks for the hypothesis ̂ h ̂ P are similar, resulting in a minimal distribution discrepancy. This condition suggests that the test samples are likely ID for the hypothesis ̂ h ̂ P . Conversely, a notable difference between P and Q indicates a larger distribution discrepancy, implying that the hypothesis ̂ h ̂ P perceives the training and test samples as originated from distinct distributions, thus categorizing the test samples as OOD.

To make the expected risk difference in Eq. (4) more pronounced, I-Div employs Jensen's inequality to consider its upper bound by

thereby more distinctly highlighting the differences between the training and test distributions.

Recall that class labels in test samples are inaccessible, thus, the principal challenge is evaluating the expected risk for the test distribution with respective to the given hypothesis without access to ground truth labels. Since direct sampling from the test distribution is not feasible, an alternative is sampling from the training distribution. To overcome this limitation, we employ the importance sampling technique [22], converting the data sampling from test to training distributions, i.e.,

where r and v denote a density ratio and a likelihood ratio, respectively. Thus, I-Div estimates distribution discrepancy between training and test distributions by merely sampling from the training one without accessing the class labels of the test samples. According to Eq. (5) and Eq. (6), we have

Accordingly, for a given labeled training dataset ̂ P and a unlabeled test dataset ̂ Q , we can then construct an empirical estimator for estimating the distribution discrepancy in Eq. (7) by

However, to estimate the distribution discrepancy I-Div, the emerging challenge involves determining the density ratio r and likelihood ratio v from observed samples, which are discussed in Section 3.2 and Section 3.3, respectively.

3.2 Hypothesis-oriented density ratio

The density ratio in the distribution discrepancy should be hypothesis-oriented. That is, it should depend on the specific hypothesis ̂ h ̂ P ∈ H selected by an algorithm A on ̂ P . This is because the criteria for judging the discrepancy between distributions P and Q vary across different algorithms. For instance, whether two datasets of cats and dogs respectively come from the same distribution depends on whether the algorithm aims to identify if the subjects are biological entities or to distinguish between these two species. Accordingly, we apply a deep neural network to model a density ratio r from the hypothesis space R ( ̂ h ̂ P ) depending on the specific hypothesis ̂ h ̂ P . Then, we select a density ratio r ∈ R ( ̂ h ̂ P ) to minimize the Kullback-Leibler divergence between the actual distribution and the estimated distribution based on this density ratio.

To construct the hypothesis space R ( ̂ h ̂ P ) , we utilize the output representations from ̂ h ̂ P with a learnable component ω to model a density ratio r . Specifically, we decompose the hypothesis into a backbone ψ and a softmax layer φ , represented as ̂ h ̂ P ( x ) = ( φ ◦ ψ ) ( x ) , where the output dimension of the backbone is O ψ . Using a learnable component ω , r is constructed as r ( x ) = ( ω ◦ ψ ) ( x ) , where the learnable component ω ( · ) = ( ω SP ◦ ω AD )( · ) contains an adapter ω AD [23] to introduce learnable parameters and a Softplus layer ω SP [24] to ensure strictly positive outputs. Specifically, the adapter ω AD, which follows the bottleneck ψ , comprises two fully connected layers with a Gaussian Error Linear Units (GELU) [25] activation layer in between. Furthermore, the Softplus layer is adopted to the adapter output, effectively mapping it to the range (0 , + ∞ ] . Furthermore, weight matrices are W 1 ∈ W 1 ∈ R O ψ × O m and W 2 ∈ W 2 ∈ R O m × O ω in the fully connected layers. The activation functions GELU and Softplus are β 1 - and β 2 -Lipschitz, respectively. Additionally, we assume τ 1 = sup W 1 ∈V 1 ‖ W 1 ‖ 1 , ∞ and τ 2 = sup W 2 ∈V 2 ‖ W 2 ‖ 1 , ∞ . Thus, the density ratio r ∈ R ( ̂ h ̂ P ) can be modeled as

where O ω represents the output dimensionality of the adapter. Without loss of generality, we further assume that r ( x ) ∈ ( b r , B r ) for any r ∈ R ( ̂ h ̂ P ) and x ∈ X .

To select a density ratio r ∈ R ( ̂ h ̂ P ) , we use it to estimate the density P and Q by

Drawing on the inspiration of importance estimation methods [26, 27], we construct objectives and constraints around ˜ P ( x ) and ˜ Q ( x ) . The estimated probability distributions are designed to approximate their actual counterparts, suggesting the minimization of the two KL divergences with normalization constraints

For convenience, we assume U = P / 2 + Q / 2 and define C = { 1 , -1 } as labels for training and test samples, respectively. A label c ∈ C , corresponding to a sample from U , indicates its distribution origin. The assignment c = 1 indicates that a sample originates from distribution P , while c = -1 signifies that a sample comes from distribution Q . We can then obtain the following objective function for learning the density ratio

where λ ≥ 0 balances the KL divergence and normalization constraints, as the detailed derivation shown in Appendix B. We further assume that f ( r ) is L f -lipschitz continuous with respect to

r ∈ R ( ̂ h ̂ P ) . This objective function Eq. (12) can be estimated by

where ̂ U = ̂ P ∪ ̂ Q . An empirical risk minimizer ̂ r is selected by

which aims to approximate the population risk minimizer r ∈ arg min r ∈R ( ̂ h ̂ P ) f ( r ) . The convergence rate of ̂ r can be guaranteed by the following theorem.

Theorem 3.1. Let λ ≥ 1 and µ be a constant related to the function f . With a probability of at least 1 -δ ,

The presence of N in the denominators of both terms suggests that the bound tightens with an increasing sample size, which aligns with the general understanding that more data can lead to more accurate estimates in statistical learning.

3.3 Adaptive likelihood ratio

Without making further assumptions, it is infeasible to estimate the likelihood ratio v due to its dependence on the unknown joint distribution Q ( x , y ) . Instead of pursuing the true likelihood ratio, our goal is to approximate an adaptive likelihood ratio v ∈ V ( ̂ r ) that enables a swift convergence of the distribution discrepancy, guided by the hypothesis-oriented density ratio ̂ r ∈ R ( ̂ h ̂ P ) . This strategy is valid since the density ratio captures the input distribution discrepancy between training and test distributions. It indicates that utilizing even a basic form of the covariate shift assumption, i.e., v ( x , y ) = 1 for all ( x , y ) ∼ P , allows the distribution discrepancy, as calculated by Eq. (8), to approximate the difference between distributions to a reasonable degree. Furthermore, since the density ratio serves primarily to gauge the distribution discrepancy, the corresponding likelihood ratio must be specifically adapted to this density ratio for precisely assessing the distribution discrepancy.

Accordingly, we reveal the generalization error bound of the distribution discrepancy, leveraging both density and likelihood ratios, through the convergence rate of the hypothesis-oriented density ratio, Rademacher complexity [28] and Talagrand's contraction lemmas [29], which is shown as follows.

Theorem 3.2. Based on the conditions and results outlined in Theorem 3.1, with a probability of at

The result shows that the estimated distribution discrepancy converges quickly with increasing the sample size. Additionally, the bound is associated with the values of the likelihood ratio. Moreover, as per Eq. (6), for any r ∈ R ( ̂ h ̂ P ) , we have

Considering the terms in Theorem 3.2 related to v and the average error over the samples from ̂ P , we have

where γ > 0 acts as a regularization parameter, influencing the trade-off, and the optimal solution is v . By using an proximal algorithm [30], we can obtain the following approximate solution

We observe that in the two boundary cases where γ → 0 and γ → + ∞ , ̂ v consistently equals 1 , thus adhering to the covariate shift assumption. Even under these extreme conditions, it is feasible to calculate the distribution discrepancy using both the hypothesis-oriented density ratio and adaptive likelihood ratio to overcome the challenge of unlabelled test samples. This approach is viable because the hypothesis-oriented density ratio ̂ r quantifies the distribution differences without relying on class labels, while employing γ ∈ (0 , + ∞ ) utilizes the class labels of training samples. Although this method may not precisely determine the likelihood ratio for each instance, it is designed in accordance with a hypothesis-oriented density ratio such that the estimated distribution discrepancy aligns with the actual value, fulfilling our primary objective.

3.4 Hypothesis applicability evaluation

I-Div quantifies the applicability of the hypothesis ̂ h ̂ P selected by the algorithm A from the training dataset ̂ P to the test dataset ̂ Q . Specifically, I-Div employs the empirical estimator in Eq. (8) with hypothesis-oriented density ratio ̂ r in Eq. (14), and adaptive likelihood ratio ̂ v in Eq. (17) to estimate the distribution between training and test distributions by

where ̂ r is chosen from the hypothesis space R ( ̂ h ̂ P ) based on ̂ h ̂ P , and ̂ v is selected from the space V ( ̂ r ) based on ̂ r . A smaller discrepancy indicates that the training and test samples are likely drawn from the same distribution relative to ̂ h ̂ P . Since ̂ h ̂ P minimizes the empirical risk on ̂ P , a reduced distribution discrepancy improves the transferability of the hypothesis from training to test samples. On the other hand, a greater discrepancy suggests a reduced likelihood of hypothesis applicability. The I-Div methodology is detailed in Algorithm 1.

Algorithm 1 Importance divergence

1: Input:

2: - Training samples ̂ P = { ( x i , y i , c i = 1) } N i =1 ∼ P

3: - Test samples ̂ Q = { ( x i , c i = -1) } N i =1 ∼ Q

4: - Empirical minimizer ̂ h ̂ P , Hyperparameters λ and γ

5: Merge datasets: ̂ U = ̂ P ∪ ̂ Q

6: Estimate the hypothesis-oriented density ratio on ̂ U :

7: Estimate the adaptive likelihood ratio on ̂ P :

8: Estimate the distribution discrepancy with importance sampling on ̂ P :

9: Output: empirical estimator ̂ d ( ̂ P , ̂ Q | ̂ h ̂ P , ̂ r, ̂ v )

4 Experimental results

This section presents a comparative analysis of I-Div 2 against existing methods for evaluating the distribution discrepancy between training and test samples. The detailed experimental setups are presented in Appendix D.1.

4.1 Experiments on different classes

Our initial experiments focus on a relatively straightforward task: assessing the applicability of a hypothesis obtained on the training dataset to the test dataset with distinctly different class labels. We utilize two datasets, CIFAR10 [31] and SVHN [32], each comprising ten semantically unique classes. For our experiments, we select samples from one class to serve as the test dataset, with the samples from the remaining nine classes forming the training dataset. This setup clearly illustrates that the knowledge learned in the training dataset cannot be transferred to the test dataset.

The results for CIFAR10 and SVHN are detailed in Table 1 and Table 6 (Appendix D.2), respectively. Our proposed I-Div algorithm consistently achieves perfect scores ( 100% ) in both AUROC and AUPR metrics across all classes of both datasets. This demonstrates its exceptional capability in distinguishing between training and test datasets, aligned with our initial hypothesis. The results unequivocally support the premise that the knowledge transfer from the training to the test datasets is ineffective, as evidenced by the flawless performance of I-Div. This starkly contrasts with the varying effectiveness of other algorithms, including NNBD and MMD-D. Notably, I-Div, MSP, and R-Div all yielded similarly impressive results. A key commonality among these algorithms is their reliance on a specific hypothesis to calculate distribution discrepancy, as opposed to NNBD and MMD-D, which use independent hypotheses. This highlights the significance of considering a particular hypothesis when evaluating distribution discrepancies. The rationale is that the hypothesis applicability depends on the specific design and its intended task.

4.2 Experiments on different datasets

We now turn to a more complex scenario where the training and test datasets may share semantic similarities in class labels, indicating an overlap in the class label spaces. In cases where semantics differ significantly, we expect the algorithm to clearly differentiate the two kinds of samples. Conversely, if their semantics are similar, the algorithm may find it challenging to

distinguish them. This outcome would suggest that the knowledge acquired from the training dataset is transferable to the test dataset, or it may indicate potential pathways to enhance the hypothesis generalization for the test distribution. We leverage CLIP [33] to align class labels between the training and test datasets, using the prompt template 'A photo of a { label } .' This helps adapt class labels across domains and captures semantic relationships between different datasets.

We conduct experiments using classic domain adaptation datasets: PACS [34] and Office-Home [35], each containing four domains. We designate one domain as the training dataset and merge the remaining three as the test dataset. The results, presented in Table 2, indicate that each hypothesis selected from a dataset performs significantly better than randomly selected hypotheses, demonstrating its applicability. Our I-Div algorithm aptly reflects this, in contrast to other algorithms that overly emphasize distribution discrepancies, thereby rigidly categorizing the difference between training and test datasets.

We use CIFAR10 [31] as the training dataset and evaluate on diverse test datasets including Randomly Generated Images (RGI), SVHN [32], DTD [36], Flowers102 [37], OxfordIIITPet [38], SEMEION [39], Caltech256 [40], CIFAR100 [31], CIFAR10.1 [41], and STL10. Since CIFAR10.1 and STL10 share similar category spaces with CIFAR10, the model shows minimal differentiation for these datasets, with ACC values of 73 . 4% and 42 . 1% , respectively. This suggests partial knowledge transferability. However, I-Div demonstrates lower AUROC values of 43 . 7% and 37 . 2% , indicating reduced discrimination. Other algorithms like MSP and R-Div show higher AUROC values, near 100% across all datasets, but these results suggest an overemphasis on distribution discrepancy rather than semantic similarity. I-Div, in contrast, better captures semantic relationships between datasets, providing a more nuanced view of class label semantics.

We use ImageNet [42] as the training dataset and evaluate on diverse test datasets using ResNet50 [43] and ViT-B/16 [44]. The test datasets include the Open Images Dataset v4 (OIDv4) [45], Caltech256 [40], Flowers102 [37], and DTD [36]. The experimental results presented in Table 4 show that the AUROC values of I-Div effectively capture the semantic similarity between ImageNet and the test datasets, yielding results that closely align with human intuition. For example, I-Div demonstrates lower AUROC values for OID and Caltech256, reflecting their semantic overlap with ImageNet, as these datasets share common object categories and scene types. In contrast, datasets such as Flowers102 and DTD, which focus on more specialized object categories and textures, show higher AUROC values with I-Div, indicating greater divergence from ImageNet. On the other hand, algorithms like MSP, NNBD, and MMD-D show consistently high AUROC values across most datasets, implying they emphasize distribution discrepancies over semantic relationships. This limits their effectiveness in distinguishing nuanced semantic differences compared to I-Div, which provides a more human-aligned understanding of dataset relationships.

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

10

20

30

40

50

60

70

80

90

ACC

Noise Added to Test Data

Salt & Pepper

Uniform

Speckle

(a) Classification

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

40

42

44

46

48

50

52

54

56

58

60

AUROC

Noise Added to Test Data

Salt & Pepper

Uniform

Speckle

(b) Discrepancy: HDR

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

99

99.2

99.4

99.6

99.8

100

100.2

100.4

100.6

100.8

101

AUROC

Noise Added to Test Data

Salt & Pepper

Uniform

Speckle

(c) Discrepancy: R-Div

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

60

65

70

75

80

85

90

95

100

AUROC

Noise Added to Test Data

Salt & Pepper

Uniform

Speckle

(d) Discrepancy: I-Div

0.001

0.01

0.1

0.3

Perturbation Magnitude

0

20

40

60

80

100

ACC

(a) Standard Network

0.001

0.01

0.1

0.3

Perturbation Magnitude

0

20

40

60

80

100

AUROC

(b) HDR

0.001

0.01

0.1

0.3

Perturbation Magnitude

0

20

40

60

80

100

AUROC

(c) R-Div

0.001

0.01

0.1

0.3

Perturbation Magnitude

0

20

40

60

80

100

AUROC

(d) I-Div

4.3 Experiments on corrupted data

This section discusses experimental results on corrupted datasets. We progressively introduce noise into a dataset that serves as the training one, treating the resultant corrupted samples as the test dataset. Intuitively, as the noise level increases, the hypothesis performs worse, which indicates the hypothesis becomes less applicable to the corrupted data. To conduct this experiment, CIFAR10 serves as the training dataset, with incremental addition of noises to the original dataset to create the test dataset. The types of noises [46] used include Gaussian, Salt & Pepper, Uniform, and Speckle, with the noise rate increasing from 0 . 1 to 0 . 9 with a 0 . 1 interval. The methods for comparison include Hypothesis-oriented Density Ratio (HDR) in I-Div and R-Div [47]. Fig. 1 presents our experimental findings, showing key performance metrics as influenced by varying noise rates. Notably, the classification performance declines with increasing noise, impacting the hypothesis predictive accuracy. Interestingly, our proposed I-Div algorithm demonstrates robustness against these challenges, with its discrimination power inversely related to the classification accuracy of the standard network in noisy conditions. A brief comparative analysis hints at the superior performance of I-Div over HDR, especially in relation to hypothesis applicability in noisy test datasets. Fig. 4 in Appendix D.3 shows the results when noise is added to the training data instead of the test data. The results are consistent with the above, as the classification accuracy decreases with increasing noise, and I-Div becomes more effective in distinguishing between clean training data and noisy test data. For a comprehensive discussion and full experimental results, please see Appendix D.3.

4.4 Experiments on adversarial data

In this experiment, we delve into a specific scenario involving adversarial samples [48]. We designate one dataset as training and its corresponding adversarial samples as the test dataset. It is a well-known phenomenon that a minimal adversarial perturbation, though visually imperceptible, can drastically alter the classification performance of a network. This suggests potential issues with the direct applicability of the hypothesis selected for a training dataset to a test dataset. However, based on human perception, which fails to distinguish original and adversarial samples visually, we would expect a negligible distribution discrepancy between the distributions of the original and adversarial samples. This outcome could guide us in enhancing network robustness against adversarial attacks and in generalizing the hypothesis to adversarial contexts. For this purpose, we use the CIFAR10 dataset to train a standard network, with adversarial perturbation magnitudes selected from the set { 0 . 001 , 0 . 01 , 0 . 1 , 0 . 3 } . The results in Fig. 2 indicate a marked decrease in standard network

accuracy against adversarial perturbations, while the consistent AUROC of I-Div suggests its limited differentiation capability. Detailed results and analyses are provided in Appendix D.4.

4.5 Experiments with different sample sizes and network architectures

We examine the impact of different sample sizes on the performance of the I-Div algorithm, focusing on its ability to generalize hypotheses from training to test datasets. Fig. 3 show that I-Div tends to maintain low AUROC values for semantically similar datasets like CIFAR10.1 and STL10, indicating effective hypothesis applicability. Conversely, for datasets with significant semantic differences, the performance of I-Div improves with larger sample sizes, highlighting its

500

1000

1500

2000

M

30

40

50

60

70

80

90

100

AUROC

CIFAR10

SVHN

DTD

Fake

Flowers102

OxfordIIITPet

SEMEION

Caltech256

CIFAR100

CIFAR101

STL10

500

1000

1500

2000

M

40

50

60

70

80

90

100

AUPR

CIFAR10

SVHN

DTD

Fake

Flowers102

OxfordIIITPet

SEMEION

Caltech256

CIFAR100

CIFAR101

STL10

capacity to recognize non-transferable knowledge. Additionally, we investigate the effect of varying network architectures as shown in Table 5. Detailed results are provided in Appendix D.5.

5 Limitations

I-Div relies on density and likelihood ratios to achieve the sampling transfer in unlabeled test data, allowing for the estimation of distribution discrepancies between training and test datasets with labeled training samples. Although the density ratio can be accurately estimated using inputs from both training and test samples, the likelihood ratio cannot be estimated precisely due to the unavailability of class labels of test samples. The strategy used by I-Div targets the estimation of distribution discrepancies between training and test distributions. It optimizes a likelihood ratio that adapts to this density ratio to ensure a rapid convergence of the distribution discrepancy, by minimizing the upper bound of the generalization error based on the density ratio. Our future research includes refining the estimation methods for likelihood ratios and exploring distribution discrepancy estimation methods that can bypass the likelihood ratio.

6 Conclusion

In the realm of complex data and machine learning tasks, a crucial question arises regarding the applicability of a hypothesis derived from a training dataset to a test dataset. This uncertainty, especially challenging when test samples lack class labels, significantly determining the hypothesis generalization. To address this, we introduce the I-Div measure for estimating the distribution discrepancy between training and test distributions. I-Div involves the hypothesis-oriented density ratio and adaptive likelihood ratio in expected risk difference to shift the sampling problem from test to training distributions. Experimentally, we validate that I-Div can effectively assess the hypothesis capability of handling test samples, yielding results consistent with prior human knowledge.

Acknowledgments and Disclosure of Funding

This work was supported in part by the Australian Research Council Linkage Grant LP230201022, the Australian Research Council Discovery Grant DP240102050, and the Australian Research Council Linkage Infrastructure, Equipment and Facilities Grant LE240100131.

References

Ilia Nouretdinov, Volodya Vovk, Michael V. Vyugin, and Alex Gammerman. Pattern recognition and density estimation under the general i.i.d. assumption. In COLT , volume 2111, pages 337353, 2001.

Longbing Cao. Beyond i.i.d.: Non-iid thinking, informatics, and learning. IEEE Intell. Syst. , 37(4):5-17, 2022.

Zhilin Zhao, Longbing Cao, and Kun-Yu Lin. Revealing the distributional vulnerability of discriminators by implicit generators. IEEE Trans. Pattern Anal. Mach. Intell. , 45(7):8888-8901, 2023.

David Krueger, Ethan Caballero, J¨ orn-Henrik Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, R´ emi Le Priol, and Aaron C. Courville. Out-of-distribution generalization via risk extrapolation (rex). In ICML , volume 139, pages 5815-5826, 2021.

Zhilin Zhao and Longbing Cao. Dual representation learning for out-of-distribution detection. Trans. Mach. Learn. Res. , pages 1-21, 2023.

Zhen Fang, Yixuan Li, Feng Liu, Bo Han, and Jie Lu. On the learnability of out-of-distribution detection. J. Mach. Learn. Res. , 25:1-84, 2024.

Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, Franc ¸ois Laviolette, Mario Marchand, and Victor S. Lempitsky. Domain-adversarial training of neural networks. J. Mach. Learn. Res. , 17:1-59, 2016.

Kun-Yu Lin, Jiaming Zhou, Yukun Qiu, and Wei-Shi Zheng. Adversarial partial domain adaptation by cycle inconsistency. In ECCV , pages 530-548, 2022.

Karl Weiss, Taghi M Khoshgoftaar, and DingDing Wang. A survey of transfer learning. Journal of Big data , 3(1):1-40, 2016.

Jesper E Van Engelen and Holger H Hoos. A survey on semi-supervised learning. Mach. Learn. , 109(2):373-440, 2020.

Nikola Konstantinov and Christoph Lampert. Robust learning from untrusted sources. In ICML , volume 97, pages 3488-3498, 2019.

Zhuozhuo Tu, Jingwei Zhang, and Dacheng Tao. Theoretical analysis of adversarial learning: A minimax approach. NeurIPS , 32:12259-12269, 2019.

Mohammadreza Salehi, Hossein Mirzaei, Dan Hendrycks, Yixuan Li, Mohammad Hossein Rohban, and Mohammad Sabokrou. A unified survey on anomaly, novelty, open-set, and out of-distribution detection: Solutions and future challenges. Trans. Mach. Learn. Res. , pages 1-81, 2022.

Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, and Chen Change Loy. Domain generalization: A survey. IEEE Trans. Pattern Anal. Mach. Intell. , 45(4):4396-4415, 2023.

Imre Csisz´ ar. Eine informationstheoretische ungleichung und ihre anwendung auf beweis der ergodizitaet von markoffschen ketten. Magyer Tud. Akad. Mat. Kutato Int. Koezl. , 8:85-108, 1964.

Rohit Agrawal and Thibaut Horel. Optimal bounds between f-divergences and integral probability metrics. J. Mach. Learn. Res. , 22:1-128, 2021.

Anuran Makur, Marios Mertzanidis, Alexandros Psomas, and Athina Terzoglou. On the robustness of mechanism design under total variation distance. In NeurIPS , pages 1-10, 2023.

Steffen Bickel, Michael Br¨ uckner, and Tobias Scheffer. Discriminative learning under covariate shift. J. Mach. Learn. Res. , 10:2137-2155, 2009.

Masashi Sugiyama, Shinichi Nakajima, Hisashi Kashima, Paul von B¨ unau, and Motoaki Kawanabe. Direct importance estimation with model selection and its application to covariate shift adaptation. In John C. Platt, Daphne Koller, Yoram Singer, and Sam T. Roweis, editors, NeurIPS , pages 1433-1440, 2007.

Shengjia Zhao, Abhishek Sinha, Yutong He, Aidan Perreault, Jiaming Song, and Stefano Ermon. Comparing distributions by measuring differences that affect decision making. In ICLR , pages 1-20, 2022.

Shai Shalev-Shwartz and Shai Ben-David. Understanding Machine Learning From Theory to Algorithms . Cambridge University Press, 2014.

Yuchen Zhang, Tianle Liu, Mingsheng Long, and Michael I. Jordan. Bridging theory and algorithm for domain adaptation. In ICML , volume 97, pages 7404-7413, 2019.

Taojiannan Yang, Yi Zhu, Yusheng Xie, Aston Zhang, Chen Chen, and Mu Li. AIM: adapting image models for efficient video action recognition. In ICLR , pages 1-18, 2023.

Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In AISTATS , volume 15, pages 315-323, 2011.

Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). CoRR , abs/1606.08415:110, 2016.

Masashi Sugiyama, Taiji Suzuki, and Takafumi Kanamori. Density ratio estimation in machine learning . Cambridge University Press, 2012.

Takafumi Kanamori, Shohei Hido, and Masashi Sugiyama. Efficient direct density ratio estimation for non-stationarity adaptation and outlier detection. In NeurIPS , pages 809-816, 2008.

Peter L. Bartlett and Shahar Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results. J. Mach. Learn. Res. , 3:463-482, 2002.

Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of Machine Learning . MIT Press, 2018.

Neal Parikh, Stephen Boyd, et al. Proximal algorithms. Foundations and trends R © in Optimization , 1(3):127-239, 2014.

Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Toronto, ON, Canada , 2009.

Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading digits in natural images with unsupervised feature learning. In NeurIPS Workshop on Deep Learning and Unsupervised Feature Learning , pages 1-9, 2011.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML , volume 139, pages 8748-8763, 2021.

Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M. Hospedales. Deeper, broader and artier domain generalization. In ICCV , pages 5543-5551, 2017.

Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In CVPR , pages 5018-5027, 2017.

Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In CVPR , pages 3606-3613, 2014.

Maria-Elena Nilsback and Andrew Zisserman. A visual vocabulary for flower classification. In CVPR , pages 1447-1454, 2006.

Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. In CVPR , pages 3498-3505, 2012.

Semeion Handwritten Digit. UCI Machine Learning Repository, 2008.

Greg Griffin, Alex Holub, and Pietro Perona. Caltech-256 object category dataset. Technical report, Technical Report 7694, California Institute of Technology Pasadena, 2007.

Antonio Torralba, Robert Fergus, and William T. Freeman. 80 million tiny images: A large data set for nonparametric object and scene recognition. IEEE Trans. Pattern Anal. Mach. Intell. , 30(11):1958-1970, 2008.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR , pages 248-255, 2009.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR , pages 770-778, 2016.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR , pages 1-21, 2021.

The Open Images Dataset V4. Alina kuznetsova and hassan rom and neil alldrin and jasper r. r. uijlings and ivan krasin and jordi pont-tuset and shahab kamali and stefan popov and matteo malloci and alexander kolesnikov and tom duerig and vittorio ferrari. Int. J. Comput. Vis. , 128(7):1956-1981, 2020.

Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei A. Efros, and Moritz Hardt. Test-time training with self-supervision for generalization under distribution shifts. In ICML , volume 119, pages 9229-9248, 2020.

Zhilin Zhao and Longbing Cao. R-divergence for estimating model-oriented distribution discrepancy. In NeurIPS , pages 1-19, 2023.

Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In ICLR , pages 1-11, 2015.

Kacper Chwialkowski, Aaditya Ramdas, Dino Sejdinovic, and Arthur Gretton. Fast two-sample testing with analytic representations of probability measures. In NeurIPS , pages 1981-1989, 2015.

Wittawat Jitkrittum, Zolt´ an Szab´ o, Kacper P. Chwialkowski, and Arthur Gretton. Interpretable distribution features with maximum testing power. In NeurIPS , pages 181-189, 2016.

Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Sch¨ olkopf, and Alexander J. Smola. A kernel two-sample test. J. Mach. Learn. Res. , 13:723-773, 2012.

Mehmet G¨ onen and Ethem Alpaydin. Multiple kernel learning algorithms. J. Mach. Learn. Res. , 12:2211-2268, 2011.

Feng Liu, Wenkai Xu, Jie Lu, Guangquan Zhang, Arthur Gretton, and Danica J. Sutherland. Learning deep kernels for non-parametric two-sample tests. In ICML , volume 119, pages 6316-6326, 2020.

David Lopez-Paz and Maxime Oquab. Revisiting classifier two-sample tests. In ICLR , pages 1-15, 2017.

Xiuyuan Cheng and Alexander Cloninger. Classification logit two-sample testing by neural networks for differentiating near manifold densities. IEEE Trans. Inf. Theory , 68(10):6631-6662, 2022.

Arthur Gretton, Alex Smola, Jiayuan Huang, Marcel Schmittfull, Karsten Borgwardt, and Bernhard Sch¨ olkopf. Covariate shift by kernel mean matching. 2008.

Takafumi Kanamori, Shohei Hido, and Masashi Sugiyama. A least-squares approach to direct importance estimation. J. Mach. Learn. Res. , 10:1391-1445, 2009.

Masahiro Kato and Takeshi Teshima. Non-negative bregman divergence minimization for deep direct density ratio estimation. In ICML , volume 139, pages 5320-5333, 2021.

Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In ICLR , pages 1-12, 2017.

Saurabh Garg, Sivaraman Balakrishnan, Zachary Chase Lipton, Behnam Neyshabur, and Hanie Sedghi. Leveraging unlabeled data to predict out-of-distribution performance. In ICLR , pages 1-29, 2022.

Christina Baek, Yiding Jiang, Aditi Raghunathan, and J. Zico Kolter. Agreement-on-the-line: Predicting the performance of neural networks under distribution shift. In NeurIPS , pages 1-16, 2022.

Yaodong Yu, Zitong Yang, Alexander Wei, Yi Ma, and Jacob Steinhardt. Predicting outof-distribution error with the projection norm. In ICML , volume 162, pages 25721-25746, 2022.

Elias M Stein. Singular integrals and differentiability properties of functions . Princeton university press, 1970.

Lan V. Truong. On rademacher complexity-based generalization bounds for deep learning. CoRR , abs/2208.04284:1-31, 2022.

Nicolo Cesa-Bianchi and G´ abor Lugosi. Prediction, learning, and games . Cambridge university press, 2006.

Stephen P Boyd and Lieven Vandenberghe. Convex optimization . Cambridge university press, 2004.

Matthias Kirchler, Shahryar Khorasani, Marius Kloft, and Christoph Lippert. Two-sample testing using deep learning. In AISTATS , volume 108, pages 1387-1398, 2020.

Tom Fawcett. An introduction to ROC analysis. Pattern recognition letters , 27(8):861-874, 2006.

Jesse Davis and Mark Goadrich. The relationship between precision-recall and ROC curves. In ICML , volume 148, pages 233-240, 2006.

A Related work

In this section, we briefly review the methods for distribution discrepancy estimation, density ratio estimation, and performance prediction.

A.1 Distribution discrepancy estimation

Mean Embedding (ME) [49] and Smooth Characteristic Functions (SCF) [50] utilize differences in Gaussian mean embeddings at optimized points and frequencies, respectively, to quantify distribution discrepancy. Building on ME, MMD-O [51] calculates Maximum Mean Discrepancy (MMD) using a Gaussian kernel [52], while MMD-D [53] enhances the performance of MMD-O by substituting the Gaussian Kernel with a learnable deep kernel. In contrast, Classifier Two-Sample Tests, including C2ST-S [54] and C2ST-L [55], classify samples from one dataset as positive and from another as negative, leveraging the classification accuracy of a binary classifier to differentiate between them. H-Divergence (H-Div) [20] identifies optimal hypotheses for both the mixture distribution and individual distributions within a specific model, positing that if two distributions are identical, the expected risk on the training samples from the mixture distribution exceeds that from each individual distribution. R-Divergence (R-Div) [47] tackles the overfitting problem of H-Div by suggesting that two distributions are likely identical if their optimal hypotheses yield the same expected risk for each. However, to estimate the discrepancy between training and test distributions, the aforementioned methods assume symmetry between these datasets, meaning either both sets are labeled or both are unlabeled.

A.2 Density ratio estimation

To estimate the density ratio, Kernel Mean Matching (KMM) [56] offers direct estimates of importance at training inputs by efficiently aligning two distributions, leveraging a unique characteristic of universal reproducing kernel Hilbert spaces. The Kullback-Leibler Importance Estimation Procedure (KLIEP) [19] seeks an importance estimate to minimize the Kullback-Leibler divergence from the true test input density to its estimate. Least-Squares Importance Fitting (LSIF) [57] addresses the direct importance estimation issue as a least-squares function fitting problem, transforming the optimization challenge into a convex quadratic program, which is solvable efficiently with standard quadratic programming solvers. Non-Negative Bregman Divergence (NNBD) [58] employs deep neural networks alongside empirical Bregman divergence minimization, addressing the train-loss hacking issue by adjusting the empirical Bregman divergence estimator. Although these methods can estimate the distribution discrepancy between test and training inputs by calculating the density ratio even when class labels for test samples are not available, this implies disregarding the class labels of the training data. Consequently, such density ratios are not suitable for evaluating the applicability of a hypothesis.

A.3 Performance prediction

Maximum over Softmax Probability (MSP) [59] gauges the class label confidence on each sample by a given model to distinguish ID and OOD samples, which can be extend by computing the average class label confidence across both training and test datasets and consider the confidence disparity as indicative of distribution discrepancy. Average Thresholded Confidence (ATC) [60] identifies a threshold such that the proportion of training samples exceeding this confidence threshold aligns with their accuracy, and then it predicts accuracy as the proportion of unlabeled test samples surpassing this threshold. Agreement-on-the-Line (AL) [61] capitalizes on the observation that if the accuracy of models on training samples linearly correlates with their accuracy on test samples, then a similar linear relationship exists between the training and test agreement of model predictions. Projection Norm (PN) [62] employs model predictions to pseudo-label test samples and trains a new model on these pseudo-labels. The variation in parameters between this new model and the original model is utilized to estimate the test error. The aforementioned algorithms estimate the effectiveness of a hypothesis selected from the training samples on test samples based on the confidence output of the hypothesis to judge its applicability. However, since the hypothesis is chosen based on a limited set of training samples, it exhibits significant uncertainty towards unseen test samples, especially when these test samples are OOD, leading to misleading results.

B Objective function for estimating density ratio

According to the KL divergence between P ( x ) and ˜ P ( x ) , we have

Similarly, for the KL divergence between Q ( x ) and ˜ Q ( x ) , we have

Note that the terms KL ( P ( x ) ‖ ˜ P ( x ) ) and KL ( Q ( x ) ‖ ˜ Q ( x ) ) are constants independent of the density ratio r . To normalize ˜ P ( x ) and ˜ Q ( x ) estimated by r , we have ∫ x ∈X ˜ P ( x ) d x = 1 and ∫ x ∈X ˜ Q ( x ) d x = 1 . According to Eq. (10), we have

By considering Eq. (19), Eq. (20) and Eq. (21), we can obtain an optimization problem with constraints for learning a density ratio r , i.e.,

However, solving such a problem with constraints is difficult. Inspired by the method of Lagrange multipliers [30], we can introduce a hyperparameter λ ≥ 0 , which allows us to relax this constraint and balance the loss function of r with the constraint conditions, i.e.,

Accordingly to the Cauchy-Schwarz inequality [63], we have

Applying Eq. (24) and the Jensen's inequality, we have

We obtain the objective function by combining Eq. (23) and Eq. (25).

C Proofs

C.1 Proof of Theorem 3.1

We rewrite f ( r ) by applying the density ratio and obtain

According to Eq. (12), we know that λ is used to ensure the constraints are satisfied as much as possible. Intuitively, its value should be relatively large. More specifically, we provide the following lemma to determine the lower bound of λ to ensure that f ( r ) is strongly convex.

Lemma C.1. f ( r ) is µ -strongly convex if λ ≥ 1 .

Thus, we have

The first inequality arises from the µ -strong convexity of u ( r ( x )) . The second inequality is because ̂ r is the minimizer of ̂ f ( r ) . According to r ( x ) ∈ ( b r , B r ) and f ( r ) is L f -lipschitz continuous on ( b r , B r ) , we know that the absolute value of the difference caused by altering one data point in | f ( r ) -̂ f ( r ) | is bounded by 2 L f B r .

To proceed, we introduce the following two lemmas.

LemmaC.2 ([21], Theorem 26.5) . Assume that ̂ P contains N i.i.d. samples from P and | l ( h, z ) | ≤ c for all z and h ∈ H . With probability of at least 1 -δ , for all h ∈ H ,

where R ( l ◦ H ◦ ̂ P ) is the Rademacher complexity of l ◦ H with respect to ̂ P .

Lemma C.3 ([29], Talagrand's Contraction Lemma) . For any L-Lipschitz loss function l ( · , · ) and hypothesis space H , we obtain

where R ( H◦ ̂ P ) = 1 N E σ ∼{± 1 } N [ sup h ∈H ∑ x ∈ ̂ P σ i h ( x i ) ] with random choice of σ .

Accordingly, with probability of at least 1 -δ , for any r ∈ R , we have

Recall that ψ is a frozen network backbone and ω is a learnable adapter containing two fully connected layers and an activation layer in the middle. The activation function used in the middle and the last layers are GELU and Softplus, respectively. According to the Rademacher bound [64], we have

We complete the proof by combining Jensen's inequality, Eq. (27), Eq. (28) and Eq. (29).

C.2 Proof of Theorem 3.2

Applying the triangle inequality, for and any v ∈ V , we have

To bound B 1 , with probability of at least 1 -δ , we have

where the first inequality follows from the triangle inequality, the second is a consequence of H¨ older's inequality, and the final inequality results from Theorem 3.1. Further, we know

By applying Hoeffding's inequality [65], with probability of at least 1 -δ , we have

We complete the proof by combining Eq. (30), Eq. (31), and Eq. (33).

C.3 Proof of Lemma C.1

The second derivative of the function f ( r ) with respect to r is

We know that

To ensure K ( r ( x )) is strongly convex [66], we can simply assume λ ≥ 1 .

D Experiments

Our experiments are designed to align with established knowledge and intuition. We hypothesize that I-Div will indicate a minimal distribution discrepancy when the training and test distributions share semantically similar class labels, meaning that samples from both distributions can be treated as ID. This suggests the applicability of the hypothesis selected by an algorithm from the training samples to the test samples. Conversely, if class labels significantly differ in semantics, with samples from the training and test distributions being categorized as ID and OOD, respectively, we expect I-Div to reveal a more pronounced distribution discrepancy. This implies that the knowledge learned from the training samples may not be transferable to the test samples.

D.1 Experimental setup

In our study, we explore whether a hypothesis selected from a training dataset retains its capacity when applied to a test dataset. Our quantitative analysis utilizes the I-Div metric to measure the distribution discrepancy between training and test distributions, pertinent to the hypothesis applicability. Unless otherwise specified, we set λ = 1 and γ = 1 . We hypothesize that data with closer semantic relationships will exhibit smaller distribution discrepancies, as opposed to those with distinct semantic differences.

For the given training dataset ̂ P and test dataset ̂ Q , we consider using subsets of these datasets to estimate the distribution discrepancy instead of the entire datasets. This is because using too many samples can trivialize the estimation task and, realistically, obtaining the entire test dataset at once is impractical. Accordingly, we randomly draw samples from ̂ P and ̂ Q , creating a subset of three smaller datasets ( ̂ P ′ , ̂ P ′′ , ̂ Q ′ ) . Adhering to the two-sample test framework [53, 67], each subset consists of M glyph[lessmuch] N samples. We then form positive pairings ( ̂ P ′ , ̂ P ′′ ) and negative pairings ( ̂ P ′ , ̂ Q ′ ) . Unless otherwise noted, our experiments use a standard sample size of M = 1000 . We generate 100 , 000 tuples to achieve significant distribution discrepancy in the positive pairs and minimal discrepancy in the negative pairs.

The effectiveness is evaluated using the Area Under the Receiver Operating Characteristic Curve (AUROC) [68] and the Area Under the Precision-Recall Curve (AUPR) [69], with higher values indicating more effective differentiation between pair types. To illustrate the effectiveness of our I-Div algorithm, we compare it with a selection of representative contrastive algorithms, each adapted to our specific task. All approaches quantitatively evaluate hypothesis applicability by measuring the distribution discrepancy between training and test distributions. These algorithms, which include Maximum over Softmax Probability (MSP) [59], Non-Negative Bregman Divergence (NNBD) [58], Maximum Mean Discrepancy with Deep kernels (MMD-D) [53], and R-Div [47], assess hypothesis applicability through the lens of distribution discrepancy.

D.2 Additional experiments on semantically dissimilar data

D.3 Additional experiments on corrupted data

Fig. 1(a) shows a steady decline in the classification performance of the standard network in the test dataset as the noise rate increases. This decline results from the loss of class label-relevant information in the samples as noises increase, hindering the ability of the hypothesis to predict class labels accurately. The expectation is that the distribution discrepancy between the training and test distributions will increase with added noises, suggesting a decrease of hypothesis applicability from the clean training dataset to the corrupted test dataset without accessing the ground truth labels of test samples. However, the results in Fig. 1(b), Fig. 1(c) and Fig. 1(d) show that the proposed I-Div algorithm performs well in this respect. As the classification accuracy of the standard network in the test samples decreases, the ability of I-Div to discriminate between the two datasets increases. While Fig. 1(c) shows the improving ability of HDR to distinguish the two datasets with an increasing noise

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

0

10

20

30

40

50

60

70

80

90

ACC

Salt & Pepper

Uniform

Speckle

(a) Classification

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

40

50

60

70

80

90

100

AUROC

Salt & Pepper

Uniform

Speckle

(b) Discrepancy: I-Div

rate, its performance is not as consistent as that of I-Div, whose discrimination power more closely aligns with changes in the classification ability of the standard network. Fig. 1(d) indicates that R-Div is extremely sensitive to data variation, achieving a complete separation of the two datasets at any noise level. R-Div can differentiate between clean and corrupted datasets even with minimal noises, but this does not indicate whether the hypothesis derived from training dataset is applicable to the test dataset.

The comparison between Fig. 1(b) and Fig. 1(d) can be seen as an ablation study, since HDR is a component of I-Div. From this analysis, it is clear that I-Div outperforms HDR in determining the hypothesis applicability in the test dataset more accurately. HDR relies only on input data for estimating the input density ratio, a metric unrelated to specific model performance. In contrast, I-Div uses this density ratio to estimate the disparity of hypothesis performance between two datasets, offering a more accurate reflection of hypothesis applicability in the test dataset.

D.4 Additional experiments on adversarial data

As shown in Fig. 2(a), the classification accuracy of a standard network on adversarial samples markedly diminishes with increasing perturbation magnitude. However, as depicted in Fig. 2(d), I-Div maintains a consistently low AUROC, indicating its inability to differentiate between the training and test datasets, akin to human visual perception. This suggests that, even when the classification performance of the standard network is compromised, I-Div still perceives the knowledge from training dataset as transferable to the test dataset. This finding underscores the need to focus on improving the generalization of a standard network to adversarial samples, corroborating our empirical understanding and existing knowledge in the field. Fig. 2(b) presents the AUROC of HDR, which remains high across various perturbation magnitudes, indicating its effectiveness in distinguishing original and adversarial samples. This observation might suggest that, while HDR is responsive to distribution changes induced by adversarial perturbations, it may not accurately reflect the hypothesis applicability from the training to the test datasets. Fig. 2(c) shows the AUROC of R-Div, exhibiting a distinct pattern where the AUROC is relatively lower at the smallest perturbation but improves with larger perturbation. This indicates that R-Div could potentially be used to assess the hypothesis applicability to the test dataset. Nonetheless, in the context of adversarial samples, the results of I-Div imply that enhancing the hypothesis generalization to adversarial samples should be a priority. This conclusion aligns more closely with our intuitive understanding and is supported by numerous advanced research efforts in this area.

D.5 Additional experiments with different sample sizes and network architectures

We explore the influence of varying sample sizes, denoted as M , on the experimental results. This investigation is conducted under the same experimental setup as the one used for the semantically

similar datasets. The findings illustrated in Fig. 3 indicate that for test datasets like CIFAR10.1 and STL10, I-Div consistently exhibits a relatively low AUROC. This pattern suggests that I-Div considers the hypothesis derived from the training dataset to be applicable to these test datasets, a conclusion further supported by the classification results in Table 3. In contrast, for other test datasets with class labels semantically dissimilar to the training dataset, the knowledge learned by the model is deemed non-transferable. As shown in Fig. 3, the ability of I-Div to distinguish these datasets, as quantified by AUROC and AUPR, improves with increasing sample size, eventually reaching a plateau.

Given the hypothesis-oriented nature of the I-Div algorithm, which bases its analysis of hypothesis applicability from training to test datasets, it is pertinent to investigate the impact of varying network architectures on the performance. The findings of these experiments are summarized in Table 5. The data reveal that, across all tested hypotheses, there are consistently low AUROC and AUPR scores for test samples with semantic similarities to the training datasets, such as CIFAR10.1 and STL10. However, a notable variance of performance is observed when examining test datasets with distinct semantic differences, like SVHN and DTD. This highlights the significant role of network architecture in determining a hypothesis capability to generalize and apply learned knowledge to novel, unseen dataset. The robust and consistent performance of I-Div across diverse network architectures further attests to the algorithm stability and broad applicability.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction clearly outline the challenge of evaluating distribution discrepancies between training and test distributions when test labels are unavailable. Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The paper discusses the limitations of the work performed by the authors. It specifically addresses the challenges in estimating likelihood ratios due to the absence of class labels in test samples and outlines future work to refine these methods.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate 'Limitations' section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The paper does provide the full set of assumptions for each theoretical result and includes complete and detailed derivations in the appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The paper fully discloses all the necessary information needed to reproduce the main experimental results, detailing the experimental setup comprehensively.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The paper utilizes publicly available datasets and makes the code publicly available.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so No is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The paper specifies all necessary training and test details, including data splits and experimental parameters, in the appendix, providing sufficient information to understand the results.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: The paper reports error bars or other appropriate measures to indicate the statistical significance of the experimental results, ensuring that the findings are presented with necessary reliability assessments.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer 'Yes' if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: All experiments are conducted using the PyTorch framework on a single 64GB GPU.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The paper adheres to the NeurIPS Code of Ethics, as it does not include any content or practices that violate ethical guidelines, and it ensures that all experimental procedures are appropriately conducted and reported.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [No]

Justification: The paper solely focuses on technical aspects and does not address potential societal impacts, such as ethical considerations or possible misuse of the technology.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA] ,

Justification: The paper does not discuss any specific safeguards related to the responsible release of data or models, particularly those that might have a high risk for misuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The paper properly credits the creators or original owners of all used assets, including data and code.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing experiments or research with human subjects, thus no details about participant instructions, screenshots, or compensation are applicable or required.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve research with human subjects, therefore discussions of potential risks, disclosures to subjects, or the need for Institutional Review Board (IRB) approvals are not applicable.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
The paper proposes a novel concept called Importance Divergence (I-Div) to address the lack of labeling of test samples and to measure the difference between training and testing distributions. Through the computation of importance samples, density ratios, and likelihood ratios, I-Div is able to assess the applicability of hypotheses on different datasets without the need for test labels. The main contribution of this method is that it provides a new way to quantify and reduce the expected risk difference between training and testing distributions, thus enhancing the generalization capability of machine learning models in the face of unknown test data.

STRENGTHS:
The study proposes a novel concept, which helps to estimate the difference between training and testing distributions, and adapts to the assessment of generalization ability between different datasets. This method can be tested for applicability on different datasets and task types, and can be widely used in a variety of machine learning scenarios, especially in practical applications when the training and testing data distributions are inconsistent.

WEAKNESSES:
Since this method relies on accurate estimation of the density ratio and likelihood ratio, this may limit the general feasibility of the method in practical applications.

QUESTIONS:
It is recommended that in future research, more simplified or optimized algorithmic processes be explored to reduce the computational cost and improve the practical feasibility of the method.

LIMITATIONS:
yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-06-21

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the applicability of hypotheses derived from training datasets to distinctly different test datasets through a series of experiments using various datasets, including CIFAR10, SVHN, PACS, and Office-Home. T\n\nSTRENGTHS:\nNew Algorithm Introduction: The paper introduces the I-Div algorithm, which demonstrates a novel approach to evaluating distribution discrepancies between training and test datasets. This innovation contributes to the understanding of hypothesis applicability in machine learning.\n\nDiverse Experimental Scenarios: The study explores various scenarios, including semantically dissimilar data, semantically similar data, corrupted data, and adversarial data. This comprehensive approach offers a fresh perspective on the generalization and robustness of machine learning algorithms.\n\nRobustness and Generalization: The paper’s exploration of the algorithm’s robustness against noise and adversarial attacks, as well as its performance across different sample sizes and network architectures, adds depth to the quality of the research.\n\nWEAKNESSES:\nOutdated Baselines: The baseline comparison, particularly using MMD-D from 2020, is quite outdated. This diminishes the paper's novelty and effectiveness compared to state-of-the-art methods. To enhance the paper's relevance and credibility, it would be beneficial to include more recent benchmark algorithms, preferably those introduced after 2021, or popular models like ViT (Vision Transformer) and CLIP (Contrastive Language-Image Pre-training), which have shown significant impact in computer vision.\n\n\nBackbone and Feature Encoder: Indeed, the choice of backbone in the paper does not align well with recent developments in computer vision. Modern visual tasks increasingly rely on Transformer-based architectures such as ViT and CLIP, which excel in handling interactions between visual and language modalities. Introducing these modern feature encoders in the experiments would allow the algorithm to be tested on more complex and diverse datasets, better reflecting its generalization ability and practical utility.\n\n\nPerformance of I-DIV in Table 1: I concern about I-DIV achieving 100% performance in Table 1 is valid. In practical applications, achieving 100% accuracy is nearly impossible, especially on complex datasets and real-world scenarios. This might suggest some idealization or overfitting in the experimental setup, particularly in how the model or algorithm is tailored to specific datasets or test conditions. Therefore, it's advisable to provide a more detailed explanation of these experimental results in the paper, discussing their applicability and generalization capability in real-world environments.\n\nQUESTIONS:\nsee weaknesses\n\nLIMITATIONS:\nsee weaknesses", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a novel approach called Importance Divergence (I-Div) to address the challenge of measuring the discrepancy between training and test distributions when test sample class labels are unavailable. I-Div transfers sampling patterns from the test distribution to the training distribution by estimating density and likelihood ratios. The density ratio is obtained by minimizing the Kullback-Leibler divergence, while the likelihood ratio is adjusted to reduce the generalization error. Experimentally, I-Div accurately quantifies distribution discrepancy across a wide range of complex data scenarios and tasks.\n\nSTRENGTHS:\n1. **Novel Approach**: I-Div offers a novel method to evaluate distribution discrepancies between training and test data without needing test sample class labels, addressing a significant challenge existing methods struggle with.\n\n2. **Enhanced Generalization Capability**: By transferring sampling patterns and adjusting ratios, I-Div enhances the model's generalization capability, which is particularly beneficial for applications such as out-of-distribution (OOD) sample detection.\n\n3. **Validation**: The paper demonstrates the effectiveness of I-Div through various datasets and scenarios, showing that the proposed method works well across different conditions.\n\nWEAKNESSES:\n1. **Limitations of Likelihood Ratio Estimation**: As the authors mention, accurately estimating the likelihood ratio is challenging since test sample class labels are unavailable. The proposed adaptive adjustment only partially mitigates this issue and may not fully resolve it.\n\n2. **Computational Complexity**: Estimating the density and likelihood ratios can be computationally expensive and potentially impractical for large datasets or complex models.\n\n3. **Limited Scope of Experiments**: The experiments still focus primarily on simple datasets. Additional validation on other types of datasets is necessary to confirm the generalizability of I-Div.\n\nQUESTIONS:\nThis paper presents a methodological advancement in evaluating distribution discrepancies without test sample class labels. However, the fundamental challenge of accurately estimating the likelihood ratio remains unresolved, potentially compromising the method's accuracy. Additionally, the computational complexity may hinder practical application, especially with large or complex datasets.\n\n1. Since the Likelihood Ratio significantly influences Importance Sampling, it greatly affects the quality of the samples. Consequently, this often results in high variance in the outcomes. I am curious about how the author addressed this stability issue during the implementation.\nFor example, Fig. 3 illustrates the effect of sample size but does not explain which factor is crucial to controlling the quality of the samples. And the size > 1000 means it requires high computational costs. So please elaborate on some more challenges on this matter.\n\n2. What challenges might be faced in extreme cases, such as when the Adaptive Likelihood Ratio is large? While proving convergence is important, discussing practical implementation difficulties, such as when the test dataset size is relatively bigger or when many classes exist, would improve this paper.\n\nLIMITATIONS:\nI believe this paper does not have any negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a discrepancy to measure the difference between two distributions within a common scenario: the labeled training set distribution and the unlabeled test set distribution. This discrepancy arises from the expected risk difference between these two distributions, considering a model pre-trained on the training samples, and involves the estimation of both the density ratio and the likelihood ratio. Experiments were conducted on different types of data splits and data corruptions to demonstrate its effectiveness.\n\nSTRENGTHS:\n1. The scenario considered for this discrepancy is very practical.\n2. The definition of discrepancy is intuitive, and the paper is well-written.\n3. The experiments demonstrate the robustness of the discrepancy across various scenarios, providing solid evidence of its effectiveness.\n\nWEAKNESSES:\n1. My primary concern revolves around the significant impact of the pre-trained classifier $\\hat{h}$ on the proposed I-Div. After obtaining the pre-trained model from the training set, the estimated density ratio is predicted by appending a branch to the last layer of this classifier, and the subsequent Adaptive Likelihood Ratio is further estimated based on the predicted $\\hat{r}(x)$. Therefore, I am concerned that inaccuracies in the initial estimation by $\\hat{h}$ may amplify errors in the final discrepancy. (Although Theorem 3.2 provides a bound, which measures the upper limit of the distance within the same classifier.) I am unsure whether this could limit the practical effectiveness of the proposed I-Div.\n   \n2. The compared methods are not presented under consistent settings regarding label utilization and network parameterization. Aligning these settings could enhance clarity for comparison, although direct application to the scenario proposed by the authors might not be feasible.\n\nQUESTIONS:\n1. Could the authors provide a more detailed explanation of how Eq.16 is derived?\n2. In Figure 3, what is the sample size of the data in the first column, and why does it perform better than the sample size of 500?\n3. Considering my first concern in Cons, what would happen if we were to swap the training set and testing set in the Corrupted Data Experiment, or if the training set itself is inherently challenging?\n4. Experiment 4.4 is intriguing; the proposed I-Div appears to be completely unaffected by adversarial samples. Could you provide a more insightful explanation?\n5. It would be beneficial to analyze the sensitivity of hyperparameters （e.g. $\\sigma$ and $\\gamma$）\n\nLIMITATIONS:\nThe proposed I-DIV, rooted in expectation risk, bears similarities to existing measures like R-Div. And due to the unknown labels of the test distribution, the authors propose to estimate the density ratio and likelihood ratio, which introduces uncontrollable errors, especially in complex distributions. Thus, I-DIV is affected not only by the estimation error of the likelihood ratio, as noted by the authors, but also the empirical minimizer, which is difficult to mitigate. I believe this limitation somewhat restricts the practical effectiveness of I-DIV.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-18"}